Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(exp): add LongMemEval and LoCoMo OpenViking benchmarks (#1937)
agent8a3f12d17414
diff --git a/benchmark/locomo/openviking/README.md b/benchmark/locomo/openviking/README.mdnew file mode 100644index 000000000..0e15a6e07--- /dev/null+++ b/benchmark/locomo/openviking/README.md@@ -0,0 +1,93 @@+# LoCoMo OpenViking Benchmark++This directory contains the OpenViking evaluation flow for LoCoMo:++1. import each conversation into an isolated OpenViking user space;+2. run one retrieval call per question;+3. optionally rerank the retrieved memories;+4. answer from the selected memory context;+5. judge and summarize the result CSV.++The benchmark expects an OpenViking server to already be running. The commands+below use the default local OpenViking client configuration. If you need a+different endpoint, pass `--openviking-url` to both import and eval.++## Data++Use the prepared LoCoMo JSON file:++```bash+DATA=result/locomo.json+```++LoCoMo samples are imported under `viking://user/sample_{idx}/memories`.+Evaluation uses the same `sample_{idx}` user id, so import and eval must use the+same dataset order.++## Import++```bash+python benchmark/locomo/openviking/import_to_ov.py \+ --input "$DATA" \+ --parallel-samples 16 \+ --success-csv result/locomo_openviking_import_success.csv \+ --error-log result/locomo_openviking_import_errors.log+```++Use `--force-ingest` only when intentionally re-importing existing samples.++## Smoke Test++Run one question before a full evaluation:++```bash+python benchmark/locomo/openviking/run_eval.py "$DATA" \+ --output result/locomo_openviking_smoke.csv \+ --sample 0 \+ --question-index 0 \+ --threads 1 \+ --single-search-context-limit 50 \+ --single-search-rerank-limit 10 \+ --single-search-max-context-chars 30000 \+ --debug-print-model-input+```++With `--debug-print-model-input`, the CSV includes the full answer prompt and+retrieval trace. Check `retrieved_uris_by_iteration` to confirm rerank is enabled+and `context_uris` contains the expected number of memories.++## Full Eval++This example retrieves 50 memories, keeps the top 10 after rerank, and caps the+memory text passed to the answer model at 30000 characters.++```bash+OUT=result/locomo_openviking_search50_rerank10_chars30000.csv++python benchmark/locomo/openviking/run_eval.py "$DATA" \+ --output "$OUT" \+ --threads 8 \+ --single-search-context-limit 50 \+ --single-search-rerank-limit 10 \+ --single-search-max-context-chars 30000+```++Set `--single-search-rerank-limit 0` to disable rerank. Set+`--single-search-max-context-chars 0` to disable the character budget.++## Judge And Stat++```bash+python benchmark/locomo/openviking/judge.py \+ --input "$OUT" \+ --parallel 40++python benchmark/locomo/openviking/stat_judge_result.py \+ --input "$OUT"+```++The judge writes `result` values back into the same CSV. Use `--force` to+re-grade rows that already have a result. Use `--strict-prompt` when you want the+stricter LoCoMo judge prompt instead of the default lenient prompt.++Category 5 adversarial questions are skipped by the LoCoMo judge/stat flow.diff --git a/benchmark/locomo/openviking/import_to_ov.py b/benchmark/locomo/openviking/import_to_ov.pynew file mode 100644index 000000000..20752ab2e--- /dev/null+++ b/benchmark/locomo/openviking/import_to_ov.py@@ -0,0 +1,804 @@+"""+OpenViking data import tool.++Import conversations from LoCoMo JSON or plain text files into OpenViking memory.++Usage:+ # Import LoCoMo JSON conversations+ uv run python import_to_ov.py locomo10.json --sample 0 --sessions 1-4++ # Import plain text conversations+ uv run python import_to_ov.py example.txt+"""++import argparse+import asyncio+import csv+import json+import sys+import time+import traceback+from datetime import datetime, timedelta+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++import openviking as ov++LOCOMO_SESSION_CHUNK_SIZE = 8+++def _get_session_number(session_key: str) -> int:+ """Extract session number from session key."""+ return int(session_key.split("_")[1])+++def parse_test_file(path: str) -> List[Dict[str, Any]]:+ """Parse txt test file into sessions.++ Each session is a dict with:+ - messages: list of user message strings+ """+ with open(path, "r", encoding="utf-8") as f:+ content = f.read()++ raw_sessions = content.split("---\n")+ sessions = []++ for raw in raw_sessions:+ lines = [line for line in raw.strip().splitlines() if line.strip()]+ if not lines:+ continue++ messages = []+ for line in lines:+ if not line.startswith("eval:"): # Skip eval lines+ messages.append(line)++ if messages:+ sessions.append({"messages": messages})++ return sessions+++def load_locomo_data(+ path: str,+ sample_index: Optional[int] = None,+) -> List[Dict[str, Any]]:+ """Load LoCoMo JSON and optionally filter to one sample."""+ with open(path, "r", encoding="utf-8") as f:+ data = json.load(f)++ def normalize_sample(item: Dict[str, Any], index: int) -> Dict[str, Any]:+ normalized = dict(item)+ normalized["original_sample_id"] = item.get("sample_id", "")+ normalized["sample_id"] = f"sample_{index}"+ return normalized++ if sample_index is not None:+ if sample_index < 0 or sample_index >= len(data):+ raise ValueError(f"Sample index {sample_index} out of range (0-{len(data) - 1})")+ return [normalize_sample(data[sample_index], sample_index)]+ return [normalize_sample(item, idx) for idx, item in enumerate(data)]+++def build_session_messages(+ item: Dict[str, Any],+ session_range: Optional[Tuple[int, int]] = None,+) -> List[Dict[str, Any]]:+ """Build session messages for one LoCoMo sample.++ Returns list of dicts with keys: messages, meta.+ Each dict represents a session with multiple messages (user/assistant role).+ """+ conv = item["conversation"]+ speaker_a = conv["speaker_a"]+ speaker_b = conv["speaker_b"]+ speakers = f"{speaker_a} & {speaker_b}"++ session_keys = sorted(+ [k for k in conv if k.startswith("session_") and not k.endswith("_date_time")],+ key=_get_session_number,+ )++ sessions = []+ for sk in session_keys:+ sess_num = _get_session_number(sk)+ if session_range:+ lo, hi = session_range+ if sess_num < lo or sess_num > hi:+ continue++ dt_key = f"{sk}_date_time"+ date_time = conv.get(dt_key, "")++ messages = []+ for idx, msg in enumerate(conv[sk]):+ speaker = msg.get("speaker", "unknown")+ text = msg.get("text", "")+ if speaker == speaker_a:+ role = "user"+ elif speaker == speaker_b:+ role = "assistant"+ else:+ role = "user"+ messages.append(+ {+ "role": role,+ "text": f"[{speaker}]: {text}",+ "speaker": speaker,+ "index": idx,+ "img_url": msg.get("img_url", []),+ "blip_caption": msg.get("blip_caption", ""),+ }+ )++ sessions.append(+ {+ "messages": messages,+ "meta": {+ "sample_id": item["sample_id"],+ "session_key": sk,+ "date_time": date_time,+ "speakers": speakers,+ },+ }+ )++ return sessions+++def chunk_messages(+ messages: List[Dict[str, Any]], chunk_size: int = LOCOMO_SESSION_CHUNK_SIZE+) -> List[List[Dict[str, Any]]]:+ """Split a session message list into smaller contiguous chunks."""+ if chunk_size <= 0:+ raise ValueError("chunk_size must be positive")+ return [+ messages[i : i + chunk_size]+ for i in range(0, len(messages), chunk_size)+ if messages[i : i + chunk_size]+ ]+++def _normalize_image_urls(value: Any) -> List[str]:+ """Normalize LoCoMo img_url values into a list of non-empty URL strings."""+ if not value:+ return []+ if isinstance(value, str):+ return [value] if value else []+ if isinstance(value, list):+ return [str(url) for url in value if url]+ return [str(value)]+++def build_openviking_message_parts(msg: Dict[str, Any]) -> List[Dict[str, Any]]:+ """Build OpenViking message parts from a LoCoMo message."""+ parts: List[Dict[str, Any]] = [{"type": "text", "text": msg["text"]}]+ caption = str(msg.get("blip_caption", "") or "")++ for image_url in _normalize_image_urls(msg.get("img_url")):+ image_part: Dict[str, Any] = {+ "type": "image_url",+ "image_url": {"url": image_url},+ }+ if caption:+ image_part["caption"] = caption+ parts.append(image_part)++ return parts+++# ---------------------------------------------------------------------------+# Ingest record helpers (avoid duplicate ingestion)+# ---------------------------------------------------------------------------+++def load_success_csv(csv_path: str = "./result/import_success.csv") -> set:+ """加载成功导入的CSV记录,返回已成功的键集合"""+ success_keys = set()+ if Path(csv_path).exists():+ with open(csv_path, "r", encoding="utf-8") as f:+ reader = csv.DictReader(f)+ for row in reader:+ key = f"viking:{row['sample_id']}:{row['session']}"+ success_keys.add(key)+ return success_keys+++def write_success_record(+ record: Dict[str, Any], csv_path: str = "./result/import_success.csv"+) -> None:+ """写入成功记录到CSV文件"""+ file_exists = Path(csv_path).exists()+ fieldnames = [+ "timestamp",+ "sample_id",+ "session",+ "date_time",+ "speakers",+ "embedding_tokens",+ "vlm_tokens",+ "llm_input_tokens",+ "llm_output_tokens",+ "total_tokens",+ ]++ with open(csv_path, "a", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=fieldnames)+ if not file_exists:+ writer.writeheader()++ writer.writerow(+ {+ "timestamp": record["timestamp"],+ "sample_id": record["sample_id"],+ "session": record["session"],+ "date_time": record.get("meta", {}).get("date_time", ""),+ "speakers": record.get("meta", {}).get("speakers", ""),+ "embedding_tokens": record["token_usage"].get("embedding", 0),+ "vlm_tokens": record["token_usage"].get("vlm", 0),+ "llm_input_tokens": record["token_usage"].get("llm_input", 0),+ "llm_output_tokens": record["token_usage"].get("llm_output", 0),+ "total_tokens": record["token_usage"].get("total", 0),+ }+ )+++def write_error_record(+ record: Dict[str, Any], error_path: str = "./result/import_errors.log"+) -> None:+ """写入错误记录到日志文件"""+ with open(error_path, "a", encoding="utf-8") as f:+ timestamp = record["timestamp"]+ sample_id = record["sample_id"]+ session = record["session"]+ error = record["error"]+ f.write(f"[{timestamp}] ERROR [{sample_id}/{session}]: {error}\n")+++def load_ingest_record(record_path: str = "./result/.ingest_record.json") -> Dict[str, Any]:+ """Load existing ingest record file, return empty dict if not exists."""+ try:+ with open(record_path, "r", encoding="utf-8") as f:+ return json.load(f)+ except (FileNotFoundError, json.JSONDecodeError):+ return {}+++def save_ingest_record(+ record: Dict[str, Any], record_path: str = "./result/.ingest_record.json"+) -> None:+ """Save ingest record to file."""+ with open(record_path, "w", encoding="utf-8") as f:+ json.dump(record, f, indent=2, ensure_ascii=False)+++def is_already_ingested(+ sample_id: str | int,+ session_key: str,+ record: Dict[str, Any],+ success_keys: Optional[set] = None,+) -> bool:+ """Check if a specific session has already been successfully ingested."""+ key = f"viking:{sample_id}:{session_key}"+ if success_keys is not None and key in success_keys:+ return True+ return key in record and record[key].get("success", False)+++def mark_ingested(+ sample_id: str | int,+ session_key: str,+ record: Dict[str, Any],+ meta: Optional[Dict[str, Any]] = None,+) -> None:+ """Mark a session as successfully ingested."""+ key = f"viking:{sample_id}:{session_key}"+ record[key] = {+ "success": True,+ "timestamp": int(time.time()),+ "meta": meta or {},+ }+++# ---------------------------------------------------------------------------+# OpenViking import+# ---------------------------------------------------------------------------+def _parse_token_usage(commit_result: Dict[str, Any]) -> Dict[str, int]:+ """解析Token使用数据(从commit返回的telemetry或task result中提取)"""+ # 尝试从 task result 中提取(task 完成后包含完整 token_usage)+ if "result" in commit_result:+ result = commit_result["result"]+ if "token_usage" in result:+ tu = result["token_usage"]+ embedding = tu.get("embedding", {})+ llm = tu.get("llm", {})+ # embedding 格式可能是 {"total": N} 或 {"total_tokens": N}+ embed_total = embedding.get("total", embedding.get("total_tokens", 0))+ llm_total = llm.get("total", llm.get("total_tokens", 0))+ return {+ "embedding": embed_total,+ "vlm": llm_total,+ "llm_input": llm.get("input", 0),+ "llm_output": llm.get("output", 0),+ "total": tu.get("total", {}).get("total_tokens", embed_total + llm_total),+ }++ # 从 commit 响应的 telemetry 中提取+ telemetry = commit_result.get("telemetry", {}).get("summary", {})+ tokens = telemetry.get("tokens", {})+ return {+ "embedding": tokens.get("embedding", {}).get("total", 0),+ "vlm": tokens.get("llm", {}).get("total", 0),+ "llm_input": tokens.get("llm", {}).get("input", 0),+ "llm_output": tokens.get("llm", {}).get("output", 0),+ "total": tokens.get("total", 0),+ }+++async def viking_ingest(+ messages: List[Dict[str, Any]],+ openviking_url: str,+ session_time: Optional[str] = None,+ user_id: Optional[str] = None,+ agent_id: Optional[str] = None,+) -> Dict[str, int]:+ """Save messages to OpenViking via OpenViking SDK client.+ Returns token usage dict with embedding and vlm token counts.++ Args:+ messages: List of message dicts with role and text+ openviking_url: OpenViking service URL+ session_time: Session time string (e.g., "9:36 am on 2 April, 2023")+ user_id: User identifier for separate userspace (e.g., "sample_26")+ agent_id: Agent identifier for separate agentspace (e.g., "sample_26")+ """+ # 解析 session_time - 为每条消息计算递增的时间戳+ base_datetime = None+ if session_time:+ try:+ base_datetime = datetime.strptime(session_time, "%I:%M %p on %d %B, %Y")+ except ValueError:+ print(f"Warning: Failed to parse session_time: {session_time}", file=sys.stderr)++ # Create client+ client = ov.AsyncHTTPClient(+ url=openviking_url,+ user=user_id,+ )+ await client.initialize()++ try:+ # Create session+ create_res = await client.create_session()+ session_id = create_res["session_id"]++ # Add messages one by one with created_at+ for idx, msg in enumerate(messages):+ msg_created_at = None+ if base_datetime:+ # 每条消息递增1秒,确保时间顺序+ msg_dt = base_datetime + timedelta(seconds=idx)+ msg_created_at = msg_dt.isoformat()++ await client.add_message(+ session_id=session_id,+ role=msg["role"],+ parts=build_openviking_message_parts(msg),+ created_at=msg_created_at,+ )++ # Commit+ result = await client.commit_session(session_id, telemetry=True)++ # Accept both "committed" and "accepted" as success - accepted means the session was archived+ if result.get("status") not in ("committed", "accepted"):+ raise RuntimeError(f"Commit failed: {result}")++ # 等待 task 完成以获取准确 token 消耗+ task_id = result.get("task_id")+ if task_id:+ # 轮询任务状态直到完成+ max_attempts = 1200 # 最多等待20分钟+ for _attempt in range(max_attempts):+ task = await client.get_task(task_id)+ status = task.get("status") if task else "unknown"+ if status == "completed":+ token_usage = _parse_token_usage(task)+ break+ elif status in ("failed", "cancelled", "unknown"):+ raise RuntimeError(f"Task {task_id} {status}: {task}")+ await asyncio.sleep(1)+ else:+ raise RuntimeError(f"Task {task_id} timed out after {max_attempts} attempts")+ else:+ token_usage = {"embedding": 0, "vlm": 0, "total": 0}++ # Get trace_id from commit result+ trace_id = result.get("trace_id", "")+ return {"token_usage": token_usage, "task_id": task_id, "trace_id": trace_id}++ finally:+ await client.close()+++def parse_session_range(s: str) -> Tuple[int, int]:+ """Parse '1-4' or '3' into (lo, hi) inclusive tuple."""+ if "-" in s:+ lo, hi = s.split("-", 1)+ return int(lo), int(hi)+ n = int(s)+ return n, n+++async def process_single_session(+ messages: List[Dict[str, Any]],+ sample_id: str | int,+ session_key: str,+ meta: Dict[str, Any],+ run_time: str,+ ingest_record: Dict[str, Any],+ args: argparse.Namespace,+) -> Dict[str, Any]:+ """处理单个会话的导入任务"""+ try:+ chunks = chunk_messages(messages)+ token_usage = {"embedding": 0, "vlm": 0, "llm_input": 0, "llm_output": 0, "total": 0}+ chunk_task_ids: List[str] = []+ chunk_trace_ids: List[str] = []++ # 使用 sample_id 作为 user_id 和 agent_id,实现独立的 userspace/agentspace+ for chunk in chunks:+ result = await viking_ingest(+ chunk,+ args.openviking_url,+ meta.get("date_time"),+ user_id=str(sample_id),+ agent_id=str(sample_id),+ )+ chunk_usage = result["token_usage"]+ for key in token_usage:+ token_usage[key] += int(chunk_usage.get(key, 0) or 0)++ task_id = result.get("task_id")+ trace_id = result.get("trace_id", "")+ if task_id:+ chunk_task_ids.append(str(task_id))+ if trace_id:+ chunk_trace_ids.append(str(trace_id))++ embedding_tokens = token_usage.get("embedding", 0)+ vlm_tokens = token_usage.get("vlm", 0)+ print(+ f" -> [COMPLETED] [{sample_id}/{session_key}] chunks={len(chunks)}, embed={embedding_tokens}, vlm={vlm_tokens}, task_ids={chunk_task_ids}, trace_ids={chunk_trace_ids}",+ file=sys.stderr,+ )++ # Write success record+ result = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "success",+ "meta": meta,+ "token_usage": token_usage,+ "embedding_tokens": embedding_tokens,+ "vlm_tokens": vlm_tokens,+ "task_ids": chunk_task_ids,+ "trace_ids": chunk_trace_ids,+ }++ # 写入成功CSV+ write_success_record(result, args.success_csv)++ # Mark as successfully ingested+ mark_ingested(sample_id, session_key, ingest_record, meta)+ save_ingest_record(ingest_record) # Save immediately after success++ return result++ except Exception as e:+ print(f" -> [ERROR] [{sample_id}/{session_key}] {e}", file=sys.stderr)+ traceback.print_exc(file=sys.stderr)++ # Write error record+ result = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "error",+ "error": str(e),+ }++ # 写入错误日志+ write_error_record(result, args.error_log)++ return result+++async def run_import(args: argparse.Namespace) -> None:+ session_range = parse_session_range(args.sessions) if args.sessions else None++ # 如果指定了 question-index,自动从 evidence 推断需要的 session+ if args.question_index is not None and not args.sessions:+ # 加载数据获取 question 的 evidence+ with open(args.input, "r", encoding="utf-8") as f:+ data = json.load(f)++ # 获取 sample+ sample_idx = args.sample if args.sample is not None else 0+ if sample_idx < 0 or sample_idx >= len(data):+ raise ValueError(f"sample index {sample_idx} out of range")+ sample = data[sample_idx]++ # 获取 question 的 evidence+ qa_items = sample.get("qa", [])+ if args.question_index < 0 or args.question_index >= len(qa_items):+ raise ValueError(f"question index {args.question_index} out of range")+ qa = qa_items[args.question_index]+ evidence_list = qa.get("evidence", [])++ # 从 evidence 提取 session 号 (D1:3 -> session 1)+ session_nums = set()+ for ev in evidence_list:+ try:+ # D1:3 -> session 1+ sess_num = int(ev.split(":")[0][1:])+ session_nums.add(sess_num)+ except (ValueError, IndexError):+ pass++ if session_nums:+ min_sess = min(session_nums)+ max_sess = max(session_nums)+ session_range = (min_sess, max_sess)+ print(+ f"[INFO] Auto-detected sessions from evidence: {min_sess}-{max_sess}",+ file=sys.stderr,+ )++ # Handle ingest record operations+ if args.clear_ingest_record:+ ingest_record = {}+ save_ingest_record(ingest_record)+ print("[INFO] All existing ingest records cleared", file=sys.stderr)+ else:+ ingest_record = load_ingest_record()++ # 加载成功CSV记录用于去重+ success_keys = set()+ if not args.force_ingest:+ success_keys = load_success_csv(args.success_csv)+ print(+ f"[INFO] Loaded {len(success_keys)} existing success records from {args.success_csv}",+ file=sys.stderr,+ )++ # Write run header+ run_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")++ skipped_count = 0+ success_count = 0+ error_count = 0+ total_embedding_tokens = 0+ total_vlm_tokens = 0++ if args.input.endswith(".json"):+ # LoCoMo JSON format+ samples = load_locomo_data(args.input, args.sample)++ # 为每个 sample 创建独立的处理协程+ async def process_sample(item):+ sample_id = item["sample_id"]+ sessions = build_session_messages(item, session_range)++ print(f"\n=== Sample {sample_id} ===", file=sys.stderr)+ print(f" {len(sessions)} session(s) to import", file=sys.stderr)++ # 同一 sample 内串行处理所有 sessions+ for sess in sessions:+ meta = sess["meta"]+ messages = sess["messages"]+ session_key = meta["session_key"]+ label = f"{session_key} ({meta['date_time']})"++ # Skip already ingested sessions unless force-ingest is enabled+ if not args.force_ingest and is_already_ingested(+ sample_id, session_key, ingest_record, success_keys+ ):+ print(+ f" [{label}] [SKIP] already imported (use --force-ingest to reprocess)",+ file=sys.stderr,+ )+ continue++ # Preview messages+ preview = " | ".join(+ [f"{msg['role']}: {msg['text'][:30]}..." for msg in messages[:3]]+ )+ print(f" [{label}] {preview}", file=sys.stderr)++ # 串行执行(等待完成后再处理下一个 session)+ await process_single_session(+ messages=messages,+ sample_id=sample_id,+ session_key=session_key,+ meta=meta,+ run_time=run_time,+ ingest_record=ingest_record,+ args=args,+ )++ # 不同 sample 之间并行执行;可用 --parallel-samples 控制并发上限。+ if args.parallel_samples:+ semaphore = asyncio.Semaphore(args.parallel_samples)++ async def process_sample_with_limit(item):+ async with semaphore:+ await process_sample(item)++ tasks = [asyncio.create_task(process_sample_with_limit(item)) for item in samples]+ else:+ tasks = [asyncio.create_task(process_sample(item)) for item in samples]++ else:+ # Plain text format+ sessions = parse_test_file(args.input)+ print(f"Found {len(sessions)} session(s) in text file", file=sys.stderr)++ for idx, session in enumerate(sessions, start=1):+ session_key = f"txt-session-{idx}"+ print(f"\n=== Text Session {idx} ===", file=sys.stderr)++ # Skip already ingested sessions unless force-ingest is enabled+ if not args.force_ingest and is_already_ingested(+ "txt", session_key, ingest_record, success_keys+ ):+ print(+ " [SKIP] already imported (use --force-ingest to reprocess)", file=sys.stderr+ )+ skipped_count += 1+ continue++ # For plain text, all messages as user role+ messages = []+ for i, text in enumerate(session["messages"]):+ messages.append(+ {"role": "user", "text": text.strip(), "speaker": "user", "index": i}+ )++ preview = " | ".join([f"{msg['role']}: {msg['text'][:30]}..." for msg in messages[:3]])+ print(f" {preview}", file=sys.stderr)++ # 创建异步任务+ task = asyncio.create_task(+ process_single_session(+ messages=messages,+ sample_id="txt",+ session_key=session_key,+ meta={"session_index": idx},+ run_time=run_time,+ ingest_record=ingest_record,+ args=args,+ )+ )+ tasks.append(task)++ # 等待所有 sample 处理完成+ print(+ f"\n[INFO] Starting import with {len(tasks)} tasks to process",+ file=sys.stderr,+ )+ await asyncio.gather(*tasks, return_exceptions=True)++ # 从成功 CSV 统计结果+ if Path(args.success_csv).exists():+ with open(args.success_csv, "r", encoding="utf-8") as f:+ reader = csv.DictReader(f)+ for row in reader:+ success_count += 1+ total_embedding_tokens += int(row.get("embedding_tokens", 0) or 0)+ total_vlm_tokens += int(row.get("vlm_tokens", 0) or 0)++ # Final summary+ total_processed = success_count + error_count + skipped_count+ print("\n=== Import summary ===", file=sys.stderr)+ print(f"Total sessions: {total_processed}", file=sys.stderr)+ print(f"Successfully imported: {success_count}", file=sys.stderr)+ print(f"Failed: {error_count}", file=sys.stderr)+ print(f"Skipped (already imported): {skipped_count}", file=sys.stderr)+ print("\n=== Token usage summary ===", file=sys.stderr)+ print(f"Total Embedding tokens: {total_embedding_tokens}", file=sys.stderr)+ print(f"Total VLM tokens: {total_vlm_tokens}", file=sys.stderr)+ if success_count > 0:+ print(+ f"Average Embedding per session: {total_embedding_tokens // success_count}",+ file=sys.stderr,+ )+ print(f"Average VLM per session: {total_vlm_tokens // success_count}", file=sys.stderr)+ print("\nResults saved to:", file=sys.stderr)+ print(f" - Success records: {args.success_csv}", file=sys.stderr)+ print(f" - Error logs: {args.error_log}", file=sys.stderr)+++# ---------------------------------------------------------------------------+# CLI+# ---------------------------------------------------------------------------+++def main():+ # 基于脚本所在目录计算默认数据文件路径+ script_dir = Path(__file__).parent.resolve()+ default_input = str(script_dir / ".." / "data" / "locomo10.json")++ parser = argparse.ArgumentParser(description="Import conversations into OpenViking")+ parser.add_argument(+ "--input",+ default=default_input,+ help="Path to input file (.txt or LoCoMo .json)",+ )+ parser.add_argument(+ "--success-csv",+ default="./result/import_success.csv",+ help="Path to success records CSV file (default: import_success.csv)",+ )+ parser.add_argument(+ "--error-log",+ default="./result/import_errors.log",+ help="Path to error log file (default: import_errors.log)",+ )+ parser.add_argument(+ "--openviking-url",+ default="http://localhost:1933",+ help="OpenViking service URL (default: http://localhost:1933)",+ )+ parser.add_argument(+ "--sample",+ type=int,+ default=None,+ help="LoCoMo JSON: sample index (0-based). Default: all samples.",+ )+ parser.add_argument(+ "--sessions",+ default=None,+ help="LoCoMo JSON: session range, e.g. '1-4' or '3'. Default: all sessions.",+ )+ parser.add_argument(+ "--question-index",+ type=int,+ default=None,+ help="LoCoMo JSON: question index (0-based). When specified, auto-detect required sessions from question's evidence.",+ )+ parser.add_argument(+ "--parallel-samples",+ type=int,+ default=None,+ help="Max number of samples to import concurrently. Default: no limit; create one task per sample.",+ )+ parser.add_argument(+ "--force-ingest",+ action="store_true",+ default=False,+ help="Force re-import even if already recorded as completed",+ )+ parser.add_argument(+ "--clear-ingest-record",+ action="store_true",+ default=False,+ help="Clear all existing ingest records before running",+ )+ args = parser.parse_args()++ # 确保输出目录存在+ Path(args.success_csv).parent.mkdir(parents=True, exist_ok=True)+ Path(args.error_log).parent.mkdir(parents=True, exist_ok=True)++ try:+ asyncio.run(run_import(args))+ except ValueError as e:+ print(f"Error: {e}", file=sys.stderr)+ sys.exit(1)+++if __name__ == "__main__":+ main()diff --git a/benchmark/locomo/openviking/judge.py b/benchmark/locomo/openviking/judge.pynew file mode 100644index 000000000..f4d2760d0--- /dev/null+++ b/benchmark/locomo/openviking/judge.py@@ -0,0 +1,313 @@+import argparse+import asyncio+import csv+import json+import os+import sys+from pathlib import Path++from dotenv import load_dotenv+from openai import AsyncAzureOpenAI, AsyncOpenAI++try:+ from benchmark.locomo.openviking.locomo_prompts import (+ JUDGE_SYSTEM_PROMPT,+ get_judge_prompt,+ get_judge_prompt_with_evidence,+ get_strict_judge_prompt,+ get_strict_judge_prompt_with_evidence,+ preprocess_answer,+ )+except ModuleNotFoundError:+ sys.path.insert(0, str(Path(__file__).resolve().parent))+ from locomo_prompts import ( # type: ignore+ JUDGE_SYSTEM_PROMPT,+ get_judge_prompt,+ get_judge_prompt_with_evidence,+ get_strict_judge_prompt,+ get_strict_judge_prompt_with_evidence,+ preprocess_answer,+ )++env_file = Path.home() / ".openviking_benchmark_env"+load_dotenv(env_file)+csv.field_size_limit(sys.maxsize)++DEFAULT_AZURE_API_VERSION = "2025-01-01-preview"+++def parse_judge_result(content: str) -> tuple[bool | None, str]:+ stripped = content.strip()+ if not stripped:+ return None, "[PARSE ERROR] Empty response"++ start_idx = stripped.find("{")+ end_idx = stripped.rfind("}")+ if start_idx != -1 and end_idx != -1:+ try:+ result = json.loads(stripped[start_idx : end_idx + 1].strip())+ label = str(result.get("label", "")).strip().upper()+ reasoning = str(result.get("reasoning", "")).strip()+ if label == "CORRECT":+ return True, reasoning or stripped+ if label == "WRONG":+ return False, reasoning or stripped+ except json.JSONDecodeError:+ pass+ return None, f"[PARSE ERROR] Invalid response: {stripped}"+++def _parse_evidence_text(raw: str) -> str:+ if not raw:+ return ""+ raw = raw.strip()+ if not raw:+ return ""+ try:+ parsed = json.loads(raw)+ except json.JSONDecodeError:+ return raw++ if isinstance(parsed, list):+ return "\n".join(str(item) for item in parsed if str(item).strip())+ if isinstance(parsed, str):+ return parsed+ return ""+++async def grade_answer(+ llm_client,+ model: str,+ category: int,+ question: str,+ gold_answer: str,+ response: str,+ evidence_text: str = "",+ temperature: float | None = None,+ strict_prompt: bool = False,+) -> tuple[bool, str]:+ processed_answer = preprocess_answer(category, gold_answer)+ if strict_prompt and evidence_text:+ accuracy_prompt = get_strict_judge_prompt_with_evidence(+ category,+ question,+ processed_answer,+ response,+ evidence_text,+ )+ elif strict_prompt:+ accuracy_prompt = get_strict_judge_prompt(+ category,+ question,+ processed_answer,+ response,+ )+ elif evidence_text:+ accuracy_prompt = get_judge_prompt_with_evidence(+ category,+ question,+ processed_answer,+ response,+ evidence_text,+ )+ else:+ accuracy_prompt = get_judge_prompt(+ category,+ question,+ processed_answer,+ response,+ )++ try:+ request_kwargs = {+ "model": model,+ "messages": [+ {"role": "system", "content": JUDGE_SYSTEM_PROMPT},+ {"role": "user", "content": accuracy_prompt},+ ],+ "timeout": 60,+ }+ if temperature is not None:+ request_kwargs["temperature"] = temperature++ resp = await llm_client.chat.completions.create(**request_kwargs)+ content = resp.choices[0].message.content.strip()+ verdict, reasoning = parse_judge_result(content)+ if verdict is not None:+ return verdict, reasoning+ return False, reasoning+ except Exception as e:+ return False, f"[API ERROR] {str(e)}"+++def load_answers(input_path: str) -> tuple[list[dict], list[str]]:+ if not os.path.exists(input_path):+ raise FileNotFoundError(f"Input file not found: {input_path}")++ with open(input_path, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ fieldnames = reader.fieldnames.copy()+ if "reasoning" not in fieldnames:+ fieldnames.append("reasoning")+ rows = list(reader)+ return rows, fieldnames+++def get_ungraded_rows(rows: list[dict], force: bool = False) -> list[int]:+ """Return row indexes to judge, excluding adversarial rows without normal answers."""+ indexes = []+ for i, row in enumerate(rows):+ if str(row.get("category", "")).strip() == "5":+ continue+ if force:+ row["result"] = ""+ row["reasoning"] = ""+ indexes.append(i)+ elif not row.get("result"):+ indexes.append(i)+ return indexes+++def create_llm_client(+ provider: str,+ *,+ base_url: str,+ token: str,+ api_version: str | None = None,+):+ if provider == "azure":+ return AsyncAzureOpenAI(+ api_key=token,+ azure_endpoint=base_url,+ api_version=api_version or DEFAULT_AZURE_API_VERSION,+ )+ return AsyncOpenAI(base_url=base_url, api_key=token)+++async def main():+ parser = argparse.ArgumentParser(+ description="VikingBot QA judge script, same logic as openclaw evaluation"+ )+ parser.add_argument(+ "--input",+ default="./result/locomo_qa_result_only_sys_memory.csv",+ help="Path to QA result csv file, default: ./result/locomo_qa_result.csv",+ )+ parser.add_argument(+ "--base-url",+ default="https://ark.cn-beijing.volces.com/api/v3",+ help="Judge model base URL",+ )+ parser.add_argument(+ "--provider",+ default=os.getenv("LOCOMO_JUDGE_PROVIDER", "openai"),+ choices=("openai", "azure"),+ help="Judge provider type, default: openai",+ )+ parser.add_argument(+ "--token",+ default=os.getenv("ARK_API_KEY", os.getenv("OPENAI_API_KEY", "")),+ help="Judge API token",+ )+ parser.add_argument(+ "--api-version",+ default=os.getenv("LOCOMO_JUDGE_API_VERSION"),+ help=f"Azure API version, default: {DEFAULT_AZURE_API_VERSION} for provider=azure",+ )+ parser.add_argument(+ "--model",+ default="doubao-seed-2-0-pro-260215",+ help="Judge model name",+ )+ parser.add_argument(+ "--temperature",+ type=float,+ default=None,+ help="Optional judge temperature. Omitted by default for models that only support their default value.",+ )+ parser.add_argument(+ "--parallel", type=int, default=5, help="Parallel request count, default: 5"+ )+ parser.add_argument(+ "--force",+ action="store_true",+ help="Force re-grade all non-adversarial rows even if result already exists",+ )+ parser.add_argument(+ "--strict-prompt",+ action="store_true",+ help="Use the strict LoCoMo judge prompt instead of the default lenient prompt",+ )+ args = parser.parse_args()++ if not args.token:+ print("Error: API token is required")+ raise SystemExit(1)++ rows, fieldnames = load_answers(args.input)+ total = len(rows)+ ungraded = get_ungraded_rows(rows, force=args.force)+ print(f"Total answers: {total}, ungraded: {len(ungraded)}")++ if not ungraded:+ print("All answers already graded, exit")+ return++ client = create_llm_client(+ args.provider,+ base_url=args.base_url,+ token=args.token,+ api_version=args.api_version,+ )+ semaphore = asyncio.Semaphore(args.parallel)+ file_lock = asyncio.Lock()++ async def save_results():+ async with file_lock:+ temp_file = f"{args.input}.tmp"+ with open(temp_file, "w", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=fieldnames)+ writer.writeheader()+ writer.writerows(rows)+ os.replace(temp_file, args.input)++ async def process_row(idx):+ async with semaphore:+ row = rows[idx]+ question = row["question"]+ gold = row["answer"]+ response = row["response"]+ category = int(str(row.get("category", "0") or "0"))+ evidence_text = _parse_evidence_text(row.get("evidence_text", ""))+ print(f"Grading {idx + 1}/{total}: {question[:60]}...")+ is_correct, reasoning = await grade_answer(+ client,+ args.model,+ category,+ question,+ gold,+ response,+ evidence_text,+ temperature=args.temperature,+ strict_prompt=args.strict_prompt,+ )+ row["result"] = "CORRECT" if is_correct else "WRONG"+ row["reasoning"] = reasoning++ await save_results()+ print(f"Saved result for {idx + 1}/{total}: {row['result']}")++ return idx, row++ tasks = [process_row(idx) for idx in ungraded]+ await asyncio.gather(*tasks)++ correct = sum(1 for row in rows if row.get("result") == "CORRECT")+ total_graded = sum(1 for row in rows if row.get("result"))+ accuracy = correct / total_graded if total_graded > 0 else 0.0+ print(f"\nGrading completed: {correct}/{total_graded} correct, accuracy: {accuracy:.2%}")+ print(f"All results saved to {args.input}")+++if __name__ == "__main__":+ asyncio.run(main())diff --git a/benchmark/locomo/openviking/locomo_prompts.py b/benchmark/locomo/openviking/locomo_prompts.pynew file mode 100644index 000000000..6023ca5a5--- /dev/null+++ b/benchmark/locomo/openviking/locomo_prompts.py@@ -0,0 +1,288 @@+"""+LoCoMo answer-generation prompt aligned with the mem0 benchmark runner.+"""++from datetime import datetime as _datetime++CATEGORY_NAMES = {+ 1: "multi-hop",+ 2: "temporal",+ 3: "open-domain",+ 4: "single-hop",+ 5: "adversarial",+}++CATEGORIES_TO_EVALUATE = [1, 2, 3, 4]+++ANSWER_GENERATION_PROMPT = """You are answering a question using retrieved memories from past conversations. Follow these reasoning steps IN ORDER.++## Step 1: SCAN ALL MEMORIES+Read EVERY memory below from first to last. For each one that contains information relevant to the question, note it. Do NOT stop after finding the first relevant memory - important details are often scattered across many memories, including ones far down the list. Give equal weight to ALL memories regardless of position - a memory near the end is just as likely to contain the answer as one near the beginning. In these memories, "User" refers to the main person whose memories these are.++## Step 2: ENTITY VERIFICATION+Confirm each relevant memory is about the correct person/entity. If the question asks "What does Person A like?" and a memory says "Person B likes X", do NOT use that memory to answer about Person A. In two-person conversations, both speakers' actions are relevant - if the question asks about person A and a memory attributes an action to person B (the other speaker), that information is still valid evidence from their shared conversations, but always check the attribution is correct.++## Step 3: COMBINE AND CROSS-REFERENCE+- COMBINE facts from multiple memories about the same topic. If one memory says "won first place" and another says "performed a piece titled X," those describe the same event - connect them.+- For listing/counting questions, extract EVERY distinct item from ALL memories. A single memory may contain multiple items. Think about what CATEGORIES of answers the question could have, then re-scan specifically for each category.+- For counting questions ("how many times", "how many X"), enumerate each distinct instance explicitly with its date or context BEFORE giving a final count. Do not estimate - list them out, then count the list.+- DECOMPOSE complex sentences: "an immersive X with Y, enjoys Z" contains multiple distinct facts. Each could be the answer.+- Connect related facts across memories: if one says "nearby lake" and another says "Lake Tahoe is great for kayaking", the nearby lake IS Lake Tahoe. If one says "bought X in Paris", infer the country is France.++## Step 4: SELECT THE BEST ANSWER+- Do NOT assume the highest-ranked memory is correct. Multiple memories may describe different events for the same topic. Compare each candidate's relevance to the SPECIFIC question, not its retrieval score. A lower-ranked memory that directly answers the question beats a higher-ranked one that is only tangentially related.+- ALWAYS choose the MOST SPECIFIC detail available. A proper name, title, or number beats a generic description. Rate each candidate as HIGH specificity (name, title, number, specific activity) or LOW (generic description), and prefer HIGH.+- Report what someone actually DID, not what was offered or available to them. "Has not tried X yet" means X was NOT done - disqualify it. "Joined X" or "has done X" means it WAS done - prefer it.+- When multiple memories repeat the same generic fact, that repetition does NOT make it more correct than a single memory with a more specific answer.+- Photos depict what was IN the photo, not facts about someone's daily life. Prefer direct statements over photo descriptions for inferences.+- Re-read the question carefully before answering. If it asks "what aspect/type/kind", answer with the specific aspect. If it asks "what did they discover they both enjoy", answer with the specific thing, not the setting.++## Step 5: TEMPORAL GROUNDING+These conversations took place around {reference_date}. All events occurred in 2022-2024.+- Calculate time relative to this date, NOT today. Never output 2025 or 2026.+- Use dates explicitly stated in memory text. Do not invent or estimate dates.+- When a question asks what someone "shared" or "mentioned" on a date, that date is when they TALKED about it - look for events shortly BEFORE that date.+- For "how long" questions, find the start and end dates explicitly, then compute the duration. Do not guess.+- TEMPORAL DISAMBIGUATION: When you find MULTIPLE instances of similar events at different dates, enumerate them all with their dates before picking. If the question uses past tense + "the" -> select the instance closest to (and before) the reference date. If future tense ("plans to", "going to") -> select the earliest planned date. NEVER default to the first-mentioned or highest-scored instance - the DATE determines the answer.++## Step 6: INCLUSION CHECK (for lists and counts)+If you found items during reasoning that you're tempted to exclude from your answer - STOP. Include them unless you have STRONG evidence they are wrong. The most common mistake is finding relevant items but then dropping them due to overly strict filtering. More items is better than fewer when there is supporting evidence.+- For counting: after enumerating, re-verify each item. Check for duplicates (same event described differently) and ensure you haven't missed items from memories late in the list.+- The question assumes something happened. Find WHAT happened, don't say nothing happened.++## Step 7: COMMIT AND ANSWER+Give a direct, specific answer. NEVER say "not specified", "not mentioned", "no record", or "the memories don't say" - if ANY memory contains relevant information, give the best answer from available evidence. No hedging, no caveats. If the question asks for a list, include ALL items found. NEVER return an empty answer when relevant memories exist.+- NEVER generate specific names, titles, places, or dates that do not appear in any memory above. If no memory contains the specific detail the question asks for, answer with what the memories DO contain rather than guessing.+- For open-domain/opinion questions ("Would X do Y?", "Is X considered Z?"):+ * Follow the DIRECT causal reasoning in the memories. Do NOT construct elaborate counter-arguments.+ * "Would X still do Y without Z?" - If memories show X does Y BECAUSE of Z, then without Z, answer "likely no."+ * "Would X do Y again soon?" - If the most recent attempt involved a bad experience (accident, scare, trauma), answer "likely no." A recent negative experience outweighs historical positive patterns.+ * For trait questions ("Is X considered Z?"): weigh ALL evidence including symbolic/indirect references. If there is SOME but not strong evidence, answer with a qualified degree ("somewhat") rather than flat "no."++# Instructions++## Misc++1. Make reasonable deductions based on your memories. Memory shows store with a lot of working people -> store employs a lot of people+2. If a memory describes something recognizable (e.g., "romantic drama about memory and relationships"), you may name it (e.g., "Eternal Sunshine of the Spotless Mind").+3. Use domain knowledge to connect facts: a game exclusive to one platform implies ownership of that platform. An unnamed company deal can be linked to a previously expressed brand preference.++{memories}++Question: {question}++Work through Steps 1-7, then give your final answer after "ANSWER:".+"""+++ANSWERER_MEMORY_LIMIT = 200+++def _to_human_date(iso_str: str) -> str:+ for fmt in ("%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%dT%H:%M:%S.%f%z", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d"):+ try:+ return _datetime.strptime(iso_str[:26].rstrip("Z"), fmt.replace("%z", "")).strftime(+ "%A, %B %d, %Y"+ )+ except ValueError:+ continue+ return iso_str[:10]+++def get_answer_generation_prompt(+ question: str,+ search_results: list,+ reference_date: str | None = None,+) -> str:+ if reference_date is None:+ reference_date = "2023"++ if not search_results:+ memories_text = "(No relevant memories found)"+ else:+ top_results = search_results[:ANSWERER_MEMORY_LIMIT]+ sorted_results = sorted(top_results, key=lambda x: x.get("created_at", ""))+ lines = [+ "The following memories are presented in chronological order (oldest to newest).",+ "",+ ]+ for result in sorted_results:+ memory = result.get("memory", "")+ created_at = result.get("created_at", "")+ if created_at:+ lines.append(f"({_to_human_date(created_at)}) {memory}")+ else:+ lines.append(f"(unknown date) {memory}")+ memories_text = "\n".join(lines)++ return ANSWER_GENERATION_PROMPT.format(+ memories=memories_text,+ question=question,+ reference_date=reference_date,+ )+++JUDGE_SYSTEM_PROMPT = (+ "You are evaluating conversational AI memory recall. "+ "Return JSON only with the format requested."+)++_EVIDENCE_CHUNK = """+## Evidence (actual conversation messages containing the answer)+{evidence_context}+"""++_EVIDENCE_RULE = """+5. **EVIDENCE SUPPORTS ANSWER**: If the evidence corroborates the generated answer, mark CORRECT — even when the generated answer diverges from the gold answer. The gold answer may be wrong or oversimplified; if the generated answer provides a more accurate or better-supported conclusion based on the evidence, that is acceptable. Use evidence only to ACCEPT answers, never to reject them more strictly.+"""++_EVIDENCE_WRONG_CLAUSE = " AND is not supported by evidence"++_JUDGE_TEMPLATE = """Label the generated answer as CORRECT or WRONG.+{evidence_section}+## Rules++1. **PARTIAL CREDIT**: If the generated answer includes AT LEAST ONE correct item from the gold answer's list, mark CORRECT. Getting 1 out of 2, 2 out of 4, etc. is always acceptable. Only mark WRONG if NONE of the gold answer items appear.++2. **PARAPHRASES COUNT**: Same concept in different words is CORRECT. "Chocolate raspberry tart" = "chocolate cake with raspberries". "Shelter meal service" = "volunteering at a homeless shelter". Emotions and sentiments in the same positive/negative family count as paraphrases: "proud" = "fulfilled" = "accomplished"; "huge success" = "relieved" = "thrilled" (all express positive achievement). Judge semantic meaning, not exact wording.++3. **EXTRA DETAIL IS FINE**: A longer answer that includes the gold answer's key facts plus additional information is CORRECT. Never penalize for being more detailed or specific. If the generated answer adds extra descriptive details beyond the gold answer while still referencing the same core entity or concept, mark CORRECT.++4. **DATE TOLERANCE**: Dates within 14 days of each other are CORRECT. Durations within 50% are CORRECT (e.g., "5 months" matches "six months"; "19 days" matches "two weeks"). Relative dates ("few days before November") match specific dates in the same window. A specific date (e.g., "February 2020") that is consistent with a vague reference (e.g., "a few years ago" relative to 2023) is CORRECT. Converting "last year" to the actual year (e.g., "2022" when conversations are in 2023) is CORRECT.+{evidence_rule}+5. **SEMANTIC OVERLAP**: Judge whether the generated answer addresses the same topic and captures the core idea of the gold answer. Different wording, phrasing, or level of detail should not result in WRONG if the underlying concept matches. For EMOTIONS and FEELINGS questions, answers expressing sentiments in the same valence (positive/negative) about the same event are CORRECT — do not require the exact same emotion word.++6. **SAME REFERENT**: If the generated answer mentions or references the same named entity, character, person, or concept as the gold answer, mark CORRECT — even if the generated answer provides a different physical description or includes additional details. The key question is: does the generated answer identify the same core entity? If yes, it is CORRECT.++7. **FOCUS ON KNOWLEDGE, NOT WORDING**: The goal is to assess whether the system recalled the right fact. Minor differences in specificity, phrasing, or scope should not result in WRONG. Only mark WRONG when the generated answer demonstrates a genuinely different or incorrect understanding.++## ONLY mark WRONG if:+- The generated answer contains ZERO correct items from the gold answer{evidence_wrong_clause}+- The answer addresses a completely different topic++## Question+Question: {{question}}+Gold answer: {{answer}}+Generated answer: {{response}}++Return JSON with "reasoning" (one sentence) and "label" (CORRECT or WRONG). Do NOT include both labels."""+++def _build_judge_prompt(evidence_context: str | None = None) -> str:+ if evidence_context:+ prompt = _JUDGE_TEMPLATE.format(+ evidence_section=_EVIDENCE_CHUNK.format(evidence_context=evidence_context),+ evidence_rule=_EVIDENCE_RULE,+ evidence_wrong_clause=_EVIDENCE_WRONG_CLAUSE,+ )+ prompt = prompt.replace("\n5. **SEMANTIC OVERLAP", "\n6. **SEMANTIC OVERLAP")+ prompt = prompt.replace("\n6. **SAME REFERENT", "\n7. **SAME REFERENT")+ prompt = prompt.replace("\n7. **FOCUS ON KNOWLEDGE", "\n8. **FOCUS ON KNOWLEDGE")+ return prompt++ return _JUDGE_TEMPLATE.format(+ evidence_section="",+ evidence_rule="",+ evidence_wrong_clause="",+ )+++JUDGE_PROMPT = _build_judge_prompt(evidence_context=None)+++_STRICT_JUDGE_TEMPLATE = """Label the generated answer as CORRECT or WRONG.+{evidence_section}+## Strict Rules++1. **MATCH THE GOLD ANSWER**: Judge the generated answer against the gold answer. The generated answer must contain the key fact(s) requested by the question. Do not mark CORRECT merely because it discusses the same topic.++2. **LISTS REQUIRE SUBSTANTIAL COVERAGE**: If the gold answer contains multiple items, the generated answer must include all or nearly all essential items. Missing a major item is WRONG. A single correct item from a multi-item answer is not enough.++3. **QUESTION-SPECIFIC FACTS REQUIRED**: The answer must match the exact attribute, action, object, person, place, count, or time asked by the question. Mentioning the same person/entity is insufficient if the stated fact is different.++4. **DATES AND TEMPORAL RELATIONS ARE STRICT**: For questions asking when, before/after, most recent, duration, or count over time, the generated answer must preserve the correct temporal relation. Exact dates should match unless the gold answer itself is vague. Do not allow broad date windows for specific-date answers.++5. **COUNTS MUST MATCH**: For counting questions, the generated answer must give the correct count and must not include extra unsupported instances.++6. **PARAPHRASES COUNT, BUT ONLY IF MEANING MATCHES**: Different wording is acceptable only when it preserves the same core meaning. Broader, narrower, or merely related statements are WRONG.++7. **EXTRA DETAILS ARE ALLOWED ONLY IF THE CORE ANSWER IS CORRECT**: Additional details do not hurt if the requested answer is correct, but extra details cannot compensate for a missing or wrong core answer. If extra details contradict the gold answer, mark WRONG.++8. **NO GOLD OVERRIDE**: Treat the gold answer as the grading target. Evidence may clarify wording or context, but do not replace the gold answer with a different answer unless the generated answer also matches the gold answer's key fact(s).++## Mark CORRECT only if:+- The generated answer directly answers the question, AND+- It contains the essential gold answer fact(s), AND+- Any dates, counts, entities, and temporal relations required by the question are correct.++## Mark WRONG if:+- The answer is missing a key gold fact.+- The answer gives a different date, count, person, place, action, object, or temporal relation.+- The answer is only topically related.+- The answer is too vague to verify against the gold answer.++## Question+Question: {{question}}+Gold answer: {{answer}}+Generated answer: {{response}}++Return JSON with "reasoning" (one sentence) and "label" (CORRECT or WRONG). Do NOT include both labels."""+++def _build_strict_judge_prompt(evidence_context: str | None = None) -> str:+ evidence_section = (+ _EVIDENCE_CHUNK.format(evidence_context=evidence_context) if evidence_context else ""+ )+ return _STRICT_JUDGE_TEMPLATE.format(evidence_section=evidence_section)+++def get_judge_prompt(+ category: int,+ question: str,+ answer: str,+ response: str,+) -> str:+ del category+ return JUDGE_PROMPT.format(question=question, answer=answer, response=response)+++def get_judge_prompt_with_evidence(+ category: int,+ question: str,+ answer: str,+ response: str,+ evidence_context: str,+) -> str:+ del category+ prompt = _build_judge_prompt(evidence_context=evidence_context)+ return prompt.format(question=question, answer=answer, response=response)+++def get_strict_judge_prompt(+ category: int,+ question: str,+ answer: str,+ response: str,+) -> str:+ del category+ prompt = _build_strict_judge_prompt(evidence_context=None)+ return prompt.format(question=question, answer=answer, response=response)+++def get_strict_judge_prompt_with_evidence(+ category: int,+ question: str,+ answer: str,+ response: str,+ evidence_context: str,+) -> str:+ del category+ prompt = _build_strict_judge_prompt(evidence_context=evidence_context)+ return prompt.format(question=question, answer=answer, response=response)+++def preprocess_answer(category: int, answer: str) -> str:+ if category == 3 and ";" in answer:+ return answer.split(";")[0].strip()+ return answerdiff --git a/benchmark/locomo/openviking/run_eval.py b/benchmark/locomo/openviking/run_eval.pynew file mode 100644index 000000000..87d07cbca--- /dev/null+++ b/benchmark/locomo/openviking/run_eval.py@@ -0,0 +1,980 @@+import argparse+import csv+import json+import os+import threading+import time+from concurrent.futures import ThreadPoolExecutor, as_completed+from datetime import datetime+from pathlib import Path+from typing import Any++from openviking_cli.client.sync_http import SyncHTTPClient++try:+ from benchmark.locomo.openviking.locomo_prompts import get_answer_generation_prompt+except ModuleNotFoundError:+ import sys++ sys.path.insert(0, str(Path(__file__).resolve().parent))+ from locomo_prompts import get_answer_generation_prompt+++def get_evidence_text(evidence_list: list, sample: dict) -> list[str]:+ """根据 evidence 列表获取原始对话文本++ evidence 格式: ['D1:3', 'D2:5'] -> session_1 第3条, session_2 第5条+ """+ if not evidence_list:+ return []++ conv = sample.get("conversation", {})+ results = []++ for ev in evidence_list:+ # 解析 D1:3 -> session_1, index 2+ try:+ parts = ev.split(":")+ session_num = int(parts[0][1:]) # D1 -> 1+ msg_index = int(parts[1]) - 1 # 3 -> index 2++ session_key = f"session_{session_num}"+ session_messages = conv.get(session_key, [])++ if msg_index < len(session_messages):+ msg = session_messages[msg_index]+ text = msg.get("text", "")+ speaker = msg.get("speaker", "")+ results.append(f"{speaker}: {text}")+ else:+ results.append(f"[{ev}: out of range]")+ except (ValueError, IndexError):+ results.append(f"[{ev}: invalid format]")++ return results+++def parse_locomo_datetime(date_str: str) -> datetime | None:+ """解析 LoCoMo 时间格式,如 '1:56 pm on 8 May, 2023'"""+ try:+ # 移除时间部分,只保留日期 "8 May, 2023"+ if " on " in date_str:+ date_part = date_str.split(" on ")[-1]+ return datetime.strptime(date_part.strip(), "%d %B, %Y")+ except ValueError:+ pass+ return None+++def get_sample_question_time(sample: dict) -> str | None:+ """从 sample 的 conversation 中提取最后一个有内容 session 的时间,返回 ISO 格式日期"""+ conversation = sample.get("conversation", {})++ # 找所有 session_N 字段(非 date_time)+ session_keys = [+ k for k in conversation.keys() if k.startswith("session_") and "date_time" not in k+ ]+ if not session_keys:+ return None++ # 按 session 编号排序,找到最后一个有内容的+ def get_session_num(key):+ try:+ return int(key.replace("session_", ""))+ except ValueError:+ return 0++ session_keys.sort(key=get_session_num, reverse=True)++ for session_key in session_keys:+ if conversation.get(session_key): # 有内容+ # 找到对应的 date_time+ session_num = get_session_num(session_key)+ dt_key = f"session_{session_num}_date_time"+ date_str = conversation.get(dt_key)+ if date_str:+ dt = parse_locomo_datetime(date_str)+ if dt:+ return dt.strftime("%Y-%m-%d")++ return None+++def load_csv_qa(+ input_path: str, count: int | None = None, default_time: str | None = None+) -> list[dict]:+ """从CSV文件加载QA数据,取sample_id和question字段"""+ qa_list = []+ with open(input_path, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ for row in reader:+ qa_list.append(+ {+ "sample_id": row.get("sample_id", ""),+ "question": row.get("question", ""),+ "answer": row.get("answer", ""),+ "category": "",+ "evidence": [],+ "question_time": default_time,+ }+ )++ if count is not None:+ qa_list = qa_list[:count]+ return qa_list+++def load_locomo_qa(+ input_path: str,+ sample_index: int | None = None,+ count: int | None = None,+ default_time: str | None = None,+ question_index: int | None = None,+ invalid_questions: set | None = None,+ sample_indices: list[int] | None = None,+) -> list[dict]:+ """加载LoCoMo数据集的QA部分,支持JSON和CSV格式++ Args:+ invalid_questions: 无效题目问题内容集合,用于标记无效题目+ """+ if input_path.lower().endswith(".csv"):+ return load_csv_qa(input_path, count, default_time)++ # 原有JSON格式处理逻辑+ with open(input_path, "r", encoding="utf-8") as f:+ data = json.load(f)++ def normalize_sample(item: dict, index: int) -> dict:+ normalized = dict(item)+ normalized["original_sample_id"] = item.get("sample_id", "")+ normalized["sample_id"] = f"sample_{index}"+ return normalized++ def parse_sample_index(raw_sample_index: str | int) -> int:+ try:+ sample_index_text = str(raw_sample_index)+ if sample_index_text.startswith("sample_"):+ sample_index_text = sample_index_text.removeprefix("sample_")+ idx = int(sample_index_text)+ except ValueError as exc:+ raise ValueError(+ f"sample '{raw_sample_index}' is invalid; use a numeric index or sample_{{idx}}"+ ) from exc+ if idx < 0 or idx >= len(data):+ raise ValueError(f"sample index {idx} out of range (0-{len(data) - 1})")+ return idx++ qa_list = []+ # 支持数字索引或 sample_id (如 "sample_26")+ if sample_indices is not None:+ validated_indices = [parse_sample_index(idx) for idx in sample_indices]+ samples = [normalize_sample(data[idx], idx) for idx in validated_indices]+ elif sample_index is not None:+ idx = parse_sample_index(sample_index)+ samples = [normalize_sample(data[idx], idx)]+ else:+ samples = [normalize_sample(sample, idx) for idx, sample in enumerate(data)]++ for sample in samples:+ sample_id = sample.get("sample_id", "")+ question_time = get_sample_question_time(sample)+ qa_items = sample.get("qa", [])++ # 如果指定了 question_index,只返回那一个问题+ if question_index is not None:+ if question_index < 0 or question_index >= len(qa_items):+ raise ValueError(+ f"question index {question_index} out of range (0-{len(qa_items) - 1})"+ )+ qa = qa_items[question_index]+ if qa.get("category") == 5:+ continue+ evidence_list = qa.get("evidence", [])+ question_id = f"{sample_id}_qa{question_index}"+ qa_list.append(+ {+ "sample_id": sample_id,+ "question_id": question_id,+ "question_index": question_index,+ "question": qa["question"],+ "answer": qa["answer"],+ "category": qa.get("category", ""),+ "evidence": evidence_list,+ "evidence_text": get_evidence_text(evidence_list, sample),+ "question_time": question_time,+ "is_invalid": qa["question"] in invalid_questions+ if invalid_questions+ else False,+ }+ )+ else:+ for q_idx, qa in enumerate(qa_items):+ if qa.get("category") == 5:+ continue+ evidence_list = qa.get("evidence", [])+ question_id = f"{sample_id}_qa{q_idx}"+ qa_list.append(+ {+ "sample_id": sample_id,+ "question_id": question_id,+ "question_index": q_idx,+ "question": qa["question"],+ "answer": qa["answer"],+ "category": qa.get("category", ""),+ "evidence": evidence_list,+ "evidence_text": get_evidence_text(evidence_list, sample),+ "question_time": question_time,+ "is_invalid": qa["question"] in invalid_questions+ if invalid_questions+ else False,+ }+ )++ if count is not None:+ qa_list = qa_list[:count]+ return qa_list+++DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT = 10+DEFAULT_SINGLE_SEARCH_RERANK_LIMIT = 10+DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS = 4000+SINGLE_SEARCH_EXCLUDED_BASENAMES = {".abstract.md", ".overview.md"}+++def get_token_encoding_name(model_name: str | None = None) -> str:+ try:+ import tiktoken++ if model_name:+ candidates = [model_name]+ if "/" in model_name:+ candidates.append(model_name.rsplit("/", 1)[-1])+ for candidate in candidates:+ try:+ return tiktoken.encoding_for_model(candidate).name+ except KeyError:+ continue+ except Exception:+ pass++ normalized = (model_name or "").lower()+ if normalized.startswith(("gpt-4o", "gpt-5", "o1", "o3", "o4")):+ return "o200k_base"+ if normalized.startswith(("gpt-4", "gpt-3.5")):+ return "cl100k_base"+ return "approx_chars_div_4"+++def count_text_tokens(text: str, model_name: str | None = None) -> int:+ encoding_name = get_token_encoding_name(model_name)+ if encoding_name == "approx_chars_div_4":+ return max(0, len(text or "") // 4)++ try:+ import tiktoken++ encoding = tiktoken.get_encoding(encoding_name)+ return len(encoding.encode(text or ""))+ except Exception:+ return max(0, len(text or "") // 4)+++def _iter_search_contexts(search_result: Any) -> list[Any]:+ if search_result is None:+ return []+ if isinstance(search_result, list):+ return search_result++ contexts = []+ for attr in ("memories", "resources", "skills"):+ contexts.extend(getattr(search_result, attr, []) or [])+ if contexts:+ return contexts++ try:+ return list(search_result)+ except TypeError:+ return []+++def is_single_search_excluded_uri(uri: str) -> bool:+ basename = str(uri or "").rstrip("/").rsplit("/", 1)[-1]+ return basename in SINGLE_SEARCH_EXCLUDED_BASENAMES+++def select_single_search_contexts(+ search_result: Any,+ limit: int = DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT,+) -> list[dict[str, Any]]:+ selected = []+ for raw_rank, context in enumerate(_iter_search_contexts(search_result), start=1):+ if raw_rank > limit:+ break+ uri = getattr(context, "uri", "")+ if not uri or is_single_search_excluded_uri(uri):+ continue+ selected.append(+ {+ "raw_rank": raw_rank,+ "uri": uri,+ "score": getattr(context, "score", 0.0),+ "abstract": getattr(context, "abstract", ""),+ "created_at": getattr(context, "created_at", ""),+ }+ )+ return selected+++def build_single_search_context_prompt(+ question: str,+ question_time: str | None,+ contexts: list[dict[str, Any]],+) -> str:+ search_results = build_single_search_prompt_search_results(contexts)+ return get_answer_generation_prompt(+ question=question,+ search_results=search_results,+ reference_date=question_time or "2023",+ )+++def build_single_search_prompt_search_results(+ contexts: list[dict[str, Any]],+) -> list[dict[str, Any]]:+ return [+ {+ "memory": str(context.get("content", "")),+ "score": context.get("score", 0.0),+ "raw_rank": context.get("raw_rank"),+ "created_at": context.get("created_at", ""),+ }+ for context in contexts+ ]+++def count_retrieved_memory_content_tokens(+ contexts: list[dict[str, Any]],+ model_name: str | None = None,+) -> tuple[int, int, str]:+ memory_texts = [str(context.get("content", "")) for context in contexts]+ tokenizer = get_token_encoding_name(model_name)+ return (+ sum(count_text_tokens(memory_text, model_name) for memory_text in memory_texts),+ sum(len(memory_text) for memory_text in memory_texts),+ tokenizer,+ )+++def filter_contexts_by_char_budget(+ contexts: list[dict[str, Any]],+ max_chars: int = DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS,+) -> tuple[list[dict[str, Any]], list[str], int]:+ if max_chars <= 0:+ return contexts, [], sum(len(str(context.get("content", ""))) for context in contexts)++ selected = []+ skipped_uris = []+ total_chars = 0+ for context in contexts:+ content = str(context.get("content", "") or "")+ content_chars = len(content)+ if total_chars + content_chars > max_chars:+ skipped_uris.append(str(context.get("uri", "")))+ continue+ selected.append(context)+ total_chars += content_chars+ return selected, skipped_uris, total_chars+++def build_single_search_reranker() -> Any | None:+ from openviking.models.rerank import RerankClient+ from openviking_cli.utils.config import get_openviking_config++ rerank_config = get_openviking_config().rerank+ if not rerank_config or not rerank_config.is_available():+ return None+ return RerankClient.from_config(rerank_config)+++def build_single_search_vlm() -> Any:+ from openviking.models.vlm import VLMFactory+ from openviking_cli.utils.config import get_openviking_config++ vlm_config = get_openviking_config().vlm+ return VLMFactory.create(vlm_config._build_vlm_config_dict())+++def _single_search_rerank_document(context: dict[str, Any]) -> str:+ content = str(context.get("content", "") or "")+ if content and not content.startswith("[READ ERROR]"):+ return content+ return str(context.get("abstract", "") or "")+++def rerank_single_search_contexts(+ question: str,+ contexts: list[dict[str, Any]],+ reranker: Any | None,+ limit: int = DEFAULT_SINGLE_SEARCH_RERANK_LIMIT,+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], str]:+ if not reranker or len(contexts) <= 1:+ return contexts, [], ""++ documents = [_single_search_rerank_document(context) for context in contexts]+ try:+ scores = reranker.rerank_batch(question, documents)+ except Exception as exc:+ return contexts, [], f"{type(exc).__name__}: {exc}"++ if not scores or len(scores) != len(contexts):+ return (+ contexts,+ [],+ (f"invalid_score_count: expected {len(contexts)}, got {len(scores) if scores else 0}"),+ )++ ranked_contexts = []+ rerank_scores = []+ for context, score in zip(contexts, scores, strict=True):+ if not isinstance(score, (int, float)):+ return contexts, [], f"invalid_score_type: {type(score).__name__}"+ next_context = dict(context)+ next_context["rerank_score"] = float(score)+ ranked_contexts.append(next_context)+ rerank_scores.append(+ {+ "uri": str(context.get("uri", "")),+ "score": float(score),+ "raw_rank": context.get("raw_rank"),+ }+ )++ ranked_contexts.sort(+ key=lambda context: (+ context.get("rerank_score", 0.0),+ -int(context.get("raw_rank", 0) or 0),+ ),+ reverse=True,+ )+ selected_contexts = ranked_contexts[: max(1, limit)]+ return selected_contexts, rerank_scores, ""+++def _response_text(response: Any) -> str:+ if hasattr(response, "content"):+ return str(response.content or "").strip()+ return str(response).strip()+++def _token_usage_from_vlm(vlm: Any, response: Any = None) -> dict[str, int]:+ usage = getattr(response, "usage", None)+ if isinstance(usage, dict) and usage:+ return {+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),+ "total_tokens": int(usage.get("total_tokens", 0) or 0),+ }+ if usage:+ prompt_tokens = int(getattr(usage, "prompt_tokens", 0) or 0)+ completion_tokens = int(getattr(usage, "completion_tokens", 0) or 0)+ return {+ "prompt_tokens": prompt_tokens,+ "completion_tokens": completion_tokens,+ "total_tokens": int(+ getattr(usage, "total_tokens", prompt_tokens + completion_tokens) or 0+ ),+ }++ if hasattr(vlm, "get_token_usage_summary"):+ summary = vlm.get_token_usage_summary()+ return {+ "prompt_tokens": int(summary.get("total_prompt_tokens", 0) or 0),+ "completion_tokens": int(summary.get("total_completion_tokens", 0) or 0),+ "total_tokens": int(summary.get("total_tokens", 0) or 0),+ }+ if hasattr(vlm, "get_token_usage"):+ usage_snapshot = vlm.get_token_usage()+ total_usage = usage_snapshot.get("total_usage", {}) if usage_snapshot else {}+ return {+ "prompt_tokens": int(total_usage.get("prompt_tokens", 0) or 0),+ "completion_tokens": int(total_usage.get("completion_tokens", 0) or 0),+ "total_tokens": int(total_usage.get("total_tokens", 0) or 0),+ }++ return {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}+++def run_vikingbot_chat(+ question: str,+ question_time: str | None = None,+ sample_id: str | None = None,+ question_id: str | None = None,+ openviking_url: str | None = None,+ single_search_context_limit: int = DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT,+ single_search_rerank_limit: int = DEFAULT_SINGLE_SEARCH_RERANK_LIMIT,+ timeout: int = 300,+ single_search_max_context_chars: int = DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS,+) -> tuple[str, dict, float, int, list, list, str]:+ """执行单轮 search + rerank + answer,返回回答、token、耗时、迭代次数、工具和检索轨迹"""+ start_time = time.time()+ client = SyncHTTPClient(+ url=openviking_url,+ user=sample_id,+ timeout=timeout,+ )+ try:+ client.initialize()+ target_uri = f"viking://user/{sample_id or 'default'}/memories"+ search_result = client.find(+ question,+ target_uri=target_uri,+ limit=single_search_context_limit,+ )+ contexts = select_single_search_contexts(+ search_result,+ limit=single_search_context_limit,+ )+ retrieved_uris = [context["uri"] for context in contexts]++ for context in contexts:+ uri = context["uri"]+ try:+ context["content"] = client.read(uri, offset=0, limit=-1)+ except Exception as exc:+ context["content"] = f"[READ ERROR] {exc}"++ reranker = build_single_search_reranker()+ rerank_enabled = reranker is not None+ rerank_limit = single_search_rerank_limit+ contexts, rerank_scores, rerank_error = rerank_single_search_contexts(+ question=question,+ contexts=contexts,+ reranker=reranker,+ limit=rerank_limit,+ )+ contexts, skipped_context_uris, context_chars = filter_contexts_by_char_budget(+ contexts,+ max_chars=single_search_max_context_chars,+ )+ context_uris = [context["uri"] for context in contexts]++ prompt = build_single_search_context_prompt(question, question_time, contexts)+ vlm = build_single_search_vlm()+ memory_prompt_tokens, memory_chars, memory_tokenizer = (+ count_retrieved_memory_content_tokens(+ contexts,+ model_name=getattr(vlm, "model", None),+ )+ )+ raw_response = vlm.get_completion(prompt)+ response = _response_text(raw_response)+ token_usage = _token_usage_from_vlm(vlm, raw_response)+ token_usage["memory_prompt_tokens"] = memory_prompt_tokens+ token_usage["memory_chars"] = memory_chars+ token_usage["memory_tokenizer"] = memory_tokenizer+ time_cost = time.time() - start_time+ return (+ response,+ token_usage,+ time_cost,+ 1,+ ["single_search", "read", "context_answer"],+ [+ {+ "iteration": 1,+ "retrieved_uris": retrieved_uris,+ "context_uris": context_uris,+ "rerank_enabled": rerank_enabled,+ "rerank_limit": rerank_limit if rerank_enabled else 0,+ "rerank_scores": rerank_scores,+ "rerank_error": rerank_error,+ "max_context_chars": single_search_max_context_chars,+ "context_chars": context_chars,+ "skipped_context_uris_by_char_limit": skipped_context_uris,+ }+ ],+ prompt,+ )+ except Exception as e:+ return (+ f"[SINGLE SEARCH ERROR] {str(e)}",+ {+ "prompt_tokens": 0,+ "completion_tokens": 0,+ "total_tokens": 0,+ "memory_prompt_tokens": 0,+ "memory_chars": 0,+ "memory_tokenizer": "approx_chars_div_4",+ },+ 0,+ 1,+ [],+ [+ {+ "iteration": 1,+ "retrieved_uris": [],+ "context_uris": [],+ "rerank_enabled": False,+ "rerank_limit": 0,+ "rerank_scores": [],+ }+ ],+ "",+ )+ finally:+ try:+ client.close()+ except Exception:+ pass+++def load_processed_questions(output_path: str) -> set:+ """加载已处理的问题集合(已禁用,每次重新运行)"""+ # 注意:去重逻辑已禁用,每次运行都会重新执行所有问题+ return set()+++def result_row_key(row: dict[str, Any]) -> str:+ question_id = str(row.get("question_id", "") or "").strip()+ if question_id:+ return f"question_id:{question_id}"++ sample_id = str(row.get("sample_id", "") or "").strip()+ question_index = str(row.get("question_index", "") or "").strip()+ if sample_id or question_index:+ return f"sample_question_index:{sample_id}:{question_index}"++ return f"question:{str(row.get('question', '') or '').strip()}"+++def parse_sample_indices(+ raw_samples: str | None, dataset_size: int | None = None+) -> list[int] | None:+ if raw_samples is None:+ return None++ indices: list[int] = []+ seen: set[int] = set()+ for raw_part in raw_samples.split(","):+ part = raw_part.strip()+ if not part:+ continue+ if part.startswith("sample_"):+ part = part.removeprefix("sample_")+ try:+ idx = int(part)+ except ValueError as exc:+ raise ValueError(+ f"sample '{raw_part}' is invalid; use numeric indices or sample_{{idx}}"+ ) from exc+ if idx < 0:+ raise ValueError(f"sample index {idx} must be >= 0")+ if dataset_size is not None and idx >= dataset_size:+ raise ValueError(f"sample index {idx} out of range (0-{dataset_size - 1})")+ if idx not in seen:+ indices.append(idx)+ seen.add(idx)++ if not indices:+ raise ValueError("--samples must include at least one sample index")+ return indices+++def main():+ # 基于脚本所在目录计算默认数据文件路径+ script_dir = Path(__file__).parent.resolve()+ default_input = str(script_dir / ".." / "data" / "locomo10.json")+ default_errors = str(script_dir / ".." / "data" / "errors.json")++ parser = argparse.ArgumentParser(description="VikingBot QA evaluation script")+ parser.add_argument(+ "input",+ nargs="?",+ default=default_input,+ help="Path to locomo10.json file",+ )+ parser.add_argument(+ "--output",+ default="./result/locomo_qa_result.csv",+ help="Path to output csv file, default: ./result/locomo_qa_result.csv",+ )+ parser.add_argument(+ "--errors",+ default=default_errors,+ help="Path to invalid questions JSON file",+ )+ parser.add_argument(+ "--sample",+ type=str,+ default=None,+ help="LoCoMo sample index (0-based) or sample_id (e.g., sample_26)",+ )+ parser.add_argument(+ "--samples",+ type=str,+ default=None,+ help="Comma-separated LoCoMo sample indices, e.g. 0,1 or sample_0,sample_1",+ )+ parser.add_argument(+ "--question-index",+ type=int,+ default=None,+ help="Question index (0-based) for single question testing",+ )+ parser.add_argument(+ "--count", type=int, default=None, help="Number of QA questions to run, default all"+ )+ parser.add_argument(+ "--threads", type=int, default=40, help="Number of concurrent threads, default: 40"+ )+ parser.add_argument(+ "--openviking-url",+ default=None,+ help="OpenViking server URL, e.g. http://127.0.0.1:1934. Defaults to ovcli.conf.",+ )+ parser.add_argument(+ "--single-search-context-limit",+ type=int,+ default=DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT,+ help=(+ "Number of memory files to read into the single-search prompt, "+ f"default: {DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT}"+ ),+ )+ parser.add_argument(+ "--single-search-rerank-limit",+ type=int,+ default=DEFAULT_SINGLE_SEARCH_RERANK_LIMIT,+ help=(+ "Number of reranked memory files to keep in the single-search prompt, "+ f"default: {DEFAULT_SINGLE_SEARCH_RERANK_LIMIT}"+ ),+ )+ parser.add_argument(+ "--single-search-max-context-chars",+ type=int,+ default=DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS,+ help=(+ "Maximum total memory content characters to concatenate into the answer prompt. "+ "Files that would exceed the budget are skipped whole; <=0 disables the budget. "+ f"default: {DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS}"+ ),+ )+ parser.add_argument(+ "--debug-print-model-input",+ action="store_true",+ help="Write full model input prompt and retrieved URI trace into the CSV.",+ )+ parser.add_argument(+ "--update-mode",+ action="store_true",+ help="Update mode: if output file exists, update matching question_id rows instead of overwriting",+ )+ args = parser.parse_args()++ # 如果指定了 question-index,单 sample/全量调试保持旧行为;多 samples 模式保留每个 sample 的这道题。+ if args.question_index is not None and args.count is None and args.samples is None:+ args.count = 1+ if args.sample is not None and args.samples is not None:+ raise ValueError("Use either --sample or --samples, not both")++ # 确保输出目录存在+ os.makedirs(os.path.dirname(args.output), exist_ok=True)++ # 加载无效题目集合(按问题内容匹配,因为 errors.json 索引可能与数据不匹配)+ invalid_questions = set()+ errors_path = os.path.expanduser(args.errors)+ if os.path.exists(errors_path):+ with open(errors_path, "r", encoding="utf-8") as f:+ errors_data = json.load(f)+ # 按问题内容建立集合+ if errors_data and isinstance(errors_data[0], dict):+ invalid_questions = {item["question"] for item in errors_data}+ else:+ invalid_questions = set(errors_data)+ print(f"Loaded {len(invalid_questions)} invalid questions from {errors_path}")+ else:+ print(f"No errors file found at {errors_path}, is_invalid will be False for all questions")++ # 加载QA数据(所有题目,包括无效题目,只标记 is_invalid)+ qa_list = load_locomo_qa(+ args.input,+ args.sample,+ args.count,+ question_index=args.question_index,+ invalid_questions=invalid_questions,+ sample_indices=parse_sample_indices(args.samples),+ )+ total = len(qa_list)++ # 过滤掉 category=5 的问题+ qa_list = [qa for qa in qa_list if str(qa.get("category")) != "5"]+ print(f"Filtered to {len(qa_list)} questions after removing category=5")++ # 加载已处理的问题+ processed_questions = load_processed_questions(args.output)+ remaining = total - len(processed_questions)+ print(+ f"Loaded {total} QA questions, {len(processed_questions)} already processed, {remaining} remaining"+ )++ fieldnames = [+ "sample_id",+ "question_id",+ "question_index",+ "result",+ "is_invalid",+ "question",+ "answer",+ "category",+ "question_time",+ "evidence",+ "evidence_text",+ "response",+ "model_input_prompt",+ "token_usage",+ "memory_prompt_tokens",+ "memory_chars",+ "memory_tokenizer",+ "time_cost",+ "iteration",+ "tools_used_names",+ "retrieved_uris_by_iteration",+ ]++ existing_rows = []+ existing_fieldnames = fieldnames+ if args.update_mode and os.path.exists(args.output):+ with open(args.output, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ existing_rows = list(reader)+ existing_fieldnames = reader.fieldnames or fieldnames+ for fieldname in fieldnames:+ if fieldname not in existing_fieldnames:+ existing_fieldnames.append(fieldname)+ elif os.path.exists(args.output):+ os.remove(args.output)++ # 创建线程锁,确保多线程写文件安全+ write_lock = threading.Lock()++ # 存储处理后的新行+ new_rows = []+ processed_count = 0++ def save_results_locked():+ temp_file = f"{args.output}.tmp"+ with open(temp_file, "w", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=existing_fieldnames)+ writer.writeheader()+ writer.writerows(existing_rows)+ os.replace(temp_file, args.output)++ # 过滤掉已经处理过的问题+ remaining_qa = [qa for qa in qa_list if result_row_key(qa) not in processed_questions]+ remaining_count = len(remaining_qa)+ print(+ f"Starting evaluation with {args.threads} concurrent threads, {remaining_count} questions to process"+ )++ def process_qa(qa_item, idx, total_count):+ """单个QA处理函数,供多线程调用"""+ question = qa_item["question"]+ answer = qa_item["answer"]+ question_time = qa_item.get("question_time")+ # 使用 question_id 作为 session_id,实现完全独立并行+ sample_id = qa_item.get("sample_id")+ question_id = qa_item.get("question_id")+ print(f"Processing {idx}/{total_count}: {question[:60]}...")+ if question_time:+ print(f" [time context: {question_time}]")++ (+ response,+ token_usage,+ time_cost,+ iteration,+ tools_used_names,+ retrieved_uris_by_iteration,+ model_input_prompt,+ ) = run_vikingbot_chat(+ question=question,+ question_time=question_time,+ sample_id=sample_id,+ question_id=question_id,+ openviking_url=args.openviking_url,+ single_search_context_limit=args.single_search_context_limit,+ single_search_rerank_limit=args.single_search_rerank_limit,+ single_search_max_context_chars=args.single_search_max_context_chars,+ )++ row = {+ "sample_id": qa_item["sample_id"],+ "question_id": question_id,+ "question_index": qa_item.get("question_index", ""),+ "result": "",+ "question": question,+ "answer": answer,+ "category": qa_item.get("category", ""),+ "question_time": question_time or "",+ "evidence": json.dumps(qa_item.get("evidence", [])),+ "evidence_text": json.dumps(qa_item.get("evidence_text", [])),+ "response": response,+ "model_input_prompt": model_input_prompt if args.debug_print_model_input else "",+ "token_usage": json.dumps(token_usage, ensure_ascii=False),+ "memory_prompt_tokens": token_usage.get("memory_prompt_tokens", 0),+ "memory_chars": token_usage.get("memory_chars", 0),+ "memory_tokenizer": token_usage.get("memory_tokenizer", ""),+ "time_cost": round(time_cost, 2),+ "iteration": iteration,+ "tools_used_names": json.dumps(tools_used_names, ensure_ascii=False),+ "retrieved_uris_by_iteration": json.dumps(+ retrieved_uris_by_iteration, ensure_ascii=False+ )+ if args.debug_print_model_input+ else "",+ "is_invalid": qa_item.get("is_invalid", False),+ }++ # 线程安全的结果收集+ with write_lock:+ nonlocal processed_count+ new_rows.append(row)+ row_key = result_row_key(row)+ found = False+ for existing_row in existing_rows:+ if result_row_key(existing_row) == row_key:+ existing_row.update(row)+ found = True+ break+ if not found:+ existing_rows.append(row)+ processed_questions.add(row_key)+ processed_count += 1+ save_results_locked()+ print(f"Completed {processed_count}/{total_count}, time cost: {round(time_cost, 2)}s")+ return True++ # 使用线程池处理:全局并行,每个 question 独立 session+ with ThreadPoolExecutor(max_workers=args.threads) as executor:+ # 提交所有任务+ futures = []+ for idx, qa_item in enumerate(remaining_qa, 1):+ futures.append(executor.submit(process_qa, qa_item, idx, remaining_count))++ # 等待所有任务完成+ for future in as_completed(futures):+ try:+ future.result()+ except Exception as e:+ print(f"Error processing QA item: {str(e)}")++ if args.update_mode:+ print(f"Updated {len(new_rows)} rows in {args.output}")+ else:+ print(f"Evaluation completed, results saved to {args.output}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/locomo/openviking/stat_judge_result.py b/benchmark/locomo/openviking/stat_judge_result.pynew file mode 100644index 000000000..a7d1eb769--- /dev/null+++ b/benchmark/locomo/openviking/stat_judge_result.py@@ -0,0 +1,259 @@+import argparse+import csv+import json+import os+import sys+from collections import defaultdict++CATEGORY_NAMES = {+ "1": "multi-hop",+ "2": "temporal",+ "3": "open-domain",+ "4": "single-hop",+ "5": "adversarial",+}++csv.field_size_limit(sys.maxsize)+++def category_label(category: str) -> str:+ category = str(category or "").strip()+ name = CATEGORY_NAMES.get(category)+ if name:+ return f"{category}-{name}"+ return category or "<missing>"+++def main():+ parser = argparse.ArgumentParser(description="Statistics for judge result csv")+ parser.add_argument(+ "--input",+ default="./result/locomo_qa_result_only_sys_memory.csv",+ help="Path to judge result csv file, default: ./result/judge_result.csv",+ )+ args = parser.parse_args()++ if not os.path.exists(args.input):+ print(f"Error: File not found: {args.input}")+ exit(1)++ # 统计所有题目 (排除 category=5)+ correct = 0+ wrong = 0+ total_time = 0.0+ total_prompt_tokens = 0+ total_memory_prompt_tokens = 0+ total_memory_chars = 0+ total_completion_tokens = 0+ total_tokens = 0+ valid_rows = 0+ total_iteration = 0+ by_category: dict[str, dict[str, int]] = defaultdict(+ lambda: {"CORRECT": 0, "WRONG": 0, "OTHER": 0}+ )++ # 统计 is_valid=True 的题目 (排除 category=5)+ valid_only_correct = 0+ valid_only_wrong = 0+ valid_only_total_time = 0.0+ valid_only_total_prompt_tokens = 0+ valid_only_total_memory_prompt_tokens = 0+ valid_only_total_memory_chars = 0+ valid_only_total_completion_tokens = 0+ valid_only_total_tokens = 0+ valid_only_rows = 0+ valid_only_total_iteration = 0++ with open(args.input, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ for row in reader:+ # 检查 category 是否为 5,跳过+ category = row.get("category", "")+ if category == "5":+ continue++ valid_rows += 1++ # 检查是否是无效题目+ is_invalid = row.get("is_invalid", "").lower() == "true"+ is_valid = not is_invalid++ # 统计结果+ result = row.get("result", "").strip().upper()+ category_key = category_label(category)+ if result == "CORRECT":+ correct += 1+ by_category[category_key]["CORRECT"] += 1+ if is_valid:+ valid_only_correct += 1+ elif result == "WRONG":+ wrong += 1+ by_category[category_key]["WRONG"] += 1+ if is_valid:+ valid_only_wrong += 1+ else:+ by_category[category_key]["OTHER"] += 1++ total_iteration += int(row.get("iteration", "0"))+ if is_valid:+ valid_only_total_iteration += int(row.get("iteration", "0"))++ # 统计耗时+ time_cost = row.get("time_cost", "")+ if time_cost:+ try:+ time_val = float(time_cost)+ total_time += time_val+ if is_valid:+ valid_only_total_time += time_val+ except (ValueError, TypeError):+ pass++ # 统计token+ token_usage = row.get("token_usage", "")+ if token_usage and token_usage.strip():+ try:+ token_data = json.loads(token_usage)+ total_prompt_tokens += token_data.get("prompt_tokens", 0)+ total_memory_prompt_tokens += token_data.get("memory_prompt_tokens", 0)+ total_memory_chars += token_data.get("memory_chars", 0)+ total_completion_tokens += token_data.get("completion_tokens", 0)+ total_tokens += token_data.get("total_tokens", 0)++ if is_valid:+ valid_only_total_prompt_tokens += token_data.get("prompt_tokens", 0)+ valid_only_total_memory_prompt_tokens += token_data.get(+ "memory_prompt_tokens", 0+ )+ valid_only_total_memory_chars += token_data.get("memory_chars", 0)+ valid_only_total_completion_tokens += token_data.get("completion_tokens", 0)+ valid_only_total_tokens += token_data.get("total_tokens", 0)+ except json.JSONDecodeError:+ pass++ if is_valid:+ valid_only_rows += 1++ total_graded = correct + wrong+ accuracy = correct / total_graded if total_graded > 0 else 0.0+ avg_time = total_time / valid_rows if valid_rows > 0 else 0.0++ # is_valid=True 题目的统计 (排除 category=5)+ valid_only_total_graded = valid_only_correct + valid_only_wrong+ valid_only_accuracy = (+ valid_only_correct / valid_only_total_graded if valid_only_total_graded > 0 else 0.0+ )+ valid_only_avg_time = valid_only_total_time / valid_only_rows if valid_only_rows > 0 else 0.0++ # 平均 token 消耗+ avg_prompt_tokens = total_prompt_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_memory_prompt_tokens = total_memory_prompt_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_memory_chars = total_memory_chars / valid_rows if valid_rows > 0 else 0.0+ avg_completion_tokens = total_completion_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_total_tokens = total_tokens / valid_rows if valid_rows > 0 else 0.0++ valid_only_avg_prompt_tokens = (+ valid_only_total_prompt_tokens / valid_only_rows if valid_only_rows > 0 else 0.0+ )+ valid_only_avg_memory_prompt_tokens = (+ valid_only_total_memory_prompt_tokens / valid_only_rows if valid_only_rows > 0 else 0.0+ )+ valid_only_avg_memory_chars = (+ valid_only_total_memory_chars / valid_only_rows if valid_only_rows > 0 else 0.0+ )+ valid_only_avg_completion_tokens = (+ valid_only_total_completion_tokens / valid_only_rows if valid_only_rows > 0 else 0.0+ )+ valid_only_avg_total_tokens = (+ valid_only_total_tokens / valid_only_rows if valid_only_rows > 0 else 0.0+ )++ output_lines = [+ "=== Judge Result Statistics (excluding category=5) ===",+ f"Total rows: {valid_rows}",+ f"Graded rows: {total_graded}",+ f"Correct: {correct}",+ f"Wrong: {wrong}",+ f"Accuracy: {accuracy:.2%}",+ f"\nAverage time cost: {avg_time:.2f}s",+ f"\nAverage iteration: {total_iteration / valid_rows if valid_rows > 0 else 0.0:.2f}",+ "\nToken usage:",+ f" Total prompt tokens: {total_prompt_tokens}",+ f" Total memory prompt tokens: {total_memory_prompt_tokens}",+ f" Total memory chars: {total_memory_chars}",+ f" Total completion tokens: {total_completion_tokens}",+ f" Total tokens: {total_tokens}",+ f" Avg prompt tokens: {avg_prompt_tokens:.2f}",+ f" Avg memory prompt tokens: {avg_memory_prompt_tokens:.2f}",+ f" Avg memory chars: {avg_memory_chars:.2f}",+ f" Avg completion tokens: {avg_completion_tokens:.2f}",+ f" Avg total tokens: {avg_total_tokens:.2f}",+ "",+ "By category:",+ f"{'category':<28} {'correct':>8} {'wrong':>8} {'other':>8} {'graded':>8} {'total':>8} {'accuracy':>10}",+ ]++ for category in sorted(by_category):+ category_correct = by_category[category]["CORRECT"]+ category_wrong = by_category[category]["WRONG"]+ category_other = by_category[category]["OTHER"]+ category_graded = category_correct + category_wrong+ category_total = category_graded + category_other+ category_accuracy = category_correct / category_graded if category_graded > 0 else 0.0+ output_lines.append(+ f"{category:<28} {category_correct:>8} {category_wrong:>8} {category_other:>8} "+ f"{category_graded:>8} {category_total:>8} {category_accuracy:>9.2%}"+ )++ valid_only_matches_overall = (+ valid_only_rows == valid_rows+ and valid_only_total_graded == total_graded+ and valid_only_correct == correct+ and valid_only_wrong == wrong+ and valid_only_total_time == total_time+ and valid_only_total_iteration == total_iteration+ and valid_only_total_prompt_tokens == total_prompt_tokens+ and valid_only_total_memory_prompt_tokens == total_memory_prompt_tokens+ and valid_only_total_memory_chars == total_memory_chars+ and valid_only_total_completion_tokens == total_completion_tokens+ and valid_only_total_tokens == total_tokens+ )+ if not valid_only_matches_overall:+ output_lines.extend(+ [+ "",+ "=== Valid Questions Only (is_valid=True, excluding category=5) ===",+ f"Valid rows: {valid_only_rows}",+ f"Valid graded rows: {valid_only_total_graded}",+ f"Valid correct: {valid_only_correct}",+ f"Valid wrong: {valid_only_wrong}",+ f"Valid accuracy: {valid_only_accuracy:.2%}",+ f"\nAverage time cost: {valid_only_avg_time:.2f}s",+ f"\nAverage iteration: {valid_only_total_iteration / valid_only_rows if valid_only_rows > 0 else 0.0:.2f}",+ "\nToken usage:",+ f" Total prompt tokens: {valid_only_total_prompt_tokens}",+ f" Total memory prompt tokens: {valid_only_total_memory_prompt_tokens}",+ f" Total memory chars: {valid_only_total_memory_chars}",+ f" Total completion tokens: {valid_only_total_completion_tokens}",+ f" Total tokens: {valid_only_total_tokens}",+ f" Avg prompt tokens: {valid_only_avg_prompt_tokens:.2f}",+ f" Avg memory prompt tokens: {valid_only_avg_memory_prompt_tokens:.2f}",+ f" Avg memory chars: {valid_only_avg_memory_chars:.2f}",+ f" Avg completion tokens: {valid_only_avg_completion_tokens:.2f}",+ f" Avg total tokens: {valid_only_avg_total_tokens:.2f}",+ ]+ )++ # 打印到控制台+ for line in output_lines:+ print(line)++ # 写入summary.txt+ summary_path = os.path.join(os.path.dirname(args.input), "summary.txt")+ with open(summary_path, "w", encoding="utf-8") as f:+ f.write("\n".join(output_lines) + "\n")+ print(f"\nSummary saved to {summary_path}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/longmemeval/openviking/README.md b/benchmark/longmemeval/openviking/README.mdnew file mode 100644index 000000000..506d0ec0d--- /dev/null+++ b/benchmark/longmemeval/openviking/README.md@@ -0,0 +1,96 @@+# LongMemEval OpenViking Benchmark++This directory contains the OpenViking evaluation flow for LongMemEval:++1. import each user's haystack sessions into OpenViking;+2. run one retrieval call per question;+3. optionally rerank the retrieved memories;+4. answer from the selected memory context;+5. judge and summarize the result CSV.++The benchmark expects an OpenViking server to already be running. The commands+below use the default local OpenViking client configuration. If you need a+different endpoint, pass `--openviking-url` to both import and eval.++## Data++Set the dataset path once:++```bash+DATA=/path/to/longmemeval_s_cleaned.json+```++The importer uses each sample's original `question_id`-derived user id, such as+`lm_user_<id>`, so the eval script searches the same user space created during+import.++## Import++```bash+python benchmark/longmemeval/openviking/import_to_ov.py \+ --input "$DATA" \+ --parallel 16 \+ --submit-parallel 16 \+ --wait-mode deferred \+ --success-csv result/longmemeval_openviking_import_success.csv \+ --error-log result/longmemeval_openviking_import_errors.log+```++Use `--force-ingest` only when intentionally re-importing existing sessions.++## Smoke Test++Run one question before a full evaluation:++```bash+python benchmark/longmemeval/openviking/run_eval.py "$DATA" \+ --output result/longmemeval_openviking_smoke.csv \+ --count 1 \+ --threads 1 \+ --single-search-context-limit 50 \+ --single-search-rerank-limit 10 \+ --single-search-max-context-chars 30000 \+ --debug-print-model-input+```++With `--debug-print-model-input`, the CSV includes the full answer prompt and+retrieval trace. Check `retrieved_uris_by_iteration` to confirm rerank is enabled+and `context_uris` contains the expected number of memories.++## Full Eval++This example retrieves 50 memories, keeps the top 10 after rerank, and caps the+memory text passed to the answer model at 30000 characters.++```bash+OUT=result/longmemeval_openviking_search50_rerank10_chars30000.csv++python benchmark/longmemeval/openviking/run_eval.py "$DATA" \+ --output "$OUT" \+ --threads 8 \+ --timeout 900 \+ --single-search-context-limit 50 \+ --single-search-rerank-limit 10 \+ --single-search-max-context-chars 30000+```++Set `--single-search-rerank-limit 0` to disable rerank. Set+`--single-search-max-context-chars 0` to disable the character budget.++## Judge And Stat++```bash+python benchmark/longmemeval/openviking/judge.py \+ --input "$OUT" \+ --parallel 40++python benchmark/longmemeval/openviking/stat_judge_result.py \+ --input "$OUT"+```++The judge writes `result` values back into the same CSV. Use `--force` to+re-grade rows that already have a result. Use `--strict-prompt` when you want the+stricter LongMemEval judge prompt instead of the default lenient prompt.++`stat_judge_result.py` prints overall accuracy, average memory token/character+usage, and accuracy grouped by LongMemEval question type.diff --git a/benchmark/longmemeval/openviking/import_to_ov.py b/benchmark/longmemeval/openviking/import_to_ov.pynew file mode 100644index 000000000..5da0a25f9--- /dev/null+++ b/benchmark/longmemeval/openviking/import_to_ov.py@@ -0,0 +1,712 @@+"""+OpenViking data import tool for LongMemEval.++Import haystack sessions from LongMemEval JSON into OpenViking memory.+"""++import argparse+import asyncio+import csv+import hashlib+import json+import sys+import time+import traceback+from datetime import datetime, timedelta+from pathlib import Path+from typing import Any, Dict, List, Optional, Tuple++import openviking as ov++LONGMEMEVAL_TIME_FORMAT = "%Y/%m/%d (%a) %H:%M"+++def build_sample_agent_id(sample_id: str | int) -> str:+ """Return the agent_id used for one sample import."""+ digest = hashlib.md5(str(sample_id).encode("utf-8")).hexdigest()[:12]+ return f"lm_{digest}"+++def build_sample_user_id(sample_id: str | int) -> str:+ """Return the user_id used for one sample import."""+ digest = hashlib.md5(f"user:{sample_id}".encode("utf-8")).hexdigest()[:12]+ return f"lm_user_{digest}"+++def load_longmemeval_data(+ path: str,+ sample_index: Optional[int] = None,+) -> List[Dict[str, Any]]:+ """Load LongMemEval JSON and optionally filter to one sample."""+ with open(path, "r", encoding="utf-8") as f:+ data = json.load(f)++ if sample_index is not None:+ if sample_index < 0 or sample_index >= len(data):+ raise ValueError(f"Sample index {sample_index} out of range (0-{len(data) - 1})")+ return [data[sample_index]]+ return data+++def build_session_messages(+ item: Dict[str, Any],+ session_range: Optional[Tuple[int, int]] = None,+) -> List[Dict[str, Any]]:+ """Build importable session messages for one LongMemEval sample."""+ sessions = []+ haystack_sessions = item.get("haystack_sessions", [])+ haystack_dates = item.get("haystack_dates", [])+ haystack_session_ids = item.get("haystack_session_ids", [])++ for index, raw_session in enumerate(haystack_sessions, start=1):+ if session_range:+ lo, hi = session_range+ if index < lo or index > hi:+ continue++ session_key = (+ haystack_session_ids[index - 1]+ if index - 1 < len(haystack_session_ids)+ else f"session_{index}"+ )+ date_time = haystack_dates[index - 1] if index - 1 < len(haystack_dates) else ""+ messages = []+ for msg_index, msg in enumerate(raw_session):+ messages.append(+ {+ "role": msg.get("role", "user"),+ "text": msg.get("content", ""),+ "index": msg_index,+ }+ )++ sessions.append(+ {+ "messages": messages,+ "meta": {+ "sample_id": item["question_id"],+ "session_key": session_key,+ "session_index": index,+ "date_time": date_time,+ "question_type": item.get("question_type", ""),+ },+ }+ )++ return sessions+++def load_success_csv(csv_path: str = "./result/longmemeval_import_success.csv") -> set:+ success_keys = set()+ if Path(csv_path).exists():+ with open(csv_path, "r", encoding="utf-8") as f:+ reader = csv.DictReader(f)+ for row in reader:+ key = f"viking:{row['sample_id']}:{row['session']}"+ success_keys.add(key)+ return success_keys+++def write_success_record(+ record: Dict[str, Any], csv_path: str = "./result/longmemeval_import_success.csv"+) -> None:+ file_exists = Path(csv_path).exists()+ fieldnames = [+ "timestamp",+ "sample_id",+ "session",+ "date_time",+ "question_type",+ "embedding_tokens",+ "vlm_tokens",+ "llm_input_tokens",+ "llm_output_tokens",+ "total_tokens",+ ]++ with open(csv_path, "a", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=fieldnames)+ if not file_exists:+ writer.writeheader()++ writer.writerow(+ {+ "timestamp": record["timestamp"],+ "sample_id": record["sample_id"],+ "session": record["session"],+ "date_time": record.get("meta", {}).get("date_time", ""),+ "question_type": record.get("meta", {}).get("question_type", ""),+ "embedding_tokens": record["token_usage"].get("embedding", 0),+ "vlm_tokens": record["token_usage"].get("vlm", 0),+ "llm_input_tokens": record["token_usage"].get("llm_input", 0),+ "llm_output_tokens": record["token_usage"].get("llm_output", 0),+ "total_tokens": record["token_usage"].get("total", 0),+ }+ )+++def write_error_record(+ record: Dict[str, Any], error_path: str = "./result/longmemeval_import_errors.log"+) -> None:+ with open(error_path, "a", encoding="utf-8") as f:+ timestamp = record["timestamp"]+ sample_id = record["sample_id"]+ session = record["session"]+ error = record["error"]+ f.write(f"[{timestamp}] ERROR [{sample_id}/{session}]: {error}\n")+++def load_ingest_record(+ record_path: str = "./result/.longmemeval_ingest_record.json",+) -> Dict[str, Any]:+ try:+ with open(record_path, "r", encoding="utf-8") as f:+ return json.load(f)+ except (FileNotFoundError, json.JSONDecodeError):+ return {}+++def save_ingest_record(+ record: Dict[str, Any], record_path: str = "./result/.longmemeval_ingest_record.json"+) -> None:+ with open(record_path, "w", encoding="utf-8") as f:+ json.dump(record, f, indent=2, ensure_ascii=False)+++def is_already_ingested(+ sample_id: str | int,+ session_key: str,+ record: Dict[str, Any],+ success_keys: Optional[set] = None,+) -> bool:+ key = f"viking:{sample_id}:{session_key}"+ if success_keys is not None and key in success_keys:+ return True+ return key in record and record[key].get("success", False)+++def mark_ingested(+ sample_id: str | int,+ session_key: str,+ record: Dict[str, Any],+ meta: Optional[Dict[str, Any]] = None,+) -> None:+ key = f"viking:{sample_id}:{session_key}"+ record[key] = {+ "success": True,+ "timestamp": int(time.time()),+ "meta": meta or {},+ }+++def _parse_token_usage(commit_result: Dict[str, Any]) -> Dict[str, int]:+ if "result" in commit_result:+ result = commit_result["result"]+ if "token_usage" in result:+ tu = result["token_usage"]+ embedding = tu.get("embedding", {})+ llm = tu.get("llm", {})+ embed_total = embedding.get("total", embedding.get("total_tokens", 0))+ llm_total = llm.get("total", llm.get("total_tokens", 0))+ return {+ "embedding": embed_total,+ "vlm": llm_total,+ "llm_input": llm.get("input", 0),+ "llm_output": llm.get("output", 0),+ "total": tu.get("total", {}).get("total_tokens", embed_total + llm_total),+ }++ telemetry = commit_result.get("telemetry", {}).get("summary", {})+ tokens = telemetry.get("tokens", {})+ return {+ "embedding": tokens.get("embedding", {}).get("total", 0),+ "vlm": tokens.get("llm", {}).get("total", 0),+ "llm_input": tokens.get("llm", {}).get("input", 0),+ "llm_output": tokens.get("llm", {}).get("output", 0),+ "total": tokens.get("total", 0),+ }+++def _resolve_parallel(value: Optional[int], fallback: int) -> int:+ resolved = fallback if value is None else value+ if resolved < 1:+ raise ValueError("Parallelism must be >= 1")+ return resolved+++async def submit_viking_ingest(+ messages: List[Dict[str, Any]],+ openviking_url: str,+ submit_semaphore: asyncio.Semaphore,+ session_time: Optional[str] = None,+ agent_id: str = "default",+ user_id: str = "default",+ sample_id: Optional[str | int] = None,+ session_key: Optional[str] = None,+) -> Dict[str, Any]:+ base_datetime = None+ if session_time:+ try:+ base_datetime = datetime.strptime(session_time, LONGMEMEVAL_TIME_FORMAT)+ except ValueError:+ print(f"Warning: Failed to parse session_time: {session_time}", file=sys.stderr)++ async with submit_semaphore:+ client = ov.AsyncHTTPClient(url=openviking_url, user=user_id)+ await client.initialize()+ try:+ create_res = await client.create_session()+ session_id = create_res["session_id"]++ for idx, msg in enumerate(messages):+ msg_created_at = None+ if base_datetime:+ msg_dt = base_datetime + timedelta(seconds=idx)+ msg_created_at = msg_dt.isoformat()++ await client.add_message(+ session_id=session_id,+ role=msg["role"],+ parts=[{"type": "text", "text": msg["text"]}],+ created_at=msg_created_at,+ )++ result = await client.commit_session(session_id, telemetry=True)+ if result.get("status") not in ("committed", "accepted"):+ raise RuntimeError(f"Commit failed: {result}")++ trace_id = result.get("trace_id", "")+ return {+ "task_id": result.get("task_id"),+ "trace_id": trace_id,+ "agent_id": agent_id,+ "user_id": user_id,+ "session_id": session_id,+ }+ finally:+ await client.close()+++async def wait_for_viking_task(+ openviking_url: str,+ task_id: Optional[str],+ wait_semaphore: asyncio.Semaphore,+ agent_id: str = "default",+ user_id: str = "default",+ sample_id: Optional[str | int] = None,+ session_key: Optional[str] = None,+) -> Dict[str, int]:+ if not task_id:+ return {"embedding": 0, "vlm": 0, "llm_input": 0, "llm_output": 0, "total": 0}++ async with wait_semaphore:+ client = ov.AsyncHTTPClient(url=openviking_url, user=user_id)+ await client.initialize()+ try:+ while True:+ task = await client.get_task(task_id)+ status = task.get("status") if task else "unknown"+ if status == "completed":+ return _parse_token_usage(task)+ if status in ("failed", "unknown"):+ raise RuntimeError(f"Task {task_id} failed: {task}")+ await asyncio.sleep(1)+ finally:+ await client.close()+++async def viking_ingest(+ messages: List[Dict[str, Any]],+ openviking_url: str,+ submit_semaphore: asyncio.Semaphore,+ wait_semaphore: asyncio.Semaphore,+ session_time: Optional[str] = None,+ agent_id: str = "default",+ user_id: str = "default",+ sample_id: Optional[str | int] = None,+ session_key: Optional[str] = None,+) -> Dict[str, Any]:+ submit_result = await submit_viking_ingest(+ messages=messages,+ openviking_url=openviking_url,+ submit_semaphore=submit_semaphore,+ session_time=session_time,+ agent_id=agent_id,+ user_id=user_id,+ sample_id=sample_id,+ session_key=session_key,+ )+ token_usage = await wait_for_viking_task(+ openviking_url=openviking_url,+ task_id=submit_result.get("task_id"),+ wait_semaphore=wait_semaphore,+ agent_id=agent_id,+ user_id=user_id,+ sample_id=sample_id,+ session_key=session_key,+ )+ submit_result["token_usage"] = token_usage+ return submit_result+++def parse_session_range(s: str) -> Tuple[int, int]:+ if "-" in s:+ lo, hi = s.split("-", 1)+ return int(lo), int(hi)+ n = int(s)+ return n, n+++async def process_single_session(+ messages: List[Dict[str, Any]],+ sample_id: str | int,+ session_key: str,+ meta: Dict[str, Any],+ run_time: str,+ ingest_record: Dict[str, Any],+ args: argparse.Namespace,+ submit_semaphore: asyncio.Semaphore,+ wait_semaphore: asyncio.Semaphore,+) -> Dict[str, Any]:+ try:+ agent_id = build_sample_agent_id(sample_id)+ user_id = build_sample_user_id(sample_id)+ result = await viking_ingest(+ messages,+ args.openviking_url,+ submit_semaphore,+ wait_semaphore,+ meta.get("date_time"),+ agent_id=agent_id,+ user_id=user_id,+ sample_id=sample_id,+ session_key=session_key,+ )+ token_usage = result["token_usage"]+ task_id = result.get("task_id")+ trace_id = result.get("trace_id", "")+ embedding_tokens = token_usage.get("embedding", 0)+ vlm_tokens = token_usage.get("vlm", 0)+ print(+ f" -> [COMPLETED] [{sample_id}/{session_key}] embed={embedding_tokens}, vlm={vlm_tokens}, task_id={task_id}, trace_id={trace_id}, user_id={user_id}, agent_id={agent_id}",+ file=sys.stderr,+ )++ record = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "success",+ "meta": meta,+ "token_usage": token_usage,+ "embedding_tokens": embedding_tokens,+ "vlm_tokens": vlm_tokens,+ "task_id": task_id,+ "trace_id": trace_id,+ "user_id": user_id,+ "agent_id": agent_id,+ }+ write_success_record(record, args.success_csv)+ mark_ingested(sample_id, session_key, ingest_record, meta)+ save_ingest_record(ingest_record)+ return record+ except Exception as e:+ print(f" -> [ERROR] [{sample_id}/{session_key}] {e}", file=sys.stderr)+ traceback.print_exc(file=sys.stderr)+ record = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "error",+ "error": str(e),+ }+ write_error_record(record, args.error_log)+ return record+++async def finalize_deferred_session(+ pending: Dict[str, Any],+ run_time: str,+ ingest_record: Dict[str, Any],+ args: argparse.Namespace,+ wait_semaphore: asyncio.Semaphore,+) -> Dict[str, Any]:+ sample_id = pending["sample_id"]+ session_key = pending["session_key"]+ meta = pending["meta"]+ task_id = pending.get("task_id")+ trace_id = pending.get("trace_id", "")+ agent_id = pending.get("agent_id", "default")+ user_id = pending.get("user_id", "default")++ try:+ token_usage = await wait_for_viking_task(+ openviking_url=args.openviking_url,+ task_id=task_id,+ wait_semaphore=wait_semaphore,+ agent_id=agent_id,+ user_id=user_id,+ sample_id=sample_id,+ session_key=session_key,+ )+ embedding_tokens = token_usage.get("embedding", 0)+ vlm_tokens = token_usage.get("vlm", 0)+ print(+ f" -> [COMPLETED] [{sample_id}/{session_key}] embed={embedding_tokens}, "+ f"vlm={vlm_tokens}, task_id={task_id}, trace_id={trace_id}, user_id={user_id}, agent_id={agent_id}",+ file=sys.stderr,+ )+ record = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "success",+ "meta": meta,+ "token_usage": token_usage,+ "embedding_tokens": embedding_tokens,+ "vlm_tokens": vlm_tokens,+ "task_id": task_id,+ "trace_id": trace_id,+ "user_id": user_id,+ "agent_id": agent_id,+ }+ write_success_record(record, args.success_csv)+ mark_ingested(sample_id, session_key, ingest_record, meta)+ save_ingest_record(ingest_record)+ return record+ except Exception as e:+ print(f" -> [ERROR] [{sample_id}/{session_key}] {e}", file=sys.stderr)+ traceback.print_exc(file=sys.stderr)+ record = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "error",+ "error": str(e),+ "task_id": task_id,+ "user_id": user_id,+ "agent_id": agent_id,+ }+ write_error_record(record, args.error_log)+ return record+++async def run_import(args: argparse.Namespace) -> None:+ submit_parallel = _resolve_parallel(getattr(args, "submit_parallel", None), args.parallel)+ wait_parallel = _resolve_parallel(None, args.parallel)+ sample_parallel = submit_parallel+ submit_semaphore = asyncio.Semaphore(submit_parallel)+ wait_semaphore = asyncio.Semaphore(wait_parallel)+ sample_semaphore = asyncio.Semaphore(sample_parallel)+ session_range = parse_session_range(args.sessions) if args.sessions else None+ deferred_tasks: List[Dict[str, Any]] = []++ if args.clear_ingest_record:+ ingest_record = {}+ save_ingest_record(ingest_record)+ print("[INFO] All existing ingest records cleared", file=sys.stderr)+ else:+ ingest_record = load_ingest_record()++ success_keys = set()+ if not args.force_ingest:+ success_keys = load_success_csv(args.success_csv)+ print(+ f"[INFO] Loaded {len(success_keys)} existing success records from {args.success_csv}",+ file=sys.stderr,+ )++ run_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")+ samples = load_longmemeval_data(args.input, args.sample)++ async def process_sample(item):+ sample_id = item["question_id"]+ agent_id = build_sample_agent_id(sample_id)+ user_id = build_sample_user_id(sample_id)+ sessions = build_session_messages(item, session_range)++ print(f"\n=== Sample {sample_id} ===", file=sys.stderr)+ print(+ f" {len(sessions)} session(s) to import with user_id={user_id}, agent_id={agent_id}",+ file=sys.stderr,+ )++ for sess in sessions:+ meta = sess["meta"]+ messages = sess["messages"]+ session_key = meta["session_key"]+ label = f"{session_key} ({meta['date_time']})"++ if not args.force_ingest and is_already_ingested(+ sample_id, session_key, ingest_record, success_keys+ ):+ print(f" [{label}] [SKIP] already imported", file=sys.stderr)+ continue++ preview = " | ".join([f"{m['role']}: {m['text'][:30]}..." for m in messages[:3]])+ print(f" [{label}] {preview}", file=sys.stderr)++ if args.wait_mode == "immediate":+ await process_single_session(+ messages=messages,+ sample_id=sample_id,+ session_key=session_key,+ meta=meta,+ run_time=run_time,+ ingest_record=ingest_record,+ args=args,+ submit_semaphore=submit_semaphore,+ wait_semaphore=wait_semaphore,+ )+ continue++ try:+ submit_result = await submit_viking_ingest(+ messages=messages,+ openviking_url=args.openviking_url,+ submit_semaphore=submit_semaphore,+ session_time=meta.get("date_time"),+ agent_id=agent_id,+ user_id=user_id,+ sample_id=sample_id,+ session_key=session_key,+ )+ deferred_tasks.append(+ {+ "sample_id": sample_id,+ "session_key": session_key,+ "meta": meta,+ "task_id": submit_result.get("task_id"),+ "trace_id": submit_result.get("trace_id", ""),+ "user_id": user_id,+ "agent_id": agent_id,+ }+ )+ print(+ f" -> [SUBMITTED] [{sample_id}/{session_key}] "+ f"task_id={submit_result.get('task_id')}, user_id={user_id}, agent_id={agent_id}",+ file=sys.stderr,+ )+ except Exception as e:+ print(f" -> [ERROR] [{sample_id}/{session_key}] {e}", file=sys.stderr)+ traceback.print_exc(file=sys.stderr)+ record = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "error",+ "error": str(e),+ }+ write_error_record(record, args.error_log)++ async def process_sample_with_limit(item):+ async with sample_semaphore:+ await process_sample(item)++ tasks = [asyncio.create_task(process_sample_with_limit(item)) for item in samples]+ print(+ f"\n[INFO] Starting import with sample_parallel={sample_parallel}, "+ f"submit_parallel={submit_parallel}, wait_parallel={wait_parallel}, "+ f"{len(tasks)} sample task(s) to process",+ file=sys.stderr,+ )+ await asyncio.gather(*tasks, return_exceptions=True)++ if args.wait_mode == "deferred" and deferred_tasks:+ print(+ f"\n[INFO] Waiting for {len(deferred_tasks)} deferred task(s) to complete",+ file=sys.stderr,+ )+ await asyncio.gather(+ *[+ asyncio.create_task(+ finalize_deferred_session(+ pending=pending,+ run_time=run_time,+ ingest_record=ingest_record,+ args=args,+ wait_semaphore=wait_semaphore,+ )+ )+ for pending in deferred_tasks+ ]+ )+++def main():+ parser = argparse.ArgumentParser(description="Import LongMemEval conversations into OpenViking")+ parser.add_argument(+ "--input",+ default="data/longmemeval_s_cleaned.json",+ help="Path to LongMemEval JSON file",+ )+ parser.add_argument(+ "--success-csv",+ default="./result/longmemeval_import_success.csv",+ help="Path to success records CSV file",+ )+ parser.add_argument(+ "--error-log",+ default="./result/longmemeval_import_errors.log",+ help="Path to error log file",+ )+ parser.add_argument(+ "--openviking-url",+ default="http://localhost:1933",+ help="OpenViking service URL",+ )+ parser.add_argument(+ "--parallel",+ type=int,+ default=5,+ help="Default concurrency for sample processing, task waiting, and submissions when --submit-parallel is not set.",+ )+ parser.add_argument(+ "--submit-parallel",+ type=int,+ default=None,+ help="Max concurrent session submissions across samples. Defaults to --parallel.",+ )+ parser.add_argument(+ "--sample",+ type=int,+ default=None,+ help="LongMemEval sample index (0-based). Default all samples.",+ )+ parser.add_argument(+ "--sessions",+ default=None,+ help="Session range by haystack order, e.g. '1-4' or '3'. Default all sessions.",+ )+ parser.add_argument(+ "--force-ingest",+ action="store_true",+ default=False,+ help="Force re-import even if already recorded as completed",+ )+ parser.add_argument(+ "--clear-ingest-record",+ action="store_true",+ default=False,+ help="Clear all existing ingest records before running",+ )+ parser.add_argument(+ "--wait-mode",+ choices=("immediate", "deferred"),+ default="deferred",+ help="When to wait for background commit tasks: immediate or deferred.",+ )+ args = parser.parse_args()++ Path(args.success_csv).parent.mkdir(parents=True, exist_ok=True)+ Path(args.error_log).parent.mkdir(parents=True, exist_ok=True)++ try:+ asyncio.run(run_import(args))+ except ValueError as e:+ print(f"Error: {e}", file=sys.stderr)+ sys.exit(1)+++if __name__ == "__main__":+ main()diff --git a/benchmark/longmemeval/openviking/judge.py b/benchmark/longmemeval/openviking/judge.pynew file mode 100644index 000000000..4ef753794--- /dev/null+++ b/benchmark/longmemeval/openviking/judge.py@@ -0,0 +1,244 @@+import argparse+import asyncio+import csv+import json+import os+import sys+from pathlib import Path++from dotenv import load_dotenv+from openai import AsyncAzureOpenAI, AsyncOpenAI++try:+ from benchmark.longmemeval.openviking.longmemeval_prompts import (+ get_judge_prompt,+ get_strict_judge_prompt,+ )+except ModuleNotFoundError:+ sys.path.insert(0, str(Path(__file__).resolve().parent))+ from longmemeval_prompts import get_judge_prompt, get_strict_judge_prompt++env_file = Path.home() / ".openviking_benchmark_env"+load_dotenv(env_file)+csv.field_size_limit(sys.maxsize)+++DEFAULT_AZURE_API_VERSION = "2025-01-01-preview"+++def parse_judge_verdict(content: str) -> bool | None:+ stripped = content.strip()+ if not stripped:+ return None++ start_idx = stripped.find("{")+ end_idx = stripped.rfind("}")+ if start_idx != -1 and end_idx != -1:+ try:+ result = json.loads(stripped[start_idx : end_idx + 1].strip())+ label = str(result.get("is_correct", "")).strip().upper()+ if label == "CORRECT":+ return True+ if label == "WRONG":+ return False+ except json.JSONDecodeError:+ pass++ for line in reversed([line.strip() for line in stripped.splitlines() if line.strip()]):+ normalized = line.strip().strip(".。").lower()+ if normalized == "yes":+ return True+ if normalized == "no":+ return False+ return None+++async def grade_answer(+ llm_client,+ model: str,+ question: str,+ gold_answer: str,+ response: str,+ question_type: str = "",+ question_id: str = "",+ question_date: str = "",+ temperature: float | None = None,+ strict_prompt: bool = False,+) -> tuple[bool, str]:+ prompt_builder = get_strict_judge_prompt if strict_prompt else get_judge_prompt+ accuracy_prompt = prompt_builder(+ question_type=question_type,+ question_id=question_id,+ question=question,+ answer=gold_answer,+ response=response,+ question_date=question_date,+ )++ try:+ request_kwargs = {+ "model": model,+ "messages": [{"role": "user", "content": accuracy_prompt}],+ "timeout": 60,+ }+ if temperature is not None:+ request_kwargs["temperature"] = temperature++ resp = await llm_client.chat.completions.create(**request_kwargs)+ content = resp.choices[0].message.content.strip()+ verdict = parse_judge_verdict(content)+ if verdict is not None:+ return verdict, content+ return False, f"[PARSE ERROR] Invalid response: {content}"+ except Exception as e:+ return False, f"[API ERROR] {str(e)}"+++def load_answers(input_path: str) -> tuple[list[dict], list[str]]:+ if not os.path.exists(input_path):+ raise FileNotFoundError(f"Input file not found: {input_path}")++ with open(input_path, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ fieldnames = reader.fieldnames.copy()+ if "reasoning" not in fieldnames:+ fieldnames.append("reasoning")+ rows = list(reader)+ return rows, fieldnames+++def create_llm_client(+ provider: str,+ *,+ base_url: str,+ token: str,+ api_version: str | None = None,+):+ if provider == "azure":+ return AsyncAzureOpenAI(+ api_key=token,+ azure_endpoint=base_url,+ api_version=api_version or DEFAULT_AZURE_API_VERSION,+ )+ return AsyncOpenAI(base_url=base_url, api_key=token)+++def get_ungraded_rows(rows: list[dict], force: bool = False) -> list[int]:+ if force:+ for row in rows:+ row["result"] = ""+ row["reasoning"] = ""+ return list(range(len(rows)))+ return [i for i, row in enumerate(rows) if not row.get("result")]+++async def main():+ parser = argparse.ArgumentParser(description="VikingBot LongMemEval judge script")+ parser.add_argument(+ "--input",+ default="./result/longmemeval_qa_result.csv",+ help="Path to QA result csv file",+ )+ parser.add_argument(+ "--base-url",+ default="https://ark.cn-beijing.volces.com/api/v3",+ help="Judge model base URL",+ )+ parser.add_argument(+ "--provider",+ default=os.getenv("LONGMEMEVAL_JUDGE_PROVIDER", "openai"),+ choices=("openai", "azure"),+ help="Judge provider type, default: openai",+ )+ parser.add_argument(+ "--token",+ default=os.getenv("ARK_API_KEY", os.getenv("OPENAI_API_KEY", "")),+ help="Judge API token",+ )+ parser.add_argument(+ "--api-version",+ default=os.getenv("LONGMEMEVAL_JUDGE_API_VERSION"),+ help=f"Azure API version, default: {DEFAULT_AZURE_API_VERSION} for provider=azure",+ )+ parser.add_argument(+ "--model",+ default="doubao-seed-2-0-pro-260215",+ help="Judge model name",+ )+ parser.add_argument(+ "--temperature",+ type=float,+ default=None,+ help="Optional judge temperature. Omitted by default for models that only support their default value.",+ )+ parser.add_argument(+ "--parallel", type=int, default=8, help="Parallel request count, default: 8"+ )+ parser.add_argument(+ "--force",+ action="store_true",+ help="Force re-grade all rows even if result already exists",+ )+ parser.add_argument(+ "--strict-prompt",+ action="store_true",+ help="Use the strict LongMemEval judge prompt instead of the default lenient prompt",+ )+ args = parser.parse_args()++ if not args.token:+ print("Error: API token is required")+ raise SystemExit(1)++ rows, fieldnames = load_answers(args.input)+ total = len(rows)+ ungraded = get_ungraded_rows(rows, force=args.force)+ print(f"Total answers: {total}, ungraded: {len(ungraded)}")++ if not ungraded:+ print("All answers already graded, exit")+ return++ client = create_llm_client(+ args.provider,+ base_url=args.base_url,+ token=args.token,+ api_version=args.api_version,+ )+ semaphore = asyncio.Semaphore(args.parallel)+ file_lock = asyncio.Lock()++ async def save_results():+ async with file_lock:+ temp_file = f"{args.input}.tmp"+ with open(temp_file, "w", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=fieldnames)+ writer.writeheader()+ writer.writerows(rows)+ os.replace(temp_file, args.input)++ async def process_row(idx):+ async with semaphore:+ row = rows[idx]+ is_correct, reasoning = await grade_answer(+ client,+ args.model,+ row["question"],+ row["answer"],+ row["response"],+ question_type=row.get("question_type", ""),+ question_id=row.get("sample_id", ""),+ question_date=row.get("question_time", ""),+ temperature=args.temperature,+ strict_prompt=args.strict_prompt,+ )+ row["result"] = "CORRECT" if is_correct else "WRONG"+ row["reasoning"] = reasoning+ await save_results()+ print(f"Saved result for {idx + 1}/{total}: {row['result']}")++ await asyncio.gather(*(process_row(idx) for idx in ungraded))+++if __name__ == "__main__":+ asyncio.run(main())diff --git a/benchmark/longmemeval/openviking/longmemeval_prompts.py b/benchmark/longmemeval/openviking/longmemeval_prompts.pynew file mode 100644index 000000000..600989c35--- /dev/null+++ b/benchmark/longmemeval/openviking/longmemeval_prompts.py@@ -0,0 +1,385 @@+from __future__ import annotations++from datetime import datetime as _datetime+from typing import Any++QUESTION_TYPES = [+ "temporal-reasoning",+ "multi-session",+ "knowledge-update",+ "single-session-user",+ "single-session-assistant",+ "single-session-preference",+]+++ANSWER_GENERATION_PROMPT = """You are a personal assistant with access to memories from past conversations with a user. Answer the question using information from the memories below. Be direct and concise.++IMPORTANT: Today's date is {question_date}. All relative time expressions MUST be computed relative to this date.++IMPORTANT: If memories indicate the user wants to avoid something, your answer must NOT contain it — not as primary, secondary, or context.++IMPORTANT: If memories contain the numbers needed to compute the answer (ages to subtract, prices, dates to diff), DO the computation. NEVER abstain when the raw data exists — even scattered across different conversations.++IMPORTANT: Keep your responses short. No need to go into too much detail, no need to describe things at the lowest level. You can generally describe events and ideas abstractly.++IMPORTANT: Pay close attention to the EXACT entity in the question. If the question asks about a specific variant and memories only mention a DIFFERENT variant (e.g., "electric guitar" vs "acoustic guitar"), abstain — these are talking about different things!++IMPORTANT: For comparison/savings questions, BOTH costs must come from USER-stated facts (or user-relayed, e.g., "my friend said"). Do NOT use assistant-provided general info. If only one side has a user-stated cost, abstain.++IMPORTANT: If the query uses a specific but WRONG role/title/entity (e.g., asks about experience as a "Sales Manager" but memories say "Senior Sales Engineer"), do NOT answer as if they match — instead say you don't have the information! Always lean towards abstention in these cases! Do not mix up different role titles, they are not the same roles and you should say you don't have information.++Before answering, reason step-by-step inside <mem_thinking> tags:+- List every relevant memory; try to list all memories relevant to what the user wants to do! Eg. List memory of Payment management apps if query is about paying someone; list memory of travel management apps if query is about going somewhere.++- For counting: enumerate each item with date. Apply the question's EXACT verb/qualifier strictly (e.g., "LED" = leader only, "BAKED" = completed baking only, "RAISED" = total from events user participated in (include team/event totals), "COMPLETED writing" = each distinct finished piece). Count multiple items in a single memory separately. Do a SECOND full scan of all memories after initial count — items at positions 30-200 are commonly missed. Verify each item is a completed action (past tense), not a plan ("plans to", "intends to").+- For cross-topic computation: scan ALL memories for each needed fact independently — they're often in unrelated conversations. List: (a) what you need, (b) where each appears, (c) the computation.+- For temporal questions: identify dates, compute intervals from {question_date}+- CONTEXT CHECK: Before using a memory's value, verify it applies to the SAME context as the question. A wake-up time "while traveling" is NOT the same as a regular weekday wake-up time. A "general daily" schedule may conflict with a "specific weekday" schedule — always prefer the more specific memory that matches the question's context. List the context of each memory (weekday routine vs. travel vs. weekend vs. specific day) and only use values from the matching context.+- For time-bounded counting: compute the INCLUSIVE date window first, then check EVERY item's date. Err on inclusion for ambiguous dates.+- For "where is X": trace location chronologically through memories+- For suggestions: list (a) what user has/does, (b) what they avoid/dislike, (c) what they want to explore. Check every suggestion against (b) before including.+- State your conclusion++The user will only see text outside the <mem_thinking> tags.++Rules:++1. **Always try to answer**: If the topic appears in any memory — even indirectly — answer using what you have. Don't refuse for one missing detail.++2. **Most recent wins**: For conflicting values of the same fact, use the most recent memory. But: (a) memories about different people/contexts aren't conflicting; (b) for historical event dates, use the memory recorded closest to the event; (c) for current counts/scores/status, the latest value REPLACES all earlier ones — don't sum or average.++Similarly, when memories give two numbers for the same metric (e.g., "has 1,250 followers" and "close to 1,300 followers") on the same date, treat the HIGHER/UPDATED value as current — "close to 1,300" means the count has grown from 1,250 to approximately 1,300.++3. **Time-bounded questions**: Compute the date window from {question_date}. Show date arithmetic in <mem_thinking>. Scan EVERY memory for events in range. "Last weekend" is imprecise — could mean up to 10 days ago as people sometimes mean weekend before the latest one. "Last 3 months" can include boundary days of the 4th month back.++"Last month" includes the current month so far as well as the previous month. Eg. "last month" in Late May includes all of April. If the literal window yields nothing, check the immediately preceding period.++4. **Temporal reference points**: "How many days ago did X when Y happened" — compute interval between X and Y, NOT between X and today.++5. **Counting and ordering**: Scan ALL memories first to last. Build a numbered list in <mem_thinking> with date and position. Deduplicate by matching dates/descriptions. Count items in a single memory separately.+Any addition to a list on the same day as a stated count is already included in the count++When asked to count all instances of an event *before* a specific one, obviously don't include the specific one in the count. Eg. "how many restaurants did i visit before eating at Pizza Hut?". Obviously don't include Pizza Hut in the count++6. **Use only the memories**: Don't invent numbers, prices, or addresses.++7. **When to abstain**: Say "The information provided is not enough" when:+ - The topic is genuinely unmentioned++- The question asks about a specific event that doesn't exist, even if a related topic does++- IMPORTANT: If the query uses a specific but WRONG role/title/entity (e.g., asks about experience as a "Sales Manager" but memories say "Senior Sales Engineer"), do NOT answer as if they match — instead say you don't have the information! Always lean towards abstention in these cases! Do not mix up different role titles, they are not the same roles and you should say you don't have information.++ - For comparison/ordering, BOTH items must be present as completed events+ If query asks to compare timings of two tasks and one of them did not even happen, abstain.+ Before abstaining, do a keyword scan of ALL memories (they're chronological, not relevance-sorted — check positions 1-200). Only abstain if NO keywords match.+ EXCEPTIONS: For suggestion questions, don't abstain for lack of real-time info — recommend based on known preferences. If you lack exact brand but have the store, output the store.++8. **Yes/no and comparison**: "Did I ever do X?" with no matching memory = "No." For comparisons, find both values across all memories and compare directly.++9. **Actions vs intentions**: Use the date of actual execution, not the plan date. "Decided to" or "took X for servicing" = action initiated. Only treat as plan if explicit future-tense ("plans to", "will"). A plan with a specified date and no update = assume completed on that date. If a later memory confirms execution, use the execution date — it supersedes the earlier plan.++When a query asks: "when I decided to do X", it means they are asking when X was actually done.++10. **User facts vs assistant advice**: "User..." = actual experience. "Assistant..." = advice. Prefer user-stated facts for personal questions. Don't convert currencies unless user stated the conversion.++11. **Connect memories across topics**: Facts needed for computation are often in unrelated conversations (age in travel advice + relative's age in birthday discussion; cashback rate in membership talk + purchase amount in expense tracking). Search ALL memories for each fact independently.++12. **Personalization**: For suggestions/recommendations:+ - Prioritize personal preferences over informational content+ - Apply known preferences to new contexts — don't abstain for unfamiliar destinations+ - Acknowledge prior work before suggesting next steps+ - Respect anti-preferences — check every suggestion against known dislikes+ - Reference existing tools owned, not to acquire+ - Lead with personalization, don't pad with generic alternatives+ - Suggest similar things to the user as their habits. Eg. Logging basketball scores in a app they do usually. Eg. Adding travel logs to a travel logging app they use usually.+ - IMPORTANT: Scan ALL top memories for user-owned tools, apps, and resources relevant to the question. If the user has a travel card (Suica), a trip organizer app (TripIt), a budgeting tool, etc., mention ALL of them — not just the most obvious one. Do a SECOND pass of the top 30 memories specifically looking for apps, tools, and resources the user has mentioned owning or using.++13. **Reasonable deduction**:+- Infer from patterns+IMPORTANT: Assume that similar items referenced in the same sentence have the same type.+Eg. "User ate lunch, which was the third meal with this chicken fajitas". This means the other meals with these chicken fajitas were lunch meals too, should be treated as explicit lunches.++14. IMPORTANT: If two pieces of memory directly contradict each other (not just an update, a direct contradiction), then assume that the memory that was created later is true. Doesn't matter if a different one "appears" more reliable. If on the same day, trust the one at a later time.++- Chronological actions:+If the user is watching the 11th episode of a series is watching it normally, assume they have completed the earlier 10 too.++- If you lack a name but have a description, answer with the description.++**Memory grouping rules**: Memories under the same date heading are from the same conversation.+- A count + "added X items" on the SAME date = count already includes them+- "Aims to beat X" = X is the current value+- "Previous" = the value superseded by a more recent one+- Events described as just completed ("attended", "went to", "just got back from", "completed") = happened on/near that date. Undated actions = assume the event happened on the memory's date.++# Misc Rules+- Count class projects too when asked about users' projects. Class projects = projects.+- Most old (Eg. ancestral, vintage, heritage) items count as antiques too!+- If you don't have chords for a song (but have notes), output the notes. Song notes count as chord progressions.+- Starting a *diorama project* (eg. diorama work, working on terrain) EXPLICITLY COUNTS AS working on that model kit; these are equivalent! Always count such items.+- Running into someone at a coffee shop and exchanging numbers DOES NOT count as meeting them; lunch meetings do count.+- Potlucks/feasts/birthday parties count as dinner parties (BBQ doesn't).+- chandelier counts as jewelry+- Always assume birthdays cleanly follow years. Ie. User was 22 in 2022; they will be 23 in 2023.+- "scratch grains" count as "new layer feed", always include them when interpreting "new layer feed"++Memories (sorted newest-first, grouped by date):+{memories}++Today's Date: {question_date}+Question: {question}++IMPORTANT: You MUST provide your full thinking in <mem_thinking> tags BEFORE giving your answer.; Reasoning and answer:"""+++JUDGE_PROMPT = """I will give you a question, a correct answer (or rubric), and a model response. Decide whether the model response is correct.++CORE PRINCIPLE — Semantic equivalence: Judge by MEANING, not exact words. Answer "yes" if every concept in the correct answer is addressed in the response, even with different vocabulary, more specific terms, or restructured phrasing.++IMPORTANT BIAS CHECK: You have a tendency to say "no" too quickly. Before concluding "no", you MUST verify the answer is truly wrong, not just differently worded. When in doubt, lean toward "yes".++Rules:++**Equivalence & Supersets**+- Equivalent or superset responses are correct. Extra details are fine unless proven to be factually wrong. Extra qualifiers are fine unless proven to be wrong. E.g., "a blue dress and a matching necklace" is correct when the answer is "a blue dress."+- If a response captures the most specific part (exact item/place/name) but omits a broader container, it's correct.+- Same factual meaning with different phrasing = correct (e.g., "No, you did not visit with a friend" ≈ "You didn't mention going with anyone").+- Adding scope qualifiers like "regular-season" or "excluding X" is fine as long as the core value is correct. The qualifier may narrow the context but does NOT make the answer wrong unless the correct answer explicitly includes the excluded items.++**Lists & Compound Terms**+- For list answers, match each item by semantic meaning. A concept is covered if restated via synonyms, sub-concepts, or related terms. Adding methodological detail or rewording verbs to near-synonyms is acceptable.+- A broad term like "A and B significance" is covered if the response addresses the topic area through related specific terms, even without naming each component literally.+- If some items as listed as "or"s, "maybe"s and potential answers, it's okay if the answer does not include those.+- If two items in a list achieve the same purpose, listing just one of them is fine.++IMPORTANT: The "anti-preference" items are very specific!+Eg. Someone "not interested in general AI topics" could be very interested in specific AI topics in general AI *conferences*; those are not the same thing and should be accepted! topics != conferences++**Numbers & Precision**+- Hedging ("at least 3", "approximately") is fine if the core number matches. A range that includes the correct answer is correct.+Generally, if the user themself would be satisfied by the response, it is acceptable. Ie. If the answer is conditional on information they would have (eg. their birthday, some hidden dependent information), and would be correct with that information, that is acceptable.+- More precise answers are correct: "22 days" matches "3 weeks"; "over $270" matches "$270."; "9 1/2 months" matches "9 months";++- Rough answers are correct: "about nine months" ≈ "9 months; "8 months and 20 days" matches "9 months";++- Off-by-one errors on days/weeks/months are acceptable.+- Approximate unit conversions are equivalent: "14 weeks" ≈ "3 months", "6 months" ≈ "half a year."+- Round time ranges generously: 7 months and 16 days ≈ 8 months.+- Notes instead of chords are acceptable when justified+- A correct number with added context (e.g., "about 5 months ago (around December 2022)") is correct — the parenthetical date is supplementary, not a contradiction.++**Dates & Temporal**+- Date format variations are equivalent: "February 1st" = "Feb 1, 2023" = "on February 1."+- Same-day event ordering swaps are acceptable.+- Outdated info alongside the correct updated answer is acceptable if the current value is identified.+- "recent" is upto 6 years ago, which means 2017++- References like "last weekend", "last Wednesday", etc. are imprecise - people sometimes mean the weekend/Wednesday before the latest one if they're near it. "Last 3 months" can include boundary days of the 4th month back. "Last month" includes the current month so far. Be flexible with such timestamps++**Counting Edge Cases**+- If correct answer is "0" or "nothing found," model saying "not enough information" is also correct.+- Similarly, If correct answer is "not enough information", model saying "0" or "nothing found," is also correct.++**Preference/Personalization Rubrics** (apply in order):+1. Correct if the response demonstrates awareness of user's personal context (preferences, habits, interests). Need not satisfy every rubric point.+2. Primary criterion: do main suggestions align with what the user WANTS?+3. Anti-preferences: evaluate the OVERALL thrust, not keyword scanning. If the response largely suggests correct options, minor incidental references to "not-preferred" things are fine.+4. Mentioning a phone app as a MEANS to a preferred activity (e.g., meditation app for sleep) is not "suggesting phone use." Judge by the activity, not delivery mechanism.+5. "May not prefer" = mild preference, not hard prohibition. Secondary/context-dependent inclusion is fine.+6. Explicit acknowledgment of anti-preferences (e.g., "keep screens off") strengthens correctness.+7. Context-dependent suggestions are acceptable (reading is fine on a bus even if rubric flags visual attention activities). Adjacent genres alongside preferred ones are additive, not contradictory.+8. If the rubric mentions specific user resources/tools (e.g., "Suica card", "TripIt app"), the response is correct if it demonstrates awareness of the user's MAIN personal context even if it does not name every specific tool. The rubric is a guide, not a checklist.++**Abstention Matching**+- If correct answer = unanswerable/abstention, ANY phrasing that conveys "I don't have this information" is correct, regardless of what partial context is mentioned or omitted.+- Saying "not enough information" while mentioning partial related context = correct abstention.+- Saying "no record of X" or "only have plans for X, not actual dates" = correct abstention.+- The key test: does the response REFUSE to answer the question? If yes, it matches an abstention ground truth, period.++FINAL CHECK: Before answering "no," you MUST reason through these steps:+1. What is the core factual claim or intent of the correct answer?+2. Does the model response address that same claim, even in different words?+3. Is the response a superset (correct answer + extra details)?+4. For numbers: does the core number match, ignoring hedging/qualifiers?+5. For abstentions: does the response effectively decline to answer?+Only answer "no" if, after this analysis, a core concept is entirely unaddressed or contradicted.++Question: {question}++Correct Answer: {answer}++Model Response: {response}++Think step-by-step in <judge_thinking> tags, then give your final verdict as exactly "yes" or "no" on a new line after the closing tag."""+++STRICT_JUDGE_PROMPT = """I will give you a LongMemEval question, the gold answer or rubric, and a model response. Decide whether the model response is correct.++Use a strict benchmark grading standard. Judge the response against the gold answer, not against what might be plausible from memory.++Rules:++1. **Gold answer is the target**: The response must include the key fact(s), constraint(s), and intent in the gold answer. Do not mark a response correct just because it is related to the same topic.++2. **Question type matters**:+- For temporal-reasoning questions, dates, order, recency, durations, and relative-time calculations must be correct.+- For multi-session and knowledge-update questions, the response must combine the relevant facts and use the latest/current fact when the gold answer depends on an update.+- For single-session-user and single-session-assistant questions, the response must identify the exact requested user/assistant-provided fact.+- For preference questions, the response must preserve positive and negative preferences; suggesting or including an anti-preference is wrong.++3. **Lists and compound answers**: If the gold answer contains multiple required items, the response must include all or nearly all essential items. A single correct item from a multi-item answer is not enough.++4. **Numbers and counts**: Counts, quantities, prices, scores, ages, and computed values must match the gold answer. Ranges or approximations are acceptable only when they clearly contain the gold value and do not change the conclusion.++5. **Dates and durations**: Exact dates must match when the gold answer is specific. Do not allow broad date windows, off-by-one errors, or swapped event ordering if the question asks for a specific time or temporal relation.++6. **Abstention**: If the gold answer is an abstention or says the information is unavailable, the response is correct only if it clearly abstains. If the gold answer contains a concrete fact, an abstention is wrong.++7. **Paraphrases count only when meaning is preserved**: Different wording is fine, but broader, narrower, merely related, or partially contradictory statements are wrong.++8. **Extra detail is allowed only after the core answer is correct**: Extra detail cannot compensate for missing the requested fact. If extra detail contradicts the gold answer, mark wrong.++Mark "yes" only when the response directly answers the question and matches the essential gold answer. Mark "no" when a key fact is missing, wrong, too vague, or only topically related.++Question Type: {question_type}+Question ID: {question_id}+Question Date: {question_date}++Question: {question}++Correct Answer: {answer}++Model Response: {response}++Think step-by-step in <judge_thinking> tags, then give your final verdict as exactly "yes" or "no" on a new line after the closing tag."""+++def _to_human_date(iso_str: str) -> str:+ try:+ from datetime import timezone as _tz++ for fmt in ("%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%dT%H:%M:%S.%f%z"):+ try:+ dt = _datetime.strptime(iso_str.replace("Z", "+0000"), fmt)+ return dt.astimezone(_tz.utc).strftime("%A, %B %d, %Y")+ except ValueError:+ continue+ except Exception:+ pass++ for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%d"):+ try:+ return _datetime.strptime(iso_str[:19], fmt).strftime("%A, %B %d, %Y")+ except ValueError:+ continue+ return iso_str[:10]+++def _format_user_profile(user_profile: dict[str, Any]) -> str:+ lines = ["## User Profile"]+ for key, value in user_profile.items():+ if value is None:+ continue+ display_key = key.replace("_", " ").title()+ if isinstance(value, list):+ if not value:+ continue+ display_value = ", ".join(str(v) for v in value)+ elif isinstance(value, str):+ if not value.strip():+ continue+ display_value = value+ else:+ display_value = str(value)+ lines.append(f"{display_key}: {display_value}")+ if len(lines) <= 1:+ return ""+ return "\n".join(lines)+++def format_memories_for_prompt(+ search_results: list[dict[str, Any]],+ user_profile: dict[str, Any] | None = None,+) -> str:+ if not search_results:+ memories_text = "(No relevant memories found)"+ else:+ lines = []+ current_date = None+ for result in search_results:+ memory = str(result.get("memory", ""))+ uri = str(result.get("uri", ""))+ created_at = str(result.get("created_at", ""))+ if created_at:+ date_str = _to_human_date(created_at)+ if date_str != current_date:+ current_date = date_str+ lines.append(f"\n--- {date_str} ---")+ prefix = f"[{uri}] " if uri else ""+ lines.append(f"- {prefix}{memory}")+ elif uri:+ lines.append(f"\n--- {uri} ---")+ lines.append(memory)+ else:+ lines.append(f"- {memory}")+ memories_text = "\n".join(lines).strip()++ profile_section = ""+ if user_profile:+ profile_section = _format_user_profile(user_profile)+ if profile_section:+ profile_section += "\n\n"++ return profile_section + memories_text+++def get_answer_generation_prompt(+ question: str,+ search_results: list[dict[str, Any]],+ question_date: str,+ user_profile: dict[str, Any] | None = None,+) -> str:+ memories_text = format_memories_for_prompt(search_results, user_profile)++ return ANSWER_GENERATION_PROMPT.format(+ memories=memories_text,+ question_date=question_date or "unknown",+ question=question,+ )+++def get_judge_prompt(+ question_type: str,+ question_id: str,+ question: str,+ answer: str,+ response: str,+ question_date: str = "",+) -> str:+ return JUDGE_PROMPT.format(+ question_type=question_type or "",+ question_id=question_id or "",+ question=question,+ answer=str(answer),+ response=response,+ question_date=question_date or "",+ )+++def get_strict_judge_prompt(+ question_type: str,+ question_id: str,+ question: str,+ answer: str,+ response: str,+ question_date: str = "",+) -> str:+ return STRICT_JUDGE_PROMPT.format(+ question_type=question_type or "",+ question_id=question_id or "",+ question=question,+ answer=str(answer),+ response=response,+ question_date=question_date or "",+ )diff --git a/benchmark/longmemeval/openviking/run_eval.py b/benchmark/longmemeval/openviking/run_eval.pynew file mode 100644index 000000000..46a6c2997--- /dev/null+++ b/benchmark/longmemeval/openviking/run_eval.py@@ -0,0 +1,716 @@+import argparse+import csv+import hashlib+import json+import sys+import threading+import time+from concurrent.futures import ThreadPoolExecutor, as_completed+from datetime import datetime+from pathlib import Path+from typing import Any++from openviking_cli.client.sync_http import SyncHTTPClient++try:+ from benchmark.longmemeval.openviking.longmemeval_prompts import (+ get_answer_generation_prompt,+ )+except ModuleNotFoundError:+ sys.path.insert(0, str(Path(__file__).resolve().parent))+ from longmemeval_prompts import get_answer_generation_prompt+++LONGMEMEVAL_TIME_FORMAT = "%Y/%m/%d (%a) %H:%M"+DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT = 10+DEFAULT_SINGLE_SEARCH_RERANK_LIMIT = 10+DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS = 4000+SINGLE_SEARCH_EXCLUDED_BASENAMES = {".abstract.md", ".overview.md"}+++def get_token_encoding_name(model_name: str | None = None) -> str:+ try:+ import tiktoken++ if model_name:+ candidates = [model_name]+ if "/" in model_name:+ candidates.append(model_name.rsplit("/", 1)[-1])+ for candidate in candidates:+ try:+ return tiktoken.encoding_for_model(candidate).name+ except KeyError:+ continue+ except Exception:+ pass++ normalized = (model_name or "").lower()+ if normalized.startswith(("gpt-4o", "gpt-5", "o1", "o3", "o4")):+ return "o200k_base"+ if normalized.startswith(("gpt-4", "gpt-3.5")):+ return "cl100k_base"+ return "approx_chars_div_4"+++def count_text_tokens(text: str, model_name: str | None = None) -> int:+ encoding_name = get_token_encoding_name(model_name)+ if encoding_name == "approx_chars_div_4":+ return max(0, len(text or "") // 4)++ try:+ import tiktoken++ encoding = tiktoken.get_encoding(encoding_name)+ return len(encoding.encode(text or ""))+ except Exception:+ return max(0, len(text or "") // 4)+++def parse_longmemeval_datetime(date_str: str) -> datetime | None:+ try:+ return datetime.strptime(date_str.strip(), LONGMEMEVAL_TIME_FORMAT)+ except ValueError:+ return None+++def build_sample_agent_id(sample_id: str | int) -> str:+ """Return the agent_id used for one sample eval."""+ digest = hashlib.md5(str(sample_id).encode("utf-8")).hexdigest()[:12]+ return f"lm_{digest}"+++def build_sample_user_id(sample_id: str | int) -> str:+ """Return the user_id used for one sample eval."""+ digest = hashlib.md5(f"user:{sample_id}".encode("utf-8")).hexdigest()[:12]+ return f"lm_user_{digest}"+++def load_csv_qa(input_path: str, count: int | None = None) -> list[dict]:+ qa_list = []+ with open(input_path, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ for row in reader:+ qa_list.append(+ {+ "sample_id": row.get("sample_id", row.get("question_id", "")),+ "question": row.get("question", ""),+ "answer": row.get("answer", ""),+ "question_type": row.get("question_type", ""),+ "question_time": row.get("question_time", ""),+ }+ )+ if count is not None:+ qa_list = qa_list[:count]+ return qa_list+++def load_longmemeval_qa(+ input_path: str,+ sample_index: int | None = None,+ count: int | None = None,+) -> list[dict]:+ if input_path.lower().endswith(".csv"):+ return load_csv_qa(input_path, count)++ with open(input_path, "r", encoding="utf-8") as f:+ data = json.load(f)++ if sample_index is not None:+ if sample_index < 0 or sample_index >= len(data):+ raise ValueError(f"sample index {sample_index} out of range (0-{len(data) - 1})")+ samples = [data[sample_index]]+ else:+ samples = data++ qa_list = []+ for sample in samples:+ question_dt = parse_longmemeval_datetime(sample.get("question_date", ""))+ question_time = question_dt.strftime("%Y-%m-%d") if question_dt else ""+ qa_list.append(+ {+ "sample_id": sample.get("question_id", ""),+ "question": sample.get("question", ""),+ "answer": sample.get("answer", ""),+ "question_type": sample.get("question_type", ""),+ "question_time": question_time,+ }+ )++ if count is not None:+ qa_list = qa_list[:count]+ return qa_list+++def _iter_search_contexts(search_result: Any) -> list[Any]:+ if search_result is None:+ return []+ if isinstance(search_result, list):+ return search_result++ contexts = []+ for attr in ("memories", "resources", "skills"):+ contexts.extend(getattr(search_result, attr, []) or [])+ if contexts:+ return contexts++ try:+ return list(search_result)+ except TypeError:+ return []+++def is_single_search_excluded_uri(uri: str) -> bool:+ basename = str(uri or "").rstrip("/").rsplit("/", 1)[-1]+ return basename in SINGLE_SEARCH_EXCLUDED_BASENAMES+++def select_single_search_contexts(+ search_result: Any,+ limit: int = DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT,+) -> list[dict[str, Any]]:+ selected = []+ for raw_rank, context in enumerate(_iter_search_contexts(search_result), start=1):+ if raw_rank > limit:+ break+ uri = getattr(context, "uri", "")+ if not uri or is_single_search_excluded_uri(uri):+ continue+ selected.append(+ {+ "raw_rank": raw_rank,+ "uri": uri,+ "score": getattr(context, "score", 0.0),+ "abstract": getattr(context, "abstract", ""),+ }+ )+ return selected+++def build_single_search_context_prompt(+ question: str,+ question_type: str | None,+ question_time: str | None,+ contexts: list[dict[str, Any]],+) -> str:+ search_results = build_single_search_prompt_search_results(contexts)++ prompt = get_answer_generation_prompt(+ question=question,+ search_results=search_results,+ question_date=question_time or "unknown",+ )+ if question_type:+ return f"Question Type: {question_type}\n\n{prompt}"+ return prompt+++def build_single_search_prompt_search_results(+ contexts: list[dict[str, Any]],+) -> list[dict[str, Any]]:+ return [+ {+ "memory": str(context.get("content", "")),+ "score": context.get("score", 0.0),+ "raw_rank": context.get("raw_rank"),+ }+ for context in contexts+ ]+++def count_retrieved_memory_content_tokens(+ contexts: list[dict[str, Any]],+ model_name: str | None = None,+) -> tuple[int, int, str]:+ memory_texts = [str(context.get("content", "")) for context in contexts]+ tokenizer = get_token_encoding_name(model_name)+ return (+ sum(count_text_tokens(memory_text, model_name) for memory_text in memory_texts),+ sum(len(memory_text) for memory_text in memory_texts),+ tokenizer,+ )+++def filter_contexts_by_char_budget(+ contexts: list[dict[str, Any]],+ max_chars: int = DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS,+) -> tuple[list[dict[str, Any]], list[str], int]:+ if max_chars <= 0:+ return contexts, [], sum(len(str(context.get("content", ""))) for context in contexts)++ selected = []+ skipped_uris = []+ total_chars = 0+ for context in contexts:+ content = str(context.get("content", "") or "")+ content_chars = len(content)+ if total_chars + content_chars > max_chars:+ skipped_uris.append(str(context.get("uri", "")))+ continue+ selected.append(context)+ total_chars += content_chars+ return selected, skipped_uris, total_chars+++def build_single_search_reranker() -> Any | None:+ from openviking.models.rerank import RerankClient+ from openviking_cli.utils.config import get_openviking_config++ rerank_config = get_openviking_config().rerank+ if not rerank_config or not rerank_config.is_available():+ return None+ return RerankClient.from_config(rerank_config)+++def _single_search_rerank_document(context: dict[str, Any]) -> str:+ content = str(context.get("content", "") or "")+ if content and not content.startswith("[READ ERROR]"):+ return content+ return str(context.get("abstract", "") or "")+++def rerank_single_search_contexts(+ question: str,+ contexts: list[dict[str, Any]],+ reranker: Any | None,+ limit: int = DEFAULT_SINGLE_SEARCH_RERANK_LIMIT,+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:+ if not reranker or len(contexts) <= 1:+ return contexts, []++ documents = [_single_search_rerank_document(context) for context in contexts]+ try:+ scores = reranker.rerank_batch(question, documents)+ except Exception:+ return contexts, []++ if not scores or len(scores) != len(contexts):+ return contexts, []++ ranked_contexts = []+ rerank_scores = []+ for context, score in zip(contexts, scores, strict=True):+ if not isinstance(score, (int, float)):+ return contexts, []+ next_context = dict(context)+ next_context["rerank_score"] = float(score)+ ranked_contexts.append(next_context)+ rerank_scores.append(+ {+ "uri": str(context.get("uri", "")),+ "score": float(score),+ "raw_rank": context.get("raw_rank"),+ }+ )++ ranked_contexts.sort(+ key=lambda context: (+ context.get("rerank_score", 0.0),+ -int(context.get("raw_rank", 0) or 0),+ ),+ reverse=True,+ )+ selected_contexts = ranked_contexts[: max(1, limit)]+ return selected_contexts, rerank_scores+++def build_single_search_vlm() -> Any:+ from openviking.models.vlm import VLMFactory+ from openviking_cli.utils.config import get_openviking_config++ vlm_config = get_openviking_config().vlm+ return VLMFactory.create(vlm_config._build_vlm_config_dict())+++def _response_text(response: Any) -> str:+ if hasattr(response, "content"):+ return str(response.content or "").strip()+ return str(response).strip()+++def _token_usage_from_vlm(vlm: Any, response: Any = None) -> dict[str, int]:+ usage = getattr(response, "usage", None)+ if isinstance(usage, dict) and usage:+ return {+ "prompt_tokens": int(usage.get("prompt_tokens", 0) or 0),+ "completion_tokens": int(usage.get("completion_tokens", 0) or 0),+ "total_tokens": int(usage.get("total_tokens", 0) or 0),+ }+ if usage:+ prompt_tokens = int(getattr(usage, "prompt_tokens", 0) or 0)+ completion_tokens = int(getattr(usage, "completion_tokens", 0) or 0)+ return {+ "prompt_tokens": prompt_tokens,+ "completion_tokens": completion_tokens,+ "total_tokens": int(+ getattr(usage, "total_tokens", prompt_tokens + completion_tokens) or 0+ ),+ }++ if hasattr(vlm, "get_token_usage_summary"):+ summary = vlm.get_token_usage_summary()+ return {+ "prompt_tokens": int(summary.get("total_prompt_tokens", 0) or 0),+ "completion_tokens": int(summary.get("total_completion_tokens", 0) or 0),+ "total_tokens": int(summary.get("total_tokens", 0) or 0),+ }+ if hasattr(vlm, "get_token_usage"):+ usage_snapshot = vlm.get_token_usage()+ total_usage = usage_snapshot.get("total_usage", {}) if usage_snapshot else {}+ return {+ "prompt_tokens": int(total_usage.get("prompt_tokens", 0) or 0),+ "completion_tokens": int(total_usage.get("completion_tokens", 0) or 0),+ "total_tokens": int(total_usage.get("total_tokens", 0) or 0),+ }++ return {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}+++def run_single_search_context_answer(+ question: str,+ question_type: str | None = None,+ question_time: str | None = None,+ sender_id: str | None = None,+ session_id: str | None = None,+ openviking_url: str | None = None,+ timeout: int = 300,+ single_search_context_limit: int = DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT,+ single_search_rerank_limit: int = DEFAULT_SINGLE_SEARCH_RERANK_LIMIT,+ single_search_max_context_chars: int = DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS,+ debug_print_model_input: bool = False,+) -> tuple[str, dict, float, int, list, list, str]:+ start_time = time.time()+ client = SyncHTTPClient(+ url=openviking_url,+ user=sender_id,+ timeout=timeout,+ )+ try:+ client.initialize()+ target_uri = f"viking://user/{sender_id or 'default'}/memories"+ search_result = client.find(+ question,+ target_uri=target_uri,+ limit=single_search_context_limit,+ )+ contexts = select_single_search_contexts(+ search_result,+ limit=single_search_context_limit,+ )++ retrieved_uris = [context["uri"] for context in contexts]++ for context in contexts:+ uri = context["uri"]+ try:+ context["content"] = client.read(+ uri,+ offset=0,+ limit=-1,+ )+ except Exception as exc:+ context["content"] = f"[READ ERROR] {exc}"++ reranker = build_single_search_reranker()+ rerank_enabled = reranker is not None+ rerank_limit = single_search_rerank_limit+ contexts, rerank_scores = rerank_single_search_contexts(+ question=question,+ contexts=contexts,+ reranker=reranker,+ limit=rerank_limit,+ )+ contexts, skipped_context_uris, context_chars = filter_contexts_by_char_budget(+ contexts,+ max_chars=single_search_max_context_chars,+ )+ context_uris = [context["uri"] for context in contexts]++ prompt = build_single_search_context_prompt(+ question=question,+ question_type=question_type,+ question_time=question_time,+ contexts=contexts,+ )+ vlm = build_single_search_vlm()+ memory_prompt_tokens, memory_chars, memory_tokenizer = (+ count_retrieved_memory_content_tokens(+ contexts,+ model_name=getattr(vlm, "model", None),+ )+ )+ if debug_print_model_input:+ print("\n===== SINGLE SEARCH MODEL INPUT BEGIN =====", flush=True)+ print(prompt, flush=True)+ print("===== SINGLE SEARCH MODEL INPUT END =====", flush=True)+ print("===== SINGLE SEARCH DEBUG =====", flush=True)+ print(f"context_count: {len(context_uris)}", flush=True)+ print(f"memory_content_tokens: {memory_prompt_tokens}", flush=True)+ print(f"memory_content_chars: {memory_chars}", flush=True)+ print(f"memory_tokenizer: {memory_tokenizer}", flush=True)+ print(f"max_context_chars: {single_search_max_context_chars}", flush=True)+ print(f"context_chars: {context_chars}", flush=True)+ print(f"rerank_enabled: {rerank_enabled}", flush=True)+ if rerank_enabled:+ print(f"rerank_limit: {rerank_limit}", flush=True)+ print("retrieved_uris:", flush=True)+ for uri in retrieved_uris:+ print(f" {uri}", flush=True)+ print("context_uris:", flush=True)+ for uri in context_uris:+ print(f" {uri}", flush=True)+ if skipped_context_uris:+ print("skipped_context_uris_by_char_limit:", flush=True)+ for uri in skipped_context_uris:+ print(f" {uri}", flush=True)+ if rerank_scores:+ print("rerank_scores:", flush=True)+ for item in rerank_scores:+ print(+ f" score={item['score']} raw_rank={item.get('raw_rank')} uri={item['uri']}",+ flush=True,+ )+ print("===== SINGLE SEARCH DEBUG END =====", flush=True)+ raw_response = vlm.get_completion(prompt)+ response = _response_text(raw_response)+ token_usage = _token_usage_from_vlm(vlm, raw_response)+ token_usage["memory_prompt_tokens"] = memory_prompt_tokens+ token_usage["memory_chars"] = memory_chars+ token_usage["memory_tokenizer"] = memory_tokenizer+ if debug_print_model_input:+ print("===== SINGLE SEARCH TOKEN USAGE =====", flush=True)+ print(json.dumps(token_usage, ensure_ascii=False, indent=2), flush=True)+ print("===== SINGLE SEARCH TOKEN USAGE END =====", flush=True)+ elapsed = time.time() - start_time+ return (+ response,+ token_usage,+ elapsed,+ 1,+ ["single_search", "read", "context_answer"],+ [+ {+ "iteration": 1,+ "retrieved_uris": retrieved_uris,+ "context_uris": context_uris,+ "rerank_enabled": rerank_enabled,+ "rerank_limit": rerank_limit if rerank_enabled else 0,+ "rerank_scores": rerank_scores,+ "max_context_chars": single_search_max_context_chars,+ "context_chars": context_chars,+ "skipped_context_uris_by_char_limit": skipped_context_uris,+ }+ ],+ prompt if debug_print_model_input else "",+ )+ except Exception as exc:+ elapsed = time.time() - start_time+ return (+ f"[SINGLE SEARCH ERROR] {exc}",+ {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},+ elapsed,+ 1,+ ["single_search", "read", "context_answer"],+ [+ {+ "iteration": 1,+ "retrieved_uris": [],+ "context_uris": [],+ "rerank_enabled": False,+ "rerank_limit": 0,+ "rerank_scores": [],+ }+ ],+ "",+ )+ finally:+ try:+ client.close()+ except Exception:+ pass+++def main():+ parser = argparse.ArgumentParser(description="VikingBot LongMemEval evaluation script")+ parser.add_argument(+ "input",+ nargs="?",+ default="data/longmemeval_s_cleaned.json",+ help="Path to LongMemEval JSON file",+ )+ parser.add_argument(+ "--output",+ default="./result/longmemeval_qa_result.csv",+ help="Path to output csv file",+ )+ parser.add_argument(+ "--sample",+ type=int,+ default=None,+ help="LongMemEval sample index (0-based), default all samples",+ )+ parser.add_argument(+ "--count", type=int, default=None, help="Number of QA questions to run, default all"+ )+ parser.add_argument(+ "--threads", type=int, default=5, help="Number of concurrent threads, default: 5"+ )+ parser.add_argument(+ "--timeout",+ type=int,+ default=300,+ help="Per-question OpenViking client timeout in seconds, default: 300",+ )+ parser.add_argument(+ "--openviking-url",+ default=None,+ help="OpenViking server URL, e.g. http://127.0.0.1:1934. Defaults to ovcli.conf.",+ )+ parser.add_argument(+ "--answer-mode",+ choices=["single-search-context"],+ default="single-search-context",+ help=argparse.SUPPRESS,+ )+ parser.add_argument(+ "--debug-print-model-input",+ action="store_true",+ help="Print the full model input prompt and retrieved memory token stats.",+ )+ parser.add_argument(+ "--single-search-context-limit",+ type=int,+ default=DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT,+ help=(+ "Number of memory files to read into the single-search-context prompt, "+ f"default: {DEFAULT_SINGLE_SEARCH_CONTEXT_LIMIT}"+ ),+ )+ parser.add_argument(+ "--single-search-rerank-limit",+ type=int,+ default=DEFAULT_SINGLE_SEARCH_RERANK_LIMIT,+ help=(+ "Number of reranked memory files to keep in the single-search-context prompt, "+ f"default: {DEFAULT_SINGLE_SEARCH_RERANK_LIMIT}"+ ),+ )+ parser.add_argument(+ "--single-search-max-context-chars",+ type=int,+ default=DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS,+ help=(+ "Maximum total memory content characters to concatenate into the answer prompt. "+ "Files that would exceed the budget are skipped whole; <=0 disables the budget. "+ f"default: {DEFAULT_SINGLE_SEARCH_MAX_CONTEXT_CHARS}"+ ),+ )+ args = parser.parse_args()++ output_path = Path(args.output)+ output_path.parent.mkdir(parents=True, exist_ok=True)+ qa_list = load_longmemeval_qa(args.input, args.sample, args.count)+ total = len(qa_list)+ print(f"Loaded {total} QA questions")++ fieldnames = [+ "sample_id",+ "question",+ "answer",+ "question_type",+ "question_time",+ "response",+ "model_input_prompt",+ "token_usage",+ "time_cost",+ "iteration",+ "tools_used_names",+ "retrieved_uris_by_iteration",+ "result",+ ]+ write_lock = threading.Lock()+ with open(output_path, "w", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=fieldnames)+ writer.writeheader()+ f.flush()++ processed_count = 0+ print(+ f"Starting evaluation with {args.threads} concurrent threads, {total} questions to process"+ )++ def process_qa(qa_item, idx, total_count):+ nonlocal processed_count+ question = qa_item["question"]+ answer = qa_item["answer"]+ sample_id = qa_item["sample_id"]+ question_time = qa_item.get("question_time")+ print(f"Processing {idx}/{total_count}: {question[:60]}...")+ if question_time:+ print(f" [time context: {question_time}]")++ sender_id = build_sample_user_id(sample_id)+ session_id = build_sample_agent_id(sample_id)+ (+ response,+ token_usage,+ time_cost,+ iteration,+ tools_used_names,+ retrieved_uris_by_iteration,+ model_input_prompt,+ ) = run_single_search_context_answer(+ question,+ qa_item.get("question_type"),+ question_time,+ sender_id=sender_id,+ session_id=session_id,+ openviking_url=args.openviking_url,+ timeout=args.timeout,+ single_search_context_limit=args.single_search_context_limit,+ single_search_rerank_limit=args.single_search_rerank_limit,+ single_search_max_context_chars=args.single_search_max_context_chars,+ debug_print_model_input=args.debug_print_model_input,+ )+ row = {+ "sample_id": sample_id,+ "question": question,+ "answer": answer,+ "question_type": qa_item.get("question_type", ""),+ "question_time": question_time or "",+ "response": response,+ "model_input_prompt": model_input_prompt,+ "token_usage": json.dumps(token_usage, ensure_ascii=False),+ "time_cost": round(time_cost, 2),+ "iteration": iteration,+ "tools_used_names": json.dumps(tools_used_names, ensure_ascii=False),+ "retrieved_uris_by_iteration": json.dumps(+ retrieved_uris_by_iteration, ensure_ascii=False+ )+ if args.debug_print_model_input+ else "",+ "result": "",+ }++ with write_lock:+ writer.writerow(row)+ f.flush()+ processed_count += 1+ print(f"Completed {processed_count}/{total}, time cost: {round(time_cost, 2)}s")+ return True++ with ThreadPoolExecutor(max_workers=args.threads) as executor:+ futures = []+ for idx, qa_item in enumerate(qa_list, 1):+ futures.append(executor.submit(process_qa, qa_item, idx, total))++ for future in as_completed(futures):+ try:+ future.result()+ except Exception as e:+ print(f"Error processing QA item: {str(e)}")++ print(f"Evaluation completed, results saved to {output_path}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/longmemeval/openviking/stat_judge_result.py b/benchmark/longmemeval/openviking/stat_judge_result.pynew file mode 100644index 000000000..5a609db37--- /dev/null+++ b/benchmark/longmemeval/openviking/stat_judge_result.py@@ -0,0 +1,130 @@+import argparse+import csv+import json+import os+import sys+from collections import defaultdict++csv.field_size_limit(sys.maxsize)+++def main():+ parser = argparse.ArgumentParser(description="Statistics for LongMemEval judge result csv")+ parser.add_argument(+ "--input",+ default="./result/longmemeval_qa_result.csv",+ help="Path to judge result csv file",+ )+ args = parser.parse_args()++ if not os.path.exists(args.input):+ print(f"Error: File not found: {args.input}")+ raise SystemExit(1)++ correct = 0+ wrong = 0+ total_time = 0.0+ total_prompt_tokens = 0+ total_memory_prompt_tokens = 0+ total_memory_chars = 0+ total_completion_tokens = 0+ total_tokens = 0+ valid_rows = 0+ total_iteration = 0+ by_type: dict[str, dict[str, int]] = defaultdict(lambda: {"CORRECT": 0, "WRONG": 0, "OTHER": 0})++ with open(args.input, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ for row in reader:+ valid_rows += 1+ result = row.get("result", "").strip().upper()+ question_type = row.get("question_type", "").strip() or "<missing>"+ if result == "CORRECT":+ correct += 1+ by_type[question_type]["CORRECT"] += 1+ elif result == "WRONG":+ wrong += 1+ by_type[question_type]["WRONG"] += 1+ else:+ by_type[question_type]["OTHER"] += 1++ total_iteration += int(row.get("iteration", "0"))+ time_cost = row.get("time_cost", "")+ if time_cost:+ try:+ total_time += float(time_cost)+ except (ValueError, TypeError):+ pass++ token_usage = row.get("token_usage", "")+ if token_usage and token_usage.strip():+ try:+ token_data = json.loads(token_usage)+ total_prompt_tokens += token_data.get("prompt_tokens", 0)+ total_memory_prompt_tokens += token_data.get("memory_prompt_tokens", 0)+ total_memory_chars += token_data.get("memory_chars", 0)+ total_completion_tokens += token_data.get("completion_tokens", 0)+ total_tokens += token_data.get("total_tokens", 0)+ except json.JSONDecodeError:+ pass++ total_graded = correct + wrong+ accuracy = correct / total_graded if total_graded > 0 else 0.0+ avg_time = total_time / valid_rows if valid_rows > 0 else 0.0+ avg_prompt_tokens = total_prompt_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_memory_prompt_tokens = total_memory_prompt_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_memory_chars = total_memory_chars / valid_rows if valid_rows > 0 else 0.0+ avg_completion_tokens = total_completion_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_total_tokens = total_tokens / valid_rows if valid_rows > 0 else 0.0++ output_lines = [+ "=== Judge Result Statistics ===",+ f"Total rows: {valid_rows}",+ f"Graded rows: {total_graded}",+ f"Correct: {correct}",+ f"Wrong: {wrong}",+ f"Accuracy: {accuracy:.2%}",+ "",+ f"Average time cost: {avg_time:.2f}s",+ "",+ f"Average iteration: {total_iteration / valid_rows if valid_rows > 0 else 0.0:.2f}",+ "",+ "Token usage:",+ f" Total prompt tokens: {total_prompt_tokens}",+ f" Total memory prompt tokens: {total_memory_prompt_tokens}",+ f" Total memory chars: {total_memory_chars}",+ f" Total completion tokens: {total_completion_tokens}",+ f" Total tokens: {total_tokens}",+ f" Average prompt tokens/row: {avg_prompt_tokens:.1f}",+ f" Average memory prompt tokens/row: {avg_memory_prompt_tokens:.1f}",+ f" Average memory chars/row: {avg_memory_chars:.1f}",+ f" Average completion tokens/row: {avg_completion_tokens:.1f}",+ f" Average total tokens/row: {avg_total_tokens:.1f}",+ "",+ "By question type:",+ f"{'question_type':<28} {'correct':>8} {'wrong':>8} {'other':>8} {'graded':>8} {'total':>8} {'accuracy':>10}",+ ]++ for question_type in sorted(by_type):+ type_correct = by_type[question_type]["CORRECT"]+ type_wrong = by_type[question_type]["WRONG"]+ type_other = by_type[question_type]["OTHER"]+ type_graded = type_correct + type_wrong+ type_total = type_graded + type_other+ type_accuracy = type_correct / type_graded if type_graded > 0 else 0.0+ output_lines.append(+ f"{question_type:<28} {type_correct:>8} {type_wrong:>8} {type_other:>8} "+ f"{type_graded:>8} {type_total:>8} {type_accuracy:>9.2%}"+ )++ for line in output_lines:+ print(line)++ summary_path = os.path.join(os.path.dirname(args.input), "longmemeval_summary.txt")+ with open(summary_path, "w", encoding="utf-8") as f:+ f.write("\n".join(output_lines) + "\n")+ print(f"\nSummary saved to {summary_path}")+++if __name__ == "__main__":+ main()