Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(benchmark): add Claude Code LoCoMo evaluation harness (#1990)

t0saki0a6c36213f87
diff --git a/benchmark/locomo/claudecode/README.md b/benchmark/locomo/claudecode/README.md
new file mode 100644
index 000000000..158886302
--- /dev/null
+++ b/benchmark/locomo/claudecode/README.md
@@ -0,0 +1,122 @@
+# LoCoMo benchmark — Claude Code memory paths
+
+Four reproducible evaluation paths over the LoCoMo-10 dataset, each
+exercising a different way of giving Claude Code long-term memory.
+
+| Mode | Ingest path | OV namespace | Entry point |
+|---|---|---|---|
+| **Prompted** | `claude -p` per session, CC writes `MEMORY.md` | — (no OpenViking) | `run_prompted.sh` |
+| **SDK iso** | OpenViking Python SDK direct import | per-sample `agent_id` | `run_sdk_iso.sh` |
+| **SDK no-iso** | OpenViking Python SDK direct import | shared (default) | `run_sdk_noiso.sh` |
+| **e2e** | `claude -p` stream-json multi-turn, auto-capture into OV | shared | `run_e2e.sh` |
+
+All four use the same QA / judge / stats pipeline (`eval.py` → `judge.py` →
+`stat_judge_result.py`) so accuracy numbers are directly comparable.
+
+## Prerequisites
+
+1. **LoCoMo data** — drop `locomo10.json` into `.tmp/`:
+ ```
+ benchmark/locomo/claudecode/.tmp/locomo10.json
+ ```
+2. **Python deps** — managed by `uv` (use `uv run` invocations as below).
+3. **Claude Code CLI** on `PATH` (the runners call `claude -p`).
+4. **An Anthropic-compatible model endpoint.** Anything that speaks the
+ Anthropic Messages API works. The original results were collected against
+ doubao-seed-2-0 via Volces ARK:
+ ```bash
+ export ANTHROPIC_AUTH_TOKEN=...
+ export ANTHROPIC_BASE_URL=https://ark.cn-beijing.volces.com/api/compatible
+ export ANTHROPIC_MODEL=doubao-seed-2-0-code-preview-260215
+ ```
+5. **OpenViking server + plugin** (only for SDK iso / SDK no-iso / e2e modes):
+ - Start `openviking-server` (defaults to `127.0.0.1:1933`). The e2e mode
+ wants it running inside a tmux session so the runner can restart it
+ across snapshots — set `OPENVIKING_SERVER_TMUX=<session>` (default
+ `ovserver`).
+ - Point at your clone of [`claude-code-memory-plugin`][plugin]:
+ ```bash
+ export OPENVIKING_PLUGIN_DIR=$HOME/Dev/OpenViking/examples/claude-code-memory-plugin
+ ```
+ This is referenced from `config/ov-hooks.json` and from
+ `scripts/auto-capture.mjs`.
+ - Optional: `export OPENVIKING_CLI_CONFIG_FILE=$HOME/.openviking/ovcli-local.conf`
+ to pin the local OV target and avoid leaking prod URLs from the
+ user-shell's `claude` wrapper.
+
+[plugin]: https://github.com/OpenViking/openviking/tree/main/examples/claude-code-memory-plugin
+
+## Running
+
+Single-sample smoke test (any mode, any conv id):
+
+```bash
+./run_prompted.sh conv-26
+./run_sdk_iso.sh conv-26
+./run_sdk_noiso.sh conv-26
+./run_e2e.sh conv-26
+```
+
+Full 10-conv run — omit the conv id:
+
+```bash
+./run_prompted.sh
+./run_sdk_iso.sh
+./run_sdk_noiso.sh
+./run_e2e.sh
+```
+
+Each script ends with the result of `stat_judge_result.py` printed to stdout
+and saved to `.tmp/result-<mode>/summary.txt`. Per-question CSVs land in
+`.tmp/result-<mode>/qa_results.csv`.
+
+## What each Python entry point does
+
+- `ingest.py` — feeds each LoCoMo session as a `claude -p` invocation;
+ CC's vanilla auto-memory writes `MEMORY.md` inside the project dir.
+- `import_to_ov.py` — uses the OpenViking Python SDK to push LoCoMo
+ conversations directly into OV. Per-sample namespace by default; pass
+ `--no-user-agent-id` for shared namespace.
+- `ingest_e2e.py` — opens one `claude -p` per LoCoMo session in
+ `--input-format stream-json` mode and streams the messages one at a time
+ through stdin. The plugin's Stop hook auto-captures per turn; SessionEnd
+ fires once per LoCoMo session (one commit per session). The benchmark's
+ `scripts/auto-capture.mjs` shim extracts the LoCoMo per-message timestamp
+ so OV event archive dates line up with conv chronology.
+- `eval.py` — drives QA. Per-question `claude -p`; reads `MEMORY.md` for
+ Prompted mode, talks to OV via auto-recall hook + MCP for the OV modes.
+- `judge.py` / `stat_judge_result.py` — LLM-based judging and per-category
+ accuracy roll-up.
+
+## Layout
+
+```
+benchmark/locomo/claudecode/
+├── README.md
+├── run_prompted.sh # 4 entry points, one per published mode
+├── run_sdk_iso.sh
+├── run_sdk_noiso.sh
+├── run_e2e.sh
+├── ingest.py # Prompted ingest
+├── ingest_e2e.py # e2e stream-json ingest
+├── import_to_ov.py # SDK pre-ingest (iso + no-iso)
+├── eval.py
+├── judge.py
+├── stat_judge_result.py
+├── config/
+│ ├── ov-hooks.json # hooks settings, references $OPENVIKING_PLUGIN_DIR
+│ ├── ov-mcp.json # MCP server (viking)
+│ ├── ov-ingest.conf # auto-capture on, auto-recall on
+│ ├── ov-qa.conf # auto-capture off, auto-recall on
+│ └── sys-prompts/ # optional system prompts (unused in default flows)
+├── scripts/
+│ └── auto-capture.mjs # LoCoMo-aware Stop-hook (per-message created_at)
+└── .tmp/ # gitignored; data, snapshots, results
+ ├── locomo10.json # ← drop dataset here
+ ├── legacy/ # historical scripts/configs/reports kept aside
+ └── result-<mode>/ # output of a run
+```
+
+Historical iterations of the experiment (`r8` … `r14b` snapshot scripts,
+fine/listed/min ingest variants, intermediate reports) live under
+`.tmp/legacy/` and are not part of the published numbers.
diff --git a/benchmark/locomo/claudecode/config/ov-hooks.json b/benchmark/locomo/claudecode/config/ov-hooks.json
new file mode 100644
index 000000000..43fb61cfa
--- /dev/null
+++ b/benchmark/locomo/claudecode/config/ov-hooks.json
@@ -0,0 +1,63 @@
+{
+ "hooks": {
+ "SessionStart": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "node ${OPENVIKING_PLUGIN_DIR}/scripts/session-start.mjs",
+ "timeout": 120
+ }
+ ]
+ }
+ ],
+ "UserPromptSubmit": [
+ {
+ "matcher": "",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "node ${OPENVIKING_PLUGIN_DIR}/scripts/auto-recall.mjs",
+ "timeout": 15
+ }
+ ]
+ }
+ ],
+ "Stop": [
+ {
+ "matcher": "",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "node ${LOCOMO_BENCHMARK_DIR}/scripts/auto-capture.mjs",
+ "timeout": 60
+ }
+ ]
+ }
+ ],
+ "PreCompact": [
+ {
+ "matcher": "",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "node ${OPENVIKING_PLUGIN_DIR}/scripts/pre-compact.mjs",
+ "timeout": 30
+ }
+ ]
+ }
+ ],
+ "SessionEnd": [
+ {
+ "matcher": "",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "node ${OPENVIKING_PLUGIN_DIR}/scripts/session-end.mjs",
+ "timeout": 30
+ }
+ ]
+ }
+ ]
+ }
+}
diff --git a/benchmark/locomo/claudecode/config/ov-ingest.conf b/benchmark/locomo/claudecode/config/ov-ingest.conf
new file mode 100644
index 000000000..3d68f4f6c
--- /dev/null
+++ b/benchmark/locomo/claudecode/config/ov-ingest.conf
@@ -0,0 +1,19 @@
+{
+ "server": { "host": "127.0.0.1", "port": 1933 },
+ "claude_code": {
+ "autoCapture": true,
+ "autoRecall": true,
+ "captureMode": "semantic",
+ "captureMaxLength": 8000000,
+ "captureAssistantTurns": true,
+ "writePathAsync": false,
+ "commitTokenThreshold": 5000,
+ "captureTimeoutMs": 120000,
+ "recallLimit": 6,
+ "scoreThreshold": 0.35,
+ "recallTokenBudget": 10000,
+ "recallMaxContentChars": 4000,
+ "recallPreferAbstract": false,
+ "debug": true
+ }
+}
diff --git a/benchmark/locomo/claudecode/config/ov-mcp.json b/benchmark/locomo/claudecode/config/ov-mcp.json
new file mode 100644
index 000000000..6611ca1d8
--- /dev/null
+++ b/benchmark/locomo/claudecode/config/ov-mcp.json
@@ -0,0 +1,12 @@
+{
+ "mcpServers": {
+ "viking": {
+ "type": "http",
+ "url": "http://127.0.0.1:1933/mcp",
+ "headers": {
+ "X-OpenViking-User": "${OPENVIKING_USER:-}",
+ "X-OpenViking-Agent": "${OPENVIKING_AGENT_ID:-}"
+ }
+ }
+ }
+}
diff --git a/benchmark/locomo/claudecode/config/ov-qa.conf b/benchmark/locomo/claudecode/config/ov-qa.conf
new file mode 100644
index 000000000..f7abcb171
--- /dev/null
+++ b/benchmark/locomo/claudecode/config/ov-qa.conf
@@ -0,0 +1,13 @@
+{
+ "server": { "host": "127.0.0.1", "port": 1933 },
+ "claude_code": {
+ "autoCapture": false,
+ "autoRecall": true,
+ "recallLimit": 6,
+ "scoreThreshold": 0.35,
+ "recallTokenBudget": 10000,
+ "recallMaxContentChars": 4000,
+ "recallPreferAbstract": false,
+ "debug": true
+ }
+}
diff --git a/benchmark/locomo/claudecode/config/sys-prompts/ack.txt b/benchmark/locomo/claudecode/config/sys-prompts/ack.txt
new file mode 100644
index 000000000..afe043a63
--- /dev/null
+++ b/benchmark/locomo/claudecode/config/sys-prompts/ack.txt
@@ -0,0 +1 @@
+You will be shown one day's conversation transcript per turn, as part of a continuing context-sharing flow. For each transcript, briefly acknowledge that you've read it (one short sentence). Do not analyze, summarize, or take notes. You may use any tools as needed but in this scenario nothing is required.
diff --git a/benchmark/locomo/claudecode/config/sys-prompts/summary.txt b/benchmark/locomo/claudecode/config/sys-prompts/summary.txt
new file mode 100644
index 000000000..ca0309acd
--- /dev/null
+++ b/benchmark/locomo/claudecode/config/sys-prompts/summary.txt
@@ -0,0 +1,8 @@
+You will be shown one day's conversation transcript per turn, as part of a continuing context-sharing flow. For each transcript, produce a detailed markdown summary that captures:
+- All specific people, pets, places, items, art works, books, songs, foods mentioned by name
+- Specific dates and time references
+- Events that occurred (who did what, when, where)
+- Relationships and feelings expressed between people
+- Decisions, plans, opinions stated
+
+Be comprehensive — a future reader should be able to answer detailed factual questions about this day's events from your summary alone. Use markdown bullet lists organized by person or topic. You may use any tools as needed but in this scenario nothing is required to fulfil the task.
diff --git a/benchmark/locomo/claudecode/eval.py b/benchmark/locomo/claudecode/eval.py
new file mode 100644
index 000000000..629f9a9b2
--- /dev/null
+++ b/benchmark/locomo/claudecode/eval.py
@@ -0,0 +1,674 @@
+"""
+Run LoCoMo QA evaluation against Claude Code.
+
+Each question is sent to `claude -p` in the corresponding sample's isolated
+project directory. Claude Code's auto-memory loads MEMORY.md from that
+project's memory directory, and it can Read individual session files.
+
+Usage:
+ python eval.py --input ../data/locomo10.json
+ python eval.py --sample 0 --parallel 5
+ python eval.py --sample conv-26 --api-url http://localhost:8000 --api-key sk-xxx
+"""
+
+import argparse
+import csv
+import json
+import os
+import subprocess
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime
+from pathlib import Path
+from threading import Lock
+from typing import Optional
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "locomo10.json")
+DEFAULT_PROJECT_ROOT = "/tmp/locomo-eval"
+DEFAULT_HOME = "/tmp/claude-eval-home"
+DEFAULT_OUTPUT = str(SCRIPT_DIR / "result" / "qa_results.csv")
+
+csv_lock = Lock()
+
+FIELDNAMES = [
+ "sample_id",
+ "question_index",
+ "question",
+ "answer",
+ "category",
+ "question_time",
+ "evidence",
+ "response",
+ "input_tokens",
+ "cache_creation_input_tokens",
+ "cache_read_input_tokens",
+ "output_tokens",
+ "reasoning_tokens",
+ "total_cost_usd",
+ "elapsed_seconds",
+ "num_turns",
+ "ov_recall_hooks",
+ "ov_mcp_calls",
+ "result",
+ "reasoning",
+]
+
+OV_HOOK_LOG = None # resolved per-call using home_dir
+OV_MCP_LOG = str(SCRIPT_DIR / ".tmp" / "mcp-calls.log") # observability only
+
+
+def _count_file_lines(path: str) -> int:
+ try:
+ with open(path, "rb") as f:
+ return sum(1 for _ in f)
+ except FileNotFoundError:
+ return 0
+
+
+# ---------------------------------------------------------------------------
+# LoCoMo data loading
+# ---------------------------------------------------------------------------
+
+
+def load_locomo_data(path: str, sample_id: Optional[str] = None) -> list[dict]:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ if sample_id is not None:
+ try:
+ idx = int(sample_id)
+ if idx < 0 or idx >= len(data):
+ raise ValueError(f"Sample index {idx} out of range (0-{len(data) - 1})")
+ return [data[idx]]
+ except ValueError:
+ pass
+ matched = [s for s in data if s.get("sample_id") == sample_id]
+ if not matched:
+ raise ValueError(f"sample_id '{sample_id}' not found")
+ return matched
+
+ return data
+
+
+def parse_locomo_datetime(date_str: str) -> Optional[datetime]:
+ try:
+ if " on " in date_str:
+ date_part = date_str.split(" on ")[-1]
+ return datetime.strptime(date_part.strip(), "%d %B, %Y")
+ except ValueError:
+ pass
+ return None
+
+
+def get_sample_question_time(sample: dict) -> Optional[str]:
+ conversation = sample.get("conversation", {})
+ session_keys = [
+ k for k in conversation.keys() if k.startswith("session_") and "date_time" not in k
+ ]
+ if not session_keys:
+ return None
+
+ def get_num(key):
+ try:
+ return int(key.replace("session_", ""))
+ except ValueError:
+ return 0
+
+ session_keys.sort(key=get_num, reverse=True)
+ for sk in session_keys:
+ if conversation.get(sk):
+ dt_key = f"{sk.split('_')[0]}_{sk.split('_')[1]}_date_time"
+ sess_num = get_num(sk)
+ dt_key = f"session_{sess_num}_date_time"
+ date_str = conversation.get(dt_key)
+ if date_str:
+ dt = parse_locomo_datetime(date_str)
+ if dt:
+ return dt.strftime("%Y-%m-%d")
+ return None
+
+
+def load_qa_items(
+ data: list[dict],
+ count: Optional[int] = None,
+ question_index: Optional[int] = None,
+) -> list[dict]:
+ items = []
+ for sample in data:
+ sample_id = sample.get("sample_id", "")
+ question_time = get_sample_question_time(sample)
+
+ for qi, qa in enumerate(sample.get("qa", [])):
+ if question_index is not None and qi != question_index:
+ continue
+ category = str(qa.get("category", ""))
+ if category == "5":
+ continue
+ items.append(
+ {
+ "sample_id": sample_id,
+ "question_index": qi,
+ "question": qa["question"],
+ "answer": str(qa["answer"]),
+ "category": category,
+ "evidence": qa.get("evidence", []),
+ "question_time": question_time,
+ }
+ )
+
+ if count is not None:
+ items = items[:count]
+ return items
+
+
+# ---------------------------------------------------------------------------
+# CSV helpers
+# ---------------------------------------------------------------------------
+
+
+def load_processed_keys(output_path: str) -> set[str]:
+ processed: set[str] = set()
+ if not os.path.exists(output_path):
+ return processed
+ try:
+ with open(output_path, "r", encoding="utf-8", newline="") as f:
+ for row in csv.DictReader(f):
+ resp = row.get("response", "")
+ if resp and not resp.startswith("[ERROR]"):
+ key = f"{row.get('sample_id', '')}_{row.get('question_index', '')}"
+ processed.add(key)
+ except Exception:
+ pass
+ return processed
+
+
+def append_row(output_path: str, row: dict) -> None:
+ with csv_lock:
+ file_exists = os.path.exists(output_path)
+ with open(output_path, "a", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=FIELDNAMES, extrasaction="ignore")
+ if not file_exists:
+ writer.writeheader()
+ writer.writerow(row)
+ f.flush()
+
+
+# ---------------------------------------------------------------------------
+# Claude Code invocation
+# ---------------------------------------------------------------------------
+
+
+def _run_claude_once(
+ prompt: str,
+ project_dir: str,
+ env: dict,
+ model: Optional[str] = None,
+ timeout_sec: int = 300,
+ extra_flags: Optional[list] = None,
+) -> dict:
+ """Single claude -p invocation."""
+ cmd = [
+ "claude",
+ "-p",
+ prompt,
+ "--output-format",
+ "json",
+ "--dangerously-skip-permissions",
+ "--setting-sources",
+ "",
+ "--disable-slash-commands",
+ "--strict-mcp-config",
+ ]
+ if extra_flags:
+ cmd.extend(extra_flags)
+ if model:
+ cmd.extend(["--model", model])
+
+ try:
+ result = subprocess.run(
+ cmd,
+ capture_output=True,
+ text=True,
+ timeout=timeout_sec,
+ cwd=project_dir,
+ env=env,
+ )
+ stdout = result.stdout.strip()
+ if not stdout:
+ return {
+ "response": f"[ERROR] empty stdout: {result.stderr[:200]}",
+ "usage": {},
+ "duration_ms": 0,
+ "num_turns": 0,
+ "cost": 0,
+ }
+
+ try:
+ body = json.loads(stdout)
+ except json.JSONDecodeError:
+ return {
+ "response": f"[ERROR] JSON parse: {stdout[:200]}",
+ "usage": {},
+ "duration_ms": 0,
+ "num_turns": 0,
+ "cost": 0,
+ }
+
+ if body.get("is_error"):
+ return {
+ "response": f"[ERROR] {body.get('result', 'unknown')}",
+ "usage": body.get("usage", {}),
+ "duration_ms": body.get("duration_ms", 0),
+ "num_turns": body.get("num_turns", 0),
+ "cost": body.get("total_cost_usd", 0),
+ }
+ return {
+ "response": body.get("result", ""),
+ "usage": body.get("usage", {}),
+ "duration_ms": body.get("duration_ms", 0),
+ "num_turns": body.get("num_turns", 0),
+ "cost": body.get("total_cost_usd", 0),
+ }
+ except subprocess.TimeoutExpired:
+ return {
+ "response": "[TIMEOUT]",
+ "usage": {},
+ "duration_ms": timeout_sec * 1000,
+ "num_turns": 0,
+ "cost": 0,
+ }
+ except Exception as e:
+ return {
+ "response": f"[ERROR] {e}",
+ "usage": {},
+ "duration_ms": 0,
+ "num_turns": 0,
+ "cost": 0,
+ }
+
+
+def run_claude_code(
+ prompt: str,
+ project_dir: str,
+ home_dir: str,
+ api_url: Optional[str] = None,
+ api_key: Optional[str] = None,
+ auth_token: Optional[str] = None,
+ model: Optional[str] = None,
+ timeout_sec: int = 300,
+ retries: int = 2,
+ hooks_settings: Optional[str] = None,
+ mcp_config: Optional[str] = None,
+ ov_config: Optional[str] = None,
+ ov_cli_config: Optional[str] = None,
+ ov_agent_id: Optional[str] = None,
+) -> dict:
+ """Run claude -p with retries on TIMEOUT/ERROR."""
+ env = os.environ.copy()
+ env["HOME"] = home_dir
+ if api_key:
+ env["ANTHROPIC_API_KEY"] = api_key
+ if auth_token:
+ env["ANTHROPIC_AUTH_TOKEN"] = auth_token
+ if api_url:
+ env["ANTHROPIC_BASE_URL"] = api_url
+ if ov_config:
+ env["OPENVIKING_CONFIG_FILE"] = ov_config
+ env["OPENVIKING_MEMORY_ENABLED"] = "1"
+ env["OPENVIKING_DEBUG"] = "1"
+ if ov_cli_config:
+ env["OPENVIKING_CLI_CONFIG_FILE"] = ov_cli_config
+ if ov_agent_id:
+ env["OPENVIKING_AGENT_ID"] = ov_agent_id
+ env["OPENVIKING_USER"] = ov_agent_id
+ # Disable detached-worker write path (subprocess.run reaps the worker
+ # before persistence). Strip any inherited prod URL/auth that the parent
+ # shell's `claude` shell function exports from the default ovcli.conf.
+ env["OPENVIKING_WRITE_PATH_ASYNC"] = "0"
+ env.pop("OPENVIKING_URL", None)
+ env.pop("OPENVIKING_BASE_URL", None)
+ env.pop("OPENVIKING_API_KEY", None)
+ env.pop("OPENVIKING_BEARER_TOKEN", None)
+
+ extra_flags = []
+ if hooks_settings:
+ extra_flags.extend(["--settings", hooks_settings])
+ if mcp_config:
+ extra_flags.extend(["--mcp-config", mcp_config])
+
+ for attempt in range(retries + 1):
+ result = _run_claude_once(prompt, project_dir, env, model, timeout_sec, extra_flags)
+ resp = result["response"]
+ if not resp.startswith("[TIMEOUT]") and not resp.startswith("[ERROR]"):
+ return result
+ if attempt < retries:
+ print(f" [retry {attempt + 1}/{retries}] {resp[:80]}", file=sys.stderr)
+ return result
+
+
+# ---------------------------------------------------------------------------
+# Question processing
+# ---------------------------------------------------------------------------
+
+
+def process_question(
+ qa: dict,
+ project_root: str,
+ home_dir: str,
+ api_url: Optional[str],
+ api_key: Optional[str],
+ auth_token: Optional[str],
+ model: Optional[str],
+ output_path: str,
+ timeout_sec: int,
+ hooks_settings: Optional[str] = None,
+ mcp_config: Optional[str] = None,
+ ov_config: Optional[str] = None,
+ ov_cli_config: Optional[str] = None,
+ ov_shared_id: Optional[str] = None,
+ ov_preamble_override: Optional[str] = None,
+ shared_cwd: bool = False,
+) -> dict:
+ sample_id = qa["sample_id"]
+ qi = qa["question_index"]
+ question = qa["question"]
+ question_time = qa.get("question_time")
+
+ project_dir = project_root if shared_cwd else os.path.join(project_root, sample_id)
+
+ if ov_preamble_override is not None:
+ ov_preamble = ov_preamble_override + "\n\n" if ov_preamble_override else ""
+ elif ov_config:
+ ov_preamble = "If context is insufficient, use OpenViking MCP tools or auto-memory files to find more information.\n\n"
+ else:
+ ov_preamble = ""
+
+ if question_time:
+ prompt = (
+ f"{ov_preamble}Current date: {question_time}. Answer the question directly: {question}"
+ )
+ else:
+ prompt = f"{ov_preamble}Answer the question directly: {question}"
+
+ print(
+ f" [{sample_id}] Q{qi}: {question[:60]}{'...' if len(question) > 60 else ''}",
+ file=sys.stderr,
+ )
+
+ hook_log = os.path.join(home_dir, ".openviking", "logs", "cc-hooks.log") if ov_config else ""
+ hook_before = _count_file_lines(hook_log) if hook_log else 0
+ mcp_before = _count_file_lines(OV_MCP_LOG) if ov_config else 0
+
+ t0 = time.perf_counter()
+ result = run_claude_code(
+ prompt,
+ project_dir,
+ home_dir,
+ api_url=api_url,
+ api_key=api_key,
+ auth_token=auth_token,
+ model=model,
+ timeout_sec=timeout_sec,
+ hooks_settings=hooks_settings,
+ mcp_config=mcp_config,
+ ov_config=ov_config,
+ ov_cli_config=ov_cli_config,
+ ov_agent_id=(ov_shared_id if ov_shared_id is not None else sample_id),
+ )
+ elapsed = time.perf_counter() - t0
+
+ hook_delta = _count_file_lines(hook_log) - hook_before if hook_log else 0
+ mcp_delta = _count_file_lines(OV_MCP_LOG) - mcp_before if ov_config else 0
+
+ usage = result.get("usage", {})
+ response = result["response"]
+
+ print(
+ f" [{sample_id}] A{qi}: {response[:60]}{'...' if len(response) > 60 else ''}"
+ f" ({elapsed:.1f}s)",
+ file=sys.stderr,
+ )
+
+ row = {
+ "sample_id": sample_id,
+ "question_index": qi,
+ "question": question,
+ "answer": qa["answer"],
+ "category": qa["category"],
+ "question_time": question_time or "",
+ "evidence": json.dumps(qa.get("evidence", [])),
+ "response": response,
+ "input_tokens": usage.get("input_tokens", 0),
+ "cache_creation_input_tokens": usage.get("cache_creation_input_tokens", 0),
+ "cache_read_input_tokens": usage.get("cache_read_input_tokens", 0),
+ "output_tokens": usage.get("output_tokens", 0),
+ "reasoning_tokens": usage.get("reasoning_tokens", 0),
+ "total_cost_usd": result.get("cost", 0),
+ "elapsed_seconds": round(elapsed, 2),
+ "num_turns": result.get("num_turns", 0),
+ "ov_recall_hooks": hook_delta,
+ "ov_mcp_calls": mcp_delta,
+ "result": "",
+ "reasoning": "",
+ }
+
+ append_row(output_path, row)
+ return row
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Run LoCoMo QA evaluation against Claude Code")
+ parser.add_argument(
+ "--input",
+ default=DEFAULT_DATA_PATH,
+ help=f"Path to locomo JSON (default: {DEFAULT_DATA_PATH})",
+ )
+ parser.add_argument(
+ "--output",
+ default=DEFAULT_OUTPUT,
+ help=f"Output CSV path (default: {DEFAULT_OUTPUT})",
+ )
+ parser.add_argument(
+ "--sample",
+ default=None,
+ help="Sample index (0-based) or sample_id. Default: all.",
+ )
+ parser.add_argument(
+ "--question-index",
+ type=int,
+ default=None,
+ help="Single question index (0-based) within each sample.",
+ )
+ parser.add_argument(
+ "--count",
+ type=int,
+ default=None,
+ help="Max number of QA items to process.",
+ )
+ parser.add_argument(
+ "--parallel",
+ type=int,
+ default=5,
+ help="Concurrent workers (default: 5, be mindful of API rate limits)",
+ )
+ parser.add_argument(
+ "--project-root",
+ default=DEFAULT_PROJECT_ROOT,
+ help=f"Root directory for sample project dirs (default: {DEFAULT_PROJECT_ROOT})",
+ )
+ parser.add_argument(
+ "--home",
+ default=DEFAULT_HOME,
+ help=f"Isolated HOME directory (default: {DEFAULT_HOME})",
+ )
+ parser.add_argument(
+ "--api-url",
+ default=None,
+ help="Custom Anthropic-compatible API base URL (ANTHROPIC_BASE_URL)",
+ )
+ parser.add_argument(
+ "--api-key",
+ default=None,
+ help="API key (ANTHROPIC_API_KEY). Also reads from env if not specified.",
+ )
+ parser.add_argument(
+ "--auth-token",
+ default=None,
+ help="Auth token (ANTHROPIC_AUTH_TOKEN), alternative to --api-key",
+ )
+ parser.add_argument(
+ "--model",
+ default=None,
+ help="Model name/alias (e.g. sonnet, opus, claude-sonnet-4-6)",
+ )
+ parser.add_argument(
+ "--timeout",
+ type=int,
+ default=300,
+ help="Timeout per question in seconds (default: 300)",
+ )
+ parser.add_argument(
+ "--hooks-settings",
+ default=None,
+ help="Path to hooks settings JSON (passed as --settings to claude)",
+ )
+ parser.add_argument(
+ "--mcp-config",
+ default=None,
+ help="Path to MCP server config JSON (passed as --mcp-config to claude)",
+ )
+ parser.add_argument(
+ "--ov-config",
+ default=None,
+ help="Path to ov.conf for OpenViking (sets OPENVIKING_CONFIG_FILE env var)",
+ )
+ parser.add_argument(
+ "--ov-cli-config",
+ default=None,
+ help="Path to ovcli.conf (sets OPENVIKING_CLI_CONFIG_FILE; pin local OV)",
+ )
+ parser.add_argument(
+ "--ov-shared-id",
+ default=None,
+ help="If set, use this single agent_id/user for all samples (no per-sample isolation). Empty string '' means do NOT set OPENVIKING_AGENT_ID at all (server falls back to 'default').",
+ )
+ parser.add_argument(
+ "--ov-preamble",
+ default=None,
+ help="Custom preamble for OV-enabled QA (overrides default)",
+ )
+ parser.add_argument(
+ "--shared-cwd",
+ action="store_true",
+ help="Use --project-root directly as cwd for every sample (no per-sample subdir). "
+ "Required when ingest stored CC MEMORY.md in a single shared cwd.",
+ )
+ args = parser.parse_args()
+
+ api_key = args.api_key or os.environ.get("ANTHROPIC_API_KEY", "")
+ auth_token = args.auth_token or os.environ.get("ANTHROPIC_AUTH_TOKEN", "")
+ api_url = args.api_url or os.environ.get("ANTHROPIC_BASE_URL")
+
+ if not api_key and not auth_token:
+ print(
+ "[INFO] No API key/token provided - assuming subscription auth in HOME", file=sys.stderr
+ )
+
+ # Verify project root has been ingested (skip in shared-cwd mode where ingest_e2e.py owns it)
+ if not args.shared_cwd:
+ mapping_path = os.path.join(args.project_root, "sample_mapping.json")
+ if not os.path.exists(mapping_path):
+ print(
+ f"Error: {mapping_path} not found. Run ingest.py first.",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ data = load_locomo_data(args.input, args.sample)
+ qa_items = load_qa_items(data, count=args.count, question_index=args.question_index)
+ print(f"[INFO] {len(qa_items)} QA items loaded", file=sys.stderr)
+
+ # Ensure output directory
+ Path(args.output).parent.mkdir(parents=True, exist_ok=True)
+
+ # Remove ERROR rows from existing output
+ if os.path.exists(args.output):
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ fieldnames = reader.fieldnames or FIELDNAMES
+ clean_rows = [r for r in reader if not r.get("response", "").startswith("[ERROR]")]
+ with open(args.output, "w", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
+ writer.writeheader()
+ writer.writerows(clean_rows)
+
+ processed = load_processed_keys(args.output)
+ remaining = [
+ qa for qa in qa_items if f"{qa['sample_id']}_{qa['question_index']}" not in processed
+ ]
+ print(
+ f"[INFO] {len(processed)} already done, {len(remaining)} remaining",
+ file=sys.stderr,
+ )
+
+ if not remaining:
+ print("[INFO] All questions already processed.", file=sys.stderr)
+ return
+
+ print(
+ f"[INFO] Running with {args.parallel} workers, "
+ f"model={args.model or 'default'}, "
+ f"api_url={api_url or 'default'}",
+ file=sys.stderr,
+ )
+
+ with ThreadPoolExecutor(max_workers=args.parallel) as executor:
+ futures = {}
+ for qa in remaining:
+ fut = executor.submit(
+ process_question,
+ qa,
+ args.project_root,
+ args.home,
+ api_url,
+ api_key,
+ auth_token,
+ args.model,
+ args.output,
+ args.timeout,
+ hooks_settings=args.hooks_settings,
+ mcp_config=args.mcp_config,
+ ov_config=args.ov_config,
+ ov_cli_config=args.ov_cli_config,
+ ov_shared_id=args.ov_shared_id,
+ ov_preamble_override=args.ov_preamble,
+ shared_cwd=args.shared_cwd,
+ )
+ futures[fut] = qa
+
+ done_count = 0
+ for fut in as_completed(futures):
+ done_count += 1
+ try:
+ fut.result()
+ except Exception as e:
+ qa = futures[fut]
+ print(
+ f" [ERROR] {qa['sample_id']} Q{qa['question_index']}: {e}",
+ file=sys.stderr,
+ )
+ if done_count % 10 == 0:
+ print(f"[INFO] Progress: {done_count}/{len(remaining)}", file=sys.stderr)
+
+ print(f"\n[INFO] Done. Results: {args.output}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/claudecode/import_to_ov.py b/benchmark/locomo/claudecode/import_to_ov.py
new file mode 120000
index 000000000..92bb12a85
--- /dev/null
+++ b/benchmark/locomo/claudecode/import_to_ov.py
@@ -0,0 +1 @@
+../openclaw/import_to_ov.py
\ No newline at end of file
diff --git a/benchmark/locomo/claudecode/ingest.py b/benchmark/locomo/claudecode/ingest.py
new file mode 100644
index 000000000..4dbc0ad80
--- /dev/null
+++ b/benchmark/locomo/claudecode/ingest.py
@@ -0,0 +1,550 @@
+"""
+Ingest LoCoMo conversations into Claude Code's auto-memory system.
+
+Same flow as openclaw: for each sample, send each session's bundled conversation
+to Claude Code via `claude -p`, letting its auto-memory system extract and persist
+memories. Each session is a separate `claude -p` invocation (independent conversation).
+
+All samples share one isolated project directory so Claude Code accumulates memories
+from all sessions into a single memory store — matching how openclaw uses one agent
+namespace per user.
+
+Usage:
+ python ingest.py
+ python ingest.py --sample 0
+ python ingest.py --sample conv-26
+ python ingest.py --api-url http://localhost:8000 --api-key sk-xxx
+ python ingest.py --force-ingest # re-ingest even if already done
+"""
+
+import argparse
+import csv
+import json
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Optional
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "locomo10.json")
+DEFAULT_PROJECT_ROOT = "/tmp/locomo-eval"
+DEFAULT_HOME = "/tmp/claude-eval-home"
+DEFAULT_INGEST_RECORD = str(SCRIPT_DIR / "result" / ".ingest_record.json")
+DEFAULT_SUCCESS_CSV = str(SCRIPT_DIR / "result" / "ingest_success.csv")
+DEFAULT_ERROR_LOG = str(SCRIPT_DIR / "result" / "ingest_errors.log")
+
+SUCCESS_CSV_FIELDS = [
+ "timestamp",
+ "sample_id",
+ "session",
+ "date_time",
+ "speakers",
+ "input_tokens",
+ "cache_creation_input_tokens",
+ "cache_read_input_tokens",
+ "output_tokens",
+ "reasoning_tokens",
+ "total_cost_usd",
+ "duration_ms",
+ "response_preview",
+]
+
+
+# ---------------------------------------------------------------------------
+# LoCoMo data loading
+# ---------------------------------------------------------------------------
+
+
+def load_locomo_data(path: str, sample_id: Optional[str] = None) -> list[dict]:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ if sample_id is not None:
+ try:
+ idx = int(sample_id)
+ if idx < 0 or idx >= len(data):
+ raise ValueError(f"Sample index {idx} out of range (0-{len(data) - 1})")
+ return [data[idx]]
+ except ValueError:
+ pass
+ matched = [s for s in data if s.get("sample_id") == sample_id]
+ if not matched:
+ raise ValueError(f"sample_id '{sample_id}' not found")
+ return matched
+
+ return data
+
+
+def format_locomo_message(msg: dict) -> str:
+ speaker = msg.get("speaker", "unknown")
+ text = msg.get("text", "")
+ line = f"{speaker}: {text}"
+
+ img_urls = msg.get("img_url", [])
+ if isinstance(img_urls, str) and img_urls:
+ img_urls = [img_urls]
+ for _url in img_urls or []:
+ caption = msg.get("blip_caption", "")
+ if caption:
+ line += f"\n (image: {caption})"
+
+ return line
+
+
+def build_session_messages(item: dict) -> list[dict]:
+ """Build bundled session messages for one LoCoMo sample.
+
+ Same format as openclaw: each session becomes a single string with all
+ messages concatenated, prefixed with date/time header.
+ """
+ conv = item["conversation"]
+ speaker_a = conv["speaker_a"]
+ speaker_b = conv["speaker_b"]
+
+ session_keys = sorted(
+ [k for k in conv if k.startswith("session_") and not k.endswith("_date_time")],
+ key=lambda k: int(k.split("_")[1]),
+ )
+
+ sessions = []
+ for sk in session_keys:
+ sess_num = int(sk.split("_")[1])
+ raw_messages = conv[sk]
+ if not isinstance(raw_messages, list) or not raw_messages:
+ continue
+
+ dt_key = f"{sk}_date_time"
+ date_time = conv.get(dt_key, "")
+
+ parts = [f"[group chat conversation: {date_time}]"]
+ for msg in raw_messages:
+ parts.append(format_locomo_message(msg))
+ combined = "\n\n".join(parts)
+
+ sessions.append(
+ {
+ "message": combined,
+ "meta": {
+ "sample_id": item["sample_id"],
+ "session_key": sk,
+ "session_num": sess_num,
+ "date_time": date_time,
+ "speakers": f"{speaker_a} & {speaker_b}",
+ },
+ }
+ )
+
+ return sessions
+
+
+# ---------------------------------------------------------------------------
+# Ingest record (deduplication)
+# ---------------------------------------------------------------------------
+
+
+def load_ingest_record(path: str) -> dict:
+ try:
+ with open(path, "r", encoding="utf-8") as f:
+ return json.load(f)
+ except (FileNotFoundError, json.JSONDecodeError):
+ return {}
+
+
+def save_ingest_record(record: dict, path: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(record, f, indent=2, ensure_ascii=False)
+
+
+def is_already_ingested(sample_id: str, session_key: str, record: dict) -> bool:
+ key = f"claudecode:{sample_id}:{session_key}"
+ return key in record and record[key].get("success", False)
+
+
+def mark_ingested(sample_id: str, session_key: str, record: dict, meta: dict) -> None:
+ key = f"claudecode:{sample_id}:{session_key}"
+ record[key] = {"success": True, "timestamp": int(time.time()), "meta": meta}
+
+
+def write_success_csv(record: dict, csv_path: str) -> None:
+ Path(csv_path).parent.mkdir(parents=True, exist_ok=True)
+ file_exists = os.path.exists(csv_path)
+ with open(csv_path, "a", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=SUCCESS_CSV_FIELDS)
+ if not file_exists:
+ writer.writeheader()
+ writer.writerow(record)
+
+
+def write_error_log(path: str, sample_id: str, session_key: str, error: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ ts = time.strftime("%Y-%m-%d %H:%M:%S")
+ with open(path, "a", encoding="utf-8") as f:
+ f.write(f"[{ts}] ERROR [{sample_id}/{session_key}]: {error}\n")
+
+
+# ---------------------------------------------------------------------------
+# Claude Code invocation
+# ---------------------------------------------------------------------------
+
+
+def _run_claude_once(
+ prompt: str,
+ project_dir: str,
+ env: dict,
+ model: Optional[str] = None,
+ timeout_sec: int = 600,
+ extra_flags: Optional[list] = None,
+) -> dict:
+ """Single claude -p invocation. Returns parsed result dict."""
+ cmd = [
+ "claude",
+ "-p",
+ prompt,
+ "--output-format",
+ "json",
+ "--dangerously-skip-permissions",
+ "--setting-sources",
+ "",
+ "--disable-slash-commands",
+ "--strict-mcp-config",
+ ]
+ if extra_flags:
+ cmd.extend(extra_flags)
+ if model:
+ cmd.extend(["--model", model])
+
+ try:
+ result = subprocess.run(
+ cmd,
+ capture_output=True,
+ text=True,
+ timeout=timeout_sec,
+ cwd=project_dir,
+ env=env,
+ )
+
+ stdout = result.stdout.strip()
+ if not stdout:
+ return {
+ "response": f"[ERROR] empty stdout: {result.stderr[:200]}",
+ "usage": {},
+ "duration_ms": 0,
+ "cost": 0,
+ }
+
+ try:
+ body = json.loads(stdout)
+ except json.JSONDecodeError:
+ return {
+ "response": f"[ERROR] JSON parse: {stdout[:200]}",
+ "usage": {},
+ "duration_ms": 0,
+ "cost": 0,
+ }
+
+ if body.get("is_error"):
+ return {
+ "response": f"[ERROR] {body.get('result', 'unknown')}",
+ "usage": body.get("usage", {}),
+ "duration_ms": body.get("duration_ms", 0),
+ "cost": body.get("total_cost_usd", 0),
+ }
+
+ return {
+ "response": body.get("result", ""),
+ "usage": body.get("usage", {}),
+ "duration_ms": body.get("duration_ms", 0),
+ "num_turns": body.get("num_turns", 0),
+ "cost": body.get("total_cost_usd", 0),
+ }
+
+ except subprocess.TimeoutExpired:
+ return {"response": "[TIMEOUT]", "usage": {}, "duration_ms": timeout_sec * 1000, "cost": 0}
+ except Exception as e:
+ return {"response": f"[ERROR] {e}", "usage": {}, "duration_ms": 0, "cost": 0}
+
+
+def _build_env(
+ home_dir: str, api_key: Optional[str], auth_token: Optional[str], api_url: Optional[str]
+) -> dict:
+ env = os.environ.copy()
+ env["HOME"] = home_dir
+ if api_key:
+ env["ANTHROPIC_API_KEY"] = api_key
+ if auth_token:
+ env["ANTHROPIC_AUTH_TOKEN"] = auth_token
+ if api_url:
+ env["ANTHROPIC_BASE_URL"] = api_url
+ return env
+
+
+def _run_with_retry(
+ prompt: str,
+ project_dir: str,
+ env: dict,
+ model: Optional[str] = None,
+ timeout_sec: int = 600,
+ retries: int = 2,
+ extra_flags: Optional[list] = None,
+) -> dict:
+ """Run claude -p with retries on TIMEOUT/ERROR."""
+ for attempt in range(retries + 1):
+ result = _run_claude_once(prompt, project_dir, env, model, timeout_sec, extra_flags)
+ resp = result["response"]
+ if not resp.startswith("[TIMEOUT]") and not resp.startswith("[ERROR]"):
+ return result
+ if attempt < retries:
+ print(f" [retry {attempt + 1}/{retries}] {resp[:80]}", file=sys.stderr)
+ return result
+
+
+def run_claude_ingest(
+ message: str,
+ project_dir: str,
+ home_dir: str,
+ api_url: Optional[str] = None,
+ api_key: Optional[str] = None,
+ auth_token: Optional[str] = None,
+ model: Optional[str] = None,
+ timeout_sec: int = 600,
+ retries: int = 2,
+) -> dict:
+ """Send a session's conversation to Claude Code and let auto-memory process it."""
+ env = _build_env(home_dir, api_key, auth_token, api_url)
+ return _run_with_retry(message, project_dir, env, model, timeout_sec, retries)
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Ingest LoCoMo conversations into Claude Code auto-memory"
+ )
+ parser.add_argument(
+ "--input",
+ default=DEFAULT_DATA_PATH,
+ help=f"Path to locomo JSON (default: {DEFAULT_DATA_PATH})",
+ )
+ parser.add_argument(
+ "--sample",
+ default=None,
+ help="Sample index (0-based) or sample_id. Default: all.",
+ )
+ parser.add_argument(
+ "--project-root",
+ default=DEFAULT_PROJECT_ROOT,
+ help=f"Root for sample project dirs (default: {DEFAULT_PROJECT_ROOT})",
+ )
+ parser.add_argument(
+ "--home",
+ default=DEFAULT_HOME,
+ help=f"Isolated HOME directory (default: {DEFAULT_HOME})",
+ )
+ parser.add_argument(
+ "--api-url",
+ default=None,
+ help="Custom API base URL (ANTHROPIC_BASE_URL)",
+ )
+ parser.add_argument(
+ "--api-key",
+ default=None,
+ help="API key (ANTHROPIC_API_KEY)",
+ )
+ parser.add_argument(
+ "--auth-token",
+ default=None,
+ help="Auth token (ANTHROPIC_AUTH_TOKEN), alternative to --api-key",
+ )
+ parser.add_argument(
+ "--model",
+ default=None,
+ help="Model name/alias (e.g. sonnet, opus)",
+ )
+ parser.add_argument(
+ "--force-ingest",
+ action="store_true",
+ help="Re-ingest even if already recorded as done",
+ )
+ parser.add_argument(
+ "--clear-ingest-record",
+ action="store_true",
+ help="Clear all ingest records before running",
+ )
+ parser.add_argument(
+ "--record",
+ default=DEFAULT_INGEST_RECORD,
+ help=f"Ingest record file (default: {DEFAULT_INGEST_RECORD})",
+ )
+ parser.add_argument(
+ "--success-csv",
+ default=DEFAULT_SUCCESS_CSV,
+ help=f"Success CSV path (default: {DEFAULT_SUCCESS_CSV})",
+ )
+ parser.add_argument(
+ "--error-log",
+ default=DEFAULT_ERROR_LOG,
+ help=f"Error log path (default: {DEFAULT_ERROR_LOG})",
+ )
+ parser.add_argument(
+ "--timeout",
+ type=int,
+ default=600,
+ help="Timeout per session in seconds (default: 600)",
+ )
+ parser.add_argument(
+ "--prompt-prefix",
+ default="",
+ help="Optional prefix prepended before each session's conversation text "
+ "(e.g. to nudge auto-memory). Default: empty (bare conversation).",
+ )
+ args = parser.parse_args()
+
+ api_key = args.api_key or os.environ.get("ANTHROPIC_API_KEY", "")
+ auth_token = args.auth_token or os.environ.get("ANTHROPIC_AUTH_TOKEN", "")
+ api_url = args.api_url or os.environ.get("ANTHROPIC_BASE_URL")
+
+ if not api_key and not auth_token:
+ print(
+ "Error: API key required (--api-key/ANTHROPIC_API_KEY or --auth-token/ANTHROPIC_AUTH_TOKEN)",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ if not os.path.exists(args.input):
+ print(f"Error: input file not found: {args.input}", file=sys.stderr)
+ sys.exit(1)
+
+ # Ingest record
+ if args.clear_ingest_record:
+ ingest_record: dict = {}
+ save_ingest_record(ingest_record, args.record)
+ print("[INFO] Cleared ingest records", file=sys.stderr)
+ else:
+ ingest_record = load_ingest_record(args.record)
+
+ samples = load_locomo_data(args.input, args.sample)
+ print(f"[INFO] Loaded {len(samples)} sample(s)", file=sys.stderr)
+ print(f"[INFO] Project root: {args.project_root}", file=sys.stderr)
+ print(f"[INFO] HOME: {args.home}", file=sys.stderr)
+
+ # Ensure isolated HOME has .claude directory
+ os.makedirs(os.path.join(args.home, ".claude"), exist_ok=True)
+
+ total_sessions = 0
+ success_count = 0
+ skip_count = 0
+ error_count = 0
+
+ for item in samples:
+ sample_id = item["sample_id"]
+ sessions = build_session_messages(item)
+
+ # All samples share one project dir so memories accumulate together
+ project_dir = os.path.join(args.project_root, sample_id)
+ os.makedirs(project_dir, exist_ok=True)
+
+ print(f"\n=== Sample {sample_id} ({len(sessions)} sessions) ===", file=sys.stderr)
+
+ for sess in sessions:
+ meta = sess["meta"]
+ msg = sess["message"]
+ session_key = meta["session_key"]
+ label = f"{session_key} ({meta['date_time']})"
+ total_sessions += 1
+
+ if not args.force_ingest and is_already_ingested(sample_id, session_key, ingest_record):
+ print(f" [{label}] SKIP (already ingested)", file=sys.stderr)
+ skip_count += 1
+ continue
+
+ preview = msg.replace("\n", " | ")[:80]
+ print(f" [{label}] {preview}...", file=sys.stderr)
+
+ send_msg = f"{args.prompt_prefix}\n\n{msg}" if args.prompt_prefix else msg
+
+ t0 = time.time()
+ result = run_claude_ingest(
+ send_msg,
+ project_dir,
+ args.home,
+ api_url=api_url,
+ api_key=api_key,
+ auth_token=auth_token,
+ model=args.model,
+ timeout_sec=args.timeout,
+ )
+ elapsed = time.time() - t0
+
+ response = result["response"]
+ usage = result.get("usage", {})
+
+ if response.startswith("[ERROR]") or response.startswith("[TIMEOUT]"):
+ print(f" -> {response[:80]} ({elapsed:.1f}s)", file=sys.stderr)
+ write_error_log(args.error_log, sample_id, session_key, response)
+ error_count += 1
+ else:
+ print(
+ f" -> {response[:80]}{'...' if len(response) > 80 else ''}"
+ f" ({elapsed:.1f}s)",
+ file=sys.stderr,
+ )
+ mark_ingested(
+ sample_id,
+ session_key,
+ ingest_record,
+ {
+ "date_time": meta["date_time"],
+ "duration_ms": result.get("duration_ms", 0),
+ },
+ )
+ save_ingest_record(ingest_record, args.record)
+
+ write_success_csv(
+ {
+ "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
+ "sample_id": sample_id,
+ "session": session_key,
+ "date_time": meta["date_time"],
+ "speakers": meta["speakers"],
+ "input_tokens": usage.get("input_tokens", 0),
+ "cache_creation_input_tokens": usage.get("cache_creation_input_tokens", 0),
+ "cache_read_input_tokens": usage.get("cache_read_input_tokens", 0),
+ "output_tokens": usage.get("output_tokens", 0),
+ "reasoning_tokens": usage.get("reasoning_tokens", 0),
+ "total_cost_usd": result.get("cost", 0),
+ "duration_ms": result.get("duration_ms", 0),
+ "response_preview": response[:100],
+ },
+ args.success_csv,
+ )
+
+ success_count += 1
+
+ print("\n=== Ingest summary ===", file=sys.stderr)
+ print(f" Total sessions: {total_sessions}", file=sys.stderr)
+ print(f" Succeeded: {success_count}", file=sys.stderr)
+ print(f" Skipped: {skip_count}", file=sys.stderr)
+ print(f" Failed: {error_count}", file=sys.stderr)
+
+ # Write sample mapping for eval.py
+ mapping_path = os.path.join(args.project_root, "sample_mapping.json")
+ mapping = {}
+ for item in samples:
+ sid = item["sample_id"]
+ mapping[sid] = {
+ "sample_id": sid,
+ "project_dir": os.path.join(args.project_root, sid),
+ }
+ with open(mapping_path, "w", encoding="utf-8") as f:
+ json.dump(mapping, f, indent=2, ensure_ascii=False)
+ print(f" Mapping: {mapping_path}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/claudecode/ingest_e2e.py b/benchmark/locomo/claudecode/ingest_e2e.py
new file mode 100644
index 000000000..ab580b6a4
--- /dev/null
+++ b/benchmark/locomo/claudecode/ingest_e2e.py
@@ -0,0 +1,511 @@
+"""
+End-to-end LoCoMo ingest via Claude Code chat (stream-json multi-turn).
+
+One `claude -p` subprocess per LoCoMo session. All speaker turns are streamed
+through stdin one at a time. Stdin close at end triggers SessionEnd hook
+(plugin's session-end.mjs commits once). Plugin's Stop hook fires per-turn
+and auto-capture incrementally pushes addMessages per turn.
+
+Result alignment with SDK import_to_ov.py:
+ - addMessage granularity: 1 per LoCoMo speaker turn (matches SDK)
+ - commit granularity: 1 per LoCoMo session at SessionEnd (matches SDK)
+ - extract input: all 30 messages of a session as one batch (matches SDK)
+
+Stream-json keeps the CC session alive across multiple turns, so SessionEnd
+fires only once per LoCoMo session — the only path that aligns BOTH
+granularity dimensions without modifying the upstream plugin.
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import List, Optional
+
+from ingest import (
+ format_locomo_message,
+ is_already_ingested,
+ load_ingest_record,
+ load_locomo_data,
+ mark_ingested,
+ save_ingest_record,
+ write_error_log,
+ write_success_csv,
+)
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".tmp" / "locomo10.json")
+
+
+def _build_env(
+ home_dir: str,
+ api_key: Optional[str],
+ auth_token: Optional[str],
+ api_url: Optional[str],
+ ov_config: Optional[str],
+ ov_cli_config: Optional[str],
+ ov_agent_id: Optional[str],
+) -> dict:
+ env = os.environ.copy()
+ env["HOME"] = home_dir
+ if api_key:
+ env["ANTHROPIC_API_KEY"] = api_key
+ if auth_token:
+ env["ANTHROPIC_AUTH_TOKEN"] = auth_token
+ if api_url:
+ env["ANTHROPIC_BASE_URL"] = api_url
+ if ov_config:
+ env["OPENVIKING_CONFIG_FILE"] = ov_config
+ env["OPENVIKING_MEMORY_ENABLED"] = "1"
+ env["OPENVIKING_DEBUG"] = "1"
+ if ov_cli_config:
+ env["OPENVIKING_CLI_CONFIG_FILE"] = ov_cli_config
+ if ov_agent_id:
+ env["OPENVIKING_AGENT_ID"] = ov_agent_id
+ env["OPENVIKING_USER"] = ov_agent_id
+ env["OPENVIKING_WRITE_PATH_ASYNC"] = "0"
+ env.pop("OPENVIKING_URL", None)
+ env.pop("OPENVIKING_BASE_URL", None)
+ env.pop("OPENVIKING_API_KEY", None)
+ env.pop("OPENVIKING_BEARER_TOKEN", None)
+ return env
+
+
+DEFAULT_SYS_PROMPT = (
+ "I'll send you messages from a group chat I want you to remember, "
+ "one message at a time. Reply with a brief acknowledgement (1-3 words) "
+ "so I know you've recorded it. Do not summarize or analyze."
+)
+
+
+def build_session_speaker_prompts(item: dict) -> List[dict]:
+ conv = item["conversation"]
+ session_keys = sorted(
+ [k for k in conv if k.startswith("session_") and not k.endswith("_date_time")],
+ key=lambda k: int(k.split("_")[1]),
+ )
+ out = []
+ for sk in session_keys:
+ msgs = conv[sk]
+ if not isinstance(msgs, list) or not msgs:
+ continue
+ date_time = conv.get(f"{sk}_date_time", "")
+ prompts = []
+ for idx, msg in enumerate(msgs):
+ line = format_locomo_message(msg)
+ if idx == 0:
+ # First message includes the dated header so the
+ # auto-capture-locomo regex picks up the LoCoMo date for
+ # created_at injection.
+ text = f"[group chat conversation: {date_time}]\n\n{line}"
+ else:
+ text = line
+ prompts.append(text)
+ out.append(
+ {
+ "sample_id": item["sample_id"],
+ "session_key": sk,
+ "date_time": date_time,
+ "speakers": f"{conv['speaker_a']} & {conv['speaker_b']}",
+ "prompts": prompts,
+ }
+ )
+ return out
+
+
+def _stream_one_session(
+ pack: dict,
+ project_dir: str,
+ env: dict,
+ model: Optional[str],
+ timeout_sec: int,
+ hooks_settings: Optional[str],
+ mcp_config: Optional[str],
+ sys_prompt: Optional[str],
+) -> dict:
+ """Open one claude subprocess in stream-json mode, feed all LoCoMo speaker
+ turns through stdin, close stdin to trigger SessionEnd, parse output."""
+ cmd = [
+ "claude",
+ "-p",
+ "--input-format",
+ "stream-json",
+ "--output-format",
+ "stream-json",
+ "--verbose", # required for stream-json output
+ "--dangerously-skip-permissions",
+ "--setting-sources",
+ "",
+ "--disable-slash-commands",
+ "--strict-mcp-config",
+ ]
+ if hooks_settings:
+ cmd.extend(["--settings", hooks_settings])
+ if mcp_config:
+ cmd.extend(["--mcp-config", mcp_config])
+ if model:
+ cmd.extend(["--model", model])
+ if sys_prompt:
+ cmd.extend(["--append-system-prompt", sys_prompt])
+
+ sample_id = pack["sample_id"]
+ session_key = pack["session_key"]
+ prompts = pack["prompts"]
+
+ t0 = time.time()
+ proc = subprocess.Popen(
+ cmd,
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ bufsize=1,
+ cwd=project_dir,
+ env=env,
+ )
+
+ # claude stream-json is a request/response protocol: send one message,
+ # read stdout until a "result" event arrives, only then send the next.
+ # Dumping all messages at once causes claude to process only the first
+ # one (or the first that arrived before its read loop saw EOF) and exit.
+ stdout_chunks = []
+ deadline = time.time() + timeout_sec
+ timed_out = False
+ try:
+ for msg_text in prompts:
+ line = json.dumps({"type": "user", "message": {"role": "user", "content": msg_text}})
+ proc.stdin.write(line + "\n")
+ proc.stdin.flush()
+ # Read stdout until we see a "result" event for this turn
+ while True:
+ if time.time() > deadline:
+ timed_out = True
+ break
+ out_line = proc.stdout.readline()
+ if not out_line:
+ # stdout closed mid-stream — claude exited unexpectedly
+ break
+ stdout_chunks.append(out_line)
+ try:
+ d = json.loads(out_line.strip())
+ except json.JSONDecodeError:
+ continue
+ if isinstance(d, dict) and d.get("type") == "result":
+ break
+ if timed_out:
+ break
+ # Done feeding messages; close stdin so SessionEnd fires.
+ try:
+ proc.stdin.close()
+ except Exception:
+ pass
+ # Drain remaining stdout/stderr by direct reads — communicate() after
+ # an explicit stdin.close() raises "I/O on closed file".
+ try:
+ for line in proc.stdout:
+ stdout_chunks.append(line)
+ except Exception:
+ pass
+ try:
+ tail_err = proc.stderr.read() if proc.stderr else ""
+ except Exception:
+ tail_err = ""
+ stderr = tail_err or ""
+ try:
+ proc.wait(timeout=max(30, deadline - time.time()))
+ except subprocess.TimeoutExpired:
+ try:
+ proc.kill()
+ except Exception:
+ pass
+ proc.wait(timeout=10)
+ stdout = "".join(stdout_chunks)
+ except (BrokenPipeError, OSError) as e:
+ try:
+ proc.kill()
+ except Exception:
+ pass
+ return {
+ "sample_id": sample_id,
+ "session_key": session_key,
+ "date_time": pack["date_time"],
+ "speakers": pack["speakers"],
+ "messages_total": len(prompts),
+ "messages_ok": 0,
+ "messages_err": len(prompts),
+ "error": f"stdin write failed: {e}",
+ "cc_session_id": "",
+ "total_input_tokens": 0,
+ "total_output_tokens": 0,
+ "total_cache_read_tokens": 0,
+ "total_cache_create_tokens": 0,
+ "total_num_turns": 0,
+ "total_cost_usd": 0,
+ "total_duration_ms": int((time.time() - t0) * 1000),
+ "stderr_tail": "",
+ }
+
+ elapsed_ms = int((time.time() - t0) * 1000)
+
+ # Parse stream-json output: count results, accumulate tokens, find session_id.
+ #
+ # Important: in stream-json multi-turn mode, each `result` event's
+ # `total_cost_usd` is CUMULATIVE across the session (verified: msg1
+ # cost=0.127, msg2=0.144, msg3=0.160 — monotonically increasing).
+ # But `usage.input_tokens` etc. are PER-TURN (verified: msg1 inp=12923,
+ # msg2 inp=2486, msg3 inp=13509 — non-monotonic).
+ # So: sum usage across results, take last total_cost_usd.
+ cc_session_id = ""
+ results_seen = 0
+ total_input = total_output = total_cache_read = total_cache_create = 0
+ last_total_cost = 0.0
+ last_num_turns = 0
+ error_results = []
+
+ for line in (stdout or "").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ d = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ if not isinstance(d, dict):
+ continue
+ sid = d.get("session_id")
+ if sid and not cc_session_id:
+ cc_session_id = sid
+ if d.get("type") == "result":
+ results_seen += 1
+ if d.get("is_error"):
+ error_results.append(str(d.get("result", ""))[:80])
+ u = d.get("usage", {}) or {}
+ total_input += u.get("input_tokens", 0) or 0
+ total_output += u.get("output_tokens", 0) or 0
+ total_cache_read += u.get("cache_read_input_tokens", 0) or 0
+ total_cache_create += u.get("cache_creation_input_tokens", 0) or 0
+ last_total_cost = d.get("total_cost_usd", 0) or 0
+ last_num_turns = d.get("num_turns", 0) or 0
+
+ total_turns = last_num_turns
+ total_cost = last_total_cost
+
+ msg_ok = results_seen - len(error_results)
+ msg_err = len(prompts) - msg_ok
+ return {
+ "sample_id": sample_id,
+ "session_key": session_key,
+ "date_time": pack["date_time"],
+ "speakers": pack["speakers"],
+ "messages_total": len(prompts),
+ "messages_ok": max(msg_ok, 0),
+ "messages_err": max(msg_err, 0),
+ "error": "; ".join(error_results) if error_results else "",
+ "cc_session_id": cc_session_id,
+ "total_input_tokens": total_input,
+ "total_output_tokens": total_output,
+ "total_cache_read_tokens": total_cache_read,
+ "total_cache_create_tokens": total_cache_create,
+ "total_num_turns": total_turns,
+ "total_cost_usd": total_cost,
+ "total_duration_ms": elapsed_ms,
+ "stderr_tail": (stderr or "")[-300:],
+ }
+
+
+def _poll_until_truly_empty(
+ base_url: str = "http://127.0.0.1:1933", report_every: int = 30
+) -> None:
+ import urllib.request
+
+ last_running = -1
+ last_report = 0
+ while True:
+ try:
+ with urllib.request.urlopen(f"{base_url}/api/v1/tasks", timeout=10) as r:
+ body = json.loads(r.read())
+ except Exception as e:
+ print(f" [poll tasks] {e}, retrying in 30s", file=sys.stderr)
+ time.sleep(30)
+ continue
+ tasks = body.get("result") or []
+ running = [t for t in tasks if t.get("status") == "running"]
+ if not running:
+ print(" [poll tasks] all extract tasks done", file=sys.stderr)
+ return
+ now = time.time()
+ if last_running != len(running) or (now - last_report) > report_every:
+ print(f" [poll tasks] {len(running)} task(s) still running", file=sys.stderr)
+ last_running = len(running)
+ last_report = now
+ time.sleep(20)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="LoCoMo e2e ingest via Claude Code stream-json multi-turn"
+ )
+ parser.add_argument("--input", default=DEFAULT_DATA_PATH)
+ parser.add_argument("--sample", default=None)
+ parser.add_argument("--project-dir", required=True)
+ parser.add_argument("--home", required=True)
+ parser.add_argument("--record", required=True)
+ parser.add_argument("--success-csv", required=True)
+ parser.add_argument("--error-log", required=True)
+ parser.add_argument("--api-url", default=None)
+ parser.add_argument("--api-key", default=None)
+ parser.add_argument("--auth-token", default=None)
+ parser.add_argument("--model", default=None)
+ parser.add_argument(
+ "--timeout",
+ type=int,
+ default=900,
+ help="Per-session timeout (covers all 30 stream-json turns).",
+ )
+ parser.add_argument("--force", action="store_true")
+ parser.add_argument("--hooks-settings", default=None)
+ parser.add_argument("--mcp-config", default=None)
+ parser.add_argument("--ov-config", default=None)
+ parser.add_argument("--ov-cli-config", default=None)
+ parser.add_argument("--ov-shared-id", default=None)
+ parser.add_argument("--sys-prompt", default=DEFAULT_SYS_PROMPT)
+ parser.add_argument("--parallel", type=int, default=1)
+ args = parser.parse_args()
+
+ api_key = args.api_key or os.environ.get("ANTHROPIC_API_KEY", "")
+ auth_token = args.auth_token or os.environ.get("ANTHROPIC_AUTH_TOKEN", "")
+ api_url = args.api_url or os.environ.get("ANTHROPIC_BASE_URL")
+ if not api_key and not auth_token:
+ print("Error: API key required", file=sys.stderr)
+ sys.exit(1)
+
+ samples = load_locomo_data(args.input, args.sample)
+ print(f"[INFO] Loaded {len(samples)} sample(s)", file=sys.stderr)
+ print(f"[INFO] Shared cwd: {args.project_dir}", file=sys.stderr)
+ print(f"[INFO] HOME: {args.home}", file=sys.stderr)
+ print(f"[INFO] Parallelism: {args.parallel} session(s) at a time", file=sys.stderr)
+
+ os.makedirs(args.project_dir, exist_ok=True)
+ os.makedirs(os.path.join(args.home, ".claude"), exist_ok=True)
+
+ record = load_ingest_record(args.record) if not args.force else {}
+ ov_agent_id = args.ov_shared_id if (args.ov_shared_id and args.ov_shared_id != "") else None
+ env = _build_env(
+ args.home, api_key, auth_token, api_url, args.ov_config, args.ov_cli_config, ov_agent_id
+ )
+
+ work = []
+ for item in samples:
+ packs = build_session_speaker_prompts(item)
+ for p in packs:
+ if not args.force and is_already_ingested(p["sample_id"], p["session_key"], record):
+ continue
+ work.append(p)
+
+ total_sessions = sum(len(build_session_speaker_prompts(item)) for item in samples)
+ print(
+ f"[INFO] Total sessions to ingest: {total_sessions} (already done: {total_sessions - len(work)})",
+ file=sys.stderr,
+ )
+ print(f"[INFO] Total messages to send: {sum(len(p['prompts']) for p in work)}", file=sys.stderr)
+
+ success = error = 0
+ sess_done = 0
+
+ def _run(pack):
+ return _stream_one_session(
+ pack,
+ args.project_dir,
+ env,
+ args.model,
+ args.timeout,
+ args.hooks_settings,
+ args.mcp_config,
+ args.sys_prompt,
+ )
+
+ if args.parallel <= 1:
+ results_iter = (_run(p) for p in work)
+ else:
+ ex = ThreadPoolExecutor(max_workers=args.parallel)
+ futs = [ex.submit(_run, p) for p in work]
+ results_iter = (f.result() for f in as_completed(futs))
+
+ for stats in results_iter:
+ sess_done += 1
+ sample_id = stats["sample_id"]
+ session_key = stats["session_key"]
+ prefix = f"[{sess_done}/{len(work)}] {sample_id}/{session_key}"
+
+ if stats["messages_ok"] == 0:
+ print(
+ f" {prefix} -> all msgs failed: {(stats.get('error', '') or '')[:80]} | stderr: {stats.get('stderr_tail', '')[:150]}",
+ file=sys.stderr,
+ )
+ write_error_log(args.error_log, sample_id, session_key, stats.get("error", ""))
+ error += 1
+ continue
+
+ print(
+ f" {prefix} -> ok={stats['messages_ok']}/{stats['messages_total']} msgs, "
+ f"turns={stats['total_num_turns']}, ${stats['total_cost_usd']:.4f}, "
+ f"{stats['total_duration_ms'] / 1000:.1f}s, cc_sid={stats['cc_session_id'][:8]}",
+ file=sys.stderr,
+ )
+ mark_ingested(
+ sample_id,
+ session_key,
+ record,
+ {
+ "date_time": stats["date_time"],
+ "duration_ms": stats["total_duration_ms"],
+ "messages_ok": stats["messages_ok"],
+ "messages_total": stats["messages_total"],
+ "cc_session_id": stats["cc_session_id"],
+ },
+ )
+ save_ingest_record(record, args.record)
+ write_success_csv(
+ {
+ "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
+ "sample_id": sample_id,
+ "session": session_key,
+ "date_time": stats["date_time"],
+ "speakers": stats["speakers"],
+ "input_tokens": stats["total_input_tokens"],
+ "cache_creation_input_tokens": stats["total_cache_create_tokens"],
+ "cache_read_input_tokens": stats["total_cache_read_tokens"],
+ "output_tokens": stats["total_output_tokens"],
+ "reasoning_tokens": 0,
+ "total_cost_usd": stats["total_cost_usd"],
+ "duration_ms": stats["total_duration_ms"],
+ "response_preview": f"streamjson ok={stats['messages_ok']}/{stats['messages_total']} sid={stats['cc_session_id'][:8]}",
+ },
+ args.success_csv,
+ )
+ if stats["messages_err"] > 0:
+ write_error_log(
+ args.error_log, sample_id, session_key, f"partial: {stats.get('error', '')}"
+ )
+ success += 1
+
+ if success > 0:
+ print(
+ "\n[poll tasks] waiting for ALL extract tasks to complete (no timeout)...",
+ file=sys.stderr,
+ )
+ _poll_until_truly_empty()
+
+ print("\n=== Ingest summary ===", file=sys.stderr)
+ print(
+ f" Sessions: total={total_sessions} ok={success} skip={total_sessions - len(work)} err={error}",
+ file=sys.stderr,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/claudecode/judge.py b/benchmark/locomo/claudecode/judge.py
new file mode 100644
index 000000000..2f4aa25fa
--- /dev/null
+++ b/benchmark/locomo/claudecode/judge.py
@@ -0,0 +1,186 @@
+"""
+LLM judge for Claude Code LoCoMo QA results.
+
+Reuses the same grading logic as openclaw/vikingbot benchmarks.
+
+Usage:
+ python judge.py --input ./result/qa_results.csv
+ python judge.py --input ./result/qa_results.csv --parallel 20
+"""
+
+import argparse
+import asyncio
+import csv
+import json
+import os
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+from openai import AsyncOpenAI
+
+env_file = Path.home() / ".openviking_benchmark_env"
+load_dotenv(env_file)
+
+
+async def grade_answer(
+ llm_client, model: str, question: str, gold_answer: str, response: str
+) -> tuple[bool, str]:
+ system_prompt = (
+ "You are an expert grader that determines if answers to questions "
+ "match a gold standard answer"
+ )
+
+ accuracy_prompt = f"""Your task is to label an answer to a question as 'CORRECT' or 'WRONG'. You will be given the following data:
+ (1) a question (posed by one user to another user),
+ (2) a 'gold' (ground truth) answer,
+ (3) a generated answer
+which you will score as CORRECT/WRONG.
+
+The point of the question is to ask about something one user should know about the other user based on their prior conversations.
+The gold answer will usually be a concise and short answer that includes the referenced topic, for example:
+Question: Do you remember what I got the last time I went to Hawaii?
+Gold answer: A shell necklace
+The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.
+
+For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date.
+
+Now it's time for the real question:
+Question: {question}
+Gold answer: {gold_answer}
+Generated answer: {response}
+
+First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG.
+Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.
+
+Respond with JSON only: {{"is_correct": "CORRECT" or "WRONG", "reasoning": "your explanation"}}"""
+
+ try:
+ resp = await llm_client.chat.completions.create(
+ model=model,
+ messages=[
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": accuracy_prompt},
+ ],
+ temperature=0,
+ timeout=60,
+ )
+ content = resp.choices[0].message.content.strip()
+ start_idx = content.find("{")
+ end_idx = content.rfind("}")
+ if start_idx != -1 and end_idx != -1:
+ result = json.loads(content[start_idx : end_idx + 1])
+ is_correct = result.get("is_correct", "WRONG").strip().upper() == "CORRECT"
+ reasoning = result.get("reasoning", "")
+ return is_correct, reasoning
+ return False, f"[PARSE ERROR] {content}"
+ except Exception as e:
+ return False, f"[API ERROR] {e}"
+
+
+async def main():
+ parser = argparse.ArgumentParser(description="LLM judge for Claude Code QA results")
+ parser.add_argument(
+ "--input",
+ default="./result/qa_results.csv",
+ help="Path to QA result CSV (default: ./result/qa_results.csv)",
+ )
+ parser.add_argument(
+ "--base-url",
+ default="https://ark.cn-beijing.volces.com/api/v3",
+ help="Judge API base URL",
+ )
+ parser.add_argument(
+ "--token",
+ default=os.getenv("ARK_API_KEY", os.getenv("OPENAI_API_KEY", "")),
+ help="Judge API token (or ARK_API_KEY / OPENAI_API_KEY env var)",
+ )
+ parser.add_argument(
+ "--model",
+ default="doubao-seed-2-0-pro-260215",
+ help="Judge model name",
+ )
+ parser.add_argument(
+ "--parallel",
+ type=int,
+ default=5,
+ help="Parallel grading requests (default: 5)",
+ )
+ args = parser.parse_args()
+
+ if not args.token:
+ print("Error: API token required.", file=sys.stderr)
+ print(
+ " Set ARK_API_KEY env var, or use --token, or create ~/.openviking_benchmark_env",
+ file=sys.stderr,
+ )
+ sys.exit(1)
+
+ if not os.path.exists(args.input):
+ print(f"Error: input file not found: {args.input}", file=sys.stderr)
+ sys.exit(1)
+
+ with open(args.input, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ fieldnames = list(reader.fieldnames)
+ if "reasoning" not in fieldnames:
+ fieldnames.append("reasoning")
+ rows = list(reader)
+
+ # Find ungraded rows (skip category=5)
+ ungraded = [
+ i for i, row in enumerate(rows) if row.get("category", "") != "5" and not row.get("result")
+ ]
+ total = len(rows)
+ print(f"Total: {total}, ungraded: {len(ungraded)}", file=sys.stderr)
+
+ if not ungraded:
+ print("All valid answers already graded.", file=sys.stderr)
+ return
+
+ client = AsyncOpenAI(base_url=args.base_url, api_key=args.token)
+ semaphore = asyncio.Semaphore(args.parallel)
+ file_lock = asyncio.Lock()
+
+ async def save_results():
+ async with file_lock:
+ tmp = f"{args.input}.tmp"
+ with open(tmp, "w", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
+ writer.writeheader()
+ writer.writerows(rows)
+ os.replace(tmp, args.input)
+
+ async def process_row(idx):
+ async with semaphore:
+ row = rows[idx]
+ question = row.get("question", "")
+ gold = row.get("answer", "")
+ response = row.get("response", "")
+ print(f"Grading {idx + 1}/{total}: {question[:60]}...", file=sys.stderr)
+
+ is_correct, reasoning = await grade_answer(client, args.model, question, gold, response)
+ row["result"] = "CORRECT" if is_correct else "WRONG"
+ row["reasoning"] = reasoning
+
+ await save_results()
+ print(f" -> {row['result']}", file=sys.stderr)
+
+ tasks = [process_row(idx) for idx in ungraded]
+ await asyncio.gather(*tasks)
+
+ correct = sum(
+ 1 for row in rows if row.get("category", "") != "5" and row.get("result") == "CORRECT"
+ )
+ graded = sum(
+ 1
+ for row in rows
+ if row.get("category", "") != "5" and row.get("result") in ("CORRECT", "WRONG")
+ )
+ accuracy = correct / graded if graded > 0 else 0.0
+ print(f"\nAccuracy: {correct}/{graded} = {accuracy:.2%}", file=sys.stderr)
+ print(f"Results saved to {args.input}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/benchmark/locomo/claudecode/run_e2e.sh b/benchmark/locomo/claudecode/run_e2e.sh
new file mode 100755
index 000000000..a6f886ace
--- /dev/null
+++ b/benchmark/locomo/claudecode/run_e2e.sh
@@ -0,0 +1,138 @@
+#!/bin/bash
+#
+# LoCoMo evaluation: end-to-end via Claude Code auto-capture (stream-json).
+#
+# Mirrors a real CC user behaviour:
+# - Each LoCoMo session is fed turn-by-turn into one `claude -p`
+# subprocess (stream-json input). Plugin's auto-capture (Stop hook) and
+# session-end (SessionEnd hook) push memories into OpenViking. The
+# benchmark's auto-capture wrapper adds per-message `created_at` so
+# event archive dates line up with conv timestamps.
+# - After ingest, snapshot HOME + OV data.
+# - QA loops over conv-* in series; before each conv we restore the OV
+# data + HOME from the post-ingest snapshot, so accumulated capture
+# during QA does not leak across convs.
+#
+# Required env:
+# ANTHROPIC_AUTH_TOKEN / ANTHROPIC_API_KEY
+# ANTHROPIC_BASE_URL
+# ANTHROPIC_MODEL
+# OPENVIKING_PLUGIN_DIR - path to claude-code-memory-plugin
+# OPENVIKING_DATA_DIR - default: ~/.openviking/data
+# OPENVIKING_SERVER_TMUX - tmux session name running openviking-server
+# (default "ovserver"); used to Ctrl-C/restart
+# OPENVIKING_CLI_CONFIG_FILE - optional ovcli.conf path
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR"
+
+: "${OPENVIKING_PLUGIN_DIR:?set to claude-code-memory-plugin dir}"
+export LOCOMO_BENCHMARK_DIR="$SCRIPT_DIR"
+
+OV_DATA_DIR="${OPENVIKING_DATA_DIR:-$HOME/.openviking/data}"
+OV_TMUX="${OPENVIKING_SERVER_TMUX:-ovserver}"
+
+INPUT="${LOCOMO_INPUT:-$SCRIPT_DIR/.tmp/locomo10.json}"
+PROJECT_DIR="$SCRIPT_DIR/.tmp/locomo-eval-e2e"
+HOME_DIR="$SCRIPT_DIR/.tmp/claude-eval-home-e2e"
+RESULT_DIR="$SCRIPT_DIR/.tmp/result-e2e"
+SNAP_DIR="$SCRIPT_DIR/.tmp/snapshots-e2e"
+mkdir -p "$RESULT_DIR" "$SNAP_DIR" "$PROJECT_DIR"
+
+server_health() {
+ curl -sf http://127.0.0.1:1933/api/v1/observer/models >/dev/null
+}
+
+restart_server() {
+ if ! tmux has-session -t "$OV_TMUX" 2>/dev/null; then
+ echo "ERROR: tmux session '$OV_TMUX' missing; start openviking-server in it first" >&2
+ exit 1
+ fi
+ tmux send-keys -t "$OV_TMUX" C-c
+ sleep 3
+ tmux send-keys -t "$OV_TMUX" "openviking-server" Enter
+ local i=0
+ until server_health; do
+ sleep 2; i=$((i+2))
+ if [ $i -gt 30 ]; then echo "ERROR: server didn't come up in 30s" >&2; exit 1; fi
+ done
+}
+
+server_health || { echo "ERROR: openviking-server not responding on :1933" >&2; exit 1; }
+
+# ---- Phase 1: ingest ------------------------------------------------------
+echo "[1/5] e2e ingest via stream-json..."
+SAMPLE_ARG=()
+[ $# -ge 1 ] && SAMPLE_ARG=(--sample "$1")
+
+uv run python "$SCRIPT_DIR/ingest_e2e.py" \
+ --input "$INPUT" \
+ --project-dir "$PROJECT_DIR" --home "$HOME_DIR" \
+ --record "$RESULT_DIR/.ingest_record.json" \
+ --success-csv "$RESULT_DIR/ingest_success.csv" \
+ --error-log "$RESULT_DIR/ingest_errors.log" \
+ --hooks-settings "$SCRIPT_DIR/config/ov-hooks.json" \
+ --mcp-config "$SCRIPT_DIR/config/ov-mcp.json" \
+ --ov-config "$SCRIPT_DIR/config/ov-ingest.conf" \
+ ${OPENVIKING_CLI_CONFIG_FILE:+--ov-cli-config "$OPENVIKING_CLI_CONFIG_FILE"} \
+ --ov-shared-id "" \
+ --parallel "${INGEST_PARALLEL:-3}" \
+ --timeout 1200 \
+ "${SAMPLE_ARG[@]}" 2>&1 | tee "$RESULT_DIR/ingest.log"
+
+# ---- Phase 2: snapshot ----------------------------------------------------
+echo "[2/5] snapshot HOME + OV data..."
+tmux send-keys -t "$OV_TMUX" C-c; sleep 3
+rm -rf "$SNAP_DIR/data.postingest" "$SNAP_DIR/HOME.postingest"
+cp -r "$OV_DATA_DIR" "$SNAP_DIR/data.postingest"
+cp -r "$HOME_DIR" "$SNAP_DIR/HOME.postingest"
+tmux send-keys -t "$OV_TMUX" "openviking-server" Enter
+sleep 8
+server_health && echo " server back up"
+
+# ---- Phase 3: QA loop with per-conv restore ------------------------------
+SAMPLES=(conv-26 conv-30 conv-41 conv-42 conv-43 conv-44 conv-47 conv-48 conv-49 conv-50)
+if [ $# -ge 1 ]; then SAMPLES=("$1"); fi
+
+for SID in "${SAMPLES[@]}"; do
+ OUT="$RESULT_DIR/qa_results_${SID}.csv"
+ if [ -f "$OUT" ]; then
+ echo "[$SID] CSV exists ($(tail -n +2 "$OUT" | wc -l | tr -d ' ') rows); skipping"
+ continue
+ fi
+
+ echo "[$SID] restoring HOME + OV data from snapshot..."
+ rm -rf "$HOME_DIR"; cp -r "$SNAP_DIR/HOME.postingest" "$HOME_DIR"
+ tmux send-keys -t "$OV_TMUX" C-c; sleep 3
+ rm -rf "$OV_DATA_DIR"; cp -r "$SNAP_DIR/data.postingest" "$OV_DATA_DIR"
+ tmux send-keys -t "$OV_TMUX" "openviking-server" Enter
+ i=0; until server_health; do sleep 2; i=$((i+2)); [ $i -gt 30 ] && exit 1; done
+
+ echo "[$SID] running eval.py..."
+ uv run python "$SCRIPT_DIR/eval.py" \
+ --input "$INPUT" --output "$OUT" \
+ --project-root "$PROJECT_DIR" --home "$HOME_DIR" --shared-cwd \
+ --sample "$SID" --parallel "${QA_PARALLEL:-3}" \
+ --hooks-settings "$SCRIPT_DIR/config/ov-hooks.json" \
+ --mcp-config "$SCRIPT_DIR/config/ov-mcp.json" \
+ --ov-config "$SCRIPT_DIR/config/ov-ingest.conf" \
+ --ov-shared-id "" \
+ ${OPENVIKING_CLI_CONFIG_FILE:+--ov-cli-config "$OPENVIKING_CLI_CONFIG_FILE"} \
+ 2>&1 | tee "$RESULT_DIR/qa_${SID}.log"
+done
+
+# ---- Phase 4: merge + judge ----------------------------------------------
+echo "[4/5] merging per-sample CSVs + judging..."
+MERGED="$RESULT_DIR/qa_results.csv"
+shopt -s nullglob
+PER_SAMPLE=( "$RESULT_DIR"/qa_results_conv-*.csv )
+head -1 "${PER_SAMPLE[0]}" > "$MERGED"
+for f in "${PER_SAMPLE[@]}"; do tail -n +2 "$f" >> "$MERGED"; done
+
+ARK_API_KEY="${ANTHROPIC_AUTH_TOKEN:-${ANTHROPIC_API_KEY:-}}" \
+ uv run python "$SCRIPT_DIR/judge.py" --input "$MERGED" --parallel 40
+
+echo "[5/5] stats..."
+uv run python "$SCRIPT_DIR/stat_judge_result.py" --input "$MERGED" \
+ | tee "$RESULT_DIR/summary.txt"
diff --git a/benchmark/locomo/claudecode/run_prompted.sh b/benchmark/locomo/claudecode/run_prompted.sh
new file mode 100755
index 000000000..21aaea573
--- /dev/null
+++ b/benchmark/locomo/claudecode/run_prompted.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+#
+# LoCoMo evaluation: Claude Code Prompted (vanilla auto-memory, no OpenViking).
+#
+# Floor reference: ingest each LoCoMo session via `claude -p`; CC writes notes
+# into MEMORY.md inside per-sample project dirs. QA reads MEMORY.md back to
+# answer questions. No hooks, no MCP, no OV.
+#
+# Required env (or pass via flags):
+# ANTHROPIC_AUTH_TOKEN - or ANTHROPIC_API_KEY
+# ANTHROPIC_BASE_URL - e.g. https://ark.cn-beijing.volces.com/api/compatible
+# ANTHROPIC_MODEL - e.g. doubao-seed-2-0-code-preview-260215
+#
+# LoCoMo data must be at $SCRIPT_DIR/.tmp/locomo10.json.
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR"
+
+INPUT="${LOCOMO_INPUT:-$SCRIPT_DIR/.tmp/locomo10.json}"
+PROJECT_ROOT="$SCRIPT_DIR/.tmp/locomo-eval-prompted"
+HOME_DIR="$SCRIPT_DIR/.tmp/claude-eval-home-prompted"
+RESULT_DIR="$SCRIPT_DIR/.tmp/result-prompted"
+mkdir -p "$RESULT_DIR"
+
+SAMPLE_ARG=()
+[ $# -ge 1 ] && SAMPLE_ARG=(--sample "$1")
+
+echo "[1/4] ingest into CC auto-memory..."
+uv run python "$SCRIPT_DIR/ingest.py" \
+ --input "$INPUT" \
+ --project-root "$PROJECT_ROOT" \
+ --home "$HOME_DIR" \
+ --record "$RESULT_DIR/.ingest_record.json" \
+ --success-csv "$RESULT_DIR/ingest_success.csv" \
+ --error-log "$RESULT_DIR/ingest_errors.log" \
+ "${SAMPLE_ARG[@]}"
+
+echo "[2/4] running QA..."
+uv run python "$SCRIPT_DIR/eval.py" \
+ --input "$INPUT" \
+ --output "$RESULT_DIR/qa_results.csv" \
+ --project-root "$PROJECT_ROOT" \
+ --home "$HOME_DIR" \
+ --parallel "${QA_PARALLEL:-5}" \
+ "${SAMPLE_ARG[@]}"
+
+echo "[3/4] judging..."
+ARK_API_KEY="${ANTHROPIC_AUTH_TOKEN:-${ANTHROPIC_API_KEY:-}}" \
+ uv run python "$SCRIPT_DIR/judge.py" --input "$RESULT_DIR/qa_results.csv" --parallel 40
+
+echo "[4/4] stats..."
+uv run python "$SCRIPT_DIR/stat_judge_result.py" --input "$RESULT_DIR/qa_results.csv" \
+ | tee "$RESULT_DIR/summary.txt"
diff --git a/benchmark/locomo/claudecode/run_sdk_iso.sh b/benchmark/locomo/claudecode/run_sdk_iso.sh
new file mode 100755
index 000000000..ba780f5ab
--- /dev/null
+++ b/benchmark/locomo/claudecode/run_sdk_iso.sh
@@ -0,0 +1,69 @@
+#!/bin/bash
+#
+# LoCoMo evaluation: SDK pre-ingest + per-sample namespace isolation.
+#
+# Each LoCoMo sample (conv-XX) gets its own OpenViking user/agent_id.
+# Ingest goes straight to the OV server via the openviking Python SDK
+# (import_to_ov.py); no Claude Code is involved at ingest time.
+# QA uses Claude Code with the plugin's auto-recall hook + MCP, talking to
+# the per-sample OV namespace.
+#
+# Required env:
+# ANTHROPIC_AUTH_TOKEN - or ANTHROPIC_API_KEY
+# ANTHROPIC_BASE_URL - e.g. https://ark.cn-beijing.volces.com/api/compatible
+# ANTHROPIC_MODEL - e.g. doubao-seed-2-0-code-preview-260215
+# OPENVIKING_PLUGIN_DIR - path to claude-code-memory-plugin (for hooks)
+# OPENVIKING_CLI_CONFIG_FILE (optional) - ovcli.conf override, e.g. ovcli-local.conf
+#
+# OpenViking server must be running at 127.0.0.1:1933.
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR"
+
+: "${OPENVIKING_PLUGIN_DIR:?set to claude-code-memory-plugin dir}"
+export LOCOMO_BENCHMARK_DIR="$SCRIPT_DIR"
+
+INPUT="${LOCOMO_INPUT:-$SCRIPT_DIR/.tmp/locomo10.json}"
+PROJECT_ROOT="$SCRIPT_DIR/.tmp/locomo-eval-sdk-iso"
+HOME_DIR="$SCRIPT_DIR/.tmp/claude-eval-home-sdk-iso"
+RESULT_DIR="$SCRIPT_DIR/.tmp/result-sdk-iso"
+mkdir -p "$RESULT_DIR" "$PROJECT_ROOT"
+
+SAMPLE_ARG=()
+[ $# -ge 1 ] && SAMPLE_ARG=(--sample "$1")
+
+if ! curl -sf http://127.0.0.1:1933/api/v1/observer/models >/dev/null; then
+ echo "ERROR: openviking-server not responding on :1933" >&2; exit 1
+fi
+
+# eval.py expects sample_mapping.json under --project-root in non-shared-cwd mode
+echo '{}' > "$PROJECT_ROOT/sample_mapping.json"
+
+echo "[1/4] SDK pre-ingest (per-sample namespace)..."
+uv run python "$SCRIPT_DIR/import_to_ov.py" \
+ --input "$INPUT" \
+ --success-csv "$RESULT_DIR/ingest_success.csv" \
+ --error-log "$RESULT_DIR/ingest_errors.log" \
+ "${SAMPLE_ARG[@]}"
+
+echo "[2/4] running QA (per-sample agent_id from sample_id)..."
+uv run python "$SCRIPT_DIR/eval.py" \
+ --input "$INPUT" \
+ --output "$RESULT_DIR/qa_results.csv" \
+ --project-root "$PROJECT_ROOT" \
+ --home "$HOME_DIR" \
+ --parallel "${QA_PARALLEL:-5}" \
+ --hooks-settings "$SCRIPT_DIR/config/ov-hooks.json" \
+ --mcp-config "$SCRIPT_DIR/config/ov-mcp.json" \
+ --ov-config "$SCRIPT_DIR/config/ov-qa.conf" \
+ ${OPENVIKING_CLI_CONFIG_FILE:+--ov-cli-config "$OPENVIKING_CLI_CONFIG_FILE"} \
+ "${SAMPLE_ARG[@]}"
+
+echo "[3/4] judging..."
+ARK_API_KEY="${ANTHROPIC_AUTH_TOKEN:-${ANTHROPIC_API_KEY:-}}" \
+ uv run python "$SCRIPT_DIR/judge.py" --input "$RESULT_DIR/qa_results.csv" --parallel 40
+
+echo "[4/4] stats..."
+uv run python "$SCRIPT_DIR/stat_judge_result.py" --input "$RESULT_DIR/qa_results.csv" \
+ | tee "$RESULT_DIR/summary.txt"
diff --git a/benchmark/locomo/claudecode/run_sdk_noiso.sh b/benchmark/locomo/claudecode/run_sdk_noiso.sh
new file mode 100755
index 000000000..1964e9c98
--- /dev/null
+++ b/benchmark/locomo/claudecode/run_sdk_noiso.sh
@@ -0,0 +1,61 @@
+#!/bin/bash
+#
+# LoCoMo evaluation: SDK pre-ingest + SHARED OpenViking namespace.
+#
+# All 10 LoCoMo samples are imported into the same OV user/agent (no
+# per-sample isolation). QA at recall time relies on semantic retrieval to
+# pick the right conv's memories out of the shared pool.
+#
+# Same env as run_sdk_iso.sh.
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR"
+
+: "${OPENVIKING_PLUGIN_DIR:?set to claude-code-memory-plugin dir}"
+export LOCOMO_BENCHMARK_DIR="$SCRIPT_DIR"
+
+INPUT="${LOCOMO_INPUT:-$SCRIPT_DIR/.tmp/locomo10.json}"
+PROJECT_ROOT="$SCRIPT_DIR/.tmp/locomo-eval-sdk-noiso"
+HOME_DIR="$SCRIPT_DIR/.tmp/claude-eval-home-sdk-noiso"
+RESULT_DIR="$SCRIPT_DIR/.tmp/result-sdk-noiso"
+mkdir -p "$RESULT_DIR" "$PROJECT_ROOT"
+
+SAMPLE_ARG=()
+[ $# -ge 1 ] && SAMPLE_ARG=(--sample "$1")
+
+if ! curl -sf http://127.0.0.1:1933/api/v1/observer/models >/dev/null; then
+ echo "ERROR: openviking-server not responding on :1933" >&2; exit 1
+fi
+
+echo '{}' > "$PROJECT_ROOT/sample_mapping.json"
+
+echo "[1/4] SDK pre-ingest (shared namespace, --no-user-agent-id)..."
+uv run python "$SCRIPT_DIR/import_to_ov.py" \
+ --input "$INPUT" \
+ --success-csv "$RESULT_DIR/ingest_success.csv" \
+ --error-log "$RESULT_DIR/ingest_errors.log" \
+ --no-user-agent-id \
+ "${SAMPLE_ARG[@]}"
+
+echo "[2/4] running QA (shared OV namespace via --ov-shared-id \"\")..."
+uv run python "$SCRIPT_DIR/eval.py" \
+ --input "$INPUT" \
+ --output "$RESULT_DIR/qa_results.csv" \
+ --project-root "$PROJECT_ROOT" \
+ --home "$HOME_DIR" \
+ --parallel "${QA_PARALLEL:-5}" \
+ --hooks-settings "$SCRIPT_DIR/config/ov-hooks.json" \
+ --mcp-config "$SCRIPT_DIR/config/ov-mcp.json" \
+ --ov-config "$SCRIPT_DIR/config/ov-qa.conf" \
+ --ov-shared-id "" \
+ ${OPENVIKING_CLI_CONFIG_FILE:+--ov-cli-config "$OPENVIKING_CLI_CONFIG_FILE"} \
+ "${SAMPLE_ARG[@]}"
+
+echo "[3/4] judging..."
+ARK_API_KEY="${ANTHROPIC_AUTH_TOKEN:-${ANTHROPIC_API_KEY:-}}" \
+ uv run python "$SCRIPT_DIR/judge.py" --input "$RESULT_DIR/qa_results.csv" --parallel 40
+
+echo "[4/4] stats..."
+uv run python "$SCRIPT_DIR/stat_judge_result.py" --input "$RESULT_DIR/qa_results.csv" \
+ | tee "$RESULT_DIR/summary.txt"
diff --git a/benchmark/locomo/claudecode/scripts/auto-capture.mjs b/benchmark/locomo/claudecode/scripts/auto-capture.mjs
new file mode 100644
index 000000000..7b09a3ce3
--- /dev/null
+++ b/benchmark/locomo/claudecode/scripts/auto-capture.mjs
@@ -0,0 +1,546 @@
+#!/usr/bin/env node
+
+/**
+ * LoCoMo benchmark companion to the plugin's auto-capture.mjs.
+ *
+ * Difference from upstream:
+ * pushTurnsToOv() extracts the LoCoMo session_time stamp from the user
+ * prompt (e.g. "[group chat conversation: 4:02 pm on 12 February, 2023]"
+ * or "=== session_3 (...) ===") and passes a per-message `created_at`
+ * to OV addMessage, mirroring the SDK ingest path (import_to_ov.py).
+ *
+ * Why: the OV server derives event archive dates from message.created_at.
+ * Without this patch, e2e auto-captured LoCoMo conversations all stamp
+ * under the ingest day, collapsing the temporal dimension that semantic
+ * search relies on.
+ *
+ * Behaviour outside LoCoMo prompts is unchanged: regex miss → omit
+ * created_at → server falls back to now() (upstream behaviour).
+ *
+ * The upstream plugin's lib is loaded dynamically from
+ * $OPENVIKING_PLUGIN_DIR so this script lives outside the plugin tree and
+ * the plugin source is untouched.
+ */
+
+import { readFile, writeFile, mkdir } from "node:fs/promises";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import { pathToFileURL } from "node:url";
+
+const pluginDir = process.env.OPENVIKING_PLUGIN_DIR;
+if (!pluginDir) {
+ process.stderr.write("auto-capture: OPENVIKING_PLUGIN_DIR not set\n");
+ process.stdout.write(JSON.stringify({ decision: "approve" }) + "\n");
+ process.exit(0);
+}
+const PLUGIN_SCRIPTS = pathToFileURL(`${pluginDir}/scripts`).href;
+
+const { isPluginEnabled, loadConfig } = await import(`${PLUGIN_SCRIPTS}/config.mjs`);
+const { createLogger } = await import(`${PLUGIN_SCRIPTS}/debug-log.mjs`);
+const {
+ addMessage,
+ commitSession,
+ deriveOvSessionId,
+ getSession,
+ isBypassed,
+ makeFetchJSON,
+} = await import(`${PLUGIN_SCRIPTS}/lib/ov-session.mjs`);
+const { maybeDetach, readHookStdin } = await import(`${PLUGIN_SCRIPTS}/lib/async-writer.mjs`);
+const { readJsonState, writeJsonState } = await import(`${PLUGIN_SCRIPTS}/lib/state.mjs`);
+
+if (!isPluginEnabled()) {
+ process.stdout.write(JSON.stringify({ decision: "approve" }) + "\n");
+ process.exit(0);
+}
+
+const cfg = loadConfig();
+const { log, logError } = createLogger("auto-capture-locomo-e2e");
+const fetchJSON = makeFetchJSON(cfg, "captureTimeoutMs");
+
+const STATE_DIR = join(tmpdir(), "openviking-cc-capture-state");
+
+function output(obj) {
+ process.stdout.write(JSON.stringify(obj) + "\n");
+}
+
+function approve(msg) {
+ const out = { decision: "approve" };
+ if (msg) out.systemMessage = msg;
+ output(out);
+}
+
+function stateFilePath(sessionId) {
+ const safe = sessionId.replace(/[^a-zA-Z0-9_-]/g, "_");
+ return join(STATE_DIR, `${safe}.json`);
+}
+
+async function loadState(sessionId) {
+ try {
+ const data = await readFile(stateFilePath(sessionId), "utf-8");
+ return JSON.parse(data);
+ } catch {
+ return { capturedTurnCount: 0 };
+ }
+}
+
+async function saveState(sessionId, state) {
+ try {
+ await mkdir(STATE_DIR, { recursive: true });
+ await writeFile(stateFilePath(sessionId), JSON.stringify(state));
+ } catch { /* best effort */ }
+}
+
+// ---------------------------------------------------------------------------
+// Text processing (verbatim from upstream)
+// ---------------------------------------------------------------------------
+
+const MEMORY_TRIGGERS = [
+ /remember|preference|prefer|important|decision|decided|always|never/i,
+ /记住|偏好|喜欢|喜爱|崇拜|讨厌|害怕|重要|决定|总是|永远|优先|习惯|爱好|擅长|最爱|不喜欢/i,
+ /[\w.-]+@[\w.-]+\.\w+/,
+ /\+\d{10,}/,
+ /(?:我|my)\s*(?:是|叫|名字|name|住在|live|来自|from|生日|birthday|电话|phone|邮箱|email)/i,
+ /(?:我|i)\s*(?:喜欢|崇拜|讨厌|害怕|擅长|不会|爱|恨|想要|需要|希望|觉得|认为|相信)/i,
+ /(?:favorite|favourite|love|hate|enjoy|dislike|admire|idol|fan of)/i,
+];
+
+const RELEVANT_MEMORIES_BLOCK_RE = /<relevant-memories>[\s\S]*?<\/relevant-memories>/gi;
+const OPENVIKING_CTX_BLOCK_RE = /<openviking-context>[\s\S]*?<\/openviking-context>/gi;
+const SYSTEM_REMINDER_BLOCK_RE = /<system-reminder>[\s\S]*?<\/system-reminder>/gi;
+const SUBAGENT_CONTEXT_LINE_RE = /^\[Subagent Context\][^\n]*$/gmi;
+const COMMAND_TEXT_RE = /^\/[a-z0-9_-]{1,64}\b/i;
+const NON_CONTENT_TEXT_RE = /^[\p{P}\p{S}\s]+$/u;
+const CJK_CHAR_RE = /[぀-ヿ㐀-鿿豈-﫿가-힯]/;
+const QUESTION_ONLY_RE = /^(who|what|when|where|why|how|is|are|does|did|can|could|would|should|may|might|will|谁|什么|何|哪|为什么|怎么|如何|是|会|能|能否)\b.{0,200}[??]$/i;
+
+function stripInjectedBlocks(text) {
+ return text
+ .replace(RELEVANT_MEMORIES_BLOCK_RE, "")
+ .replace(OPENVIKING_CTX_BLOCK_RE, "")
+ .replace(SYSTEM_REMINDER_BLOCK_RE, "")
+ .replace(SUBAGENT_CONTEXT_LINE_RE, "")
+ .replace(/\x00/g, "");
+}
+
+// CC's `claude -p --resume <id>` injects synthetic bridging turns:
+// user: "Continue from where you left off."
+// assistant: "No response requested."
+// These are not real conversation content and contaminate per-message
+// ingest if passed to OV — filter them out here.
+const CC_RESUME_BRIDGE_RE = /^(continue from where you left off\.?|no response requested\.?)$/i;
+function isCcResumeBridge(text) {
+ if (!text) return false;
+ const trimmed = text.trim();
+ if (trimmed.length > 60) return false;
+ return CC_RESUME_BRIDGE_RE.test(trimmed);
+}
+
+function sanitize(text) {
+ return stripInjectedBlocks(text)
+ .replace(/\s+/g, " ")
+ .trim();
+}
+
+// ---------------------------------------------------------------------------
+// LoCoMo session_time extraction (the only behavioural delta from upstream)
+// ---------------------------------------------------------------------------
+
+const MONTHS = {
+ january: 0, february: 1, march: 2, april: 3, may: 4, june: 5,
+ july: 6, august: 7, september: 8, october: 9, november: 10, december: 11,
+};
+
+// Matches LoCoMo session_X_date_time strings like "7:55 pm on 9 June, 2023"
+// whether wrapped in `(...)` (v12 prompt) or `[...]` (v14 group-chat header)
+// or appearing bare. Boundaries are loose because LoCoMo prompts vary.
+const LOCOMO_TS_RE =
+ /(\d{1,2}):(\d{2})\s*([ap]m)\s+on\s+(\d{1,2})\s+([A-Za-z]+),\s*(\d{4})/i;
+
+function parseLocomoStamp(text) {
+ const m = text.match(LOCOMO_TS_RE);
+ if (!m) return null;
+ let h = parseInt(m[1], 10);
+ const min = parseInt(m[2], 10);
+ const ampm = m[3].toLowerCase();
+ if (ampm === "pm" && h < 12) h += 12;
+ if (ampm === "am" && h === 12) h = 0;
+ const day = parseInt(m[4], 10);
+ const mo = MONTHS[m[5].toLowerCase()];
+ const year = parseInt(m[6], 10);
+ if (mo === undefined) return null;
+ // Use UTC to avoid host TZ skewing the day boundary.
+ return new Date(Date.UTC(year, mo, day, h, min, 0));
+}
+
+function extractBaseDate(turns) {
+ for (const t of turns) {
+ if (t.role !== "user") continue;
+ const d = parseLocomoStamp(t.text || "");
+ if (d) return d;
+ }
+ return null;
+}
+
+function shouldCapture(text) {
+ const normalized = sanitize(text);
+ if (!normalized) return { capture: false, reason: "empty", text: "" };
+
+ const compact = normalized.replace(/\s+/g, "");
+ const minLen = CJK_CHAR_RE.test(compact) ? 4 : 10;
+ if (compact.length < minLen || normalized.length > cfg.captureMaxLength) {
+ return { capture: false, reason: "length_out_of_range", text: normalized };
+ }
+
+ if (COMMAND_TEXT_RE.test(normalized)) {
+ return { capture: false, reason: "command", text: normalized };
+ }
+
+ if (NON_CONTENT_TEXT_RE.test(normalized)) {
+ return { capture: false, reason: "non_content", text: normalized };
+ }
+
+ if (QUESTION_ONLY_RE.test(normalized)) {
+ return { capture: false, reason: "question_only", text: normalized };
+ }
+
+ if (cfg.captureMode === "keyword") {
+ for (const trigger of MEMORY_TRIGGERS) {
+ if (trigger.test(normalized)) {
+ return { capture: true, reason: `trigger:${trigger}`, text: normalized };
+ }
+ }
+ return { capture: false, reason: "no_trigger", text: normalized };
+ }
+
+ return { capture: true, reason: "semantic", text: normalized };
+}
+
+// ---------------------------------------------------------------------------
+// Transcript parsing (verbatim from upstream)
+// ---------------------------------------------------------------------------
+
+function parseTranscript(content) {
+ try {
+ const data = JSON.parse(content);
+ if (Array.isArray(data)) return data;
+ } catch { /* not a JSON array */ }
+
+ const lines = content.split("\n").filter(l => l.trim());
+ const messages = [];
+ for (const line of lines) {
+ try { messages.push(JSON.parse(line)); } catch { /* skip */ }
+ }
+ return messages;
+}
+
+const TOOL_BLOCK_MAX_CHARS = 4096;
+
+function truncateForLog(value) {
+ let s;
+ if (typeof value === "string") {
+ s = value;
+ } else {
+ try {
+ s = JSON.stringify(value, null, 2);
+ } catch {
+ s = String(value);
+ }
+ }
+ if (typeof s !== "string") s = "";
+ if (s.length <= TOOL_BLOCK_MAX_CHARS) return s;
+ return (
+ s.slice(0, TOOL_BLOCK_MAX_CHARS) +
+ `\n... [truncated, ${s.length - TOOL_BLOCK_MAX_CHARS} more chars]`
+ );
+}
+
+function extractToolResultText(content) {
+ if (typeof content === "string") return content;
+ if (!Array.isArray(content)) return "";
+ return content
+ .filter((b) => b && b.type === "text" && typeof b.text === "string")
+ .map((b) => b.text)
+ .join("\n");
+}
+
+function extractAllTurns(messages) {
+ const turns = [];
+ for (const msg of messages) {
+ if (!msg || typeof msg !== "object") continue;
+
+ let role = msg.role;
+ let text = "";
+ let toolNames = [];
+
+ const harvestContent = (content) => {
+ if (typeof content === "string") {
+ text = content;
+ } else if (Array.isArray(content)) {
+ const parts = [];
+ for (const block of content) {
+ if (!block || typeof block !== "object") continue;
+ if (block.type === "text" && typeof block.text === "string") {
+ parts.push(block.text);
+ } else if (block.type === "tool_use" && typeof block.name === "string") {
+ toolNames.push(block.name);
+ parts.push(`[tool: ${block.name}]\n${truncateForLog(block.input)}`);
+ } else if (block.type === "tool_result") {
+ const resultText = extractToolResultText(block.content);
+ if (resultText) {
+ parts.push(`[tool result]\n${truncateForLog(resultText)}`);
+ }
+ }
+ }
+ text = parts.join("\n\n");
+ }
+ };
+
+ if (msg.content !== undefined) {
+ harvestContent(msg.content);
+ } else if (typeof msg.message === "object" && msg.message) {
+ role = msg.message.role || role;
+ harvestContent(msg.message.content);
+ }
+
+ if (role !== "user" && role !== "assistant") continue;
+ if (!text.trim() && toolNames.length === 0) continue;
+ turns.push({ role, text: text.trim(), toolNames });
+ }
+ return turns;
+}
+
+function formatTurnsAsText(turns) {
+ const lines = [];
+ for (const t of turns) {
+ if (t.role === "assistant" && t.toolNames.length > 0) {
+ const uniq = Array.from(new Set(t.toolNames)).join(", ");
+ if (t.text) lines.push(`[assistant]: ${t.text}`);
+ lines.push(`[assistant used tools: ${uniq}]`);
+ } else if (t.text) {
+ lines.push(`[${t.role}]: ${t.text}`);
+ }
+ }
+ return lines.join("\n");
+}
+
+// ---------------------------------------------------------------------------
+// Persistent-session capture (LoCoMo-aware created_at injection)
+// ---------------------------------------------------------------------------
+
+async function pushTurnsToOv(ovSessionId, turns, baseDate) {
+ let ok = 0;
+ let failed = 0;
+ let idx = 0;
+ let bridgesSkipped = 0;
+ for (const turn of turns) {
+ const content = stripInjectedBlocks(turn.text).trim();
+ if (!content) continue;
+ if (isCcResumeBridge(content)) { bridgesSkipped++; continue; }
+
+ const payload = { role: turn.role, content };
+ if (baseDate) {
+ const ts = new Date(baseDate.getTime() + idx * 1000).toISOString();
+ payload.created_at = ts;
+ }
+
+ const res = await addMessage(fetchJSON, ovSessionId, payload);
+ if (res.ok) { ok++; idx++; }
+ else failed++;
+ }
+ return { ok, failed, bridgesSkipped };
+}
+
+// ---------------------------------------------------------------------------
+// Main
+// ---------------------------------------------------------------------------
+
+async function main() {
+ if (!cfg.autoCapture) {
+ log("skip", { stage: "init", reason: "autoCapture disabled" });
+ approve();
+ return;
+ }
+
+ if (await maybeDetach(cfg, { approve })) return;
+
+ let input;
+ try {
+ input = JSON.parse(await readHookStdin());
+ } catch {
+ log("skip", { stage: "stdin_parse", reason: "invalid input" });
+ approve();
+ return;
+ }
+
+ const transcriptPath = input.transcript_path;
+ const sessionId = input.session_id || "unknown";
+ const cwd = input.cwd;
+ const ovSessionId = sessionId !== "unknown" ? deriveOvSessionId(sessionId) : null;
+ log("start", { sessionId, ovSessionId, transcriptPath });
+
+ if (isBypassed(cfg, { sessionId, cwd })) {
+ log("skip", { reason: "bypass_session_pattern" });
+ approve();
+ return;
+ }
+
+ const health = await fetchJSON("/health");
+ if (!health.ok) {
+ logError("health_check", "server unreachable or unhealthy");
+ approve();
+ return;
+ }
+
+ if (!transcriptPath || !ovSessionId) {
+ log("skip", { stage: "input_check", reason: "no transcript_path or session_id" });
+ approve();
+ return;
+ }
+
+ let transcriptContent;
+ try {
+ transcriptContent = await readFile(transcriptPath, "utf-8");
+ } catch (err) {
+ logError("transcript_read", err);
+ approve();
+ return;
+ }
+
+ if (!transcriptContent.trim()) {
+ log("skip", { stage: "transcript_read", reason: "empty transcript" });
+ approve();
+ return;
+ }
+
+ const messages = parseTranscript(transcriptContent);
+ const allTurns = extractAllTurns(messages);
+ if (allTurns.length === 0) {
+ log("skip", { stage: "transcript_parse", reason: "no user/assistant turns found" });
+ approve();
+ return;
+ }
+
+ const state = await loadState(sessionId);
+ const newTurns = allTurns.slice(state.capturedTurnCount);
+ const captureTurns = cfg.captureAssistantTurns
+ ? newTurns
+ : newTurns.filter(turn => turn.role === "user");
+
+ // LoCoMo timestamp: scan ALL captured turns (not just newTurns) so that on
+ // an incremental Stop firing, the date — which lives in the very first user
+ // turn header — is still recoverable. In our ingest path each `claude -p`
+ // is a fresh CC session with capturedTurnCount=0, so allTurns == newTurns
+ // here in practice; the fallback exists for robustness.
+ const baseDate = extractBaseDate(allTurns);
+
+ log("transcript_parse", {
+ totalTurns: allTurns.length,
+ previouslyCaptured: state.capturedTurnCount,
+ newTurns: newTurns.length,
+ captureTurns: captureTurns.length,
+ assistantTurnsSkipped: newTurns.length - captureTurns.length,
+ locomoBaseDate: baseDate ? baseDate.toISOString() : null,
+ });
+
+ if (newTurns.length === 0) {
+ log("skip", { stage: "incremental_check", reason: "no new turns" });
+ approve();
+ return;
+ }
+
+ if (captureTurns.length === 0) {
+ await saveState(sessionId, { capturedTurnCount: allTurns.length });
+ log("state_update", { newCapturedTurnCount: allTurns.length, reason: "assistant_only_increment" });
+ approve();
+ return;
+ }
+
+ const combined = formatTurnsAsText(captureTurns);
+ if (!sanitize(combined)) {
+ log("skip", { stage: "batch_empty" });
+ await saveState(sessionId, { capturedTurnCount: allTurns.length });
+ approve();
+ return;
+ }
+
+ if (cfg.captureMode === "keyword") {
+ const hasTrigger = captureTurns.some(
+ (t) =>
+ t.role === "user" &&
+ MEMORY_TRIGGERS.some((re) => re.test(sanitize(t.text))),
+ );
+ if (!hasTrigger) {
+ log("skip", { stage: "keyword_mode_no_trigger", turns: captureTurns.length });
+ await saveState(sessionId, { capturedTurnCount: allTurns.length });
+ approve();
+ return;
+ }
+ }
+
+ log("should_capture", {
+ capture: true,
+ reason: cfg.captureMode === "keyword" ? "keyword_trigger_matched" : "semantic",
+ combinedLength: combined.length,
+ });
+
+ const result = await pushTurnsToOv(ovSessionId, captureTurns, baseDate);
+ log("push_turns", {
+ ovSessionId,
+ ok: result.ok,
+ failed: result.failed,
+ locomoBaseDate: baseDate ? baseDate.toISOString() : null,
+ });
+
+ await saveState(sessionId, { capturedTurnCount: allTurns.length });
+ log("state_update", { newCapturedTurnCount: allTurns.length });
+
+ let committed = false;
+ let pendingTokens = 0;
+ let commitCount = 0;
+ let totalMessageCount = 0;
+ if (result.ok > 0) {
+ const meta = await getSession(fetchJSON, ovSessionId);
+ pendingTokens = Number(meta?.pending_tokens || 0);
+ commitCount = Number(meta?.commit_count || 0);
+ totalMessageCount = Number(meta?.total_message_count || 0);
+ log("pending_tokens", { ovSessionId, pending: pendingTokens, threshold: cfg.commitTokenThreshold });
+ if (pendingTokens >= cfg.commitTokenThreshold) {
+ const commitRes = await commitSession(fetchJSON, ovSessionId);
+ committed = commitRes.ok;
+ if (committed) commitCount += 1;
+ log("commit", { ovSessionId, ok: commitRes.ok, pending: pendingTokens });
+ }
+ }
+
+ writeJsonState("last-capture.json", {
+ turns_captured: result.ok,
+ turns_failed: result.failed,
+ pending_tokens: pendingTokens,
+ commit_threshold: cfg.commitTokenThreshold,
+ committed,
+ commit_count: commitCount,
+ total_message_count: totalMessageCount,
+ ov_session_id: ovSessionId,
+ cc_session_id: sessionId,
+ locomo_base_date: baseDate ? baseDate.toISOString() : null,
+ });
+
+ if (committed) {
+ const today = new Date().toISOString().slice(0, 10);
+ const prior = readJsonState("daily-stats.json") || {};
+ const archives = prior.date === today ? Number(prior.archives || 0) + 1 : 1;
+ writeJsonState("daily-stats.json", { date: today, archives });
+ }
+
+ if (result.ok > 0) {
+ approve(
+ `captured ${result.ok} turns to ov session ${ovSessionId}` +
+ (committed ? " (committed)" : "") +
+ (baseDate ? ` [locomo:${baseDate.toISOString().slice(0,10)}]` : ""),
+ );
+ } else {
+ approve();
+ }
+}
+
+main().catch((err) => { logError("uncaught", err); approve(); });
diff --git a/benchmark/locomo/claudecode/stat_judge_result.py b/benchmark/locomo/claudecode/stat_judge_result.py
new file mode 100644
index 000000000..b0a3ec343
--- /dev/null
+++ b/benchmark/locomo/claudecode/stat_judge_result.py
@@ -0,0 +1,265 @@
+"""
+Statistics for Claude Code LoCoMo QA judge results.
+
+Reports accuracy by category, token/cost/latency for both ingest and QA phases.
+
+Usage:
+ python stat_judge_result.py --input ./result/qa_results.csv
+ python stat_judge_result.py --input ./result/qa_results.csv --ingest-csv ./result/ingest_success.csv
+"""
+
+import argparse
+import csv
+import os
+import sys
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Statistics for Claude Code QA results")
+ parser.add_argument(
+ "--input",
+ default="./result/qa_results.csv",
+ help="Path to judge result CSV",
+ )
+ parser.add_argument(
+ "--ingest-csv",
+ default=None,
+ help="Path to ingest_success.csv for ingest-phase token stats. "
+ "Default: auto-detect from --input's directory.",
+ )
+ args = parser.parse_args()
+
+ if not os.path.exists(args.input):
+ print(f"Error: file not found: {args.input}", file=sys.stderr)
+ sys.exit(1)
+
+ ingest_csv = args.ingest_csv
+ if ingest_csv is None:
+ ingest_csv = os.path.join(os.path.dirname(args.input), "ingest_success.csv")
+
+ output_lines = []
+
+ output_lines.extend(process_qa_results(args.input))
+
+ if os.path.exists(ingest_csv):
+ output_lines.append("")
+ output_lines.extend(process_ingest_csv(ingest_csv))
+ else:
+ output_lines.append(f"\n(Ingest CSV not found: {ingest_csv}, skipping ingest stats)")
+
+ for line in output_lines:
+ print(line)
+
+ summary_path = os.path.join(os.path.dirname(args.input), "summary.txt")
+ os.makedirs(os.path.dirname(summary_path), exist_ok=True)
+ with open(summary_path, "w", encoding="utf-8") as f:
+ f.write("\n".join(output_lines) + "\n")
+ print(f"\nSummary saved to {summary_path}")
+
+
+def process_qa_results(input_path: str) -> list[str]:
+ correct = 0
+ wrong = 0
+ total_input_tokens = 0
+ total_cache_creation = 0
+ total_cache_read = 0
+ total_output_tokens = 0
+ total_reasoning_tokens = 0
+ total_cost = 0.0
+ total_elapsed = 0.0
+ total_turns = 0
+ min_elapsed = None
+ max_elapsed = None
+ elapsed_rows = 0
+ valid_rows = 0
+
+ by_category: dict[str, dict] = {}
+
+ with open(input_path, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ for row in reader:
+ category = row.get("category", "")
+ if category == "5":
+ continue
+
+ valid_rows += 1
+
+ result = row.get("result", "").strip().upper()
+ if result == "CORRECT":
+ correct += 1
+ elif result == "WRONG":
+ wrong += 1
+
+ # Per-category stats
+ if category not in by_category:
+ by_category[category] = {"correct": 0, "wrong": 0, "total": 0}
+ by_category[category]["total"] += 1
+ if result == "CORRECT":
+ by_category[category]["correct"] += 1
+ elif result == "WRONG":
+ by_category[category]["wrong"] += 1
+
+ try:
+ total_input_tokens += int(row.get("input_tokens", 0))
+ total_cache_creation += int(row.get("cache_creation_input_tokens", 0))
+ total_cache_read += int(row.get("cache_read_input_tokens", 0))
+ total_output_tokens += int(row.get("output_tokens", 0))
+ total_reasoning_tokens += int(row.get("reasoning_tokens", 0))
+ except (ValueError, TypeError):
+ pass
+
+ try:
+ total_cost += float(row.get("total_cost_usd", 0))
+ except (ValueError, TypeError):
+ pass
+
+ try:
+ total_turns += int(row.get("num_turns", 0))
+ except (ValueError, TypeError):
+ pass
+
+ try:
+ elapsed_raw = row.get("elapsed_seconds")
+ if elapsed_raw is not None and str(elapsed_raw).strip():
+ elapsed = float(elapsed_raw)
+ total_elapsed += elapsed
+ min_elapsed = elapsed if min_elapsed is None else min(min_elapsed, elapsed)
+ max_elapsed = elapsed if max_elapsed is None else max(max_elapsed, elapsed)
+ elapsed_rows += 1
+ except (ValueError, TypeError):
+ pass
+
+ total_graded = correct + wrong
+ accuracy = correct / total_graded if total_graded > 0 else 0.0
+
+ avg_input = total_input_tokens / valid_rows if valid_rows > 0 else 0.0
+ avg_cache_creation = total_cache_creation / valid_rows if valid_rows > 0 else 0.0
+ avg_cache_read = total_cache_read / valid_rows if valid_rows > 0 else 0.0
+ avg_output = total_output_tokens / valid_rows if valid_rows > 0 else 0.0
+ avg_reasoning = total_reasoning_tokens / valid_rows if valid_rows > 0 else 0.0
+ avg_elapsed = total_elapsed / elapsed_rows if elapsed_rows > 0 else 0.0
+ avg_turns = total_turns / valid_rows if valid_rows > 0 else 0.0
+ avg_cost = total_cost / valid_rows if valid_rows > 0 else 0.0
+
+ category_names = {
+ "1": "multi-hop",
+ "2": "temporal",
+ "3": "open-domain",
+ "4": "single-hop",
+ }
+
+ output = [
+ "=== Claude Code LoCoMo QA Statistics (excluding category=5) ===",
+ f"Total rows: {valid_rows:,}",
+ f"Graded rows: {total_graded:,}",
+ f"Correct: {correct:,}",
+ f"Wrong: {wrong:,}",
+ f"Accuracy: {accuracy:.2%}",
+ ]
+
+ if by_category:
+ output.append("\nAccuracy by category:")
+ for cat in sorted(by_category.keys()):
+ stats = by_category[cat]
+ cat_graded = stats["correct"] + stats["wrong"]
+ cat_acc = stats["correct"] / cat_graded if cat_graded > 0 else 0.0
+ name = category_names.get(cat, f"cat-{cat}")
+ output.append(
+ f" {name:20s}: {stats['correct']}/{cat_graded} = {cat_acc:.2%}"
+ f" (total: {stats['total']})"
+ )
+
+ output.extend(
+ [
+ "\nQA latency:",
+ f" Total elapsed: {total_elapsed:,.1f}s",
+ f" Avg elapsed: {avg_elapsed:,.1f}s",
+ f" Min elapsed: {(min_elapsed or 0.0):,.1f}s",
+ f" Max elapsed: {(max_elapsed or 0.0):,.1f}s",
+ "\nQA token usage:",
+ f" Total input tokens: {total_input_tokens:,}",
+ f" Total cache creation: {total_cache_creation:,}",
+ f" Total cache read: {total_cache_read:,}",
+ f" Total output tokens: {total_output_tokens:,}",
+ f" Total reasoning tokens: {total_reasoning_tokens:,}",
+ f" Avg input tokens: {avg_input:,.0f}",
+ f" Avg cache creation: {avg_cache_creation:,.0f}",
+ f" Avg cache read: {avg_cache_read:,.0f}",
+ f" Avg output tokens: {avg_output:,.0f}",
+ f" Avg reasoning tokens: {avg_reasoning:,.0f}",
+ "\nQA cost:",
+ f" Total cost: ${total_cost:,.4f}",
+ f" Avg cost per question: ${avg_cost:,.4f}",
+ "\nTurns:",
+ f" Total turns: {total_turns:,}",
+ f" Avg turns per question: {avg_turns:,.1f}",
+ ]
+ )
+
+ return output
+
+
+def process_ingest_csv(input_path: str) -> list[str]:
+ total_input_tokens = 0
+ total_cache_creation = 0
+ total_cache_read = 0
+ total_output_tokens = 0
+ total_reasoning_tokens = 0
+ total_cost = 0.0
+ total_duration_ms = 0
+ valid_rows = 0
+
+ with open(input_path, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ for row in reader:
+ valid_rows += 1
+ try:
+ total_input_tokens += int(row.get("input_tokens", 0))
+ total_cache_creation += int(row.get("cache_creation_input_tokens", 0))
+ total_cache_read += int(row.get("cache_read_input_tokens", 0))
+ total_output_tokens += int(row.get("output_tokens", 0))
+ total_reasoning_tokens += int(row.get("reasoning_tokens", 0))
+ except (ValueError, TypeError):
+ pass
+ try:
+ total_cost += float(row.get("total_cost_usd", 0))
+ except (ValueError, TypeError):
+ pass
+ try:
+ total_duration_ms += int(row.get("duration_ms", 0))
+ except (ValueError, TypeError):
+ pass
+
+ total_elapsed = total_duration_ms / 1000.0
+ avg_input = total_input_tokens / valid_rows if valid_rows > 0 else 0.0
+ avg_cache_creation = total_cache_creation / valid_rows if valid_rows > 0 else 0.0
+ avg_cache_read = total_cache_read / valid_rows if valid_rows > 0 else 0.0
+ avg_output = total_output_tokens / valid_rows if valid_rows > 0 else 0.0
+ avg_reasoning = total_reasoning_tokens / valid_rows if valid_rows > 0 else 0.0
+ avg_cost = total_cost / valid_rows if valid_rows > 0 else 0.0
+
+ return [
+ "=== Ingest Phase Statistics ===",
+ f"Total sessions: {valid_rows:,}",
+ "\nIngest latency:",
+ f" Total elapsed: {total_elapsed:,.1f}s",
+ f" Avg elapsed: {(total_elapsed / valid_rows if valid_rows > 0 else 0.0):,.1f}s",
+ "\nIngest token usage:",
+ f" Total input tokens: {total_input_tokens:,}",
+ f" Total cache creation: {total_cache_creation:,}",
+ f" Total cache read: {total_cache_read:,}",
+ f" Total output tokens: {total_output_tokens:,}",
+ f" Total reasoning tokens: {total_reasoning_tokens:,}",
+ f" Avg input tokens: {avg_input:,.0f}",
+ f" Avg cache creation: {avg_cache_creation:,.0f}",
+ f" Avg cache read: {avg_cache_read:,.0f}",
+ f" Avg output tokens: {avg_output:,.0f}",
+ f" Avg reasoning tokens: {avg_reasoning:,.0f}",
+ "\nIngest cost:",
+ f" Total cost: ${total_cost:,.4f}",
+ f" Avg cost per session: ${avg_cost:,.4f}",
+ ]
+
+
+if __name__ == "__main__":
+ main()