Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat: bailian (#1664)

yangxinxin-7e706a8cdc62d
diff --git a/benchmark/locomo/bailian_memory/README.md b/benchmark/locomo/bailian_memory/README.md
new file mode 100644
index 000000000..e8c6ee0c9
--- /dev/null
+++ b/benchmark/locomo/bailian_memory/README.md
@@ -0,0 +1,188 @@
+# 百炼记忆库(ModelStudio Memory)LoCoMo 评测
+
+本目录使用阿里云百炼记忆库对 LoCoMo 长期对话记忆数据集进行评测。
+
+## 目录结构
+
+```
+bailian_memory/
+├── ingest.py # 将对话数据导入百炼记忆库
+├── eval.py # QA 评测(SearchMemory + Qwen LLM 回答)
+├── delete_user.py # 删除指定用户的全部记忆节点
+└── result/ # 评测结果输出目录
+```
+
+## 环境准备
+
+### 1. 获取 API Key 和记忆库 ID
+
+- **DASHSCOPE_API_KEY**:在[百炼控制台](https://bailian.console.aliyun.com)获取 DashScope API Key(`sk-xxx` 格式)
+- **BAILIAN_MEMORY_LIBRARY_ID**:在百炼控制台「记忆库」页面创建记忆库后,从记忆库卡片获取 ID
+- **BAILIAN_PROFILE_SCHEMA_ID**:可选,见下方「用户画像说明」
+
+推荐写入 `~/.openviking_benchmark_env`:
+
+```bash
+DASHSCOPE_API_KEY=sk-xxx
+BAILIAN_MEMORY_LIBRARY_ID=xxx
+# BAILIAN_PROFILE_SCHEMA_ID=xxx # 可选,不建议在 LoCoMo 场景下开启,见下方说明
+ARK_API_KEY=xxx # 裁判模型(可选,doubao)
+```
+
+### 2. 安装依赖
+
+```bash
+pip install requests python-dotenv
+```
+
+---
+
+## 用户画像说明
+
+百炼记忆库支持两类数据:**事件记忆**(memory nodes)和**用户画像**(user profile)。
+
+### LoCoMo 场景下不建议开启画像
+
+LoCoMo 是双人群聊数据集(Alice & Bob 互相对话),而百炼画像设计上针对**单用户**场景:
+
+- ingest 时所有消息均为 `role: user`,API 无法区分两个说话人
+- 画像会把两个人的属性混在同一个 `user_id` 下,导致错误归属
+
+**实测结果**(全 `role: user` + `profile_schema`):
+
+| 画像字段 | 提取值 | 实际归属 |
+|---------|--------|---------|
+| 性别 | 女性 | Caroline(transgender woman) |
+| 爱好 | 绘画,尤其是日出主题 | **Melanie**(串了) |
+| 人生理想 | 学习心理学或心理咨询 | Caroline |
+
+事件记忆提取正常(两个人的事件都能正确提取),但画像是乱的。
+
+**结论:`BAILIAN_PROFILE_SCHEMA_ID` 不设置,只走事件记忆。**
+
+### 如需测试画像功能
+
+设置 `BAILIAN_PROFILE_SCHEMA_ID` 后,ingest 时会自动同时提取事件记忆和用户画像:
+
+```bash
+# ~/.openviking_benchmark_env
+BAILIAN_PROFILE_SCHEMA_ID=your_schema_id
+```
+
+### 画像数据的限制
+
+- **无法通过 API 删除**:`GetUserProfile` 是只读接口,没有删除/更新画像的公开 API
+- **无法删除整个用户**:`DELETE /entities/{user_id}` 接口存在但尚未对外稳定开放(返回 500)
+- 如需清理画像数据,只能在百炼控制台页面手动操作
+
+---
+
+## 评测流程
+
+### 步骤一:导入对话数据
+
+将 LoCoMo 对话导入百炼记忆库,每个 sample 以 `sample_id` 作为 `user_id` 隔离记忆空间:
+
+```bash
+# 导入所有数据
+python ingest.py
+
+# 只导入第 0 个 sample
+python ingest.py --sample 0
+
+# 指定会话范围(1-4 session)
+python ingest.py --sample conv-26 --sessions 1-4
+
+# 强制重新导入
+python ingest.py --force-ingest
+```
+
+**参数说明:**
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| `--input` | locomo10.json 路径 | `../data/locomo10.json` |
+| `--sample` | 样本索引(int)或 sample_id | 全部 |
+| `--sessions` | 会话范围,如 `1-4` 或 `3` | 全部 |
+| `--force-ingest` | 强制重新导入(忽略已导入记录) | false |
+| `--clear-ingest-record` | 清空所有导入进度记录 | false |
+| `--limit` | 最多处理样本数 | 全部 |
+
+### 步骤二:运行 QA 评测
+
+对每道题执行 SearchMemory 检索 → Qwen LLM 生成答案:
+
+```bash
+# 运行全部评测
+python eval.py
+
+# 只测第 0 个 sample 的前 10 题
+python eval.py --sample 0 --count 10
+
+# 指定模型和检索数量
+python eval.py --model qwen-max --top-k 15 --threads 20
+```
+
+**参数说明:**
+
+| 参数 | 说明 | 默认值 |
+|------|------|--------|
+| `--input` | locomo10.json 路径 | `../data/locomo10.json` |
+| `--output` | 输出 CSV 路径 | `./result/qa_results.csv` |
+| `--model` | 回答用 Qwen 模型 | `qwen-plus` |
+| `--top-k` | 每题检索的记忆条数 | `10` |
+| `--threads` | 并发线程数 | `10` |
+| `--sample` | 指定样本 | 全部 |
+| `--count` | 最多处理题目数 | 全部 |
+
+> 评测支持断点续传,重新运行会自动跳过已完成的题目。
+
+### 步骤三:裁判打分
+
+使用上层目录的 judge 脚本打分:
+
+```bash
+python ../vikingbot/judge.py --input ./result/qa_results.csv --token $ARK_API_KEY
+
+# 统计结果
+python ../vikingbot/stat_judge_result.py --input ./result/qa_results.csv
+```
+
+---
+
+## 清理数据
+
+```bash
+# 删除指定用户的全部记忆节点
+python delete_user.py conv-26
+
+# 删除多个用户
+python delete_user.py conv-26 conv-31 conv-45
+
+# 删除 locomo10.json 中的所有用户
+python delete_user.py --from-data
+
+# 只删前 2 个
+python delete_user.py --from-data --limit 2
+```
+
+> **注意**:`delete_user.py` 只能删除 memory nodes(事件记忆),用户画像数据无法通过 API 删除,需要在百炼控制台手动清理。
+
+---
+
+## 输出文件
+
+| 文件 | 说明 |
+|------|------|
+| `result/qa_results.csv` | QA 评测结果(含检索到的记忆、模型回答、token 消耗) |
+| `result/.ingest_record.json` | 导入进度记录(断点续传用) |
+| `result/ingest_errors.log` | 导入错误日志 |
+
+## 环境变量
+
+| 变量名 | 说明 |
+|--------|------|
+| `DASHSCOPE_API_KEY` | 百炼 DashScope API Key(`sk-xxx`),必填 |
+| `BAILIAN_MEMORY_LIBRARY_ID` | 百炼记忆库 ID,必填 |
+| `BAILIAN_PROFILE_SCHEMA_ID` | 画像模板 ID,可选,LoCoMo 场景不建议设置 |
+| `ARK_API_KEY` | 裁判模型 API Key(Volcengine doubao),可选 |
diff --git a/benchmark/locomo/bailian_memory/delete_user.py b/benchmark/locomo/bailian_memory/delete_user.py
new file mode 100644
index 000000000..50a072288
--- /dev/null
+++ b/benchmark/locomo/bailian_memory/delete_user.py
@@ -0,0 +1,142 @@
+"""
+Delete all memories for one or more users from Bailian (ModelStudio) Memory.
+
+Usage:
+ # Delete a single user
+ python delete_user.py conv-26
+
+ # Delete multiple users
+ python delete_user.py conv-26 conv-31 conv-45
+
+ # Delete first N users from locomo10.json
+ python delete_user.py --from-data --limit 2
+
+ # Delete all users from locomo10.json
+ python delete_user.py --from-data
+
+ DASHSCOPE_API_KEY=sk-xxx BAILIAN_MEMORY_LIBRARY_ID=xxx python delete_user.py --from-data
+"""
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+import requests
+from dotenv import load_dotenv
+
+load_dotenv(Path.home() / ".openviking_benchmark_env")
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "data" / "locomo10.json")
+
+BAILIAN_MEMORY_BASE_URL = "https://dashscope.aliyuncs.com/api/v2/apps/memory"
+
+
+def _auth_headers(api_key: str) -> dict:
+ return {
+ "Authorization": f"Bearer {api_key}",
+ "Content-Type": "application/json",
+ }
+
+
+def list_memory_nodes(api_key: str, memory_library_id: str, user_id: str) -> list[str]:
+ """Return all memory_node_ids for the given user."""
+ node_ids = []
+ page_num = 1
+ page_size = 100
+
+ while True:
+ resp = requests.get(
+ f"{BAILIAN_MEMORY_BASE_URL}/memory_nodes",
+ headers=_auth_headers(api_key),
+ params={
+ "memory_library_id": memory_library_id,
+ "user_id": user_id,
+ "page_num": page_num,
+ "page_size": page_size,
+ },
+ timeout=30,
+ )
+ resp.raise_for_status()
+ body = resp.json()
+ nodes = body.get("memory_nodes", [])
+ for n in nodes:
+ nid = n.get("memory_node_id")
+ if nid:
+ node_ids.append(nid)
+ if len(nodes) < page_size:
+ break
+ page_num += 1
+
+ return node_ids
+
+
+def delete_memory_node(api_key: str, memory_library_id: str, node_id: str) -> None:
+ resp = requests.delete(
+ f"{BAILIAN_MEMORY_BASE_URL}/memory_nodes/{node_id}",
+ headers=_auth_headers(api_key),
+ params={"memory_library_id": memory_library_id},
+ timeout=30,
+ )
+ resp.raise_for_status()
+
+
+def delete_user(api_key: str, memory_library_id: str, user_id: str) -> bool:
+ try:
+ node_ids = list_memory_nodes(api_key, memory_library_id, user_id)
+ if not node_ids:
+ print(f" [OK] {user_id} (no memories found)")
+ return True
+ for nid in node_ids:
+ delete_memory_node(api_key, memory_library_id, nid)
+ print(f" [OK] {user_id} (deleted {len(node_ids)} nodes)")
+ return True
+ except Exception as e:
+ print(f" [ERROR] {user_id}: {e}")
+ return False
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Delete all Bailian Memory nodes for given user(s)")
+ parser.add_argument("users", nargs="*", help="user_id(s) to delete (e.g. conv-26 conv-31)")
+ parser.add_argument("--api-key", default=None, help="DashScope API key (or DASHSCOPE_API_KEY env var)")
+ parser.add_argument("--memory-library-id", default=None, help="Bailian memory library ID (or BAILIAN_MEMORY_LIBRARY_ID env var)")
+ parser.add_argument("--from-data", action="store_true", help="load user_ids from locomo10.json")
+ parser.add_argument("--input", default=DEFAULT_DATA_PATH, help="path to locomo10.json")
+ parser.add_argument("--limit", type=int, default=None, help="max users to delete (with --from-data)")
+ args = parser.parse_args()
+
+ api_key = args.api_key or os.environ.get("DASHSCOPE_API_KEY", "")
+ if not api_key:
+ print("Error: DashScope API key required (--api-key or DASHSCOPE_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ memory_library_id = args.memory_library_id or os.environ.get("BAILIAN_MEMORY_LIBRARY_ID", "")
+ if not memory_library_id:
+ print("Error: memory_library_id required (--memory-library-id or BAILIAN_MEMORY_LIBRARY_ID env var)", file=sys.stderr)
+ sys.exit(1)
+
+ user_ids: list[str] = list(args.users)
+
+ if args.from_data:
+ with open(args.input, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ if args.limit:
+ data = data[: args.limit]
+ user_ids += [s["sample_id"] for s in data]
+
+ if not user_ids:
+ print("Error: no users specified. Pass user_ids or use --from-data", file=sys.stderr)
+ sys.exit(1)
+
+ user_ids = list(dict.fromkeys(user_ids)) # deduplicate, preserve order
+ print(f"Deleting memories for {len(user_ids)} user(s)...")
+
+ ok = sum(delete_user(api_key, memory_library_id, uid) for uid in user_ids)
+ print(f"\nDone: {ok}/{len(user_ids)} succeeded")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/bailian_memory/eval.py b/benchmark/locomo/bailian_memory/eval.py
new file mode 100644
index 000000000..577981930
--- /dev/null
+++ b/benchmark/locomo/bailian_memory/eval.py
@@ -0,0 +1,752 @@
+"""
+Evaluate LoCoMo QA via OpenClaw + modelstudio-memory-for-openclaw plugin.
+
+For each sample, updates ~/.openclaw/openclaw.json to set userId = sample_id,
+restarts the OpenClaw gateway, then sends questions via /v1/responses API.
+
+Prerequisites:
+ - Conversations already ingested via ingest.py (userId = sample_id)
+ - OpenClaw installed with modelstudio-memory-for-openclaw plugin configured
+
+Usage:
+ # Run QA + auto-judge
+ python eval.py --openclaw-url http://127.0.0.1:18789 --openclaw-token xxx \\
+ --judge --judge-token xxx
+
+ # Single sample
+ python eval.py --sample conv-26
+
+ # Only judge an existing result CSV (skip QA)
+ python eval.py --judge-only --output result/qa_results.csv --judge-token xxx
+"""
+
+import argparse
+import csv
+import json
+import os
+import subprocess
+import sys
+import threading
+import time
+from collections import defaultdict
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+import requests
+from dotenv import load_dotenv
+
+load_dotenv(Path.home() / ".openviking_benchmark_env")
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "data" / "locomo10.json")
+DEFAULT_OUTPUT_PATH = str(SCRIPT_DIR / "result" / "qa_results.csv")
+DEFAULT_OPENCLAW_URL = "http://127.0.0.1:18789"
+OPENCLAW_CONFIG_PATH = Path.home() / ".openclaw" / "openclaw.json"
+PLUGIN_KEY = "modelstudio-memory-for-openclaw"
+
+_openclaw_config_lock = threading.Lock()
+
+# ---------------------------------------------------------------------------
+# openclaw.json config helpers
+# ---------------------------------------------------------------------------
+
+def _update_plugin_user(sample_id: str) -> None:
+ """
+ Rewrite ~/.openclaw/openclaw.json so that the plugin uses sample_id as userId.
+ Must be called while holding _openclaw_config_lock.
+ """
+ with open(OPENCLAW_CONFIG_PATH, "r", encoding="utf-8") as f:
+ config = json.load(f)
+
+ entries = config.setdefault("plugins", {}).setdefault("entries", {})
+ plugin_entry = entries.setdefault(PLUGIN_KEY, {})
+ plugin_entry["enabled"] = True
+ plugin_entry.setdefault("config", {})["userId"] = sample_id
+
+ tmp = str(OPENCLAW_CONFIG_PATH) + ".tmp"
+ with open(tmp, "w", encoding="utf-8") as f:
+ json.dump(config, f, indent=2, ensure_ascii=False)
+ os.replace(tmp, str(OPENCLAW_CONFIG_PATH))
+
+
+def _restart_gateway(base_url: str, startup_timeout: int = 30) -> None:
+ """
+ Kill and restart the OpenClaw gateway, wait until healthy.
+ Must be called while holding _openclaw_config_lock.
+ """
+ try:
+ subprocess.run(["pkill", "-f", "openclaw gateway"], capture_output=True)
+ except Exception as e:
+ print(f" [gateway] pkill failed: {e}", file=sys.stderr)
+
+ try:
+ subprocess.Popen(
+ ["openclaw", "gateway"],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ )
+ except Exception as e:
+ raise RuntimeError(f"Failed to start openclaw gateway: {e}")
+
+ time.sleep(5)
+
+ health_url = f"{base_url.rstrip('/')}/health"
+ deadline = time.time() + startup_timeout
+ while time.time() < deadline:
+ try:
+ resp = requests.get(health_url, timeout=2)
+ if resp.status_code < 500:
+ return
+ except Exception:
+ pass
+ time.sleep(0.5)
+ raise RuntimeError(f"OpenClaw gateway did not become ready within {startup_timeout}s")
+
+
+CATEGORY_NAMES = {
+ 1: "single-hop",
+ 2: "multi-hop",
+ 3: "temporal",
+ 4: "world-knowledge",
+ 5: "adversarial",
+}
+
+# ---------------------------------------------------------------------------
+# LoCoMo data loading
+# ---------------------------------------------------------------------------
+
+def load_locomo_data(path: str, sample_id: Optional[str] = None) -> list[dict]:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ if sample_id is not None:
+ try:
+ idx = int(sample_id)
+ if idx < 0 or idx >= len(data):
+ raise ValueError(f"Sample index {idx} out of range (0-{len(data) - 1})")
+ return [data[idx]]
+ except ValueError:
+ pass
+ matched = [s for s in data if s.get("sample_id") == sample_id]
+ if not matched:
+ raise ValueError(f"sample_id '{sample_id}' not found")
+ return matched
+
+ return data
+
+
+def get_sample_last_session_date(sample: dict) -> Optional[str]:
+ conv = sample.get("conversation", {})
+ session_keys = [k for k in conv if k.startswith("session_") and "date_time" not in k]
+ if not session_keys:
+ return None
+
+ def sess_num(k: str) -> int:
+ try:
+ return int(k.split("_")[1])
+ except ValueError:
+ return 0
+
+ for sk in sorted(session_keys, key=sess_num, reverse=True):
+ if conv.get(sk):
+ dt_key = f"{sk}_date_time"
+ date_str = conv.get(dt_key, "")
+ if date_str and " on " in date_str:
+ try:
+ date_part = date_str.split(" on ")[-1]
+ dt = datetime.strptime(date_part.strip(), "%d %B, %Y")
+ return dt.strftime("%Y-%m-%d")
+ except ValueError:
+ pass
+ return None
+
+
+def load_qa_items(
+ data: list[dict],
+ skip_adversarial: bool = True,
+ question_index: Optional[int] = None,
+ count: Optional[int] = None,
+) -> list[dict]:
+ items = []
+ for sample in data:
+ sample_id = sample.get("sample_id", "")
+ question_time = get_sample_last_session_date(sample)
+
+ for q_idx, qa in enumerate(sample.get("qa", [])):
+ if question_index is not None and q_idx != question_index:
+ continue
+ category = qa.get("category", 0)
+ if skip_adversarial and str(category) == "5":
+ continue
+ items.append(
+ {
+ "sample_id": sample_id,
+ "question_index": q_idx,
+ "question_id": f"{sample_id}_qa{q_idx}",
+ "question": qa["question"],
+ "answer": str(qa["answer"]),
+ "category": category,
+ "category_name": CATEGORY_NAMES.get(category, "unknown"),
+ "evidence": qa.get("evidence", []),
+ "question_time": question_time,
+ }
+ )
+
+ if count is not None:
+ items = items[:count]
+ return items
+
+
+# ---------------------------------------------------------------------------
+# CSV helpers
+# ---------------------------------------------------------------------------
+
+QA_FIELDNAMES = [
+ "sample_id",
+ "question_index",
+ "question_id",
+ "question",
+ "answer",
+ "category",
+ "category_name",
+ "question_time",
+ "evidence",
+ "response",
+ "input_tokens",
+ "output_tokens",
+ "total_tokens",
+ "time_cost",
+ "result",
+ "reasoning",
+ "timestamp",
+]
+
+
+def load_processed_ids(output_path: str) -> set[str]:
+ processed: set[str] = set()
+ if not os.path.exists(output_path):
+ return processed
+ try:
+ with open(output_path, "r", encoding="utf-8", newline="") as f:
+ for row in csv.DictReader(f):
+ if row.get("response"):
+ processed.add(row.get("question_id", ""))
+ except Exception as e:
+ print(f"[WARN] Error reading {output_path}: {e}", file=sys.stderr)
+ return processed
+
+
+def save_row(output_path: str, row: dict, write_lock: threading.Lock) -> None:
+ with write_lock:
+ file_exists = os.path.exists(output_path)
+ with open(output_path, "a", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=QA_FIELDNAMES, extrasaction="ignore")
+ if not file_exists:
+ writer.writeheader()
+ writer.writerow(row)
+ f.flush()
+
+
+# ---------------------------------------------------------------------------
+# OpenClaw agent
+# ---------------------------------------------------------------------------
+
+def extract_response_text(body: dict) -> str:
+ try:
+ for item in body.get("output", []):
+ if item.get("type") == "message":
+ for content in item.get("content", []):
+ if content.get("type") == "output_text":
+ return content.get("text", "")
+ for item in body.get("output", []):
+ if "text" in item:
+ return item["text"]
+ for content in item.get("content", []):
+ if "text" in content:
+ return content["text"]
+ except Exception:
+ pass
+ return f"[ERROR: could not parse response: {body}]"
+
+
+def get_openclaw_session_id(session_key: str) -> Optional[str]:
+ sessions_file = os.path.expanduser("~/.openclaw/agents/main/sessions/sessions.json")
+ try:
+ with open(sessions_file, "r") as f:
+ data = json.load(f)
+ return data.get(session_key, {}).get("sessionId")
+ except Exception:
+ return None
+
+
+def parse_session_tokens(session_id: str) -> dict:
+ sessions_dir = os.path.expanduser("~/.openclaw/agents/main/sessions")
+ src = os.path.join(sessions_dir, f"{session_id}.jsonl")
+ total_input = total_output = total_cache_read = 0
+ try:
+ with open(src, "r", encoding="utf-8") as f:
+ for line in f:
+ line = line.strip()
+ if not line:
+ continue
+ obj = json.loads(line)
+ if obj.get("type") == "message" and obj.get("message", {}).get("role") == "assistant":
+ usage = obj["message"].get("usage", {})
+ total_input += usage.get("input", 0)
+ total_output += usage.get("output", 0)
+ total_cache_read += usage.get("cacheRead", 0)
+ except Exception:
+ pass
+ return {
+ "input_tokens": total_input,
+ "output_tokens": total_output,
+ "total_tokens": total_input + total_output + total_cache_read,
+ }
+
+
+def send_to_openclaw(
+ question: str,
+ sample_id: str,
+ base_url: str,
+ token: str,
+ question_time: Optional[str] = None,
+ question_id: Optional[str] = None,
+ retries: int = 2,
+) -> tuple[str, dict, float]:
+ """Send a question to OpenClaw. Returns (response_text, usage, time_cost)."""
+ if question_time:
+ input_text = f"Current date: {question_time}. Answer the question directly: {question}"
+ else:
+ input_text = f"Answer the question directly: {question}"
+
+ import uuid
+ suffix = uuid.uuid4().hex[:8]
+ session_key = f"locomo-eval-{question_id}-{suffix}" if question_id else f"locomo-eval-{suffix}"
+ url = f"{base_url.rstrip('/')}/v1/responses"
+ headers = {
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {token}",
+ "X-OpenClaw-Session-Key": session_key,
+ }
+ payload = {
+ "model": "openclaw",
+ "input": input_text,
+ "stream": False,
+ "user": sample_id,
+ }
+
+ last_exc: Optional[Exception] = None
+ t0 = time.time()
+ for attempt in range(retries + 1):
+ try:
+ resp = requests.post(url, json=payload, headers=headers, timeout=300)
+ resp.raise_for_status()
+ body = resp.json()
+ response_text = extract_response_text(body)
+
+ time.sleep(1)
+ session_id = get_openclaw_session_id(session_key)
+ if session_id:
+ usage = parse_session_tokens(session_id)
+ else:
+ usage = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
+
+ return response_text, usage, time.time() - t0
+ except Exception as e:
+ last_exc = e
+ if attempt < retries:
+ print(f" [retry {attempt + 1}/{retries}] {e}", file=sys.stderr)
+
+ raise RuntimeError(f"OpenClaw request failed after {retries + 1} attempts: {last_exc}")
+
+
+# ---------------------------------------------------------------------------
+# LLM judge
+# ---------------------------------------------------------------------------
+
+JUDGE_SYSTEM_PROMPT = "You are an expert grader that determines if answers to questions match a gold standard answer"
+
+JUDGE_ACCURACY_PROMPT = """Your task is to label an answer to a question as 'CORRECT' or 'WRONG'. You will be given the following data:
+ (1) a question (posed by one user to another user),
+ (2) a 'gold' (ground truth) answer,
+ (3) a generated answer
+which you will score as CORRECT/WRONG.
+
+The point of the question is to ask about something one user should know about the other user based on their prior conversations.
+The gold answer will usually be a concise and short answer that includes the referenced topic, for example:
+Question: Do you remember what I got the last time I went to Hawaii?
+Gold answer: A shell necklace
+The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.
+
+For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date.
+
+Now it's time for the real question:
+Question: {question}
+Gold answer: {gold_answer}
+Generated answer: {response}
+
+First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG.
+Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.
+
+Respond with JSON only: {{"reasoning": "your explanation", "is_correct": "CORRECT" or "WRONG"}}"""
+
+
+def judge_answer(
+ question: str,
+ gold_answer: str,
+ response: str,
+ judge_base_url: str,
+ judge_token: str,
+ judge_model: str,
+) -> tuple[str, str]:
+ from openai import OpenAI
+ client = OpenAI(base_url=judge_base_url, api_key=judge_token)
+ prompt = JUDGE_ACCURACY_PROMPT.format(
+ question=question, gold_answer=gold_answer, response=response
+ )
+ try:
+ resp = client.chat.completions.create(
+ model=judge_model,
+ messages=[
+ {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
+ {"role": "user", "content": prompt},
+ ],
+ temperature=0,
+ timeout=60,
+ )
+ content = resp.choices[0].message.content.strip()
+ start, end = content.find("{"), content.rfind("}")
+ if start != -1 and end != -1:
+ parsed = json.loads(content[start : end + 1])
+ label = "CORRECT" if parsed.get("is_correct", "WRONG").strip().upper() == "CORRECT" else "WRONG"
+ return label, parsed.get("reasoning", "")
+ return "WRONG", f"[PARSE ERROR] {content}"
+ except Exception as e:
+ return "WRONG", f"[API ERROR] {e}"
+
+
+# ---------------------------------------------------------------------------
+# Accuracy summary
+# ---------------------------------------------------------------------------
+
+def print_accuracy(rows: list[dict]) -> None:
+ graded = [r for r in rows if r.get("result") in ("CORRECT", "WRONG")]
+ if not graded:
+ print("\n[INFO] No graded results to summarize.", file=sys.stderr)
+ return
+
+ correct_total = sum(1 for r in graded if r["result"] == "CORRECT")
+ print("\n=== Accuracy Summary ===", file=sys.stderr)
+ print(f" Overall: {correct_total}/{len(graded)} = {correct_total/len(graded):.2%}", file=sys.stderr)
+
+ by_cat: dict[str, list[str]] = {}
+ for r in graded:
+ cat = r.get("category_name") or str(r.get("category", "?"))
+ by_cat.setdefault(cat, []).append(r["result"])
+
+ print(" By category:", file=sys.stderr)
+ for cat, results in sorted(by_cat.items()):
+ n = sum(1 for r in results if r == "CORRECT")
+ print(f" {cat:20s}: {n}/{len(results)} = {n/len(results):.2%}", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------------
+# Main runners
+# ---------------------------------------------------------------------------
+
+def run_qa(args: argparse.Namespace) -> None:
+ openclaw_token = args.openclaw_token or os.environ.get("OPENCLAW_GATEWAY_TOKEN", "")
+
+ judge_token = args.judge_token or os.environ.get("ARK_API_KEY", os.environ.get("OPENAI_API_KEY", ""))
+ if args.judge and not judge_token:
+ print("Error: judge token required (--judge-token or ARK_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ data = load_locomo_data(args.input, args.sample)
+ qa_items = load_qa_items(
+ data,
+ skip_adversarial=args.skip_adversarial,
+ question_index=args.question_index,
+ count=args.count,
+ )
+ print(f"[INFO] {len(qa_items)} QA items loaded", file=sys.stderr)
+
+ Path(args.output).parent.mkdir(parents=True, exist_ok=True)
+
+ # Remove ERROR rows from CSV before loading processed ids
+ if os.path.exists(args.output):
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ fieldnames = reader.fieldnames or QA_FIELDNAMES
+ clean_rows = [r for r in reader if not r.get("response", "").startswith("[ERROR]")]
+ with open(args.output, "w", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
+ writer.writeheader()
+ writer.writerows(clean_rows)
+
+ processed_ids = load_processed_ids(args.output)
+ remaining = [qa for qa in qa_items if qa["question_id"] not in processed_ids]
+ print(
+ f"[INFO] {len(processed_ids)} already done, {len(remaining)} remaining",
+ file=sys.stderr,
+ )
+
+ if not remaining:
+ print("[INFO] All questions already processed.", file=sys.stderr)
+ else:
+ write_lock = threading.Lock()
+ total = len(remaining)
+
+ # Group remaining questions by sample_id to minimize gateway restarts
+ by_sample: dict[str, list[tuple[int, dict]]] = defaultdict(list)
+ for i, qa in enumerate(remaining):
+ by_sample[qa["sample_id"]].append((i + 1, qa))
+
+ def run_one(qa: dict, idx: int) -> None:
+ print(
+ f" [{idx}/{total}] {qa['question_id']}: {qa['question'][:60]}...",
+ file=sys.stderr,
+ )
+ try:
+ response, usage, time_cost = send_to_openclaw(
+ qa["question"],
+ qa["sample_id"],
+ args.openclaw_url,
+ openclaw_token,
+ qa.get("question_time"),
+ qa["question_id"],
+ )
+ except Exception as e:
+ response = f"[ERROR] {e}"
+ usage = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
+ time_cost = 0.0
+
+ result_label, reasoning = "", ""
+ if args.judge and response and not response.startswith("[ERROR]"):
+ result_label, reasoning = judge_answer(
+ qa["question"],
+ qa["answer"],
+ response,
+ args.judge_base_url,
+ judge_token,
+ args.judge_model,
+ )
+
+ row = {
+ "sample_id": qa["sample_id"],
+ "question_index": qa["question_index"],
+ "question_id": qa["question_id"],
+ "question": qa["question"],
+ "answer": qa["answer"],
+ "category": qa["category"],
+ "category_name": qa["category_name"],
+ "question_time": qa.get("question_time", ""),
+ "evidence": json.dumps(qa.get("evidence", [])),
+ "response": response,
+ "input_tokens": usage.get("input_tokens", 0),
+ "output_tokens": usage.get("output_tokens", 0),
+ "total_tokens": usage.get("total_tokens", 0),
+ "time_cost": round(time_cost, 2),
+ "result": result_label,
+ "reasoning": reasoning,
+ "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
+ }
+ save_row(args.output, row, write_lock)
+ label_str = f" → {result_label}" if result_label else ""
+ print(f" [{idx}/{total}] done {time_cost:.1f}s{label_str}", file=sys.stderr)
+
+ # Process sample by sample: restart gateway once per sample to pick up new userId
+ for sample_id, qa_list in by_sample.items():
+ print(f"\n[INFO] Switching to sample {sample_id}, restarting openclaw gateway...", file=sys.stderr)
+ with _openclaw_config_lock:
+ _update_plugin_user(sample_id)
+ _restart_gateway(args.openclaw_url)
+ print(f"[INFO] Gateway ready, running {len(qa_list)} questions for {sample_id}", file=sys.stderr)
+
+ with ThreadPoolExecutor(max_workers=args.threads) as executor:
+ futures = {
+ executor.submit(run_one, qa, idx): qa
+ for idx, qa in qa_list
+ }
+ for fut in as_completed(futures):
+ try:
+ fut.result()
+ except Exception as e:
+ qa = futures[fut]
+ print(f" [ERROR] {qa['question_id']}: {e}", file=sys.stderr)
+
+ # Print token and latency summary
+ try:
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ rows = list(csv.DictReader(f))
+ total_input = sum(int(r.get("input_tokens") or 0) for r in rows)
+ total_input_with_cache = sum(
+ int(r.get("total_tokens") or 0) - int(r.get("output_tokens") or 0) for r in rows
+ )
+ times = [float(r["time_cost"]) for r in rows if r.get("time_cost")]
+ avg_time = sum(times) / len(times) if times else 0.0
+ print(f"\n=== Token & Latency Summary ===", file=sys.stderr)
+ print(f" Total input tokens : {total_input}", file=sys.stderr)
+ print(f" Total input tokens (with cache): {total_input_with_cache}", file=sys.stderr)
+ print(f" Avg time per query : {avg_time:.1f}s", file=sys.stderr)
+ except Exception:
+ pass
+
+ if args.judge:
+ try:
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ print_accuracy(list(csv.DictReader(f)))
+ except Exception:
+ pass
+
+
+def run_judge_only(args: argparse.Namespace) -> None:
+ """Grade responses in an existing CSV that lack a result label."""
+ if not os.path.exists(args.output):
+ print(f"Error: output file not found: {args.output}", file=sys.stderr)
+ sys.exit(1)
+
+ judge_token = args.judge_token or os.environ.get("ARK_API_KEY", os.environ.get("OPENAI_API_KEY", ""))
+ if not judge_token:
+ print("Error: judge token required (--judge-token or ARK_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ fieldnames = list(reader.fieldnames or QA_FIELDNAMES)
+ rows = list(reader)
+
+ for extra in ("result", "reasoning"):
+ if extra not in fieldnames:
+ fieldnames.append(extra)
+
+ ungraded_indices = [i for i, row in enumerate(rows) if not row.get("result")]
+ print(f"[INFO] {len(rows)} rows total, {len(ungraded_indices)} ungraded", file=sys.stderr)
+
+ if not ungraded_indices:
+ print("[INFO] All rows already graded.", file=sys.stderr)
+ print_accuracy(rows)
+ return
+
+ file_lock = threading.Lock()
+
+ def grade_one(idx: int) -> None:
+ row = rows[idx]
+ label, reasoning = judge_answer(
+ row.get("question", ""),
+ row.get("answer", ""),
+ row.get("response", ""),
+ args.judge_base_url,
+ judge_token,
+ args.judge_model,
+ )
+ row["result"] = label
+ row["reasoning"] = reasoning
+ with file_lock:
+ tmp = args.output + ".tmp"
+ with open(tmp, "w", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
+ writer.writeheader()
+ writer.writerows(rows)
+ os.replace(tmp, args.output)
+ print(f" Graded {row.get('question_id', '?')}: {label}", file=sys.stderr)
+
+ with ThreadPoolExecutor(max_workers=args.threads) as executor:
+ futures = [executor.submit(grade_one, idx) for idx in ungraded_indices]
+ for fut in as_completed(futures):
+ try:
+ fut.result()
+ except Exception as e:
+ print(f"[ERROR] grading failed: {e}", file=sys.stderr)
+
+ print_accuracy(rows)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ description="Evaluate LoCoMo QA via OpenClaw + modelstudio-memory-for-openclaw plugin",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=__doc__,
+ )
+
+ # Data selection
+ parser.add_argument("--input", default=DEFAULT_DATA_PATH, help="Path to locomo10.json")
+ parser.add_argument("--output", default=DEFAULT_OUTPUT_PATH, help="Path to output CSV")
+ parser.add_argument(
+ "--sample",
+ default=None,
+ help="Sample index (int) or sample_id (e.g. conv-26). Default: all.",
+ )
+ parser.add_argument(
+ "--question-index",
+ type=int,
+ default=None,
+ help="Single question index (0-based) within the sample.",
+ )
+ parser.add_argument("--count", type=int, default=None, help="Max QA items to process.")
+ parser.add_argument(
+ "--no-skip-adversarial",
+ dest="skip_adversarial",
+ action="store_false",
+ default=True,
+ help="Include category-5 adversarial questions (skipped by default).",
+ )
+ parser.add_argument("--threads", type=int, default=10, help="Concurrent threads (default: 10)")
+
+ # OpenClaw
+ parser.add_argument(
+ "--openclaw-url",
+ default=DEFAULT_OPENCLAW_URL,
+ help=f"OpenClaw gateway URL (default: {DEFAULT_OPENCLAW_URL})",
+ )
+ parser.add_argument(
+ "--openclaw-token",
+ default=None,
+ help="OpenClaw auth token (or OPENCLAW_GATEWAY_TOKEN env var)",
+ )
+
+ # Judge
+ parser.add_argument(
+ "--judge",
+ action="store_true",
+ default=False,
+ help="Auto-judge each response right after answering.",
+ )
+ parser.add_argument(
+ "--judge-only",
+ action="store_true",
+ default=False,
+ help="Skip QA; only grade ungraded responses in the existing --output CSV.",
+ )
+ parser.add_argument(
+ "--judge-base-url",
+ default="https://ark.cn-beijing.volces.com/api/v3",
+ help="OpenAI-compatible API base URL for judge (default: Volcengine ARK)",
+ )
+ parser.add_argument(
+ "--judge-token",
+ default=None,
+ help="API token for judge (or ARK_API_KEY / OPENAI_API_KEY env var)",
+ )
+ parser.add_argument(
+ "--judge-model",
+ default="doubao-seed-2-0-pro-260215",
+ help="Judge model (default: doubao-seed-2-0-pro-260215)",
+ )
+
+ args = parser.parse_args()
+
+ if args.judge_only:
+ run_judge_only(args)
+ else:
+ run_qa(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/bailian_memory/ingest.py b/benchmark/locomo/bailian_memory/ingest.py
new file mode 100644
index 000000000..4305d610c
--- /dev/null
+++ b/benchmark/locomo/bailian_memory/ingest.py
@@ -0,0 +1,366 @@
+"""
+Ingest LoCoMo conversations into Alibaba Cloud Bailian (ModelStudio) Memory.
+
+Each sample gets an isolated memory namespace keyed by sample_id.
+Speaker messages are formatted with speaker names for better memory extraction.
+
+Config via ~/.openviking_benchmark_env:
+ DASHSCOPE_API_KEY=sk-xxx
+ BAILIAN_MEMORY_LIBRARY_ID=xxx
+ BAILIAN_PROFILE_SCHEMA_ID=xxx # optional, omit to skip profile extraction
+
+Usage:
+ python ingest.py
+ python ingest.py --sample conv-26
+ python ingest.py --sample conv-26 --sessions 1-4
+ python ingest.py --force-ingest
+"""
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Optional
+
+import requests
+from dotenv import load_dotenv
+
+load_dotenv(Path.home() / ".openviking_benchmark_env")
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "data" / "locomo10.json")
+DEFAULT_RECORD_PATH = str(SCRIPT_DIR / "result" / ".ingest_record.json")
+DEFAULT_LOG_PATH = str(SCRIPT_DIR / "result" / "ingest_errors.log")
+
+BAILIAN_MEMORY_BASE_URL = "https://dashscope.aliyuncs.com/api/v2/apps/memory"
+
+
+# ---------------------------------------------------------------------------
+# Bailian Memory API
+# ---------------------------------------------------------------------------
+
+def _auth_headers(api_key: str) -> dict:
+ return {
+ "Authorization": f"Bearer {api_key}",
+ "Content-Type": "application/json",
+ }
+
+
+def add_memory(
+ api_key: str,
+ memory_library_id: str,
+ messages: list[dict],
+ user_id: str,
+ profile_schema_id: Optional[str] = None,
+ meta_data: Optional[dict] = None,
+) -> dict:
+ """Call Bailian Memory AddMemory API."""
+ payload: dict = {
+ "messages": messages,
+ "user_id": user_id,
+ "memory_library_id": memory_library_id,
+ }
+ if profile_schema_id:
+ payload["profile_schema"] = profile_schema_id
+ if meta_data:
+ payload["meta_data"] = meta_data
+
+ resp = requests.post(
+ f"{BAILIAN_MEMORY_BASE_URL}/add",
+ headers=_auth_headers(api_key),
+ json=payload,
+ timeout=120,
+ )
+ resp.raise_for_status()
+ return resp.json()
+
+
+# ---------------------------------------------------------------------------
+# LoCoMo data loading
+# ---------------------------------------------------------------------------
+
+def load_locomo_data(path: str, sample_id: Optional[str] = None) -> list[dict]:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ if sample_id is not None:
+ try:
+ idx = int(sample_id)
+ if idx < 0 or idx >= len(data):
+ raise ValueError(f"Sample index {idx} out of range (0-{len(data) - 1})")
+ return [data[idx]]
+ except ValueError:
+ pass
+ matched = [s for s in data if s.get("sample_id") == sample_id]
+ if not matched:
+ raise ValueError(f"sample_id '{sample_id}' not found")
+ return matched
+
+ return data
+
+
+def parse_session_range(s: str) -> tuple[int, int]:
+ if "-" in s:
+ lo, hi = s.split("-", 1)
+ return int(lo), int(hi)
+ n = int(s)
+ return n, n
+
+
+def build_session_messages(
+ item: dict,
+ session_range: Optional[tuple[int, int]] = None,
+) -> list[dict]:
+ conv = item["conversation"]
+
+ session_keys = sorted(
+ [k for k in conv if k.startswith("session_") and not k.endswith("_date_time")],
+ key=lambda k: int(k.split("_")[1]),
+ )
+
+ sessions = []
+ for sk in session_keys:
+ sess_num = int(sk.split("_")[1])
+ if session_range:
+ lo, hi = session_range
+ if sess_num < lo or sess_num > hi:
+ continue
+
+ raw_messages = conv[sk]
+ if not isinstance(raw_messages, list) or not raw_messages:
+ continue
+
+ dt_key = f"{sk}_date_time"
+ date_time = conv.get(dt_key, "")
+
+ messages = []
+ if date_time:
+ messages.append({
+ "role": "user",
+ "content": f"[System]: This conversation took place on {date_time}.",
+ })
+ for msg in raw_messages:
+ speaker = msg.get("speaker", "")
+ text = msg.get("text", "")
+ blip = msg.get("blip_caption", "")
+ content = f"[{speaker}]: {text}"
+ if blip:
+ content += f" (image: {blip})"
+ messages.append({"role": "user", "content": content})
+
+ sessions.append({
+ "messages": messages,
+ "meta": {
+ "sample_id": item["sample_id"],
+ "session_key": sk,
+ "date_time": date_time,
+ "speaker_a": conv.get("speaker_a", ""),
+ "speaker_b": conv.get("speaker_b", ""),
+ },
+ })
+
+ return sessions
+
+
+# ---------------------------------------------------------------------------
+# Ingest record (progress tracking)
+# ---------------------------------------------------------------------------
+
+def load_ingest_record(path: str) -> dict:
+ try:
+ with open(path, "r", encoding="utf-8") as f:
+ return json.load(f)
+ except (FileNotFoundError, json.JSONDecodeError):
+ return {}
+
+
+def save_ingest_record(record: dict, path: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(record, f, indent=2, ensure_ascii=False)
+
+
+def is_already_ingested(sample_id: str, session_key: str, record: dict) -> bool:
+ key = f"bailian:{sample_id}:{session_key}"
+ return key in record and record[key].get("success", False)
+
+
+def mark_ingested(
+ sample_id: str,
+ session_key: str,
+ record: dict,
+ memory_node_ids: list[str],
+ meta: Optional[dict] = None,
+) -> None:
+ key = f"bailian:{sample_id}:{session_key}"
+ record[key] = {
+ "success": True,
+ "timestamp": int(time.time()),
+ "memory_node_ids": memory_node_ids,
+ "meta": meta or {},
+ }
+
+
+def write_error_log(path: str, sample_id: str, session_key: str, error: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ ts = time.strftime("%Y-%m-%d %H:%M:%S")
+ with open(path, "a", encoding="utf-8") as f:
+ f.write(f"[{ts}] ERROR [{sample_id}/{session_key}]: {error}\n")
+
+
+# ---------------------------------------------------------------------------
+# Core ingest logic
+# ---------------------------------------------------------------------------
+
+def run_ingest(args: argparse.Namespace) -> None:
+ api_key = os.environ.get("DASHSCOPE_API_KEY", "")
+ if not api_key:
+ print("Error: DASHSCOPE_API_KEY not set in ~/.openviking_benchmark_env or environment", file=sys.stderr)
+ sys.exit(1)
+
+ memory_library_id = os.environ.get("BAILIAN_MEMORY_LIBRARY_ID", "")
+ if not memory_library_id:
+ print("Error: BAILIAN_MEMORY_LIBRARY_ID not set in ~/.openviking_benchmark_env or environment", file=sys.stderr)
+ sys.exit(1)
+
+ # Optional: only used if set in env
+ profile_schema_id = os.environ.get("BAILIAN_PROFILE_SCHEMA_ID", "") or None
+ if profile_schema_id:
+ print(f"[INFO] profile_schema_id={profile_schema_id} (user portrait extraction enabled)", file=sys.stderr)
+ else:
+ print("[INFO] BAILIAN_PROFILE_SCHEMA_ID not set, skipping profile extraction", file=sys.stderr)
+
+ session_range = parse_session_range(args.sessions) if args.sessions else None
+
+ if args.clear_ingest_record:
+ ingest_record: dict = {}
+ save_ingest_record(ingest_record, args.record)
+ print("[INFO] Cleared existing ingest records", file=sys.stderr)
+ else:
+ ingest_record = load_ingest_record(args.record)
+
+ samples = load_locomo_data(args.input, args.sample)
+ if args.limit:
+ samples = samples[: args.limit]
+ print(f"[INFO] Loaded {len(samples)} sample(s)", file=sys.stderr)
+
+ total_sessions = 0
+ success_count = 0
+ skip_count = 0
+ error_count = 0
+
+ for item in samples:
+ sample_id: str = item["sample_id"]
+ sessions = build_session_messages(item, session_range)
+ print(f"\n=== Sample {sample_id} ({len(sessions)} sessions) ===", file=sys.stderr)
+
+ for sess in sessions:
+ meta = sess["meta"]
+ session_key = meta["session_key"]
+ label = f"{session_key} ({meta['date_time']})"
+ total_sessions += 1
+
+ if not args.force_ingest and is_already_ingested(sample_id, session_key, ingest_record):
+ print(f" [{label}] SKIP (already ingested)", file=sys.stderr)
+ skip_count += 1
+ continue
+
+ print(f" [{label}] ingesting {len(sess['messages'])} messages ...", file=sys.stderr)
+ t0 = time.time()
+
+ try:
+ result = add_memory(
+ api_key=api_key,
+ memory_library_id=memory_library_id,
+ messages=sess["messages"],
+ user_id=sample_id,
+ profile_schema_id=profile_schema_id,
+ meta_data={
+ "session_key": session_key,
+ "date_time": meta["date_time"],
+ "speaker_a": meta["speaker_a"],
+ "speaker_b": meta["speaker_b"],
+ },
+ )
+ elapsed = time.time() - t0
+
+ memory_node_ids = [
+ n.get("memory_node_id", "")
+ for n in result.get("memory_nodes", [])
+ if n.get("memory_node_id")
+ ]
+ mark_ingested(sample_id, session_key, ingest_record, memory_node_ids, meta)
+ save_ingest_record(ingest_record, args.record)
+ print(f" [{label}] OK nodes={len(memory_node_ids)} {elapsed:.1f}s", file=sys.stderr)
+ success_count += 1
+ except Exception as e:
+ elapsed = time.time() - t0
+ print(f" [{label}] ERROR: {e} {elapsed:.1f}s", file=sys.stderr)
+ write_error_log(args.error_log, sample_id, session_key, str(e))
+ error_count += 1
+
+ print(f"\n=== Ingest summary ===", file=sys.stderr)
+ print(f" Total sessions: {total_sessions}", file=sys.stderr)
+ print(f" Succeeded: {success_count}", file=sys.stderr)
+ print(f" Skipped: {skip_count}", file=sys.stderr)
+ print(f" Failed: {error_count}", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Ingest LoCoMo conversations into Bailian Memory")
+ parser.add_argument(
+ "--input",
+ default=DEFAULT_DATA_PATH,
+ help="Path to locomo10.json (default: ../data/locomo10.json)",
+ )
+ parser.add_argument(
+ "--sample",
+ default=None,
+ help="Sample index (0-based int) or sample_id string (e.g. conv-26). Default: all.",
+ )
+ parser.add_argument(
+ "--limit",
+ type=int,
+ default=None,
+ help="Max number of samples to ingest. Default: all.",
+ )
+ parser.add_argument(
+ "--sessions",
+ default=None,
+ help="Session range, e.g. '1-4' or '3'. Default: all.",
+ )
+ parser.add_argument(
+ "--record",
+ default=DEFAULT_RECORD_PATH,
+ help=f"Path to ingest progress record (default: {DEFAULT_RECORD_PATH})",
+ )
+ parser.add_argument(
+ "--error-log",
+ default=DEFAULT_LOG_PATH,
+ help=f"Path to error log (default: {DEFAULT_LOG_PATH})",
+ )
+ parser.add_argument(
+ "--force-ingest",
+ action="store_true",
+ default=False,
+ help="Re-ingest even if already recorded as done",
+ )
+ parser.add_argument(
+ "--clear-ingest-record",
+ action="store_true",
+ default=False,
+ help="Clear all existing ingest records before running",
+ )
+
+ args = parser.parse_args()
+ run_ingest(args)
+
+
+if __name__ == "__main__":
+ main()