Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

benchmark: add LoCoMo evaluation for Supermemory (#1401)

yangxinxin-726bbfd2c244e
diff --git a/.gitignore b/.gitignore
index 79b3a12ad..a5499bd75 100644
--- a/.gitignore
+++ b/.gitignore
@@ -195,3 +195,5 @@ examples/data/
openviking/_version.py
specs/
.trae/
+.codex/
+.ttadk/
diff --git a/benchmark/locomo/mem0/eval.py b/benchmark/locomo/mem0/eval.py
index fa273a718..24b1c155a 100644
--- a/benchmark/locomo/mem0/eval.py
+++ b/benchmark/locomo/mem0/eval.py
@@ -96,7 +96,7 @@ def _restart_openclaw_gateway(base_url: str, sample_id: str, startup_timeout: in
raise RuntimeError(f"Failed to start openclaw gateway: {e}")
# Wait for process to fully start before checking health
- time.sleep(3)
+ time.sleep(5)
# Wait until gateway is ready
health_url = f"{base_url.rstrip('/')}/health"
diff --git a/benchmark/locomo/supermemory/README.md b/benchmark/locomo/supermemory/README.md
new file mode 100644
index 000000000..d193378e6
--- /dev/null
+++ b/benchmark/locomo/supermemory/README.md
@@ -0,0 +1,168 @@
+# LoCoMo Benchmark — Supermemory Evaluation
+
+Evaluate [Supermemory](https://supermemory.ai) on the [LoCoMo](https://github.com/snap-stanford/locomo) benchmark using OpenClaw as the agent (same approach as the mem0 eval).
+
+## Overview
+
+Two-phase pipeline:
+
+1. **Ingest** — Import LoCoMo conversations into Supermemory (one `containerTag` per sample)
+2. **Eval** — Send QA questions to OpenClaw agent (which recalls from Supermemory internally), then judge answers with an LLM
+
+Before each sample, `eval.py` automatically:
+1. Updates `~/.openclaw/openclaw.json` to set `openclaw-supermemory.config.containerTag = sanitize(sample_id)`
+2. Switches `plugins.slots.memory` to `"openclaw-supermemory"`
+3. Restarts the OpenClaw gateway to pick up the new config
+
+> **Tag sanitization**: `conv-26` → `conv_26` (matches openclaw-supermemory's internal `sanitizeTag` logic). Both `ingest.py` and `eval.py` apply the same transformation automatically.
+
+## Prerequisites
+
+- [OpenClaw](https://openclaw.ai) installed and configured
+- `openclaw-supermemory` plugin installed (`~/.openclaw/extensions/openclaw-supermemory`)
+- `~/.openclaw/openclaw.json` with `openclaw-supermemory.config.apiKey` set
+- API keys in `~/.openviking_benchmark_env`:
+
+```env
+SUPERMEMORY_API_KEY=sm-...
+ARK_API_KEY=... # Volcengine ARK, used for judge LLM
+```
+
+- Python dependencies:
+
+```bash
+uv sync --frozen --extra dev
+pip install supermemory openai python-dotenv
+```
+
+## Data
+
+LoCoMo 10-sample dataset at `benchmark/locomo/data/locomo10.json`:
+
+- 10 samples (conversations between two people)
+- 1986 QA pairs across 5 categories:
+ - 1: single-hop
+ - 2: multi-hop
+ - 3: temporal
+ - 4: world-knowledge
+ - 5: adversarial (skipped by default)
+
+## Step 1 — Ingest
+
+Import conversations into Supermemory. Each sample is stored under `containerTag = sample_id` (e.g. `conv-26`).
+
+Sessions are formatted as date-prefixed JSON strings, matching the memorybench supermemory provider convention. Indexing is polled until both document and memory reach `done` status.
+
+```bash
+# Ingest all 10 samples
+python ingest.py
+
+# Ingest a single sample
+python ingest.py --sample conv-26
+
+# Ingest specific sessions only
+python ingest.py --sample conv-26 --sessions 1-4
+
+# Force re-ingest (ignore existing records)
+python ingest.py --sample conv-26 --force-ingest
+
+# Clear all ingest records and start fresh
+python ingest.py --clear-ingest-record
+```
+
+Key options:
+
+| Option | Description |
+|--------|-------------|
+| `--sample` | Sample ID (e.g. `conv-26`) or index (0-based). Default: all |
+| `--sessions` | Session range, e.g. `1-4` or `3`. Default: all |
+| `--limit` | Max samples to process |
+| `--force-ingest` | Re-ingest even if already recorded |
+| `--clear-ingest-record` | Clear `.ingest_record.json` before running |
+| `--no-wait-indexing` | Skip indexing poll (faster, no status check) |
+
+Ingest records are saved to `result/.ingest_record.json` to avoid duplicate ingestion.
+
+## Step 2 — Eval
+
+Send QA questions to OpenClaw agent and optionally judge answers.
+
+```bash
+# Run QA + judge for all samples (6 concurrent threads)
+python eval.py --threads 6 --judge
+
+# Single sample
+python eval.py --sample conv-26 --threads 6 --judge
+
+# First 12 questions only
+python eval.py --sample conv-26 --count 12 --threads 6 --judge
+
+# Judge-only (grade existing responses in CSV)
+python eval.py --judge-only
+```
+
+Key options:
+
+| Option | Description |
+|--------|-------------|
+| `--sample` | Sample ID or index. Default: all |
+| `--count` | Max QA items to process |
+| `--threads` | Concurrent threads per sample (default: 10) |
+| `--judge` | Auto-judge each response after answering |
+| `--judge-only` | Skip QA, only grade ungraded rows in existing CSV |
+| `--openclaw-url` | OpenClaw gateway URL (default: `http://127.0.0.1:18789`) |
+| `--openclaw-token` | Auth token (or `OPENCLAW_GATEWAY_TOKEN` env var) |
+| `--judge-base-url` | Judge API base URL (default: Volcengine ARK) |
+| `--judge-model` | Judge model (default: `doubao-seed-2-0-pro-260215`) |
+| `--output` | Output CSV path (default: `result/qa_results.csv`) |
+
+Results are written to `result/qa_results.csv`. Failed (`[ERROR]`) rows are automatically removed at the start of each run and retried.
+
+## Output
+
+`result/qa_results.csv` columns:
+
+| Column | Description |
+|--------|-------------|
+| `sample_id` | Conversation sample ID |
+| `question_id` | Unique question ID (e.g. `conv-26_qa0`) |
+| `question` / `answer` | Question and gold answer |
+| `category` / `category_name` | Question category |
+| `response` | Agent response |
+| `input_tokens` / `output_tokens` / `total_tokens` | LLM token usage |
+| `time_cost` | End-to-end latency (seconds) |
+| `result` | `CORRECT` or `WRONG` |
+| `reasoning` | Judge's reasoning |
+
+## Summary Output
+
+After eval completes:
+
+```
+=== Token & Latency Summary ===
+ Total input tokens : 123456
+ Avg time per query : 18.3s
+
+=== Accuracy Summary ===
+ Overall: 512/1540 = 33.25%
+ By category:
+ multi-hop : 120/321 = 37.38%
+ single-hop : 98/282 = 34.75%
+ temporal : 28/96 = 29.17%
+ world-knowledge : 266/841 = 31.63%
+```
+
+## Delete Supermemory Data
+
+```bash
+# Delete a specific sample's documents
+python delete_container.py conv-26
+
+# Delete all samples from the dataset
+python delete_container.py --from-data
+
+# Delete first N samples
+python delete_container.py --from-data --limit 3
+```
+
+> **Note:** `delete_container.py` uses `documents.list(containerTags=[tag])` + `documents.deleteBulk(ids=[...])` in batches of 100, and also clears the corresponding ingest records from `result/.ingest_record.json`.
diff --git a/benchmark/locomo/supermemory/delete_container.py b/benchmark/locomo/supermemory/delete_container.py
new file mode 100644
index 000000000..a1a08a4ba
--- /dev/null
+++ b/benchmark/locomo/supermemory/delete_container.py
@@ -0,0 +1,188 @@
+"""
+Delete all Supermemory documents for one or more containerTags (sample_ids).
+
+Usage:
+ # Delete a single container
+ python delete_container.py conv-26
+
+ # Delete multiple containers
+ python delete_container.py conv-26 conv-31 conv-45
+
+ # Delete first N samples from locomo10.json
+ python delete_container.py --from-data --limit 2
+
+ # Delete all samples from locomo10.json
+ python delete_container.py --from-data
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+load_dotenv(Path.home() / ".openviking_benchmark_env")
+
+try:
+ from supermemory import Supermemory
+except ImportError:
+ print("Error: supermemory package not installed. Run: pip install supermemory", file=sys.stderr)
+ sys.exit(1)
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "data" / "locomo10.json")
+DEFAULT_RECORD_PATH = str(SCRIPT_DIR / "result" / ".ingest_record.json")
+
+
+def sanitize_tag(raw: str) -> str:
+ """Sanitize a tag string to match openclaw-supermemory convention.
+ e.g. 'conv-26' -> 'conv_26'
+ """
+ tag = re.sub(r"[^a-zA-Z0-9_]", "_", raw)
+ tag = re.sub(r"_+", "_", tag)
+ tag = tag.strip("_")
+ return tag
+
+
+def wipe_container(client: Supermemory, container_tag: str) -> int:
+ """
+ Delete all documents in a containerTag using documents.list + deleteBulk.
+ Returns number of documents deleted.
+ """
+ all_ids: list[str] = []
+ page = 1
+
+ while True:
+ response = client.documents.list(
+ container_tags=[container_tag],
+ limit=100,
+ page=page,
+ )
+
+ memories = getattr(response, "memories", None)
+ if memories is None and isinstance(response, dict):
+ memories = response.get("memories", [])
+
+ if not memories:
+ break
+
+ for doc in memories:
+ doc_id = getattr(doc, "id", None) or (doc.get("id") if isinstance(doc, dict) else None)
+ if doc_id:
+ all_ids.append(doc_id)
+
+ # Check pagination
+ pagination = getattr(response, "pagination", None) or (response.get("pagination") if isinstance(response, dict) else None)
+ total_pages = None
+ if pagination:
+ total_pages = getattr(pagination, "totalPages", None) or (pagination.get("totalPages") if isinstance(pagination, dict) else None)
+
+ if total_pages is None or page >= total_pages:
+ break
+ page += 1
+
+ if not all_ids:
+ return 0
+
+ # Delete in batches of 100
+ deleted = 0
+ for i in range(0, len(all_ids), 100):
+ batch = all_ids[i : i + 100]
+ client.documents.delete_bulk(ids=batch)
+ deleted += len(batch)
+
+ return deleted
+
+
+def clear_ingest_records(container_tag: str, record_path: str) -> int:
+ """Remove ingest records for the given container_tag. Returns count removed."""
+ try:
+ with open(record_path, "r", encoding="utf-8") as f:
+ record = json.load(f)
+ except (FileNotFoundError, json.JSONDecodeError):
+ return 0
+
+ # Records are keyed as "supermemory:{sample_id}:{session_key}"
+ # Match by sanitized sample_id to handle keys like "conv-26" vs "conv_26"
+ keys_to_remove = [k for k in record if len(k.split(":")) >= 2 and sanitize_tag(k.split(":")[1]) == container_tag]
+
+ for k in keys_to_remove:
+ del record[k]
+
+ with open(record_path, "w", encoding="utf-8") as f:
+ json.dump(record, f, indent=2, ensure_ascii=False)
+
+ return len(keys_to_remove)
+
+
+def delete_container(client: Supermemory, sample_id: str, record_path: str) -> bool:
+ container_tag = sanitize_tag(sample_id)
+ print(f" [containerTag={container_tag}] listing documents...", file=sys.stderr)
+
+ try:
+ deleted = wipe_container(client, container_tag)
+ if deleted == 0:
+ print(f" [WARN] No documents found (may already be deleted)", file=sys.stderr)
+ else:
+ print(f" [OK] Deleted {deleted} documents", file=sys.stderr)
+ except Exception as e:
+ print(f" [ERROR] Failed to delete documents: {e}", file=sys.stderr)
+ return False
+
+ removed = clear_ingest_records(container_tag, record_path)
+ if removed:
+ print(f" Cleared {removed} ingest record(s)", file=sys.stderr)
+
+ return True
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Delete all Supermemory documents for given sample(s)")
+ parser.add_argument("samples", nargs="*", help="sample_id(s) to delete (e.g. conv-26 conv-31)")
+ parser.add_argument("--api-key", default=None, help="Supermemory API key (or SUPERMEMORY_API_KEY env var)")
+ parser.add_argument("--from-data", action="store_true", help="load sample_ids from locomo10.json")
+ parser.add_argument("--input", default=DEFAULT_DATA_PATH, help="path to locomo10.json")
+ parser.add_argument("--limit", type=int, default=None, help="max samples to delete (with --from-data)")
+ parser.add_argument(
+ "--record",
+ default=DEFAULT_RECORD_PATH,
+ help=f"Path to ingest progress record (default: {DEFAULT_RECORD_PATH})",
+ )
+ args = parser.parse_args()
+
+ api_key = args.api_key or os.environ.get("SUPERMEMORY_API_KEY", "")
+ if not api_key:
+ print("Error: Supermemory API key required (--api-key or SUPERMEMORY_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ sample_ids: list[str] = list(args.samples)
+
+ if args.from_data:
+ with open(args.input, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ if args.limit:
+ data = data[: args.limit]
+ sample_ids += [s["sample_id"] for s in data]
+
+ if not sample_ids:
+ print("Error: no sample_ids specified. Pass sample_ids or use --from-data", file=sys.stderr)
+ sys.exit(1)
+
+ sample_ids = list(dict.fromkeys(sample_ids)) # deduplicate, preserve order
+ print(f"Deleting documents for {len(sample_ids)} sample(s)...", file=sys.stderr)
+
+ client = Supermemory(api_key=api_key)
+ ok = 0
+ for sid in sample_ids:
+ print(f"\n=== {sid} ===", file=sys.stderr)
+ if delete_container(client, sid, args.record):
+ ok += 1
+
+ print(f"\nDone: {ok}/{len(sample_ids)} succeeded", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/supermemory/eval.py b/benchmark/locomo/supermemory/eval.py
new file mode 100644
index 000000000..b9fbbe257
--- /dev/null
+++ b/benchmark/locomo/supermemory/eval.py
@@ -0,0 +1,797 @@
+"""
+Evaluate LoCoMo QA via Supermemory + OpenClaw (agent mode).
+
+Questions are sent to an OpenClaw agent which calls Supermemory internally.
+Before each sample, eval.py automatically:
+ 1. Updates ~/.openclaw/openclaw.json to set openclaw-supermemory.config.containerTag = sample_id
+ (and switches plugins.slots.memory to "openclaw-supermemory")
+ 2. Restarts the OpenClaw gateway to pick up the new config
+ 3. Runs all questions for that sample concurrently
+
+Prerequisites:
+ - Conversations already ingested into Supermemory via ingest.py (containerTag = sanitize(sample_id))
+ - OpenClaw running locally with the openclaw-supermemory plugin installed
+ - ~/.openclaw/openclaw.json configured with openclaw-supermemory apiKey
+
+Usage:
+ # Run QA + judge for all samples (6 concurrent threads)
+ python eval.py --threads 6 --judge
+
+ # Single sample
+ python eval.py --sample conv-26 --threads 6 --judge
+
+ # First 12 questions only
+ python eval.py --sample conv-26 --count 12 --threads 6 --judge
+
+ # Judge-only (grade existing responses in CSV)
+ python eval.py --judge-only
+"""
+
+import argparse
+import csv
+import json
+import os
+import re
+import subprocess
+import sys
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Optional
+
+import requests
+from dotenv import load_dotenv
+
+load_dotenv(Path.home() / ".openviking_benchmark_env")
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "data" / "locomo10.json")
+DEFAULT_OUTPUT_PATH = str(SCRIPT_DIR / "result" / "qa_results.csv")
+DEFAULT_OPENCLAW_URL = "http://127.0.0.1:18789"
+DEFAULT_SESSION_KEY = "locomo-eval"
+OPENCLAW_CONFIG_PATH = Path.home() / ".openclaw" / "openclaw.json"
+
+# Serialize openclaw config updates across threads so each sample sees the right containerTag
+_openclaw_config_lock = threading.Lock()
+
+# ---------------------------------------------------------------------------
+# Tag sanitization (must match openclaw-supermemory's sanitizeTag logic)
+# ---------------------------------------------------------------------------
+
+def sanitize_tag(raw: str) -> str:
+ """Sanitize a tag string to match openclaw-supermemory convention.
+ e.g. 'conv-26' -> 'conv_26'
+ """
+ tag = re.sub(r"[^a-zA-Z0-9_]", "_", raw)
+ tag = re.sub(r"_+", "_", tag)
+ tag = tag.strip("_")
+ return tag
+
+
+# ---------------------------------------------------------------------------
+# openclaw.json config helpers
+# ---------------------------------------------------------------------------
+
+def _update_openclaw_supermemory_container(sample_id: str) -> None:
+ """
+ Rewrite ~/.openclaw/openclaw.json so that openclaw-supermemory uses
+ sanitize(sample_id) as containerTag. Also switches slots.memory to
+ 'openclaw-supermemory' and disables autoCapture.
+ Must be called while holding _openclaw_config_lock.
+ """
+ container_tag = sanitize_tag(sample_id)
+
+ with open(OPENCLAW_CONFIG_PATH, "r", encoding="utf-8") as f:
+ config = json.load(f)
+
+ plugins = config.setdefault("plugins", {})
+ entries = plugins.setdefault("entries", {})
+
+ sm_entry = entries.setdefault("openclaw-supermemory", {})
+ sm_entry["enabled"] = True
+ sm_entry.setdefault("config", {})["containerTag"] = container_tag
+ sm_entry["config"]["autoCapture"] = False
+ sm_entry["config"]["autoRecall"] = True
+
+ # Switch memory slot to supermemory
+ plugins.setdefault("slots", {})["memory"] = "openclaw-supermemory"
+
+ tmp = str(OPENCLAW_CONFIG_PATH) + ".tmp"
+ with open(tmp, "w", encoding="utf-8") as f:
+ json.dump(config, f, indent=2, ensure_ascii=False)
+ os.replace(tmp, str(OPENCLAW_CONFIG_PATH))
+
+
+def _restart_openclaw_gateway(base_url: str, startup_timeout: int = 30) -> None:
+ """
+ Kill the running openclaw gateway process and restart it.
+ Waits until the gateway is ready to accept requests.
+ Must be called while holding _openclaw_config_lock.
+ """
+ try:
+ subprocess.run(["pkill", "-f", "openclaw gateway"], capture_output=True)
+ except Exception as e:
+ print(f" [gateway] pkill failed: {e}", file=sys.stderr)
+
+ try:
+ subprocess.Popen(
+ ["openclaw", "gateway"],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ )
+ except Exception as e:
+ raise RuntimeError(f"Failed to start openclaw gateway: {e}")
+
+ # Wait for process to fully start
+ time.sleep(5)
+
+ # Wait until gateway is ready
+ health_url = f"{base_url.rstrip('/')}/health"
+ deadline = time.time() + startup_timeout
+ while time.time() < deadline:
+ try:
+ resp = requests.get(health_url, timeout=2)
+ if resp.status_code < 500:
+ return
+ except Exception:
+ pass
+ time.sleep(0.5)
+
+ raise RuntimeError(f"openclaw gateway did not become ready within {startup_timeout}s")
+
+
+CATEGORY_NAMES = {
+ 1: "single-hop",
+ 2: "multi-hop",
+ 3: "temporal",
+ 4: "world-knowledge",
+ 5: "adversarial",
+}
+
+# ---------------------------------------------------------------------------
+# LoCoMo data loading
+# ---------------------------------------------------------------------------
+
+def load_locomo_data(path: str, sample_id: Optional[str] = None) -> list[dict]:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ if sample_id is not None:
+ try:
+ idx = int(sample_id)
+ if idx < 0 or idx >= len(data):
+ raise ValueError(f"Sample index {idx} out of range (0-{len(data) - 1})")
+ return [data[idx]]
+ except ValueError:
+ pass
+ matched = [s for s in data if s.get("sample_id") == sample_id]
+ if not matched:
+ raise ValueError(f"sample_id '{sample_id}' not found")
+ return matched
+
+ return data
+
+
+def get_sample_last_session_date(sample: dict) -> Optional[str]:
+ """Return the date of the last session as YYYY-MM-DD, or None."""
+ conv = sample.get("conversation", {})
+ session_keys = [k for k in conv if k.startswith("session_") and "date_time" not in k]
+ if not session_keys:
+ return None
+
+ def sess_num(k: str) -> int:
+ try:
+ return int(k.split("_")[1])
+ except ValueError:
+ return 0
+
+ for sk in sorted(session_keys, key=sess_num, reverse=True):
+ if conv.get(sk):
+ dt_key = f"{sk}_date_time"
+ date_str = conv.get(dt_key, "")
+ if date_str and " on " in date_str:
+ try:
+ from datetime import datetime
+ date_part = date_str.split(" on ")[-1]
+ dt = datetime.strptime(date_part.strip(), "%d %B, %Y")
+ return dt.strftime("%Y-%m-%d")
+ except ValueError:
+ pass
+ return None
+
+
+def load_qa_items(
+ data: list[dict],
+ skip_adversarial: bool = True,
+ question_index: Optional[int] = None,
+ count: Optional[int] = None,
+) -> list[dict]:
+ items = []
+ for sample in data:
+ sample_id = sample.get("sample_id", "")
+ question_time = get_sample_last_session_date(sample)
+
+ for q_idx, qa in enumerate(sample.get("qa", [])):
+ if question_index is not None and q_idx != question_index:
+ continue
+ category = qa.get("category", 0)
+ if skip_adversarial and str(category) == "5":
+ continue
+ items.append(
+ {
+ "sample_id": sample_id,
+ "question_index": q_idx,
+ "question_id": f"{sample_id}_qa{q_idx}",
+ "question": qa["question"],
+ "answer": str(qa["answer"]),
+ "category": category,
+ "category_name": CATEGORY_NAMES.get(category, "unknown"),
+ "evidence": qa.get("evidence", []),
+ "question_time": question_time,
+ }
+ )
+
+ if count is not None:
+ items = items[:count]
+ return items
+
+
+# ---------------------------------------------------------------------------
+# CSV helpers
+# ---------------------------------------------------------------------------
+
+QA_FIELDNAMES = [
+ "sample_id",
+ "question_index",
+ "question_id",
+ "question",
+ "answer",
+ "category",
+ "category_name",
+ "question_time",
+ "evidence",
+ "response",
+ "input_tokens",
+ "output_tokens",
+ "total_tokens",
+ "time_cost",
+ "result",
+ "reasoning",
+ "timestamp",
+]
+
+
+def load_processed_ids(output_path: str) -> set[str]:
+ processed: set[str] = set()
+ if not os.path.exists(output_path):
+ return processed
+ try:
+ with open(output_path, "r", encoding="utf-8", newline="") as f:
+ for row in csv.DictReader(f):
+ if row.get("response"):
+ processed.add(row.get("question_id", ""))
+ except Exception as e:
+ print(f"[WARN] Error reading {output_path}: {e}", file=sys.stderr)
+ return processed
+
+
+def save_row(output_path: str, row: dict, write_lock: threading.Lock) -> None:
+ with write_lock:
+ file_exists = os.path.exists(output_path)
+ with open(output_path, "a", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=QA_FIELDNAMES, extrasaction="ignore")
+ if not file_exists:
+ writer.writeheader()
+ writer.writerow(row)
+ f.flush()
+
+
+# ---------------------------------------------------------------------------
+# OpenClaw agent
+# ---------------------------------------------------------------------------
+
+def extract_openclaw_text(body: dict) -> str:
+ """Extract assistant text from /v1/responses API response."""
+ try:
+ for item in body.get("output", []):
+ if item.get("type") == "message":
+ for content in item.get("content", []):
+ if content.get("type") == "output_text":
+ return content.get("text", "")
+ for item in body.get("output", []):
+ if "text" in item:
+ return item["text"]
+ for content in item.get("content", []):
+ if "text" in content:
+ return content["text"]
+ except Exception:
+ pass
+ return f"[ERROR: could not parse response: {body}]"
+
+
+def get_openclaw_session_id(session_key: str) -> Optional[str]:
+ sessions_file = os.path.expanduser("~/.openclaw/agents/main/sessions/sessions.json")
+ try:
+ with open(sessions_file, "r") as f:
+ data = json.load(f)
+ return data.get(session_key, {}).get("sessionId")
+ except Exception:
+ return None
+
+
+def parse_session_tokens(session_id: str, agent_id: str = "main") -> dict:
+ """Sum up all LLM usage across all assistant messages in the session jsonl."""
+ sessions_dir = os.path.expanduser(f"~/.openclaw/agents/{agent_id}/sessions")
+ src = os.path.join(sessions_dir, f"{session_id}.jsonl")
+ total_input = total_output = total_cache_read = 0
+ try:
+ with open(src, "r", encoding="utf-8") as f:
+ for line in f:
+ line = line.strip()
+ if not line:
+ continue
+ obj = json.loads(line)
+ if obj.get("type") == "message" and obj.get("message", {}).get("role") == "assistant":
+ usage = obj["message"].get("usage", {})
+ total_input += usage.get("input", 0)
+ total_output += usage.get("output", 0)
+ total_cache_read += usage.get("cacheRead", 0)
+ except Exception:
+ pass
+ return {
+ "input_tokens": total_input,
+ "output_tokens": total_output,
+ "total_tokens": total_input + total_output + total_cache_read,
+ }
+
+
+def send_to_openclaw(
+ question: str,
+ sample_id: str,
+ base_url: str,
+ token: str,
+ question_id: Optional[str] = None,
+ question_time: Optional[str] = None,
+ retries: int = 2,
+) -> tuple[str, dict, float]:
+ """
+ Send a question to an OpenClaw agent.
+ Returns (response_text, usage, time_cost).
+ """
+ session_key = f"{DEFAULT_SESSION_KEY}-{question_id}" if question_id else DEFAULT_SESSION_KEY
+
+ url = f"{base_url.rstrip('/')}/v1/responses"
+ headers = {
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {token}",
+ "X-OpenClaw-Session-Key": session_key,
+ }
+ payload = {
+ "model": "openclaw",
+ # Force explicit search: the openclaw-supermemory plugin skips semantic search
+ # on the first turn of a session, so autoRecall alone is not enough here.
+ "input": f"Use supermemory_search to find relevant memories, then Answer the question directly: {question}",
+ "stream": False,
+ "user": sample_id,
+ }
+
+ last_exc: Optional[Exception] = None
+ t0 = time.time()
+ for attempt in range(retries + 1):
+ try:
+ resp = requests.post(url, json=payload, headers=headers, timeout=300)
+ resp.raise_for_status()
+ body = resp.json()
+ response_text = extract_openclaw_text(body)
+
+ # Wait for openclaw to flush the session jsonl before parsing tokens
+ time.sleep(1)
+ session_id = get_openclaw_session_id(session_key)
+ if session_id:
+ usage = parse_session_tokens(session_id)
+ else:
+ usage = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
+
+ return response_text, usage, time.time() - t0
+ except Exception as e:
+ last_exc = e
+ if attempt < retries:
+ print(f" [retry {attempt + 1}/{retries}] {e}", file=sys.stderr)
+
+ raise RuntimeError(f"OpenClaw request failed after {retries + 1} attempts: {last_exc}")
+
+
+# ---------------------------------------------------------------------------
+# LLM judge
+# ---------------------------------------------------------------------------
+
+JUDGE_SYSTEM_PROMPT = "You are an expert grader that determines if answers to questions match a gold standard answer"
+
+JUDGE_ACCURACY_PROMPT = """Your task is to label an answer to a question as 'CORRECT' or 'WRONG'. You will be given the following data:
+ (1) a question (posed by one user to another user),
+ (2) a 'gold' (ground truth) answer,
+ (3) a generated answer
+which you will score as CORRECT/WRONG.
+
+The point of the question is to ask about something one user should know about the other user based on their prior conversations.
+The gold answer will usually be a concise and short answer that includes the referenced topic, for example:
+Question: Do you remember what I got the last time I went to Hawaii?
+Gold answer: A shell necklace
+The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.
+
+For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date.
+
+Now it's time for the real question:
+Question: {question}
+Gold answer: {gold_answer}
+Generated answer: {response}
+
+First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG.
+Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.
+
+Respond with JSON only: {{"reasoning": "your explanation", "is_correct": "CORRECT" or "WRONG"}}"""
+
+
+def judge_answer(
+ question: str,
+ gold_answer: str,
+ response: str,
+ judge_base_url: str,
+ judge_token: str,
+ judge_model: str,
+) -> tuple[str, str]:
+ from openai import OpenAI
+ client = OpenAI(base_url=judge_base_url, api_key=judge_token)
+ prompt = JUDGE_ACCURACY_PROMPT.format(
+ question=question, gold_answer=gold_answer, response=response
+ )
+ try:
+ resp = client.chat.completions.create(
+ model=judge_model,
+ messages=[
+ {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
+ {"role": "user", "content": prompt},
+ ],
+ temperature=0,
+ timeout=60,
+ )
+ content = resp.choices[0].message.content.strip()
+ start, end = content.find("{"), content.rfind("}")
+ if start != -1 and end != -1:
+ parsed = json.loads(content[start : end + 1])
+ label = "CORRECT" if parsed.get("is_correct", "WRONG").strip().upper() == "CORRECT" else "WRONG"
+ return label, parsed.get("reasoning", "")
+ return "WRONG", f"[PARSE ERROR] {content}"
+ except Exception as e:
+ return "WRONG", f"[API ERROR] {e}"
+
+
+# ---------------------------------------------------------------------------
+# Accuracy summary
+# ---------------------------------------------------------------------------
+
+def print_accuracy(rows: list[dict]) -> None:
+ graded = [r for r in rows if r.get("result") in ("CORRECT", "WRONG")]
+ if not graded:
+ print("\n[INFO] No graded results to summarize.", file=sys.stderr)
+ return
+
+ correct_total = sum(1 for r in graded if r["result"] == "CORRECT")
+ print("\n=== Accuracy Summary ===", file=sys.stderr)
+ print(f" Overall: {correct_total}/{len(graded)} = {correct_total/len(graded):.2%}", file=sys.stderr)
+
+ by_cat: dict[str, list[str]] = {}
+ for r in graded:
+ cat = r.get("category_name") or str(r.get("category", "?"))
+ by_cat.setdefault(cat, []).append(r["result"])
+
+ print(" By category:", file=sys.stderr)
+ for cat, results in sorted(by_cat.items()):
+ n = sum(1 for r in results if r == "CORRECT")
+ print(f" {cat:20s}: {n}/{len(results)} = {n/len(results):.2%}", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------------
+# Main runners
+# ---------------------------------------------------------------------------
+
+def run_qa(args: argparse.Namespace) -> None:
+ openclaw_token = args.openclaw_token or os.environ.get("OPENCLAW_GATEWAY_TOKEN", "")
+
+ judge_token = args.judge_token or os.environ.get("ARK_API_KEY", os.environ.get("OPENAI_API_KEY", ""))
+ if args.judge and not judge_token:
+ print("Error: judge token required (--judge-token or ARK_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ data = load_locomo_data(args.input, args.sample)
+ qa_items = load_qa_items(
+ data,
+ skip_adversarial=args.skip_adversarial,
+ question_index=args.question_index,
+ count=args.count,
+ )
+ print(f"[INFO] {len(qa_items)} QA items loaded", file=sys.stderr)
+
+ Path(args.output).parent.mkdir(parents=True, exist_ok=True)
+
+ # Remove ERROR rows from CSV before loading processed ids
+ if os.path.exists(args.output):
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ fieldnames = reader.fieldnames or QA_FIELDNAMES
+ clean_rows = [r for r in reader if not r.get("response", "").startswith("[ERROR]")]
+ with open(args.output, "w", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
+ writer.writeheader()
+ writer.writerows(clean_rows)
+
+ processed_ids = load_processed_ids(args.output)
+ remaining = [qa for qa in qa_items if qa["question_id"] not in processed_ids]
+ print(
+ f"[INFO] {len(processed_ids)} already done, {len(remaining)} remaining",
+ file=sys.stderr,
+ )
+
+ if not remaining:
+ print("[INFO] All questions already processed.", file=sys.stderr)
+ else:
+ write_lock = threading.Lock()
+ total = len(remaining)
+
+ # Group remaining questions by sample_id to minimize gateway restarts
+ from collections import defaultdict
+ by_sample: dict[str, list[tuple[int, dict]]] = defaultdict(list)
+ for i, qa in enumerate(remaining):
+ by_sample[qa["sample_id"]].append((i + 1, qa))
+
+ def run_one(qa: dict, idx: int) -> None:
+ print(
+ f" [{idx}/{total}] {qa['question_id']}: {qa['question'][:60]}...",
+ file=sys.stderr,
+ )
+ try:
+ response, usage, time_cost = send_to_openclaw(
+ qa["question"],
+ qa["sample_id"],
+ args.openclaw_url,
+ openclaw_token,
+ qa["question_id"],
+ qa.get("question_time"),
+ )
+ except Exception as e:
+ response = f"[ERROR] {e}"
+ usage = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
+ time_cost = 0.0
+
+ result_label, reasoning = "", ""
+ if args.judge and response and not response.startswith("[ERROR]"):
+ result_label, reasoning = judge_answer(
+ qa["question"],
+ qa["answer"],
+ response,
+ args.judge_base_url,
+ judge_token,
+ args.judge_model,
+ )
+
+ row = {
+ "sample_id": qa["sample_id"],
+ "question_index": qa["question_index"],
+ "question_id": qa["question_id"],
+ "question": qa["question"],
+ "answer": qa["answer"],
+ "category": qa["category"],
+ "category_name": qa["category_name"],
+ "question_time": qa.get("question_time", ""),
+ "evidence": json.dumps(qa.get("evidence", [])),
+ "response": response,
+ "input_tokens": usage.get("input_tokens", 0),
+ "output_tokens": usage.get("output_tokens", 0),
+ "total_tokens": usage.get("total_tokens", 0),
+ "time_cost": round(time_cost, 2),
+ "result": result_label,
+ "reasoning": reasoning,
+ "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
+ }
+ save_row(args.output, row, write_lock)
+ label_str = f" → {result_label}" if result_label else ""
+ print(f" [{idx}/{total}] done {time_cost:.1f}s{label_str}", file=sys.stderr)
+
+ # Process sample by sample: restart gateway once per sample to pick up new containerTag
+ for sample_id, qa_list in by_sample.items():
+ container_tag = sanitize_tag(sample_id)
+ print(
+ f"\n[INFO] Switching to sample {sample_id} (containerTag={container_tag}), restarting openclaw gateway...",
+ file=sys.stderr,
+ )
+ with _openclaw_config_lock:
+ _update_openclaw_supermemory_container(sample_id)
+ _restart_openclaw_gateway(args.openclaw_url)
+ print(f"[INFO] Gateway ready, running {len(qa_list)} questions for {sample_id}", file=sys.stderr)
+
+ with ThreadPoolExecutor(max_workers=args.threads) as executor:
+ futures = {
+ executor.submit(run_one, qa, idx): qa
+ for idx, qa in qa_list
+ }
+ for fut in as_completed(futures):
+ try:
+ fut.result()
+ except Exception as e:
+ qa = futures[fut]
+ print(f" [ERROR] {qa['question_id']}: {e}", file=sys.stderr)
+
+ # Print token and latency summary
+ try:
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ rows = list(csv.DictReader(f))
+ total_input = sum(int(r.get("input_tokens") or 0) for r in rows)
+ total_input_with_cache = sum(
+ int(r.get("total_tokens") or 0) - int(r.get("output_tokens") or 0) for r in rows
+ )
+ times = [float(r["time_cost"]) for r in rows if r.get("time_cost")]
+ avg_time = sum(times) / len(times) if times else 0.0
+ print(f"\n=== Token & Latency Summary ===", file=sys.stderr)
+ print(f" Total input tokens : {total_input}", file=sys.stderr)
+ print(f" Total input tokens (with cache): {total_input_with_cache}", file=sys.stderr)
+ print(f" Avg time per query : {avg_time:.1f}s", file=sys.stderr)
+ except Exception:
+ pass
+
+ if args.judge:
+ try:
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ print_accuracy(list(csv.DictReader(f)))
+ except Exception:
+ pass
+
+
+def run_judge_only(args: argparse.Namespace) -> None:
+ """Grade responses in an existing CSV that lack a result label."""
+ if not os.path.exists(args.output):
+ print(f"Error: output file not found: {args.output}", file=sys.stderr)
+ sys.exit(1)
+
+ judge_token = args.judge_token or os.environ.get("ARK_API_KEY", os.environ.get("OPENAI_API_KEY", ""))
+ if not judge_token:
+ print("Error: judge token required (--judge-token or ARK_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ with open(args.output, "r", encoding="utf-8", newline="") as f:
+ reader = csv.DictReader(f)
+ fieldnames = list(reader.fieldnames or QA_FIELDNAMES)
+ rows = list(reader)
+
+ for extra in ("result", "reasoning"):
+ if extra not in fieldnames:
+ fieldnames.append(extra)
+
+ ungraded_indices = [i for i, row in enumerate(rows) if not row.get("result")]
+ print(f"[INFO] {len(rows)} rows total, {len(ungraded_indices)} ungraded", file=sys.stderr)
+
+ if not ungraded_indices:
+ print("[INFO] All rows already graded.", file=sys.stderr)
+ print_accuracy(rows)
+ return
+
+ file_lock = threading.Lock()
+
+ def grade_one(idx: int) -> None:
+ row = rows[idx]
+ label, reasoning = judge_answer(
+ row.get("question", ""),
+ row.get("answer", ""),
+ row.get("response", ""),
+ args.judge_base_url,
+ judge_token,
+ args.judge_model,
+ )
+ row["result"] = label
+ row["reasoning"] = reasoning
+ with file_lock:
+ tmp = args.output + ".tmp"
+ with open(tmp, "w", encoding="utf-8", newline="") as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
+ writer.writeheader()
+ writer.writerows(rows)
+ os.replace(tmp, args.output)
+ print(f" Graded {row.get('question_id','?')}: {label}", file=sys.stderr)
+
+ with ThreadPoolExecutor(max_workers=args.threads) as executor:
+ futures = [executor.submit(grade_one, idx) for idx in ungraded_indices]
+ for fut in as_completed(futures):
+ try:
+ fut.result()
+ except Exception as e:
+ print(f"[ERROR] grading failed: {e}", file=sys.stderr)
+
+ print_accuracy(rows)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ description="Evaluate LoCoMo QA via OpenClaw agent (Supermemory-backed)",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=__doc__,
+ )
+
+ # Data selection
+ parser.add_argument("--input", default=DEFAULT_DATA_PATH, help="Path to locomo10.json")
+ parser.add_argument("--output", default=DEFAULT_OUTPUT_PATH, help="Path to output CSV")
+ parser.add_argument(
+ "--sample",
+ default=None,
+ help="Sample index (int) or sample_id (e.g. conv-26). Default: all.",
+ )
+ parser.add_argument(
+ "--question-index",
+ type=int,
+ default=None,
+ help="Single question index (0-based) within the sample.",
+ )
+ parser.add_argument("--count", type=int, default=None, help="Max QA items to process.")
+ parser.add_argument(
+ "--no-skip-adversarial",
+ dest="skip_adversarial",
+ action="store_false",
+ default=True,
+ help="Include category-5 adversarial questions (skipped by default).",
+ )
+ parser.add_argument("--threads", type=int, default=10, help="Concurrent threads per sample (default: 10)")
+
+ # OpenClaw
+ parser.add_argument(
+ "--openclaw-url",
+ default=DEFAULT_OPENCLAW_URL,
+ help=f"OpenClaw gateway URL (default: {DEFAULT_OPENCLAW_URL})",
+ )
+ parser.add_argument(
+ "--openclaw-token",
+ default=None,
+ help="OpenClaw auth token (or OPENCLAW_GATEWAY_TOKEN env var)",
+ )
+
+ # Judge
+ parser.add_argument(
+ "--judge",
+ action="store_true",
+ default=False,
+ help="Auto-judge each response right after answering.",
+ )
+ parser.add_argument(
+ "--judge-only",
+ action="store_true",
+ default=False,
+ help="Skip QA; only grade ungraded responses in the existing --output CSV.",
+ )
+ parser.add_argument(
+ "--judge-base-url",
+ default="https://ark.cn-beijing.volces.com/api/v3",
+ help="OpenAI-compatible API base URL for judge (default: Volcengine ARK)",
+ )
+ parser.add_argument(
+ "--judge-token",
+ default=None,
+ help="API token for judge (or ARK_API_KEY / OPENAI_API_KEY env var)",
+ )
+ parser.add_argument(
+ "--judge-model",
+ default="doubao-seed-2-0-pro-260215",
+ help="Judge model (default: doubao-seed-2-0-pro-260215)",
+ )
+
+ args = parser.parse_args()
+
+ if args.judge_only:
+ run_judge_only(args)
+ else:
+ run_qa(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/locomo/supermemory/ingest.py b/benchmark/locomo/supermemory/ingest.py
new file mode 100644
index 000000000..a2656f464
--- /dev/null
+++ b/benchmark/locomo/supermemory/ingest.py
@@ -0,0 +1,534 @@
+"""
+Ingest LoCoMo conversations into Supermemory.
+
+Each sample gets an isolated Supermemory namespace keyed by containerTag = sample_id
+(e.g. "conv-26"). Sessions are formatted as date-prefixed JSON content strings,
+matching the memorybench supermemory provider convention.
+
+Usage:
+ # Ingest all samples
+ python ingest.py
+
+ # Ingest a specific sample
+ python ingest.py --sample conv-26
+
+ # Ingest specific sessions
+ python ingest.py --sample conv-26 --sessions 1-4
+
+ # Force re-ingest even if already done
+ python ingest.py --force-ingest
+
+ # Set Supermemory API key via env or flag
+ SUPERMEMORY_API_KEY=xxx python ingest.py
+ python ingest.py --api-key xxx
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+import time
+import threading
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any, Optional
+
+from dotenv import load_dotenv
+
+load_dotenv(Path.home() / ".openviking_benchmark_env")
+
+try:
+ from supermemory import Supermemory
+except ImportError:
+ print("Error: supermemory package not installed. Run: pip install supermemory", file=sys.stderr)
+ sys.exit(1)
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+
+SCRIPT_DIR = Path(__file__).parent.resolve()
+DEFAULT_DATA_PATH = str(SCRIPT_DIR / ".." / "data" / "locomo10.json")
+DEFAULT_RECORD_PATH = str(SCRIPT_DIR / "result" / ".ingest_record.json")
+DEFAULT_LOG_PATH = str(SCRIPT_DIR / "result" / "ingest_errors.log")
+
+
+# ---------------------------------------------------------------------------
+# Tag sanitization (must match openclaw-supermemory's sanitizeTag logic)
+# ---------------------------------------------------------------------------
+
+def sanitize_tag(raw: str) -> str:
+ """Sanitize a tag string to match openclaw-supermemory convention.
+ Replaces non-alphanumeric/underscore chars with '_', collapses runs, strips edges.
+ e.g. 'conv-26' -> 'conv_26'
+ """
+ tag = re.sub(r"[^a-zA-Z0-9_]", "_", raw)
+ tag = re.sub(r"_+", "_", tag)
+ tag = tag.strip("_")
+ return tag
+
+
+# ---------------------------------------------------------------------------
+# LoCoMo data loading
+# ---------------------------------------------------------------------------
+
+def load_locomo_data(path: str, sample_id: Optional[str] = None) -> list[dict]:
+ """Load LoCoMo JSON and optionally filter to one sample by sample_id or numeric index."""
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ if sample_id is not None:
+ try:
+ idx = int(sample_id)
+ if idx < 0 or idx >= len(data):
+ raise ValueError(f"Sample index {idx} out of range (0-{len(data) - 1})")
+ return [data[idx]]
+ except ValueError:
+ pass
+ matched = [s for s in data if s.get("sample_id") == sample_id]
+ if not matched:
+ raise ValueError(f"sample_id '{sample_id}' not found")
+ return matched
+
+ return data
+
+
+def parse_session_range(s: str) -> tuple[int, int]:
+ """Parse '1-4' or '3' into (lo, hi) inclusive tuple."""
+ if "-" in s:
+ lo, hi = s.split("-", 1)
+ return int(lo), int(hi)
+ n = int(s)
+ return n, n
+
+
+def parse_locomo_date(date_time: str) -> Optional[str]:
+ """Parse a LoCoMo date_time string to ISO 8601 format.
+ e.g. '1:56 pm on 8 May, 2023' -> '2023-05-08T13:56:00.000Z'
+ """
+ import re
+ from datetime import datetime, timezone
+ match = re.search(r"(\d+):(\d+)\s*(am|pm)\s*on\s*(\d+)\s*(\w+),?\s*(\d+)", date_time, re.IGNORECASE)
+ if not match:
+ return None
+ hour_str, minute, ampm, day, month_name, year = match.groups()
+ hour = int(hour_str)
+ if ampm.lower() == "pm" and hour != 12:
+ hour += 12
+ if ampm.lower() == "am" and hour == 12:
+ hour = 0
+ month_names = ["january","february","march","april","may","june",
+ "july","august","september","october","november","december"]
+ try:
+ month = next(i + 1 for i, n in enumerate(month_names) if n.startswith(month_name.lower()))
+ dt = datetime(int(year), month, int(day), hour, int(minute), tzinfo=timezone.utc)
+ return dt.strftime("%Y-%m-%dT%H:%M:%S.000Z")
+ except (StopIteration, ValueError):
+ return None
+
+
+def build_session_content(
+ item: dict,
+ session_key: str,
+ date_time: str,
+) -> str:
+ conv = item["conversation"]
+ raw_messages = conv[session_key]
+ lines = [f"[{msg.get('speaker', '').capitalize()}]: {msg.get('text', '')}" for msg in raw_messages]
+ session_str = "\n".join(lines)
+
+ if date_time:
+ return (
+ f"Here is the date the following session took place: {date_time}\n\n"
+ f"Here is the session as a stringified JSON:\n{session_str}"
+ )
+ else:
+ return f"Here is the session as a stringified JSON:\n{session_str}"
+
+
+def build_sessions(
+ item: dict,
+ session_range: Optional[tuple[int, int]] = None,
+) -> list[dict]:
+ """
+ Extract sessions from a LoCoMo sample.
+
+ Returns list of dicts with keys:
+ - content: formatted string for supermemory
+ - meta: session metadata
+ """
+ conv = item["conversation"]
+
+ session_keys = sorted(
+ [k for k in conv if k.startswith("session_") and not k.endswith("_date_time")],
+ key=lambda k: int(k.split("_")[1]),
+ )
+
+ sessions = []
+ for sk in session_keys:
+ sess_num = int(sk.split("_")[1])
+ if session_range:
+ lo, hi = session_range
+ if sess_num < lo or sess_num > hi:
+ continue
+
+ raw_messages = conv[sk]
+ if not isinstance(raw_messages, list) or not raw_messages:
+ continue
+
+ dt_key = f"{sk}_date_time"
+ date_time = conv.get(dt_key, "")
+ date_iso = parse_locomo_date(date_time) if date_time else None
+
+ content = build_session_content(item, sk, date_time)
+
+ sessions.append(
+ {
+ "content": content,
+ "meta": {
+ "sample_id": item["sample_id"],
+ "session_key": sk,
+ "date_time": date_time,
+ "date_iso": date_iso or "",
+ "speaker_a": conv["speaker_a"],
+ "speaker_b": conv["speaker_b"],
+ },
+ }
+ )
+
+ return sessions
+
+
+# ---------------------------------------------------------------------------
+# Ingest record (progress tracking)
+# ---------------------------------------------------------------------------
+
+def load_ingest_record(path: str) -> dict:
+ try:
+ with open(path, "r", encoding="utf-8") as f:
+ return json.load(f)
+ except (FileNotFoundError, json.JSONDecodeError):
+ return {}
+
+
+def save_ingest_record(record: dict, path: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(record, f, indent=2, ensure_ascii=False)
+
+
+def is_already_ingested(sample_id: str, session_key: str, record: dict) -> bool:
+ key = f"supermemory:{sample_id}:{session_key}"
+ return key in record and record[key].get("success", False)
+
+
+def mark_ingested(
+ sample_id: str,
+ session_key: str,
+ record: dict,
+ doc_id: str,
+ meta: Optional[dict] = None,
+) -> None:
+ key = f"supermemory:{sample_id}:{session_key}"
+ record[key] = {
+ "success": True,
+ "timestamp": int(time.time()),
+ "doc_id": doc_id,
+ "meta": meta or {},
+ }
+
+
+def write_error_log(path: str, sample_id: str, session_key: str, error: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ ts = time.strftime("%Y-%m-%d %H:%M:%S")
+ with open(path, "a", encoding="utf-8") as f:
+ f.write(f"[{ts}] ERROR [{sample_id}/{session_key}]: {error}\n")
+
+
+# ---------------------------------------------------------------------------
+# Supermemory indexing poll
+# ---------------------------------------------------------------------------
+
+def poll_document(client: Supermemory, doc_id: str, timeout_sec: int = 600) -> str:
+ """
+ Poll a single document until done/failed/TIMEOUT.
+ Returns final status string.
+ """
+ backoff = 1.0
+ start = time.time()
+
+ while True:
+ if time.time() - start > timeout_sec:
+ return "TIMEOUT"
+
+ try:
+ doc = client.documents.get(doc_id)
+ doc_status = getattr(doc, "status", None) or (doc.get("status") if isinstance(doc, dict) else None)
+
+ if doc_status == "failed":
+ return "failed"
+ if doc_status == "done":
+ return "done"
+
+ except Exception as e:
+ print(f" [poll] Error checking doc {doc_id}: {e}", file=sys.stderr)
+
+ time.sleep(backoff)
+ backoff = min(backoff * 1.2, 5.0)
+
+
+def poll_all_documents(
+ client: Supermemory,
+ pending: dict[str, dict], # doc_id -> {"session_key", "label", ...}
+ timeout_sec: int = 600,
+ threads: int = 8,
+) -> tuple[dict[str, str], dict[str, str]]:
+ """
+ Poll all pending doc_ids in parallel.
+ Returns (done_map, failed_map): doc_id -> label.
+ """
+ done_map: dict[str, str] = {}
+ failed_map: dict[str, str] = {}
+
+ with ThreadPoolExecutor(max_workers=threads) as executor:
+ futures = {
+ executor.submit(poll_document, client, doc_id, timeout_sec): doc_id
+ for doc_id in pending
+ }
+ for fut in as_completed(futures):
+ doc_id = futures[fut]
+ label = pending[doc_id]["label"]
+ try:
+ status = fut.result()
+ except Exception as e:
+ status = f"ERROR: {e}"
+ if status == "done":
+ done_map[doc_id] = label
+ print(f" [{label}] indexed OK doc_id={doc_id}", file=sys.stderr)
+ else:
+ failed_map[doc_id] = label
+ print(f" [{label}] indexing {status} doc_id={doc_id}", file=sys.stderr)
+
+ return done_map, failed_map
+
+
+# ---------------------------------------------------------------------------
+# Core ingest logic
+# ---------------------------------------------------------------------------
+
+def upload_session(
+ client: Supermemory,
+ content: str,
+ container_tag: str,
+ meta: dict,
+) -> str:
+ """Upload one session to Supermemory (no indexing wait). Returns doc_id."""
+ response = client.add(
+ content=content,
+ container_tag=container_tag,
+ metadata={
+ "session_key": meta.get("session_key", ""),
+ "date_time": meta.get("date_time", ""),
+ "date": meta.get("date_iso", ""),
+ "speaker_a": meta.get("speaker_a", ""),
+ "speaker_b": meta.get("speaker_b", ""),
+ },
+ )
+ doc_id = getattr(response, "id", None) or (response.get("id") if isinstance(response, dict) else None)
+ if not doc_id:
+ raise RuntimeError(f"Supermemory add() returned no id: {response}")
+ return doc_id
+
+
+def ingest_sample(
+ item: dict,
+ client: Supermemory,
+ ingest_record: dict,
+ record_lock: threading.Lock,
+ args: argparse.Namespace,
+ session_range: Optional[tuple[int, int]],
+) -> tuple[int, int, int, int]:
+ """Ingest one sample. Returns (total, success, skip, error) counts."""
+ sample_id: str = item["sample_id"]
+ container_tag = sanitize_tag(sample_id)
+ sessions = build_sessions(item, session_range)
+ print(f"\n=== Sample {sample_id} ({len(sessions)} sessions) [containerTag={container_tag}] ===", file=sys.stderr)
+
+ total = len(sessions)
+ success = skip = error = 0
+
+ # Phase 1: upload all sessions serially (within this sample)
+ to_poll: dict[str, dict] = {}
+
+ for sess in sessions:
+ meta = sess["meta"]
+ session_key = meta["session_key"]
+ label = f"{session_key} ({meta['date_time']})"
+
+ with record_lock:
+ already = not args.force_ingest and is_already_ingested(sample_id, session_key, ingest_record)
+ if already:
+ print(f" [{label}] SKIP (already ingested)", file=sys.stderr)
+ skip += 1
+ continue
+
+ try:
+ doc_id = upload_session(client, sess["content"], container_tag, meta)
+ print(f" [{label}] uploaded doc_id={doc_id}", file=sys.stderr)
+ to_poll[doc_id] = {"session_key": session_key, "label": label, "meta": meta}
+ except Exception as e:
+ print(f" [{label}] UPLOAD ERROR: {e}", file=sys.stderr)
+ write_error_log(args.error_log, sample_id, session_key, str(e))
+ error += 1
+
+ if not to_poll:
+ return total, success, skip, error
+
+ # Phase 2: poll all uploaded docs in parallel until all done
+ print(f" [INFO] Polling {len(to_poll)} docs for indexing completion...", file=sys.stderr)
+ t0 = time.time()
+ done_map, failed_map = poll_all_documents(client, to_poll, threads=args.threads)
+ elapsed = time.time() - t0
+ print(f" [INFO] Indexing done in {elapsed:.1f}s: {len(done_map)} OK, {len(failed_map)} failed", file=sys.stderr)
+
+ # Phase 3: save ingest records (thread-safe)
+ with record_lock:
+ for doc_id in done_map:
+ info = to_poll[doc_id]
+ mark_ingested(sample_id, info["session_key"], ingest_record, doc_id, info["meta"])
+ success += 1
+ save_ingest_record(ingest_record, args.record)
+
+ for doc_id in failed_map:
+ info = to_poll[doc_id]
+ write_error_log(args.error_log, sample_id, info["session_key"], f"indexing failed/timeout for doc {doc_id}")
+ error += 1
+
+ return total, success, skip, error
+
+
+def run_ingest(args: argparse.Namespace) -> None:
+ api_key = args.api_key or os.environ.get("SUPERMEMORY_API_KEY", "")
+ if not api_key:
+ print("Error: Supermemory API key required (--api-key or SUPERMEMORY_API_KEY env var)", file=sys.stderr)
+ sys.exit(1)
+
+ client = Supermemory(api_key=api_key)
+
+ session_range = parse_session_range(args.sessions) if args.sessions else None
+
+ if args.clear_ingest_record:
+ ingest_record: dict = {}
+ save_ingest_record(ingest_record, args.record)
+ print("[INFO] Cleared existing ingest records", file=sys.stderr)
+ else:
+ ingest_record = load_ingest_record(args.record)
+
+ samples = load_locomo_data(args.input, args.sample)
+ if args.limit:
+ samples = samples[: args.limit]
+ print(f"[INFO] Loaded {len(samples)} sample(s) (sample_concurrency={args.sample_concurrency})", file=sys.stderr)
+
+ total_sessions = success_count = skip_count = error_count = 0
+ record_lock = threading.Lock()
+
+ with ThreadPoolExecutor(max_workers=args.sample_concurrency) as executor:
+ futures = {
+ executor.submit(
+ ingest_sample, item, client, ingest_record, record_lock, args, session_range
+ ): item["sample_id"]
+ for item in samples
+ }
+ for fut in as_completed(futures):
+ sample_id = futures[fut]
+ try:
+ t, s, sk, e = fut.result()
+ total_sessions += t
+ success_count += s
+ skip_count += sk
+ error_count += e
+ except Exception as exc:
+ print(f"[ERROR] Sample {sample_id} failed: {exc}", file=sys.stderr)
+ error_count += 1
+
+ print(f"\n=== Ingest summary ===", file=sys.stderr)
+ print(f" Total sessions: {total_sessions}", file=sys.stderr)
+ print(f" Succeeded: {success_count}", file=sys.stderr)
+ print(f" Skipped: {skip_count}", file=sys.stderr)
+ print(f" Failed: {error_count}", file=sys.stderr)
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Ingest LoCoMo conversations into Supermemory")
+ parser.add_argument(
+ "--input",
+ default=DEFAULT_DATA_PATH,
+ help="Path to locomo10.json (default: ../data/locomo10.json)",
+ )
+ parser.add_argument(
+ "--api-key",
+ default=None,
+ help="Supermemory API key (or set SUPERMEMORY_API_KEY env var)",
+ )
+ parser.add_argument(
+ "--sample",
+ default=None,
+ help="Sample index (0-based int) or sample_id string (e.g. conv-26). Default: all.",
+ )
+ parser.add_argument(
+ "--limit",
+ type=int,
+ default=None,
+ help="Max number of samples to ingest. Default: all.",
+ )
+ parser.add_argument(
+ "--sessions",
+ default=None,
+ help="Session range, e.g. '1-4' or '3'. Default: all.",
+ )
+ parser.add_argument(
+ "--record",
+ default=DEFAULT_RECORD_PATH,
+ help=f"Path to ingest progress record (default: {DEFAULT_RECORD_PATH})",
+ )
+ parser.add_argument(
+ "--error-log",
+ default=DEFAULT_LOG_PATH,
+ help=f"Path to error log (default: {DEFAULT_LOG_PATH})",
+ )
+ parser.add_argument(
+ "--force-ingest",
+ action="store_true",
+ default=False,
+ help="Re-ingest even if already recorded as done",
+ )
+ parser.add_argument(
+ "--clear-ingest-record",
+ action="store_true",
+ default=False,
+ help="Clear all existing ingest records before running",
+ )
+ parser.add_argument(
+ "--threads",
+ type=int,
+ default=8,
+ help="Concurrent threads for indexing poll within a sample (default: 8)",
+ )
+ parser.add_argument(
+ "--sample-concurrency",
+ type=int,
+ default=3,
+ help="Number of samples to ingest concurrently (default: 3)",
+ )
+
+ args = parser.parse_args()
+ run_ingest(args)
+
+
+if __name__ == "__main__":
+ main()