Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(benchmark): add AML adapter and public evaluation workflows (#5276)

agent71d705cf9850
diff --git a/benchmark/aml/README.md b/benchmark/aml/README.md
new file mode 100644
index 000000000..c3667e79f
--- /dev/null
+++ b/benchmark/aml/README.md
@@ -0,0 +1,204 @@
+# Agent Memory Leaderboard adapter
+
+This directory contains three deliberately separated stages:
+
+1. `server.py` exposes the competition-facing AML Add/Search API;
+2. `eval/run_public.py` reads the already prepared datasets under
+ `result/aml_data` and produces retrieval results;
+3. `eval/evaluate.py` optionally runs each dataset's public Answer/Judge pipeline
+ after retrieval, including deterministic PersonaMem MCQ grading.
+
+The deployed adapter never imports the local evaluation code. In a formal AML
+run, the platform supplies the datasets and controls Answer, Judge, and score
+aggregation.
+
+## Add/Search adapter
+
+For a local OpenViking server using `server.auth_mode: dev`:
+
+```bash
+export OPENVIKING_URL=http://127.0.0.1:1933
+export AML_API_KEY='<memory-system key>'
+
+python -m benchmark.aml.server --host 127.0.0.1 --port 8088
+```
+
+Do not set an OpenViking key or account for this local `dev` setup. The SDK
+passes each AML `user_id` directly as the OpenViking user; there is no hash or
+second identity mapping. Like other OpenViking SDK clients, it also reads an
+existing `~/.openviking/ovcli.conf`, so that file must match the server being
+used (or be disabled with `OPENVIKING_CLI_CONFIG_FILE` pointing to a missing
+file).
+
+For a remote multi-user deployment, configure the OpenViking server with
+`server.auth_mode: trusted`, then set the trusted-mode root key and account:
+
+```bash
+export OPENVIKING_URL='https://openviking.example.com'
+export OPENVIKING_API_KEY='<trusted-mode root key>'
+export OPENVIKING_ACCOUNT=default
+export AML_API_KEY='<memory-system key>'
+
+python -m benchmark.aml.server --host 0.0.0.0 --port 8088
+```
+
+OpenViking's standard `api_key` mode cannot use one root key to switch among
+dynamic AML users: in that mode a request must use the key belonging to the
+target user. `AML_API_KEY` is unrelated; it only protects this adapter's
+incoming `/add` and `/search` endpoints.
+
+The adapter provides:
+
+- `GET /health`
+- `POST /add`
+- `POST /search`
+
+`/add` calls the OpenViking SDK's `batch_add_messages`, `commit_session`, and
+`get_task` in that order. `/search` calls OpenViking `find`, restricted to
+`viking://~/memories` and level 2 under the same AML user. Each dataset case is
+isolated by its `user_id`; session IDs only separate histories inside that
+user's space. Transient SDK calls are retried up to three times by
+default; use `AML_RETRY_ATTEMPTS` and `AML_RETRY_DELAY_SECONDS` to adjust that
+behavior. `Token`, `Bearer`, and `X-Api-Key` authentication are accepted when
+`AML_API_KEY` is set.
+
+The native `find` call uses quick retrieval without reranking, even when the
+OpenViking server has a reranker configured. The adapter returns each hit's
+`content` when present, otherwise its `abstract`, without an additional
+full-text read.
+
+Run these modules from a source checkout with the project's Python dependencies
+installed. The standard OpenViking image and installed package do not include
+`benchmark/aml`.
+
+## Local Add/Search
+
+The input files already exist locally and are not generated by this code:
+
+```bash
+python -m benchmark.aml.eval.run_public \
+ --prepared result/aml_data/prepared/lme_locomo_script.json \
+ --prepared result/aml_data/prepared/persona_cl.json \
+ --prepared result/aml_data/prepared/beam.json \
+ --namespace local-check-v1 \
+ --base-url http://127.0.0.1:8088 \
+ --output-dir result/aml-public/retrieval \
+ --add-concurrency 32 --search-concurrency 16
+```
+
+The main flow stops after Add/Search and writes one `retrieval.jsonl` under
+each runnable dataset directory, plus a small top-level `summary.json`. Empty
+private-data declarations are ignored because they contain nothing that can be
+sent to Add or Search.
+
+The prepared files contain `history_key`, not an AML `user_id`. The local
+runner therefore creates one stable AML user for each unique
+`namespace + unit + history_key + history contents`: questions sharing a
+history share that user, while different histories stay isolated. This is ID
+creation in the local runner; `server.py` forwards the resulting ID unchanged.
+
+Use repeated `--unit` arguments to select individual datasets. Add runs up to
+16 histories (AML users) concurrently by default, while Search runs up to 16 questions
+concurrently. Override them independently with `--add-concurrency` and
+`--search-concurrency`; the existing `--concurrency` option remains a shorthand
+baseline for both. Phase-specific options take precedence. Within each history,
+sessions run in prepared order, and chunks within each session run in order.
+Every Add waits for its commit to finish before the next request for that user;
+Search starts after all histories finish importing. For BEAM, this preserves
+batch order within a conversation while different conversations run concurrently.
+
+For smoke/full presets, `run_ecs` accepts a provider URL through `--base-url`
+or `AML_BASE_URL`; it defaults to the local adapter. It reads authentication
+from `AML_API_KEY` or `--api-key-file` and creates a fresh namespace per run:
+
+```bash
+python -m benchmark.aml.eval.run_ecs list
+python -m benchmark.aml.eval.run_ecs full beam_100k \
+ --base-url http://127.0.0.1:8088 --add-concurrency 32 --dry-run
+```
+
+Remove `--dry-run` to import. The generic callers do not resume an interrupted
+import. Reconcile the original server tasks and archives before retrying an
+ambiguous Add; a new namespace starts a separate experiment.
+
+## PersonaMem-v2 text preparation
+
+`eval/prepare_personamem.py` follows the exact `chat_history_32k_link` and
+`chat_history_128k_link` paths in `benchmark/text/benchmark.csv`. Each size is a
+separate condition; their histories are not concatenated. Shared histories
+are imported once per run and reused by their questions.
+
+The preparer removes all system persona messages and blank messages, preserves
+the remaining user/assistant text and order, and omits timestamps absent from
+the source. It does not assemble histories from raw persona records or gold
+snippets. Reference answers, preference annotations and distractors stay in
+case metadata and never enter Add messages.
+
+```bash
+python -m benchmark.aml.eval.prepare_personamem \
+ --source-dir result/aml_data/datasets/personamem_v2 \
+ --verification result/aml_data/personamem_v2_download/verification.json \
+ --output-dir result/aml_data/prepared/full
+```
+
+The downloaded source and its verification manifest must already exist. The
+manifest contains `success: true`, the source `revision`, and a `files` array
+with relative `path` and `sha256` for the CSV and linked chat files. Preparation
+checks those hashes, writes both size-specific prepared files and backs up any
+replaced outputs. Optional `--smoke-file` refreshes existing 32K smoke cases.
+Preparation makes no Add/Search or model calls and covers text inputs only.
+
+## Optional Answer/Eval
+
+The optional evaluator consumes one completed retrieval artifact and never
+repeats Add or Search. It reads the artifact's `unit` and dispatches to the
+matching public AML pipeline for LongMemEval-S, LoCoMo-Refined, BEAM,
+CLBench, or PersonaMem-v2:
+
+```bash
+export ANSWER_API_BASE='https://answer-endpoint/v1'
+export ANSWER_API_KEY='...'
+export ANSWER_MODEL='...'
+export JUDGE_API_BASE='https://judge-endpoint/v1'
+export JUDGE_API_KEY='...'
+export JUDGE_MODEL='...'
+
+python -m benchmark.aml.eval.evaluate \
+ --retrieval result/aml-public/retrieval/longmemeval_s/retrieval.jsonl \
+ --aml-repo result/aml-public-eval/agent-memory-leaderboard \
+ --answer-concurrency 16 --eval-concurrency 32
+```
+
+This produces `answers.jsonl`, `judgments.jsonl`, and `summary.json` beside the
+retrieval file. Answer prompts and scoring stay inside the pinned public
+dataset pipeline; `evaluate.py` only selects it and adapts the generic
+`retrieved_context` field where CLBench or PersonaMem expects a different input
+shape. Answer and Judge each run up to 32 pipeline shards concurrently by
+default; use `--answer-concurrency` and `--eval-concurrency` to tune them
+independently. Outputs are merged back into retrieval order and published only
+after every shard succeeds. These remain local proxy results rather than
+official leaderboard scores. Model options and retries follow the selected
+public pipeline; this wrapper does not provide the experiment-specific pacing
+or resume logic. The pinned pipelines currently combine
+an asynchronous HTTP client and a regular output file in one `async with`
+statement; the wrapper corrects only that context-manager syntax in a temporary
+runtime copy, leaving the verified checkout, prompts, and scoring code unchanged.
+
+PersonaMem defaults to `--persona-mode mcq`, even for older retrieval files
+marked as generative. Answer receives only retrieved memory, the question and
+four shuffled choices from `correct_answer` plus `incorrect_answers`; preference
+labels and the correct option marker are not added to the prompt. Search does
+not receive these choices. The public pipeline maps the predicted letter to
+its option text and grades it locally; invalid answers count as incorrect.
+Only the three `ANSWER_*` variables are required for MCQ. Use
+`--persona-mode generative` explicitly for the public narrow LLM judge instead.
+Subprocesses default to `PYTHONHASHSEED=0` to keep option mappings stable across
+shard counts. These are OV retrieval-based evaluations, not reproductions of
+the authors' trained MemAgent implementation.
+
+## Static checks
+
+```bash
+python -m ruff check benchmark/aml
+python -m ruff format --check benchmark/aml
+```
diff --git a/benchmark/aml/__init__.py b/benchmark/aml/__init__.py
new file mode 100644
index 000000000..ca42f6335
--- /dev/null
+++ b/benchmark/aml/__init__.py
@@ -0,0 +1 @@
+"""Agent Memory Leaderboard adapter benchmark utilities."""
diff --git a/benchmark/aml/eval/__init__.py b/benchmark/aml/eval/__init__.py
new file mode 100644
index 000000000..3bf85df07
--- /dev/null
+++ b/benchmark/aml/eval/__init__.py
@@ -0,0 +1,6 @@
+"""Local, public-data evaluation utilities for the AML adapter.
+
+These helpers reproduce the public Add/Search contract and reuse AML's public
+prompt renderers. They do not reproduce the private AML dataset bundle,
+production evidence assembler, model deployment, or leaderboard aggregation.
+"""
diff --git a/benchmark/aml/eval/evaluate.py b/benchmark/aml/eval/evaluate.py
new file mode 100644
index 000000000..2578a54bb
--- /dev/null
+++ b/benchmark/aml/eval/evaluate.py
@@ -0,0 +1,597 @@
+"""Optionally run the public AML Answer/Eval pipeline for one retrieval file."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any, Callable, Sequence
+
+DEFAULT_AML_COMMIT = "1b8142bfe0f20f1c5218d6b554aa0012de34e504"
+DEFAULT_ANSWER_CONCURRENCY = 32
+DEFAULT_EVAL_CONCURRENCY = 32
+
+_MIXED_OUTPUT_CONTEXT = re.compile(
+ r"^(\s*)async with (httpx\.AsyncClient\(.*\) as client), "
+ r"(.+\.open\(.*\) as \w+):\s*$"
+)
+
+# kind controls only the retrieval-to-pipeline input adapter. Answer prompts and
+# scoring remain owned by the corresponding public AML pipeline.
+PIPELINES = {
+ "longmemeval_s": ("data/longmemeval-s/pipeline.py", "generic"),
+ "locomo_refined": ("data/locomo-refined/pipeline.py", "generic"),
+ "beam_100k": ("data/beam/pipeline.py", "generic"),
+ "beam_1m": ("data/beam/pipeline.py", "generic"),
+ "clbench_0_4k": ("data/clbench/pipeline.py", "clbench"),
+ "clbench_16_32k": ("data/clbench/pipeline.py", "clbench"),
+ "personamem_v2_32k": ("data/personamem/pipeline_v2.py", "personamem_v2"),
+ "personamem_v2_128k": ("data/personamem/pipeline_v2.py", "personamem_v2"),
+}
+
+
+def _git(repo: Path, *args: str) -> str:
+ try:
+ result = subprocess.run(
+ ["git", "-C", str(repo), *args],
+ check=True,
+ capture_output=True,
+ text=True,
+ timeout=10,
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ raise RuntimeError(f"cannot inspect AML checkout: {exc}") from exc
+ return result.stdout.strip()
+
+
+def _read_jsonl(path: Path) -> list[dict[str, Any]]:
+ rows = []
+ seen = set()
+ for line_number, line in enumerate(path.read_text(encoding="utf-8").split("\n"), start=1):
+ if not line.strip():
+ continue
+ try:
+ row = json.loads(line)
+ except json.JSONDecodeError as exc:
+ raise ValueError(f"{path}:{line_number}: invalid JSON") from exc
+ if not isinstance(row, dict):
+ raise ValueError(f"{path}:{line_number}: expected an object")
+ raw_id = row.get("id", row.get("idx"))
+ if raw_id is None:
+ raise ValueError(f"{path}:{line_number}: expected id or idx")
+ ident = str(raw_id)
+ if ident in seen:
+ raise ValueError(f"{path}:{line_number}: duplicate id {ident!r}")
+ seen.add(ident)
+ rows.append(row)
+ if not rows:
+ raise ValueError(f"JSONL input is empty: {path}")
+ return rows
+
+
+def _write_jsonl(path: Path, rows: Sequence[dict[str, Any]]) -> None:
+ path.write_text(
+ "".join(json.dumps(row, ensure_ascii=True) + "\n" for row in rows),
+ encoding="utf-8",
+ )
+
+
+def _row_id(row: dict[str, Any]) -> str:
+ raw_id = row.get("id", row.get("idx"))
+ if raw_id is None:
+ raise ValueError("pipeline row has no id or idx")
+ return str(raw_id)
+
+
+def _pipeline_row_id(row: dict[str, Any], kind: str) -> str:
+ if kind != "clbench":
+ return _row_id(row)
+ for key in ("idx", "id", "question_id", "task_id"):
+ if row.get(key) is not None:
+ return str(row[key])
+ metadata = row.get("metadata")
+ if isinstance(metadata, dict):
+ for key in ("task_id", "id"):
+ if metadata.get(key) is not None:
+ return str(metadata[key])
+ raise ValueError("CLBench pipeline row has no supported identifier")
+
+
+def _positive_int(value: str) -> int:
+ parsed = int(value)
+ if parsed < 1:
+ raise argparse.ArgumentTypeError("must be at least 1")
+ return parsed
+
+
+def _resolve_unit(rows: Sequence[dict[str, Any]], requested: str | None) -> str:
+ declared = {str(row["unit"]) for row in rows if row.get("unit") is not None}
+ if requested is not None:
+ if declared and declared != {requested}:
+ raise ValueError(
+ f"--unit {requested!r} does not match retrieval units: {sorted(declared)}"
+ )
+ unit = requested
+ elif len(declared) == 1:
+ unit = declared.pop()
+ else:
+ raise ValueError("retrieval must contain exactly one unit, or pass --unit")
+ if unit not in PIPELINES:
+ raise ValueError(f"no public Answer/Eval pipeline configured for unit {unit!r}")
+ return unit
+
+
+def _contexts(row: dict[str, Any]) -> list[str]:
+ value = row.get("retrieved_context")
+ if not isinstance(value, list) or any(not isinstance(item, str) for item in value):
+ raise ValueError(f"record {row.get('id')!r} has invalid retrieved_context")
+ return value
+
+
+def _adapt_rows(
+ rows: Sequence[dict[str, Any]], kind: str, *, persona_mode: str | None = None
+) -> tuple[list[dict[str, Any]], str | None]:
+ if persona_mode is not None:
+ if kind != "personamem_v2":
+ raise ValueError("--persona-mode is only supported for PersonaMem v2")
+ if persona_mode not in {"generative", "mcq"}:
+ raise ValueError(f"unsupported PersonaMem mode: {persona_mode!r}")
+ if kind == "generic":
+ for row in rows:
+ _contexts(row)
+ return [dict(row) for row in rows], None
+
+ adapted = []
+ modes = set()
+ for row in rows:
+ item = dict(row)
+ contexts = _contexts(row)
+ if kind == "clbench":
+ item["retrieval"] = {"selected": [{"text": content} for content in contexts]}
+ elif kind == "personamem_v2":
+ # Apply the default to older retrieval files as well; generation
+ # requires an explicit mode choice.
+ mode = persona_mode or "mcq"
+ item["answer_mode"] = mode
+ if mode == "mcq":
+ correct = item.get("correct_answer")
+ incorrect = item.get("incorrect_answers")
+ if isinstance(incorrect, str):
+ try:
+ incorrect = json.loads(incorrect)
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"PersonaMem row {_row_id(item)!r}: invalid MCQ options"
+ ) from exc
+ if (
+ not isinstance(correct, str)
+ or not correct.strip()
+ or not isinstance(incorrect, list)
+ or len(incorrect) != 3
+ or any(
+ not isinstance(option, str) or not option.strip() for option in incorrect
+ )
+ or len({option.strip() for option in [correct, *incorrect]}) != 4
+ ):
+ raise ValueError(
+ f"PersonaMem row {_row_id(item)!r}: MCQ requires a correct answer "
+ "and three distinct incorrect answers"
+ )
+ item["incorrect_answers"] = list(incorrect)
+ memory_text = "\n\n".join(
+ f"[Retrieved memory {index}]\n{content}"
+ for index, content in enumerate(contexts, start=1)
+ )
+ item["chat_history"] = [
+ {
+ "role": "system",
+ "content": memory_text or "No relevant memories were retrieved.",
+ }
+ ]
+ modes.add(mode)
+ else:
+ raise AssertionError(f"unknown pipeline adapter: {kind}")
+ adapted.append(item)
+
+ if kind == "personamem_v2":
+ if not modes <= {"generative", "mcq"} or len(modes) != 1:
+ raise ValueError(f"PersonaMem retrieval has unsupported mixed modes: {sorted(modes)}")
+ return adapted, modes.pop()
+ return adapted, None
+
+
+def _verify_pipeline(
+ repo: Path,
+ relative_path: str,
+ expected_commit: str,
+) -> tuple[Path, dict[str, str]]:
+ repo = repo.expanduser().resolve()
+ pipeline = repo / relative_path
+ if not pipeline.is_file():
+ raise ValueError(f"AML pipeline not found: {pipeline}")
+ actual_commit = _git(repo, "rev-parse", "HEAD")
+ if actual_commit != expected_commit:
+ raise ValueError(
+ f"AML checkout commit mismatch: expected {expected_commit}, got {actual_commit}"
+ )
+ if _git(repo, "status", "--short", "--", relative_path):
+ raise ValueError(f"AML pipeline has local modifications: {pipeline}")
+ return pipeline, {
+ "commit": actual_commit,
+ "path": relative_path,
+ "sha256": hashlib.sha256(pipeline.read_bytes()).hexdigest(),
+ }
+
+
+def _nest_synchronous_output_contexts(source: str) -> tuple[str, int]:
+ """Make the pinned pipelines' regular output files valid inside async functions."""
+ lines = source.splitlines()
+ result = []
+ replacements = 0
+ index = 0
+ while index < len(lines):
+ line = lines[index]
+ match = _MIXED_OUTPUT_CONTEXT.match(line)
+ if match is None:
+ result.append(line)
+ index += 1
+ continue
+
+ indent, async_manager, sync_manager = match.groups()
+ result.append(f"{indent}async with {async_manager}:")
+ result.append(f"{indent} with {sync_manager}:")
+ replacements += 1
+ index += 1
+ while index < len(lines):
+ body_line = lines[index]
+ stripped = body_line.lstrip()
+ if stripped and len(body_line) - len(stripped) <= len(indent):
+ break
+ result.append(f" {body_line}" if stripped else body_line)
+ index += 1
+
+ return "\n".join(result) + "\n", replacements
+
+
+def _runtime_pipeline(repo: Path, pipeline: Path, temporary: Path) -> Path:
+ source = pipeline.read_text(encoding="utf-8")
+ patched_source, replacements = _nest_synchronous_output_contexts(source)
+ if replacements == 0:
+ return pipeline
+ try:
+ compile(patched_source, str(pipeline), "exec")
+ except SyntaxError as exc:
+ raise RuntimeError(f"cannot apply AML pipeline output compatibility fix: {exc}") from exc
+
+ runtime_root = temporary / "pipeline-runtime"
+ runtime_pipeline = runtime_root / pipeline.relative_to(repo)
+ runtime_pipeline.parent.mkdir(parents=True)
+ runtime_pipeline.write_text(patched_source, encoding="utf-8")
+ api_config = repo / "api_config.py"
+ if not api_config.is_file():
+ raise ValueError(f"AML API config not found: {api_config}")
+ (runtime_root / "api_config.py").write_bytes(api_config.read_bytes())
+ return runtime_pipeline
+
+
+def _require_model_environment(*, needs_judge: bool) -> None:
+ names = ["ANSWER_API_BASE", "ANSWER_API_KEY", "ANSWER_MODEL"]
+ if needs_judge:
+ names.extend(["JUDGE_API_BASE", "JUDGE_API_KEY", "JUDGE_MODEL"])
+ missing = [name for name in names if not os.getenv(name)]
+ if missing:
+ raise ValueError(f"missing model environment: {', '.join(missing)}")
+
+
+def _run(repo: Path, arguments: Sequence[str]) -> None:
+ environment = os.environ.copy()
+ environment.setdefault("PYTHONHASHSEED", "0")
+ try:
+ result = subprocess.run(
+ [sys.executable, *arguments],
+ cwd=repo,
+ capture_output=True,
+ text=True,
+ env=environment,
+ )
+ except OSError as exc:
+ raise RuntimeError(f"cannot start AML pipeline: {exc}") from exc
+ if result.returncode:
+ detail = (result.stderr or result.stdout).strip()[-2_000:]
+ raise RuntimeError(f"AML pipeline failed with exit {result.returncode}: {detail}")
+
+
+def _partition_rows(rows: Sequence[dict[str, Any]], concurrency: int) -> list[list[dict[str, Any]]]:
+ shard_count = min(len(rows), concurrency)
+ if shard_count == 0:
+ return []
+ return [list(rows[index::shard_count]) for index in range(shard_count)]
+
+
+def _run_parallel(
+ repo: Path,
+ stage: str,
+ commands: Sequence[Sequence[str]],
+ concurrency: int,
+) -> None:
+ if not commands:
+ return
+ with ThreadPoolExecutor(max_workers=min(concurrency, len(commands))) as executor:
+ futures = {
+ executor.submit(_run, repo, command): index for index, command in enumerate(commands)
+ }
+ for future in as_completed(futures):
+ try:
+ future.result()
+ except RuntimeError as exc:
+ for pending in futures:
+ pending.cancel()
+ index = futures[future]
+ raise RuntimeError(
+ f"AML {stage} shard {index + 1}/{len(commands)} failed: {exc}"
+ ) from exc
+
+
+def _merge_outputs(
+ paths: Sequence[Path],
+ expected_rows: Sequence[dict[str, Any]],
+ stage: str,
+ identify: Callable[[dict[str, Any]], str] = _row_id,
+) -> list[dict[str, Any]]:
+ expected_ids = [identify(row) for row in expected_rows]
+ if len(expected_ids) != len(set(expected_ids)):
+ raise ValueError(f"AML {stage} input contains duplicate pipeline identifiers")
+ merged: dict[str, dict[str, Any]] = {}
+ for path in paths:
+ try:
+ rows = _read_jsonl(path)
+ except (OSError, ValueError) as exc:
+ raise RuntimeError(f"AML {stage} produced invalid output {path}: {exc}") from exc
+ for row in rows:
+ ident = identify(row)
+ if ident in merged:
+ raise RuntimeError(f"AML {stage} produced duplicate id {ident!r}")
+ merged[ident] = row
+
+ expected = set(expected_ids)
+ actual = set(merged)
+ if expected != actual:
+ missing = sorted(expected - actual)[:10]
+ unexpected = sorted(actual - expected)[:10]
+ raise RuntimeError(
+ f"AML {stage} output ID mismatch: missing={missing}, unexpected={unexpected}"
+ )
+ return [merged[ident] for ident in expected_ids]
+
+
+ShardCommand = Callable[[Path, Path, Sequence[dict[str, Any]]], Sequence[str]]
+
+
+def _run_sharded_stage(
+ repo: Path,
+ rows: Sequence[dict[str, Any]],
+ destination: Path,
+ temporary: Path,
+ *,
+ stage: str,
+ concurrency: int,
+ command: ShardCommand,
+ identify: Callable[[dict[str, Any]], str] = _row_id,
+) -> list[dict[str, Any]]:
+ identifiers = [identify(row) for row in rows]
+ if len(identifiers) != len(set(identifiers)):
+ raise ValueError(f"AML {stage} input contains duplicate pipeline identifiers")
+ stage_dir = temporary / stage
+ stage_dir.mkdir()
+ commands = []
+ outputs = []
+ for index, shard_rows in enumerate(_partition_rows(rows, concurrency)):
+ shard_dir = stage_dir / f"{index:04d}"
+ shard_dir.mkdir()
+ input_path = shard_dir / "input.jsonl"
+ output_path = shard_dir / "output.jsonl"
+ _write_jsonl(input_path, shard_rows)
+ commands.append(command(input_path, output_path, shard_rows))
+ outputs.append(output_path)
+
+ _run_parallel(repo, stage, commands, concurrency)
+ merged = _merge_outputs(outputs, rows, stage, identify)
+ _write_jsonl(destination, merged)
+ return merged
+
+
+def evaluate(
+ retrieval_path: Path,
+ output_dir: Path,
+ *,
+ aml_repo: Path,
+ unit: str | None = None,
+ aml_commit: str = DEFAULT_AML_COMMIT,
+ answer_concurrency: int = DEFAULT_ANSWER_CONCURRENCY,
+ eval_concurrency: int = DEFAULT_EVAL_CONCURRENCY,
+ persona_mode: str | None = None,
+) -> dict[str, Any]:
+ """Dispatch one retrieval file to its public dataset pipeline."""
+ retrieval_path = retrieval_path.expanduser().resolve()
+ if not retrieval_path.is_file():
+ raise ValueError(f"retrieval file does not exist: {retrieval_path}")
+ if answer_concurrency < 1:
+ raise ValueError("answer_concurrency must be positive")
+ if eval_concurrency < 1:
+ raise ValueError("eval_concurrency must be positive")
+ rows = _read_jsonl(retrieval_path)
+ unit = _resolve_unit(rows, unit)
+ relative_path, kind = PIPELINES[unit]
+ repo = aml_repo.expanduser().resolve()
+ pipeline, pipeline_info = _verify_pipeline(repo, relative_path, aml_commit)
+ adapted_rows, persona_mode = _adapt_rows(rows, kind, persona_mode=persona_mode)
+
+ output_dir = output_dir.expanduser().resolve()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ answers_path = output_dir / "answers.jsonl"
+ judgments_path = output_dir / "judgments.jsonl"
+ summary_path = output_dir / "summary.json"
+ existing = [path.name for path in (answers_path, judgments_path, summary_path) if path.exists()]
+ if existing:
+ raise ValueError(f"output already exists: {', '.join(existing)}")
+
+ needs_judge = not (kind == "personamem_v2" and persona_mode == "mcq")
+ _require_model_environment(needs_judge=needs_judge)
+
+ def identify(row: dict[str, Any]) -> str:
+ return _pipeline_row_id(row, kind)
+
+ with tempfile.TemporaryDirectory(prefix=".aml-eval-", dir=output_dir) as temporary:
+ temporary_path = Path(temporary)
+ staged_answers_path = temporary_path / "answers.jsonl"
+ staged_judgments_path = temporary_path / "judgments.jsonl"
+ runtime_pipeline = _runtime_pipeline(repo, pipeline, temporary_path)
+
+ def answer_command(
+ input_path: Path,
+ output_path: Path,
+ _shard_rows: Sequence[dict[str, Any]],
+ ) -> Sequence[str]:
+ arguments = [
+ str(runtime_pipeline),
+ "answer",
+ "--input",
+ str(input_path),
+ "--output",
+ str(output_path),
+ ]
+ if kind == "personamem_v2":
+ arguments.extend(["--mode", persona_mode or "generative"])
+ return arguments
+
+ answer_rows = _run_sharded_stage(
+ repo,
+ adapted_rows,
+ staged_answers_path,
+ temporary_path,
+ stage="answer",
+ concurrency=answer_concurrency,
+ command=answer_command,
+ identify=identify,
+ )
+ answers_by_id = {identify(row): row for row in answer_rows}
+
+ def judge_command(
+ input_path: Path,
+ output_path: Path,
+ shard_rows: Sequence[dict[str, Any]],
+ ) -> Sequence[str]:
+ shard_answers = input_path.with_name("answers.jsonl")
+ _write_jsonl(shard_answers, [answers_by_id[identify(row)] for row in shard_rows])
+ if kind == "personamem_v2" and persona_mode == "mcq":
+ return [
+ str(runtime_pipeline),
+ "evaluate-mcq",
+ "--answers",
+ str(shard_answers),
+ "--output",
+ str(output_path),
+ ]
+ return [
+ str(runtime_pipeline),
+ "evaluate-narrow" if kind == "personamem_v2" else "evaluate",
+ "--input",
+ str(input_path),
+ "--answers",
+ str(shard_answers),
+ "--output",
+ str(output_path),
+ ]
+
+ judgment_rows = _run_sharded_stage(
+ repo,
+ adapted_rows,
+ staged_judgments_path,
+ temporary_path,
+ stage="eval",
+ concurrency=eval_concurrency,
+ command=judge_command,
+ identify=identify,
+ )
+ staged_answers_path.replace(answers_path)
+ staged_judgments_path.replace(judgments_path)
+
+ summary = {
+ "unit": unit,
+ "retrieval": str(retrieval_path),
+ "pipeline": pipeline_info,
+ "answer_rows": len(answer_rows),
+ "evaluation_rows": len(judgment_rows),
+ "answers": str(answers_path),
+ "judgments": str(judgments_path),
+ }
+ if kind == "personamem_v2":
+ summary["persona_mode"] = persona_mode
+ if persona_mode == "mcq":
+ correct_count = sum(row["is_correct"] is True for row in judgment_rows)
+ summary.update(
+ scoring_protocol="personamem-v2-public-mcq",
+ correct_count=correct_count,
+ incorrect_count=len(judgment_rows) - correct_count,
+ invalid_answer_count=sum(row["predicted_answer"] is None for row in judgment_rows),
+ accuracy=correct_count / len(judgment_rows),
+ shuffle_pythonhashseed=os.environ.get("PYTHONHASHSEED", "0"),
+ )
+ summary_path.write_text(
+ json.dumps(summary, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ return summary
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--retrieval", required=True, type=Path)
+ parser.add_argument("--output-dir", type=Path)
+ parser.add_argument("--unit", choices=sorted(PIPELINES))
+ parser.add_argument("--aml-repo", required=True, type=Path)
+ parser.add_argument("--aml-commit", default=DEFAULT_AML_COMMIT)
+ parser.add_argument(
+ "--persona-mode",
+ choices=["mcq", "generative"],
+ help="PersonaMem Answer/scoring mode using existing retrieval (default: mcq)",
+ )
+ parser.add_argument(
+ "--answer-concurrency",
+ type=_positive_int,
+ default=DEFAULT_ANSWER_CONCURRENCY,
+ help=f"parallel Answer shards (default: {DEFAULT_ANSWER_CONCURRENCY})",
+ )
+ parser.add_argument(
+ "--eval-concurrency",
+ type=_positive_int,
+ default=DEFAULT_EVAL_CONCURRENCY,
+ help=f"parallel Eval/Judge shards (default: {DEFAULT_EVAL_CONCURRENCY})",
+ )
+ args = parser.parse_args(argv)
+ try:
+ summary = evaluate(
+ args.retrieval,
+ args.output_dir or args.retrieval.parent,
+ aml_repo=args.aml_repo,
+ unit=args.unit,
+ aml_commit=args.aml_commit,
+ answer_concurrency=args.answer_concurrency,
+ eval_concurrency=args.eval_concurrency,
+ persona_mode=args.persona_mode,
+ )
+ except (OSError, RuntimeError, ValueError) as exc:
+ parser.exit(1, f"{type(exc).__name__}: {exc}\n")
+ print(json.dumps(summary, ensure_ascii=False, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmark/aml/eval/prepare_personamem.py b/benchmark/aml/eval/prepare_personamem.py
new file mode 100644
index 000000000..0fac2fe4f
--- /dev/null
+++ b/benchmark/aml/eval/prepare_personamem.py
@@ -0,0 +1,371 @@
+"""Prepare PersonaMem-v2 text benchmark histories without system persona profiles.
+
+Each size follows its exact benchmark CSV link. Raw persona records and gold
+conversation snippets are never used to assemble the Add messages.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import csv
+import hashlib
+import json
+import shutil
+from datetime import datetime, timezone
+from pathlib import Path, PurePosixPath
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[3]
+DATA_ROOT = ROOT / "result/aml_data"
+POLICY = "text-chat-only-drop-system-v1"
+SIZES = ("32k", "128k")
+BENCHMARK = "benchmark/text/benchmark.csv"
+
+
+def read_json(path: Path) -> Any:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def dump_json(path: Path, value: Any) -> None:
+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+
+
+def sha256(path: Path) -> str:
+ return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def source_file(root: Path, relative: str, expected: dict[str, str]) -> Path:
+ parts = PurePosixPath(relative)
+ if parts.is_absolute() or ".." in parts.parts or "\\" in relative:
+ raise ValueError(f"Invalid source path: {relative}")
+ path = root / relative
+ if not path.resolve().is_relative_to(root.resolve()):
+ raise ValueError(f"Source escapes dataset directory: {relative}")
+ if relative not in expected or sha256(path) != expected[relative]:
+ raise ValueError(f"Source differs from verified download: {relative}")
+ return path
+
+
+def literal(value: str) -> Any:
+ try:
+ return json.loads(value)
+ except json.JSONDecodeError:
+ return ast.literal_eval(value)
+
+
+def chat_messages(chat: Any) -> tuple[list[dict[str, str]], list[int], list[int]]:
+ if not isinstance(chat, list) or not chat:
+ raise ValueError("Expected a non-empty chat_history array")
+ messages = []
+ removed = []
+ blank = []
+ for index, message in enumerate(chat):
+ if not isinstance(message, dict):
+ raise ValueError(f"Invalid message at source index {index}")
+ role = message.get("role")
+ if role == "system":
+ removed.append(index)
+ continue
+ content = message.get("content")
+ if role not in {"user", "assistant"} or not isinstance(content, str):
+ raise ValueError(f"Unsupported text message at source index {index}")
+ if not content.strip():
+ blank.append(index)
+ continue
+ # Preserve text, role and array order exactly. The source has no real
+ # timestamps; do not manufacture dates or infer session boundaries.
+ messages.append({"role": role, "content": content})
+ if not messages:
+ raise ValueError("No conversation messages remain after dropping system")
+ return messages, removed, blank
+
+
+def build_prepared(
+ root: Path,
+ rows: list[dict[str, str]],
+ size: str,
+ revision: str,
+ expected: dict[str, str],
+) -> dict[str, Any]:
+ if size not in SIZES:
+ raise ValueError(f"Unsupported text history size: {size}")
+ unit = f"personamem_v2_{size}"
+ column = f"chat_history_{size}_link"
+ benchmark = source_file(root, BENCHMARK, expected)
+ histories: dict[tuple[str, str], dict[str, Any]] = {}
+ provenance: dict[tuple[str, str], dict[str, Any]] = {}
+ cases = []
+ for index, row in enumerate(rows):
+ persona_id = row["persona_id"]
+ relative = row[column]
+ if not relative.startswith(f"data/chat_history_{size}/"):
+ raise ValueError(f"Row {index}: wrong text history directory for {column}")
+ identity = (persona_id, relative)
+ if identity not in histories:
+ path = source_file(root, relative, expected)
+ source = read_json(path)
+ metadata = source["metadata"]
+ if str(metadata["persona_id"]) != persona_id:
+ raise ValueError(f"Row {index}: history persona does not match CSV")
+ if metadata.get("input_filename") != row["raw_persona_file"]:
+ raise ValueError(f"Row {index}: raw persona reference does not match history")
+ messages, removed, blank = chat_messages(source["chat_history"])
+ history_key = f"personamem-v2-{persona_id}-{path.stem}"
+ histories[identity] = {
+ "unit": unit,
+ "history_key": history_key,
+ "sessions": [{"session_id": f"persona-{persona_id}", "messages": messages}],
+ }
+ provenance[identity] = {
+ "source_path": str(path.resolve()),
+ "source_relative_path": relative,
+ "source_url": f"https://huggingface.co/datasets/bowen-upenn/PersonaMem-v2/resolve/{revision}/{relative}",
+ "revision": revision,
+ "sha256": expected[relative],
+ "history_link_column": column,
+ "raw_persona_file": row["raw_persona_file"],
+ "history_scope": f"complete linked public text {size} history excluding system messages",
+ "preprocessing_policy": POLICY,
+ "dropped_system_message_indices": removed,
+ "dropped_blank_message_indices": blank,
+ "source_counts": {
+ "sessions": 1,
+ "messages_including_system": len(source["chat_history"]),
+ "retained_messages": len(messages),
+ "source_reported_tokens_including_system": metadata.get("final_token_count"),
+ "source_reported_irrelevant_tokens": metadata.get("num_irrelevant_tokens", 0),
+ },
+ "transformations": [
+ "drop system and blank messages; preserve remaining role/content/order exactly"
+ ],
+ "timestamp_policy": "absent in source; preserve sequence without synthetic dates",
+ "aml_equivalence": "public-source proxy; private AML retrieval-to-chat assembly unavailable",
+ }
+ if provenance[identity]["raw_persona_file"] != row["raw_persona_file"]:
+ raise ValueError(f"Row {index}: conflicting raw persona reference")
+ query = literal(row["user_query"])
+ incorrect = literal(row["incorrect_answers"])
+ if (
+ not isinstance(query, dict)
+ or query.get("role") != "user"
+ or not isinstance(query.get("content"), str)
+ or not query["content"].strip()
+ ):
+ raise ValueError(f"Row {index}: invalid user_query")
+ if not isinstance(incorrect, list) or any(not isinstance(a, str) for a in incorrect):
+ raise ValueError(f"Row {index}: invalid incorrect_answers")
+ # Evaluation labels stay on cases. They are not part of history.messages.
+ extra = {
+ key: row[key]
+ for key in (
+ "preference",
+ "sensitive_info",
+ "who",
+ "updated",
+ "prev_pref",
+ "topic_query",
+ "topic_preference",
+ "conversation_scenario",
+ "pref_type",
+ )
+ if key in row
+ }
+ extra.update(
+ persona_id=persona_id,
+ user_query=query,
+ correct_answer=row["correct_answer"],
+ incorrect_answers=incorrect,
+ answer_mode="mcq",
+ )
+ cases.append(
+ {
+ "unit": unit,
+ "case_id": f"personamem-v2-row-{index}-persona-{persona_id}",
+ "history_key": histories[identity]["history_key"],
+ "question": query["content"],
+ "gold_answer": row["correct_answer"],
+ "options": [],
+ "extra": extra,
+ "provenance": {**provenance[identity], "benchmark_row_offset": index},
+ }
+ )
+ dataset = {
+ "unit": unit,
+ "profile": "full",
+ "revision": revision,
+ "source": {
+ "path": str(benchmark.resolve()),
+ "bytes": benchmark.stat().st_size,
+ "sha256": expected[BENCHMARK],
+ },
+ "source_rows": len(rows),
+ "eligible_cases": len(cases),
+ "selected_cases": len(cases),
+ "selection": {"method": "all-text-benchmark-rows-in-source-order"},
+ "history_link_column": column,
+ "referenced_histories": len(histories),
+ "loaded_histories": len(histories),
+ "preprocessing_policy": POLICY,
+ "answer_mode": "mcq",
+ "dropped_system_messages": sum(
+ len(p["dropped_system_message_indices"]) for p in provenance.values()
+ ),
+ "dropped_blank_messages": sum(
+ len(p["dropped_blank_message_indices"]) for p in provenance.values()
+ ),
+ "retained_messages": sum(
+ p["source_counts"]["retained_messages"] for p in provenance.values()
+ ),
+ "timestamp_policy": "absent in source; preserve sequence without synthetic dates",
+ "aml_equivalence": "public-source proxy; private AML retrieval-to-chat assembly unavailable",
+ }
+ return {
+ "schema_version": 2,
+ "profile": "full",
+ "unit": unit,
+ "datasets": [dataset],
+ "histories": list(histories.values()),
+ "cases": cases,
+ }
+
+
+def refresh_smoke(smoke: dict[str, Any], prepared: dict[str, Any]) -> dict[str, Any]:
+ unit = prepared["unit"]
+ cases = {case["case_id"]: case for case in prepared["cases"]}
+ histories = {h["history_key"]: h["sessions"] for h in prepared["histories"]}
+ updated = []
+ for case in smoke["cases"]:
+ if case["unit"] != unit:
+ updated.append(case)
+ continue
+ replacement = cases[case["case_id"]]
+ updated.append({**replacement, "histories": histories[replacement["history_key"]]})
+ metadata = {
+ **prepared["datasets"][0],
+ "profile": "smoke",
+ "selected_cases": sum(c["unit"] == unit for c in updated),
+ }
+ return {
+ **smoke,
+ "cases": updated,
+ "datasets": [metadata if d.get("unit") == unit else d for d in smoke["datasets"]],
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--source-dir", type=Path, default=DATA_ROOT / "datasets/personamem_v2")
+ parser.add_argument(
+ "--verification", type=Path, default=DATA_ROOT / "personamem_v2_download/verification.json"
+ )
+ parser.add_argument("--output-dir", type=Path, default=DATA_ROOT / "prepared/full")
+ parser.add_argument(
+ "--smoke-file",
+ type=Path,
+ help="Also refresh existing 32K smoke cases, preserving other units",
+ )
+ args = parser.parse_args()
+ root = args.source_dir.resolve()
+ verification = read_json(args.verification)
+ if not verification["success"]:
+ raise ValueError("Source download verification did not succeed")
+ expected = {entry["path"]: entry["sha256"] for entry in verification["files"]}
+ benchmark = source_file(root, BENCHMARK, expected)
+ with benchmark.open(newline="", encoding="utf-8") as stream:
+ rows = list(csv.DictReader(stream))
+ if not rows:
+ raise ValueError("Text benchmark is empty")
+ destination = args.output_dir.resolve()
+ destination.mkdir(parents=True, exist_ok=True)
+ stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
+ stage = destination.parent / f".personamem-text-{stamp}"
+ stage.mkdir()
+ outputs = []
+ artifacts = []
+ for size in SIZES:
+ payload = build_prepared(root, rows, size, verification["revision"], expected)
+ name = f"personamem_v2_{size}.json"
+ target = destination / name
+ staged = stage / name
+ dump_json(staged, payload)
+ outputs.append((staged, target))
+ artifacts.append(
+ {
+ "unit": payload["unit"],
+ "profile": "full",
+ "cases": len(payload["cases"]),
+ "histories": len(payload["histories"]),
+ "bytes": staged.stat().st_size,
+ "sha256": sha256(staged),
+ "path": str(target),
+ "dataset": payload["datasets"][0],
+ }
+ )
+ if size == "32k" and args.smoke_file:
+ smoke = refresh_smoke(read_json(args.smoke_file), payload)
+ staged_smoke = stage / "persona_cl.json"
+ dump_json(staged_smoke, smoke)
+ outputs.append((staged_smoke, args.smoke_file.resolve()))
+ print(
+ json.dumps(
+ {
+ "size": size,
+ "cases": len(payload["cases"]),
+ **{
+ k: payload["datasets"][0][k]
+ for k in [
+ "loaded_histories",
+ "retained_messages",
+ "dropped_system_messages",
+ ]
+ },
+ }
+ ),
+ flush=True,
+ )
+ del payload
+ manifest_path = destination / "manifest.json"
+ manifest = (
+ read_json(manifest_path)
+ if manifest_path.exists()
+ else {"schema_version": 1, "artifacts": []}
+ )
+ replaced = {a["unit"] for a in artifacts}
+ manifest["artifacts"] = [
+ a for a in manifest["artifacts"] if a["unit"] not in replaced
+ ] + artifacts
+ staged_manifest = stage / "manifest.json"
+ dump_json(staged_manifest, manifest)
+ outputs.append((staged_manifest, manifest_path))
+ backup = destination.parent / "backups" / f"personamem-before-chat-only-{stamp}"
+ backup.mkdir(parents=True)
+ backups = []
+ for _, target in outputs:
+ if target.exists():
+ saved = backup / target.name
+ shutil.copy2(target, saved)
+ backups.append({"original": str(target), "backup": str(saved), "sha256": sha256(saved)})
+ dump_json(backup / "backup-manifest.json", {"files": backups})
+ for staged, target in outputs:
+ staged.replace(target)
+ stage.rmdir()
+ report = {
+ "prepared_at": datetime.now(timezone.utc).isoformat(),
+ "preprocessing_policy": POLICY,
+ "revision": verification["revision"],
+ "scope": "text benchmark 32K and 128K",
+ "backup_directory": str(backup),
+ "artifacts": artifacts,
+ "normalizer": str(Path(__file__).resolve()),
+ "normalizer_sha256": sha256(Path(__file__)),
+ "ov_import_started": False,
+ }
+ dump_json(destination / "personamem_text_preparation.json", report)
+ print(
+ json.dumps({"prepared": [a["path"] for a in artifacts], "backup": str(backup)}), flush=True
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/aml/eval/run_ecs.py b/benchmark/aml/eval/run_ecs.py
new file mode 100644
index 000000000..61cfa9176
--- /dev/null
+++ b/benchmark/aml/eval/run_ecs.py
@@ -0,0 +1,287 @@
+#!/usr/bin/env python3
+"""Run local AML smoke/full datasets against a public Add/Search provider."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import shlex
+import sys
+import time
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Sequence
+from urllib.parse import urlsplit
+
+import httpx
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+if __package__ in {None, ""}:
+ sys.path.insert(0, str(REPO_ROOT))
+
+from benchmark.aml.eval import run_public as runner # noqa: E402
+
+DEFAULT_BASE_URL = runner.DEFAULT_BASE_URL
+DEFAULT_DATA_ROOT = REPO_ROOT / "result" / "aml_data"
+DEFAULT_OUTPUT_ROOT = REPO_ROOT / "result" / "aml_ecs"
+DEFAULT_KEY_FILE = REPO_ROOT / ".codex" / "ops" / "aml-api-key"
+
+# Every preset selects the named unit, even when a smoke file contains others.
+DATASETS = {
+ "longmemeval_s": "lme_locomo_script.json",
+ "locomo_refined": "lme_locomo_script.json",
+ "personamem_v2_32k": "persona_cl.json",
+ "clbench_0_4k": "persona_cl.json",
+ "clbench_16_32k": "persona_cl.json",
+ "beam_100k": "beam.json",
+ "beam_1m": "beam.json",
+}
+UNAVAILABLE = {
+ "longmemeval_refined": "AML has not published the transformed production data",
+ "angry": "ScriptMem has no published script history",
+ "enemy": "ScriptMem has no published script history",
+ "friends": "ScriptMem has no published script history",
+ "man_earth": "ScriptMem has no published script history",
+}
+
+
+def _now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _select_datasets(names: Sequence[str]) -> list[str]:
+ if not names:
+ raise ValueError("specify a dataset or 'all'; use 'list' to see every command")
+ if "all" in names:
+ if len(names) != 1:
+ raise ValueError("'all' cannot be combined with individual datasets")
+ return list(DATASETS)
+ for name in names:
+ if name in UNAVAILABLE:
+ raise ValueError(f"{name} is unavailable: {UNAVAILABLE[name]}")
+ if name not in DATASETS:
+ raise ValueError(f"unknown dataset {name!r}; use 'list' to see supported datasets")
+ return list(dict.fromkeys(names))
+
+
+def _base_url(value: str) -> str:
+ parsed = urlsplit(value)
+ if parsed.scheme not in {"http", "https"} or not parsed.hostname:
+ raise ValueError("base URL must be an http:// or https:// provider address")
+ if parsed.username or parsed.password or parsed.query or parsed.fragment:
+ raise ValueError("base URL must not contain credentials, a query, or a fragment")
+ if parsed.port == 0:
+ raise ValueError("base URL port must be at least 1")
+ return value.rstrip("/")
+
+
+def _api_key(path: Path) -> str:
+ value = os.getenv("AML_API_KEY", "").strip()
+ if not value and path.is_file():
+ value = path.read_text(encoding="utf-8").strip()
+ if not value:
+ raise ValueError("set AML_API_KEY or provide --api-key-file; see benchmark/aml/README.md")
+ if any(character.isspace() for character in value):
+ raise ValueError("AML API key must not contain whitespace")
+ return value
+
+
+def _write_report(path: Path, report: dict[str, Any]) -> None:
+ temporary = path.with_suffix(".tmp")
+ temporary.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ temporary.replace(path)
+
+
+def build_plan(args: argparse.Namespace) -> dict[str, Any]:
+ units = _select_datasets(args.datasets)
+ base_url = _base_url(args.base_url)
+ run_id = f"{args.mode}-{datetime.now(timezone.utc):%Y%m%dT%H%M%SZ}-{uuid.uuid4().hex[:12]}"
+ output = (args.output_dir or DEFAULT_OUTPUT_ROOT / run_id).expanduser().resolve()
+ if output.exists() and (not output.is_dir() or any(output.iterdir())):
+ raise ValueError(f"output directory must be new or empty: {output}")
+
+ sources: dict[Path, list[str]] = {}
+ for unit in units:
+ relative = Path("full") / f"{unit}.json" if args.mode == "full" else Path(DATASETS[unit])
+ source = (args.data_root.expanduser() / "prepared" / relative).resolve()
+ if not source.is_file():
+ raise FileNotFoundError(f"prepared data is missing: {source}; check --data-root")
+ sources.setdefault(source, []).append(unit)
+
+ counts = {}
+ for source, selected in sources.items():
+ # Load one input at a time for the offline plan, including validation of
+ # history references and exactly the same message splitting as execution.
+ loaded = runner._load_cases([source], selected)
+ for unit, cases in loaded.items():
+ groups, searches = runner._prepare_unit(unit, cases, f"ecs-{run_id}")
+ counts[unit] = {
+ "unit": unit,
+ "prepared": str(source),
+ "histories": len({case.user_id for case in searches}),
+ "sessions": len(groups),
+ "add_requests": sum(len(group) for group in groups),
+ "search_requests": len(searches),
+ }
+ del groups, searches
+ del loaded
+
+ default_search_concurrency = 4 if args.mode == "smoke" else 16
+ return {
+ "status": "planned",
+ "scope": "local public-source Add/Search; no Answer/Judge or official AML score",
+ "run_id": run_id,
+ "namespace": f"ecs-{run_id}",
+ "mode": args.mode,
+ "base_url": base_url,
+ "output_dir": str(output),
+ "prepared": [str(path) for path in sources],
+ "units": [counts[unit] for unit in units],
+ # Add holds the request open through commit extraction, including queue wait.
+ "add_concurrency": args.add_concurrency or 16,
+ "search_concurrency": args.search_concurrency or default_search_concurrency,
+ "timeout_seconds": args.timeout_seconds,
+ }
+
+
+async def execute(plan: dict[str, Any], api_key: str) -> dict[str, Any]:
+ output = Path(plan["output_dir"])
+ output.mkdir(parents=True, exist_ok=True)
+ if any(output.iterdir()):
+ raise ValueError(f"output directory is not empty: {output}")
+ report = dict(plan, status="running", started_at=_now())
+ report_path = output / "run.json"
+ # Reserve this destination before sending any requests; never overwrite an
+ # earlier invocation, including one that stopped after a partial import.
+ with report_path.open("x", encoding="utf-8") as stream:
+ json.dump(report, stream, ensure_ascii=False, indent=2)
+ stream.write("\n")
+ start = time.monotonic()
+ totals = {
+ "/add": sum(unit["add_requests"] for unit in plan["units"]),
+ "/search": sum(unit["search_requests"] for unit in plan["units"]),
+ }
+ completed = dict.fromkeys(totals, 0)
+ last_progress = dict.fromkeys(totals, start)
+
+ async def progress(response: httpx.Response) -> None:
+ route = "/" + response.request.url.path.rsplit("/", 1)[-1]
+ if route not in totals or response.status_code != 200:
+ return
+ completed[route] += 1
+ now = time.monotonic()
+ count = completed[route]
+ if count in {1, totals[route]} or count % 25 == 0 or now - last_progress[route] >= 30:
+ print(f" {route} HTTP 200 responses: {count}/{totals[route]}", flush=True)
+ last_progress[route] = now
+
+ try:
+ async with httpx.AsyncClient(
+ timeout=plan["timeout_seconds"], event_hooks={"response": [progress]}
+ ) as client:
+ response = await client.get(plan["base_url"] + "/health", timeout=15)
+ if response.status_code != 200:
+ raise RuntimeError(f"provider /health returned HTTP {response.status_code}")
+ print("Provider /health: HTTP 200", flush=True)
+ result = await runner.run_public(
+ plan["prepared"],
+ output_dir=output / "retrieval",
+ namespace=plan["namespace"],
+ units=[item["unit"] for item in plan["units"]],
+ base_url=plan["base_url"],
+ api_key=api_key,
+ add_concurrency=plan["add_concurrency"],
+ search_concurrency=plan["search_concurrency"],
+ timeout_seconds=plan["timeout_seconds"],
+ client=client,
+ )
+ report.update(status="completed", success=True, result=result)
+ except (Exception, asyncio.CancelledError) as exc:
+ report.update(
+ status="interrupted" if isinstance(exc, asyncio.CancelledError) else "failed",
+ success=False,
+ error=f"{type(exc).__name__}: {exc}".replace(api_key, "[REDACTED]"),
+ )
+ raise
+ finally:
+ report.update(
+ finished_at=_now(),
+ elapsed_seconds=round(time.monotonic() - start, 3),
+ http_200_responses=completed,
+ )
+ _write_report(report_path, report)
+ return report
+
+
+def _print_commands() -> None:
+ script = str(Path(__file__).resolve())
+ print("Local public-source presets (Add/Search only):")
+ for unit in DATASETS:
+ for mode in ("smoke", "full"):
+ print(shlex.join([sys.executable, script, mode, unit]))
+ print("\nAll available datasets:")
+ for mode in ("smoke", "full"):
+ print(shlex.join([sys.executable, script, mode, "all"]))
+ print("\nUnavailable data:")
+ for unit, reason in UNAVAILABLE.items():
+ print(f" {unit}: {reason}")
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("mode", choices=("list", "smoke", "full"))
+ parser.add_argument("datasets", nargs="*", help="dataset names, or all")
+ parser.add_argument("--base-url", default=os.getenv("AML_BASE_URL") or DEFAULT_BASE_URL)
+ parser.add_argument("--data-root", type=Path, default=DEFAULT_DATA_ROOT)
+ parser.add_argument("--output-dir", type=Path)
+ parser.add_argument("--api-key-file", type=Path, default=DEFAULT_KEY_FILE)
+ parser.add_argument(
+ "--add-concurrency",
+ type=runner._positive_int,
+ help="parallel histories/users; sessions within each history run in prepared order",
+ )
+ parser.add_argument("--search-concurrency", type=runner._positive_int)
+ parser.add_argument(
+ "--timeout-seconds", type=runner._positive_float, default=runner.DEFAULT_TIMEOUT_SECONDS
+ )
+ parser.add_argument("--dry-run", action="store_true", help="validate and show plan, no network")
+ args = parser.parse_args(argv)
+ if args.mode == "list":
+ if args.datasets:
+ parser.error("list does not accept dataset names")
+ _print_commands()
+ return 0
+
+ api_key = ""
+ try:
+ plan = build_plan(args)
+ if args.dry_run:
+ print(json.dumps(plan, ensure_ascii=False, indent=2))
+ return 0
+ api_key = _api_key(args.api_key_file.expanduser())
+ print(f"Run: {plan['run_id']} -> {plan['base_url']}", flush=True)
+ for unit in plan["units"]:
+ print(
+ f" {unit['unit']}: {unit['add_requests']} Add / {unit['search_requests']} Search",
+ flush=True,
+ )
+ print(f"Output: {plan['output_dir']}", flush=True)
+ result = asyncio.run(execute(plan, api_key))
+ except KeyboardInterrupt:
+ print(
+ "Interrupted. Remote writes may already exist; the next run uses new users.",
+ file=sys.stderr,
+ )
+ return 130
+ except (httpx.HTTPError, OSError, RuntimeError, ValueError) as exc:
+ message = f"{type(exc).__name__}: {exc}"
+ parser.exit(1, (message.replace(api_key, "[REDACTED]") if api_key else message) + "\n")
+ print(f"Completed in {result['elapsed_seconds']}s: {plan['output_dir']}/run.json")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmark/aml/eval/run_public.py b/benchmark/aml/eval/run_public.py
new file mode 100644
index 000000000..000025fe9
--- /dev/null
+++ b/benchmark/aml/eval/run_public.py
@@ -0,0 +1,512 @@
+"""Run existing prepared AML data through Add, then Search."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import hashlib
+import json
+import math
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Awaitable, Callable, Sequence, TypeVar
+
+import httpx
+
+DEFAULT_BASE_URL = "http://127.0.0.1:8088"
+DEFAULT_TIMEOUT_SECONDS = 1_000.0
+DEFAULT_ADD_CONCURRENCY = 16
+DEFAULT_SEARCH_CONCURRENCY = 16
+MAX_MESSAGES = 20
+MAX_WORDS = 2_000
+TOP_K = 100
+
+_WORD = re.compile(r"\S+")
+T = TypeVar("T")
+R = TypeVar("R")
+
+
+@dataclass(frozen=True, slots=True)
+class AddOperation:
+ user_id: str
+ session_id: str
+ request_id: str
+ messages: list[dict[str, Any]]
+
+ def payload(self) -> dict[str, Any]:
+ return {
+ "request_id": self.request_id,
+ "messages": self.messages,
+ "user_id": self.user_id,
+ "session_id": self.session_id,
+ }
+
+
+@dataclass(frozen=True, slots=True)
+class SearchCase:
+ user_id: str
+ query: str
+ options: list[str]
+ output: dict[str, Any]
+
+
+def _digest(value: Any) -> str:
+ encoded = json.dumps(
+ value,
+ ensure_ascii=False,
+ sort_keys=True,
+ separators=(",", ":"),
+ ).encode()
+ return hashlib.sha256(encoded).hexdigest()
+
+
+def _cached_digest(value: Any, cache: dict[int, str]) -> str:
+ identity = id(value)
+ digest = cache.get(identity)
+ if digest is None:
+ digest = _digest(value)
+ cache[identity] = digest
+ return digest
+
+
+def _text(value: Any, label: str) -> str:
+ if not isinstance(value, str) or not value.strip():
+ raise ValueError(f"{label} must be a non-blank string")
+ return value
+
+
+def _positive_int(value: str) -> int:
+ parsed = int(value)
+ if parsed < 1:
+ raise argparse.ArgumentTypeError("must be at least 1")
+ return parsed
+
+
+def _positive_float(value: str) -> float:
+ parsed = float(value)
+ if not math.isfinite(parsed) or parsed <= 0:
+ raise argparse.ArgumentTypeError("must be a finite number greater than 0")
+ return parsed
+
+
+def _resolve_concurrency(specific: int | None, shared: int | None, default: int) -> int:
+ if specific is not None:
+ return specific
+ if shared is not None:
+ return shared
+ return default
+
+
+def _load_cases(
+ paths: Sequence[str | Path], selected_units: Sequence[str] | None
+) -> dict[str, list[dict[str, Any]]]:
+ selected = set(selected_units) if selected_units else None
+ units: dict[str, list[dict[str, Any]]] = {}
+ seen: dict[tuple[str, str], str] = {}
+ digest_cache: dict[int, str] = {}
+
+ for raw_path in paths:
+ path = Path(raw_path)
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ cases = payload.get("cases") if isinstance(payload, dict) else None
+ if not isinstance(cases, list):
+ raise ValueError(f"{path}: expected an object with a cases array")
+ history_references = {}
+ for history in payload.get("histories", []):
+ if not isinstance(history, dict):
+ raise ValueError(f"{path}: every history must be an object")
+ key = (
+ _text(history.get("unit"), "history.unit"),
+ _text(history.get("history_key"), "history.history_key"),
+ )
+ sessions = history.get("sessions")
+ if not isinstance(sessions, list) or not sessions:
+ raise ValueError(f"{path}: history.sessions must be a non-empty array")
+ history_references[key] = sessions
+ for raw in cases:
+ if not isinstance(raw, dict):
+ raise ValueError(f"{path}: every case must be an object")
+ unit = _text(raw.get("unit"), "case.unit")
+ if selected is not None and unit not in selected:
+ continue
+ case_id = _text(raw.get("case_id"), f"{unit}.case_id")
+ normalized = dict(raw)
+ if "histories" not in normalized:
+ history_key = _text(raw.get("history_key"), f"{case_id}.history_key")
+ try:
+ normalized["histories"] = history_references[(unit, history_key)]
+ except KeyError as exc:
+ raise ValueError(f"missing history reference: {unit}/{history_key}") from exc
+ key = (unit, case_id)
+ fingerprint_case = dict(normalized)
+ histories = fingerprint_case.pop("histories")
+ fingerprint = _digest([fingerprint_case, _cached_digest(histories, digest_cache)])
+ if key in seen:
+ if seen[key] != fingerprint:
+ raise ValueError(f"conflicting duplicate case: {unit}/{case_id}")
+ continue
+ seen[key] = fingerprint
+ units.setdefault(unit, []).append(normalized)
+
+ missing = sorted((selected or set()) - units.keys())
+ if missing:
+ raise ValueError(f"selected unit has no prepared cases: {', '.join(missing)}")
+ if not units:
+ raise ValueError("no prepared cases selected")
+ return units
+
+
+def _split_content(content: str) -> list[str]:
+ words = list(_WORD.finditer(content))
+ if not words:
+ raise ValueError("message.content must be non-blank")
+ parts = []
+ for first in range(0, len(words), MAX_WORDS):
+ last = min(first + MAX_WORDS, len(words)) - 1
+ parts.append(content[words[first].start() : words[last].end()])
+ return parts
+
+
+def _message_chunks(raw_messages: Any, label: str) -> list[list[dict[str, Any]]]:
+ if not isinstance(raw_messages, list) or not raw_messages:
+ raise ValueError(f"{label}.messages must be a non-empty array")
+
+ messages: list[tuple[dict[str, Any], int]] = []
+ for index, raw in enumerate(raw_messages):
+ if not isinstance(raw, dict):
+ raise ValueError(f"{label}.messages[{index}] must be an object")
+ role = raw.get("role")
+ if role not in {"user", "assistant"}:
+ raise ValueError(f"{label}.messages[{index}].role is invalid")
+ content = _text(raw.get("content"), f"{label}.messages[{index}].content")
+ timestamp = raw.get("timestamp")
+ if timestamp is not None and (
+ isinstance(timestamp, bool) or not isinstance(timestamp, int) or timestamp < 0
+ ):
+ raise ValueError(f"{label}.messages[{index}].timestamp is invalid")
+ for part in _split_content(content):
+ message: dict[str, Any] = {"role": role, "content": part}
+ if timestamp is not None:
+ message["timestamp"] = timestamp
+ messages.append((message, len(_WORD.findall(part))))
+
+ chunks: list[list[dict[str, Any]]] = []
+ chunk: list[dict[str, Any]] = []
+ chunk_words = 0
+ for message, words in messages:
+ if chunk and (len(chunk) == MAX_MESSAGES or chunk_words + words > MAX_WORDS):
+ chunks.append(chunk)
+ chunk = []
+ chunk_words = 0
+ chunk.append(message)
+ chunk_words += words
+ if chunk:
+ chunks.append(chunk)
+ return chunks
+
+
+def _prepare_unit(
+ unit: str,
+ raw_cases: Sequence[dict[str, Any]],
+ namespace: str,
+) -> tuple[list[list[AddOperation]], list[SearchCase]]:
+ histories: dict[tuple[str, str], str] = {}
+ digest_cache: dict[int, str] = {}
+ session_groups: dict[tuple[str, str], list[AddOperation]] = {}
+ searches: list[SearchCase] = []
+
+ for raw in raw_cases:
+ case_id = _text(raw.get("case_id"), f"{unit}.case_id")
+ history_key = _text(raw.get("history_key"), f"{case_id}.history_key")
+ raw_histories = raw.get("histories")
+ if not isinstance(raw_histories, list) or not raw_histories:
+ raise ValueError(f"{case_id}.histories must be a non-empty array")
+ history_hash = _cached_digest(raw_histories, digest_cache)
+ history_identity = (history_key, history_hash)
+ user_id = histories.get(history_identity)
+
+ if user_id is None:
+ user_id = f"aml-{_digest([namespace, unit, history_key, history_hash])[:32]}"
+ histories[history_identity] = user_id
+ for session_index, raw_session in enumerate(raw_histories):
+ if not isinstance(raw_session, dict):
+ raise ValueError(f"{case_id}.histories[{session_index}] must be an object")
+ source_session_id = _text(
+ raw_session.get("session_id"),
+ f"{case_id}.histories[{session_index}].session_id",
+ )
+ session_id = "session-" + _digest([user_id, session_index, source_session_id])[:32]
+ operations = []
+ for chunk_index, messages in enumerate(
+ _message_chunks(raw_session.get("messages"), source_session_id)
+ ):
+ operations.append(
+ AddOperation(
+ user_id=user_id,
+ session_id=session_id,
+ request_id="request-" + _digest([session_id, chunk_index])[:32],
+ messages=messages,
+ )
+ )
+ session_groups[(user_id, session_id)] = operations
+
+ options = raw.get("options", [])
+ if not isinstance(options, list) or any(
+ not isinstance(option, str) or not option.strip() for option in options
+ ):
+ raise ValueError(f"{case_id}.options must contain strings")
+ extra = raw.get("extra", {})
+ if not isinstance(extra, dict):
+ raise ValueError(f"{case_id}.extra must be an object")
+ question = _text(raw.get("question"), f"{case_id}.question")
+ searches.append(
+ SearchCase(
+ user_id=user_id,
+ query=question,
+ options=list(options),
+ output={
+ **extra,
+ "id": case_id,
+ "unit": unit,
+ "case_id": case_id,
+ "question": question,
+ "gold_answer": raw.get("gold_answer"),
+ "options": options,
+ },
+ )
+ )
+
+ return list(session_groups.values()), searches
+
+
+async def _bounded_map(
+ values: Sequence[T], limit: int, function: Callable[[T], Awaitable[R]]
+) -> list[R]:
+ semaphore = asyncio.Semaphore(limit)
+
+ async def invoke(value: T) -> R:
+ async with semaphore:
+ return await function(value)
+
+ tasks = [asyncio.create_task(invoke(value)) for value in values]
+ try:
+ return list(await asyncio.gather(*tasks))
+ except BaseException:
+ # Stop queued histories and drain in-flight coroutines before the caller
+ # closes its HTTP client. Remote Add writes still require reconciliation.
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ raise
+
+
+def _headers(api_key: str | None) -> dict[str, str]:
+ return {"Authorization": f"Token {api_key}"} if api_key else {}
+
+
+async def _post(
+ client: httpx.AsyncClient,
+ base_url: str,
+ path: str,
+ headers: dict[str, str],
+ payload: dict[str, Any],
+) -> dict[str, Any]:
+ response = await client.post(base_url.rstrip("/") + path, headers=headers, json=payload)
+ if response.status_code != 200:
+ raise RuntimeError(f"{path} returned HTTP {response.status_code}: {response.text[:500]}")
+ try:
+ body = response.json()
+ except ValueError as exc:
+ raise RuntimeError(f"{path} returned non-JSON data") from exc
+ if not isinstance(body, dict):
+ raise RuntimeError(f"{path} response must be an object")
+ return body
+
+
+async def _add_session(
+ client: httpx.AsyncClient,
+ base_url: str,
+ headers: dict[str, str],
+ operations: Sequence[AddOperation],
+) -> None:
+ # One AML session may span multiple requests. Each request waits for its
+ # commit to complete before the next chunk is sent.
+ for operation in operations:
+ body = await _post(client, base_url, "/add", headers, operation.payload())
+ if body.get("success") is not True:
+ raise RuntimeError(f"/add rejected {operation.request_id}")
+
+
+async def _search(
+ client: httpx.AsyncClient,
+ base_url: str,
+ headers: dict[str, str],
+ case: SearchCase,
+) -> dict[str, Any]:
+ payload: dict[str, Any] = {
+ "query": case.query,
+ "user_id": case.user_id,
+ "top_k": TOP_K,
+ }
+ if case.options:
+ payload["options"] = case.options
+ body = await _post(client, base_url, "/search", headers, payload)
+ data = body.get("data")
+ if not isinstance(data, list):
+ raise RuntimeError("/search response must contain a data array")
+ contexts = []
+ for index, item in enumerate(data):
+ if not isinstance(item, dict) or not isinstance(item.get("id"), str):
+ raise RuntimeError(f"/search data[{index}] has no valid id")
+ content = item.get("content")
+ if not isinstance(content, str) or not content:
+ raise RuntimeError(f"/search data[{index}] has no valid content")
+ contexts.append(content)
+ return {**case.output, "retrieved_context": contexts}
+
+
+def _write_json(path: Path, value: Any) -> None:
+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+
+
+def _write_jsonl(path: Path, rows: Sequence[dict[str, Any]]) -> None:
+ path.write_text(
+ "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows),
+ encoding="utf-8",
+ )
+
+
+async def run_public(
+ prepared: Sequence[str | Path],
+ *,
+ output_dir: str | Path,
+ namespace: str,
+ units: Sequence[str] | None = None,
+ base_url: str = DEFAULT_BASE_URL,
+ api_key: str | None = None,
+ concurrency: int | None = None,
+ add_concurrency: int | None = None,
+ search_concurrency: int | None = None,
+ timeout_seconds: float = DEFAULT_TIMEOUT_SECONDS,
+ client: httpx.AsyncClient | None = None,
+) -> dict[str, Any]:
+ """Add histories concurrently, preserving session order, then Search."""
+ _text(namespace, "namespace")
+ if concurrency is not None and concurrency < 1:
+ raise ValueError("concurrency must be positive")
+ if add_concurrency is not None and add_concurrency < 1:
+ raise ValueError("add_concurrency must be positive")
+ if search_concurrency is not None and search_concurrency < 1:
+ raise ValueError("search_concurrency must be positive")
+ resolved_add_concurrency = _resolve_concurrency(
+ add_concurrency, concurrency, DEFAULT_ADD_CONCURRENCY
+ )
+ resolved_search_concurrency = _resolve_concurrency(
+ search_concurrency, concurrency, DEFAULT_SEARCH_CONCURRENCY
+ )
+ if not math.isfinite(timeout_seconds) or timeout_seconds <= 0:
+ raise ValueError("timeout_seconds must be finite and positive")
+
+ destination = Path(output_dir)
+ if destination.exists() and any(destination.iterdir()):
+ raise ValueError("output directory is not empty")
+ destination.mkdir(parents=True, exist_ok=True)
+
+ prepared_units = _load_cases(prepared, units)
+ headers = _headers(api_key)
+ own_client = client is None
+ active_client = client or httpx.AsyncClient(timeout=timeout_seconds)
+ summaries = []
+ try:
+ for unit, raw_cases in prepared_units.items():
+ session_groups, searches = _prepare_unit(unit, raw_cases, namespace)
+ add_count = sum(len(group) for group in session_groups)
+ print(f"[{unit}] Add {add_count} requests", flush=True)
+
+ # Sessions belonging to one user update shared memories. Preserve
+ # their prepared order and parallelize only across isolated users.
+ history_groups: dict[str, list[list[AddOperation]]] = {}
+ for group in session_groups:
+ history_groups.setdefault(group[0].user_id, []).append(group)
+
+ async def add_history(sessions: list[list[AddOperation]]) -> None:
+ for group in sessions:
+ await _add_session(active_client, base_url, headers, group)
+
+ await _bounded_map(list(history_groups.values()), resolved_add_concurrency, add_history)
+ print(f"[{unit}] Search {len(searches)} questions", flush=True)
+
+ async def search_case(case: SearchCase) -> dict[str, Any]:
+ return await _search(active_client, base_url, headers, case)
+
+ rows = await _bounded_map(searches, resolved_search_concurrency, search_case)
+ unit_dir = destination / unit
+ unit_dir.mkdir()
+ retrieval_path = unit_dir / "retrieval.jsonl"
+ _write_jsonl(retrieval_path, rows)
+ summaries.append(
+ {
+ "unit": unit,
+ "cases": len(rows),
+ "add_requests": add_count,
+ "retrieval": str(retrieval_path.resolve()),
+ }
+ )
+ finally:
+ if own_client:
+ await active_client.aclose()
+
+ summary = {"success": True, "units": summaries}
+ _write_json(destination / "summary.json", summary)
+ return summary
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--prepared", action="append", required=True, type=Path)
+ parser.add_argument("--unit", action="append")
+ parser.add_argument("--namespace", required=True)
+ parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
+ parser.add_argument("--output-dir", required=True, type=Path)
+ parser.add_argument(
+ "--add-concurrency",
+ type=_positive_int,
+ help=f"parallel Add histories/users (default: {DEFAULT_ADD_CONCURRENCY})",
+ )
+ parser.add_argument(
+ "--search-concurrency",
+ type=_positive_int,
+ help=f"parallel Search requests (default: {DEFAULT_SEARCH_CONCURRENCY})",
+ )
+ parser.add_argument(
+ "--concurrency",
+ type=_positive_int,
+ help="set both Add and Search concurrency; phase-specific options take precedence",
+ )
+ parser.add_argument("--timeout-seconds", type=_positive_float, default=DEFAULT_TIMEOUT_SECONDS)
+ args = parser.parse_args(argv)
+ try:
+ summary = asyncio.run(
+ run_public(
+ args.prepared,
+ output_dir=args.output_dir,
+ namespace=args.namespace,
+ units=args.unit,
+ base_url=args.base_url,
+ api_key=os.getenv("AML_API_KEY") or None,
+ concurrency=args.concurrency,
+ add_concurrency=args.add_concurrency,
+ search_concurrency=args.search_concurrency,
+ timeout_seconds=args.timeout_seconds,
+ )
+ )
+ except (httpx.HTTPError, OSError, RuntimeError, ValueError) as exc:
+ parser.exit(1, f"{type(exc).__name__}: {exc}\n")
+ print(json.dumps(summary, ensure_ascii=False, indent=2))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmark/aml/server.py b/benchmark/aml/server.py
new file mode 100644
index 000000000..537213261
--- /dev/null
+++ b/benchmark/aml/server.py
@@ -0,0 +1,409 @@
+#!/usr/bin/env python3
+"""Minimal Agent Memory Leaderboard bridge for OpenViking."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import hmac
+import math
+import os
+import time
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from typing import Any, AsyncIterator, Awaitable, Callable, Literal, TypeVar
+
+import httpx
+import uvicorn
+from fastapi import Depends, FastAPI, HTTPException, Request, status
+from openviking_sdk import AsyncHTTPClient, OpenVikingError
+from pydantic import BaseModel, ConfigDict, Field, field_validator
+
+DEFAULT_HOST = "127.0.0.1"
+DEFAULT_PORT = 8088
+DEFAULT_OPENVIKING_URL = "http://127.0.0.1:1933"
+MEMORY_ROOT_URI = "viking://~/memories"
+RETRYABLE_OPENVIKING_CODES = {
+ "ABORTED",
+ "DEADLINE_EXCEEDED",
+ "INTERNAL",
+ "RESOURCE_EXHAUSTED",
+ "UNAVAILABLE",
+ "UNKNOWN",
+}
+
+T = TypeVar("T")
+
+
+def _positive_int_env(name: str, default: int) -> int:
+ value = int(os.getenv(name, str(default)))
+ if value < 1:
+ raise ValueError(f"{name} must be at least 1")
+ return value
+
+
+def _positive_float_env(name: str, default: float) -> float:
+ value = float(os.getenv(name, str(default)))
+ if not math.isfinite(value) or value <= 0:
+ raise ValueError(f"{name} must be a finite number greater than 0")
+ return value
+
+
+@dataclass(frozen=True)
+class AMLSettings:
+ openviking_url: str = DEFAULT_OPENVIKING_URL
+ openviking_api_key: str | None = None
+ openviking_account: str | None = None
+ aml_api_key: str | None = None
+ retry_attempts: int = 3
+ retry_delay_seconds: float = 0.5
+ request_timeout_seconds: float = 120.0
+ extraction_timeout_seconds: float = 900.0
+ task_poll_seconds: float = 1.0
+
+ def __post_init__(self) -> None:
+ object.__setattr__(self, "openviking_url", self.openviking_url.rstrip("/"))
+ if not self.openviking_url.startswith(("http://", "https://")):
+ raise ValueError("OPENVIKING_URL must use http:// or https://")
+ if self.openviking_account is not None and not self.openviking_account.strip():
+ raise ValueError("OPENVIKING_ACCOUNT must not be blank")
+ if self.retry_attempts < 1:
+ raise ValueError("retry_attempts must be at least 1")
+ for name in (
+ "retry_delay_seconds",
+ "request_timeout_seconds",
+ "extraction_timeout_seconds",
+ "task_poll_seconds",
+ ):
+ value = getattr(self, name)
+ if not math.isfinite(value) or value <= 0:
+ raise ValueError(f"{name} must be a finite number greater than 0")
+
+ @classmethod
+ def from_env(cls) -> "AMLSettings":
+ return cls(
+ openviking_url=os.getenv("OPENVIKING_URL", DEFAULT_OPENVIKING_URL),
+ openviking_api_key=os.getenv("OPENVIKING_API_KEY") or None,
+ openviking_account=os.getenv("OPENVIKING_ACCOUNT") or None,
+ aml_api_key=os.getenv("AML_API_KEY") or None,
+ retry_attempts=_positive_int_env("AML_RETRY_ATTEMPTS", 3),
+ retry_delay_seconds=_positive_float_env("AML_RETRY_DELAY_SECONDS", 0.5),
+ request_timeout_seconds=_positive_float_env("AML_OV_REQUEST_TIMEOUT_SECONDS", 120.0),
+ extraction_timeout_seconds=_positive_float_env(
+ "AML_OV_EXTRACTION_TIMEOUT_SECONDS", 900.0
+ ),
+ task_poll_seconds=_positive_float_env("AML_OV_TASK_POLL_SECONDS", 1.0),
+ )
+
+
+def _not_blank(value: str) -> str:
+ if not value.strip():
+ raise ValueError("value must not be blank")
+ return value
+
+
+class AMLMessage(BaseModel):
+ model_config = ConfigDict(extra="forbid")
+
+ role: Literal["user", "assistant"]
+ content: str = Field(min_length=1)
+ timestamp: int | None = Field(default=None, ge=0, strict=True)
+
+ _content_not_blank = field_validator("content")(_not_blank)
+
+ @field_validator("timestamp")
+ @classmethod
+ def timestamp_is_representable(cls, value: int | None) -> int | None:
+ if value is not None:
+ try:
+ datetime.fromtimestamp(value / 1000, tz=timezone.utc)
+ except (OverflowError, OSError, ValueError) as exc:
+ raise ValueError(
+ "timestamp is outside the supported Unix millisecond range"
+ ) from exc
+ return value
+
+
+class AMLAddRequest(BaseModel):
+ model_config = ConfigDict(extra="forbid")
+
+ request_id: str = Field(min_length=1)
+ messages: list[AMLMessage] = Field(min_length=1, max_length=20)
+ user_id: str = Field(min_length=1)
+ session_id: str = Field(min_length=1)
+
+ _request_id_not_blank = field_validator("request_id")(_not_blank)
+ _user_id_not_blank = field_validator("user_id")(_not_blank)
+ _session_id_not_blank = field_validator("session_id")(_not_blank)
+
+
+class AMLSearchRequest(BaseModel):
+ model_config = ConfigDict(extra="forbid")
+
+ query: str = Field(min_length=1)
+ options: list[str] | None = None
+ user_id: str = Field(min_length=1)
+ top_k: int = Field(ge=1, strict=True)
+
+ _query_not_blank = field_validator("query")(_not_blank)
+ _user_id_not_blank = field_validator("user_id")(_not_blank)
+
+
+class OpenVikingBackendError(RuntimeError):
+ pass
+
+
+class OpenVikingBackend:
+ def __init__(
+ self,
+ settings: AMLSettings,
+ *,
+ client_factory: Callable[[str], AsyncHTTPClient] | None = None,
+ ) -> None:
+ self.settings = settings
+ self._client_factory = client_factory
+
+ def _new_client(self, user_id: str) -> AsyncHTTPClient:
+ if self._client_factory is not None:
+ return self._client_factory(user_id)
+ return AsyncHTTPClient(
+ url=self.settings.openviking_url,
+ api_key=self.settings.openviking_api_key,
+ account=self.settings.openviking_account,
+ user=user_id,
+ timeout=self.settings.request_timeout_seconds,
+ extra_headers={},
+ profile_enabled=False,
+ )
+
+ @asynccontextmanager
+ async def _client(self, user_id: str) -> AsyncIterator[AsyncHTTPClient]:
+ client = self._new_client(user_id)
+ await client.initialize()
+ try:
+ yield client
+ finally:
+ await client.close()
+
+ async def _retry(self, operation: str, call: Callable[[], Awaitable[T]]) -> T:
+ for attempt in range(self.settings.retry_attempts):
+ try:
+ return await call()
+ except (httpx.RequestError, OpenVikingError) as exc:
+ retryable = isinstance(exc, httpx.RequestError) or (
+ exc.code in RETRYABLE_OPENVIKING_CODES
+ )
+ if not retryable or attempt + 1 == self.settings.retry_attempts:
+ raise OpenVikingBackendError(f"OpenViking {operation} failed: {exc}") from exc
+ await asyncio.sleep(self.settings.retry_delay_seconds * (2**attempt))
+ raise AssertionError("retry loop exited unexpectedly")
+
+ async def ready(self) -> bool:
+ try:
+ async with self._client("aml-health") as client:
+ return await client.health()
+ except (httpx.RequestError, OpenVikingError, RuntimeError, ValueError):
+ return False
+
+ async def add_and_commit(
+ self, *, user_id: str, session_id: str, messages: list[dict[str, Any]]
+ ) -> None:
+ async with self._client(user_id) as client:
+ result = await self._retry(
+ "batch add",
+ lambda: client.batch_add_messages(session_id, messages),
+ )
+ if not isinstance(result, dict) or result.get("added") != len(messages):
+ raise OpenVikingBackendError("OpenViking did not add every message")
+
+ commit = await self._retry(
+ "commit",
+ lambda: client.commit_session(session_id, keep_recent_count=0),
+ )
+ if not isinstance(commit, dict):
+ raise OpenVikingBackendError("OpenViking did not accept the commit")
+ task_id = commit.get("task_id")
+ if commit.get("status") != "accepted" or not isinstance(task_id, str) or not task_id:
+ raise OpenVikingBackendError("OpenViking did not accept the commit")
+
+ deadline = time.monotonic() + self.settings.extraction_timeout_seconds
+ while True:
+ task = await self._retry("task poll", lambda: client.get_task(task_id))
+ if not isinstance(task, dict):
+ raise OpenVikingBackendError("OpenViking returned an invalid commit task")
+ task_status = task.get("status")
+ if task_status == "completed":
+ return
+ if task_status in {"failed", "cancelled"}:
+ raise OpenVikingBackendError(
+ f"OpenViking commit {task_status}: {task.get('error') or 'unknown error'}"
+ )
+ if task_status not in {"pending", "running", "cancelling"}:
+ raise OpenVikingBackendError("OpenViking returned an unknown commit status")
+ remaining = deadline - time.monotonic()
+ if remaining <= 0:
+ raise OpenVikingBackendError("OpenViking commit timed out")
+ await asyncio.sleep(min(self.settings.task_poll_seconds, remaining))
+
+ async def find(self, *, user_id: str, query: str, limit: int) -> list[dict[str, Any]]:
+ async with self._client(user_id) as client:
+ result = await self._retry(
+ "find",
+ lambda: client.find(
+ query=query,
+ target_uri=MEMORY_ROOT_URI,
+ limit=limit,
+ options={
+ "context_type": ["memory"],
+ "level": 2,
+ },
+ ),
+ )
+ memories = result.get("memories") if isinstance(result, dict) else None
+ if not isinstance(memories, list) or any(not isinstance(hit, dict) for hit in memories):
+ raise OpenVikingBackendError("OpenViking returned an invalid find result")
+ return memories
+
+
+def _format_timestamp(timestamp_ms: int) -> str:
+ return (
+ datetime.fromtimestamp(timestamp_ms / 1000, tz=timezone.utc)
+ .isoformat(timespec="milliseconds")
+ .replace("+00:00", "Z")
+ )
+
+
+class AMLAdapter:
+ def __init__(self, settings: AMLSettings, backend: OpenVikingBackend) -> None:
+ self.settings = settings
+ self.backend = backend
+
+ def authorize(self, request: Request) -> None:
+ if self.settings.aml_api_key is None:
+ return
+ candidates = [request.headers.get("x-api-key", "")]
+ scheme, separator, value = request.headers.get("authorization", "").partition(" ")
+ if separator and scheme.lower() in {"bearer", "token"}:
+ candidates.append(value)
+ if any(
+ hmac.compare_digest(candidate, self.settings.aml_api_key) for candidate in candidates
+ ):
+ return
+ raise HTTPException(
+ status_code=status.HTTP_401_UNAUTHORIZED,
+ detail="Invalid or missing Memory System Key",
+ )
+
+ async def add(self, request: AMLAddRequest) -> dict[str, Any]:
+ messages: list[dict[str, Any]] = []
+ for message in request.messages:
+ item: dict[str, Any] = {"role": message.role, "content": message.content}
+ if message.timestamp is not None:
+ item["created_at"] = _format_timestamp(message.timestamp)
+ messages.append(item)
+ try:
+ await self.backend.add_and_commit(
+ user_id=request.user_id,
+ session_id=request.session_id,
+ messages=messages,
+ )
+ except OpenVikingBackendError as exc:
+ raise HTTPException(
+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail=str(exc)
+ ) from exc
+ return {
+ "success": True,
+ "request_id": request.request_id,
+ "user_id": request.user_id,
+ "session_id": request.session_id,
+ }
+
+ async def search(self, request: AMLSearchRequest) -> dict[str, list[dict[str, Any]]]:
+ query = request.query
+ if request.options:
+ options = "\n".join(
+ f"{index}. {option}" for index, option in enumerate(request.options, start=1)
+ )
+ query = f"{query}\n\nAnswer options:\n{options}"
+ try:
+ hits = await self.backend.find(
+ user_id=request.user_id,
+ query=query,
+ limit=request.top_k,
+ )
+ except OpenVikingBackendError as exc:
+ raise HTTPException(
+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail=str(exc)
+ ) from exc
+
+ data: list[dict[str, Any]] = []
+ for hit in hits:
+ content = hit.get("content") or hit.get("abstract")
+ uri = hit.get("uri")
+ if not isinstance(uri, str) or not uri or not isinstance(content, str) or not content:
+ continue
+ data.append({"id": uri, "content": content})
+ if len(data) == request.top_k:
+ break
+ return {"data": data}
+
+
+def create_app(
+ *,
+ settings: AMLSettings | None = None,
+ backend: OpenVikingBackend | None = None,
+) -> FastAPI:
+ resolved_settings = settings or AMLSettings.from_env()
+ resolved_backend = backend or OpenVikingBackend(resolved_settings)
+ adapter = AMLAdapter(resolved_settings, resolved_backend)
+ application = FastAPI(title="OpenViking AML Adapter", version="0.1.0")
+
+ async def require_key(request: Request) -> None:
+ adapter.authorize(request)
+
+ @application.get("/health")
+ async def health() -> dict[str, str]:
+ if not await resolved_backend.ready():
+ raise HTTPException(
+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+ detail="OpenViking is not ready",
+ )
+ return {"status": "ok"}
+
+ @application.post("/add", dependencies=[Depends(require_key)])
+ async def add(request: AMLAddRequest) -> dict[str, Any]:
+ return await adapter.add(request)
+
+ @application.post("/search", dependencies=[Depends(require_key)])
+ async def search(request: AMLSearchRequest) -> dict[str, list[dict[str, Any]]]:
+ return await adapter.search(request)
+
+ application.state.aml_adapter = adapter
+ return application
+
+
+app = create_app()
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description="Start the OpenViking AML adapter")
+ parser.add_argument("--host", default=os.getenv("AML_HOST", DEFAULT_HOST))
+ parser.add_argument("--port", type=int, default=int(os.getenv("AML_PORT", str(DEFAULT_PORT))))
+ parser.add_argument("--log-level", default=os.getenv("AML_LOG_LEVEL", "info"))
+ return parser.parse_args()
+
+
+def main() -> None:
+ args = parse_args()
+ uvicorn.run(
+ create_app(),
+ host=args.host,
+ port=args.port,
+ log_level=args.log_level,
+ access_log=False,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/unit/test_aml_run_public.py b/tests/unit/test_aml_run_public.py
new file mode 100644
index 000000000..f86353aac
--- /dev/null
+++ b/tests/unit/test_aml_run_public.py
@@ -0,0 +1,111 @@
+import asyncio
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+import httpx
+
+from benchmark.aml.eval import run_public as runner
+
+
+class PublicRunnerTests(unittest.IsolatedAsyncioTestCase):
+ async def test_histories_overlap_but_each_users_sessions_and_chunks_stay_ordered(self):
+ cases = []
+ for history in range(2):
+ sessions = [
+ {
+ "session_id": f"session-{session}",
+ "messages": [
+ {"role": "user", "content": f"{history}/{session}/{turn}"}
+ for turn in range(21)
+ ],
+ }
+ for session in range(2)
+ ]
+ cases.append(
+ {
+ "unit": "beam_100k",
+ "case_id": f"case-{history}",
+ "history_key": f"history-{history}",
+ "histories": sessions,
+ "question": "What happened?",
+ }
+ )
+ cases.append({**cases[0], "case_id": "same-history-second-question"})
+ groups, searches = runner._prepare_unit("beam_100k", cases, "test-run")
+ expected = {}
+ for group in groups:
+ expected.setdefault(group[0].user_id, []).extend(op.payload() for op in group)
+ observed = {user: [] for user in expected}
+ active = set()
+ started = set()
+ both_started = asyncio.Event()
+
+ async def handle(request):
+ payload = json.loads(request.content)
+ self.assertEqual(request.headers["Authorization"], "Token test-only")
+ if request.url.path == "/add":
+ user = payload["user_id"]
+ self.assertNotIn(user, active)
+ active.add(user)
+ started.add(user)
+ if len(started) == 2:
+ both_started.set()
+ await both_started.wait()
+ await asyncio.sleep(0)
+ observed[user].append(payload)
+ active.remove(user)
+ return httpx.Response(200, json={"success": True})
+ self.assertEqual(request.url.path, "/search")
+ self.assertEqual(observed, expected)
+ return httpx.Response(200, json={"data": [{"id": "memory", "content": "event"}]})
+
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory)
+ prepared = root / "prepared.json"
+ prepared.write_text(json.dumps({"cases": cases}))
+ async with httpx.AsyncClient(transport=httpx.MockTransport(handle)) as client:
+ summary = await asyncio.wait_for(
+ runner.run_public(
+ [prepared],
+ output_dir=root / "output",
+ namespace="test-run",
+ base_url="http://test-provider",
+ api_key="test-only",
+ add_concurrency=2,
+ search_concurrency=2,
+ client=client,
+ ),
+ timeout=5,
+ )
+ self.assertEqual(observed, expected)
+ self.assertEqual(summary["units"][0]["add_requests"], 8)
+ rows = [
+ json.loads(line)
+ for line in Path(summary["units"][0]["retrieval"]).read_text().splitlines()
+ ]
+ self.assertEqual([row["id"] for row in rows], [case.output["id"] for case in searches])
+ self.assertEqual(len(started), 2)
+
+ async def test_failed_batch_drains_other_workers_before_returning(self):
+ slow_started = asyncio.Event()
+ slow_cancelled = asyncio.Event()
+
+ async def work(value):
+ if value == "fail":
+ await slow_started.wait()
+ raise RuntimeError("Add failed")
+ slow_started.set()
+ try:
+ await asyncio.Event().wait()
+ finally:
+ slow_cancelled.set()
+
+ with self.assertRaisesRegex(RuntimeError, "Add failed"):
+ await asyncio.wait_for(runner._bounded_map(["fail", "slow"], 2, work), timeout=5)
+ self.assertTrue(slow_cancelled.is_set())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit/test_aml_server.py b/tests/unit/test_aml_server.py
new file mode 100644
index 000000000..7d25a73d1
--- /dev/null
+++ b/tests/unit/test_aml_server.py
@@ -0,0 +1,112 @@
+"""Competition API contract through the real OpenViking SDK request builder."""
+
+import asyncio
+import json
+
+import httpx
+import pytest
+
+from benchmark.aml.server import AMLSettings, create_app
+
+
+def adapter_client(monkeypatch, tmp_path):
+ monkeypatch.setenv("OPENVIKING_CLI_CONFIG_FILE", str(tmp_path / "missing.conf"))
+ client_type = httpx.AsyncClient
+
+ def build(handler):
+ def sdk_client(**kwargs):
+ return client_type(**kwargs, transport=httpx.MockTransport(handler))
+
+ monkeypatch.setattr("openviking_sdk.client.httpx.AsyncClient", sdk_client)
+ app = create_app(
+ settings=AMLSettings(
+ openviking_url="http://ov.test",
+ openviking_account="competition",
+ openviking_api_key="test-ov-key",
+ aml_api_key="test-aml-key",
+ retry_delay_seconds=0.001,
+ )
+ )
+ return client_type(
+ transport=httpx.ASGITransport(app=app),
+ base_url="http://adapter.test",
+ headers={"Authorization": "Token test-aml-key"},
+ )
+
+ return build
+
+
+async def test_search_uses_native_find_and_preserves_user_scope_and_aml_response(adapter_client):
+ seen = []
+
+ async def handle(request):
+ assert request.method == "POST"
+ assert request.url.path == "/api/v1/search/find"
+ assert request.headers["X-OpenViking-Account"] == "competition"
+ assert request.headers["X-API-Key"] == "test-ov-key"
+ user = request.headers["X-OpenViking-User"]
+ body = json.loads(request.content)
+ assert body["target_uri"] == "viking://~/memories"
+ assert body["context_type"] == ["memory"]
+ assert body["level"] == 2
+ assert body["limit"] == 2
+ assert "mode" not in body
+ assert not body.get("session_id")
+ assert not body.get("read_content")
+ seen.append((user, body["query"]))
+ await asyncio.sleep(0)
+ return httpx.Response(
+ 200,
+ json={
+ "status": "ok",
+ "result": {
+ "memories": [
+ {"uri": f"viking://user/{user}/memories/a", "content": "body"},
+ {"uri": f"viking://user/{user}/memories/b", "abstract": "summary"},
+ {"uri": f"viking://user/{user}/memories/c", "abstract": "extra"},
+ ],
+ "resources": [{"uri": "viking://resources/other", "content": "other"}],
+ "skills": [],
+ },
+ },
+ )
+
+ async with adapter_client(handle) as client:
+ responses = await asyncio.gather(
+ client.post("/search", json={"user_id": "alice", "query": "Where?", "top_k": 2}),
+ client.post(
+ "/search",
+ json={"user_id": "bob", "query": "Which?", "options": ["A", "B"], "top_k": 2},
+ ),
+ )
+ assert sorted(seen) == [("alice", "Where?"), ("bob", "Which?\n\nAnswer options:\n1. A\n2. B")]
+ for user, response in zip(("alice", "bob"), responses, strict=True):
+ assert response.status_code == 200
+ assert response.json() == {
+ "data": [
+ {"id": f"viking://user/{user}/memories/a", "content": "body"},
+ {"id": f"viking://user/{user}/memories/b", "content": "summary"},
+ ]
+ }
+
+
+async def test_search_retries_transient_native_find_failure(adapter_client):
+ requests = []
+
+ def handle(request):
+ requests.append(request)
+ if len(requests) == 1:
+ raise httpx.ReadTimeout("temporary failure", request=request)
+ return httpx.Response(200, json={"status": "ok", "result": {"memories": []}})
+
+ async with adapter_client(handle) as client:
+ response = await client.post(
+ "/search", json={"user_id": "alice", "query": "Where?", "top_k": 2}
+ )
+ assert response.status_code == 200
+ assert response.json() == {"data": []}
+ assert len(requests) == 2
+ assert all(request.url.path == "/api/v1/search/find" for request in requests)
+ assert requests[0].content == requests[1].content
+ assert requests[0].headers == requests[1].headers
diff --git a/tests/unit/test_personamem_evaluate.py b/tests/unit/test_personamem_evaluate.py
new file mode 100644
index 000000000..162b64676
--- /dev/null
+++ b/tests/unit/test_personamem_evaluate.py
@@ -0,0 +1,184 @@
+import copy
+import json
+import os
+import re
+import tempfile
+import threading
+import unittest
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from unittest.mock import patch
+
+from benchmark.aml.eval import evaluate as evaluator
+
+
+def retrieval_row(unit="personamem_v2_32k", question="QUESTION_CORRECT"):
+ return {
+ "id": question,
+ "unit": unit,
+ "persona_id": "7",
+ "question": question,
+ "user_query": {"role": "user", "content": question},
+ "retrieved_context": ["The user enjoys hiking."],
+ "chat_history": [{"role": "user", "content": "UNRETRIEVED_HISTORY"}],
+ "preference": "LABEL_ONLY_PREFERENCE",
+ "correct_answer": "Take a hiking trip.",
+ "gold_answer": "Take a hiking trip.",
+ "incorrect_answers": ["Go shopping.", "Watch television.", "Play a video game."],
+ "answer_mode": "generative",
+ "options": [],
+ }
+
+
+class PersonaMemEvaluateTests(unittest.TestCase):
+ def test_mode_override_uses_only_retrieved_memory_and_preserves_input(self):
+ row = retrieval_row()
+ original = copy.deepcopy(row)
+ adapted, mode = evaluator._adapt_rows([row], "personamem_v2", persona_mode="mcq")
+ self.assertEqual(mode, "mcq")
+ self.assertEqual(adapted[0]["answer_mode"], "mcq")
+ self.assertEqual(
+ adapted[0]["chat_history"],
+ [{"role": "system", "content": "[Retrieved memory 1]\nThe user enjoys hiking."}],
+ )
+ self.assertEqual(row, original)
+ _, default_mode = evaluator._adapt_rows([row], "personamem_v2")
+ self.assertEqual(default_mode, "mcq")
+ _, explicit_mode = evaluator._adapt_rows([row], "personamem_v2", persona_mode="generative")
+ self.assertEqual(explicit_mode, "generative")
+ generic, generic_mode = evaluator._adapt_rows([row], "generic")
+ self.assertIsNone(generic_mode)
+ self.assertEqual(generic, [row])
+ for size in ("32k", "128k"):
+ unit = f"personamem_v2_{size}"
+ self.assertEqual(evaluator._resolve_unit([retrieval_row(unit)], None), unit)
+ adapted, _ = evaluator._adapt_rows(
+ [{**row, "retrieved_context": []}], "personamem_v2", persona_mode="mcq"
+ )
+ self.assertEqual(
+ adapted[0]["chat_history"][0]["content"], "No relevant memories were retrieved."
+ )
+
+ def test_invalid_options_or_unrelated_mode_override_fail_before_models(self):
+ for incorrect in (
+ [],
+ ["only one"],
+ ["a", "a", "b"],
+ ["a", "b", " "],
+ ["a", "b", "Take a hiking trip."],
+ ["a", "b", 123],
+ "not JSON",
+ ):
+ with self.subTest(incorrect=incorrect), self.assertRaises(ValueError):
+ evaluator._adapt_rows(
+ [{**retrieval_row(), "incorrect_answers": incorrect}],
+ "personamem_v2",
+ persona_mode="mcq",
+ )
+ with self.assertRaisesRegex(ValueError, "only supported for PersonaMem"):
+ evaluator._adapt_rows([retrieval_row()], "generic", persona_mode="mcq")
+ with self.assertRaisesRegex(ValueError, "unsupported PersonaMem mode"):
+ evaluator._adapt_rows([retrieval_row()], "personamem_v2", persona_mode="invalid")
+
+ def test_public_mcq_pipeline_with_local_http_and_no_judge_credentials(self):
+ repo = (
+ Path(__file__).resolve().parents[2] / "result/aml-public-eval/agent-memory-leaderboard"
+ )
+ if not (repo / "data/personamem/pipeline_v2.py").is_file():
+ self.skipTest("pinned public AML checkout is required for the integration check")
+ requests = []
+
+ class Handler(BaseHTTPRequestHandler):
+ def do_POST(self):
+ payload = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
+ requests.append((self.path, payload))
+ messages = payload["messages"]
+ options = dict(re.findall(r"^([A-D])\. (.+)$", messages[-1]["content"], re.M))
+ question = messages[1]["content"]
+ if "QUESTION_INVALID" in question:
+ answer = "I cannot select an option."
+ else:
+ target = (
+ "Take a hiking trip." if "QUESTION_CORRECT" in question else "Go shopping."
+ )
+ letter = next(key for key, value in options.items() if value == target)
+ answer = f"Final Answer: {letter}"
+ body = json.dumps({"choices": [{"message": {"content": answer}}]}).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *_args):
+ pass
+
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ environment = {
+ "ANSWER_API_BASE": f"http://127.0.0.1:{server.server_port}/v1",
+ "ANSWER_API_KEY": "local-test-only",
+ "ANSWER_MODEL": "local-test-model",
+ "JUDGE_API_BASE": "",
+ "JUDGE_API_KEY": "",
+ "JUDGE_MODEL": "",
+ "PYTHONHASHSEED": "0",
+ "NO_PROXY": "127.0.0.1,localhost",
+ }
+ try:
+ with tempfile.TemporaryDirectory() as directory, patch.dict(os.environ, environment):
+ root = Path(directory)
+ previous_mapping = None
+ for size, concurrency in (("32k", 1), ("32k", 2), ("128k", 2)):
+ unit = f"personamem_v2_{size}"
+ rows = [
+ retrieval_row(unit, question)
+ for question in ("QUESTION_CORRECT", "QUESTION_WRONG", "QUESTION_INVALID")
+ ]
+ retrieval = root / f"{size}-{concurrency}.jsonl"
+ evaluator._write_jsonl(retrieval, rows)
+ original = retrieval.read_bytes()
+ summary = evaluator.evaluate(
+ retrieval,
+ root / f"output-{size}-{concurrency}",
+ aml_repo=repo,
+ answer_concurrency=concurrency,
+ eval_concurrency=concurrency,
+ )
+ self.assertEqual(retrieval.read_bytes(), original)
+ self.assertEqual(summary["persona_mode"], "mcq")
+ self.assertEqual(summary["correct_count"], 1)
+ self.assertEqual(summary["incorrect_count"], 2)
+ self.assertEqual(summary["invalid_answer_count"], 1)
+ self.assertEqual(summary["accuracy"], 1 / 3)
+ answers = evaluator._read_jsonl(Path(summary["answers"]))
+ self.assertEqual([row["id"] for row in answers], [row["id"] for row in rows])
+ mappings = [row["option_mapping"] for row in answers]
+ if previous_mapping is not None:
+ self.assertEqual(mappings, previous_mapping)
+ previous_mapping = mappings
+ judgments = evaluator._read_jsonl(Path(summary["judgments"]))
+ self.assertEqual([r["is_correct"] for r in judgments], [True, False, False])
+ self.assertTrue(
+ all(r["gold_answer"] == "Take a hiking trip." for r in judgments)
+ )
+ self.assertEqual(len(requests), 9) # Only Answer calls; scoring is local.
+ for path, payload in requests:
+ self.assertEqual(path, "/v1/chat/completions")
+ self.assertEqual(payload["model"], "local-test-model")
+ prompt = json.dumps(payload["messages"])
+ self.assertIn("The user enjoys hiking.", prompt)
+ self.assertNotIn("UNRETRIEVED_HISTORY", prompt)
+ self.assertNotIn("LABEL_ONLY_PREFERENCE", prompt)
+ self.assertNotIn("correct_answer", prompt)
+ self.assertNotIn("correct_letter", prompt)
+ self.assertEqual(prompt.count("Take a hiking trip."), 1)
+ finally:
+ server.shutdown()
+ server.server_close()
+ thread.join(timeout=5)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit/test_personamem_prepare.py b/tests/unit/test_personamem_prepare.py
new file mode 100644
index 000000000..b19882a6d
--- /dev/null
+++ b/tests/unit/test_personamem_prepare.py
@@ -0,0 +1,119 @@
+import hashlib
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+from benchmark.aml.eval.prepare_personamem import build_prepared, chat_messages, refresh_smoke
+from benchmark.aml.eval.run_public import _prepare_unit
+
+
+class PersonaMemPrepareTests(unittest.TestCase):
+ def test_system_is_dropped_and_chat_text_order_is_exact(self):
+ messages, removed, blank = chat_messages(
+ [
+ {"role": "system", "content": "EXPLICIT PERSONA"},
+ {"role": "user", "content": " Original text\n"},
+ {"role": "system", "content": "ANOTHER PROFILE"},
+ {"role": "assistant", "content": "reply"},
+ {"role": "assistant", "content": " \n"},
+ ]
+ )
+ self.assertEqual(removed, [0, 2])
+ self.assertEqual(blank, [4])
+ self.assertEqual(
+ messages,
+ [
+ {"role": "user", "content": " Original text\n"},
+ {"role": "assistant", "content": "reply"},
+ ],
+ )
+
+ def test_multimodal_or_unknown_messages_fail_in_text_mode(self):
+ for message in [
+ {"role": "user", "content": [{"type": "image"}]},
+ {"role": "tool", "content": "text"},
+ ]:
+ with self.assertRaises(ValueError):
+ chat_messages([message])
+
+ def test_csv_links_labels_and_user_isolation(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory)
+ expected = {}
+
+ def write(relative, content):
+ path = root / relative
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(content)
+ expected[relative] = hashlib.sha256(path.read_bytes()).hexdigest()
+
+ write("benchmark/text/benchmark.csv", "test fixture")
+ row = {
+ "persona_id": "7",
+ "raw_persona_file": "data/raw_data/persona7.json",
+ "user_query": repr({"role": "user", "content": "question"}),
+ "correct_answer": "GOLD ONLY",
+ "incorrect_answers": repr(["WRONG ONLY"]),
+ "preference": "ANNOTATION ONLY",
+ }
+ for size in ["32k", "128k"]:
+ relative = f"data/chat_history_{size}/exact7.json"
+ row[f"chat_history_{size}_link"] = relative
+ write(
+ relative,
+ json.dumps(
+ {
+ "metadata": {
+ "persona_id": 7,
+ "input_filename": row["raw_persona_file"],
+ "final_token_count": 100,
+ },
+ "chat_history": [
+ {"role": "system", "content": "EXPLICIT PERSONA"},
+ {"role": "user", "content": size + " original chat"},
+ ],
+ }
+ ),
+ )
+ # Filename guessing or reading raw_data must not replace this link.
+ write(f"data/chat_history_{size}/newer7.json", "invalid alternative")
+ second = {**row, "user_query": repr({"role": "user", "content": "question two"})}
+ units = [
+ build_prepared(root, [row, second], size, "revision", expected)
+ for size in ["32k", "128k"]
+ ]
+ users = []
+ for payload in units:
+ self.assertEqual(len(payload["histories"]), 1)
+ session = payload["histories"][0]["sessions"][0]
+ self.assertNotIn("GOLD ONLY", json.dumps(session))
+ self.assertNotIn("ANNOTATION ONLY", json.dumps(session))
+ cases = [{**case, "histories": [session]} for case in payload["cases"]]
+ groups, searches = _prepare_unit(payload["unit"], cases, "same-run")
+ self.assertEqual(len(groups), 1)
+ self.assertEqual(searches[0].user_id, searches[1].user_id)
+ users.append(searches[0].user_id)
+ self.assertEqual(payload["cases"][0]["gold_answer"], "GOLD ONLY")
+ self.assertEqual(payload["cases"][0]["extra"]["answer_mode"], "mcq")
+ self.assertEqual(payload["cases"][0]["options"], [])
+ self.assertNotEqual(*users)
+ unchanged = {"unit": "clbench_0_4k", "case_id": "unrelated", "histories": []}
+ smoke = {
+ "cases": [
+ unchanged,
+ {"unit": units[0]["unit"], "case_id": units[0]["cases"][0]["case_id"]},
+ ],
+ "datasets": [{"unit": units[0]["unit"]}, {"unit": "clbench_0_4k"}],
+ }
+ refreshed = refresh_smoke(smoke, units[0])
+ self.assertEqual(refreshed["cases"][0], unchanged)
+ self.assertNotIn("system", json.dumps(refreshed["cases"][1]["histories"]))
+ # A modified or wrong-sized linked file must fail before preparation.
+ (root / row["chat_history_128k_link"]).write_text("tampered")
+ with self.assertRaises(ValueError):
+ build_prepared(root, [row], "128k", "revision", expected)
+
+
+if __name__ == "__main__":
+ unittest.main()