Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(benchmark): add AML adapter and public evaluation workflows (#5276)
agent71d705cf9850
diff --git a/benchmark/aml/README.md b/benchmark/aml/README.mdnew file mode 100644index 000000000..c3667e79f--- /dev/null+++ b/benchmark/aml/README.md@@ -0,0 +1,204 @@+# Agent Memory Leaderboard adapter++This directory contains three deliberately separated stages:++1. `server.py` exposes the competition-facing AML Add/Search API;+2. `eval/run_public.py` reads the already prepared datasets under+ `result/aml_data` and produces retrieval results;+3. `eval/evaluate.py` optionally runs each dataset's public Answer/Judge pipeline+ after retrieval, including deterministic PersonaMem MCQ grading.++The deployed adapter never imports the local evaluation code. In a formal AML+run, the platform supplies the datasets and controls Answer, Judge, and score+aggregation.++## Add/Search adapter++For a local OpenViking server using `server.auth_mode: dev`:++```bash+export OPENVIKING_URL=http://127.0.0.1:1933+export AML_API_KEY='<memory-system key>'++python -m benchmark.aml.server --host 127.0.0.1 --port 8088+```++Do not set an OpenViking key or account for this local `dev` setup. The SDK+passes each AML `user_id` directly as the OpenViking user; there is no hash or+second identity mapping. Like other OpenViking SDK clients, it also reads an+existing `~/.openviking/ovcli.conf`, so that file must match the server being+used (or be disabled with `OPENVIKING_CLI_CONFIG_FILE` pointing to a missing+file).++For a remote multi-user deployment, configure the OpenViking server with+`server.auth_mode: trusted`, then set the trusted-mode root key and account:++```bash+export OPENVIKING_URL='https://openviking.example.com'+export OPENVIKING_API_KEY='<trusted-mode root key>'+export OPENVIKING_ACCOUNT=default+export AML_API_KEY='<memory-system key>'++python -m benchmark.aml.server --host 0.0.0.0 --port 8088+```++OpenViking's standard `api_key` mode cannot use one root key to switch among+dynamic AML users: in that mode a request must use the key belonging to the+target user. `AML_API_KEY` is unrelated; it only protects this adapter's+incoming `/add` and `/search` endpoints.++The adapter provides:++- `GET /health`+- `POST /add`+- `POST /search`++`/add` calls the OpenViking SDK's `batch_add_messages`, `commit_session`, and+`get_task` in that order. `/search` calls OpenViking `find`, restricted to+`viking://~/memories` and level 2 under the same AML user. Each dataset case is+isolated by its `user_id`; session IDs only separate histories inside that+user's space. Transient SDK calls are retried up to three times by+default; use `AML_RETRY_ATTEMPTS` and `AML_RETRY_DELAY_SECONDS` to adjust that+behavior. `Token`, `Bearer`, and `X-Api-Key` authentication are accepted when+`AML_API_KEY` is set.++The native `find` call uses quick retrieval without reranking, even when the+OpenViking server has a reranker configured. The adapter returns each hit's+`content` when present, otherwise its `abstract`, without an additional+full-text read.++Run these modules from a source checkout with the project's Python dependencies+installed. The standard OpenViking image and installed package do not include+`benchmark/aml`.++## Local Add/Search++The input files already exist locally and are not generated by this code:++```bash+python -m benchmark.aml.eval.run_public \+ --prepared result/aml_data/prepared/lme_locomo_script.json \+ --prepared result/aml_data/prepared/persona_cl.json \+ --prepared result/aml_data/prepared/beam.json \+ --namespace local-check-v1 \+ --base-url http://127.0.0.1:8088 \+ --output-dir result/aml-public/retrieval \+ --add-concurrency 32 --search-concurrency 16+```++The main flow stops after Add/Search and writes one `retrieval.jsonl` under+each runnable dataset directory, plus a small top-level `summary.json`. Empty+private-data declarations are ignored because they contain nothing that can be+sent to Add or Search.++The prepared files contain `history_key`, not an AML `user_id`. The local+runner therefore creates one stable AML user for each unique+`namespace + unit + history_key + history contents`: questions sharing a+history share that user, while different histories stay isolated. This is ID+creation in the local runner; `server.py` forwards the resulting ID unchanged.++Use repeated `--unit` arguments to select individual datasets. Add runs up to+16 histories (AML users) concurrently by default, while Search runs up to 16 questions+concurrently. Override them independently with `--add-concurrency` and+`--search-concurrency`; the existing `--concurrency` option remains a shorthand+baseline for both. Phase-specific options take precedence. Within each history,+sessions run in prepared order, and chunks within each session run in order.+Every Add waits for its commit to finish before the next request for that user;+Search starts after all histories finish importing. For BEAM, this preserves+batch order within a conversation while different conversations run concurrently.++For smoke/full presets, `run_ecs` accepts a provider URL through `--base-url`+or `AML_BASE_URL`; it defaults to the local adapter. It reads authentication+from `AML_API_KEY` or `--api-key-file` and creates a fresh namespace per run:++```bash+python -m benchmark.aml.eval.run_ecs list+python -m benchmark.aml.eval.run_ecs full beam_100k \+ --base-url http://127.0.0.1:8088 --add-concurrency 32 --dry-run+```++Remove `--dry-run` to import. The generic callers do not resume an interrupted+import. Reconcile the original server tasks and archives before retrying an+ambiguous Add; a new namespace starts a separate experiment.++## PersonaMem-v2 text preparation++`eval/prepare_personamem.py` follows the exact `chat_history_32k_link` and+`chat_history_128k_link` paths in `benchmark/text/benchmark.csv`. Each size is a+separate condition; their histories are not concatenated. Shared histories+are imported once per run and reused by their questions.++The preparer removes all system persona messages and blank messages, preserves+the remaining user/assistant text and order, and omits timestamps absent from+the source. It does not assemble histories from raw persona records or gold+snippets. Reference answers, preference annotations and distractors stay in+case metadata and never enter Add messages.++```bash+python -m benchmark.aml.eval.prepare_personamem \+ --source-dir result/aml_data/datasets/personamem_v2 \+ --verification result/aml_data/personamem_v2_download/verification.json \+ --output-dir result/aml_data/prepared/full+```++The downloaded source and its verification manifest must already exist. The+manifest contains `success: true`, the source `revision`, and a `files` array+with relative `path` and `sha256` for the CSV and linked chat files. Preparation+checks those hashes, writes both size-specific prepared files and backs up any+replaced outputs. Optional `--smoke-file` refreshes existing 32K smoke cases.+Preparation makes no Add/Search or model calls and covers text inputs only.++## Optional Answer/Eval++The optional evaluator consumes one completed retrieval artifact and never+repeats Add or Search. It reads the artifact's `unit` and dispatches to the+matching public AML pipeline for LongMemEval-S, LoCoMo-Refined, BEAM,+CLBench, or PersonaMem-v2:++```bash+export ANSWER_API_BASE='https://answer-endpoint/v1'+export ANSWER_API_KEY='...'+export ANSWER_MODEL='...'+export JUDGE_API_BASE='https://judge-endpoint/v1'+export JUDGE_API_KEY='...'+export JUDGE_MODEL='...'++python -m benchmark.aml.eval.evaluate \+ --retrieval result/aml-public/retrieval/longmemeval_s/retrieval.jsonl \+ --aml-repo result/aml-public-eval/agent-memory-leaderboard \+ --answer-concurrency 16 --eval-concurrency 32+```++This produces `answers.jsonl`, `judgments.jsonl`, and `summary.json` beside the+retrieval file. Answer prompts and scoring stay inside the pinned public+dataset pipeline; `evaluate.py` only selects it and adapts the generic+`retrieved_context` field where CLBench or PersonaMem expects a different input+shape. Answer and Judge each run up to 32 pipeline shards concurrently by+default; use `--answer-concurrency` and `--eval-concurrency` to tune them+independently. Outputs are merged back into retrieval order and published only+after every shard succeeds. These remain local proxy results rather than+official leaderboard scores. Model options and retries follow the selected+public pipeline; this wrapper does not provide the experiment-specific pacing+or resume logic. The pinned pipelines currently combine+an asynchronous HTTP client and a regular output file in one `async with`+statement; the wrapper corrects only that context-manager syntax in a temporary+runtime copy, leaving the verified checkout, prompts, and scoring code unchanged.++PersonaMem defaults to `--persona-mode mcq`, even for older retrieval files+marked as generative. Answer receives only retrieved memory, the question and+four shuffled choices from `correct_answer` plus `incorrect_answers`; preference+labels and the correct option marker are not added to the prompt. Search does+not receive these choices. The public pipeline maps the predicted letter to+its option text and grades it locally; invalid answers count as incorrect.+Only the three `ANSWER_*` variables are required for MCQ. Use+`--persona-mode generative` explicitly for the public narrow LLM judge instead.+Subprocesses default to `PYTHONHASHSEED=0` to keep option mappings stable across+shard counts. These are OV retrieval-based evaluations, not reproductions of+the authors' trained MemAgent implementation.++## Static checks++```bash+python -m ruff check benchmark/aml+python -m ruff format --check benchmark/aml+```diff --git a/benchmark/aml/__init__.py b/benchmark/aml/__init__.pynew file mode 100644index 000000000..ca42f6335--- /dev/null+++ b/benchmark/aml/__init__.py@@ -0,0 +1 @@+"""Agent Memory Leaderboard adapter benchmark utilities."""diff --git a/benchmark/aml/eval/__init__.py b/benchmark/aml/eval/__init__.pynew file mode 100644index 000000000..3bf85df07--- /dev/null+++ b/benchmark/aml/eval/__init__.py@@ -0,0 +1,6 @@+"""Local, public-data evaluation utilities for the AML adapter.++These helpers reproduce the public Add/Search contract and reuse AML's public+prompt renderers. They do not reproduce the private AML dataset bundle,+production evidence assembler, model deployment, or leaderboard aggregation.+"""diff --git a/benchmark/aml/eval/evaluate.py b/benchmark/aml/eval/evaluate.pynew file mode 100644index 000000000..2578a54bb--- /dev/null+++ b/benchmark/aml/eval/evaluate.py@@ -0,0 +1,597 @@+"""Optionally run the public AML Answer/Eval pipeline for one retrieval file."""++from __future__ import annotations++import argparse+import hashlib+import json+import os+import re+import subprocess+import sys+import tempfile+from concurrent.futures import ThreadPoolExecutor, as_completed+from pathlib import Path+from typing import Any, Callable, Sequence++DEFAULT_AML_COMMIT = "1b8142bfe0f20f1c5218d6b554aa0012de34e504"+DEFAULT_ANSWER_CONCURRENCY = 32+DEFAULT_EVAL_CONCURRENCY = 32++_MIXED_OUTPUT_CONTEXT = re.compile(+ r"^(\s*)async with (httpx\.AsyncClient\(.*\) as client), "+ r"(.+\.open\(.*\) as \w+):\s*$"+)++# kind controls only the retrieval-to-pipeline input adapter. Answer prompts and+# scoring remain owned by the corresponding public AML pipeline.+PIPELINES = {+ "longmemeval_s": ("data/longmemeval-s/pipeline.py", "generic"),+ "locomo_refined": ("data/locomo-refined/pipeline.py", "generic"),+ "beam_100k": ("data/beam/pipeline.py", "generic"),+ "beam_1m": ("data/beam/pipeline.py", "generic"),+ "clbench_0_4k": ("data/clbench/pipeline.py", "clbench"),+ "clbench_16_32k": ("data/clbench/pipeline.py", "clbench"),+ "personamem_v2_32k": ("data/personamem/pipeline_v2.py", "personamem_v2"),+ "personamem_v2_128k": ("data/personamem/pipeline_v2.py", "personamem_v2"),+}+++def _git(repo: Path, *args: str) -> str:+ try:+ result = subprocess.run(+ ["git", "-C", str(repo), *args],+ check=True,+ capture_output=True,+ text=True,+ timeout=10,+ )+ except (OSError, subprocess.SubprocessError) as exc:+ raise RuntimeError(f"cannot inspect AML checkout: {exc}") from exc+ return result.stdout.strip()+++def _read_jsonl(path: Path) -> list[dict[str, Any]]:+ rows = []+ seen = set()+ for line_number, line in enumerate(path.read_text(encoding="utf-8").split("\n"), start=1):+ if not line.strip():+ continue+ try:+ row = json.loads(line)+ except json.JSONDecodeError as exc:+ raise ValueError(f"{path}:{line_number}: invalid JSON") from exc+ if not isinstance(row, dict):+ raise ValueError(f"{path}:{line_number}: expected an object")+ raw_id = row.get("id", row.get("idx"))+ if raw_id is None:+ raise ValueError(f"{path}:{line_number}: expected id or idx")+ ident = str(raw_id)+ if ident in seen:+ raise ValueError(f"{path}:{line_number}: duplicate id {ident!r}")+ seen.add(ident)+ rows.append(row)+ if not rows:+ raise ValueError(f"JSONL input is empty: {path}")+ return rows+++def _write_jsonl(path: Path, rows: Sequence[dict[str, Any]]) -> None:+ path.write_text(+ "".join(json.dumps(row, ensure_ascii=True) + "\n" for row in rows),+ encoding="utf-8",+ )+++def _row_id(row: dict[str, Any]) -> str:+ raw_id = row.get("id", row.get("idx"))+ if raw_id is None:+ raise ValueError("pipeline row has no id or idx")+ return str(raw_id)+++def _pipeline_row_id(row: dict[str, Any], kind: str) -> str:+ if kind != "clbench":+ return _row_id(row)+ for key in ("idx", "id", "question_id", "task_id"):+ if row.get(key) is not None:+ return str(row[key])+ metadata = row.get("metadata")+ if isinstance(metadata, dict):+ for key in ("task_id", "id"):+ if metadata.get(key) is not None:+ return str(metadata[key])+ raise ValueError("CLBench pipeline row has no supported identifier")+++def _positive_int(value: str) -> int:+ parsed = int(value)+ if parsed < 1:+ raise argparse.ArgumentTypeError("must be at least 1")+ return parsed+++def _resolve_unit(rows: Sequence[dict[str, Any]], requested: str | None) -> str:+ declared = {str(row["unit"]) for row in rows if row.get("unit") is not None}+ if requested is not None:+ if declared and declared != {requested}:+ raise ValueError(+ f"--unit {requested!r} does not match retrieval units: {sorted(declared)}"+ )+ unit = requested+ elif len(declared) == 1:+ unit = declared.pop()+ else:+ raise ValueError("retrieval must contain exactly one unit, or pass --unit")+ if unit not in PIPELINES:+ raise ValueError(f"no public Answer/Eval pipeline configured for unit {unit!r}")+ return unit+++def _contexts(row: dict[str, Any]) -> list[str]:+ value = row.get("retrieved_context")+ if not isinstance(value, list) or any(not isinstance(item, str) for item in value):+ raise ValueError(f"record {row.get('id')!r} has invalid retrieved_context")+ return value+++def _adapt_rows(+ rows: Sequence[dict[str, Any]], kind: str, *, persona_mode: str | None = None+) -> tuple[list[dict[str, Any]], str | None]:+ if persona_mode is not None:+ if kind != "personamem_v2":+ raise ValueError("--persona-mode is only supported for PersonaMem v2")+ if persona_mode not in {"generative", "mcq"}:+ raise ValueError(f"unsupported PersonaMem mode: {persona_mode!r}")+ if kind == "generic":+ for row in rows:+ _contexts(row)+ return [dict(row) for row in rows], None++ adapted = []+ modes = set()+ for row in rows:+ item = dict(row)+ contexts = _contexts(row)+ if kind == "clbench":+ item["retrieval"] = {"selected": [{"text": content} for content in contexts]}+ elif kind == "personamem_v2":+ # Apply the default to older retrieval files as well; generation+ # requires an explicit mode choice.+ mode = persona_mode or "mcq"+ item["answer_mode"] = mode+ if mode == "mcq":+ correct = item.get("correct_answer")+ incorrect = item.get("incorrect_answers")+ if isinstance(incorrect, str):+ try:+ incorrect = json.loads(incorrect)+ except json.JSONDecodeError as exc:+ raise ValueError(+ f"PersonaMem row {_row_id(item)!r}: invalid MCQ options"+ ) from exc+ if (+ not isinstance(correct, str)+ or not correct.strip()+ or not isinstance(incorrect, list)+ or len(incorrect) != 3+ or any(+ not isinstance(option, str) or not option.strip() for option in incorrect+ )+ or len({option.strip() for option in [correct, *incorrect]}) != 4+ ):+ raise ValueError(+ f"PersonaMem row {_row_id(item)!r}: MCQ requires a correct answer "+ "and three distinct incorrect answers"+ )+ item["incorrect_answers"] = list(incorrect)+ memory_text = "\n\n".join(+ f"[Retrieved memory {index}]\n{content}"+ for index, content in enumerate(contexts, start=1)+ )+ item["chat_history"] = [+ {+ "role": "system",+ "content": memory_text or "No relevant memories were retrieved.",+ }+ ]+ modes.add(mode)+ else:+ raise AssertionError(f"unknown pipeline adapter: {kind}")+ adapted.append(item)++ if kind == "personamem_v2":+ if not modes <= {"generative", "mcq"} or len(modes) != 1:+ raise ValueError(f"PersonaMem retrieval has unsupported mixed modes: {sorted(modes)}")+ return adapted, modes.pop()+ return adapted, None+++def _verify_pipeline(+ repo: Path,+ relative_path: str,+ expected_commit: str,+) -> tuple[Path, dict[str, str]]:+ repo = repo.expanduser().resolve()+ pipeline = repo / relative_path+ if not pipeline.is_file():+ raise ValueError(f"AML pipeline not found: {pipeline}")+ actual_commit = _git(repo, "rev-parse", "HEAD")+ if actual_commit != expected_commit:+ raise ValueError(+ f"AML checkout commit mismatch: expected {expected_commit}, got {actual_commit}"+ )+ if _git(repo, "status", "--short", "--", relative_path):+ raise ValueError(f"AML pipeline has local modifications: {pipeline}")+ return pipeline, {+ "commit": actual_commit,+ "path": relative_path,+ "sha256": hashlib.sha256(pipeline.read_bytes()).hexdigest(),+ }+++def _nest_synchronous_output_contexts(source: str) -> tuple[str, int]:+ """Make the pinned pipelines' regular output files valid inside async functions."""+ lines = source.splitlines()+ result = []+ replacements = 0+ index = 0+ while index < len(lines):+ line = lines[index]+ match = _MIXED_OUTPUT_CONTEXT.match(line)+ if match is None:+ result.append(line)+ index += 1+ continue++ indent, async_manager, sync_manager = match.groups()+ result.append(f"{indent}async with {async_manager}:")+ result.append(f"{indent} with {sync_manager}:")+ replacements += 1+ index += 1+ while index < len(lines):+ body_line = lines[index]+ stripped = body_line.lstrip()+ if stripped and len(body_line) - len(stripped) <= len(indent):+ break+ result.append(f" {body_line}" if stripped else body_line)+ index += 1++ return "\n".join(result) + "\n", replacements+++def _runtime_pipeline(repo: Path, pipeline: Path, temporary: Path) -> Path:+ source = pipeline.read_text(encoding="utf-8")+ patched_source, replacements = _nest_synchronous_output_contexts(source)+ if replacements == 0:+ return pipeline+ try:+ compile(patched_source, str(pipeline), "exec")+ except SyntaxError as exc:+ raise RuntimeError(f"cannot apply AML pipeline output compatibility fix: {exc}") from exc++ runtime_root = temporary / "pipeline-runtime"+ runtime_pipeline = runtime_root / pipeline.relative_to(repo)+ runtime_pipeline.parent.mkdir(parents=True)+ runtime_pipeline.write_text(patched_source, encoding="utf-8")+ api_config = repo / "api_config.py"+ if not api_config.is_file():+ raise ValueError(f"AML API config not found: {api_config}")+ (runtime_root / "api_config.py").write_bytes(api_config.read_bytes())+ return runtime_pipeline+++def _require_model_environment(*, needs_judge: bool) -> None:+ names = ["ANSWER_API_BASE", "ANSWER_API_KEY", "ANSWER_MODEL"]+ if needs_judge:+ names.extend(["JUDGE_API_BASE", "JUDGE_API_KEY", "JUDGE_MODEL"])+ missing = [name for name in names if not os.getenv(name)]+ if missing:+ raise ValueError(f"missing model environment: {', '.join(missing)}")+++def _run(repo: Path, arguments: Sequence[str]) -> None:+ environment = os.environ.copy()+ environment.setdefault("PYTHONHASHSEED", "0")+ try:+ result = subprocess.run(+ [sys.executable, *arguments],+ cwd=repo,+ capture_output=True,+ text=True,+ env=environment,+ )+ except OSError as exc:+ raise RuntimeError(f"cannot start AML pipeline: {exc}") from exc+ if result.returncode:+ detail = (result.stderr or result.stdout).strip()[-2_000:]+ raise RuntimeError(f"AML pipeline failed with exit {result.returncode}: {detail}")+++def _partition_rows(rows: Sequence[dict[str, Any]], concurrency: int) -> list[list[dict[str, Any]]]:+ shard_count = min(len(rows), concurrency)+ if shard_count == 0:+ return []+ return [list(rows[index::shard_count]) for index in range(shard_count)]+++def _run_parallel(+ repo: Path,+ stage: str,+ commands: Sequence[Sequence[str]],+ concurrency: int,+) -> None:+ if not commands:+ return+ with ThreadPoolExecutor(max_workers=min(concurrency, len(commands))) as executor:+ futures = {+ executor.submit(_run, repo, command): index for index, command in enumerate(commands)+ }+ for future in as_completed(futures):+ try:+ future.result()+ except RuntimeError as exc:+ for pending in futures:+ pending.cancel()+ index = futures[future]+ raise RuntimeError(+ f"AML {stage} shard {index + 1}/{len(commands)} failed: {exc}"+ ) from exc+++def _merge_outputs(+ paths: Sequence[Path],+ expected_rows: Sequence[dict[str, Any]],+ stage: str,+ identify: Callable[[dict[str, Any]], str] = _row_id,+) -> list[dict[str, Any]]:+ expected_ids = [identify(row) for row in expected_rows]+ if len(expected_ids) != len(set(expected_ids)):+ raise ValueError(f"AML {stage} input contains duplicate pipeline identifiers")+ merged: dict[str, dict[str, Any]] = {}+ for path in paths:+ try:+ rows = _read_jsonl(path)+ except (OSError, ValueError) as exc:+ raise RuntimeError(f"AML {stage} produced invalid output {path}: {exc}") from exc+ for row in rows:+ ident = identify(row)+ if ident in merged:+ raise RuntimeError(f"AML {stage} produced duplicate id {ident!r}")+ merged[ident] = row++ expected = set(expected_ids)+ actual = set(merged)+ if expected != actual:+ missing = sorted(expected - actual)[:10]+ unexpected = sorted(actual - expected)[:10]+ raise RuntimeError(+ f"AML {stage} output ID mismatch: missing={missing}, unexpected={unexpected}"+ )+ return [merged[ident] for ident in expected_ids]+++ShardCommand = Callable[[Path, Path, Sequence[dict[str, Any]]], Sequence[str]]+++def _run_sharded_stage(+ repo: Path,+ rows: Sequence[dict[str, Any]],+ destination: Path,+ temporary: Path,+ *,+ stage: str,+ concurrency: int,+ command: ShardCommand,+ identify: Callable[[dict[str, Any]], str] = _row_id,+) -> list[dict[str, Any]]:+ identifiers = [identify(row) for row in rows]+ if len(identifiers) != len(set(identifiers)):+ raise ValueError(f"AML {stage} input contains duplicate pipeline identifiers")+ stage_dir = temporary / stage+ stage_dir.mkdir()+ commands = []+ outputs = []+ for index, shard_rows in enumerate(_partition_rows(rows, concurrency)):+ shard_dir = stage_dir / f"{index:04d}"+ shard_dir.mkdir()+ input_path = shard_dir / "input.jsonl"+ output_path = shard_dir / "output.jsonl"+ _write_jsonl(input_path, shard_rows)+ commands.append(command(input_path, output_path, shard_rows))+ outputs.append(output_path)++ _run_parallel(repo, stage, commands, concurrency)+ merged = _merge_outputs(outputs, rows, stage, identify)+ _write_jsonl(destination, merged)+ return merged+++def evaluate(+ retrieval_path: Path,+ output_dir: Path,+ *,+ aml_repo: Path,+ unit: str | None = None,+ aml_commit: str = DEFAULT_AML_COMMIT,+ answer_concurrency: int = DEFAULT_ANSWER_CONCURRENCY,+ eval_concurrency: int = DEFAULT_EVAL_CONCURRENCY,+ persona_mode: str | None = None,+) -> dict[str, Any]:+ """Dispatch one retrieval file to its public dataset pipeline."""+ retrieval_path = retrieval_path.expanduser().resolve()+ if not retrieval_path.is_file():+ raise ValueError(f"retrieval file does not exist: {retrieval_path}")+ if answer_concurrency < 1:+ raise ValueError("answer_concurrency must be positive")+ if eval_concurrency < 1:+ raise ValueError("eval_concurrency must be positive")+ rows = _read_jsonl(retrieval_path)+ unit = _resolve_unit(rows, unit)+ relative_path, kind = PIPELINES[unit]+ repo = aml_repo.expanduser().resolve()+ pipeline, pipeline_info = _verify_pipeline(repo, relative_path, aml_commit)+ adapted_rows, persona_mode = _adapt_rows(rows, kind, persona_mode=persona_mode)++ output_dir = output_dir.expanduser().resolve()+ output_dir.mkdir(parents=True, exist_ok=True)+ answers_path = output_dir / "answers.jsonl"+ judgments_path = output_dir / "judgments.jsonl"+ summary_path = output_dir / "summary.json"+ existing = [path.name for path in (answers_path, judgments_path, summary_path) if path.exists()]+ if existing:+ raise ValueError(f"output already exists: {', '.join(existing)}")++ needs_judge = not (kind == "personamem_v2" and persona_mode == "mcq")+ _require_model_environment(needs_judge=needs_judge)++ def identify(row: dict[str, Any]) -> str:+ return _pipeline_row_id(row, kind)++ with tempfile.TemporaryDirectory(prefix=".aml-eval-", dir=output_dir) as temporary:+ temporary_path = Path(temporary)+ staged_answers_path = temporary_path / "answers.jsonl"+ staged_judgments_path = temporary_path / "judgments.jsonl"+ runtime_pipeline = _runtime_pipeline(repo, pipeline, temporary_path)++ def answer_command(+ input_path: Path,+ output_path: Path,+ _shard_rows: Sequence[dict[str, Any]],+ ) -> Sequence[str]:+ arguments = [+ str(runtime_pipeline),+ "answer",+ "--input",+ str(input_path),+ "--output",+ str(output_path),+ ]+ if kind == "personamem_v2":+ arguments.extend(["--mode", persona_mode or "generative"])+ return arguments++ answer_rows = _run_sharded_stage(+ repo,+ adapted_rows,+ staged_answers_path,+ temporary_path,+ stage="answer",+ concurrency=answer_concurrency,+ command=answer_command,+ identify=identify,+ )+ answers_by_id = {identify(row): row for row in answer_rows}++ def judge_command(+ input_path: Path,+ output_path: Path,+ shard_rows: Sequence[dict[str, Any]],+ ) -> Sequence[str]:+ shard_answers = input_path.with_name("answers.jsonl")+ _write_jsonl(shard_answers, [answers_by_id[identify(row)] for row in shard_rows])+ if kind == "personamem_v2" and persona_mode == "mcq":+ return [+ str(runtime_pipeline),+ "evaluate-mcq",+ "--answers",+ str(shard_answers),+ "--output",+ str(output_path),+ ]+ return [+ str(runtime_pipeline),+ "evaluate-narrow" if kind == "personamem_v2" else "evaluate",+ "--input",+ str(input_path),+ "--answers",+ str(shard_answers),+ "--output",+ str(output_path),+ ]++ judgment_rows = _run_sharded_stage(+ repo,+ adapted_rows,+ staged_judgments_path,+ temporary_path,+ stage="eval",+ concurrency=eval_concurrency,+ command=judge_command,+ identify=identify,+ )+ staged_answers_path.replace(answers_path)+ staged_judgments_path.replace(judgments_path)++ summary = {+ "unit": unit,+ "retrieval": str(retrieval_path),+ "pipeline": pipeline_info,+ "answer_rows": len(answer_rows),+ "evaluation_rows": len(judgment_rows),+ "answers": str(answers_path),+ "judgments": str(judgments_path),+ }+ if kind == "personamem_v2":+ summary["persona_mode"] = persona_mode+ if persona_mode == "mcq":+ correct_count = sum(row["is_correct"] is True for row in judgment_rows)+ summary.update(+ scoring_protocol="personamem-v2-public-mcq",+ correct_count=correct_count,+ incorrect_count=len(judgment_rows) - correct_count,+ invalid_answer_count=sum(row["predicted_answer"] is None for row in judgment_rows),+ accuracy=correct_count / len(judgment_rows),+ shuffle_pythonhashseed=os.environ.get("PYTHONHASHSEED", "0"),+ )+ summary_path.write_text(+ json.dumps(summary, ensure_ascii=False, indent=2) + "\n",+ encoding="utf-8",+ )+ return summary+++def main(argv: Sequence[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--retrieval", required=True, type=Path)+ parser.add_argument("--output-dir", type=Path)+ parser.add_argument("--unit", choices=sorted(PIPELINES))+ parser.add_argument("--aml-repo", required=True, type=Path)+ parser.add_argument("--aml-commit", default=DEFAULT_AML_COMMIT)+ parser.add_argument(+ "--persona-mode",+ choices=["mcq", "generative"],+ help="PersonaMem Answer/scoring mode using existing retrieval (default: mcq)",+ )+ parser.add_argument(+ "--answer-concurrency",+ type=_positive_int,+ default=DEFAULT_ANSWER_CONCURRENCY,+ help=f"parallel Answer shards (default: {DEFAULT_ANSWER_CONCURRENCY})",+ )+ parser.add_argument(+ "--eval-concurrency",+ type=_positive_int,+ default=DEFAULT_EVAL_CONCURRENCY,+ help=f"parallel Eval/Judge shards (default: {DEFAULT_EVAL_CONCURRENCY})",+ )+ args = parser.parse_args(argv)+ try:+ summary = evaluate(+ args.retrieval,+ args.output_dir or args.retrieval.parent,+ aml_repo=args.aml_repo,+ unit=args.unit,+ aml_commit=args.aml_commit,+ answer_concurrency=args.answer_concurrency,+ eval_concurrency=args.eval_concurrency,+ persona_mode=args.persona_mode,+ )+ except (OSError, RuntimeError, ValueError) as exc:+ parser.exit(1, f"{type(exc).__name__}: {exc}\n")+ print(json.dumps(summary, ensure_ascii=False, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/aml/eval/prepare_personamem.py b/benchmark/aml/eval/prepare_personamem.pynew file mode 100644index 000000000..0fac2fe4f--- /dev/null+++ b/benchmark/aml/eval/prepare_personamem.py@@ -0,0 +1,371 @@+"""Prepare PersonaMem-v2 text benchmark histories without system persona profiles.++Each size follows its exact benchmark CSV link. Raw persona records and gold+conversation snippets are never used to assemble the Add messages.+"""++from __future__ import annotations++import argparse+import ast+import csv+import hashlib+import json+import shutil+from datetime import datetime, timezone+from pathlib import Path, PurePosixPath+from typing import Any++ROOT = Path(__file__).resolve().parents[3]+DATA_ROOT = ROOT / "result/aml_data"+POLICY = "text-chat-only-drop-system-v1"+SIZES = ("32k", "128k")+BENCHMARK = "benchmark/text/benchmark.csv"+++def read_json(path: Path) -> Any:+ return json.loads(path.read_text(encoding="utf-8"))+++def dump_json(path: Path, value: Any) -> None:+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")+++def sha256(path: Path) -> str:+ return hashlib.sha256(path.read_bytes()).hexdigest()+++def source_file(root: Path, relative: str, expected: dict[str, str]) -> Path:+ parts = PurePosixPath(relative)+ if parts.is_absolute() or ".." in parts.parts or "\\" in relative:+ raise ValueError(f"Invalid source path: {relative}")+ path = root / relative+ if not path.resolve().is_relative_to(root.resolve()):+ raise ValueError(f"Source escapes dataset directory: {relative}")+ if relative not in expected or sha256(path) != expected[relative]:+ raise ValueError(f"Source differs from verified download: {relative}")+ return path+++def literal(value: str) -> Any:+ try:+ return json.loads(value)+ except json.JSONDecodeError:+ return ast.literal_eval(value)+++def chat_messages(chat: Any) -> tuple[list[dict[str, str]], list[int], list[int]]:+ if not isinstance(chat, list) or not chat:+ raise ValueError("Expected a non-empty chat_history array")+ messages = []+ removed = []+ blank = []+ for index, message in enumerate(chat):+ if not isinstance(message, dict):+ raise ValueError(f"Invalid message at source index {index}")+ role = message.get("role")+ if role == "system":+ removed.append(index)+ continue+ content = message.get("content")+ if role not in {"user", "assistant"} or not isinstance(content, str):+ raise ValueError(f"Unsupported text message at source index {index}")+ if not content.strip():+ blank.append(index)+ continue+ # Preserve text, role and array order exactly. The source has no real+ # timestamps; do not manufacture dates or infer session boundaries.+ messages.append({"role": role, "content": content})+ if not messages:+ raise ValueError("No conversation messages remain after dropping system")+ return messages, removed, blank+++def build_prepared(+ root: Path,+ rows: list[dict[str, str]],+ size: str,+ revision: str,+ expected: dict[str, str],+) -> dict[str, Any]:+ if size not in SIZES:+ raise ValueError(f"Unsupported text history size: {size}")+ unit = f"personamem_v2_{size}"+ column = f"chat_history_{size}_link"+ benchmark = source_file(root, BENCHMARK, expected)+ histories: dict[tuple[str, str], dict[str, Any]] = {}+ provenance: dict[tuple[str, str], dict[str, Any]] = {}+ cases = []+ for index, row in enumerate(rows):+ persona_id = row["persona_id"]+ relative = row[column]+ if not relative.startswith(f"data/chat_history_{size}/"):+ raise ValueError(f"Row {index}: wrong text history directory for {column}")+ identity = (persona_id, relative)+ if identity not in histories:+ path = source_file(root, relative, expected)+ source = read_json(path)+ metadata = source["metadata"]+ if str(metadata["persona_id"]) != persona_id:+ raise ValueError(f"Row {index}: history persona does not match CSV")+ if metadata.get("input_filename") != row["raw_persona_file"]:+ raise ValueError(f"Row {index}: raw persona reference does not match history")+ messages, removed, blank = chat_messages(source["chat_history"])+ history_key = f"personamem-v2-{persona_id}-{path.stem}"+ histories[identity] = {+ "unit": unit,+ "history_key": history_key,+ "sessions": [{"session_id": f"persona-{persona_id}", "messages": messages}],+ }+ provenance[identity] = {+ "source_path": str(path.resolve()),+ "source_relative_path": relative,+ "source_url": f"https://huggingface.co/datasets/bowen-upenn/PersonaMem-v2/resolve/{revision}/{relative}",+ "revision": revision,+ "sha256": expected[relative],+ "history_link_column": column,+ "raw_persona_file": row["raw_persona_file"],+ "history_scope": f"complete linked public text {size} history excluding system messages",+ "preprocessing_policy": POLICY,+ "dropped_system_message_indices": removed,+ "dropped_blank_message_indices": blank,+ "source_counts": {+ "sessions": 1,+ "messages_including_system": len(source["chat_history"]),+ "retained_messages": len(messages),+ "source_reported_tokens_including_system": metadata.get("final_token_count"),+ "source_reported_irrelevant_tokens": metadata.get("num_irrelevant_tokens", 0),+ },+ "transformations": [+ "drop system and blank messages; preserve remaining role/content/order exactly"+ ],+ "timestamp_policy": "absent in source; preserve sequence without synthetic dates",+ "aml_equivalence": "public-source proxy; private AML retrieval-to-chat assembly unavailable",+ }+ if provenance[identity]["raw_persona_file"] != row["raw_persona_file"]:+ raise ValueError(f"Row {index}: conflicting raw persona reference")+ query = literal(row["user_query"])+ incorrect = literal(row["incorrect_answers"])+ if (+ not isinstance(query, dict)+ or query.get("role") != "user"+ or not isinstance(query.get("content"), str)+ or not query["content"].strip()+ ):+ raise ValueError(f"Row {index}: invalid user_query")+ if not isinstance(incorrect, list) or any(not isinstance(a, str) for a in incorrect):+ raise ValueError(f"Row {index}: invalid incorrect_answers")+ # Evaluation labels stay on cases. They are not part of history.messages.+ extra = {+ key: row[key]+ for key in (+ "preference",+ "sensitive_info",+ "who",+ "updated",+ "prev_pref",+ "topic_query",+ "topic_preference",+ "conversation_scenario",+ "pref_type",+ )+ if key in row+ }+ extra.update(+ persona_id=persona_id,+ user_query=query,+ correct_answer=row["correct_answer"],+ incorrect_answers=incorrect,+ answer_mode="mcq",+ )+ cases.append(+ {+ "unit": unit,+ "case_id": f"personamem-v2-row-{index}-persona-{persona_id}",+ "history_key": histories[identity]["history_key"],+ "question": query["content"],+ "gold_answer": row["correct_answer"],+ "options": [],+ "extra": extra,+ "provenance": {**provenance[identity], "benchmark_row_offset": index},+ }+ )+ dataset = {+ "unit": unit,+ "profile": "full",+ "revision": revision,+ "source": {+ "path": str(benchmark.resolve()),+ "bytes": benchmark.stat().st_size,+ "sha256": expected[BENCHMARK],+ },+ "source_rows": len(rows),+ "eligible_cases": len(cases),+ "selected_cases": len(cases),+ "selection": {"method": "all-text-benchmark-rows-in-source-order"},+ "history_link_column": column,+ "referenced_histories": len(histories),+ "loaded_histories": len(histories),+ "preprocessing_policy": POLICY,+ "answer_mode": "mcq",+ "dropped_system_messages": sum(+ len(p["dropped_system_message_indices"]) for p in provenance.values()+ ),+ "dropped_blank_messages": sum(+ len(p["dropped_blank_message_indices"]) for p in provenance.values()+ ),+ "retained_messages": sum(+ p["source_counts"]["retained_messages"] for p in provenance.values()+ ),+ "timestamp_policy": "absent in source; preserve sequence without synthetic dates",+ "aml_equivalence": "public-source proxy; private AML retrieval-to-chat assembly unavailable",+ }+ return {+ "schema_version": 2,+ "profile": "full",+ "unit": unit,+ "datasets": [dataset],+ "histories": list(histories.values()),+ "cases": cases,+ }+++def refresh_smoke(smoke: dict[str, Any], prepared: dict[str, Any]) -> dict[str, Any]:+ unit = prepared["unit"]+ cases = {case["case_id"]: case for case in prepared["cases"]}+ histories = {h["history_key"]: h["sessions"] for h in prepared["histories"]}+ updated = []+ for case in smoke["cases"]:+ if case["unit"] != unit:+ updated.append(case)+ continue+ replacement = cases[case["case_id"]]+ updated.append({**replacement, "histories": histories[replacement["history_key"]]})+ metadata = {+ **prepared["datasets"][0],+ "profile": "smoke",+ "selected_cases": sum(c["unit"] == unit for c in updated),+ }+ return {+ **smoke,+ "cases": updated,+ "datasets": [metadata if d.get("unit") == unit else d for d in smoke["datasets"]],+ }+++def main() -> None:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--source-dir", type=Path, default=DATA_ROOT / "datasets/personamem_v2")+ parser.add_argument(+ "--verification", type=Path, default=DATA_ROOT / "personamem_v2_download/verification.json"+ )+ parser.add_argument("--output-dir", type=Path, default=DATA_ROOT / "prepared/full")+ parser.add_argument(+ "--smoke-file",+ type=Path,+ help="Also refresh existing 32K smoke cases, preserving other units",+ )+ args = parser.parse_args()+ root = args.source_dir.resolve()+ verification = read_json(args.verification)+ if not verification["success"]:+ raise ValueError("Source download verification did not succeed")+ expected = {entry["path"]: entry["sha256"] for entry in verification["files"]}+ benchmark = source_file(root, BENCHMARK, expected)+ with benchmark.open(newline="", encoding="utf-8") as stream:+ rows = list(csv.DictReader(stream))+ if not rows:+ raise ValueError("Text benchmark is empty")+ destination = args.output_dir.resolve()+ destination.mkdir(parents=True, exist_ok=True)+ stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")+ stage = destination.parent / f".personamem-text-{stamp}"+ stage.mkdir()+ outputs = []+ artifacts = []+ for size in SIZES:+ payload = build_prepared(root, rows, size, verification["revision"], expected)+ name = f"personamem_v2_{size}.json"+ target = destination / name+ staged = stage / name+ dump_json(staged, payload)+ outputs.append((staged, target))+ artifacts.append(+ {+ "unit": payload["unit"],+ "profile": "full",+ "cases": len(payload["cases"]),+ "histories": len(payload["histories"]),+ "bytes": staged.stat().st_size,+ "sha256": sha256(staged),+ "path": str(target),+ "dataset": payload["datasets"][0],+ }+ )+ if size == "32k" and args.smoke_file:+ smoke = refresh_smoke(read_json(args.smoke_file), payload)+ staged_smoke = stage / "persona_cl.json"+ dump_json(staged_smoke, smoke)+ outputs.append((staged_smoke, args.smoke_file.resolve()))+ print(+ json.dumps(+ {+ "size": size,+ "cases": len(payload["cases"]),+ **{+ k: payload["datasets"][0][k]+ for k in [+ "loaded_histories",+ "retained_messages",+ "dropped_system_messages",+ ]+ },+ }+ ),+ flush=True,+ )+ del payload+ manifest_path = destination / "manifest.json"+ manifest = (+ read_json(manifest_path)+ if manifest_path.exists()+ else {"schema_version": 1, "artifacts": []}+ )+ replaced = {a["unit"] for a in artifacts}+ manifest["artifacts"] = [+ a for a in manifest["artifacts"] if a["unit"] not in replaced+ ] + artifacts+ staged_manifest = stage / "manifest.json"+ dump_json(staged_manifest, manifest)+ outputs.append((staged_manifest, manifest_path))+ backup = destination.parent / "backups" / f"personamem-before-chat-only-{stamp}"+ backup.mkdir(parents=True)+ backups = []+ for _, target in outputs:+ if target.exists():+ saved = backup / target.name+ shutil.copy2(target, saved)+ backups.append({"original": str(target), "backup": str(saved), "sha256": sha256(saved)})+ dump_json(backup / "backup-manifest.json", {"files": backups})+ for staged, target in outputs:+ staged.replace(target)+ stage.rmdir()+ report = {+ "prepared_at": datetime.now(timezone.utc).isoformat(),+ "preprocessing_policy": POLICY,+ "revision": verification["revision"],+ "scope": "text benchmark 32K and 128K",+ "backup_directory": str(backup),+ "artifacts": artifacts,+ "normalizer": str(Path(__file__).resolve()),+ "normalizer_sha256": sha256(Path(__file__)),+ "ov_import_started": False,+ }+ dump_json(destination / "personamem_text_preparation.json", report)+ print(+ json.dumps({"prepared": [a["path"] for a in artifacts], "backup": str(backup)}), flush=True+ )+++if __name__ == "__main__":+ main()diff --git a/benchmark/aml/eval/run_ecs.py b/benchmark/aml/eval/run_ecs.pynew file mode 100644index 000000000..61cfa9176--- /dev/null+++ b/benchmark/aml/eval/run_ecs.py@@ -0,0 +1,287 @@+#!/usr/bin/env python3+"""Run local AML smoke/full datasets against a public Add/Search provider."""++from __future__ import annotations++import argparse+import asyncio+import json+import os+import shlex+import sys+import time+import uuid+from datetime import datetime, timezone+from pathlib import Path+from typing import Any, Sequence+from urllib.parse import urlsplit++import httpx++REPO_ROOT = Path(__file__).resolve().parents[3]+if __package__ in {None, ""}:+ sys.path.insert(0, str(REPO_ROOT))++from benchmark.aml.eval import run_public as runner # noqa: E402++DEFAULT_BASE_URL = runner.DEFAULT_BASE_URL+DEFAULT_DATA_ROOT = REPO_ROOT / "result" / "aml_data"+DEFAULT_OUTPUT_ROOT = REPO_ROOT / "result" / "aml_ecs"+DEFAULT_KEY_FILE = REPO_ROOT / ".codex" / "ops" / "aml-api-key"++# Every preset selects the named unit, even when a smoke file contains others.+DATASETS = {+ "longmemeval_s": "lme_locomo_script.json",+ "locomo_refined": "lme_locomo_script.json",+ "personamem_v2_32k": "persona_cl.json",+ "clbench_0_4k": "persona_cl.json",+ "clbench_16_32k": "persona_cl.json",+ "beam_100k": "beam.json",+ "beam_1m": "beam.json",+}+UNAVAILABLE = {+ "longmemeval_refined": "AML has not published the transformed production data",+ "angry": "ScriptMem has no published script history",+ "enemy": "ScriptMem has no published script history",+ "friends": "ScriptMem has no published script history",+ "man_earth": "ScriptMem has no published script history",+}+++def _now() -> str:+ return datetime.now(timezone.utc).isoformat()+++def _select_datasets(names: Sequence[str]) -> list[str]:+ if not names:+ raise ValueError("specify a dataset or 'all'; use 'list' to see every command")+ if "all" in names:+ if len(names) != 1:+ raise ValueError("'all' cannot be combined with individual datasets")+ return list(DATASETS)+ for name in names:+ if name in UNAVAILABLE:+ raise ValueError(f"{name} is unavailable: {UNAVAILABLE[name]}")+ if name not in DATASETS:+ raise ValueError(f"unknown dataset {name!r}; use 'list' to see supported datasets")+ return list(dict.fromkeys(names))+++def _base_url(value: str) -> str:+ parsed = urlsplit(value)+ if parsed.scheme not in {"http", "https"} or not parsed.hostname:+ raise ValueError("base URL must be an http:// or https:// provider address")+ if parsed.username or parsed.password or parsed.query or parsed.fragment:+ raise ValueError("base URL must not contain credentials, a query, or a fragment")+ if parsed.port == 0:+ raise ValueError("base URL port must be at least 1")+ return value.rstrip("/")+++def _api_key(path: Path) -> str:+ value = os.getenv("AML_API_KEY", "").strip()+ if not value and path.is_file():+ value = path.read_text(encoding="utf-8").strip()+ if not value:+ raise ValueError("set AML_API_KEY or provide --api-key-file; see benchmark/aml/README.md")+ if any(character.isspace() for character in value):+ raise ValueError("AML API key must not contain whitespace")+ return value+++def _write_report(path: Path, report: dict[str, Any]) -> None:+ temporary = path.with_suffix(".tmp")+ temporary.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")+ temporary.replace(path)+++def build_plan(args: argparse.Namespace) -> dict[str, Any]:+ units = _select_datasets(args.datasets)+ base_url = _base_url(args.base_url)+ run_id = f"{args.mode}-{datetime.now(timezone.utc):%Y%m%dT%H%M%SZ}-{uuid.uuid4().hex[:12]}"+ output = (args.output_dir or DEFAULT_OUTPUT_ROOT / run_id).expanduser().resolve()+ if output.exists() and (not output.is_dir() or any(output.iterdir())):+ raise ValueError(f"output directory must be new or empty: {output}")++ sources: dict[Path, list[str]] = {}+ for unit in units:+ relative = Path("full") / f"{unit}.json" if args.mode == "full" else Path(DATASETS[unit])+ source = (args.data_root.expanduser() / "prepared" / relative).resolve()+ if not source.is_file():+ raise FileNotFoundError(f"prepared data is missing: {source}; check --data-root")+ sources.setdefault(source, []).append(unit)++ counts = {}+ for source, selected in sources.items():+ # Load one input at a time for the offline plan, including validation of+ # history references and exactly the same message splitting as execution.+ loaded = runner._load_cases([source], selected)+ for unit, cases in loaded.items():+ groups, searches = runner._prepare_unit(unit, cases, f"ecs-{run_id}")+ counts[unit] = {+ "unit": unit,+ "prepared": str(source),+ "histories": len({case.user_id for case in searches}),+ "sessions": len(groups),+ "add_requests": sum(len(group) for group in groups),+ "search_requests": len(searches),+ }+ del groups, searches+ del loaded++ default_search_concurrency = 4 if args.mode == "smoke" else 16+ return {+ "status": "planned",+ "scope": "local public-source Add/Search; no Answer/Judge or official AML score",+ "run_id": run_id,+ "namespace": f"ecs-{run_id}",+ "mode": args.mode,+ "base_url": base_url,+ "output_dir": str(output),+ "prepared": [str(path) for path in sources],+ "units": [counts[unit] for unit in units],+ # Add holds the request open through commit extraction, including queue wait.+ "add_concurrency": args.add_concurrency or 16,+ "search_concurrency": args.search_concurrency or default_search_concurrency,+ "timeout_seconds": args.timeout_seconds,+ }+++async def execute(plan: dict[str, Any], api_key: str) -> dict[str, Any]:+ output = Path(plan["output_dir"])+ output.mkdir(parents=True, exist_ok=True)+ if any(output.iterdir()):+ raise ValueError(f"output directory is not empty: {output}")+ report = dict(plan, status="running", started_at=_now())+ report_path = output / "run.json"+ # Reserve this destination before sending any requests; never overwrite an+ # earlier invocation, including one that stopped after a partial import.+ with report_path.open("x", encoding="utf-8") as stream:+ json.dump(report, stream, ensure_ascii=False, indent=2)+ stream.write("\n")+ start = time.monotonic()+ totals = {+ "/add": sum(unit["add_requests"] for unit in plan["units"]),+ "/search": sum(unit["search_requests"] for unit in plan["units"]),+ }+ completed = dict.fromkeys(totals, 0)+ last_progress = dict.fromkeys(totals, start)++ async def progress(response: httpx.Response) -> None:+ route = "/" + response.request.url.path.rsplit("/", 1)[-1]+ if route not in totals or response.status_code != 200:+ return+ completed[route] += 1+ now = time.monotonic()+ count = completed[route]+ if count in {1, totals[route]} or count % 25 == 0 or now - last_progress[route] >= 30:+ print(f" {route} HTTP 200 responses: {count}/{totals[route]}", flush=True)+ last_progress[route] = now++ try:+ async with httpx.AsyncClient(+ timeout=plan["timeout_seconds"], event_hooks={"response": [progress]}+ ) as client:+ response = await client.get(plan["base_url"] + "/health", timeout=15)+ if response.status_code != 200:+ raise RuntimeError(f"provider /health returned HTTP {response.status_code}")+ print("Provider /health: HTTP 200", flush=True)+ result = await runner.run_public(+ plan["prepared"],+ output_dir=output / "retrieval",+ namespace=plan["namespace"],+ units=[item["unit"] for item in plan["units"]],+ base_url=plan["base_url"],+ api_key=api_key,+ add_concurrency=plan["add_concurrency"],+ search_concurrency=plan["search_concurrency"],+ timeout_seconds=plan["timeout_seconds"],+ client=client,+ )+ report.update(status="completed", success=True, result=result)+ except (Exception, asyncio.CancelledError) as exc:+ report.update(+ status="interrupted" if isinstance(exc, asyncio.CancelledError) else "failed",+ success=False,+ error=f"{type(exc).__name__}: {exc}".replace(api_key, "[REDACTED]"),+ )+ raise+ finally:+ report.update(+ finished_at=_now(),+ elapsed_seconds=round(time.monotonic() - start, 3),+ http_200_responses=completed,+ )+ _write_report(report_path, report)+ return report+++def _print_commands() -> None:+ script = str(Path(__file__).resolve())+ print("Local public-source presets (Add/Search only):")+ for unit in DATASETS:+ for mode in ("smoke", "full"):+ print(shlex.join([sys.executable, script, mode, unit]))+ print("\nAll available datasets:")+ for mode in ("smoke", "full"):+ print(shlex.join([sys.executable, script, mode, "all"]))+ print("\nUnavailable data:")+ for unit, reason in UNAVAILABLE.items():+ print(f" {unit}: {reason}")+++def main(argv: Sequence[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("mode", choices=("list", "smoke", "full"))+ parser.add_argument("datasets", nargs="*", help="dataset names, or all")+ parser.add_argument("--base-url", default=os.getenv("AML_BASE_URL") or DEFAULT_BASE_URL)+ parser.add_argument("--data-root", type=Path, default=DEFAULT_DATA_ROOT)+ parser.add_argument("--output-dir", type=Path)+ parser.add_argument("--api-key-file", type=Path, default=DEFAULT_KEY_FILE)+ parser.add_argument(+ "--add-concurrency",+ type=runner._positive_int,+ help="parallel histories/users; sessions within each history run in prepared order",+ )+ parser.add_argument("--search-concurrency", type=runner._positive_int)+ parser.add_argument(+ "--timeout-seconds", type=runner._positive_float, default=runner.DEFAULT_TIMEOUT_SECONDS+ )+ parser.add_argument("--dry-run", action="store_true", help="validate and show plan, no network")+ args = parser.parse_args(argv)+ if args.mode == "list":+ if args.datasets:+ parser.error("list does not accept dataset names")+ _print_commands()+ return 0++ api_key = ""+ try:+ plan = build_plan(args)+ if args.dry_run:+ print(json.dumps(plan, ensure_ascii=False, indent=2))+ return 0+ api_key = _api_key(args.api_key_file.expanduser())+ print(f"Run: {plan['run_id']} -> {plan['base_url']}", flush=True)+ for unit in plan["units"]:+ print(+ f" {unit['unit']}: {unit['add_requests']} Add / {unit['search_requests']} Search",+ flush=True,+ )+ print(f"Output: {plan['output_dir']}", flush=True)+ result = asyncio.run(execute(plan, api_key))+ except KeyboardInterrupt:+ print(+ "Interrupted. Remote writes may already exist; the next run uses new users.",+ file=sys.stderr,+ )+ return 130+ except (httpx.HTTPError, OSError, RuntimeError, ValueError) as exc:+ message = f"{type(exc).__name__}: {exc}"+ parser.exit(1, (message.replace(api_key, "[REDACTED]") if api_key else message) + "\n")+ print(f"Completed in {result['elapsed_seconds']}s: {plan['output_dir']}/run.json")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/aml/eval/run_public.py b/benchmark/aml/eval/run_public.pynew file mode 100644index 000000000..000025fe9--- /dev/null+++ b/benchmark/aml/eval/run_public.py@@ -0,0 +1,512 @@+"""Run existing prepared AML data through Add, then Search."""++from __future__ import annotations++import argparse+import asyncio+import hashlib+import json+import math+import os+import re+from dataclasses import dataclass+from pathlib import Path+from typing import Any, Awaitable, Callable, Sequence, TypeVar++import httpx++DEFAULT_BASE_URL = "http://127.0.0.1:8088"+DEFAULT_TIMEOUT_SECONDS = 1_000.0+DEFAULT_ADD_CONCURRENCY = 16+DEFAULT_SEARCH_CONCURRENCY = 16+MAX_MESSAGES = 20+MAX_WORDS = 2_000+TOP_K = 100++_WORD = re.compile(r"\S+")+T = TypeVar("T")+R = TypeVar("R")+++@dataclass(frozen=True, slots=True)+class AddOperation:+ user_id: str+ session_id: str+ request_id: str+ messages: list[dict[str, Any]]++ def payload(self) -> dict[str, Any]:+ return {+ "request_id": self.request_id,+ "messages": self.messages,+ "user_id": self.user_id,+ "session_id": self.session_id,+ }+++@dataclass(frozen=True, slots=True)+class SearchCase:+ user_id: str+ query: str+ options: list[str]+ output: dict[str, Any]+++def _digest(value: Any) -> str:+ encoded = json.dumps(+ value,+ ensure_ascii=False,+ sort_keys=True,+ separators=(",", ":"),+ ).encode()+ return hashlib.sha256(encoded).hexdigest()+++def _cached_digest(value: Any, cache: dict[int, str]) -> str:+ identity = id(value)+ digest = cache.get(identity)+ if digest is None:+ digest = _digest(value)+ cache[identity] = digest+ return digest+++def _text(value: Any, label: str) -> str:+ if not isinstance(value, str) or not value.strip():+ raise ValueError(f"{label} must be a non-blank string")+ return value+++def _positive_int(value: str) -> int:+ parsed = int(value)+ if parsed < 1:+ raise argparse.ArgumentTypeError("must be at least 1")+ return parsed+++def _positive_float(value: str) -> float:+ parsed = float(value)+ if not math.isfinite(parsed) or parsed <= 0:+ raise argparse.ArgumentTypeError("must be a finite number greater than 0")+ return parsed+++def _resolve_concurrency(specific: int | None, shared: int | None, default: int) -> int:+ if specific is not None:+ return specific+ if shared is not None:+ return shared+ return default+++def _load_cases(+ paths: Sequence[str | Path], selected_units: Sequence[str] | None+) -> dict[str, list[dict[str, Any]]]:+ selected = set(selected_units) if selected_units else None+ units: dict[str, list[dict[str, Any]]] = {}+ seen: dict[tuple[str, str], str] = {}+ digest_cache: dict[int, str] = {}++ for raw_path in paths:+ path = Path(raw_path)+ payload = json.loads(path.read_text(encoding="utf-8"))+ cases = payload.get("cases") if isinstance(payload, dict) else None+ if not isinstance(cases, list):+ raise ValueError(f"{path}: expected an object with a cases array")+ history_references = {}+ for history in payload.get("histories", []):+ if not isinstance(history, dict):+ raise ValueError(f"{path}: every history must be an object")+ key = (+ _text(history.get("unit"), "history.unit"),+ _text(history.get("history_key"), "history.history_key"),+ )+ sessions = history.get("sessions")+ if not isinstance(sessions, list) or not sessions:+ raise ValueError(f"{path}: history.sessions must be a non-empty array")+ history_references[key] = sessions+ for raw in cases:+ if not isinstance(raw, dict):+ raise ValueError(f"{path}: every case must be an object")+ unit = _text(raw.get("unit"), "case.unit")+ if selected is not None and unit not in selected:+ continue+ case_id = _text(raw.get("case_id"), f"{unit}.case_id")+ normalized = dict(raw)+ if "histories" not in normalized:+ history_key = _text(raw.get("history_key"), f"{case_id}.history_key")+ try:+ normalized["histories"] = history_references[(unit, history_key)]+ except KeyError as exc:+ raise ValueError(f"missing history reference: {unit}/{history_key}") from exc+ key = (unit, case_id)+ fingerprint_case = dict(normalized)+ histories = fingerprint_case.pop("histories")+ fingerprint = _digest([fingerprint_case, _cached_digest(histories, digest_cache)])+ if key in seen:+ if seen[key] != fingerprint:+ raise ValueError(f"conflicting duplicate case: {unit}/{case_id}")+ continue+ seen[key] = fingerprint+ units.setdefault(unit, []).append(normalized)++ missing = sorted((selected or set()) - units.keys())+ if missing:+ raise ValueError(f"selected unit has no prepared cases: {', '.join(missing)}")+ if not units:+ raise ValueError("no prepared cases selected")+ return units+++def _split_content(content: str) -> list[str]:+ words = list(_WORD.finditer(content))+ if not words:+ raise ValueError("message.content must be non-blank")+ parts = []+ for first in range(0, len(words), MAX_WORDS):+ last = min(first + MAX_WORDS, len(words)) - 1+ parts.append(content[words[first].start() : words[last].end()])+ return parts+++def _message_chunks(raw_messages: Any, label: str) -> list[list[dict[str, Any]]]:+ if not isinstance(raw_messages, list) or not raw_messages:+ raise ValueError(f"{label}.messages must be a non-empty array")++ messages: list[tuple[dict[str, Any], int]] = []+ for index, raw in enumerate(raw_messages):+ if not isinstance(raw, dict):+ raise ValueError(f"{label}.messages[{index}] must be an object")+ role = raw.get("role")+ if role not in {"user", "assistant"}:+ raise ValueError(f"{label}.messages[{index}].role is invalid")+ content = _text(raw.get("content"), f"{label}.messages[{index}].content")+ timestamp = raw.get("timestamp")+ if timestamp is not None and (+ isinstance(timestamp, bool) or not isinstance(timestamp, int) or timestamp < 0+ ):+ raise ValueError(f"{label}.messages[{index}].timestamp is invalid")+ for part in _split_content(content):+ message: dict[str, Any] = {"role": role, "content": part}+ if timestamp is not None:+ message["timestamp"] = timestamp+ messages.append((message, len(_WORD.findall(part))))++ chunks: list[list[dict[str, Any]]] = []+ chunk: list[dict[str, Any]] = []+ chunk_words = 0+ for message, words in messages:+ if chunk and (len(chunk) == MAX_MESSAGES or chunk_words + words > MAX_WORDS):+ chunks.append(chunk)+ chunk = []+ chunk_words = 0+ chunk.append(message)+ chunk_words += words+ if chunk:+ chunks.append(chunk)+ return chunks+++def _prepare_unit(+ unit: str,+ raw_cases: Sequence[dict[str, Any]],+ namespace: str,+) -> tuple[list[list[AddOperation]], list[SearchCase]]:+ histories: dict[tuple[str, str], str] = {}+ digest_cache: dict[int, str] = {}+ session_groups: dict[tuple[str, str], list[AddOperation]] = {}+ searches: list[SearchCase] = []++ for raw in raw_cases:+ case_id = _text(raw.get("case_id"), f"{unit}.case_id")+ history_key = _text(raw.get("history_key"), f"{case_id}.history_key")+ raw_histories = raw.get("histories")+ if not isinstance(raw_histories, list) or not raw_histories:+ raise ValueError(f"{case_id}.histories must be a non-empty array")+ history_hash = _cached_digest(raw_histories, digest_cache)+ history_identity = (history_key, history_hash)+ user_id = histories.get(history_identity)++ if user_id is None:+ user_id = f"aml-{_digest([namespace, unit, history_key, history_hash])[:32]}"+ histories[history_identity] = user_id+ for session_index, raw_session in enumerate(raw_histories):+ if not isinstance(raw_session, dict):+ raise ValueError(f"{case_id}.histories[{session_index}] must be an object")+ source_session_id = _text(+ raw_session.get("session_id"),+ f"{case_id}.histories[{session_index}].session_id",+ )+ session_id = "session-" + _digest([user_id, session_index, source_session_id])[:32]+ operations = []+ for chunk_index, messages in enumerate(+ _message_chunks(raw_session.get("messages"), source_session_id)+ ):+ operations.append(+ AddOperation(+ user_id=user_id,+ session_id=session_id,+ request_id="request-" + _digest([session_id, chunk_index])[:32],+ messages=messages,+ )+ )+ session_groups[(user_id, session_id)] = operations++ options = raw.get("options", [])+ if not isinstance(options, list) or any(+ not isinstance(option, str) or not option.strip() for option in options+ ):+ raise ValueError(f"{case_id}.options must contain strings")+ extra = raw.get("extra", {})+ if not isinstance(extra, dict):+ raise ValueError(f"{case_id}.extra must be an object")+ question = _text(raw.get("question"), f"{case_id}.question")+ searches.append(+ SearchCase(+ user_id=user_id,+ query=question,+ options=list(options),+ output={+ **extra,+ "id": case_id,+ "unit": unit,+ "case_id": case_id,+ "question": question,+ "gold_answer": raw.get("gold_answer"),+ "options": options,+ },+ )+ )++ return list(session_groups.values()), searches+++async def _bounded_map(+ values: Sequence[T], limit: int, function: Callable[[T], Awaitable[R]]+) -> list[R]:+ semaphore = asyncio.Semaphore(limit)++ async def invoke(value: T) -> R:+ async with semaphore:+ return await function(value)++ tasks = [asyncio.create_task(invoke(value)) for value in values]+ try:+ return list(await asyncio.gather(*tasks))+ except BaseException:+ # Stop queued histories and drain in-flight coroutines before the caller+ # closes its HTTP client. Remote Add writes still require reconciliation.+ for task in tasks:+ task.cancel()+ await asyncio.gather(*tasks, return_exceptions=True)+ raise+++def _headers(api_key: str | None) -> dict[str, str]:+ return {"Authorization": f"Token {api_key}"} if api_key else {}+++async def _post(+ client: httpx.AsyncClient,+ base_url: str,+ path: str,+ headers: dict[str, str],+ payload: dict[str, Any],+) -> dict[str, Any]:+ response = await client.post(base_url.rstrip("/") + path, headers=headers, json=payload)+ if response.status_code != 200:+ raise RuntimeError(f"{path} returned HTTP {response.status_code}: {response.text[:500]}")+ try:+ body = response.json()+ except ValueError as exc:+ raise RuntimeError(f"{path} returned non-JSON data") from exc+ if not isinstance(body, dict):+ raise RuntimeError(f"{path} response must be an object")+ return body+++async def _add_session(+ client: httpx.AsyncClient,+ base_url: str,+ headers: dict[str, str],+ operations: Sequence[AddOperation],+) -> None:+ # One AML session may span multiple requests. Each request waits for its+ # commit to complete before the next chunk is sent.+ for operation in operations:+ body = await _post(client, base_url, "/add", headers, operation.payload())+ if body.get("success") is not True:+ raise RuntimeError(f"/add rejected {operation.request_id}")+++async def _search(+ client: httpx.AsyncClient,+ base_url: str,+ headers: dict[str, str],+ case: SearchCase,+) -> dict[str, Any]:+ payload: dict[str, Any] = {+ "query": case.query,+ "user_id": case.user_id,+ "top_k": TOP_K,+ }+ if case.options:+ payload["options"] = case.options+ body = await _post(client, base_url, "/search", headers, payload)+ data = body.get("data")+ if not isinstance(data, list):+ raise RuntimeError("/search response must contain a data array")+ contexts = []+ for index, item in enumerate(data):+ if not isinstance(item, dict) or not isinstance(item.get("id"), str):+ raise RuntimeError(f"/search data[{index}] has no valid id")+ content = item.get("content")+ if not isinstance(content, str) or not content:+ raise RuntimeError(f"/search data[{index}] has no valid content")+ contexts.append(content)+ return {**case.output, "retrieved_context": contexts}+++def _write_json(path: Path, value: Any) -> None:+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")+++def _write_jsonl(path: Path, rows: Sequence[dict[str, Any]]) -> None:+ path.write_text(+ "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows),+ encoding="utf-8",+ )+++async def run_public(+ prepared: Sequence[str | Path],+ *,+ output_dir: str | Path,+ namespace: str,+ units: Sequence[str] | None = None,+ base_url: str = DEFAULT_BASE_URL,+ api_key: str | None = None,+ concurrency: int | None = None,+ add_concurrency: int | None = None,+ search_concurrency: int | None = None,+ timeout_seconds: float = DEFAULT_TIMEOUT_SECONDS,+ client: httpx.AsyncClient | None = None,+) -> dict[str, Any]:+ """Add histories concurrently, preserving session order, then Search."""+ _text(namespace, "namespace")+ if concurrency is not None and concurrency < 1:+ raise ValueError("concurrency must be positive")+ if add_concurrency is not None and add_concurrency < 1:+ raise ValueError("add_concurrency must be positive")+ if search_concurrency is not None and search_concurrency < 1:+ raise ValueError("search_concurrency must be positive")+ resolved_add_concurrency = _resolve_concurrency(+ add_concurrency, concurrency, DEFAULT_ADD_CONCURRENCY+ )+ resolved_search_concurrency = _resolve_concurrency(+ search_concurrency, concurrency, DEFAULT_SEARCH_CONCURRENCY+ )+ if not math.isfinite(timeout_seconds) or timeout_seconds <= 0:+ raise ValueError("timeout_seconds must be finite and positive")++ destination = Path(output_dir)+ if destination.exists() and any(destination.iterdir()):+ raise ValueError("output directory is not empty")+ destination.mkdir(parents=True, exist_ok=True)++ prepared_units = _load_cases(prepared, units)+ headers = _headers(api_key)+ own_client = client is None+ active_client = client or httpx.AsyncClient(timeout=timeout_seconds)+ summaries = []+ try:+ for unit, raw_cases in prepared_units.items():+ session_groups, searches = _prepare_unit(unit, raw_cases, namespace)+ add_count = sum(len(group) for group in session_groups)+ print(f"[{unit}] Add {add_count} requests", flush=True)++ # Sessions belonging to one user update shared memories. Preserve+ # their prepared order and parallelize only across isolated users.+ history_groups: dict[str, list[list[AddOperation]]] = {}+ for group in session_groups:+ history_groups.setdefault(group[0].user_id, []).append(group)++ async def add_history(sessions: list[list[AddOperation]]) -> None:+ for group in sessions:+ await _add_session(active_client, base_url, headers, group)++ await _bounded_map(list(history_groups.values()), resolved_add_concurrency, add_history)+ print(f"[{unit}] Search {len(searches)} questions", flush=True)++ async def search_case(case: SearchCase) -> dict[str, Any]:+ return await _search(active_client, base_url, headers, case)++ rows = await _bounded_map(searches, resolved_search_concurrency, search_case)+ unit_dir = destination / unit+ unit_dir.mkdir()+ retrieval_path = unit_dir / "retrieval.jsonl"+ _write_jsonl(retrieval_path, rows)+ summaries.append(+ {+ "unit": unit,+ "cases": len(rows),+ "add_requests": add_count,+ "retrieval": str(retrieval_path.resolve()),+ }+ )+ finally:+ if own_client:+ await active_client.aclose()++ summary = {"success": True, "units": summaries}+ _write_json(destination / "summary.json", summary)+ return summary+++def main(argv: Sequence[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--prepared", action="append", required=True, type=Path)+ parser.add_argument("--unit", action="append")+ parser.add_argument("--namespace", required=True)+ parser.add_argument("--base-url", default=DEFAULT_BASE_URL)+ parser.add_argument("--output-dir", required=True, type=Path)+ parser.add_argument(+ "--add-concurrency",+ type=_positive_int,+ help=f"parallel Add histories/users (default: {DEFAULT_ADD_CONCURRENCY})",+ )+ parser.add_argument(+ "--search-concurrency",+ type=_positive_int,+ help=f"parallel Search requests (default: {DEFAULT_SEARCH_CONCURRENCY})",+ )+ parser.add_argument(+ "--concurrency",+ type=_positive_int,+ help="set both Add and Search concurrency; phase-specific options take precedence",+ )+ parser.add_argument("--timeout-seconds", type=_positive_float, default=DEFAULT_TIMEOUT_SECONDS)+ args = parser.parse_args(argv)+ try:+ summary = asyncio.run(+ run_public(+ args.prepared,+ output_dir=args.output_dir,+ namespace=args.namespace,+ units=args.unit,+ base_url=args.base_url,+ api_key=os.getenv("AML_API_KEY") or None,+ concurrency=args.concurrency,+ add_concurrency=args.add_concurrency,+ search_concurrency=args.search_concurrency,+ timeout_seconds=args.timeout_seconds,+ )+ )+ except (httpx.HTTPError, OSError, RuntimeError, ValueError) as exc:+ parser.exit(1, f"{type(exc).__name__}: {exc}\n")+ print(json.dumps(summary, ensure_ascii=False, indent=2))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/aml/server.py b/benchmark/aml/server.pynew file mode 100644index 000000000..537213261--- /dev/null+++ b/benchmark/aml/server.py@@ -0,0 +1,409 @@+#!/usr/bin/env python3+"""Minimal Agent Memory Leaderboard bridge for OpenViking."""++from __future__ import annotations++import argparse+import asyncio+import hmac+import math+import os+import time+from contextlib import asynccontextmanager+from dataclasses import dataclass+from datetime import datetime, timezone+from typing import Any, AsyncIterator, Awaitable, Callable, Literal, TypeVar++import httpx+import uvicorn+from fastapi import Depends, FastAPI, HTTPException, Request, status+from openviking_sdk import AsyncHTTPClient, OpenVikingError+from pydantic import BaseModel, ConfigDict, Field, field_validator++DEFAULT_HOST = "127.0.0.1"+DEFAULT_PORT = 8088+DEFAULT_OPENVIKING_URL = "http://127.0.0.1:1933"+MEMORY_ROOT_URI = "viking://~/memories"+RETRYABLE_OPENVIKING_CODES = {+ "ABORTED",+ "DEADLINE_EXCEEDED",+ "INTERNAL",+ "RESOURCE_EXHAUSTED",+ "UNAVAILABLE",+ "UNKNOWN",+}++T = TypeVar("T")+++def _positive_int_env(name: str, default: int) -> int:+ value = int(os.getenv(name, str(default)))+ if value < 1:+ raise ValueError(f"{name} must be at least 1")+ return value+++def _positive_float_env(name: str, default: float) -> float:+ value = float(os.getenv(name, str(default)))+ if not math.isfinite(value) or value <= 0:+ raise ValueError(f"{name} must be a finite number greater than 0")+ return value+++@dataclass(frozen=True)+class AMLSettings:+ openviking_url: str = DEFAULT_OPENVIKING_URL+ openviking_api_key: str | None = None+ openviking_account: str | None = None+ aml_api_key: str | None = None+ retry_attempts: int = 3+ retry_delay_seconds: float = 0.5+ request_timeout_seconds: float = 120.0+ extraction_timeout_seconds: float = 900.0+ task_poll_seconds: float = 1.0++ def __post_init__(self) -> None:+ object.__setattr__(self, "openviking_url", self.openviking_url.rstrip("/"))+ if not self.openviking_url.startswith(("http://", "https://")):+ raise ValueError("OPENVIKING_URL must use http:// or https://")+ if self.openviking_account is not None and not self.openviking_account.strip():+ raise ValueError("OPENVIKING_ACCOUNT must not be blank")+ if self.retry_attempts < 1:+ raise ValueError("retry_attempts must be at least 1")+ for name in (+ "retry_delay_seconds",+ "request_timeout_seconds",+ "extraction_timeout_seconds",+ "task_poll_seconds",+ ):+ value = getattr(self, name)+ if not math.isfinite(value) or value <= 0:+ raise ValueError(f"{name} must be a finite number greater than 0")++ @classmethod+ def from_env(cls) -> "AMLSettings":+ return cls(+ openviking_url=os.getenv("OPENVIKING_URL", DEFAULT_OPENVIKING_URL),+ openviking_api_key=os.getenv("OPENVIKING_API_KEY") or None,+ openviking_account=os.getenv("OPENVIKING_ACCOUNT") or None,+ aml_api_key=os.getenv("AML_API_KEY") or None,+ retry_attempts=_positive_int_env("AML_RETRY_ATTEMPTS", 3),+ retry_delay_seconds=_positive_float_env("AML_RETRY_DELAY_SECONDS", 0.5),+ request_timeout_seconds=_positive_float_env("AML_OV_REQUEST_TIMEOUT_SECONDS", 120.0),+ extraction_timeout_seconds=_positive_float_env(+ "AML_OV_EXTRACTION_TIMEOUT_SECONDS", 900.0+ ),+ task_poll_seconds=_positive_float_env("AML_OV_TASK_POLL_SECONDS", 1.0),+ )+++def _not_blank(value: str) -> str:+ if not value.strip():+ raise ValueError("value must not be blank")+ return value+++class AMLMessage(BaseModel):+ model_config = ConfigDict(extra="forbid")++ role: Literal["user", "assistant"]+ content: str = Field(min_length=1)+ timestamp: int | None = Field(default=None, ge=0, strict=True)++ _content_not_blank = field_validator("content")(_not_blank)++ @field_validator("timestamp")+ @classmethod+ def timestamp_is_representable(cls, value: int | None) -> int | None:+ if value is not None:+ try:+ datetime.fromtimestamp(value / 1000, tz=timezone.utc)+ except (OverflowError, OSError, ValueError) as exc:+ raise ValueError(+ "timestamp is outside the supported Unix millisecond range"+ ) from exc+ return value+++class AMLAddRequest(BaseModel):+ model_config = ConfigDict(extra="forbid")++ request_id: str = Field(min_length=1)+ messages: list[AMLMessage] = Field(min_length=1, max_length=20)+ user_id: str = Field(min_length=1)+ session_id: str = Field(min_length=1)++ _request_id_not_blank = field_validator("request_id")(_not_blank)+ _user_id_not_blank = field_validator("user_id")(_not_blank)+ _session_id_not_blank = field_validator("session_id")(_not_blank)+++class AMLSearchRequest(BaseModel):+ model_config = ConfigDict(extra="forbid")++ query: str = Field(min_length=1)+ options: list[str] | None = None+ user_id: str = Field(min_length=1)+ top_k: int = Field(ge=1, strict=True)++ _query_not_blank = field_validator("query")(_not_blank)+ _user_id_not_blank = field_validator("user_id")(_not_blank)+++class OpenVikingBackendError(RuntimeError):+ pass+++class OpenVikingBackend:+ def __init__(+ self,+ settings: AMLSettings,+ *,+ client_factory: Callable[[str], AsyncHTTPClient] | None = None,+ ) -> None:+ self.settings = settings+ self._client_factory = client_factory++ def _new_client(self, user_id: str) -> AsyncHTTPClient:+ if self._client_factory is not None:+ return self._client_factory(user_id)+ return AsyncHTTPClient(+ url=self.settings.openviking_url,+ api_key=self.settings.openviking_api_key,+ account=self.settings.openviking_account,+ user=user_id,+ timeout=self.settings.request_timeout_seconds,+ extra_headers={},+ profile_enabled=False,+ )++ @asynccontextmanager+ async def _client(self, user_id: str) -> AsyncIterator[AsyncHTTPClient]:+ client = self._new_client(user_id)+ await client.initialize()+ try:+ yield client+ finally:+ await client.close()++ async def _retry(self, operation: str, call: Callable[[], Awaitable[T]]) -> T:+ for attempt in range(self.settings.retry_attempts):+ try:+ return await call()+ except (httpx.RequestError, OpenVikingError) as exc:+ retryable = isinstance(exc, httpx.RequestError) or (+ exc.code in RETRYABLE_OPENVIKING_CODES+ )+ if not retryable or attempt + 1 == self.settings.retry_attempts:+ raise OpenVikingBackendError(f"OpenViking {operation} failed: {exc}") from exc+ await asyncio.sleep(self.settings.retry_delay_seconds * (2**attempt))+ raise AssertionError("retry loop exited unexpectedly")++ async def ready(self) -> bool:+ try:+ async with self._client("aml-health") as client:+ return await client.health()+ except (httpx.RequestError, OpenVikingError, RuntimeError, ValueError):+ return False++ async def add_and_commit(+ self, *, user_id: str, session_id: str, messages: list[dict[str, Any]]+ ) -> None:+ async with self._client(user_id) as client:+ result = await self._retry(+ "batch add",+ lambda: client.batch_add_messages(session_id, messages),+ )+ if not isinstance(result, dict) or result.get("added") != len(messages):+ raise OpenVikingBackendError("OpenViking did not add every message")++ commit = await self._retry(+ "commit",+ lambda: client.commit_session(session_id, keep_recent_count=0),+ )+ if not isinstance(commit, dict):+ raise OpenVikingBackendError("OpenViking did not accept the commit")+ task_id = commit.get("task_id")+ if commit.get("status") != "accepted" or not isinstance(task_id, str) or not task_id:+ raise OpenVikingBackendError("OpenViking did not accept the commit")++ deadline = time.monotonic() + self.settings.extraction_timeout_seconds+ while True:+ task = await self._retry("task poll", lambda: client.get_task(task_id))+ if not isinstance(task, dict):+ raise OpenVikingBackendError("OpenViking returned an invalid commit task")+ task_status = task.get("status")+ if task_status == "completed":+ return+ if task_status in {"failed", "cancelled"}:+ raise OpenVikingBackendError(+ f"OpenViking commit {task_status}: {task.get('error') or 'unknown error'}"+ )+ if task_status not in {"pending", "running", "cancelling"}:+ raise OpenVikingBackendError("OpenViking returned an unknown commit status")+ remaining = deadline - time.monotonic()+ if remaining <= 0:+ raise OpenVikingBackendError("OpenViking commit timed out")+ await asyncio.sleep(min(self.settings.task_poll_seconds, remaining))++ async def find(self, *, user_id: str, query: str, limit: int) -> list[dict[str, Any]]:+ async with self._client(user_id) as client:+ result = await self._retry(+ "find",+ lambda: client.find(+ query=query,+ target_uri=MEMORY_ROOT_URI,+ limit=limit,+ options={+ "context_type": ["memory"],+ "level": 2,+ },+ ),+ )+ memories = result.get("memories") if isinstance(result, dict) else None+ if not isinstance(memories, list) or any(not isinstance(hit, dict) for hit in memories):+ raise OpenVikingBackendError("OpenViking returned an invalid find result")+ return memories+++def _format_timestamp(timestamp_ms: int) -> str:+ return (+ datetime.fromtimestamp(timestamp_ms / 1000, tz=timezone.utc)+ .isoformat(timespec="milliseconds")+ .replace("+00:00", "Z")+ )+++class AMLAdapter:+ def __init__(self, settings: AMLSettings, backend: OpenVikingBackend) -> None:+ self.settings = settings+ self.backend = backend++ def authorize(self, request: Request) -> None:+ if self.settings.aml_api_key is None:+ return+ candidates = [request.headers.get("x-api-key", "")]+ scheme, separator, value = request.headers.get("authorization", "").partition(" ")+ if separator and scheme.lower() in {"bearer", "token"}:+ candidates.append(value)+ if any(+ hmac.compare_digest(candidate, self.settings.aml_api_key) for candidate in candidates+ ):+ return+ raise HTTPException(+ status_code=status.HTTP_401_UNAUTHORIZED,+ detail="Invalid or missing Memory System Key",+ )++ async def add(self, request: AMLAddRequest) -> dict[str, Any]:+ messages: list[dict[str, Any]] = []+ for message in request.messages:+ item: dict[str, Any] = {"role": message.role, "content": message.content}+ if message.timestamp is not None:+ item["created_at"] = _format_timestamp(message.timestamp)+ messages.append(item)+ try:+ await self.backend.add_and_commit(+ user_id=request.user_id,+ session_id=request.session_id,+ messages=messages,+ )+ except OpenVikingBackendError as exc:+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail=str(exc)+ ) from exc+ return {+ "success": True,+ "request_id": request.request_id,+ "user_id": request.user_id,+ "session_id": request.session_id,+ }++ async def search(self, request: AMLSearchRequest) -> dict[str, list[dict[str, Any]]]:+ query = request.query+ if request.options:+ options = "\n".join(+ f"{index}. {option}" for index, option in enumerate(request.options, start=1)+ )+ query = f"{query}\n\nAnswer options:\n{options}"+ try:+ hits = await self.backend.find(+ user_id=request.user_id,+ query=query,+ limit=request.top_k,+ )+ except OpenVikingBackendError as exc:+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail=str(exc)+ ) from exc++ data: list[dict[str, Any]] = []+ for hit in hits:+ content = hit.get("content") or hit.get("abstract")+ uri = hit.get("uri")+ if not isinstance(uri, str) or not uri or not isinstance(content, str) or not content:+ continue+ data.append({"id": uri, "content": content})+ if len(data) == request.top_k:+ break+ return {"data": data}+++def create_app(+ *,+ settings: AMLSettings | None = None,+ backend: OpenVikingBackend | None = None,+) -> FastAPI:+ resolved_settings = settings or AMLSettings.from_env()+ resolved_backend = backend or OpenVikingBackend(resolved_settings)+ adapter = AMLAdapter(resolved_settings, resolved_backend)+ application = FastAPI(title="OpenViking AML Adapter", version="0.1.0")++ async def require_key(request: Request) -> None:+ adapter.authorize(request)++ @application.get("/health")+ async def health() -> dict[str, str]:+ if not await resolved_backend.ready():+ raise HTTPException(+ status_code=status.HTTP_503_SERVICE_UNAVAILABLE,+ detail="OpenViking is not ready",+ )+ return {"status": "ok"}++ @application.post("/add", dependencies=[Depends(require_key)])+ async def add(request: AMLAddRequest) -> dict[str, Any]:+ return await adapter.add(request)++ @application.post("/search", dependencies=[Depends(require_key)])+ async def search(request: AMLSearchRequest) -> dict[str, list[dict[str, Any]]]:+ return await adapter.search(request)++ application.state.aml_adapter = adapter+ return application+++app = create_app()+++def parse_args() -> argparse.Namespace:+ parser = argparse.ArgumentParser(description="Start the OpenViking AML adapter")+ parser.add_argument("--host", default=os.getenv("AML_HOST", DEFAULT_HOST))+ parser.add_argument("--port", type=int, default=int(os.getenv("AML_PORT", str(DEFAULT_PORT))))+ parser.add_argument("--log-level", default=os.getenv("AML_LOG_LEVEL", "info"))+ return parser.parse_args()+++def main() -> None:+ args = parse_args()+ uvicorn.run(+ create_app(),+ host=args.host,+ port=args.port,+ log_level=args.log_level,+ access_log=False,+ )+++if __name__ == "__main__":+ main()diff --git a/tests/unit/test_aml_run_public.py b/tests/unit/test_aml_run_public.pynew file mode 100644index 000000000..f86353aac--- /dev/null+++ b/tests/unit/test_aml_run_public.py@@ -0,0 +1,111 @@+import asyncio+import json+import tempfile+import unittest+from pathlib import Path++import httpx++from benchmark.aml.eval import run_public as runner+++class PublicRunnerTests(unittest.IsolatedAsyncioTestCase):+ async def test_histories_overlap_but_each_users_sessions_and_chunks_stay_ordered(self):+ cases = []+ for history in range(2):+ sessions = [+ {+ "session_id": f"session-{session}",+ "messages": [+ {"role": "user", "content": f"{history}/{session}/{turn}"}+ for turn in range(21)+ ],+ }+ for session in range(2)+ ]+ cases.append(+ {+ "unit": "beam_100k",+ "case_id": f"case-{history}",+ "history_key": f"history-{history}",+ "histories": sessions,+ "question": "What happened?",+ }+ )+ cases.append({**cases[0], "case_id": "same-history-second-question"})+ groups, searches = runner._prepare_unit("beam_100k", cases, "test-run")+ expected = {}+ for group in groups:+ expected.setdefault(group[0].user_id, []).extend(op.payload() for op in group)+ observed = {user: [] for user in expected}+ active = set()+ started = set()+ both_started = asyncio.Event()++ async def handle(request):+ payload = json.loads(request.content)+ self.assertEqual(request.headers["Authorization"], "Token test-only")+ if request.url.path == "/add":+ user = payload["user_id"]+ self.assertNotIn(user, active)+ active.add(user)+ started.add(user)+ if len(started) == 2:+ both_started.set()+ await both_started.wait()+ await asyncio.sleep(0)+ observed[user].append(payload)+ active.remove(user)+ return httpx.Response(200, json={"success": True})+ self.assertEqual(request.url.path, "/search")+ self.assertEqual(observed, expected)+ return httpx.Response(200, json={"data": [{"id": "memory", "content": "event"}]})++ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ prepared = root / "prepared.json"+ prepared.write_text(json.dumps({"cases": cases}))+ async with httpx.AsyncClient(transport=httpx.MockTransport(handle)) as client:+ summary = await asyncio.wait_for(+ runner.run_public(+ [prepared],+ output_dir=root / "output",+ namespace="test-run",+ base_url="http://test-provider",+ api_key="test-only",+ add_concurrency=2,+ search_concurrency=2,+ client=client,+ ),+ timeout=5,+ )+ self.assertEqual(observed, expected)+ self.assertEqual(summary["units"][0]["add_requests"], 8)+ rows = [+ json.loads(line)+ for line in Path(summary["units"][0]["retrieval"]).read_text().splitlines()+ ]+ self.assertEqual([row["id"] for row in rows], [case.output["id"] for case in searches])+ self.assertEqual(len(started), 2)++ async def test_failed_batch_drains_other_workers_before_returning(self):+ slow_started = asyncio.Event()+ slow_cancelled = asyncio.Event()++ async def work(value):+ if value == "fail":+ await slow_started.wait()+ raise RuntimeError("Add failed")+ slow_started.set()+ try:+ await asyncio.Event().wait()+ finally:+ slow_cancelled.set()++ with self.assertRaisesRegex(RuntimeError, "Add failed"):+ await asyncio.wait_for(runner._bounded_map(["fail", "slow"], 2, work), timeout=5)+ self.assertTrue(slow_cancelled.is_set())+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/unit/test_aml_server.py b/tests/unit/test_aml_server.pynew file mode 100644index 000000000..7d25a73d1--- /dev/null+++ b/tests/unit/test_aml_server.py@@ -0,0 +1,112 @@+"""Competition API contract through the real OpenViking SDK request builder."""++import asyncio+import json++import httpx+import pytest++from benchmark.aml.server import AMLSettings, create_app+++def adapter_client(monkeypatch, tmp_path):+ monkeypatch.setenv("OPENVIKING_CLI_CONFIG_FILE", str(tmp_path / "missing.conf"))+ client_type = httpx.AsyncClient++ def build(handler):+ def sdk_client(**kwargs):+ return client_type(**kwargs, transport=httpx.MockTransport(handler))++ monkeypatch.setattr("openviking_sdk.client.httpx.AsyncClient", sdk_client)+ app = create_app(+ settings=AMLSettings(+ openviking_url="http://ov.test",+ openviking_account="competition",+ openviking_api_key="test-ov-key",+ aml_api_key="test-aml-key",+ retry_delay_seconds=0.001,+ )+ )+ return client_type(+ transport=httpx.ASGITransport(app=app),+ base_url="http://adapter.test",+ headers={"Authorization": "Token test-aml-key"},+ )++ return build+++async def test_search_uses_native_find_and_preserves_user_scope_and_aml_response(adapter_client):+ seen = []++ async def handle(request):+ assert request.method == "POST"+ assert request.url.path == "/api/v1/search/find"+ assert request.headers["X-OpenViking-Account"] == "competition"+ assert request.headers["X-API-Key"] == "test-ov-key"+ user = request.headers["X-OpenViking-User"]+ body = json.loads(request.content)+ assert body["target_uri"] == "viking://~/memories"+ assert body["context_type"] == ["memory"]+ assert body["level"] == 2+ assert body["limit"] == 2+ assert "mode" not in body+ assert not body.get("session_id")+ assert not body.get("read_content")+ seen.append((user, body["query"]))+ await asyncio.sleep(0)+ return httpx.Response(+ 200,+ json={+ "status": "ok",+ "result": {+ "memories": [+ {"uri": f"viking://user/{user}/memories/a", "content": "body"},+ {"uri": f"viking://user/{user}/memories/b", "abstract": "summary"},+ {"uri": f"viking://user/{user}/memories/c", "abstract": "extra"},+ ],+ "resources": [{"uri": "viking://resources/other", "content": "other"}],+ "skills": [],+ },+ },+ )++ async with adapter_client(handle) as client:+ responses = await asyncio.gather(+ client.post("/search", json={"user_id": "alice", "query": "Where?", "top_k": 2}),+ client.post(+ "/search",+ json={"user_id": "bob", "query": "Which?", "options": ["A", "B"], "top_k": 2},+ ),+ )+ assert sorted(seen) == [("alice", "Where?"), ("bob", "Which?\n\nAnswer options:\n1. A\n2. B")]+ for user, response in zip(("alice", "bob"), responses, strict=True):+ assert response.status_code == 200+ assert response.json() == {+ "data": [+ {"id": f"viking://user/{user}/memories/a", "content": "body"},+ {"id": f"viking://user/{user}/memories/b", "content": "summary"},+ ]+ }+++async def test_search_retries_transient_native_find_failure(adapter_client):+ requests = []++ def handle(request):+ requests.append(request)+ if len(requests) == 1:+ raise httpx.ReadTimeout("temporary failure", request=request)+ return httpx.Response(200, json={"status": "ok", "result": {"memories": []}})++ async with adapter_client(handle) as client:+ response = await client.post(+ "/search", json={"user_id": "alice", "query": "Where?", "top_k": 2}+ )+ assert response.status_code == 200+ assert response.json() == {"data": []}+ assert len(requests) == 2+ assert all(request.url.path == "/api/v1/search/find" for request in requests)+ assert requests[0].content == requests[1].content+ assert requests[0].headers == requests[1].headersdiff --git a/tests/unit/test_personamem_evaluate.py b/tests/unit/test_personamem_evaluate.pynew file mode 100644index 000000000..162b64676--- /dev/null+++ b/tests/unit/test_personamem_evaluate.py@@ -0,0 +1,184 @@+import copy+import json+import os+import re+import tempfile+import threading+import unittest+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from pathlib import Path+from unittest.mock import patch++from benchmark.aml.eval import evaluate as evaluator+++def retrieval_row(unit="personamem_v2_32k", question="QUESTION_CORRECT"):+ return {+ "id": question,+ "unit": unit,+ "persona_id": "7",+ "question": question,+ "user_query": {"role": "user", "content": question},+ "retrieved_context": ["The user enjoys hiking."],+ "chat_history": [{"role": "user", "content": "UNRETRIEVED_HISTORY"}],+ "preference": "LABEL_ONLY_PREFERENCE",+ "correct_answer": "Take a hiking trip.",+ "gold_answer": "Take a hiking trip.",+ "incorrect_answers": ["Go shopping.", "Watch television.", "Play a video game."],+ "answer_mode": "generative",+ "options": [],+ }+++class PersonaMemEvaluateTests(unittest.TestCase):+ def test_mode_override_uses_only_retrieved_memory_and_preserves_input(self):+ row = retrieval_row()+ original = copy.deepcopy(row)+ adapted, mode = evaluator._adapt_rows([row], "personamem_v2", persona_mode="mcq")+ self.assertEqual(mode, "mcq")+ self.assertEqual(adapted[0]["answer_mode"], "mcq")+ self.assertEqual(+ adapted[0]["chat_history"],+ [{"role": "system", "content": "[Retrieved memory 1]\nThe user enjoys hiking."}],+ )+ self.assertEqual(row, original)+ _, default_mode = evaluator._adapt_rows([row], "personamem_v2")+ self.assertEqual(default_mode, "mcq")+ _, explicit_mode = evaluator._adapt_rows([row], "personamem_v2", persona_mode="generative")+ self.assertEqual(explicit_mode, "generative")+ generic, generic_mode = evaluator._adapt_rows([row], "generic")+ self.assertIsNone(generic_mode)+ self.assertEqual(generic, [row])+ for size in ("32k", "128k"):+ unit = f"personamem_v2_{size}"+ self.assertEqual(evaluator._resolve_unit([retrieval_row(unit)], None), unit)+ adapted, _ = evaluator._adapt_rows(+ [{**row, "retrieved_context": []}], "personamem_v2", persona_mode="mcq"+ )+ self.assertEqual(+ adapted[0]["chat_history"][0]["content"], "No relevant memories were retrieved."+ )++ def test_invalid_options_or_unrelated_mode_override_fail_before_models(self):+ for incorrect in (+ [],+ ["only one"],+ ["a", "a", "b"],+ ["a", "b", " "],+ ["a", "b", "Take a hiking trip."],+ ["a", "b", 123],+ "not JSON",+ ):+ with self.subTest(incorrect=incorrect), self.assertRaises(ValueError):+ evaluator._adapt_rows(+ [{**retrieval_row(), "incorrect_answers": incorrect}],+ "personamem_v2",+ persona_mode="mcq",+ )+ with self.assertRaisesRegex(ValueError, "only supported for PersonaMem"):+ evaluator._adapt_rows([retrieval_row()], "generic", persona_mode="mcq")+ with self.assertRaisesRegex(ValueError, "unsupported PersonaMem mode"):+ evaluator._adapt_rows([retrieval_row()], "personamem_v2", persona_mode="invalid")++ def test_public_mcq_pipeline_with_local_http_and_no_judge_credentials(self):+ repo = (+ Path(__file__).resolve().parents[2] / "result/aml-public-eval/agent-memory-leaderboard"+ )+ if not (repo / "data/personamem/pipeline_v2.py").is_file():+ self.skipTest("pinned public AML checkout is required for the integration check")+ requests = []++ class Handler(BaseHTTPRequestHandler):+ def do_POST(self):+ payload = json.loads(self.rfile.read(int(self.headers["Content-Length"])))+ requests.append((self.path, payload))+ messages = payload["messages"]+ options = dict(re.findall(r"^([A-D])\. (.+)$", messages[-1]["content"], re.M))+ question = messages[1]["content"]+ if "QUESTION_INVALID" in question:+ answer = "I cannot select an option."+ else:+ target = (+ "Take a hiking trip." if "QUESTION_CORRECT" in question else "Go shopping."+ )+ letter = next(key for key, value in options.items() if value == target)+ answer = f"Final Answer: {letter}"+ body = json.dumps({"choices": [{"message": {"content": answer}}]}).encode()+ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def log_message(self, *_args):+ pass++ server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)+ thread = threading.Thread(target=server.serve_forever, daemon=True)+ thread.start()+ environment = {+ "ANSWER_API_BASE": f"http://127.0.0.1:{server.server_port}/v1",+ "ANSWER_API_KEY": "local-test-only",+ "ANSWER_MODEL": "local-test-model",+ "JUDGE_API_BASE": "",+ "JUDGE_API_KEY": "",+ "JUDGE_MODEL": "",+ "PYTHONHASHSEED": "0",+ "NO_PROXY": "127.0.0.1,localhost",+ }+ try:+ with tempfile.TemporaryDirectory() as directory, patch.dict(os.environ, environment):+ root = Path(directory)+ previous_mapping = None+ for size, concurrency in (("32k", 1), ("32k", 2), ("128k", 2)):+ unit = f"personamem_v2_{size}"+ rows = [+ retrieval_row(unit, question)+ for question in ("QUESTION_CORRECT", "QUESTION_WRONG", "QUESTION_INVALID")+ ]+ retrieval = root / f"{size}-{concurrency}.jsonl"+ evaluator._write_jsonl(retrieval, rows)+ original = retrieval.read_bytes()+ summary = evaluator.evaluate(+ retrieval,+ root / f"output-{size}-{concurrency}",+ aml_repo=repo,+ answer_concurrency=concurrency,+ eval_concurrency=concurrency,+ )+ self.assertEqual(retrieval.read_bytes(), original)+ self.assertEqual(summary["persona_mode"], "mcq")+ self.assertEqual(summary["correct_count"], 1)+ self.assertEqual(summary["incorrect_count"], 2)+ self.assertEqual(summary["invalid_answer_count"], 1)+ self.assertEqual(summary["accuracy"], 1 / 3)+ answers = evaluator._read_jsonl(Path(summary["answers"]))+ self.assertEqual([row["id"] for row in answers], [row["id"] for row in rows])+ mappings = [row["option_mapping"] for row in answers]+ if previous_mapping is not None:+ self.assertEqual(mappings, previous_mapping)+ previous_mapping = mappings+ judgments = evaluator._read_jsonl(Path(summary["judgments"]))+ self.assertEqual([r["is_correct"] for r in judgments], [True, False, False])+ self.assertTrue(+ all(r["gold_answer"] == "Take a hiking trip." for r in judgments)+ )+ self.assertEqual(len(requests), 9) # Only Answer calls; scoring is local.+ for path, payload in requests:+ self.assertEqual(path, "/v1/chat/completions")+ self.assertEqual(payload["model"], "local-test-model")+ prompt = json.dumps(payload["messages"])+ self.assertIn("The user enjoys hiking.", prompt)+ self.assertNotIn("UNRETRIEVED_HISTORY", prompt)+ self.assertNotIn("LABEL_ONLY_PREFERENCE", prompt)+ self.assertNotIn("correct_answer", prompt)+ self.assertNotIn("correct_letter", prompt)+ self.assertEqual(prompt.count("Take a hiking trip."), 1)+ finally:+ server.shutdown()+ server.server_close()+ thread.join(timeout=5)+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/unit/test_personamem_prepare.py b/tests/unit/test_personamem_prepare.pynew file mode 100644index 000000000..b19882a6d--- /dev/null+++ b/tests/unit/test_personamem_prepare.py@@ -0,0 +1,119 @@+import hashlib+import json+import tempfile+import unittest+from pathlib import Path++from benchmark.aml.eval.prepare_personamem import build_prepared, chat_messages, refresh_smoke+from benchmark.aml.eval.run_public import _prepare_unit+++class PersonaMemPrepareTests(unittest.TestCase):+ def test_system_is_dropped_and_chat_text_order_is_exact(self):+ messages, removed, blank = chat_messages(+ [+ {"role": "system", "content": "EXPLICIT PERSONA"},+ {"role": "user", "content": " Original text\n"},+ {"role": "system", "content": "ANOTHER PROFILE"},+ {"role": "assistant", "content": "reply"},+ {"role": "assistant", "content": " \n"},+ ]+ )+ self.assertEqual(removed, [0, 2])+ self.assertEqual(blank, [4])+ self.assertEqual(+ messages,+ [+ {"role": "user", "content": " Original text\n"},+ {"role": "assistant", "content": "reply"},+ ],+ )++ def test_multimodal_or_unknown_messages_fail_in_text_mode(self):+ for message in [+ {"role": "user", "content": [{"type": "image"}]},+ {"role": "tool", "content": "text"},+ ]:+ with self.assertRaises(ValueError):+ chat_messages([message])++ def test_csv_links_labels_and_user_isolation(self):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ expected = {}++ def write(relative, content):+ path = root / relative+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(content)+ expected[relative] = hashlib.sha256(path.read_bytes()).hexdigest()++ write("benchmark/text/benchmark.csv", "test fixture")+ row = {+ "persona_id": "7",+ "raw_persona_file": "data/raw_data/persona7.json",+ "user_query": repr({"role": "user", "content": "question"}),+ "correct_answer": "GOLD ONLY",+ "incorrect_answers": repr(["WRONG ONLY"]),+ "preference": "ANNOTATION ONLY",+ }+ for size in ["32k", "128k"]:+ relative = f"data/chat_history_{size}/exact7.json"+ row[f"chat_history_{size}_link"] = relative+ write(+ relative,+ json.dumps(+ {+ "metadata": {+ "persona_id": 7,+ "input_filename": row["raw_persona_file"],+ "final_token_count": 100,+ },+ "chat_history": [+ {"role": "system", "content": "EXPLICIT PERSONA"},+ {"role": "user", "content": size + " original chat"},+ ],+ }+ ),+ )+ # Filename guessing or reading raw_data must not replace this link.+ write(f"data/chat_history_{size}/newer7.json", "invalid alternative")+ second = {**row, "user_query": repr({"role": "user", "content": "question two"})}+ units = [+ build_prepared(root, [row, second], size, "revision", expected)+ for size in ["32k", "128k"]+ ]+ users = []+ for payload in units:+ self.assertEqual(len(payload["histories"]), 1)+ session = payload["histories"][0]["sessions"][0]+ self.assertNotIn("GOLD ONLY", json.dumps(session))+ self.assertNotIn("ANNOTATION ONLY", json.dumps(session))+ cases = [{**case, "histories": [session]} for case in payload["cases"]]+ groups, searches = _prepare_unit(payload["unit"], cases, "same-run")+ self.assertEqual(len(groups), 1)+ self.assertEqual(searches[0].user_id, searches[1].user_id)+ users.append(searches[0].user_id)+ self.assertEqual(payload["cases"][0]["gold_answer"], "GOLD ONLY")+ self.assertEqual(payload["cases"][0]["extra"]["answer_mode"], "mcq")+ self.assertEqual(payload["cases"][0]["options"], [])+ self.assertNotEqual(*users)+ unchanged = {"unit": "clbench_0_4k", "case_id": "unrelated", "histories": []}+ smoke = {+ "cases": [+ unchanged,+ {"unit": units[0]["unit"], "case_id": units[0]["cases"][0]["case_id"]},+ ],+ "datasets": [{"unit": units[0]["unit"]}, {"unit": "clbench_0_4k"}],+ }+ refreshed = refresh_smoke(smoke, units[0])+ self.assertEqual(refreshed["cases"][0], unchanged)+ self.assertNotIn("system", json.dumps(refreshed["cases"][1]["histories"]))+ # A modified or wrong-sized linked file must fail before preparation.+ (root / row["chat_history_128k_link"]).write_text("tampered")+ with self.assertRaises(ValueError):+ build_prepared(root, [row], "128k", "revision", expected)+++if __name__ == "__main__":+ unittest.main()