Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(benchmark): add TAU-2 Memory V2 eval runner (#2003)
huangruitengc9ebba07347d
diff --git a/benchmark/tau2/.gitignore b/benchmark/tau2/.gitignorenew file mode 100644index 000000000..2577e5885--- /dev/null+++ b/benchmark/tau2/.gitignore@@ -0,0 +1,5 @@+result/+.env.tau2+.external/+.venv-tau2/+__pycache__/diff --git a/benchmark/tau2/README.md b/benchmark/tau2/README.mdnew file mode 100644index 000000000..7ebdb0807--- /dev/null+++ b/benchmark/tau2/README.md@@ -0,0 +1,146 @@+# TAU-2 Benchmark++This directory contains a small OpenViking-style entry point for TAU-2 memory+evaluation. The first version is intentionally narrow:++- fresh OpenViking Memory V2 experience-only baseline;+- Memory V2 pre-write recall treatment.++Trajectory / procedure-view prompts, category rerank, and other harness-only+diagnostics are intentionally left out of this first PR.++## Layout++```text+benchmark/tau2/+├── config/+│ ├── baseline.yaml+│ ├── official.yaml+│ └── prewrite.yaml+├── scripts/+│ ├── run_eval.py+│ ├── setup_tau2_repo.sh+│ └── tau2_common.py+└── run_full_eval.sh+```++Generated artifacts are written to `benchmark/tau2/result/<run_id>/`.++## Quick Start++This benchmark delegates task simulation and scoring to an external TAU-2+checkout. Point the runner at that checkout and CLI explicitly when they are not+on the default path:++```bash+export TAU2_REPO=/path/to/tau2-bench+export TAU2_CLI=/path/to/tau2+```++For a local one-command setup, clone and install TAU-2 into ignored benchmark+directories:++```bash+benchmark/tau2/scripts/setup_tau2_repo.sh+source benchmark/tau2/.env.tau2+```++Plan the default benchmark without running TAU-2:++```bash+python benchmark/tau2/scripts/run_eval.py --config benchmark/tau2/config/baseline.yaml --plan-only+```++Add `--preflight` or `--strict-preflight` when you want the runner to write a+small environment/config check next to the run plan.++After setup, verify the local TAU-2 link and write a one-cell run plan:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/baseline.yaml \+ --strict-preflight \+ --domain retail \+ --strategy-id memory_v2_experience_only \+ --task-id 5 \+ --repeat-count 1+```++Plan a one-cell Memory V2 pre-write smoke:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/baseline.yaml \+ --domain retail \+ --strategy-id memory_v2_prewrite \+ --num-tasks 1 \+ --repeat-count 1+```++Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/baseline.yaml \+ --execute+```++For a small E2E smoke, keep both the eval and train slices tiny:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/baseline.yaml \+ --domain retail \+ --strategy-id memory_v2_experience_only \+ --num-tasks 1 \+ --train-num-tasks 1 \+ --repeat-count 1 \+ --execute+```++When using Doubao through an OpenAI-compatible endpoint, set `OPENAI_API_KEY`+and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.++Start the OpenViking service before executing memory cells, and verify it with+`ov status`. For evidence runs, use a clean OpenViking workspace/config and set+`OPENVIKING_URL` explicitly so local custom memory templates do not pollute the+Memory V2 baseline.++## Memory Adapter++`memory_v2_experience_only` and `memory_v2_prewrite` cells run through a small+TAU-2 agent adapter in this directory:++- train by writing TAU-2 training conversations into OpenViking sessions;+- evaluate by retrieving OpenViking experience memory at the first user turn;+- for pre-write recall, retrieve again before write-like tool calls and+ regenerate that step with the matched memories;+- emit artifact metadata to identify the OpenViking account, agent,+ corpus, retrieval mode, and simulator policy used by each cell.++## User Simulator Policy++The runner default is the official TAU-2 user simulator if+`eval.user_simulator_policy` is omitted. The bundled OpenViking memory benchmark+config sets `confirmation_aware`, because a memory benchmark should not treat+user confirmation as task completion before the backend write has happened.++`confirmation_aware` applies a small idempotent prompt patch to the configured+TAU-2 checkout before planning or running. The patch appends only the behavioral+confirmation boundary to the TAU-2 user simulator guidelines; metadata such as+the upstream PR link is kept in run artifacts, not in the simulator prompt.+Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).++Use `config/official.yaml` with a clean TAU-2 checkout when you need an+official-user-simulator parity run. If the checkout was already patched, the+artifact records that boundary instead of labeling the run pure official.++## Evidence Boundary++Only completed `retail + airline` runs with the same config, same seeds/repeats,+and non-empty artifacts should be read as benchmark evidence. Partial runs,+single-task probes, or missing OpenViking corpus identity are diagnostics.+Executed runs write per-cell JSON under `cell_results/` and a strategy/domain+aggregate under `scoreboard.json`. Memory training artifacts are shared by+domain and strategy under `memory_corpora/`, so repeated eval cells reuse the+same fresh corpus instead of rewriting it.diff --git a/benchmark/tau2/config/baseline.yaml b/benchmark/tau2/config/baseline.yamlnew file mode 100644index 000000000..4c4a5060e--- /dev/null+++ b/benchmark/tau2/config/baseline.yaml@@ -0,0 +1,53 @@+benchmark:+ name: tau2_openviking_baseline+ domains:+ - retail+ - airline+ train_split_name: train+ eval_split_name: test+ repeat_count: 8+ task_max_concurrency: 10+ max_steps: 200+ seed: 300+ agent: llm_agent+ user: user_simulator+ reasoning_effort: high++paths:+ tau2_repo: ${TAU2_REPO:-data/external_benchmarks/tau2-bench}+ tau2_cli: ${TAU2_CLI:-tau2}+ output_dir: benchmark/tau2/result++eval:+ # The runner default is official if this field is omitted. The OpenViking+ # memory benchmark config opts into a confirmation-aware TAU-2 user simulator+ # prompt; run_eval.py applies that small prompt patch idempotently when needed.+ user_simulator_policy: confirmation_aware++model:+ agent_llm: ${TAU2_AGENT_LLM:-openai/doubao-seed-2-0-pro-260215}+ user_llm: ${TAU2_USER_LLM:-openai/doubao-seed-2-0-pro-260215}+ temperature: 0.0++openviking:+ url: ${OPENVIKING_URL:-http://localhost:1933}+ account: ${OPENVIKING_ACCOUNT:-default}+ agent_id: ${OPENVIKING_AGENT_ID:-tau2-openviking-agent}+ retrieval_top_k: 4+ replay_write_policy: read_only++strategies:+ - id: memory_v2_experience_only+ label: OpenViking Memory V2 experience-only+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_experience_only+ train_memory_mode: experience_only+ retrieval_mode: first_user+ - id: memory_v2_prewrite+ label: OpenViking Memory V2 pre-write recall+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_experience_only+ train_memory_mode: experience_only+ retrieval_mode: first_user_prewritediff --git a/benchmark/tau2/config/official.yaml b/benchmark/tau2/config/official.yamlnew file mode 100644index 000000000..d10bee872--- /dev/null+++ b/benchmark/tau2/config/official.yaml@@ -0,0 +1,7 @@+extends: baseline.yaml++benchmark:+ name: tau2_openviking_official_user_simulator++eval:+ user_simulator_policy: officialdiff --git a/benchmark/tau2/config/prewrite.yaml b/benchmark/tau2/config/prewrite.yamlnew file mode 100644index 000000000..834963b41--- /dev/null+++ b/benchmark/tau2/config/prewrite.yaml@@ -0,0 +1,13 @@+extends: baseline.yaml++benchmark:+ name: tau2_openviking_prewrite++strategies:+ - id: memory_v2_prewrite+ label: OpenViking Memory V2 pre-write recall+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_experience_only+ train_memory_mode: experience_only+ retrieval_mode: first_user_prewritediff --git a/benchmark/tau2/run_full_eval.sh b/benchmark/tau2/run_full_eval.shnew file mode 100755index 000000000..ca69a7a32--- /dev/null+++ b/benchmark/tau2/run_full_eval.sh@@ -0,0 +1,72 @@+#!/usr/bin/env bash+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"+PYTHON_BIN="${PYTHON_BIN:-python3}"+CONFIG="$SCRIPT_DIR/config/baseline.yaml"+EXECUTE=false+PREFLIGHT=false+STRICT_PREFLIGHT=false+RUN_ID=""+RUN_EVAL_EXTRA=()++while [[ $# -gt 0 ]]; do+ case "$1" in+ --config)+ CONFIG="$2"+ shift 2+ ;;+ --run-id)+ RUN_ID="$2"+ shift 2+ ;;+ --execute)+ EXECUTE=true+ shift+ ;;+ --preflight)+ PREFLIGHT=true+ shift+ ;;+ --strict-preflight)+ STRICT_PREFLIGHT=true+ shift+ ;;+ --domain|--repeat-count|--strategy-id|--task-id|--num-tasks|--train-num-tasks)+ RUN_EVAL_EXTRA+=("$1" "$2")+ shift 2+ ;;+ --help|-h)+ cat <<'EOF'+Usage:+ benchmark/tau2/run_full_eval.sh [--config PATH] [--run-id ID] [--execute] [--preflight]++Without --execute the script only writes run_plan artifacts.+EOF+ exit 0+ ;;+ *)+ echo "Unknown argument: $1" >&2+ exit 1+ ;;+ esac+done++RUN_ARGS=()+if [[ -n "$RUN_ID" ]]; then+ RUN_ARGS+=(--run-id "$RUN_ID")+fi++cd "$REPO_ROOT"+if [[ "$STRICT_PREFLIGHT" == true ]]; then+ RUN_EVAL_EXTRA+=(--strict-preflight)+elif [[ "$PREFLIGHT" == true ]]; then+ RUN_EVAL_EXTRA+=(--preflight)+fi++if [[ "$EXECUTE" == true ]]; then+ "$PYTHON_BIN" "$SCRIPT_DIR/scripts/run_eval.py" --config "$CONFIG" "${RUN_ARGS[@]}" "${RUN_EVAL_EXTRA[@]}" --execute+else+ "$PYTHON_BIN" "$SCRIPT_DIR/scripts/run_eval.py" --config "$CONFIG" "${RUN_ARGS[@]}" "${RUN_EVAL_EXTRA[@]}" --plan-only+fidiff --git a/benchmark/tau2/scripts/run_eval.py b/benchmark/tau2/scripts/run_eval.pynew file mode 100755index 000000000..5458ba61a--- /dev/null+++ b/benchmark/tau2/scripts/run_eval.py@@ -0,0 +1,531 @@+#!/usr/bin/env python3+from __future__ import annotations++import argparse+import importlib.util+import json+import subprocess+import sys+from pathlib import Path+from typing import Any++from tau2_common import (+ domains,+ load_config,+ output_dir,+ normalize_litellm_env,+ run_id,+ simulator_policy_report,+ split_file,+ strategy_ids,+ tau2_cli,+ tau2_context,+ tau2_repo,+ user_simulator_policy,+ write_json,+)+++def _reward(sim: dict[str, Any]) -> float:+ info = sim.get("reward_info") or {}+ value = info.get("reward", sim.get("reward", 0.0))+ try:+ return float(value)+ except (TypeError, ValueError):+ return 0.0+++def _db_match(sim: dict[str, Any]) -> bool | None:+ info = sim.get("reward_info") or {}+ db = info.get("db_check") or {}+ if isinstance(db, dict):+ if "score" in db:+ return bool(db["score"])+ if "db_match" in db:+ return bool(db["db_match"])+ return sim.get("db_match")+++def _metrics_from_tau2_results(results_path: Path) -> dict[str, Any]:+ data = json.loads(results_path.read_text(encoding="utf-8"))+ sims = data.get("simulations") or []+ rewards = [_reward(sim) for sim in sims]+ db_values = [_db_match(sim) for sim in sims]+ db_known = [value for value in db_values if value is not None]+ return {+ "simulation_count": len(sims),+ "avg_reward": sum(rewards) / len(rewards) if rewards else 0.0,+ "db_match_rate": (sum(1 for value in db_known if value) / len(db_known)) if db_known else None,+ }+++def _tau2_command(+ config: dict[str, Any],+ *,+ domain: str,+ strategy: dict[str, Any],+ configured_run_id: str,+ run_label: str,+ task_ids: list[str] | None,+ num_tasks: int | None,+ train_num_tasks: int | None,+ seed: int,+) -> list[str] | None:+ benchmark = config["benchmark"]+ model = config["model"]++ reasoning_effort = benchmark.get("reasoning_effort")+ agent_llm_args = '{"temperature":0.0}'+ user_llm_args = '{"temperature":0.0}'+ if reasoning_effort:+ agent_llm_args = f'{{"temperature":0.0,"reasoning_effort":"{reasoning_effort}"}}'+ user_llm_args = f'{{"temperature":0.0,"reasoning_effort":"{reasoning_effort}"}}'++ if (+ strategy.get("memory_backend") == "openviking"+ and strategy.get("train_memory_mode") == "experience_only"+ ):+ openviking = config["openviking"]+ corpus_id = str(strategy.get("corpus_id") or strategy["id"])+ account = f"{openviking['account']}-{configured_run_id}-{domain}-{corpus_id}"+ agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"+ user = f"tau2-{domain}-{corpus_id}"+ search_uri = f"viking://agent/{agent_id}/memories/experiences"+ command = [+ sys.executable,+ str(Path(__file__).with_name("run_memory_v2_eval.py")),+ "--tau2-repo",+ str(tau2_repo(config)),+ "--run-dir",+ str(output_dir(config, configured_run_id) / "memory_cells" / run_label),+ "--corpus-dir",+ str(+ output_dir(config, configured_run_id)+ / "memory_corpora"+ / f"{domain}_{corpus_id}"+ ),+ "--run-label",+ run_label,+ "--strategy-id",+ strategy["id"],+ "--domain",+ domain,+ "--train-split-name",+ str(benchmark.get("train_split_name", "train")),+ "--eval-split-name",+ str(benchmark.get("eval_split_name", "test")),+ "--max-steps",+ str(benchmark.get("max_steps", 200)),+ "--max-concurrency",+ str(benchmark.get("task_max_concurrency", 10)),+ "--agent-llm",+ str(model["agent_llm"]),+ "--user-llm",+ str(model["user_llm"]),+ "--agent-llm-args",+ agent_llm_args,+ "--user-llm-args",+ user_llm_args,+ "--openviking-url",+ str(openviking["url"]),+ "--openviking-account",+ account,+ "--openviking-user",+ user,+ "--openviking-agent-id",+ agent_id,+ "--search-uri",+ search_uri,+ "--retrieval-top-k",+ str(openviking.get("retrieval_top_k", 4)),+ "--retrieval-mode",+ str(strategy.get("retrieval_mode", "first_user")),+ "--seed",+ str(seed),+ ]+ if task_ids:+ for task_id in task_ids:+ command.extend(["--task-id", task_id])+ elif num_tasks is not None:+ command.extend(["--num-tasks", str(num_tasks)])+ train_num_tasks = train_num_tasks if train_num_tasks is not None else strategy.get("train_num_tasks")+ if train_num_tasks is not None:+ command.extend(["--train-num-tasks", str(train_num_tasks)])+ return command++ if strategy.get("memory_backend") != "none":+ return None++ command = [+ tau2_cli(config),+ "run",+ "--domain",+ domain,+ "--agent",+ str(benchmark.get("agent", "llm_agent")),+ "--user",+ str(benchmark.get("user", "user_simulator")),+ "--task-split-name",+ str(benchmark.get("eval_split_name", "test")),+ "--num-trials",+ "1",+ "--max-steps",+ str(benchmark.get("max_steps", 200)),+ "--max-concurrency",+ str(benchmark.get("task_max_concurrency", 10)),+ "--agent-llm",+ str(model["agent_llm"]),+ "--user-llm",+ str(model["user_llm"]),+ "--save-to",+ run_label,+ "--seed",+ str(seed),+ ]++ command.extend(["--agent-llm-args", agent_llm_args])+ command.extend(["--user-llm-args", user_llm_args])++ if task_ids:+ command.append("--task-ids")+ command.extend(task_ids)+ elif num_tasks is not None:+ command.extend(["--num-tasks", str(num_tasks)])++ return command+++def _build_plan(+ config: dict[str, Any],+ configured_run_id: str,+ *,+ selected_domains: set[str] | None,+ selected_strategy_ids: set[str] | None,+ task_ids: list[str] | None,+ num_tasks: int | None,+ train_num_tasks: int | None,+ repeat_count_override: int | None,+) -> dict[str, Any]:+ repeat_count = repeat_count_override or int(config["benchmark"].get("repeat_count", 8))+ base_seed = int(config["benchmark"].get("seed", 300))+ policy_report = simulator_policy_report(config)+ strategies = config.get("strategies") or []+ if selected_strategy_ids:+ unknown = selected_strategy_ids - set(strategy_ids(config))+ if unknown:+ raise ValueError(f"unknown strategy ids: {sorted(unknown)}")+ strategies = [strategy for strategy in strategies if strategy["id"] in selected_strategy_ids]+ cells = []+ plan_domains = domains(config)+ if selected_domains:+ unknown_domains = selected_domains - set(plan_domains)+ if unknown_domains:+ raise ValueError(f"unknown domains: {sorted(unknown_domains)}")+ plan_domains = [domain for domain in plan_domains if domain in selected_domains]+ for domain in plan_domains:+ split_path = split_file(config, domain)+ for strategy in strategies:+ for repeat_index in range(repeat_count):+ seed = base_seed + repeat_index+ run_label = f"{configured_run_id}_{domain}_{strategy['id']}_r{repeat_index + 1}"+ command = _tau2_command(+ config,+ domain=domain,+ strategy=strategy,+ configured_run_id=configured_run_id,+ run_label=run_label,+ task_ids=task_ids,+ num_tasks=num_tasks,+ train_num_tasks=train_num_tasks,+ seed=seed,+ )+ non_executable_reason = None+ if command is None:+ non_executable_reason = (+ "This OpenViking memory strategy is planned but not wired to "+ "the TAU-2 adapter in this PR."+ )+ cells.append(+ {+ "domain": domain,+ "strategy_id": strategy["id"],+ "strategy_label": strategy.get("label", strategy["id"]),+ "repeat_index": repeat_index + 1,+ "seed": seed,+ "run_label": run_label,+ "train_required": bool(strategy.get("train_required")),+ "memory_backend": strategy.get("memory_backend"),+ "corpus_id": strategy.get("corpus_id", strategy["id"]),+ "retrieval_mode": strategy.get("retrieval_mode"),+ "adapter_status": strategy.get("adapter_status", "ready"),+ "executable": command is not None,+ "user_simulator_policy": user_simulator_policy(config),+ "user_simulator_policy_supported": policy_report["supported"],+ "split_file": str(split_path),+ "command": command,+ "non_executable_reason": non_executable_reason,+ }+ )+ executable_cell_count = sum(1 for cell in cells if cell["executable"])+ return {+ "schema_version": "openviking.tau2.run_plan.v0",+ "run_id": configured_run_id,+ "status": "planned",+ "strategy_ids": strategy_ids(config),+ "domains": plan_domains,+ "tau2": tau2_context(config),+ "simulator_policy": policy_report,+ "cell_count": len(cells),+ "executable_cell_count": executable_cell_count,+ "pending_cell_count": len(cells) - executable_cell_count,+ "cells": cells,+ }+++def _cell_artifacts(cell: dict[str, Any], repo: Path, out: Path) -> dict[str, str]:+ if cell.get("memory_backend") == "openviking":+ run_dir = out / "memory_cells" / cell["run_label"]+ corpus_id = str(cell.get("corpus_id") or cell["strategy_id"])+ corpus_dir = out / "memory_corpora" / f"{cell['domain']}_{corpus_id}"+ return {+ "summary": str(run_dir / f"{cell['run_label']}.summary.json"),+ "results": str(run_dir / f"{cell['run_label']}.json"),+ "retrieval_trace": str(run_dir / f"{cell['run_label']}.retrieval_trace.jsonl"),+ "corpus_manifest": str(corpus_dir / "corpus_manifest.json"),+ }+ return {+ "results": str(repo / "data" / "simulations" / f"{cell['run_label']}.json")+ }+++def _cell_metrics(cell: dict[str, Any], artifacts: dict[str, str]) -> dict[str, Any] | None:+ if cell.get("memory_backend") == "openviking":+ summary_path = Path(artifacts["summary"])+ if not summary_path.is_file():+ return None+ summary = json.loads(summary_path.read_text(encoding="utf-8"))+ return summary.get("metrics")++ results_path = Path(artifacts["results"])+ if not results_path.is_file():+ return None+ return _metrics_from_tau2_results(results_path)+++def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:+ def weighted(rows_for_group: list[dict[str, Any]]) -> dict[str, Any]:+ metric_rows = [row for row in rows_for_group if row.get("metrics")]+ sim_count = sum(int(row["metrics"].get("simulation_count") or 0) for row in metric_rows)+ reward_sum = sum(+ float(row["metrics"].get("avg_reward") or 0.0)+ * int(row["metrics"].get("simulation_count") or 0)+ for row in metric_rows+ )+ db_weighted_rows = [+ row+ for row in metric_rows+ if row["metrics"].get("db_match_rate") is not None+ and int(row["metrics"].get("simulation_count") or 0) > 0+ ]+ db_weight = sum(int(row["metrics"].get("simulation_count") or 0) for row in db_weighted_rows)+ db_sum = sum(+ float(row["metrics"]["db_match_rate"])+ * int(row["metrics"].get("simulation_count") or 0)+ for row in db_weighted_rows+ )+ return {+ "cell_count": len(rows_for_group),+ "completed_cell_count": len(metric_rows),+ "simulation_count": sim_count,+ "avg_reward": reward_sum / sim_count if sim_count else None,+ "db_match_rate": db_sum / db_weight if db_weight else None,+ }++ by_strategy: dict[str, dict[str, Any]] = {}+ for row in rows:+ strategy_id = row["strategy_id"]+ strategy_summary = by_strategy.setdefault(+ strategy_id,+ {+ "strategy_id": strategy_id,+ "domains": {},+ "task_weighted_total": {},+ },+ )+ strategy_summary["domains"].setdefault(row["domain"], []).append(row)++ for strategy_summary in by_strategy.values():+ all_rows = []+ for domain, domain_rows in list(strategy_summary["domains"].items()):+ strategy_summary["domains"][domain] = weighted(domain_rows)+ all_rows.extend(domain_rows)+ strategy_summary["task_weighted_total"] = weighted(all_rows)++ return {+ "schema_version": "openviking.tau2.scoreboard.v0",+ "strategies": by_strategy,+ }+++def _execute_cells(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str, Any]]:+ policy_report = plan.get("simulator_policy") or {}+ if not policy_report.get("supported", False):+ raise RuntimeError(+ "configured user simulator policy is not supported by this TAU-2 checkout: "+ f"{policy_report}"+ )+ rows = []+ for cell in plan["cells"]:+ if not cell.get("executable"):+ raise RuntimeError(+ f"cell is not executable yet: {cell['run_label']} "+ f"(strategy_id={cell['strategy_id']}, adapter_status={cell.get('adapter_status')})"+ )+ print(f"[tau2] running {cell['run_label']}")+ completed = subprocess.run(+ cell["command"],+ cwd=repo,+ text=True,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ check=False,+ )+ row = {+ "run_label": cell["run_label"],+ "domain": cell["domain"],+ "strategy_id": cell["strategy_id"],+ "returncode": completed.returncode,+ "stdout_tail": completed.stdout[-4000:],+ "stderr_tail": completed.stderr[-4000:],+ }+ row["artifacts"] = _cell_artifacts(cell, repo, out)+ row["metrics"] = _cell_metrics(cell, row["artifacts"])+ rows.append(row)+ write_json(out / "cell_results" / f"{cell['run_label']}.json", row)+ if completed.returncode != 0:+ raise RuntimeError(f"cell failed: {cell['run_label']} returncode={completed.returncode}")+ return rows+++def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:+ errors: list[str] = []+ llm_env = normalize_litellm_env()+ tau2_info = tau2_context(config)+ policy_report = simulator_policy_report(config)+ if strict and not tau2_info["tau2_repo_exists"]:+ errors.append(f"missing TAU-2 repo: {tau2_info['tau2_repo']}")+ if strict and not tau2_info["tau2_cli_resolved"]:+ errors.append(f"missing TAU-2 CLI: {tau2_info['tau2_cli']}")+ if strict and not llm_env["has_api_key"]:+ errors.append("missing LLM API key: set OPENAI_API_KEY or ARK_API_KEY")+ if strict and not llm_env["has_base_url"]:+ errors.append("missing OpenAI-compatible base URL: set OPENAI_API_BASE, OPENAI_BASE_URL, or ARK_BASE_URL")+ if strict and not policy_report["supported"]:+ errors.append(+ "configured confirmation-aware user simulator policy requires a TAU-2 "+ f"checkout with the prompt fix: {policy_report['prompt_files']}"+ )+ split_rows = []+ for domain in domains(config):+ path = split_file(config, domain)+ exists = path.is_file()+ split_rows.append({"domain": domain, "path": str(path), "exists": exists})+ if strict and not exists:+ errors.append(f"missing split file for {domain}: {path}")++ import_rows = []+ for module in ("openviking", "openviking_cli", "tau2"):+ ok = importlib.util.find_spec(module) is not None+ import_rows.append({"module": module, "ok": ok})+ if strict and not ok:+ errors.append(f"missing Python module: {module}")++ report = {+ "status": "failed" if errors else "ok",+ "strict": strict,+ "tau2": tau2_info,+ "llm_env": llm_env,+ "simulator_policy": policy_report,+ "domains": domains(config),+ "strategies": strategy_ids(config),+ "imports": import_rows,+ "split_files": split_rows,+ "errors": errors,+ }+ write_json(out / "preflight.json", report)+ if errors:+ for error in errors:+ print(f"[preflight][ERROR] {error}", file=sys.stderr)+ return 1+ print(f"[preflight][OK] wrote {out / 'preflight.json'}")+ return 0+++def main() -> int:+ parser = argparse.ArgumentParser(description="Plan or run TAU-2 benchmark cells.")+ parser.add_argument("--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml")+ parser.add_argument("--run-id", default=run_id())+ parser.add_argument("--domain", action="append", help="Run only this configured domain; may be repeated.")+ parser.add_argument("--repeat-count", type=int, help="Override benchmark.repeat_count for smoke runs.")+ parser.add_argument("--strategy-id", action="append", help="Run only this strategy id; may be repeated.")+ parser.add_argument("--task-id", action="append", help="Run only this TAU-2 task id; may be repeated.")+ parser.add_argument("--num-tasks", type=int, help="Run the first N tasks from the selected split.")+ parser.add_argument("--train-num-tasks", type=int, help="Train OpenViking memory on the first N train tasks.")+ parser.add_argument(+ "--preflight",+ action="store_true",+ help="Write a lightweight environment/config preflight report.",+ )+ parser.add_argument(+ "--strict-preflight",+ action="store_true",+ help="Fail if optional runtime imports or split files are missing.",+ )+ parser.add_argument("--plan-only", action="store_true", help="Only write run_plan.json.")+ parser.add_argument("--execute", action="store_true", help="Execute planned cells.")+ args = parser.parse_args()+ normalize_litellm_env()++ if args.plan_only and args.execute:+ raise SystemExit("--plan-only and --execute are mutually exclusive")++ config = load_config(args.config)+ out = output_dir(config, args.run_id)+ out.mkdir(parents=True, exist_ok=True)+ if args.preflight or args.strict_preflight:+ preflight_status = _preflight(config, out, strict=args.strict_preflight)+ if args.strict_preflight and preflight_status != 0:+ return preflight_status++ plan = _build_plan(+ config,+ args.run_id,+ selected_domains=set(args.domain) if args.domain else None,+ selected_strategy_ids=set(args.strategy_id) if args.strategy_id else None,+ task_ids=args.task_id,+ num_tasks=args.num_tasks,+ train_num_tasks=args.train_num_tasks,+ repeat_count_override=args.repeat_count,+ )+ write_json(out / "run_plan.json", plan)+ write_json(out / "resolved_config.json", config)+ print(f"[tau2] wrote {out / 'run_plan.json'}")++ if args.execute:+ try:+ rows = _execute_cells(plan, tau2_repo(config), out)+ plan["status"] = "succeeded"+ plan["executed_cell_count"] = len(rows)+ write_json(out / "run_plan.json", plan)+ write_json(out / "scoreboard.json", _summarize(rows))+ except Exception as exc:+ plan["status"] = "failed"+ plan["error"] = str(exc)+ write_json(out / "run_plan.json", plan)+ print(f"[tau2][ERROR] {exc}", file=sys.stderr)+ return 1+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/tau2/scripts/run_memory_v2_eval.py b/benchmark/tau2/scripts/run_memory_v2_eval.pynew file mode 100644index 000000000..de5ef5441--- /dev/null+++ b/benchmark/tau2/scripts/run_memory_v2_eval.py@@ -0,0 +1,645 @@+#!/usr/bin/env python3+from __future__ import annotations++import argparse+import json+import shutil+import sys+import time+from pathlib import Path+from typing import Any++from tau2_common import normalize_litellm_env+++AGENT_NAME = "openviking_memory_agent"+REPO_ROOT = Path(__file__).resolve().parents[3]+WRITE_TOOL_PREFIXES = (+ "toggle_",+ "enable_",+ "disable_",+ "set_",+ "reset_",+ "update_",+ "modify_",+ "cancel_",+ "book_",+ "exchange_",+ "return_",+ "grant_",+ "reboot_",+)+++def _json(text: str) -> dict[str, Any]:+ return json.loads(text) if text else {}+++def _write_json(path: Path, payload: Any) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n")+++def _add_tau2_to_path(tau2_repo: Path) -> None:+ src = tau2_repo / "src"+ sys.path.insert(0, str(REPO_ROOT))+ sys.path.insert(0, str(src if src.is_dir() else tau2_repo))+++def _save_to_arg(path: Path) -> str:+ # Some TAU-2 versions append ".json"; newer versions treat save_to as a+ # run directory and write results.json under it.+ return str(path.with_suffix("") if path.suffix == ".json" else path)+++def _compat_results_path(path: Path) -> Path:+ run_dir = path.with_suffix("") if path.suffix == ".json" else path+ return run_dir / "results.json"+++def _reward(sim: dict[str, Any]) -> float:+ info = sim.get("reward_info") or {}+ value = info.get("reward", sim.get("reward", 0.0))+ try:+ return float(value)+ except (TypeError, ValueError):+ return 0.0+++def _db_match(sim: dict[str, Any]) -> bool | None:+ info = sim.get("reward_info") or {}+ db = info.get("db_check") or {}+ if isinstance(db, dict):+ if "score" in db:+ return bool(db["score"])+ if "db_match" in db:+ return bool(db["db_match"])+ return sim.get("db_match")+++def _metrics(results_path: Path) -> dict[str, Any]:+ data = json.loads(results_path.read_text())+ sims = data.get("simulations") or []+ rewards = [_reward(sim) for sim in sims]+ db_values = [_db_match(sim) for sim in sims]+ db_known = [value for value in db_values if value is not None]+ return {+ "simulation_count": len(sims),+ "avg_reward": sum(rewards) / len(rewards) if rewards else 0.0,+ "db_match_rate": (sum(1 for value in db_known if value) / len(db_known)) if db_known else None,+ }+++def _tool_call_name(tool_call: Any) -> str:+ if isinstance(tool_call, dict):+ return str(tool_call.get("name") or tool_call.get("function", {}).get("name") or "")+ return str(getattr(tool_call, "name", "") or "")+++def _tool_call_arguments(tool_call: Any) -> Any:+ if isinstance(tool_call, dict):+ return tool_call.get("arguments") or tool_call.get("function", {}).get("arguments") or {}+ return getattr(tool_call, "arguments", {}) or {}+++def _is_write_tool_call(tool_call: Any) -> bool:+ name = _tool_call_name(tool_call)+ return bool(name) and name.startswith(WRITE_TOOL_PREFIXES)+++def _tool_call_query(tool_calls: list[Any], state_messages: list[Any]) -> str:+ rendered = []+ for call in tool_calls:+ rendered.append(+ f"{_tool_call_name(call) or 'unknown_tool'}("+ f"{json.dumps(_tool_call_arguments(call), ensure_ascii=False, sort_keys=True)}"+ ")"+ )+ recent_user = [+ str(getattr(message, "content", "") or "")+ for message in state_messages[-8:]+ if str(getattr(message, "role", "")) == "user" and str(getattr(message, "content", "") or "").strip()+ ]+ recent_observations = [+ str(getattr(message, "content", "") or "")[:600]+ for message in state_messages[-12:]+ if str(getattr(message, "role", "")) == "tool" and str(getattr(message, "content", "") or "").strip()+ ]+ parts = [+ "Before executing write-like tool call(s): " + "; ".join(rendered),+ "Recent user context: " + " | ".join(recent_user[-3:]),+ ]+ if recent_observations:+ parts.append("Recent tool observations: " + " | ".join(recent_observations[-4:]))+ return "\n".join(parts)+++def _message_text(message: dict[str, Any]) -> tuple[str, str]:+ role = str(message.get("role") or "assistant")+ if role == "user":+ return "user", str(message.get("content") or "")+ if role == "tool":+ return "assistant", f"Tool result: {message.get('content') or ''}"+ calls = message.get("tool_calls") or []+ if calls:+ rendered = []+ for call in calls:+ name = call.get("name") or call.get("function", {}).get("name") or "unknown_tool"+ arguments = call.get("arguments") or call.get("function", {}).get("arguments") or {}+ rendered.append(f"{name}({json.dumps(arguments, ensure_ascii=False, sort_keys=True)})")+ return "assistant", "Assistant tool call: " + "; ".join(rendered)+ return "assistant", str(message.get("content") or "")+++def _run_tau2(+ *,+ tau2_repo: Path,+ domain: str,+ split: str,+ task_ids: list[str] | None,+ num_tasks: int | None,+ trials: int,+ max_steps: int,+ max_concurrency: int,+ agent: str,+ user: str,+ agent_llm: str,+ user_llm: str,+ agent_llm_args: dict[str, Any],+ user_llm_args: dict[str, Any],+ seed: int,+ save_to: Path,+):+ _add_tau2_to_path(tau2_repo)+ from tau2.data_model.simulation import RunConfig, TextRunConfig+ from tau2.run import run_domain++ compat_results = _compat_results_path(save_to)+ if save_to.exists():+ save_to.unlink()+ if compat_results.parent.is_dir():+ shutil.rmtree(compat_results.parent)+ config_cls = TextRunConfig if getattr(RunConfig, "__origin__", None) is not None else RunConfig+ result = run_domain(+ config_cls(+ domain=domain,+ task_split_name=split,+ task_ids=task_ids,+ num_tasks=num_tasks,+ agent=agent,+ llm_agent=agent_llm,+ llm_args_agent=agent_llm_args,+ user=user,+ llm_user=user_llm,+ llm_args_user=user_llm_args,+ num_trials=trials,+ max_steps=max_steps,+ save_to=_save_to_arg(save_to),+ max_concurrency=max_concurrency,+ seed=seed,+ log_level="INFO",+ )+ )+ if not save_to.exists() and compat_results.exists():+ shutil.copyfile(compat_results, save_to)+ return result+++def _client(args: argparse.Namespace):+ import openviking as ov++ client = ov.SyncHTTPClient(+ url=args.openviking_url,+ api_key="",+ user=args.openviking_user,+ agent_id=args.openviking_agent_id,+ account=args.openviking_account,+ timeout=args.openviking_timeout,+ extra_headers={},+ )+ client.initialize()+ return client+++def _wait_task(client: Any, task_id: str | None, timeout: int) -> dict[str, Any]:+ if not task_id:+ return {"status": "no_task"}+ deadline = time.time() + timeout+ last = None+ while time.time() < deadline:+ last = client.get_task(task_id)+ status = (last or {}).get("status")+ if status == "completed":+ return last or {"status": status}+ if status in {"failed", "cancelled"}:+ raise RuntimeError(f"OpenViking task {task_id} {status}: {last}")+ time.sleep(2)+ raise TimeoutError(f"OpenViking task {task_id} did not finish within {timeout}s: {last}")+++def _read_memory_text(client: Any, match: Any) -> tuple[str, str | None]:+ try:+ return client.read(getattr(match, "uri", "")), None+ except Exception as exc:+ fallback = getattr(match, "abstract", "") or getattr(match, "overview", "") or ""+ return fallback, f"{type(exc).__name__}: {exc}"+++def _probe_corpus(args: argparse.Namespace, client: Any) -> dict[str, Any]:+ result = client.search(+ query=f"{args.domain} customer service order reservation booking cancellation exchange return update",+ target_uri=args.search_uri,+ limit=args.retrieval_top_k,+ )+ memories = list(getattr(result, "memories", []) or [])+ reads = []+ for match in memories[: args.retrieval_top_k]:+ uri = getattr(match, "uri", "")+ text, read_error = _read_memory_text(client, match)+ row = {+ "uri": uri,+ "score": getattr(match, "score", None),+ "text_chars": len(text),+ "non_empty": bool(str(text).strip()),+ }+ if read_error:+ row["read_error"] = read_error+ reads.append(row)+ return {+ "query": f"{args.domain} customer service order reservation booking cancellation exchange return update",+ "match_count": len(memories),+ "read_non_empty_count": sum(1 for row in reads if row["non_empty"]),+ "matches": reads,+ }+++def _train(args: argparse.Namespace, train_results: Path, corpus_manifest: Path) -> dict[str, Any]:+ if corpus_manifest.is_file() and not args.force_train:+ return json.loads(corpus_manifest.read_text())++ _run_tau2(+ tau2_repo=args.tau2_repo,+ domain=args.domain,+ split=args.train_split_name,+ task_ids=args.train_task_ids,+ num_tasks=args.train_num_tasks,+ trials=1,+ max_steps=args.max_steps,+ max_concurrency=args.max_concurrency,+ agent=args.base_agent,+ user=args.user,+ agent_llm=args.agent_llm,+ user_llm=args.user_llm,+ agent_llm_args=args.agent_llm_args,+ user_llm_args=args.user_llm_args,+ seed=args.seed,+ save_to=train_results,+ )++ data = json.loads(train_results.read_text())+ client = _client(args)+ committed = []+ try:+ for sim in data.get("simulations") or []:+ session_id = f"tau2-{args.domain}-train-{sim.get('task_id')}-trial-{sim.get('trial', 0)}"+ created = client.create_session(session_id=session_id)+ sid = created.get("session_id", session_id)+ for msg in sim.get("messages") or []:+ role, text = _message_text(msg)+ if not text.strip():+ continue+ client.add_message(+ sid,+ role=role,+ parts=[{"type": "text", "text": text}],+ created_at=msg.get("timestamp"),+ )+ result = client.commit_session(sid, telemetry=True)+ task = _wait_task(client, result.get("task_id"), args.openviking_wait_timeout)+ committed.append(+ {+ "session_id": sid,+ "task_id": sim.get("task_id"),+ "commit_status": result.get("status"),+ "openviking_task_id": result.get("task_id"),+ "openviking_task_status": task.get("status"),+ }+ )+ finally:+ client.close()++ client = _client(args)+ try:+ corpus_probe = _probe_corpus(args, client)+ finally:+ client.close()++ manifest = {+ "domain": args.domain,+ "train_results": str(train_results),+ "openviking": {+ "url": args.openviking_url,+ "account": args.openviking_account,+ "user": args.openviking_user,+ "agent_id": args.openviking_agent_id,+ "search_uri": args.search_uri,+ },+ "committed_sessions": committed,+ "committed_session_count": len(committed),+ "corpus_probe": corpus_probe,+ }+ _write_json(corpus_manifest, manifest)+ return manifest+++def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:+ _add_tau2_to_path(args.tau2_repo)++ from tau2.agent.llm_agent import LLMAgent, LLMAgentState+ from tau2.data_model.message import AssistantMessage, MultiToolMessage, SystemMessage+ from tau2.registry import registry+ from tau2.utils.llm_utils import generate++ class OpenVikingMemoryAgent(LLMAgent):+ def get_init_state(self, message_history=None):+ state = super().get_init_state(message_history)+ if args.retrieval_mode in {"first_user", "first_user_prewrite"}:+ state.system_messages.append(+ SystemMessage(role="system", content="<openviking_memory_not_loaded/>")+ )+ return state++ def _retrieve(self, query: str) -> tuple[str, list[dict[str, Any]]]:+ client = _client(args)+ rows: list[dict[str, Any]] = []+ try:+ result = client.search(query=query, target_uri=args.search_uri, limit=args.retrieval_top_k)+ memories = list(getattr(result, "memories", []) or [])+ blocks = []+ for index, match in enumerate(memories[: args.retrieval_top_k], 1):+ uri = getattr(match, "uri", "")+ text, read_error = _read_memory_text(client, match)+ row = {+ "uri": uri,+ "score": getattr(match, "score", None),+ "level": getattr(match, "level", None),+ "text_chars": len(text),+ }+ if read_error:+ row["read_error"] = read_error+ rows.append(row)+ if text.strip():+ blocks.append(f"Memory {index} ({uri}):\n{text.strip()}")+ return "\n\n".join(blocks), rows+ finally:+ client.close()++ def _trace(self, event: dict[str, Any]) -> None:+ with trace_path.open("a", encoding="utf-8") as handle:+ handle.write(json.dumps(event, ensure_ascii=False, sort_keys=True) + "\n")++ @staticmethod+ def _trace_injection_fields(block: str, matches: list[dict[str, Any]]) -> dict[str, Any]:+ injected_count = sum(1 for row in matches if int(row.get("text_chars") or 0) > 0)+ return {+ "injected": bool(block.strip()),+ "injected_count": injected_count if block.strip() else 0,+ "retrieval_action_taken": "retrieve_and_inject" if block.strip() else "retrieve_no_injection",+ }++ def _generate(self, messages):+ def _is_empty_assistant(response) -> bool:+ content = str(getattr(response, "content", "") or "")+ tool_calls = getattr(response, "tool_calls", None) or []+ return not content.strip() and not tool_calls++ try:+ response = generate(+ model=self.llm,+ tools=self.tools,+ messages=messages,+ **self.llm_args,+ )+ if not _is_empty_assistant(response):+ return response+ except json.JSONDecodeError:+ retry_messages = messages + [+ SystemMessage(+ role="system",+ content=(+ "Retry the last assistant step once. If you call a tool, "+ "the tool arguments must be syntactically valid JSON."+ ),+ )+ ]+ else:+ retry_messages = messages + [+ SystemMessage(+ role="system",+ content=(+ "Retry the last assistant step once. Return either a useful "+ "natural language response or a valid tool call; do not return "+ "an empty assistant message."+ ),+ )+ ]+ try:+ response = generate(+ model=self.llm,+ tools=self.tools,+ messages=retry_messages,+ **self.llm_args,+ )+ if not _is_empty_assistant(response):+ return response+ return AssistantMessage(+ role="assistant",+ content="I need to continue with the available task information.",+ raw_data={"openviking_memory_agent_error": "empty_assistant_message"},+ )+ except json.JSONDecodeError as exc:+ return AssistantMessage(+ role="assistant",+ content="I need to continue with the available task information.",+ raw_data={+ "openviking_memory_agent_error": "invalid_tool_call_json",+ "error": str(exc),+ },+ )++ def generate_next_message(self, message, state: LLMAgentState):+ if isinstance(message, MultiToolMessage):+ state.messages.extend(message.tool_messages)+ else:+ state.messages.append(message)+ marker_index = next(+ (+ i+ for i, item in enumerate(state.system_messages)+ if isinstance(item, SystemMessage) and item.content == "<openviking_memory_not_loaded/>"+ ),+ None,+ )+ role = getattr(message, "role", "")+ role_value = getattr(role, "value", role)+ if marker_index is not None and str(role_value) == "user":+ query = str(getattr(message, "content", "") or "")+ block, matches = self._retrieve(query)+ prompt = (+ "No OpenViking memory matched this user request."+ if not block+ else "Use these OpenViking experience memories only when they match the current task:\n\n"+ + block+ )+ state.system_messages[marker_index] = SystemMessage(role="system", content=prompt)+ self._trace(+ {+ "decision_node": "first_user",+ "query": query,+ "match_count": len(matches),+ "matches": matches,+ **self._trace_injection_fields(block, matches),+ }+ )++ assistant_message = self._generate(state.system_messages + state.messages)+ if args.retrieval_mode in {"prewrite", "first_user_prewrite"}:+ tool_calls = list(getattr(assistant_message, "tool_calls", None) or [])+ write_calls = [call for call in tool_calls if _is_write_tool_call(call)]+ if write_calls:+ query = _tool_call_query(write_calls, state.messages)+ block, matches = self._retrieve(query)+ self._trace(+ {+ "decision_node": "before_write_tool_call",+ "query": query,+ "match_count": len(matches),+ "matches": matches,+ **self._trace_injection_fields(block, matches),+ "tool_calls": [+ {+ "name": _tool_call_name(call),+ "arguments": _tool_call_arguments(call),+ }+ for call in write_calls+ ],+ }+ )+ if block:+ prompt = (+ "Before executing the pending write-like tool call, use these "+ "OpenViking experience memories only when they match the current task:\n\n"+ + block+ )+ assistant_message = self._generate(+ state.system_messages+ + state.messages+ + [SystemMessage(role="system", content=prompt)]+ )+ state.messages.append(assistant_message)+ return assistant_message, state++ if AGENT_NAME not in registry.get_agents():+ def create_openviking_memory_agent(tools, domain_policy, **kwargs):+ return OpenVikingMemoryAgent(+ tools=tools,+ domain_policy=domain_policy,+ llm=kwargs.get("llm"),+ llm_args=kwargs.get("llm_args"),+ )++ if hasattr(registry, "register_agent"):+ registry.register_agent(OpenVikingMemoryAgent, AGENT_NAME)+ else:+ registry.register_agent_factory(create_openviking_memory_agent, AGENT_NAME)+++def main() -> int:+ parser = argparse.ArgumentParser(description="Run TAU-2 with OpenViking Memory V2.")+ parser.add_argument("--tau2-repo", type=Path, required=True)+ parser.add_argument("--run-dir", type=Path, required=True)+ parser.add_argument("--corpus-dir", type=Path)+ parser.add_argument("--run-label", required=True)+ parser.add_argument("--strategy-id", default="memory_v2_experience_only")+ parser.add_argument("--domain", required=True)+ parser.add_argument("--train-split-name", default="train")+ parser.add_argument("--eval-split-name", default="test")+ parser.add_argument("--task-id", dest="task_ids", action="append")+ parser.add_argument("--num-tasks", type=int)+ parser.add_argument("--train-task-id", dest="train_task_ids", action="append")+ parser.add_argument("--train-num-tasks", type=int)+ parser.add_argument("--max-steps", type=int, default=200)+ parser.add_argument("--max-concurrency", type=int, default=10)+ parser.add_argument("--seed", type=int, default=300)+ parser.add_argument("--base-agent", default="llm_agent")+ parser.add_argument("--user", default="user_simulator")+ parser.add_argument("--agent-llm", required=True)+ parser.add_argument("--user-llm", required=True)+ parser.add_argument("--agent-llm-args", type=_json, default={})+ parser.add_argument("--user-llm-args", type=_json, default={})+ parser.add_argument("--openviking-url", required=True)+ parser.add_argument("--openviking-account", required=True)+ parser.add_argument("--openviking-user", required=True)+ parser.add_argument("--openviking-agent-id", required=True)+ parser.add_argument("--openviking-timeout", type=float, default=600.0)+ parser.add_argument("--openviking-wait-timeout", type=int, default=600)+ parser.add_argument("--search-uri", required=True)+ parser.add_argument("--retrieval-top-k", type=int, default=4)+ parser.add_argument(+ "--retrieval-mode",+ choices=["first_user", "prewrite", "first_user_prewrite"],+ default="first_user",+ )+ parser.add_argument("--force-train", action="store_true")+ args = parser.parse_args()+ normalize_litellm_env()++ args.tau2_repo = args.tau2_repo.resolve()+ args.run_dir.mkdir(parents=True, exist_ok=True)+ corpus_dir = args.corpus_dir or args.run_dir+ corpus_dir.mkdir(parents=True, exist_ok=True)+ train_results = corpus_dir / "train_results.json"+ corpus_manifest = corpus_dir / "corpus_manifest.json"+ eval_results = args.run_dir / f"{args.run_label}.json"+ trace_path = args.run_dir / f"{args.run_label}.retrieval_trace.jsonl"+ summary_path = args.run_dir / f"{args.run_label}.summary.json"++ corpus = _train(args, train_results, corpus_manifest)+ trace_path.touch()+ _register_memory_agent(args, trace_path)+ _run_tau2(+ tau2_repo=args.tau2_repo,+ domain=args.domain,+ split=args.eval_split_name,+ task_ids=args.task_ids,+ num_tasks=args.num_tasks,+ trials=1,+ max_steps=args.max_steps,+ max_concurrency=args.max_concurrency,+ agent=AGENT_NAME,+ user=args.user,+ agent_llm=args.agent_llm,+ user_llm=args.user_llm,+ agent_llm_args=args.agent_llm_args,+ user_llm_args=args.user_llm_args,+ seed=args.seed,+ save_to=eval_results,+ )+ summary = {+ "run_label": args.run_label,+ "domain": args.domain,+ "strategy_id": args.strategy_id,+ "retrieval_mode": args.retrieval_mode,+ "seed": args.seed,+ "corpus": corpus,+ "eval_results": str(eval_results),+ "retrieval_trace": str(trace_path),+ "metrics": _metrics(eval_results),+ }+ _write_json(summary_path, summary)+ print(json.dumps(summary, ensure_ascii=False, sort_keys=True))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/tau2/scripts/setup_tau2_repo.sh b/benchmark/tau2/scripts/setup_tau2_repo.shnew file mode 100755index 000000000..3cee2655a--- /dev/null+++ b/benchmark/tau2/scripts/setup_tau2_repo.sh@@ -0,0 +1,82 @@+#!/usr/bin/env bash+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"+TAU2_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"+DEFAULT_REPO_DIR="$TAU2_DIR/.external/tau2-bench"+DEFAULT_VENV_DIR="$TAU2_DIR/.venv-tau2"++REPO_URL="${TAU2_REPO_URL:-https://github.com/sierra-research/tau2-bench.git}"+REPO_DIR="${TAU2_REPO:-$DEFAULT_REPO_DIR}"+VENV_DIR="${TAU2_VENV:-$DEFAULT_VENV_DIR}"+REF="${TAU2_REF:-}"+INSTALL=true++while [[ $# -gt 0 ]]; do+ case "$1" in+ --repo-url)+ REPO_URL="$2"+ shift 2+ ;;+ --repo-dir)+ REPO_DIR="$2"+ shift 2+ ;;+ --venv)+ VENV_DIR="$2"+ shift 2+ ;;+ --ref)+ REF="$2"+ shift 2+ ;;+ --no-install)+ INSTALL=false+ shift+ ;;+ --help|-h)+ cat <<'EOF'+Usage:+ benchmark/tau2/scripts/setup_tau2_repo.sh [--repo-url URL] [--repo-dir DIR] [--venv DIR] [--ref REF] [--no-install]++Clones TAU-2 into a local ignored directory and optionally installs it into a+local virtualenv. The script writes benchmark/tau2/.env.tau2 with TAU2_REPO and+TAU2_CLI for the benchmark runner.+EOF+ exit 0+ ;;+ *)+ echo "Unknown argument: $1" >&2+ exit 1+ ;;+ esac+done++mkdir -p "$(dirname "$REPO_DIR")"+if [[ ! -d "$REPO_DIR/.git" ]]; then+ git clone "$REPO_URL" "$REPO_DIR"+else+ git -C "$REPO_DIR" fetch --all --prune+fi++if [[ -n "$REF" ]]; then+ git -C "$REPO_DIR" checkout "$REF"+fi++TAU2_CLI="tau2"+if [[ "$INSTALL" == true ]]; then+ python3 -m venv "$VENV_DIR"+ "$VENV_DIR/bin/python" -m pip install --upgrade pip+ "$VENV_DIR/bin/python" -m pip install -e "$REPO_DIR"+ TAU2_CLI="$VENV_DIR/bin/tau2"+fi++cat > "$TAU2_DIR/.env.tau2" <<EOF+export TAU2_REPO="$REPO_DIR"+export TAU2_CLI="$TAU2_CLI"+EOF++echo "[tau2-setup] repo: $REPO_DIR"+echo "[tau2-setup] cli: $TAU2_CLI"+echo "[tau2-setup] wrote $TAU2_DIR/.env.tau2"+echo "[tau2-setup] next: source $TAU2_DIR/.env.tau2"diff --git a/benchmark/tau2/scripts/tau2_common.py b/benchmark/tau2/scripts/tau2_common.pynew file mode 100755index 000000000..a8b5ce201--- /dev/null+++ b/benchmark/tau2/scripts/tau2_common.py@@ -0,0 +1,268 @@+from __future__ import annotations++import json+import os+import re+import shutil+import subprocess+from datetime import datetime, timezone+from pathlib import Path+from typing import Any++import yaml+++TAU2_DIR = Path(__file__).resolve().parents[1]+REPO_ROOT = TAU2_DIR.parents[1]+CONFIRMATION_AWARE_UPSTREAM_PR = "https://github.com/sierra-research/tau2-bench/pull/297"+CONFIRMATION_AWARE_APPENDIX = """++- If the agent asks you to confirm, authorize, or approve a backend action,+ reply with the requested confirmation but do not emit `###STOP###` in the+ same turn.+- Emit `###STOP###` only after the agent clearly reports that the requested+ backend action has been completed, or when the official transfer /+ out-of-scope rules apply.+"""+++_ENV_PATTERN = re.compile(r"\$\{([^}:]+)(?::-([^}]*))?\}")+++def run_id() -> str:+ return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")+++def normalize_litellm_env() -> dict[str, Any]:+ aliases = []+ if not os.environ.get("OPENAI_API_KEY") and os.environ.get("ARK_API_KEY"):+ os.environ["OPENAI_API_KEY"] = os.environ["ARK_API_KEY"]+ aliases.append("OPENAI_API_KEY<-ARK_API_KEY")+ ark_base = os.environ.get("ARK_BASE_URL")+ openai_base = os.environ.get("OPENAI_API_BASE") or os.environ.get("OPENAI_BASE_URL")+ if not openai_base and ark_base:+ os.environ["OPENAI_API_BASE"] = ark_base+ os.environ["OPENAI_BASE_URL"] = ark_base+ aliases.append("OPENAI_API_BASE<-ARK_BASE_URL")+ elif os.environ.get("OPENAI_API_BASE") and not os.environ.get("OPENAI_BASE_URL"):+ os.environ["OPENAI_BASE_URL"] = os.environ["OPENAI_API_BASE"]+ aliases.append("OPENAI_BASE_URL<-OPENAI_API_BASE")+ elif os.environ.get("OPENAI_BASE_URL") and not os.environ.get("OPENAI_API_BASE"):+ os.environ["OPENAI_API_BASE"] = os.environ["OPENAI_BASE_URL"]+ aliases.append("OPENAI_API_BASE<-OPENAI_BASE_URL")+ return {+ "aliases": aliases,+ "has_api_key": bool(os.environ.get("OPENAI_API_KEY") or os.environ.get("ARK_API_KEY")),+ "has_base_url": bool(+ os.environ.get("OPENAI_API_BASE")+ or os.environ.get("OPENAI_BASE_URL")+ or os.environ.get("ARK_BASE_URL")+ ),+ }+++def render_env(value: Any) -> Any:+ if isinstance(value, str):+ def replace(match: re.Match[str]) -> str:+ name = match.group(1)+ default = match.group(2) or ""+ return os.environ.get(name, default)++ return _ENV_PATTERN.sub(replace, value)+ if isinstance(value, list):+ return [render_env(item) for item in value]+ if isinstance(value, dict):+ return {key: render_env(item) for key, item in value.items()}+ return value+++def deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:+ merged = dict(base)+ for key, value in override.items():+ if (+ key in merged+ and isinstance(merged[key], dict)+ and isinstance(value, dict)+ ):+ merged[key] = deep_merge(merged[key], value)+ else:+ merged[key] = value+ return merged+++def load_config(path: Path) -> dict[str, Any]:+ path = path.expanduser().resolve()+ with path.open("r", encoding="utf-8") as handle:+ raw = yaml.safe_load(handle) or {}+ if not isinstance(raw, dict):+ raise ValueError(f"Config must be a mapping: {path}")++ parent_name = raw.pop("extends", None)+ if parent_name:+ parent_path = (path.parent / str(parent_name)).resolve()+ parent = load_config(parent_path)+ raw = deep_merge(parent, raw)+ return render_env(raw)+++def resolve_path(path_value: str | Path, *, base: Path | None = None) -> Path:+ path = Path(path_value).expanduser()+ if path.is_absolute():+ return path+ return ((base or REPO_ROOT) / path).resolve()+++def output_dir(config: dict[str, Any], configured_run_id: str) -> Path:+ raw = config.get("paths", {}).get("output_dir", TAU2_DIR / "result")+ return resolve_path(raw) / configured_run_id+++def write_json(path: Path, payload: Any) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(+ json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n",+ encoding="utf-8",+ )+++def strategy_ids(config: dict[str, Any]) -> list[str]:+ strategies = config.get("strategies") or []+ if not isinstance(strategies, list):+ raise ValueError("strategies must be a list")+ ids = []+ for item in strategies:+ if not isinstance(item, dict) or not item.get("id"):+ raise ValueError("each strategy must be a mapping with id")+ ids.append(str(item["id"]))+ if len(ids) != len(set(ids)):+ raise ValueError(f"duplicate strategy ids: {ids}")+ return ids+++def domains(config: dict[str, Any]) -> list[str]:+ values = config.get("benchmark", {}).get("domains") or []+ if not isinstance(values, list) or not values:+ raise ValueError("benchmark.domains must be a non-empty list")+ return [str(item) for item in values]+++def tau2_repo(config: dict[str, Any]) -> Path:+ raw = config.get("paths", {}).get("tau2_repo")+ if not raw:+ raise ValueError("paths.tau2_repo is required")+ return resolve_path(raw)+++def tau2_cli(config: dict[str, Any]) -> str:+ return str(config.get("paths", {}).get("tau2_cli") or "tau2")+++def _git_commit(path: Path) -> str | None:+ if not path.exists():+ return None+ completed = subprocess.run(+ ["git", "-C", str(path), "rev-parse", "HEAD"],+ text=True,+ stdout=subprocess.PIPE,+ stderr=subprocess.DEVNULL,+ check=False,+ )+ if completed.returncode != 0:+ return None+ return completed.stdout.strip() or None+++def tau2_context(config: dict[str, Any]) -> dict[str, Any]:+ repo = tau2_repo(config)+ cli = tau2_cli(config)+ return {+ "tau2_repo": str(repo),+ "tau2_repo_exists": repo.exists(),+ "tau2_commit": _git_commit(repo),+ "tau2_cli": cli,+ "tau2_cli_resolved": shutil.which(cli),+ }+++def _prompt_paths(repo: Path) -> list[Path]:+ return [+ repo / "data" / "tau2" / "user_simulator" / "simulation_guidelines.md",+ repo / "data" / "tau2" / "user_simulator" / "simulation_guidelines_tools.md",+ ]+++def _has_confirmation_aware_prompt(prompt_text: str) -> bool:+ normalized = " ".join(prompt_text.split())+ return (+ "reply with the requested confirmation" in normalized+ and "do not emit `###STOP###` in the same turn" in normalized+ )+++def _ensure_confirmation_aware_prompt(repo: Path) -> bool:+ patched = False+ for path in _prompt_paths(repo):+ if not path.is_file():+ continue+ text = path.read_text(encoding="utf-8")+ if _has_confirmation_aware_prompt(text):+ continue+ backup = path.with_suffix(path.suffix + ".openviking.bak")+ if not backup.exists():+ backup.write_text(text, encoding="utf-8")+ path.write_text(text.rstrip() + CONFIRMATION_AWARE_APPENDIX + "\n", encoding="utf-8")+ patched = True+ return patched+++def user_simulator_policy(config: dict[str, Any]) -> str:+ policy = config.get("eval", {}).get("user_simulator_policy", "official")+ policy = str(policy)+ if policy not in {"official", "confirmation_aware"}:+ raise ValueError(+ "eval.user_simulator_policy must be 'official' or 'confirmation_aware'"+ )+ return policy+++def simulator_policy_report(config: dict[str, Any]) -> dict[str, Any]:+ policy = user_simulator_policy(config)+ repo = tau2_repo(config)+ patch_applied = policy == "confirmation_aware" and _ensure_confirmation_aware_prompt(repo)+ patch_mode = "direct_prompt_append" if patch_applied else "none"+ if policy == "confirmation_aware":+ if not patch_applied:+ patch_mode = "upstream_or_existing_prompt"++ prompt_paths = _prompt_paths(repo)+ prompt_text = "\n".join(+ path.read_text(encoding="utf-8") for path in prompt_paths if path.is_file()+ )+ confirmation_aware_prompt = _has_confirmation_aware_prompt(prompt_text)+ supported = policy == "official" or confirmation_aware_prompt+ claim_boundary = "confirmation_aware_user_simulator_prompt"+ if policy == "official":+ claim_boundary = (+ "official_policy_with_confirmation_aware_checkout"+ if confirmation_aware_prompt+ else "official_tau2_user_simulator"+ )+ return {+ "user_simulator_policy": policy,+ "supported": supported,+ "confirmation_aware_prompt_detected": confirmation_aware_prompt,+ "confirmation_aware_upstream_pr": CONFIRMATION_AWARE_UPSTREAM_PR,+ "patch_applied": patch_applied,+ "patch_mode": patch_mode,+ "prompt_files": [str(path) for path in prompt_paths],+ "backup_files": [+ str(path.with_suffix(path.suffix + ".openviking.bak"))+ for path in prompt_paths+ if path.with_suffix(path.suffix + ".openviking.bak").exists()+ ],+ "claim_boundary": claim_boundary,+ }+++def split_file(config: dict[str, Any], domain: str) -> Path:+ return tau2_repo(config) / "data" / "tau2" / "domains" / domain / "split_tasks.json"