Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(tau2/vikingbot): benchmark updates (#2244)
nicoleqiwte0ce670f5cb7
diff --git a/benchmark/tau2/.gitignore b/benchmark/tau2/llm/.gitignoresimilarity index 100%rename from benchmark/tau2/.gitignorerename to benchmark/tau2/llm/.gitignorediff --git a/benchmark/tau2/README.md b/benchmark/tau2/llm/README.mdsimilarity index 80%rename from benchmark/tau2/README.mdrename to benchmark/tau2/llm/README.mdindex dd96b4430..afc519cd5 100644--- a/benchmark/tau2/README.md+++ b/benchmark/tau2/llm/README.md@@ -12,7 +12,7 @@ Category rerank and other harness-only diagnostics are intentionally left out.## Layout```text-benchmark/tau2/+benchmark/tau2/llm/├── config/│ ├── baseline.yaml│ ├── official.yaml@@ -25,9 +25,9 @@ benchmark/tau2/└── run_full_eval.sh```-Generated eval artifacts are written to `benchmark/tau2/result/<run_id>/`.+Generated eval artifacts are written to `benchmark/tau2/llm/result/<run_id>/`.Memory corpus artifacts are cached outside the run id at-`benchmark/tau2/result/memory_corpora/` by default.+`benchmark/tau2/llm/result/memory_corpora/` by default.## Quick Start@@ -59,8 +59,8 @@ For a local one-command setup, clone and install TAU-2 into ignored benchmarkdirectories:```bash-benchmark/tau2/scripts/setup_tau2_repo.sh-source benchmark/tau2/.env.tau2+benchmark/tau2/llm/scripts/setup_tau2_repo.sh+source benchmark/tau2/llm/.env.tau2```For PR-B-compatible reproduction, pin the TAU-2 checkout to a ref that includes@@ -68,9 +68,9 @@ the confirmation-aware text-user-simulator prompt. The original PR-B evidenceused the open TAU-2 fix PR head (`79dbf0c18ac7637aedf869cb3122babcd57aaf17`):```bash-benchmark/tau2/scripts/setup_tau2_repo.sh \+benchmark/tau2/llm/scripts/setup_tau2_repo.sh \--ref refs/pull/297/head-source benchmark/tau2/.env.tau2+source benchmark/tau2/llm/.env.tau2```Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).@@ -78,7 +78,7 @@ Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tPlan the default benchmark without running TAU-2:```bash-python benchmark/tau2/scripts/run_eval.py --config benchmark/tau2/config/baseline.yaml --plan-only+python benchmark/tau2/llm/scripts/run_eval.py --config benchmark/tau2/llm/config/baseline.yaml --plan-only```Add `--preflight` or `--strict-preflight` when you want the runner to write a@@ -87,8 +87,8 @@ small environment/config check next to the run plan.After setup, verify the local TAU-2 link and write a one-cell run plan:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/baseline.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/baseline.yaml \--strict-preflight \--domain retail \--strategy-id memory_v2_experience_only \@@ -99,8 +99,8 @@ benchmark/tau2/run_full_eval.sh \Plan a one-cell Memory V2 pre-write smoke:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/baseline.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/baseline.yaml \--domain retail \--strategy-id memory_v2_prewrite \--num-tasks 1 \@@ -110,8 +110,8 @@ benchmark/tau2/run_full_eval.sh \Plan a one-cell trajectory memory smoke:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/trajectory.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/trajectory.yaml \--domain retail \--strategy-id memory_v2_trajectory_view \--num-tasks 1 \@@ -122,8 +122,8 @@ benchmark/tau2/run_full_eval.sh \Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/baseline.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/baseline.yaml \--execute```@@ -143,15 +143,15 @@ the same confirmation-aware simulator policy but does not require fixed fixturesRun one bootstrap pass per domain:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml \--domain retail \--run-id fixed_first_user_bootstrap_retail \--strict-preflight \--execute-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml \--domain airline \--run-id fixed_first_user_bootstrap_airline \--strict-preflight \@@ -161,31 +161,31 @@ benchmark/tau2/run_full_eval.sh \Then convert each bootstrap `results.json` into a fixture:```bash-RETAIL_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_retail/memory_cells/fixed_first_user_bootstrap_retail_retail_no_memory_r1/fixed_first_user_bootstrap_retail_retail_no_memory_r1.json-AIRLINE_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_airline/memory_cells/fixed_first_user_bootstrap_airline_airline_no_memory_r1/fixed_first_user_bootstrap_airline_airline_no_memory_r1.json+RETAIL_RESULTS=benchmark/tau2/llm/result/fixed_first_user_bootstrap_retail/memory_cells/fixed_first_user_bootstrap_retail_retail_no_memory_r1/fixed_first_user_bootstrap_retail_retail_no_memory_r1.json+AIRLINE_RESULTS=benchmark/tau2/llm/result/fixed_first_user_bootstrap_airline/memory_cells/fixed_first_user_bootstrap_airline_airline_no_memory_r1/fixed_first_user_bootstrap_airline_airline_no_memory_r1.json-python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \+python benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py \--repo "$TAU2_REPO" \--results-json "$RETAIL_RESULTS" \--domain retail \--task-split-name test \- --output benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json \+ --output benchmark/tau2/llm/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json \--require-full-split-python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \+python benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py \--repo "$TAU2_REPO" \--results-json "$AIRLINE_RESULTS" \--domain airline \--task-split-name test \- --output benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json \+ --output benchmark/tau2/llm/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json \--require-full-split```Export the generated fixture paths for subsequent strict runs:```bash-export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json"-export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"+export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json"+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"```### 2. Run smoke and full PR-B matrix@@ -193,8 +193,8 @@ export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_firsFirst run one tiny end-to-end smoke against a clean local OpenViking service:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \--domain retail \--strategy-id new_traj_fixed_first_user_prewrite \--num-tasks 1 \@@ -207,15 +207,15 @@ benchmark/tau2/run_full_eval.sh \Then run the full PR-B matrix:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \--run-id prb_content_matrix_new_prompt_full8 \--strict-preflight \--execute```The main result is written to-`benchmark/tau2/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.+`benchmark/tau2/llm/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.Per-cell execution records live under `cell_results/`, raw TAU-2 result JSONlives under `memory_cells/`, and corpus identity / generated memory checks liveunder `memory_corpora/`.@@ -223,8 +223,8 @@ under `memory_corpora/`.For a small E2E smoke, keep both the eval and train slices tiny:```bash-benchmark/tau2/run_full_eval.sh \- --config benchmark/tau2/config/baseline.yaml \+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/baseline.yaml \--domain retail \--strategy-id memory_v2_experience_only \--num-tasks 1 \diff --git a/benchmark/tau2/config/baseline.yaml b/benchmark/tau2/llm/config/baseline.yamlsimilarity index 95%rename from benchmark/tau2/config/baseline.yamlrename to benchmark/tau2/llm/config/baseline.yamlindex 0672322bf..31b054dc9 100644--- a/benchmark/tau2/config/baseline.yaml+++ b/benchmark/tau2/llm/config/baseline.yaml@@ -18,10 +18,10 @@ benchmark:paths:tau2_repo: ${TAU2_REPO:-data/external_benchmarks/tau2-bench}tau2_cli: ${TAU2_CLI:-tau2}- output_dir: benchmark/tau2/result+ output_dir: benchmark/tau2/llm/result# Corpus writes are expensive and should be reused across eval run ids when# the train split and memory prompt/config did not change.- corpus_cache_dir: benchmark/tau2/result/memory_corpora+ corpus_cache_dir: benchmark/tau2/llm/result/memory_corporaeval:# Default OpenViking TAU-2 memory evidence uses the fixed-first-user full8diff --git a/benchmark/tau2/config/fixed_first_user_bootstrap.yaml b/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yamlsimilarity index 100%rename from benchmark/tau2/config/fixed_first_user_bootstrap.yamlrename to benchmark/tau2/llm/config/fixed_first_user_bootstrap.yamldiff --git a/benchmark/tau2/config/no_memory.yaml b/benchmark/tau2/llm/config/no_memory.yamlsimilarity index 100%rename from benchmark/tau2/config/no_memory.yamlrename to benchmark/tau2/llm/config/no_memory.yamldiff --git a/benchmark/tau2/config/official.yaml b/benchmark/tau2/llm/config/official.yamlsimilarity index 100%rename from benchmark/tau2/config/official.yamlrename to benchmark/tau2/llm/config/official.yamldiff --git a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yamlsimilarity index 86%rename from benchmark/tau2/config/prb_content_matrix_new_prompt.yamlrename to benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yamlindex 55607c479..be128c15d 100644--- a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml+++ b/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml@@ -24,7 +24,7 @@ strategies:retrieval_mode: first_userretrieval_top_k: 4first_user_inject_top_k: 4- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_traj_fixed_prewrite_onlylabel: PR-B new trajectory fixed-count prewrite top2@@ -40,7 +40,7 @@ strategies:retrieval_top_k: 4prewrite_retrieval_top_k: 2prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_traj_fixed_first_user_prewritelabel: PR-B new trajectory fixed-count first-user top4 + prewrite top2@@ -57,7 +57,7 @@ strategies:first_user_inject_top_k: 4prewrite_retrieval_top_k: 2prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_exp_fixed_first_userlabel: PR-B new experience fixed-count first-user top2@@ -72,7 +72,7 @@ strategies:retrieval_mode: first_userretrieval_top_k: 2first_user_inject_top_k: 2- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_exp_fixed_prewrite_onlylabel: PR-B new experience fixed-count prewrite top2@@ -88,7 +88,7 @@ strategies:retrieval_top_k: 2prewrite_retrieval_top_k: 2prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_exp_fixed_first_user_prewritelabel: PR-B new experience fixed-count first-user + prewrite top2@@ -105,7 +105,7 @@ strategies:first_user_inject_top_k: 2prewrite_retrieval_top_k: 2prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_traj_4000_prewrite_onlylabel: PR-B new trajectory 4000-char prewrite@@ -122,7 +122,7 @@ strategies:prewrite_retrieval_top_k: 8prewrite_inject_top_k: 8memory_inject_max_chars: 4000- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: new_exp_4000_first_user_prewritelabel: PR-B new experience 4000-char first-user + prewrite@@ -140,4 +140,4 @@ strategies:prewrite_retrieval_top_k: 8prewrite_inject_top_k: 8memory_inject_max_chars: 4000- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/config/prb_scope_fairness.yaml b/benchmark/tau2/llm/config/prb_scope_fairness.yamlsimilarity index 84%rename from benchmark/tau2/config/prb_scope_fairness.yamlrename to benchmark/tau2/llm/config/prb_scope_fairness.yamlindex 8d044d216..bee9ed94f 100644--- a/benchmark/tau2/config/prb_scope_fairness.yaml+++ b/benchmark/tau2/llm/config/prb_scope_fairness.yaml@@ -17,7 +17,7 @@ strategies:- id: no_memory_generic_scopelabel: TAU-2 no-memory same-seed baseline with generic memory scope promptmemory_backend: none- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: trajectory_top4_first_user_prewrite_generic_scopelabel: Trajectory top4 first-user + pre-write top2 with generic memory scope prompt@@ -31,4 +31,4 @@ strategies:first_user_inject_top_k: 4prewrite_retrieval_top_k: 2prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/config/prewrite.yaml b/benchmark/tau2/llm/config/prewrite.yamlsimilarity index 100%rename from benchmark/tau2/config/prewrite.yamlrename to benchmark/tau2/llm/config/prewrite.yamldiff --git a/benchmark/tau2/config/scope_prompts/generic_memory_scope.md b/benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mdsimilarity index 100%rename from benchmark/tau2/config/scope_prompts/generic_memory_scope.mdrename to benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/config/trajectory.yaml b/benchmark/tau2/llm/config/trajectory.yamlsimilarity index 88%rename from benchmark/tau2/config/trajectory.yamlrename to benchmark/tau2/llm/config/trajectory.yamlindex 2538d2639..aa0396e12 100644--- a/benchmark/tau2/config/trajectory.yaml+++ b/benchmark/tau2/llm/config/trajectory.yaml@@ -20,7 +20,7 @@ strategies:train_memory_mode: experience_onlysearch_memory_type: trajectoriesretrieval_mode: first_user- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: memory_v2_trajectory_prewrite_onlylabel: OpenViking Memory V2 trajectory pre-write-only recallmemory_backend: openviking@@ -45,7 +45,7 @@ strategies:train_memory_mode: experience_onlysearch_memory_type: trajectoriesretrieval_mode: prewrite- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- id: memory_v2_trajectory_prewrite_scopelabel: OpenViking Memory V2 trajectory first-user + pre-write recall with scope promptmemory_backend: openviking@@ -54,4 +54,4 @@ strategies:train_memory_mode: experience_onlysearch_memory_type: trajectoriesretrieval_mode: first_user_prewrite- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/run_full_eval.sh b/benchmark/tau2/llm/run_full_eval.shsimilarity index 91%rename from benchmark/tau2/run_full_eval.shrename to benchmark/tau2/llm/run_full_eval.shindex 728f3ce46..a3b63ab8f 100755--- a/benchmark/tau2/run_full_eval.sh+++ b/benchmark/tau2/llm/run_full_eval.sh@@ -2,7 +2,7 @@set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"-REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"PYTHON_BIN="${PYTHON_BIN:-python3}"CONFIG="$SCRIPT_DIR/config/baseline.yaml"EXECUTE=false@@ -40,7 +40,7 @@ while [[ $# -gt 0 ]]; do--help|-h)cat <<'EOF'Usage:- benchmark/tau2/run_full_eval.sh [--config PATH] [--run-id ID] [--execute] [--preflight]+ benchmark/tau2/llm/run_full_eval.sh [--config PATH] [--run-id ID] [--execute] [--preflight]Without --execute the script only writes run_plan artifacts.EOFdiff --git a/benchmark/tau2/scripts/build_fixed_first_user_fixture.py b/benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.pysimilarity index 99%rename from benchmark/tau2/scripts/build_fixed_first_user_fixture.pyrename to benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.pyindex 8b4c0cc49..e9576b861 100644--- a/benchmark/tau2/scripts/build_fixed_first_user_fixture.py+++ b/benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py@@ -94,7 +94,7 @@ def build_fixture(def main() -> None:parser = argparse.ArgumentParser()- parser.add_argument("--repo", type=Path, default=Path("benchmark/tau2/.external/tau2-bench"))+ parser.add_argument("--repo", type=Path, default=Path("benchmark/tau2/llm/.external/tau2-bench"))parser.add_argument("--results-json", type=Path, required=True)parser.add_argument("--domain", required=True)parser.add_argument("--task-split-name", default="test")diff --git a/benchmark/tau2/scripts/run_eval.py b/benchmark/tau2/llm/scripts/run_eval.pysimilarity index 100%rename from benchmark/tau2/scripts/run_eval.pyrename to benchmark/tau2/llm/scripts/run_eval.pydiff --git a/benchmark/tau2/scripts/run_memory_v2_eval.py b/benchmark/tau2/llm/scripts/run_memory_v2_eval.pysimilarity index 99%rename from benchmark/tau2/scripts/run_memory_v2_eval.pyrename to benchmark/tau2/llm/scripts/run_memory_v2_eval.pyindex 26b2d69e7..dd9ed0633 100644--- a/benchmark/tau2/scripts/run_memory_v2_eval.py+++ b/benchmark/tau2/llm/scripts/run_memory_v2_eval.py@@ -15,7 +15,7 @@ from typing import Anyfrom tau2_common import assert_tau2_results_complete, normalize_litellm_envAGENT_NAME = "openviking_memory_agent"-REPO_ROOT = Path(__file__).resolve().parents[3]+REPO_ROOT = Path(__file__).resolve().parents[4]WRITE_TOOL_PREFIXES = ("toggle_","enable_",diff --git a/benchmark/tau2/scripts/setup_tau2_repo.sh b/benchmark/tau2/llm/scripts/setup_tau2_repo.shsimilarity index 90%rename from benchmark/tau2/scripts/setup_tau2_repo.shrename to benchmark/tau2/llm/scripts/setup_tau2_repo.shindex 815f9bc0f..e4d0320be 100755--- a/benchmark/tau2/scripts/setup_tau2_repo.sh+++ b/benchmark/tau2/llm/scripts/setup_tau2_repo.sh@@ -37,10 +37,10 @@ while [[ $# -gt 0 ]]; do--help|-h)cat <<'EOF'Usage:- benchmark/tau2/scripts/setup_tau2_repo.sh [--repo-url URL] [--repo-dir DIR] [--venv DIR] [--ref REF] [--no-install]+ benchmark/tau2/llm/scripts/setup_tau2_repo.sh [--repo-url URL] [--repo-dir DIR] [--venv DIR] [--ref REF] [--no-install]Clones TAU-2 into a local ignored directory and optionally installs it into a-local virtualenv. The script writes benchmark/tau2/.env.tau2 with TAU2_REPO and+local virtualenv. The script writes benchmark/tau2/llm/.env.tau2 with TAU2_REPO andTAU2_CLI for the benchmark runner.EOFexit 0diff --git a/benchmark/tau2/scripts/tau2_common.py b/benchmark/tau2/llm/scripts/tau2_common.pysimilarity index 99%rename from benchmark/tau2/scripts/tau2_common.pyrename to benchmark/tau2/llm/scripts/tau2_common.pyindex 4f4505bd8..6a657be39 100755--- a/benchmark/tau2/scripts/tau2_common.py+++ b/benchmark/tau2/llm/scripts/tau2_common.py@@ -13,7 +13,7 @@ from typing import Anyimport yamlTAU2_DIR = Path(__file__).resolve().parents[1]-REPO_ROOT = TAU2_DIR.parents[1]+REPO_ROOT = TAU2_DIR.parents[2]CONFIRMATION_AWARE_UPSTREAM_PR = "https://github.com/sierra-research/tau2-bench/pull/297"CONFIRMATION_AWARE_APPENDIX = """diff --git a/benchmark/tau2/vikingbot/.gitignore b/benchmark/tau2/vikingbot/.gitignorenew file mode 100644index 000000000..c7821be79--- /dev/null+++ b/benchmark/tau2/vikingbot/.gitignore@@ -0,0 +1,16 @@+# Generated eval artifacts+result/+result_*/+trajectory/+trajectory_*/++# External tau2-bench checkout (see README / setup_env.sh)+tau2-bench/++# Reports & logs+full_test_report*.txt+*.log++# Python caches+__pycache__/+*.pycdiff --git a/benchmark/tau2/vikingbot/README.md b/benchmark/tau2/vikingbot/README.mdnew file mode 100644index 000000000..d9d573c67--- /dev/null+++ b/benchmark/tau2/vikingbot/README.md@@ -0,0 +1,321 @@+# VikingBot × tau2-bench Runner++This folder runs the **full VikingBot agent** (`bot/vikingbot` `AgentLoop`) end-to-end+on [tau2-bench](https://github.com/sierra-research/tau2-bench) tasks, then commits the+resulting trajectories back into OpenViking memory so the agent can **self-improve across+epochs** (cold start → memory-augmented runs).++> **Memory is extracted only from the `train` split.** Each epoch runs both splits, but only+> `train` trajectories are committed to OpenViking memory. The `test` split is **held out** and+> used purely to measure the task-success improvement once that (train-derived) memory is injected+> — so the reported gains reflect learning transferred from train to test, with no test-set leakage.++It is a sibling to the harness in [`../llm/`](../llm/README.md). Both are multi-turn and exercise+OpenViking memory extraction + retrieval; they differ in *which agent* drives the tasks:++- **[`../llm/`](../llm/README.md)** uses tau2-bench's **native ReAct agent**, wired to OpenViking+ memory, to measure the effect of that memory on task performance.+- **`vikingbot/`** (this folder) is an **end-to-end, self-improving agent** evaluation: it runs the+ full VikingBot agent loop on the tasks and commits trajectories back into memory so the agent+ improves across epochs.++The pipeline is: **run tasks → evaluate reward → commit train trajectories to memory** (see+`run_full_test.sh`).++---++## Install & Config++One step sets up everything. `setup_env.sh` creates a fresh `.venv` at the OpenViking repo+root, clones tau2-bench into `./tau2-bench` (external dependency, gitignored), installs+openviking + vikingbot (`pip install -e .`, which also runs the Cargo build) and tau2-bench,+installs smolagents, then activates the venv and exports the runtime env vars:++```bash+source benchmark/tau2/vikingbot/setup_env.sh # first run: install, then activate + export+source benchmark/tau2/vikingbot/setup_env.sh --reinstall # rebuild the .venv from scratch+```++Safe to `source` in every new shell: the install phase runs only when the venv is missing;+later sources just activate and re-export.++It exports `PYTHONPATH` for `openviking` + `bot/vikingbot`, `TAU2_DATA_ROOT`+(defaults to `./tau2-bench/data/tau2`), `OPENVIKING_CONFIG_FILE`, and the user-simulator LLM+env vars. Override any of these by exporting before sourcing:++- `TAU2_BENCH_ROOT` — tau2-bench checkout location (if it lives elsewhere)+- `TAU2_BENCH_REPO` / `TAU2_BENCH_REF` — git URL / ref to clone (e.g. pin a specific checkout)+- `VIKINGBOT_ROOT`+- `ARK_API_KEY` (mapped to `OPENAI_API_KEY`), `OPENAI_API_BASE`++The tau2 **user simulator** talks to an OpenAI-compatible endpoint — set `ARK_API_KEY` (e.g.+Doubao through volcengine ARK) before sourcing, or the simulator will fail. The user-simulator+model is configured in [`tau2_env/tau2_environment.py`](tau2_env/tau2_environment.py).++> Note: the sibling `llm/` harness ([`../llm/README.md`](../llm/README.md)) pins a tau2-bench ref+> with a confirmation-aware user-simulator prompt (sierra-research/tau2-bench#297). Set+> `TAU2_BENCH_REF` to a comparable checkout if you want results aligned with that protocol.++Then start the OpenViking server with the bot enabled:++```bash+openviking-server --config "${OPENVIKING_CONFIG_FILE}" --with-bot+```+++---++## One-click full run (recommended)++Run one epoch — **1 train run + 8 test runs in parallel** (`--test-repeats`, default 8) — then+evaluate (test accuracy is averaged over the repeats) and commit the train trajectories to memory:++```bash+bash benchmark/tau2/vikingbot/run_full_test.sh --domain airline --epoch 0 --result-dir result+```++The async memory commit happens on the server, so wait for it to+finish before starting the next epoch. The per-domain report is appended to+`full_test_report_<domain>.txt`.++Multi-epoch examples (cold start → memory-augmented epochs):++```bash+bash benchmark/tau2/vikingbot/run_airline_2epochs.sh+```++---++## Run each step separately++### 1) Run tasks (train / test)++**train** runs **once per epoch** — one trajectory per task, which is what gets committed to memory+as the experience corpus. A single pass mirrors real usage, where the agent learns from one attempt+at each task.++**test** runs **8 times per epoch (in parallel) and is averaged**. Agent execution is stochastic, so+averaging several independent repeats gives a more confident accuracy estimate. Each repeat is a+separate `--try-no`; `run_eval_reward.sh` scores one repeat, and the per-repeat accuracies are+averaged. The one-click `run_full_test.sh` does all of this for you (`--test-repeats`, default 8).++```bash+# train: a single run (try 0)+bash scripts/run_tau2_domain.sh \+ --domain airline --split train --epoch 0 --try-no 0 \+ --result-dir result --concurrency 5 --use-continue --agent-id airline_v0++# test: 8 independent runs (try 0..7)+for t in 0 1 2 3 4 5 6 7; do+ bash scripts/run_tau2_domain.sh \+ --domain airline --split test --epoch 0 --try-no "$t" \+ --result-dir result --concurrency 5 --use-continue --agent-id airline_v0+done+```++Results are written to `result/<domain>_<split>/task_<n>_<epoch>_<try>_trajectory.json`,+and full message dumps to the mirrored `trajectory/...` path.++### 2) Evaluate rewards++```bash+bash scripts/run_eval_reward.sh result/airline_train 0 0+bash scripts/run_eval_reward.sh result/airline_test 0 0+```++### 3) Commit trajectories to memory++> VikingBot natively commits a task's trajectory into memory automatically as soon as that task finishes.+> For these experiments that auto-commit is **disabled**, so that all+> tasks within an epoch (train + test, run in parallel) execute under identical memory conditions — no run sees memory written by a sibling run mid-experiment.+> Instead, the commit is performed explicitly as a separate, controlled step via the script below (run once between epochs).++```bash+python scripts/commit_trajectory_to_memory.py \+ --input result/airline_train \+ --domain airline_v0 \+ --pattern "*_0_0_trajectory.json" \+ --include-eval-result+```++---++## How the runner adapts VikingBot for tau2++There are **two layers** of adaptation:++1. **Runner-level** (this folder only, no core edits) — swap the agent's tool set over to the tau2+ environment tools, gate OpenViking memory by epoch (cold start vs. memory-augmented), and commit+ train trajectories between epochs. Detailed below.+2. **Core `bot/vikingbot`** edits — required for per-domain workspace isolation and for reading+ *agent* (experience) memory instead of *user* memory. See+ [Core `bot/vikingbot` changes for tau2](#core-botvikingbot-changes-for-tau2) below.++### Runner-level adaptations:++- **Tool registry swap** — the tau2 environment tools are injected into VikingBot's `ToolRegistry`+ via `agent.tools.register(Tau2Tool(...))`, so the agent drives the task through tau2's own tools+ (plus `communicate_with_user` and `done`). `openviking_memory_commit` is **always** unregistered+ here — that is the mechanism that disables VikingBot's per-task auto-commit (see step 3 above).+ - **`--keep-default-tools` controls memory availability, tied to the epoch.** The flag decides+ whether VikingBot's built-in memory tools — **OpenViking memory tools** — stay+ registered, and whether agent-experience memory is retrieved into the system prompt+ (`ov_tools_enable`). `run_full_test.sh` sets it by epoch: **epoch 0 omits the flag**, so all+ built-in memory tools are unregistered (only tau2 tools remain) and no memory is injected — a clean+ **cold-start / no-memory** run; **epoch > 0 passes the flag**, so the memory tools and retrieved+ experiences are available (memory-augmented).+- **Epoch-based memory commit** — `commit_trajectory_to_memory.py` writes train trajectories+ (optionally only failed ones, via `--only-wrong`) into OpenViking memory between epochs.+++### Core `bot/vikingbot` changes for tau2++Two behaviours in the VikingBot core+(`bot/vikingbot`) had to be changed for tau2:++1. **Per-domain workspace isolation via `agent_id`**+ By default VikingBot do not support agent id+ isolation in local mode. In order to easily test in local, changes are made so that each tau2 domain (airline, retail, …) read/write its own OpenViking namespace so experiences learned on one domain don't leak into another.+2. **Read *agent* (experience) memory at system-prompt build time**+ By default VikingBot retrieves *user* memory when assembling the system message. For tau2 self-improvement we want to isolate the effect of the agent's own accumulated **experience** agent memory instead.+++### Change 1 — `agent_id` isolation in `VikingClient` (`openviking_mount/ov_server.py`)++In `local` mode the client used to hardcode `agent_id = "default"` for every caller, so all+domains shared one namespace. It now reads the incoming `agent_id` and only falls back to+`"default"` when the id is a normal session key (which contains `__`, e.g. `cli__default`):++```diff+ if openviking_config.mode == "local":+- self.client = ov.AsyncHTTPClient(url=openviking_config.server_url)+- self.agent_id = "default"++ if agent_id is None or "__" in agent_id:++ self.client = ov.AsyncHTTPClient(url=openviking_config.server_url)++ self.agent_id = "default"++ else:++ self.client = ov.AsyncHTTPClient(++ url=openviking_config.server_url,++ agent_id=agent_id,++ )++ self.agent_id = agent_id+ self.account_id = "default"+ self.user_id = "default"+ self.admin_user_id = "default"+ return+```++(The `agent_id is None` guard is needed because `agent_id` defaults to `None`, and `"__" in None`+would raise `TypeError`.)++`search_experiences` then resolves the experience URI from this per-instance `agent_id` (a clean+domain id like `airline_v1` contains a single `_`), instead of always using the global config+`agent_id`:++```diff+ async def search_experiences(self, query: str, limit: int = 5) -> list[Any]:+ """用 query 检索 agent experience 记忆。"""+ effective_agent_id = self.openviking_config.agent_id or "default"++ if self.agent_id and "_" in self.agent_id:++ effective_agent_id = self.agent_id+ exp_uri = f"viking://agent/{effective_agent_id}/memories/experiences/"+ result = await self.search(query=query, target_uri=exp_uri, limit=limit)+ return result.get("memories", [])+```+++#### How `agent_id` flows from the runner into the core++The domain id is threaded all the way down into the OpenViking client:++```+run_full_test.sh AGENT_ID=${DOMAIN}_v1 # e.g. airline_v1, retail_v1+ └─ run_tau2_domain.sh --agent-id airline_v1+ └─ vikingbot_tau2_runner.py+ build_messages(..., memory_users=agent_id) # memory_users == "airline_v1"+ └─ context.py _build_user_memory+ workspace_id = memory_users # workspace_id == "airline_v1"+ └─ memory.py get_viking_experience_context(query, workspace_id)+ └─ _create_client(workspace_id)+ └─ VikingClient.create(agent_id=workspace_id)+ └─ ov_server.py VikingClient.__init__ / search_experiences+ viking://agent/airline_v1/memories/experiences/+```++So `--agent-id airline_v1` ⇒ the agent reads/writes `viking://agent/airline_v1/...`, giving each+domain an isolated workspace.++### Change 2 — system prompt reads *agent experience* memory (`agent/context.py`)++`_build_user_memory` (called by `build_messages` when assembling the system prompt) used to call+`get_viking_memory_context`, which retrieves **user** memory. For tau2 it now binds `workspace_id`+to the passed-in `memory_users` (the domain `agent_id`) and calls `get_viking_experience_context`+to retrieve the agent's **experience** memory:++```diff++ # Viking agent memory (only if ov tools are enabled)+ if ov_tools_enable:+ start = _time.time()+ # Use provided memory_users or fall back to [sender_id]+ search_user_ids = memory_users if memory_users else [sender_id]+- viking_memory = await self.memory.get_viking_memory_context(+- current_message=current_message,+- workspace_id=workspace_id,+- sender_id=sender_id,+- user_ids=search_user_ids,+- )++ workspace_id = memory_users++++ viking_memory = await self.memory.get_viking_experience_context(++ query=current_message,++ workspace_id=workspace_id,++ )+ logger.info(f"viking_memory={viking_memory}")+ cost = round(_time.time() - start, 2)+ logger.info(+- f"[READ_USER_MEMORY]: cost {cost}s, memory={viking_memory[:50] if viking_memory else 'None'}"++ f"[READ_AGENT_MEMORY]: cost {cost}s, memory={viking_memory[:50] if viking_memory else 'None'}"+ )+ if viking_memory:+ self.latest_relevant_memories = viking_memory+- parts.append(f"## openviking_search(query=[user_query])\n{viking_memory}")++ parts.append(f"## openviking_search(query=[user_query])\n ## Agent Experience (relevant to this task)\n {viking_memory}")+ else:+ self.latest_relevant_memories = None+```++The key line is `workspace_id = memory_users`: the runner passes the domain id as `memory_users`,+and it becomes the `agent_id` used to open the (isolated) OpenViking client downstream.++### Change 3 — supporting edits in `agent/memory.py`++`get_viking_experience_context` is the experience-retrieval path that Change 2 now calls.++Retrieval limits in `get_viking_experience_context` were also tuned for tau2 (fewer experiences,+more characters per experience):++```diff+- experiences = await client.search_experiences(query, limit=5)++ experiences = await client.search_experiences(query, limit=2)+ ...+ return await self._parse_viking_memory(+- experiences, client, min_score=0.3, max_chars=2000++ experiences, client, min_score=0.3, max_chars=10000+ )+```+++---++## Layout++- `setup_env.sh` — environment setup (PYTHONPATH, tau2 data root, simulator LLM)+- `run_full_test.sh` — full pipeline for one epoch (run → eval → commit)+- `run_airline_2epochs.sh` — multi-epoch example (cold start → memory-augmented epochs)+- `tau2_env/` — tau2 environment integration (`tau2_environment.py`, `tau2_tool_provider.py`)+- `scripts/`+ - `vikingbot_tau2_runner.py` — runs a single tau2 task through the VikingBot agent loop+ - `run_tau2_domain.sh` — runs all tasks in a `{domain}_{split}` slice with bounded concurrency+ - `run_eval_reward.sh` — average reward over a result folder+ - `commit_trajectory_to_memory.py` — commit trajectories into OpenViking memorydiff --git a/benchmark/tau2/vikingbot/run_airline_2epochs.sh b/benchmark/tau2/vikingbot/run_airline_2epochs.shnew file mode 100755index 000000000..6971c2148--- /dev/null+++ b/benchmark/tau2/vikingbot/run_airline_2epochs.sh@@ -0,0 +1,44 @@+#!/usr/bin/env bash+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"+LOG_FILE="${SCRIPT_DIR}/run_airline_2epochs.log"++# --- env setup (inline, no source needed) ---+source "${SCRIPT_DIR}/setup_env.sh"++CONCURRENCY=5+DOMAIN=airline+RESULT_DIR=result++log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG_FILE}"; }++log "===== Start airline 2-epoch run ====="++# ---------- Epoch 0 ----------+log ">>> Epoch 0 start (cold start, no memory)"+bash "${SCRIPT_DIR}/run_full_test.sh" \+ --domain "${DOMAIN}" \+ --epoch 0 \+ --concurrency "${CONCURRENCY}" \+ --result-dir "${RESULT_DIR}"+log ">>> Epoch 0 done"++# Wait for OpenViking server to finish async memory processing+WAIT_SECS=9000+log ">>> Waiting ${WAIT_SECS}s for server async memory commit to finish..."+sleep "${WAIT_SECS}"+log ">>> Wait done, starting Epoch 1"++# ---------- Epoch 1 ----------+log ">>> Epoch 1 start (with memory from epoch 0 train)"+bash "${SCRIPT_DIR}/run_full_test.sh" \+ --domain "${DOMAIN}" \+ --epoch 1 \+ --concurrency "${CONCURRENCY}" \+ --result-dir "${RESULT_DIR}"+log ">>> Epoch 1 done"++log "===== All done. Results in: ${SCRIPT_DIR}/${RESULT_DIR} ====="+log "===== Report: ${SCRIPT_DIR}/full_test_report_${DOMAIN}.txt ====="diff --git a/benchmark/tau2/vikingbot/run_full_test.sh b/benchmark/tau2/vikingbot/run_full_test.shnew file mode 100755index 000000000..0e0fa4c2f--- /dev/null+++ b/benchmark/tau2/vikingbot/run_full_test.sh@@ -0,0 +1,168 @@+#!/usr/bin/env bash+set -euo pipefail++# Run one epoch of a tau2 domain:+# - train: a single run (used to extract experience into memory; mirrors real usage)+# - test : TEST_REPEATS independent runs in parallel, averaged (agent execution is stochastic,+# so averaging repeats gives a more confident accuracy estimate)+# Then evaluate rewards and commit the train trajectories to memory.+# Usage:+# bash run_full_test.sh --domain airline [--epoch 0] [--test-repeats 8] [--result-dir result] [--concurrency N] [--commit|--no-commit]++CUR_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+SCRIPTS_DIR="${CUR_DIR}/scripts"++PIDS=()+kill_tree() {+ local pid="$1"+ local sig="$2"+ local children=""+ children=$(pgrep -P "${pid}" 2>/dev/null || true)+ for child in ${children}; do+ kill_tree "${child}" "${sig}"+ done+ kill "-${sig}" "${pid}" 2>/dev/null || true+}+cleanup() {+ local exit_code=$?+ if [[ ${#PIDS[@]} -gt 0 ]]; then+ echo "[run_full_test] Caught interrupt, stopping child processes..."+ for pid in "${PIDS[@]}"; do+ kill_tree "${pid}" "TERM"+ done+ sleep 1+ for pid in "${PIDS[@]}"; do+ kill_tree "${pid}" "KILL"+ done+ fi+ exit "${exit_code}"+}+trap cleanup INT TERM++DOMAIN=""+EPOCH=0+TEST_REPEATS=8+RESULT_DIR="result"+CONCURRENCY=1+DO_COMMIT=1++while [[ $# -gt 0 ]]; do+ case "$1" in+ --domain)+ DOMAIN="$2"; shift 2 ;;+ --epoch)+ EPOCH="$2"; shift 2 ;;+ --test-repeats)+ TEST_REPEATS="$2"; shift 2 ;;+ --result-dir)+ RESULT_DIR="$2"; shift 2 ;;+ --concurrency)+ CONCURRENCY="$2"; shift 2 ;;+ --no-commit)+ DO_COMMIT=0; shift ;;+ --commit)+ DO_COMMIT=1; shift ;;+ -h|--help)+ echo "Usage: bash run_full_test.sh --domain DOMAIN [--epoch N] [--test-repeats N] [--result-dir DIR] [--concurrency N] [--commit|--no-commit]"+ exit 0 ;;+ *)+ echo "Unknown argument: $1" >&2; exit 1 ;;+ esac+done++if [[ -z "${DOMAIN}" ]]; then+ echo "Missing --domain" >&2+ exit 1+fi++KEEP_DEFAULT_TOOLS_FLAG=""+ONLY_WRONG_FLAG=""+if [[ "${EPOCH}" -gt 0 ]]; then+ KEEP_DEFAULT_TOOLS_FLAG="--keep-default-tools"+ ONLY_WRONG_FLAG="--only-wrong"+fi++if [[ "${RESULT_DIR}" = /* ]]; then+ OUTPUT_ROOT="${RESULT_DIR}"+else+ OUTPUT_ROOT="${CUR_DIR}/${RESULT_DIR}"+fi+TRAIN_DIR="${OUTPUT_ROOT}/${DOMAIN}_train"+TEST_DIR="${OUTPUT_ROOT}/${DOMAIN}_test"+AGENT_ID=${DOMAIN}_v0+AGENT_ID_FLAG="--agent-id ${AGENT_ID}"+# train runs once per epoch; test repeats are averaged.+TRAIN_TRY_NO=0++echo "[run_full_test] Start ${DOMAIN}: 1 train run + ${TEST_REPEATS} test run(s), in parallel..."++# train: a single run per epoch+bash "${SCRIPTS_DIR}/run_tau2_domain.sh" \+ --domain "${DOMAIN}" \+ --split train \+ --epoch "${EPOCH}" \+ --try-no "${TRAIN_TRY_NO}" \+ --result-dir "${RESULT_DIR}" \+ --concurrency "${CONCURRENCY}" \+ ${KEEP_DEFAULT_TOOLS_FLAG} \+ --use-continue ${AGENT_ID_FLAG} &+PIDS+=("$!")++# test: TEST_REPEATS independent runs (try-no 0 .. TEST_REPEATS-1), averaged at eval time+for ((t=0; t<TEST_REPEATS; t++)); do+ bash "${SCRIPTS_DIR}/run_tau2_domain.sh" \+ --domain "${DOMAIN}" \+ --split test \+ --epoch "${EPOCH}" \+ --try-no "${t}" \+ --result-dir "${RESULT_DIR}" \+ --concurrency "${CONCURRENCY}" \+ ${KEEP_DEFAULT_TOOLS_FLAG} \+ --use-continue ${AGENT_ID_FLAG} &+ PIDS+=("$!")+done++for pid in "${PIDS[@]}"; do+ wait "${pid}" || true+done+PIDS=()++REPORT_PATH="${CUR_DIR}/full_test_report_${DOMAIN}.txt"+{+ echo "==== Tau2 Full Test Report ===="+ echo "Domain: ${DOMAIN}"+ echo "Epoch: ${EPOCH} Test repeats: ${TEST_REPEATS}"+ echo "Result dir: ${RESULT_DIR}"+ echo "Timestamp: $(date '+%Y-%m-%d %H:%M:%S')"+ echo+ echo "-- Train Reward (try ${TRAIN_TRY_NO}) --"+ bash "${SCRIPTS_DIR}/run_eval_reward.sh" "${TRAIN_DIR}" "${EPOCH}" "${TRAIN_TRY_NO}" || true+ echo+ echo "-- Test Reward (averaged over ${TEST_REPEATS} repeat(s)) --"+ acc_file="$(mktemp)"+ for ((t=0; t<TEST_REPEATS; t++)); do+ echo "[test try=${t}]"+ test_out="$(bash "${SCRIPTS_DIR}/run_eval_reward.sh" "${TEST_DIR}" "${EPOCH}" "${t}" || true)"+ echo "${test_out}"+ echo "${test_out}" | awk -F': ' '/Average reward:/ {print $2}' | tail -n 1 >> "${acc_file}"+ echo+ done+ echo -n "Test average accuracy over ${TEST_REPEATS} repeat(s): "+ awk '{sum += $1; n++} END {if (n > 0) printf "%.6f\n", sum / n; else print "n/a"}' "${acc_file}"+ rm -f "${acc_file}"+} | tee -a "${REPORT_PATH}"++echo "[run_full_test] Report saved to: ${REPORT_PATH}"++# train: commit trajectories to extract memory (consumed by the next epoch)+if [[ "${DO_COMMIT}" -eq 1 ]]; then+ echo "[run_full_test] Commit train trajectories to memory..."+ python "${SCRIPTS_DIR}/commit_trajectory_to_memory.py" \+ --input "${TRAIN_DIR}" \+ --domain "${AGENT_ID}" \+ --pattern "*_${EPOCH}_${TRAIN_TRY_NO}_trajectory.json" \+ --include-eval-result \+ ${ONLY_WRONG_FLAG}+else+ echo "[run_full_test] Skip commit (--no-commit)"+fidiff --git a/benchmark/tau2/vikingbot/scripts/commit_trajectory_to_memory.py b/benchmark/tau2/vikingbot/scripts/commit_trajectory_to_memory.pynew file mode 100644index 000000000..c8af81da9--- /dev/null+++ b/benchmark/tau2/vikingbot/scripts/commit_trajectory_to_memory.py@@ -0,0 +1,204 @@+#!/usr/bin/env python3+"""Commit runner trajectories into OpenViking memory.++This script reads trajectory JSON files produced by vikingbot_tau2_runner.py+and commits a minimal conversation (user -> assistant) into OpenViking.++Usage:+ python3 commit_trajectory_to_memory.py --input /path/to/result_dir+ python3 commit_trajectory_to_memory.py --input /path/to/file.json+"""++from __future__ import annotations++import argparse+import asyncio+import json+from pathlib import Path+from typing import Any, Iterable++from vikingbot.config.loader import load_config+from vikingbot.openviking_mount.ov_server import VikingClient+++def _iter_files(root: Path, pattern: str) -> Iterable[Path]:+ if root.is_file():+ yield root+ return+ for path in sorted(root.glob(pattern)):+ if path.is_file():+ yield path+++def _build_messages(+ data: dict[str, Any],+ include_eval_result: bool,+) -> list[dict[str, str]]:+ messages: list[dict[str, str]] = []++ system_prompt = data.get("system_prompt") or ""+ if system_prompt:+ messages.append({"role": "user", "content": f"system:\n{system_prompt}"})++ user_text = data.get("user_prompt") or data.get("user_query") or ""+ if user_text:+ messages.append({"role": "user", "content": user_text})++ tools_used = data.get("tools_used") or []+ if isinstance(tools_used, list):+ for tool_info in tools_used:+ if not isinstance(tool_info, dict):+ continue+ tool_name = tool_info.get("tool_name", "")+ args = tool_info.get("args", "")+ assistant_content = tool_name+ if args:+ assistant_content = f"tool-call: \n name: {tool_name}\n arguments: {args}"+ if assistant_content:+ messages.append({"role": "assistant", "content": assistant_content})++ result = tool_info.get("result")+ if result is not None:+ messages.append({"role": "user", "content": f"tool-response: \n{result}"})+ messages.append({"role": "assistant", "content": data.get("final_content", "")})+ reward = data.get("reward")+ success = bool(reward == 1)+ result = f"task_success: {success}"+ if include_eval_result and data.get("evaluation_result") is not None:+ evaluation_result = data.get("evaluation_result")+ if not isinstance(evaluation_result, str):+ evaluation_result = json.dumps(evaluation_result, ensure_ascii=False)+ result += f"\n evaluation report: {evaluation_result}"+ messages.append({"role": "user", "content": result})+ return messages+++def _get_session_id(data: dict[str, Any], prefix: str | None) -> str | None:+ session_id = data.get("session_id")+ if session_id:+ return f"{prefix}{session_id}" if prefix else session_id+ data_split = data.get("data_split")+ task_no = data.get("task_no")+ if data_split is not None and task_no is not None:+ base = f"tau2_{data_split}_{task_no}"+ return f"{prefix}{base}" if prefix else base+ return None+++async def _commit_single(+ client: VikingClient,+ path: Path,+ user_id: str,+ session_prefix: str | None,+ dry_run: bool,+ only_wrong: bool,+ include_eval_result: bool,+) -> tuple[Path, bool, str]:+ try:+ data = json.loads(path.read_text(encoding="utf-8"))+ except Exception as exc:+ return path, False, f"Failed to load JSON: {exc}"+ if only_wrong and data.get("reward") == 1:+ return path, False, "Skipped: reward == 1"+ messages = _build_messages(data, include_eval_result)+ if not messages:+ return path, False, "No messages to commit"++ session_id = _get_session_id(data, session_prefix)+ if not session_id:+ return path, False, "Missing session_id/data_split/task_no"++ if dry_run:+ print(messages)+ return path, True, f"DRY RUN: would commit {len(messages)} messages to {session_id}"+ await client.commit(session_id, messages, user_id)++ return path, True, f"Committed {len(messages)} messages to {session_id}"+++async def main_async() -> int:+ parser = argparse.ArgumentParser(description="Commit runner trajectories to OpenViking")+ parser.add_argument("--input", required=True, help="Trajectory file or directory")+ parser.add_argument(+ "--pattern",+ default="*.json",+ help="Glob pattern when input is a directory (default: *.json)",+ )+ parser.add_argument(+ "--user-id",+ default=None,+ help="User id for commit (default: ov.conf admin_user_id)",+ )+ parser.add_argument(+ "--session-prefix",+ default=None,+ help="Prefix to prepend to session_id",+ )+ parser.add_argument(+ "--domain",+ default="telecom",+ help="Use as agent_id when creating VikingClient",+ )+ parser.add_argument(+ "--dry-run",+ action="store_true",+ help="Only print what would be committed",+ )+ parser.add_argument(+ "--only-wrong",+ action="store_true",+ help="Only commit trajectories with reward != 1",+ )+ parser.add_argument(+ "--include-eval-result",+ action="store_true",+ help="Include evaluation_result in the final user message",+ )+ args = parser.parse_args()++ root = Path(args.input).expanduser()+ if not root.exists():+ print(f"Input not found: {root}")+ return 1++ config = load_config()+ user_id = args.user_id or config.ov_server.admin_user_id+ if not user_id:+ print("Missing user_id (pass --user-id or set ov.conf admin_user_id)")+ return 1++ client = await VikingClient.create(args.domain)+ print(client.agent_id)+ print(client.get_agent_space_name("default"))+ print(client.client._agent_id)+ files = list(_iter_files(root, args.pattern))+ if not files:+ print("No files matched.")+ return 1++ ok = 0+ for path in files:+ _, success, msg = await _commit_single(+ client,+ path,+ user_id=user_id,+ session_prefix=args.session_prefix,+ dry_run=args.dry_run,+ only_wrong=args.only_wrong,+ include_eval_result=args.include_eval_result,+ )+ status = "OK" if success else "SKIP"+ print(f"[{status}] {path.name} - {msg}")+ if success:+ ok += 1++ print(f"Committed {ok}/{len(files)} files")+ return 0+++def main() -> None:+ raise SystemExit(asyncio.run(main_async()))+++if __name__ == "__main__":+ main()diff --git a/benchmark/tau2/vikingbot/scripts/run_eval_reward.sh b/benchmark/tau2/vikingbot/scripts/run_eval_reward.shnew file mode 100755index 000000000..67516a169--- /dev/null+++ b/benchmark/tau2/vikingbot/scripts/run_eval_reward.sh@@ -0,0 +1,61 @@+#!/usr/bin/env bash+set -euo pipefail++# Evaluate average reward for a given result folder, epoch, and try_no.+# Usage: bash run_eval_reward.sh <result_dir> [epoch] [try_no]++RESULT_DIR=${1:-""}+EPOCH=${2:-0}+TRY_NO=${3:-0}++if [[ -z "${RESULT_DIR}" ]]; then+ echo "Usage: bash run_eval_reward.sh <result_dir> [epoch] [try_no]" >&2+ exit 1+fi++if [[ ! -d "${RESULT_DIR}" ]]; then+ echo "Result dir not found: ${RESULT_DIR}" >&2+ exit 1+fi++RESULT_DIR="${RESULT_DIR}" EPOCH="${EPOCH}" TRY_NO="${TRY_NO}" python3 - <<'PY'+import glob+import json+import os+import sys++result_dir = os.environ.get("RESULT_DIR")+epoch = os.environ.get("EPOCH")+try_no = os.environ.get("TRY_NO")++pattern = os.path.join(result_dir, f"task_*_{epoch}_{try_no}_trajectory.json")+files = sorted(glob.glob(pattern))++if not files:+ print(f"No files matched: {pattern}")+ sys.exit(1)++total = 0.0+count = 0+missing = 0++for path in files:+ try:+ with open(path, "r", encoding="utf-8") as f:+ data = json.load(f)+ reward = data.get("reward")+ if reward is None:+ missing += 1+ continue+ total += float(reward)+ count += 1+ except Exception:+ missing += 1++avg = total / count if count else 0.0+print(f"Result dir: {result_dir}")+print(f"Epoch: {epoch} Try: {try_no}")+print(f"Matched files: {len(files)}")+print(f"Used rewards: {count} Missing/invalid: {missing}")+print(f"Average reward: {avg:.6f}")+PYdiff --git a/benchmark/tau2/vikingbot/scripts/run_tau2_domain.sh b/benchmark/tau2/vikingbot/scripts/run_tau2_domain.shnew file mode 100755index 000000000..9be4bc79b--- /dev/null+++ b/benchmark/tau2/vikingbot/scripts/run_tau2_domain.sh@@ -0,0 +1,119 @@+#!/usr/bin/env bash+set -euo pipefail++# Run VikingBot tau2 runner for all tasks in {domain}_{train|test} split.+# Usage:+# bash run_tau2_domain.sh --domain telecom --split train --epoch 0 --try-no 0 \+# --keep-default-tools --result-dir result --use-continue --concurrency 5+++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"+RUNNER="${SCRIPT_DIR}/vikingbot_tau2_runner.py"++DOMAIN=""+SPLIT=""+EPOCH=0+TRY_NO=0+KEEP_DEFAULT_TOOLS=0+RESULT_DIR="result"+USE_CONTINUE=0+AGENT_ID=""+CONCURRENCY=1++while [[ $# -gt 0 ]]; do+ case "$1" in+ --domain)+ DOMAIN="$2"; shift 2 ;;+ --split)+ SPLIT="$2"; shift 2 ;;+ --epoch)+ EPOCH="$2"; shift 2 ;;+ --try-no)+ TRY_NO="$2"; shift 2 ;;+ --keep-default-tools)+ KEEP_DEFAULT_TOOLS=1; shift 1 ;;+ --result-dir)+ RESULT_DIR="$2"; shift 2 ;;+ --use-continue)+ USE_CONTINUE=1; shift 1 ;;+ --agent-id)+ AGENT_ID="$2"; shift 2 ;;+ --concurrency)+ CONCURRENCY="$2"; shift 2 ;;+ -h|--help)+ echo "Usage: bash run_tau2_domain.sh --domain DOMAIN --split train|test --epoch N --try-no N --keep-default-tools --result-dir DIR --use-continue --agent-id ID --concurrency N"; exit 0 ;;+ *)+ echo "Unknown argument: $1" >&2; exit 1 ;;+ esac+done++if [[ -z "${DOMAIN}" || ( "${SPLIT}" != "train" && "${SPLIT}" != "test" ) ]]; then+ echo "Usage: bash run_tau2_domain.sh --domain DOMAIN --split train|test --epoch N --try-no N --keep-default-tools --result-dir DIR --use-continue --agent-id ID --concurrency N" >&2+ exit 1+fi++DATA_ROOT="${TAU2_DATA_ROOT}"+SPLIT_TASKS_JSON="${DATA_ROOT}/domains/${DOMAIN}/split_tasks.json"++if [[ "${RESULT_DIR}" = /* ]]; then+ OUTPUT_ROOT="${RESULT_DIR}/${DOMAIN}_${SPLIT}"+else+ OUTPUT_ROOT="${REPO_ROOT}/${RESULT_DIR}/${DOMAIN}_${SPLIT}"+fi+mkdir -p "${OUTPUT_ROOT}"++if [[ ! -f "${SPLIT_TASKS_JSON}" ]]; then+ echo "Split file not found: ${SPLIT_TASKS_JSON}" >&2+ exit 1+fi++TASK_COUNT=$(SPLIT_TASKS_JSON="${SPLIT_TASKS_JSON}" SPLIT_NAME="${SPLIT}" python3 - <<'PY'+import json+import os++path = os.environ.get("SPLIT_TASKS_JSON")+split_name = os.environ.get("SPLIT_NAME")+with open(path, "r", encoding="utf-8") as f:+ data = json.load(f)+print(len(data[split_name]))+PY+)++echo "${DOMAIN}_${SPLIT} task count: ${TASK_COUNT}, concurrency: ${CONCURRENCY}"++# build reusable flags+KEEP_DEFAULT_TOOLS_FLAG=""+CONTINUE_FLAG=""+AGENT_ID_FLAG=""+[[ "${KEEP_DEFAULT_TOOLS}" == "1" ]] && KEEP_DEFAULT_TOOLS_FLAG="--keep-default-tools"+[[ "${USE_CONTINUE}" == "1" ]] && CONTINUE_FLAG="--continue"+[[ -n "${AGENT_ID}" ]] && AGENT_ID_FLAG="--agent-id ${AGENT_ID}"++# Use a tmpdir as a semaphore: each running task holds a slot file.+SEM_DIR=$(mktemp -d)+trap 'rm -rf "${SEM_DIR}"' EXIT++active_count() { ls "${SEM_DIR}" 2>/dev/null | wc -l | tr -d ' '; }++for ((task_no=0; task_no < TASK_COUNT; task_no++)); do+ out_path="${OUTPUT_ROOT}/task_${task_no}_${EPOCH}_${TRY_NO}_trajectory.json"+ echo "[${DOMAIN}_${SPLIT}] task_no=${task_no} epoch=${EPOCH} try_no=${TRY_NO}"++ # wait for a free slot+ while [[ $(active_count) -ge ${CONCURRENCY} ]]; do sleep 0.5; done++ SLOT="${SEM_DIR}/${task_no}"+ touch "${SLOT}"+ (+ python "${RUNNER}" \+ --data-split "${DOMAIN}_${SPLIT}" \+ --task-no "${task_no}" \+ --output "${out_path}" ${AGENT_ID_FLAG} ${KEEP_DEFAULT_TOOLS_FLAG} ${CONTINUE_FLAG} \+ || echo "[WARN] task_no=${task_no} failed, skipping"+ rm -f "${SLOT}"+ ) &+done++wait || true+rm -rf "${SEM_DIR}"diff --git a/benchmark/tau2/vikingbot/scripts/vikingbot_tau2_runner.py b/benchmark/tau2/vikingbot/scripts/vikingbot_tau2_runner.pynew file mode 100644index 000000000..39aa9de41--- /dev/null+++ b/benchmark/tau2/vikingbot/scripts/vikingbot_tau2_runner.py@@ -0,0 +1,309 @@+#!/usr/bin/env python3+"""Run a single tau2-bench task with VikingBot AgentLoop + Tau2 tools.++Key points:+1) Tau2BenchEnv is initialized once and exposes tools via Tau2BenchToolProvider.call_tool().+2) VikingBot already has its own multi-iteration agent loop, so we call it once.+3) We register Tau2 tools into VikingBot ToolRegistry and let the agent decide.+"""++from __future__ import annotations++import argparse+import asyncio+import json+import sys+from pathlib import Path+from typing import Any++SCRIPT_DIR = Path(__file__).resolve().parent+REPO_ROOT = SCRIPT_DIR.parent+if str(REPO_ROOT) not in sys.path:+ sys.path.insert(0, str(REPO_ROOT))++from tau2_env.tau2_tool_provider import Tau2BenchToolProvider, load_task_id++try:+ from vikingbot.agent.loop import AgentLoop+ from vikingbot.agent.tools.base import Tool, ToolContext+ from vikingbot.bus.queue import MessageBus+ from vikingbot.cli.commands import _init_bot_data, _make_provider+ from vikingbot.config.loader import ensure_config+ from vikingbot.config.schema import SessionKey+ from vikingbot.sandbox.manager import SandboxManager+ from vikingbot.session.manager import SessionManager+ from vikingbot.utils.helpers import get_source_workspace_path+except ImportError as exc:+ raise RuntimeError(+ "Failed to import vikingbot. Make sure OpenViking/vikingbot is installed "+ "or available on PYTHONPATH."+ ) from exc+++class Tau2Tool(Tool):+ """Bridge tau2 tool schema into VikingBot Tool interface."""++ def __init__(self, schema: dict[str, Any], provider: Tau2BenchToolProvider):+ self._schema = schema+ self._provider = provider+ function_def = schema.get("function", {}) if isinstance(schema, dict) else {}+ self._name = function_def.get("name", "")+ self._description = function_def.get("description", "")+ self._parameters = function_def.get("parameters", {})++ @property+ def name(self) -> str:+ return self._name++ @property+ def description(self) -> str:+ return self._description++ @property+ def parameters(self) -> dict[str, Any]:+ return self._parameters++ async def execute(self, tool_context: ToolContext, **kwargs: Any) -> str:+ return self._provider.call_tool(self._name, kwargs)+++def _build_agent(config_path: str | None) -> AgentLoop:+ config = ensure_config(Path(config_path).expanduser() if config_path else None)+ _init_bot_data(config)++ bus = MessageBus()+ session_manager = SessionManager(config.bot_data_path)++ sandbox_parent_path = config.workspace_path+ source_workspace_path = get_source_workspace_path()+ sandbox_manager = SandboxManager(config, sandbox_parent_path, source_workspace_path)++ provider = _make_provider(config)++ return AgentLoop(+ bus=bus,+ provider=provider,+ workspace=config.workspace_path,+ model=config.agents.model,+ # max_iterations=config.agents.max_tool_iterations,+ max_iterations=30,+ memory_window=config.agents.memory_window,+ brave_api_key=config.tools.web.search.api_key or None,+ exa_api_key=None,+ gen_image_model=config.agents.gen_image_model,+ exec_config=config.tools.exec,+ cron_service=None,+ session_manager=session_manager,+ sandbox_manager=sandbox_manager,+ config=config,+ eval=True,+ mcp_servers=None,+ )+++# Stable markers around the injected memory block in VikingBot's prompt wrapper.+# The wrapper looks like:+# ## Current Time: <system time>\n\n---\n\n## Current Session+# Channel: cli\n\n---\n\n<MEMORY>---\n\nReply in the same language ... User's query:+# The current time is generated by the system at runtime and may be anything, so we+# anchor only on the session header and the trailing reply instruction — never on the time.+MEMORY_PROMPT_PREFIX = "## Current Session\nChannel: cli\n\n---\n\n"+MEMORY_PROMPT_SUFFIX = (+ "---\n\nReply in the same language as the user's query, ignoring the language of "+ "the reference materials. User's query:"+)+++def _extract_memory_content(content: str) -> str | None:+ """Extract the injected memory block from VikingBot's prompt wrapper.++ Time-independent: locates the block between the session header and the trailing+ reply instruction, regardless of the (system-generated) current time.+ """+ start = content.find(MEMORY_PROMPT_PREFIX)+ end = content.rfind(MEMORY_PROMPT_SUFFIX)+ if start == -1 or end == -1:+ return None+ start += len(MEMORY_PROMPT_PREFIX)+ if start > end:+ return None+ return content[start:end]+++++async def _run_agent(+ agent: AgentLoop,+ system_prompt: str,+ user_prompt: str,+ session_key: SessionKey,+ sender_id: str | None,+ agent_id: str | None,+ keep_default_tools: bool,+ messages_output_path: Path | None,+) -> tuple[str | None, str | None, dict[str, int], int, str | None]:+ messages = await agent.context.build_messages(+ history=[],+ current_message=user_prompt,+ session_key=session_key,+ ov_tools_enable=keep_default_tools,+ media=None,+ profile_user_list=[],+ memory_users=agent_id,+ )+ if system_prompt:+ messages.insert(1, {"role": "system", "content": system_prompt})++ memory_content = None+ if len(messages) > 2 and isinstance(messages[2].get("content"), str):+ memory_content = _extract_memory_content(messages[2]["content"])++ result = await agent._run_agent_loop(+ messages=messages,+ session_key=session_key,+ publish_events=False,+ sender_id=sender_id,+ ov_tools_enable=keep_default_tools,+ )+ if messages_output_path is not None:+ messages_output_path.parent.mkdir(parents=True, exist_ok=True)+ with open(messages_output_path, "w", encoding="utf-8") as f:+ json.dump(messages, f, ensure_ascii=False, indent=2)+ print(f"Saved Full Messages: {messages_output_path}")+ return (*result, memory_content)+++def _derive_messages_path(output_path: Path) -> Path:+ parent_parts = [part.replace("result", "trajectory") for part in output_path.parent.parts]+ base_path = Path(*parent_parts) / output_path.name++ if base_path.name.endswith("_trajectory.json"):+ return base_path.with_name(base_path.name.replace("_trajectory.json", "_messages.json"))+ return base_path.with_name(f"{base_path.stem}_messages{base_path.suffix}")+++def main() -> None:+ parser = argparse.ArgumentParser(description="VikingBot tau2-bench runner")+ parser.add_argument("--data-split", required=True, help="e.g. telecom_test")+ parser.add_argument("--task-no", type=int, required=True, help="index in split list")+ parser.add_argument("--output", default="./result/tau2_trajectory.json")+ parser.add_argument(+ "--continue",+ dest="continue_run",+ action="store_true",+ help="If output file exists, skip running and exit",+ )+ parser.add_argument("--sender", default="tau2_user")+ parser.add_argument("--agent-id", default="", help="airline_v0 domain split workspace")+ parser.add_argument("--session", default=None)+ parser.add_argument("--config", default=None, help="ov.conf path (optional)")+ parser.add_argument(+ "--keep-default-tools",+ action="store_true",+ help="Keep VikingBot default tools (default: only tau2 tools)",+ )+ parser.add_argument(+ "--max-iterations",+ type=int,+ default=None,+ help="Override agent max iterations for this run",+ )+ args = parser.parse_args()++ output_path = Path(args.output)+ if output_path.exists():+ if args.continue_run:+ print(f"[Runner] Output exists, skip run: {output_path}")+ return+ print(f"[Runner] Output exists, will overwrite: {output_path}")++ domain, task_id = load_task_id(args.data_split, args.task_no)+ print("[Runner] Initializing tau2 environment...", flush=True)+ provider = Tau2BenchToolProvider(domain, task_id)+ provider.reset()++ user_query = provider.user_query+ policy = provider.policy+ ground_truth = provider.ground_truth++ agent = _build_agent(args.config)+ if args.max_iterations is not None:+ agent.max_iterations = args.max_iterations++ if not args.keep_default_tools:+ for tool_name in list(agent.tools.tool_names):+ # print(tool_name)+ agent.tools.unregister(tool_name)++ agent.tools.unregister("openviking_memory_commit")+ for schema in provider.list_openai_tools():+ agent.tools.register(Tau2Tool(schema, provider))++ instructions = []+ if policy:+ instructions.append(policy)+ instructions.append("Use the provided tools to interact with the environment.")+ if args.keep_default_tools:+ instructions.append("Before you attend to customer, you MUST read relevant agent memory that stores experiences distilled from similar tasks and carefully learn them.")+ instructions.append(+ "If you need to communicate with the user, you MUST call tool `communicate_with_user`."+ )+ instructions.append("When the task is finished or terminated, call tool `done` first and output an ending content without using any tool calling for the next round to exit.")++ system_prompt = "\n".join(instructions)+ user_prompt = user_query++ session_id = args.session or f"tau2_{args.data_split}_{args.task_no}"+ session_key = SessionKey(type="cli", channel_id="tau2", chat_id=session_id)++ messages_output_path = _derive_messages_path(output_path)++ final_content, final_reasoning_content, tools_used, token_usage, iteration, memory_content = asyncio.run(+ _run_agent(+ agent,+ system_prompt,+ user_prompt,+ session_key,+ args.sender,+ args.agent_id,+ args.keep_default_tools,+ messages_output_path,+ )+ )++ reward = None+ evaluation_result = None+ if provider.env is not None:+ try:+ reward, evaluation_result = provider.env.env._get_reward()+ except Exception:+ pass++ trajectory = {+ "data_split": args.data_split,+ "task_no": args.task_no,+ "domain": domain,+ "task_id": task_id,+ "user_query": user_query,+ "policy": policy,+ "ground_truth": ground_truth,+ "system_prompt": system_prompt,+ "user_prompt": user_prompt,+ "final_content": final_content,+ "reward": reward,+ "evaluation_result": evaluation_result,+ "tools_used": tools_used,+ "token_usage": token_usage,+ "iterations": iteration,+ "memory": memory_content,+ }++ output_path.parent.mkdir(parents=True, exist_ok=True)+ with open(output_path, "w", encoding="utf-8") as f:+ json.dump(trajectory, f, ensure_ascii=False, indent=2)++ print(f"Saved: {output_path}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/tau2/vikingbot/setup_env.sh b/benchmark/tau2/vikingbot/setup_env.shnew file mode 100755index 000000000..8d9536074--- /dev/null+++ b/benchmark/tau2/vikingbot/setup_env.sh@@ -0,0 +1,198 @@+#!/usr/bin/env bash+# One-step environment setup for the VikingBot × tau2-bench runner.+#+# Does everything in a single step:+# 1. creates a fresh .venv at the OpenViking repo root (if missing)+# 2. clones tau2-bench into ./tau2-bench (if missing; external dependency)+# 3. installs openviking + vikingbot (pip install -e .[bot] -> runs the Cargo build;+# the [bot] extra provides prompt_toolkit/gradio/mcp/... needed by the runner and+# `openviking-server --with-bot`)+# 4. ensures the ragfs_python native binding is built (via maturin) and bundled into+# openviking/lib/ (the editable install can skip it under pip build isolation)+# 5. installs tau2-bench with the gym extra (pip install -e ./tau2-bench[gym] -> gymnasium)+# 6. installs smolagents+# 7. activates the venv and exports the runtime env vars+#+# Usage:+# source setup_env.sh # install on first run, then activate + export+# source setup_env.sh --reinstall # delete .venv and rebuild from scratch+#+# Safe to source repeatedly: the install phase runs only when the venv is missing+# (or when --reinstall is passed); later sources just activate + export.+#+# Overridable via env vars (export before sourcing):+# TAU2_BENCH_ROOT tau2-bench checkout location (default ./tau2-bench)+# TAU2_BENCH_REPO git URL to clone tau2-bench (default sierra-research/tau2-bench)+# TAU2_BENCH_REF git ref/branch/tag to check out after clone (default: repo default)+# VIKINGBOT_ROOT vikingbot package dir (default REPO_ROOT/bot)+# OPENVIKING_CONFIG_FILE, OPENAI_API_KEY / ARK_API_KEY, OPENAI_API_BASE++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# OpenViking repo root (this folder lives at benchmark/tau2/vikingbot/).+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"+VENV="${REPO_ROOT}/.venv"+SETUP_MARKER="${VENV}/.tau2_setup_complete"++# tau2-bench checkout (external dependency; gitignored). Resolved early so the+# install phase can clone/install it and the exports can derive TAU2_DATA_ROOT.+TAU2_BENCH_ROOT="${TAU2_BENCH_ROOT:-${SCRIPT_DIR}/tau2-bench}"+TAU2_BENCH_REPO="${TAU2_BENCH_REPO:-https://github.com/sierra-research/tau2-bench}"++# Detect source vs execute so errors abort cleanly without killing the user's shell.+(return 0 2>/dev/null) && _SOURCED=1 || _SOURCED=0+_abort() { echo "[setup_env] ERROR: $*" >&2; if [[ "${_SOURCED}" -eq 1 ]]; then return 1; else exit 1; fi; }++# --- parse args ---+REINSTALL=0+for _arg in "$@"; do+ case "${_arg}" in+ --reinstall|--force) REINSTALL=1 ;;+ -h|--help)+ echo "Usage: source setup_env.sh [--reinstall]"+ { [[ "${_SOURCED}" -eq 1 ]] && return 0; } 2>/dev/null || exit 0 ;;+ esac+done++# --- install phase (one-time) ---------------------------------------------------+_setup_install() {+ if [[ "${REINSTALL}" -eq 1 && -d "${VENV}" ]]; then+ echo "[setup_env] --reinstall: removing ${VENV}"+ rm -rf "${VENV}"+ fi++ if [[ -f "${SETUP_MARKER}" && "${REINSTALL}" -eq 0 ]]; then+ return 0 # already installed; nothing to do+ fi++ if [[ ! -f "${VENV}/bin/activate" ]]; then+ echo "[setup_env] Creating venv at ${VENV}"+ python3 -m venv "${VENV}" || { echo "[setup_env] failed to create venv"; return 1; }+ fi++ local PY="${VENV}/bin/python"++ echo "[setup_env] Upgrading pip/setuptools/wheel"+ "${PY}" -m pip install --upgrade pip setuptools wheel || return 1++ # tau2-bench: external checkout. Clone it if it isn't present yet.+ if [[ ! -d "${TAU2_BENCH_ROOT}" ]]; then+ if ! command -v git >/dev/null 2>&1; then+ echo "[setup_env] 'git' not found; cannot clone tau2-bench into ${TAU2_BENCH_ROOT}"; return 1+ fi+ echo "[setup_env] Cloning tau2-bench: ${TAU2_BENCH_REPO} -> ${TAU2_BENCH_ROOT}"+ git clone "${TAU2_BENCH_REPO}" "${TAU2_BENCH_ROOT}" || { echo "[setup_env] tau2-bench clone failed"; return 1; }+ if [[ -n "${TAU2_BENCH_REF:-}" ]]; then+ echo "[setup_env] Checking out tau2-bench ref: ${TAU2_BENCH_REF}"+ git -C "${TAU2_BENCH_ROOT}" checkout "${TAU2_BENCH_REF}" || { echo "[setup_env] tau2-bench checkout failed"; return 1; }+ fi+ fi++ # openviking + vikingbot: editable install of the repo root (pyproject finds both).+ # NOTE: this triggers a Cargo/Rust release build of the `ov` CLI.+ if ! command -v cargo >/dev/null 2>&1; then+ echo "[setup_env] WARNING: 'cargo' not found on PATH; the openviking Rust build may fail." >&2+ fi+ # Install with the [bot] extra: the vikingbot runner imports vikingbot.cli and+ # `openviking-server --with-bot` needs the bot deps (prompt_toolkit, gradio, mcp, ...).+ echo "[setup_env] Installing openviking + vikingbot with bot extras (pip install -e ${REPO_ROOT}[bot])"+ "${PY}" -m pip install -e "${REPO_ROOT}[bot]" || { echo "[setup_env] openviking install failed"; return 1; }++ # Ensure the ragfs_python native (PyO3) binding is bundled. The editable install+ # can skip the maturin build under pip build isolation, leaving the server unable+ # to load RAGFS ("ragfs_python native library is not available"). If it's not+ # importable, build it via maturin and copy the extension into openviking/lib/.+ if ! "${PY}" -c "from openviking.pyagfs import get_binding_client; get_binding_client()" >/dev/null 2>&1; then+ echo "[setup_env] ragfs_python native binding missing; building via maturin..."+ "${PY}" -m pip install "maturin>=1.0,<2.0" || { echo "[setup_env] maturin install failed"; return 1; }+ _ragfs_out="$(mktemp -d)"+ if "${PY}" -m maturin build --release -m "${REPO_ROOT}/crates/ragfs-python/Cargo.toml" --out "${_ragfs_out}"; then+ "${PY}" - "${_ragfs_out}" "${REPO_ROOT}/openviking/lib" <<'PYEOF'+import sys, os, glob, zipfile+from pathlib import Path+out_dir, lib_dir = sys.argv[1], Path(sys.argv[2])+whls = sorted(glob.glob(os.path.join(out_dir, "ragfs_python-*.whl")))+if not whls:+ sys.exit("maturin produced no ragfs_python wheel")+lib_dir.mkdir(parents=True, exist_ok=True)+for pat in ("ragfs_python*.so", "ragfs_python*.pyd", "ragfs_python*.dylib"):+ for stale in lib_dir.glob(pat):+ stale.unlink()+with zipfile.ZipFile(whls[0]) as zf:+ for name in zf.namelist():+ base = Path(name).name+ if base == "ragfs_python.pyd" or (base.startswith("ragfs_python.abi3.") and base.endswith((".so", ".pyd"))):+ target = lib_dir / base+ with zf.open(name) as src, open(target, "wb") as dst:+ dst.write(src.read())+ if not sys.platform.startswith("win"):+ os.chmod(target, 0o755)+ print(f"[setup_env] ragfs_python: bundled {base} -> {target}")+ break+ else:+ sys.exit("ragfs_python native extension not found in wheel")+PYEOF+ _ragfs_rc=$?+ rm -rf "${_ragfs_out}"+ [[ "${_ragfs_rc}" -eq 0 ]] || { echo "[setup_env] ragfs_python bundling failed"; return 1; }+ else+ rm -rf "${_ragfs_out}"+ echo "[setup_env] maturin build of ragfs_python failed"; return 1+ fi+ else+ echo "[setup_env] ragfs_python native binding present"+ fi++ # tau2-bench: install the [gym] extra so tau2.gym (gymnasium) is available to the runner.+ echo "[setup_env] Installing tau2-bench with gym extra (pip install -e ${TAU2_BENCH_ROOT}[gym])"+ "${PY}" -m pip install -e "${TAU2_BENCH_ROOT}[gym]" || { echo "[setup_env] tau2-bench install failed"; return 1; }++ echo "[setup_env] Installing smolagents"+ "${PY}" -m pip install smolagents || { echo "[setup_env] smolagents install failed"; return 1; }++ touch "${SETUP_MARKER}"+ echo "[setup_env] Install complete."+ return 0+}++if ! _setup_install; then+ _abort "environment install failed (see messages above)"+ unset -f _setup_install _abort+ return 1 2>/dev/null || exit 1+fi+unset -f _setup_install++# --- activate venv --------------------------------------------------------------+if [[ ! -f "${VENV}/bin/activate" ]]; then+ _abort "venv not found at ${VENV} after install"+ return 1 2>/dev/null || exit 1+fi+# shellcheck disable=SC1090+source "${VENV}/bin/activate"+echo "[setup_env] venv activated: ${VENV}"++# --- runtime env vars -----------------------------------------------------------+# openviking + vikingbot come from the editable install; REPO_ROOT + bot/ kept on+# PYTHONPATH as a harmless import fallback.+OPENVIKING_TAU2_ROOT="${REPO_ROOT}"+VIKINGBOT_ROOT="${VIKINGBOT_ROOT:-${REPO_ROOT}/bot}"+export PYTHONPATH="${OPENVIKING_TAU2_ROOT}:${VIKINGBOT_ROOT}:${PYTHONPATH:-}"++# tau2 dataset root (derived from the tau2-bench checkout)+export TAU2_DATA_ROOT="${TAU2_DATA_ROOT:-${TAU2_BENCH_ROOT}/data/tau2}"++# OpenViking server config+export OPENVIKING_CONFIG_FILE="${OPENVIKING_CONFIG_FILE:-${HOME}/.openviking/ov.conf}"++# LLM for the tau2 user simulator (e.g. Doubao via volcengine ARK, OpenAI-compatible).+# Provide your own key via ARK_API_KEY (do NOT commit real keys).+export OPENAI_API_KEY="${OPENAI_API_KEY:-${ARK_API_KEY:-}}"+export OPENAI_API_BASE="${OPENAI_API_BASE:-https://ark.cn-beijing.volces.com/api/v3}"+if [[ -z "${OPENAI_API_KEY}" ]]; then+ echo "[setup_env] WARNING: OPENAI_API_KEY/ARK_API_KEY is empty; the tau2 user simulator will fail."+fi++echo "[setup_env] PYTHONPATH includes openviking (${OPENVIKING_TAU2_ROOT}) and vikingbot (${VIKINGBOT_ROOT})"+echo "[setup_env] TAU2_DATA_ROOT=${TAU2_DATA_ROOT}"+echo "[setup_env] OPENAI_API_BASE=${OPENAI_API_BASE}"++unset -f _abort 2>/dev/null || truediff --git a/benchmark/tau2/vikingbot/tau2_env/tau2_environment.py b/benchmark/tau2/vikingbot/tau2_env/tau2_environment.pynew file mode 100644index 000000000..27a89143a--- /dev/null+++ b/benchmark/tau2/vikingbot/tau2_env/tau2_environment.py@@ -0,0 +1,96 @@+#!/usr/bin/env python3++from __future__ import annotations++import json++from tau2.gym.gym_agent import AgentGymEnv+++class CommunicateWithUser:+ """The agent's only channel for talking to the user.++ tau2's environment has no native "speak to the user" action, so we add this+ tool: whatever ``content`` the agent passes is delivered to the tau2 user+ simulator, and the simulator's reply comes back as the tool result. This class+ owns both the tool's schema (``openai_schema``, consumed by+ ``Tau2BenchToolProvider``) and its execution (``forward``, invoked by+ ``Tau2BenchEnv.tool_call``).+ """++ name = "communicate_with_user"+ description = (+ "say something to the user. Note that the customer cannot see the answer "+ "returned in `final_answer`. You must communicate with the customer "+ "exclusively through this tool."+ )+ parameters = {+ "type": "object",+ "properties": {+ "content": {+ "type": "string",+ "description": "the content to say to the user",+ }+ },+ "required": ["content"],+ }++ def __init__(self, env):+ # ``env`` is the underlying tau2 AgentGymEnv.+ self.env = env++ def forward(self, content: str):+ """Deliver ``content`` to the tau2 user simulator.++ Returns the raw gym step tuple ``(obs, reward, terminated, truncated, info)``;+ ``Tau2BenchEnv.tool_call`` cleans the observation and tracks termination.+ """+ response = self.env.tool_call(self.name, {"content": content})+ return response++ @classmethod+ def openai_schema(cls) -> dict:+ return {+ "type": "function",+ "function": {+ "name": cls.name,+ "description": cls.description,+ "parameters": cls.parameters,+ },+ }+++class Tau2BenchEnv:+ def __init__(self, domain: str, task_id: str):+ self.env = AgentGymEnv(domain=domain, task_id=task_id, user_llm="openai/doubao-seed-2-0-pro-260215")+ self.terminated = False++ def reset(self):+ user_query, info_dict = self.env.reset()+ self.user_query = user_query.lstrip("user: ")+ self.task = info_dict["task"]+ self.simulation_run = info_dict["simulation_run"]+ self.policy = info_dict["policy"]+ # All OpenAI tool schemas exposed to the agent: tau2's native tools plus+ # communicate_with_user (the agent's only channel to the user).+ self.tool_schemas = [tool.openai_schema for tool in info_dict["tools"]]+ self.tool_schemas.append(CommunicateWithUser.openai_schema())+ self.ground_truth = str(self.task.evaluation_criteria)+ self.user_scenario = self.task.user_scenario++ def tool_call(self, tool_name: str, arguments: dict) -> str:+ if self.terminated:+ return "Task Terminated"++ if tool_name == CommunicateWithUser.name:+ obs, reward, terminated, truncated, info = self.env.step(arguments["content"])+ else:+ action = {"name": tool_name, "arguments": arguments}+ obs, reward, terminated, truncated, info = self.env.step(json.dumps(action))++ if "tool: " in obs:+ obs = obs.lstrip("tool: ")+ if "user: " in obs:+ obs = obs.lstrip("user: ")+ self.terminated = terminated+ return obsdiff --git a/benchmark/tau2/vikingbot/tau2_env/tau2_tool_provider.py b/benchmark/tau2/vikingbot/tau2_env/tau2_tool_provider.pynew file mode 100644index 000000000..fec8e5a26--- /dev/null+++ b/benchmark/tau2/vikingbot/tau2_env/tau2_tool_provider.py@@ -0,0 +1,73 @@+#!/usr/bin/env python3+"""Tau2-bench tool provider that exposes environment tools in OpenAI schema."""++from __future__ import annotations++import json+import os+from typing import Any, Iterable++from .tau2_environment import Tau2BenchEnv+++class Tau2BenchToolProvider:+ """Wrap Tau2BenchEnv tools into OpenAI tool schemas for MCP."""++ def __init__(self, domain: str, task_id: str, data_root: str | None = None):+ self.domain = domain+ self.task_id = task_id+ self.data_root = data_root+ self.env: Tau2BenchEnv | None = None+ self._openai_tools: list[dict[str, Any]] = []++ def reset(self) -> None:+ env = Tau2BenchEnv(self.domain, self.task_id)+ env.reset()+ self.env = env+ # Tau2BenchEnv.tool_schemas already includes tau2's native tools plus+ # communicate_with_user.+ self._openai_tools = list(env.tool_schemas)++ @property+ def user_query(self) -> str:+ if not self.env:+ return ""+ return self.env.user_query++ @property+ def policy(self) -> str:+ if not self.env:+ return ""+ return self.env.policy++ @property+ def ground_truth(self) -> str:+ if not self.env:+ return ""+ return self.env.ground_truth++ def list_openai_tools(self) -> Iterable[dict[str, Any]]:+ return self._openai_tools++ def call_tool(self, name: str, arguments: dict[str, Any]) -> str:+ if not self.env:+ raise RuntimeError("Environment not initialized. Call reset() first.")+ normalized = name.replace("-", "_")+ return self.env.tool_call(normalized, arguments)+++def load_task_id(data_split: str, task_no: int) -> tuple[str, str]:+ """Resolve task_id from split file and return (domain, task_id)."""+ domain, split = data_split.split("_", 1)+ data_root = os.getenv("TAU2_DATA_ROOT")+ if not data_root:+ raise RuntimeError(+ "TAU2_DATA_ROOT is not set. Point it at your tau2-bench data dir, e.g. "+ "export TAU2_DATA_ROOT=<tau2-bench>/data/tau2 (see setup_env.sh)."+ )+ split_path = os.path.join(data_root, "domains", domain, "split_tasks.json")+ with open(split_path, "r", encoding="utf-8") as f:+ data = json.load(f)+ task_ids = data[split]+ task_id = task_ids[task_no]+ return domain, task_id