Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(tau2/vikingbot): benchmark updates (#2244)

nicoleqiwte0ce670f5cb7
diff --git a/benchmark/tau2/.gitignore b/benchmark/tau2/llm/.gitignore
similarity index 100%
rename from benchmark/tau2/.gitignore
rename to benchmark/tau2/llm/.gitignore
diff --git a/benchmark/tau2/README.md b/benchmark/tau2/llm/README.md
similarity index 80%
rename from benchmark/tau2/README.md
rename to benchmark/tau2/llm/README.md
index dd96b4430..afc519cd5 100644
--- a/benchmark/tau2/README.md
+++ b/benchmark/tau2/llm/README.md
@@ -12,7 +12,7 @@ Category rerank and other harness-only diagnostics are intentionally left out.
## Layout
```text
-benchmark/tau2/
+benchmark/tau2/llm/
├── config/
│ ├── baseline.yaml
│ ├── official.yaml
@@ -25,9 +25,9 @@ benchmark/tau2/
└── run_full_eval.sh
```
-Generated eval artifacts are written to `benchmark/tau2/result/<run_id>/`.
+Generated eval artifacts are written to `benchmark/tau2/llm/result/<run_id>/`.
Memory corpus artifacts are cached outside the run id at
-`benchmark/tau2/result/memory_corpora/` by default.
+`benchmark/tau2/llm/result/memory_corpora/` by default.
## Quick Start
@@ -59,8 +59,8 @@ For a local one-command setup, clone and install TAU-2 into ignored benchmark
directories:
```bash
-benchmark/tau2/scripts/setup_tau2_repo.sh
-source benchmark/tau2/.env.tau2
+benchmark/tau2/llm/scripts/setup_tau2_repo.sh
+source benchmark/tau2/llm/.env.tau2
```
For PR-B-compatible reproduction, pin the TAU-2 checkout to a ref that includes
@@ -68,9 +68,9 @@ the confirmation-aware text-user-simulator prompt. The original PR-B evidence
used the open TAU-2 fix PR head (`79dbf0c18ac7637aedf869cb3122babcd57aaf17`):
```bash
-benchmark/tau2/scripts/setup_tau2_repo.sh \
+benchmark/tau2/llm/scripts/setup_tau2_repo.sh \
--ref refs/pull/297/head
-source benchmark/tau2/.env.tau2
+source benchmark/tau2/llm/.env.tau2
```
Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).
@@ -78,7 +78,7 @@ Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/t
Plan the default benchmark without running TAU-2:
```bash
-python benchmark/tau2/scripts/run_eval.py --config benchmark/tau2/config/baseline.yaml --plan-only
+python benchmark/tau2/llm/scripts/run_eval.py --config benchmark/tau2/llm/config/baseline.yaml --plan-only
```
Add `--preflight` or `--strict-preflight` when you want the runner to write a
@@ -87,8 +87,8 @@ small environment/config check next to the run plan.
After setup, verify the local TAU-2 link and write a one-cell run plan:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/baseline.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/baseline.yaml \
--strict-preflight \
--domain retail \
--strategy-id memory_v2_experience_only \
@@ -99,8 +99,8 @@ benchmark/tau2/run_full_eval.sh \
Plan a one-cell Memory V2 pre-write smoke:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/baseline.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/baseline.yaml \
--domain retail \
--strategy-id memory_v2_prewrite \
--num-tasks 1 \
@@ -110,8 +110,8 @@ benchmark/tau2/run_full_eval.sh \
Plan a one-cell trajectory memory smoke:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/trajectory.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/trajectory.yaml \
--domain retail \
--strategy-id memory_v2_trajectory_view \
--num-tasks 1 \
@@ -122,8 +122,8 @@ benchmark/tau2/run_full_eval.sh \
Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/baseline.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/baseline.yaml \
--execute
```
@@ -143,15 +143,15 @@ the same confirmation-aware simulator policy but does not require fixed fixtures
Run one bootstrap pass per domain:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml \
--domain retail \
--run-id fixed_first_user_bootstrap_retail \
--strict-preflight \
--execute
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml \
--domain airline \
--run-id fixed_first_user_bootstrap_airline \
--strict-preflight \
@@ -161,31 +161,31 @@ benchmark/tau2/run_full_eval.sh \
Then convert each bootstrap `results.json` into a fixture:
```bash
-RETAIL_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_retail/memory_cells/fixed_first_user_bootstrap_retail_retail_no_memory_r1/fixed_first_user_bootstrap_retail_retail_no_memory_r1.json
-AIRLINE_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_airline/memory_cells/fixed_first_user_bootstrap_airline_airline_no_memory_r1/fixed_first_user_bootstrap_airline_airline_no_memory_r1.json
+RETAIL_RESULTS=benchmark/tau2/llm/result/fixed_first_user_bootstrap_retail/memory_cells/fixed_first_user_bootstrap_retail_retail_no_memory_r1/fixed_first_user_bootstrap_retail_retail_no_memory_r1.json
+AIRLINE_RESULTS=benchmark/tau2/llm/result/fixed_first_user_bootstrap_airline/memory_cells/fixed_first_user_bootstrap_airline_airline_no_memory_r1/fixed_first_user_bootstrap_airline_airline_no_memory_r1.json
-python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \
+python benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py \
--repo "$TAU2_REPO" \
--results-json "$RETAIL_RESULTS" \
--domain retail \
--task-split-name test \
- --output benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json \
+ --output benchmark/tau2/llm/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json \
--require-full-split
-python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \
+python benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py \
--repo "$TAU2_REPO" \
--results-json "$AIRLINE_RESULTS" \
--domain airline \
--task-split-name test \
- --output benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json \
+ --output benchmark/tau2/llm/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json \
--require-full-split
```
Export the generated fixture paths for subsequent strict runs:
```bash
-export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json"
-export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"
+export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json"
+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"
```
### 2. Run smoke and full PR-B matrix
@@ -193,8 +193,8 @@ export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_firs
First run one tiny end-to-end smoke against a clean local OpenViking service:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \
--domain retail \
--strategy-id new_traj_fixed_first_user_prewrite \
--num-tasks 1 \
@@ -207,15 +207,15 @@ benchmark/tau2/run_full_eval.sh \
Then run the full PR-B matrix:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \
--run-id prb_content_matrix_new_prompt_full8 \
--strict-preflight \
--execute
```
The main result is written to
-`benchmark/tau2/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.
+`benchmark/tau2/llm/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.
Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON
lives under `memory_cells/`, and corpus identity / generated memory checks live
under `memory_corpora/`.
@@ -223,8 +223,8 @@ under `memory_corpora/`.
For a small E2E smoke, keep both the eval and train slices tiny:
```bash
-benchmark/tau2/run_full_eval.sh \
- --config benchmark/tau2/config/baseline.yaml \
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/baseline.yaml \
--domain retail \
--strategy-id memory_v2_experience_only \
--num-tasks 1 \
diff --git a/benchmark/tau2/config/baseline.yaml b/benchmark/tau2/llm/config/baseline.yaml
similarity index 95%
rename from benchmark/tau2/config/baseline.yaml
rename to benchmark/tau2/llm/config/baseline.yaml
index 0672322bf..31b054dc9 100644
--- a/benchmark/tau2/config/baseline.yaml
+++ b/benchmark/tau2/llm/config/baseline.yaml
@@ -18,10 +18,10 @@ benchmark:
paths:
tau2_repo: ${TAU2_REPO:-data/external_benchmarks/tau2-bench}
tau2_cli: ${TAU2_CLI:-tau2}
- output_dir: benchmark/tau2/result
+ output_dir: benchmark/tau2/llm/result
# Corpus writes are expensive and should be reused across eval run ids when
# the train split and memory prompt/config did not change.
- corpus_cache_dir: benchmark/tau2/result/memory_corpora
+ corpus_cache_dir: benchmark/tau2/llm/result/memory_corpora
eval:
# Default OpenViking TAU-2 memory evidence uses the fixed-first-user full8
diff --git a/benchmark/tau2/config/fixed_first_user_bootstrap.yaml b/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml
similarity index 100%
rename from benchmark/tau2/config/fixed_first_user_bootstrap.yaml
rename to benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml
diff --git a/benchmark/tau2/config/no_memory.yaml b/benchmark/tau2/llm/config/no_memory.yaml
similarity index 100%
rename from benchmark/tau2/config/no_memory.yaml
rename to benchmark/tau2/llm/config/no_memory.yaml
diff --git a/benchmark/tau2/config/official.yaml b/benchmark/tau2/llm/config/official.yaml
similarity index 100%
rename from benchmark/tau2/config/official.yaml
rename to benchmark/tau2/llm/config/official.yaml
diff --git a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml
similarity index 86%
rename from benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
rename to benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml
index 55607c479..be128c15d 100644
--- a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
+++ b/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml
@@ -24,7 +24,7 @@ strategies:
retrieval_mode: first_user
retrieval_top_k: 4
first_user_inject_top_k: 4
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_traj_fixed_prewrite_only
label: PR-B new trajectory fixed-count prewrite top2
@@ -40,7 +40,7 @@ strategies:
retrieval_top_k: 4
prewrite_retrieval_top_k: 2
prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_traj_fixed_first_user_prewrite
label: PR-B new trajectory fixed-count first-user top4 + prewrite top2
@@ -57,7 +57,7 @@ strategies:
first_user_inject_top_k: 4
prewrite_retrieval_top_k: 2
prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_exp_fixed_first_user
label: PR-B new experience fixed-count first-user top2
@@ -72,7 +72,7 @@ strategies:
retrieval_mode: first_user
retrieval_top_k: 2
first_user_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_exp_fixed_prewrite_only
label: PR-B new experience fixed-count prewrite top2
@@ -88,7 +88,7 @@ strategies:
retrieval_top_k: 2
prewrite_retrieval_top_k: 2
prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_exp_fixed_first_user_prewrite
label: PR-B new experience fixed-count first-user + prewrite top2
@@ -105,7 +105,7 @@ strategies:
first_user_inject_top_k: 2
prewrite_retrieval_top_k: 2
prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_traj_4000_prewrite_only
label: PR-B new trajectory 4000-char prewrite
@@ -122,7 +122,7 @@ strategies:
prewrite_retrieval_top_k: 8
prewrite_inject_top_k: 8
memory_inject_max_chars: 4000
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: new_exp_4000_first_user_prewrite
label: PR-B new experience 4000-char first-user + prewrite
@@ -140,4 +140,4 @@ strategies:
prewrite_retrieval_top_k: 8
prewrite_inject_top_k: 8
memory_inject_max_chars: 4000
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/config/prb_scope_fairness.yaml b/benchmark/tau2/llm/config/prb_scope_fairness.yaml
similarity index 84%
rename from benchmark/tau2/config/prb_scope_fairness.yaml
rename to benchmark/tau2/llm/config/prb_scope_fairness.yaml
index 8d044d216..bee9ed94f 100644
--- a/benchmark/tau2/config/prb_scope_fairness.yaml
+++ b/benchmark/tau2/llm/config/prb_scope_fairness.yaml
@@ -17,7 +17,7 @@ strategies:
- id: no_memory_generic_scope
label: TAU-2 no-memory same-seed baseline with generic memory scope prompt
memory_backend: none
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: trajectory_top4_first_user_prewrite_generic_scope
label: Trajectory top4 first-user + pre-write top2 with generic memory scope prompt
@@ -31,4 +31,4 @@ strategies:
first_user_inject_top_k: 4
prewrite_retrieval_top_k: 2
prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/config/prewrite.yaml b/benchmark/tau2/llm/config/prewrite.yaml
similarity index 100%
rename from benchmark/tau2/config/prewrite.yaml
rename to benchmark/tau2/llm/config/prewrite.yaml
diff --git a/benchmark/tau2/config/scope_prompts/generic_memory_scope.md b/benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
similarity index 100%
rename from benchmark/tau2/config/scope_prompts/generic_memory_scope.md
rename to benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/config/trajectory.yaml b/benchmark/tau2/llm/config/trajectory.yaml
similarity index 88%
rename from benchmark/tau2/config/trajectory.yaml
rename to benchmark/tau2/llm/config/trajectory.yaml
index 2538d2639..aa0396e12 100644
--- a/benchmark/tau2/config/trajectory.yaml
+++ b/benchmark/tau2/llm/config/trajectory.yaml
@@ -20,7 +20,7 @@ strategies:
train_memory_mode: experience_only
search_memory_type: trajectories
retrieval_mode: first_user
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: memory_v2_trajectory_prewrite_only
label: OpenViking Memory V2 trajectory pre-write-only recall
memory_backend: openviking
@@ -45,7 +45,7 @@ strategies:
train_memory_mode: experience_only
search_memory_type: trajectories
retrieval_mode: prewrite
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- id: memory_v2_trajectory_prewrite_scope
label: OpenViking Memory V2 trajectory first-user + pre-write recall with scope prompt
memory_backend: openviking
@@ -54,4 +54,4 @@ strategies:
train_memory_mode: experience_only
search_memory_type: trajectories
retrieval_mode: first_user_prewrite
- scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
+ scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/run_full_eval.sh b/benchmark/tau2/llm/run_full_eval.sh
similarity index 91%
rename from benchmark/tau2/run_full_eval.sh
rename to benchmark/tau2/llm/run_full_eval.sh
index 728f3ce46..a3b63ab8f 100755
--- a/benchmark/tau2/run_full_eval.sh
+++ b/benchmark/tau2/llm/run_full_eval.sh
@@ -2,7 +2,7 @@
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
-REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-python3}"
CONFIG="$SCRIPT_DIR/config/baseline.yaml"
EXECUTE=false
@@ -40,7 +40,7 @@ while [[ $# -gt 0 ]]; do
--help|-h)
cat <<'EOF'
Usage:
- benchmark/tau2/run_full_eval.sh [--config PATH] [--run-id ID] [--execute] [--preflight]
+ benchmark/tau2/llm/run_full_eval.sh [--config PATH] [--run-id ID] [--execute] [--preflight]
Without --execute the script only writes run_plan artifacts.
EOF
diff --git a/benchmark/tau2/scripts/build_fixed_first_user_fixture.py b/benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py
similarity index 99%
rename from benchmark/tau2/scripts/build_fixed_first_user_fixture.py
rename to benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py
index 8b4c0cc49..e9576b861 100644
--- a/benchmark/tau2/scripts/build_fixed_first_user_fixture.py
+++ b/benchmark/tau2/llm/scripts/build_fixed_first_user_fixture.py
@@ -94,7 +94,7 @@ def build_fixture(
def main() -> None:
parser = argparse.ArgumentParser()
- parser.add_argument("--repo", type=Path, default=Path("benchmark/tau2/.external/tau2-bench"))
+ parser.add_argument("--repo", type=Path, default=Path("benchmark/tau2/llm/.external/tau2-bench"))
parser.add_argument("--results-json", type=Path, required=True)
parser.add_argument("--domain", required=True)
parser.add_argument("--task-split-name", default="test")
diff --git a/benchmark/tau2/scripts/run_eval.py b/benchmark/tau2/llm/scripts/run_eval.py
similarity index 100%
rename from benchmark/tau2/scripts/run_eval.py
rename to benchmark/tau2/llm/scripts/run_eval.py
diff --git a/benchmark/tau2/scripts/run_memory_v2_eval.py b/benchmark/tau2/llm/scripts/run_memory_v2_eval.py
similarity index 99%
rename from benchmark/tau2/scripts/run_memory_v2_eval.py
rename to benchmark/tau2/llm/scripts/run_memory_v2_eval.py
index 26b2d69e7..dd9ed0633 100644
--- a/benchmark/tau2/scripts/run_memory_v2_eval.py
+++ b/benchmark/tau2/llm/scripts/run_memory_v2_eval.py
@@ -15,7 +15,7 @@ from typing import Any
from tau2_common import assert_tau2_results_complete, normalize_litellm_env
AGENT_NAME = "openviking_memory_agent"
-REPO_ROOT = Path(__file__).resolve().parents[3]
+REPO_ROOT = Path(__file__).resolve().parents[4]
WRITE_TOOL_PREFIXES = (
"toggle_",
"enable_",
diff --git a/benchmark/tau2/scripts/setup_tau2_repo.sh b/benchmark/tau2/llm/scripts/setup_tau2_repo.sh
similarity index 90%
rename from benchmark/tau2/scripts/setup_tau2_repo.sh
rename to benchmark/tau2/llm/scripts/setup_tau2_repo.sh
index 815f9bc0f..e4d0320be 100755
--- a/benchmark/tau2/scripts/setup_tau2_repo.sh
+++ b/benchmark/tau2/llm/scripts/setup_tau2_repo.sh
@@ -37,10 +37,10 @@ while [[ $# -gt 0 ]]; do
--help|-h)
cat <<'EOF'
Usage:
- benchmark/tau2/scripts/setup_tau2_repo.sh [--repo-url URL] [--repo-dir DIR] [--venv DIR] [--ref REF] [--no-install]
+ benchmark/tau2/llm/scripts/setup_tau2_repo.sh [--repo-url URL] [--repo-dir DIR] [--venv DIR] [--ref REF] [--no-install]
Clones TAU-2 into a local ignored directory and optionally installs it into a
-local virtualenv. The script writes benchmark/tau2/.env.tau2 with TAU2_REPO and
+local virtualenv. The script writes benchmark/tau2/llm/.env.tau2 with TAU2_REPO and
TAU2_CLI for the benchmark runner.
EOF
exit 0
diff --git a/benchmark/tau2/scripts/tau2_common.py b/benchmark/tau2/llm/scripts/tau2_common.py
similarity index 99%
rename from benchmark/tau2/scripts/tau2_common.py
rename to benchmark/tau2/llm/scripts/tau2_common.py
index 4f4505bd8..6a657be39 100755
--- a/benchmark/tau2/scripts/tau2_common.py
+++ b/benchmark/tau2/llm/scripts/tau2_common.py
@@ -13,7 +13,7 @@ from typing import Any
import yaml
TAU2_DIR = Path(__file__).resolve().parents[1]
-REPO_ROOT = TAU2_DIR.parents[1]
+REPO_ROOT = TAU2_DIR.parents[2]
CONFIRMATION_AWARE_UPSTREAM_PR = "https://github.com/sierra-research/tau2-bench/pull/297"
CONFIRMATION_AWARE_APPENDIX = """
diff --git a/benchmark/tau2/vikingbot/.gitignore b/benchmark/tau2/vikingbot/.gitignore
new file mode 100644
index 000000000..c7821be79
--- /dev/null
+++ b/benchmark/tau2/vikingbot/.gitignore
@@ -0,0 +1,16 @@
+# Generated eval artifacts
+result/
+result_*/
+trajectory/
+trajectory_*/
+
+# External tau2-bench checkout (see README / setup_env.sh)
+tau2-bench/
+
+# Reports & logs
+full_test_report*.txt
+*.log
+
+# Python caches
+__pycache__/
+*.pyc
diff --git a/benchmark/tau2/vikingbot/README.md b/benchmark/tau2/vikingbot/README.md
new file mode 100644
index 000000000..d9d573c67
--- /dev/null
+++ b/benchmark/tau2/vikingbot/README.md
@@ -0,0 +1,321 @@
+# VikingBot × tau2-bench Runner
+
+This folder runs the **full VikingBot agent** (`bot/vikingbot` `AgentLoop`) end-to-end
+on [tau2-bench](https://github.com/sierra-research/tau2-bench) tasks, then commits the
+resulting trajectories back into OpenViking memory so the agent can **self-improve across
+epochs** (cold start → memory-augmented runs).
+
+> **Memory is extracted only from the `train` split.** Each epoch runs both splits, but only
+> `train` trajectories are committed to OpenViking memory. The `test` split is **held out** and
+> used purely to measure the task-success improvement once that (train-derived) memory is injected
+> — so the reported gains reflect learning transferred from train to test, with no test-set leakage.
+
+It is a sibling to the harness in [`../llm/`](../llm/README.md). Both are multi-turn and exercise
+OpenViking memory extraction + retrieval; they differ in *which agent* drives the tasks:
+
+- **[`../llm/`](../llm/README.md)** uses tau2-bench's **native ReAct agent**, wired to OpenViking
+ memory, to measure the effect of that memory on task performance.
+- **`vikingbot/`** (this folder) is an **end-to-end, self-improving agent** evaluation: it runs the
+ full VikingBot agent loop on the tasks and commits trajectories back into memory so the agent
+ improves across epochs.
+
+The pipeline is: **run tasks → evaluate reward → commit train trajectories to memory** (see
+`run_full_test.sh`).
+
+---
+
+## Install & Config
+
+One step sets up everything. `setup_env.sh` creates a fresh `.venv` at the OpenViking repo
+root, clones tau2-bench into `./tau2-bench` (external dependency, gitignored), installs
+openviking + vikingbot (`pip install -e .`, which also runs the Cargo build) and tau2-bench,
+installs smolagents, then activates the venv and exports the runtime env vars:
+
+```bash
+source benchmark/tau2/vikingbot/setup_env.sh # first run: install, then activate + export
+source benchmark/tau2/vikingbot/setup_env.sh --reinstall # rebuild the .venv from scratch
+```
+
+Safe to `source` in every new shell: the install phase runs only when the venv is missing;
+later sources just activate and re-export.
+
+It exports `PYTHONPATH` for `openviking` + `bot/vikingbot`, `TAU2_DATA_ROOT`
+(defaults to `./tau2-bench/data/tau2`), `OPENVIKING_CONFIG_FILE`, and the user-simulator LLM
+env vars. Override any of these by exporting before sourcing:
+
+- `TAU2_BENCH_ROOT` — tau2-bench checkout location (if it lives elsewhere)
+- `TAU2_BENCH_REPO` / `TAU2_BENCH_REF` — git URL / ref to clone (e.g. pin a specific checkout)
+- `VIKINGBOT_ROOT`
+- `ARK_API_KEY` (mapped to `OPENAI_API_KEY`), `OPENAI_API_BASE`
+
+The tau2 **user simulator** talks to an OpenAI-compatible endpoint — set `ARK_API_KEY` (e.g.
+Doubao through volcengine ARK) before sourcing, or the simulator will fail. The user-simulator
+model is configured in [`tau2_env/tau2_environment.py`](tau2_env/tau2_environment.py).
+
+> Note: the sibling `llm/` harness ([`../llm/README.md`](../llm/README.md)) pins a tau2-bench ref
+> with a confirmation-aware user-simulator prompt (sierra-research/tau2-bench#297). Set
+> `TAU2_BENCH_REF` to a comparable checkout if you want results aligned with that protocol.
+
+Then start the OpenViking server with the bot enabled:
+
+```bash
+openviking-server --config "${OPENVIKING_CONFIG_FILE}" --with-bot
+```
+
+
+---
+
+## One-click full run (recommended)
+
+Run one epoch — **1 train run + 8 test runs in parallel** (`--test-repeats`, default 8) — then
+evaluate (test accuracy is averaged over the repeats) and commit the train trajectories to memory:
+
+```bash
+bash benchmark/tau2/vikingbot/run_full_test.sh --domain airline --epoch 0 --result-dir result
+```
+
+The async memory commit happens on the server, so wait for it to
+finish before starting the next epoch. The per-domain report is appended to
+`full_test_report_<domain>.txt`.
+
+Multi-epoch examples (cold start → memory-augmented epochs):
+
+```bash
+bash benchmark/tau2/vikingbot/run_airline_2epochs.sh
+```
+
+---
+
+## Run each step separately
+
+### 1) Run tasks (train / test)
+
+**train** runs **once per epoch** — one trajectory per task, which is what gets committed to memory
+as the experience corpus. A single pass mirrors real usage, where the agent learns from one attempt
+at each task.
+
+**test** runs **8 times per epoch (in parallel) and is averaged**. Agent execution is stochastic, so
+averaging several independent repeats gives a more confident accuracy estimate. Each repeat is a
+separate `--try-no`; `run_eval_reward.sh` scores one repeat, and the per-repeat accuracies are
+averaged. The one-click `run_full_test.sh` does all of this for you (`--test-repeats`, default 8).
+
+```bash
+# train: a single run (try 0)
+bash scripts/run_tau2_domain.sh \
+ --domain airline --split train --epoch 0 --try-no 0 \
+ --result-dir result --concurrency 5 --use-continue --agent-id airline_v0
+
+# test: 8 independent runs (try 0..7)
+for t in 0 1 2 3 4 5 6 7; do
+ bash scripts/run_tau2_domain.sh \
+ --domain airline --split test --epoch 0 --try-no "$t" \
+ --result-dir result --concurrency 5 --use-continue --agent-id airline_v0
+done
+```
+
+Results are written to `result/<domain>_<split>/task_<n>_<epoch>_<try>_trajectory.json`,
+and full message dumps to the mirrored `trajectory/...` path.
+
+### 2) Evaluate rewards
+
+```bash
+bash scripts/run_eval_reward.sh result/airline_train 0 0
+bash scripts/run_eval_reward.sh result/airline_test 0 0
+```
+
+### 3) Commit trajectories to memory
+
+> VikingBot natively commits a task's trajectory into memory automatically as soon as that task finishes.
+> For these experiments that auto-commit is **disabled**, so that all
+> tasks within an epoch (train + test, run in parallel) execute under identical memory conditions — no run sees memory written by a sibling run mid-experiment.
+> Instead, the commit is performed explicitly as a separate, controlled step via the script below (run once between epochs).
+
+```bash
+python scripts/commit_trajectory_to_memory.py \
+ --input result/airline_train \
+ --domain airline_v0 \
+ --pattern "*_0_0_trajectory.json" \
+ --include-eval-result
+```
+
+---
+
+## How the runner adapts VikingBot for tau2
+
+There are **two layers** of adaptation:
+
+1. **Runner-level** (this folder only, no core edits) — swap the agent's tool set over to the tau2
+ environment tools, gate OpenViking memory by epoch (cold start vs. memory-augmented), and commit
+ train trajectories between epochs. Detailed below.
+2. **Core `bot/vikingbot`** edits — required for per-domain workspace isolation and for reading
+ *agent* (experience) memory instead of *user* memory. See
+ [Core `bot/vikingbot` changes for tau2](#core-botvikingbot-changes-for-tau2) below.
+
+### Runner-level adaptations:
+
+- **Tool registry swap** — the tau2 environment tools are injected into VikingBot's `ToolRegistry`
+ via `agent.tools.register(Tau2Tool(...))`, so the agent drives the task through tau2's own tools
+ (plus `communicate_with_user` and `done`). `openviking_memory_commit` is **always** unregistered
+ here — that is the mechanism that disables VikingBot's per-task auto-commit (see step 3 above).
+ - **`--keep-default-tools` controls memory availability, tied to the epoch.** The flag decides
+ whether VikingBot's built-in memory tools — **OpenViking memory tools** — stay
+ registered, and whether agent-experience memory is retrieved into the system prompt
+ (`ov_tools_enable`). `run_full_test.sh` sets it by epoch: **epoch 0 omits the flag**, so all
+ built-in memory tools are unregistered (only tau2 tools remain) and no memory is injected — a clean
+ **cold-start / no-memory** run; **epoch > 0 passes the flag**, so the memory tools and retrieved
+ experiences are available (memory-augmented).
+- **Epoch-based memory commit** — `commit_trajectory_to_memory.py` writes train trajectories
+ (optionally only failed ones, via `--only-wrong`) into OpenViking memory between epochs.
+
+
+### Core `bot/vikingbot` changes for tau2
+
+Two behaviours in the VikingBot core
+(`bot/vikingbot`) had to be changed for tau2:
+
+1. **Per-domain workspace isolation via `agent_id`**
+ By default VikingBot do not support agent id
+ isolation in local mode. In order to easily test in local, changes are made so that each tau2 domain (airline, retail, …) read/write its own OpenViking namespace so experiences learned on one domain don't leak into another.
+2. **Read *agent* (experience) memory at system-prompt build time**
+ By default VikingBot retrieves *user* memory when assembling the system message. For tau2 self-improvement we want to isolate the effect of the agent's own accumulated **experience** agent memory instead.
+
+
+### Change 1 — `agent_id` isolation in `VikingClient` (`openviking_mount/ov_server.py`)
+
+In `local` mode the client used to hardcode `agent_id = "default"` for every caller, so all
+domains shared one namespace. It now reads the incoming `agent_id` and only falls back to
+`"default"` when the id is a normal session key (which contains `__`, e.g. `cli__default`):
+
+```diff
+ if openviking_config.mode == "local":
+- self.client = ov.AsyncHTTPClient(url=openviking_config.server_url)
+- self.agent_id = "default"
++ if agent_id is None or "__" in agent_id:
++ self.client = ov.AsyncHTTPClient(url=openviking_config.server_url)
++ self.agent_id = "default"
++ else:
++ self.client = ov.AsyncHTTPClient(
++ url=openviking_config.server_url,
++ agent_id=agent_id,
++ )
++ self.agent_id = agent_id
+ self.account_id = "default"
+ self.user_id = "default"
+ self.admin_user_id = "default"
+ return
+```
+
+(The `agent_id is None` guard is needed because `agent_id` defaults to `None`, and `"__" in None`
+would raise `TypeError`.)
+
+`search_experiences` then resolves the experience URI from this per-instance `agent_id` (a clean
+domain id like `airline_v1` contains a single `_`), instead of always using the global config
+`agent_id`:
+
+```diff
+ async def search_experiences(self, query: str, limit: int = 5) -> list[Any]:
+ """用 query 检索 agent experience 记忆。"""
+ effective_agent_id = self.openviking_config.agent_id or "default"
++ if self.agent_id and "_" in self.agent_id:
++ effective_agent_id = self.agent_id
+ exp_uri = f"viking://agent/{effective_agent_id}/memories/experiences/"
+ result = await self.search(query=query, target_uri=exp_uri, limit=limit)
+ return result.get("memories", [])
+```
+
+
+#### How `agent_id` flows from the runner into the core
+
+The domain id is threaded all the way down into the OpenViking client:
+
+```
+run_full_test.sh AGENT_ID=${DOMAIN}_v1 # e.g. airline_v1, retail_v1
+ └─ run_tau2_domain.sh --agent-id airline_v1
+ └─ vikingbot_tau2_runner.py
+ build_messages(..., memory_users=agent_id) # memory_users == "airline_v1"
+ └─ context.py _build_user_memory
+ workspace_id = memory_users # workspace_id == "airline_v1"
+ └─ memory.py get_viking_experience_context(query, workspace_id)
+ └─ _create_client(workspace_id)
+ └─ VikingClient.create(agent_id=workspace_id)
+ └─ ov_server.py VikingClient.__init__ / search_experiences
+ viking://agent/airline_v1/memories/experiences/
+```
+
+So `--agent-id airline_v1` ⇒ the agent reads/writes `viking://agent/airline_v1/...`, giving each
+domain an isolated workspace.
+
+### Change 2 — system prompt reads *agent experience* memory (`agent/context.py`)
+
+`_build_user_memory` (called by `build_messages` when assembling the system prompt) used to call
+`get_viking_memory_context`, which retrieves **user** memory. For tau2 it now binds `workspace_id`
+to the passed-in `memory_users` (the domain `agent_id`) and calls `get_viking_experience_context`
+to retrieve the agent's **experience** memory:
+
+```diff
+
+ # Viking agent memory (only if ov tools are enabled)
+ if ov_tools_enable:
+ start = _time.time()
+ # Use provided memory_users or fall back to [sender_id]
+ search_user_ids = memory_users if memory_users else [sender_id]
+- viking_memory = await self.memory.get_viking_memory_context(
+- current_message=current_message,
+- workspace_id=workspace_id,
+- sender_id=sender_id,
+- user_ids=search_user_ids,
+- )
++ workspace_id = memory_users
++
++ viking_memory = await self.memory.get_viking_experience_context(
++ query=current_message,
++ workspace_id=workspace_id,
++ )
+ logger.info(f"viking_memory={viking_memory}")
+ cost = round(_time.time() - start, 2)
+ logger.info(
+- f"[READ_USER_MEMORY]: cost {cost}s, memory={viking_memory[:50] if viking_memory else 'None'}"
++ f"[READ_AGENT_MEMORY]: cost {cost}s, memory={viking_memory[:50] if viking_memory else 'None'}"
+ )
+ if viking_memory:
+ self.latest_relevant_memories = viking_memory
+- parts.append(f"## openviking_search(query=[user_query])\n{viking_memory}")
++ parts.append(f"## openviking_search(query=[user_query])\n ## Agent Experience (relevant to this task)\n {viking_memory}")
+ else:
+ self.latest_relevant_memories = None
+```
+
+The key line is `workspace_id = memory_users`: the runner passes the domain id as `memory_users`,
+and it becomes the `agent_id` used to open the (isolated) OpenViking client downstream.
+
+### Change 3 — supporting edits in `agent/memory.py`
+
+`get_viking_experience_context` is the experience-retrieval path that Change 2 now calls.
+
+Retrieval limits in `get_viking_experience_context` were also tuned for tau2 (fewer experiences,
+more characters per experience):
+
+```diff
+- experiences = await client.search_experiences(query, limit=5)
++ experiences = await client.search_experiences(query, limit=2)
+ ...
+ return await self._parse_viking_memory(
+- experiences, client, min_score=0.3, max_chars=2000
++ experiences, client, min_score=0.3, max_chars=10000
+ )
+```
+
+
+---
+
+## Layout
+
+- `setup_env.sh` — environment setup (PYTHONPATH, tau2 data root, simulator LLM)
+- `run_full_test.sh` — full pipeline for one epoch (run → eval → commit)
+- `run_airline_2epochs.sh` — multi-epoch example (cold start → memory-augmented epochs)
+- `tau2_env/` — tau2 environment integration (`tau2_environment.py`, `tau2_tool_provider.py`)
+- `scripts/`
+ - `vikingbot_tau2_runner.py` — runs a single tau2 task through the VikingBot agent loop
+ - `run_tau2_domain.sh` — runs all tasks in a `{domain}_{split}` slice with bounded concurrency
+ - `run_eval_reward.sh` — average reward over a result folder
+ - `commit_trajectory_to_memory.py` — commit trajectories into OpenViking memory
diff --git a/benchmark/tau2/vikingbot/run_airline_2epochs.sh b/benchmark/tau2/vikingbot/run_airline_2epochs.sh
new file mode 100755
index 000000000..6971c2148
--- /dev/null
+++ b/benchmark/tau2/vikingbot/run_airline_2epochs.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
+LOG_FILE="${SCRIPT_DIR}/run_airline_2epochs.log"
+
+# --- env setup (inline, no source needed) ---
+source "${SCRIPT_DIR}/setup_env.sh"
+
+CONCURRENCY=5
+DOMAIN=airline
+RESULT_DIR=result
+
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG_FILE}"; }
+
+log "===== Start airline 2-epoch run ====="
+
+# ---------- Epoch 0 ----------
+log ">>> Epoch 0 start (cold start, no memory)"
+bash "${SCRIPT_DIR}/run_full_test.sh" \
+ --domain "${DOMAIN}" \
+ --epoch 0 \
+ --concurrency "${CONCURRENCY}" \
+ --result-dir "${RESULT_DIR}"
+log ">>> Epoch 0 done"
+
+# Wait for OpenViking server to finish async memory processing
+WAIT_SECS=9000
+log ">>> Waiting ${WAIT_SECS}s for server async memory commit to finish..."
+sleep "${WAIT_SECS}"
+log ">>> Wait done, starting Epoch 1"
+
+# ---------- Epoch 1 ----------
+log ">>> Epoch 1 start (with memory from epoch 0 train)"
+bash "${SCRIPT_DIR}/run_full_test.sh" \
+ --domain "${DOMAIN}" \
+ --epoch 1 \
+ --concurrency "${CONCURRENCY}" \
+ --result-dir "${RESULT_DIR}"
+log ">>> Epoch 1 done"
+
+log "===== All done. Results in: ${SCRIPT_DIR}/${RESULT_DIR} ====="
+log "===== Report: ${SCRIPT_DIR}/full_test_report_${DOMAIN}.txt ====="
diff --git a/benchmark/tau2/vikingbot/run_full_test.sh b/benchmark/tau2/vikingbot/run_full_test.sh
new file mode 100755
index 000000000..0e0fa4c2f
--- /dev/null
+++ b/benchmark/tau2/vikingbot/run_full_test.sh
@@ -0,0 +1,168 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Run one epoch of a tau2 domain:
+# - train: a single run (used to extract experience into memory; mirrors real usage)
+# - test : TEST_REPEATS independent runs in parallel, averaged (agent execution is stochastic,
+# so averaging repeats gives a more confident accuracy estimate)
+# Then evaluate rewards and commit the train trajectories to memory.
+# Usage:
+# bash run_full_test.sh --domain airline [--epoch 0] [--test-repeats 8] [--result-dir result] [--concurrency N] [--commit|--no-commit]
+
+CUR_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SCRIPTS_DIR="${CUR_DIR}/scripts"
+
+PIDS=()
+kill_tree() {
+ local pid="$1"
+ local sig="$2"
+ local children=""
+ children=$(pgrep -P "${pid}" 2>/dev/null || true)
+ for child in ${children}; do
+ kill_tree "${child}" "${sig}"
+ done
+ kill "-${sig}" "${pid}" 2>/dev/null || true
+}
+cleanup() {
+ local exit_code=$?
+ if [[ ${#PIDS[@]} -gt 0 ]]; then
+ echo "[run_full_test] Caught interrupt, stopping child processes..."
+ for pid in "${PIDS[@]}"; do
+ kill_tree "${pid}" "TERM"
+ done
+ sleep 1
+ for pid in "${PIDS[@]}"; do
+ kill_tree "${pid}" "KILL"
+ done
+ fi
+ exit "${exit_code}"
+}
+trap cleanup INT TERM
+
+DOMAIN=""
+EPOCH=0
+TEST_REPEATS=8
+RESULT_DIR="result"
+CONCURRENCY=1
+DO_COMMIT=1
+
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --domain)
+ DOMAIN="$2"; shift 2 ;;
+ --epoch)
+ EPOCH="$2"; shift 2 ;;
+ --test-repeats)
+ TEST_REPEATS="$2"; shift 2 ;;
+ --result-dir)
+ RESULT_DIR="$2"; shift 2 ;;
+ --concurrency)
+ CONCURRENCY="$2"; shift 2 ;;
+ --no-commit)
+ DO_COMMIT=0; shift ;;
+ --commit)
+ DO_COMMIT=1; shift ;;
+ -h|--help)
+ echo "Usage: bash run_full_test.sh --domain DOMAIN [--epoch N] [--test-repeats N] [--result-dir DIR] [--concurrency N] [--commit|--no-commit]"
+ exit 0 ;;
+ *)
+ echo "Unknown argument: $1" >&2; exit 1 ;;
+ esac
+done
+
+if [[ -z "${DOMAIN}" ]]; then
+ echo "Missing --domain" >&2
+ exit 1
+fi
+
+KEEP_DEFAULT_TOOLS_FLAG=""
+ONLY_WRONG_FLAG=""
+if [[ "${EPOCH}" -gt 0 ]]; then
+ KEEP_DEFAULT_TOOLS_FLAG="--keep-default-tools"
+ ONLY_WRONG_FLAG="--only-wrong"
+fi
+
+if [[ "${RESULT_DIR}" = /* ]]; then
+ OUTPUT_ROOT="${RESULT_DIR}"
+else
+ OUTPUT_ROOT="${CUR_DIR}/${RESULT_DIR}"
+fi
+TRAIN_DIR="${OUTPUT_ROOT}/${DOMAIN}_train"
+TEST_DIR="${OUTPUT_ROOT}/${DOMAIN}_test"
+AGENT_ID=${DOMAIN}_v0
+AGENT_ID_FLAG="--agent-id ${AGENT_ID}"
+# train runs once per epoch; test repeats are averaged.
+TRAIN_TRY_NO=0
+
+echo "[run_full_test] Start ${DOMAIN}: 1 train run + ${TEST_REPEATS} test run(s), in parallel..."
+
+# train: a single run per epoch
+bash "${SCRIPTS_DIR}/run_tau2_domain.sh" \
+ --domain "${DOMAIN}" \
+ --split train \
+ --epoch "${EPOCH}" \
+ --try-no "${TRAIN_TRY_NO}" \
+ --result-dir "${RESULT_DIR}" \
+ --concurrency "${CONCURRENCY}" \
+ ${KEEP_DEFAULT_TOOLS_FLAG} \
+ --use-continue ${AGENT_ID_FLAG} &
+PIDS+=("$!")
+
+# test: TEST_REPEATS independent runs (try-no 0 .. TEST_REPEATS-1), averaged at eval time
+for ((t=0; t<TEST_REPEATS; t++)); do
+ bash "${SCRIPTS_DIR}/run_tau2_domain.sh" \
+ --domain "${DOMAIN}" \
+ --split test \
+ --epoch "${EPOCH}" \
+ --try-no "${t}" \
+ --result-dir "${RESULT_DIR}" \
+ --concurrency "${CONCURRENCY}" \
+ ${KEEP_DEFAULT_TOOLS_FLAG} \
+ --use-continue ${AGENT_ID_FLAG} &
+ PIDS+=("$!")
+done
+
+for pid in "${PIDS[@]}"; do
+ wait "${pid}" || true
+done
+PIDS=()
+
+REPORT_PATH="${CUR_DIR}/full_test_report_${DOMAIN}.txt"
+{
+ echo "==== Tau2 Full Test Report ===="
+ echo "Domain: ${DOMAIN}"
+ echo "Epoch: ${EPOCH} Test repeats: ${TEST_REPEATS}"
+ echo "Result dir: ${RESULT_DIR}"
+ echo "Timestamp: $(date '+%Y-%m-%d %H:%M:%S')"
+ echo
+ echo "-- Train Reward (try ${TRAIN_TRY_NO}) --"
+ bash "${SCRIPTS_DIR}/run_eval_reward.sh" "${TRAIN_DIR}" "${EPOCH}" "${TRAIN_TRY_NO}" || true
+ echo
+ echo "-- Test Reward (averaged over ${TEST_REPEATS} repeat(s)) --"
+ acc_file="$(mktemp)"
+ for ((t=0; t<TEST_REPEATS; t++)); do
+ echo "[test try=${t}]"
+ test_out="$(bash "${SCRIPTS_DIR}/run_eval_reward.sh" "${TEST_DIR}" "${EPOCH}" "${t}" || true)"
+ echo "${test_out}"
+ echo "${test_out}" | awk -F': ' '/Average reward:/ {print $2}' | tail -n 1 >> "${acc_file}"
+ echo
+ done
+ echo -n "Test average accuracy over ${TEST_REPEATS} repeat(s): "
+ awk '{sum += $1; n++} END {if (n > 0) printf "%.6f\n", sum / n; else print "n/a"}' "${acc_file}"
+ rm -f "${acc_file}"
+} | tee -a "${REPORT_PATH}"
+
+echo "[run_full_test] Report saved to: ${REPORT_PATH}"
+
+# train: commit trajectories to extract memory (consumed by the next epoch)
+if [[ "${DO_COMMIT}" -eq 1 ]]; then
+ echo "[run_full_test] Commit train trajectories to memory..."
+ python "${SCRIPTS_DIR}/commit_trajectory_to_memory.py" \
+ --input "${TRAIN_DIR}" \
+ --domain "${AGENT_ID}" \
+ --pattern "*_${EPOCH}_${TRAIN_TRY_NO}_trajectory.json" \
+ --include-eval-result \
+ ${ONLY_WRONG_FLAG}
+else
+ echo "[run_full_test] Skip commit (--no-commit)"
+fi
diff --git a/benchmark/tau2/vikingbot/scripts/commit_trajectory_to_memory.py b/benchmark/tau2/vikingbot/scripts/commit_trajectory_to_memory.py
new file mode 100644
index 000000000..c8af81da9
--- /dev/null
+++ b/benchmark/tau2/vikingbot/scripts/commit_trajectory_to_memory.py
@@ -0,0 +1,204 @@
+#!/usr/bin/env python3
+"""Commit runner trajectories into OpenViking memory.
+
+This script reads trajectory JSON files produced by vikingbot_tau2_runner.py
+and commits a minimal conversation (user -> assistant) into OpenViking.
+
+Usage:
+ python3 commit_trajectory_to_memory.py --input /path/to/result_dir
+ python3 commit_trajectory_to_memory.py --input /path/to/file.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+from pathlib import Path
+from typing import Any, Iterable
+
+from vikingbot.config.loader import load_config
+from vikingbot.openviking_mount.ov_server import VikingClient
+
+
+def _iter_files(root: Path, pattern: str) -> Iterable[Path]:
+ if root.is_file():
+ yield root
+ return
+ for path in sorted(root.glob(pattern)):
+ if path.is_file():
+ yield path
+
+
+def _build_messages(
+ data: dict[str, Any],
+ include_eval_result: bool,
+) -> list[dict[str, str]]:
+ messages: list[dict[str, str]] = []
+
+ system_prompt = data.get("system_prompt") or ""
+ if system_prompt:
+ messages.append({"role": "user", "content": f"system:\n{system_prompt}"})
+
+ user_text = data.get("user_prompt") or data.get("user_query") or ""
+ if user_text:
+ messages.append({"role": "user", "content": user_text})
+
+ tools_used = data.get("tools_used") or []
+ if isinstance(tools_used, list):
+ for tool_info in tools_used:
+ if not isinstance(tool_info, dict):
+ continue
+ tool_name = tool_info.get("tool_name", "")
+ args = tool_info.get("args", "")
+ assistant_content = tool_name
+ if args:
+ assistant_content = f"tool-call: \n name: {tool_name}\n arguments: {args}"
+ if assistant_content:
+ messages.append({"role": "assistant", "content": assistant_content})
+
+ result = tool_info.get("result")
+ if result is not None:
+ messages.append({"role": "user", "content": f"tool-response: \n{result}"})
+ messages.append({"role": "assistant", "content": data.get("final_content", "")})
+ reward = data.get("reward")
+ success = bool(reward == 1)
+ result = f"task_success: {success}"
+ if include_eval_result and data.get("evaluation_result") is not None:
+ evaluation_result = data.get("evaluation_result")
+ if not isinstance(evaluation_result, str):
+ evaluation_result = json.dumps(evaluation_result, ensure_ascii=False)
+ result += f"\n evaluation report: {evaluation_result}"
+ messages.append({"role": "user", "content": result})
+ return messages
+
+
+def _get_session_id(data: dict[str, Any], prefix: str | None) -> str | None:
+ session_id = data.get("session_id")
+ if session_id:
+ return f"{prefix}{session_id}" if prefix else session_id
+ data_split = data.get("data_split")
+ task_no = data.get("task_no")
+ if data_split is not None and task_no is not None:
+ base = f"tau2_{data_split}_{task_no}"
+ return f"{prefix}{base}" if prefix else base
+ return None
+
+
+async def _commit_single(
+ client: VikingClient,
+ path: Path,
+ user_id: str,
+ session_prefix: str | None,
+ dry_run: bool,
+ only_wrong: bool,
+ include_eval_result: bool,
+) -> tuple[Path, bool, str]:
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ return path, False, f"Failed to load JSON: {exc}"
+ if only_wrong and data.get("reward") == 1:
+ return path, False, "Skipped: reward == 1"
+ messages = _build_messages(data, include_eval_result)
+ if not messages:
+ return path, False, "No messages to commit"
+
+ session_id = _get_session_id(data, session_prefix)
+ if not session_id:
+ return path, False, "Missing session_id/data_split/task_no"
+
+ if dry_run:
+ print(messages)
+ return path, True, f"DRY RUN: would commit {len(messages)} messages to {session_id}"
+ await client.commit(session_id, messages, user_id)
+
+ return path, True, f"Committed {len(messages)} messages to {session_id}"
+
+
+async def main_async() -> int:
+ parser = argparse.ArgumentParser(description="Commit runner trajectories to OpenViking")
+ parser.add_argument("--input", required=True, help="Trajectory file or directory")
+ parser.add_argument(
+ "--pattern",
+ default="*.json",
+ help="Glob pattern when input is a directory (default: *.json)",
+ )
+ parser.add_argument(
+ "--user-id",
+ default=None,
+ help="User id for commit (default: ov.conf admin_user_id)",
+ )
+ parser.add_argument(
+ "--session-prefix",
+ default=None,
+ help="Prefix to prepend to session_id",
+ )
+ parser.add_argument(
+ "--domain",
+ default="telecom",
+ help="Use as agent_id when creating VikingClient",
+ )
+ parser.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="Only print what would be committed",
+ )
+ parser.add_argument(
+ "--only-wrong",
+ action="store_true",
+ help="Only commit trajectories with reward != 1",
+ )
+ parser.add_argument(
+ "--include-eval-result",
+ action="store_true",
+ help="Include evaluation_result in the final user message",
+ )
+ args = parser.parse_args()
+
+ root = Path(args.input).expanduser()
+ if not root.exists():
+ print(f"Input not found: {root}")
+ return 1
+
+ config = load_config()
+ user_id = args.user_id or config.ov_server.admin_user_id
+ if not user_id:
+ print("Missing user_id (pass --user-id or set ov.conf admin_user_id)")
+ return 1
+
+ client = await VikingClient.create(args.domain)
+ print(client.agent_id)
+ print(client.get_agent_space_name("default"))
+ print(client.client._agent_id)
+ files = list(_iter_files(root, args.pattern))
+ if not files:
+ print("No files matched.")
+ return 1
+
+ ok = 0
+ for path in files:
+ _, success, msg = await _commit_single(
+ client,
+ path,
+ user_id=user_id,
+ session_prefix=args.session_prefix,
+ dry_run=args.dry_run,
+ only_wrong=args.only_wrong,
+ include_eval_result=args.include_eval_result,
+ )
+ status = "OK" if success else "SKIP"
+ print(f"[{status}] {path.name} - {msg}")
+ if success:
+ ok += 1
+
+ print(f"Committed {ok}/{len(files)} files")
+ return 0
+
+
+def main() -> None:
+ raise SystemExit(asyncio.run(main_async()))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/tau2/vikingbot/scripts/run_eval_reward.sh b/benchmark/tau2/vikingbot/scripts/run_eval_reward.sh
new file mode 100755
index 000000000..67516a169
--- /dev/null
+++ b/benchmark/tau2/vikingbot/scripts/run_eval_reward.sh
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Evaluate average reward for a given result folder, epoch, and try_no.
+# Usage: bash run_eval_reward.sh <result_dir> [epoch] [try_no]
+
+RESULT_DIR=${1:-""}
+EPOCH=${2:-0}
+TRY_NO=${3:-0}
+
+if [[ -z "${RESULT_DIR}" ]]; then
+ echo "Usage: bash run_eval_reward.sh <result_dir> [epoch] [try_no]" >&2
+ exit 1
+fi
+
+if [[ ! -d "${RESULT_DIR}" ]]; then
+ echo "Result dir not found: ${RESULT_DIR}" >&2
+ exit 1
+fi
+
+RESULT_DIR="${RESULT_DIR}" EPOCH="${EPOCH}" TRY_NO="${TRY_NO}" python3 - <<'PY'
+import glob
+import json
+import os
+import sys
+
+result_dir = os.environ.get("RESULT_DIR")
+epoch = os.environ.get("EPOCH")
+try_no = os.environ.get("TRY_NO")
+
+pattern = os.path.join(result_dir, f"task_*_{epoch}_{try_no}_trajectory.json")
+files = sorted(glob.glob(pattern))
+
+if not files:
+ print(f"No files matched: {pattern}")
+ sys.exit(1)
+
+total = 0.0
+count = 0
+missing = 0
+
+for path in files:
+ try:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ reward = data.get("reward")
+ if reward is None:
+ missing += 1
+ continue
+ total += float(reward)
+ count += 1
+ except Exception:
+ missing += 1
+
+avg = total / count if count else 0.0
+print(f"Result dir: {result_dir}")
+print(f"Epoch: {epoch} Try: {try_no}")
+print(f"Matched files: {len(files)}")
+print(f"Used rewards: {count} Missing/invalid: {missing}")
+print(f"Average reward: {avg:.6f}")
+PY
diff --git a/benchmark/tau2/vikingbot/scripts/run_tau2_domain.sh b/benchmark/tau2/vikingbot/scripts/run_tau2_domain.sh
new file mode 100755
index 000000000..9be4bc79b
--- /dev/null
+++ b/benchmark/tau2/vikingbot/scripts/run_tau2_domain.sh
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Run VikingBot tau2 runner for all tasks in {domain}_{train|test} split.
+# Usage:
+# bash run_tau2_domain.sh --domain telecom --split train --epoch 0 --try-no 0 \
+# --keep-default-tools --result-dir result --use-continue --concurrency 5
+
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
+RUNNER="${SCRIPT_DIR}/vikingbot_tau2_runner.py"
+
+DOMAIN=""
+SPLIT=""
+EPOCH=0
+TRY_NO=0
+KEEP_DEFAULT_TOOLS=0
+RESULT_DIR="result"
+USE_CONTINUE=0
+AGENT_ID=""
+CONCURRENCY=1
+
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --domain)
+ DOMAIN="$2"; shift 2 ;;
+ --split)
+ SPLIT="$2"; shift 2 ;;
+ --epoch)
+ EPOCH="$2"; shift 2 ;;
+ --try-no)
+ TRY_NO="$2"; shift 2 ;;
+ --keep-default-tools)
+ KEEP_DEFAULT_TOOLS=1; shift 1 ;;
+ --result-dir)
+ RESULT_DIR="$2"; shift 2 ;;
+ --use-continue)
+ USE_CONTINUE=1; shift 1 ;;
+ --agent-id)
+ AGENT_ID="$2"; shift 2 ;;
+ --concurrency)
+ CONCURRENCY="$2"; shift 2 ;;
+ -h|--help)
+ echo "Usage: bash run_tau2_domain.sh --domain DOMAIN --split train|test --epoch N --try-no N --keep-default-tools --result-dir DIR --use-continue --agent-id ID --concurrency N"; exit 0 ;;
+ *)
+ echo "Unknown argument: $1" >&2; exit 1 ;;
+ esac
+done
+
+if [[ -z "${DOMAIN}" || ( "${SPLIT}" != "train" && "${SPLIT}" != "test" ) ]]; then
+ echo "Usage: bash run_tau2_domain.sh --domain DOMAIN --split train|test --epoch N --try-no N --keep-default-tools --result-dir DIR --use-continue --agent-id ID --concurrency N" >&2
+ exit 1
+fi
+
+DATA_ROOT="${TAU2_DATA_ROOT}"
+SPLIT_TASKS_JSON="${DATA_ROOT}/domains/${DOMAIN}/split_tasks.json"
+
+if [[ "${RESULT_DIR}" = /* ]]; then
+ OUTPUT_ROOT="${RESULT_DIR}/${DOMAIN}_${SPLIT}"
+else
+ OUTPUT_ROOT="${REPO_ROOT}/${RESULT_DIR}/${DOMAIN}_${SPLIT}"
+fi
+mkdir -p "${OUTPUT_ROOT}"
+
+if [[ ! -f "${SPLIT_TASKS_JSON}" ]]; then
+ echo "Split file not found: ${SPLIT_TASKS_JSON}" >&2
+ exit 1
+fi
+
+TASK_COUNT=$(SPLIT_TASKS_JSON="${SPLIT_TASKS_JSON}" SPLIT_NAME="${SPLIT}" python3 - <<'PY'
+import json
+import os
+
+path = os.environ.get("SPLIT_TASKS_JSON")
+split_name = os.environ.get("SPLIT_NAME")
+with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+print(len(data[split_name]))
+PY
+)
+
+echo "${DOMAIN}_${SPLIT} task count: ${TASK_COUNT}, concurrency: ${CONCURRENCY}"
+
+# build reusable flags
+KEEP_DEFAULT_TOOLS_FLAG=""
+CONTINUE_FLAG=""
+AGENT_ID_FLAG=""
+[[ "${KEEP_DEFAULT_TOOLS}" == "1" ]] && KEEP_DEFAULT_TOOLS_FLAG="--keep-default-tools"
+[[ "${USE_CONTINUE}" == "1" ]] && CONTINUE_FLAG="--continue"
+[[ -n "${AGENT_ID}" ]] && AGENT_ID_FLAG="--agent-id ${AGENT_ID}"
+
+# Use a tmpdir as a semaphore: each running task holds a slot file.
+SEM_DIR=$(mktemp -d)
+trap 'rm -rf "${SEM_DIR}"' EXIT
+
+active_count() { ls "${SEM_DIR}" 2>/dev/null | wc -l | tr -d ' '; }
+
+for ((task_no=0; task_no < TASK_COUNT; task_no++)); do
+ out_path="${OUTPUT_ROOT}/task_${task_no}_${EPOCH}_${TRY_NO}_trajectory.json"
+ echo "[${DOMAIN}_${SPLIT}] task_no=${task_no} epoch=${EPOCH} try_no=${TRY_NO}"
+
+ # wait for a free slot
+ while [[ $(active_count) -ge ${CONCURRENCY} ]]; do sleep 0.5; done
+
+ SLOT="${SEM_DIR}/${task_no}"
+ touch "${SLOT}"
+ (
+ python "${RUNNER}" \
+ --data-split "${DOMAIN}_${SPLIT}" \
+ --task-no "${task_no}" \
+ --output "${out_path}" ${AGENT_ID_FLAG} ${KEEP_DEFAULT_TOOLS_FLAG} ${CONTINUE_FLAG} \
+ || echo "[WARN] task_no=${task_no} failed, skipping"
+ rm -f "${SLOT}"
+ ) &
+done
+
+wait || true
+rm -rf "${SEM_DIR}"
diff --git a/benchmark/tau2/vikingbot/scripts/vikingbot_tau2_runner.py b/benchmark/tau2/vikingbot/scripts/vikingbot_tau2_runner.py
new file mode 100644
index 000000000..39aa9de41
--- /dev/null
+++ b/benchmark/tau2/vikingbot/scripts/vikingbot_tau2_runner.py
@@ -0,0 +1,309 @@
+#!/usr/bin/env python3
+"""Run a single tau2-bench task with VikingBot AgentLoop + Tau2 tools.
+
+Key points:
+1) Tau2BenchEnv is initialized once and exposes tools via Tau2BenchToolProvider.call_tool().
+2) VikingBot already has its own multi-iteration agent loop, so we call it once.
+3) We register Tau2 tools into VikingBot ToolRegistry and let the agent decide.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+REPO_ROOT = SCRIPT_DIR.parent
+if str(REPO_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPO_ROOT))
+
+from tau2_env.tau2_tool_provider import Tau2BenchToolProvider, load_task_id
+
+try:
+ from vikingbot.agent.loop import AgentLoop
+ from vikingbot.agent.tools.base import Tool, ToolContext
+ from vikingbot.bus.queue import MessageBus
+ from vikingbot.cli.commands import _init_bot_data, _make_provider
+ from vikingbot.config.loader import ensure_config
+ from vikingbot.config.schema import SessionKey
+ from vikingbot.sandbox.manager import SandboxManager
+ from vikingbot.session.manager import SessionManager
+ from vikingbot.utils.helpers import get_source_workspace_path
+except ImportError as exc:
+ raise RuntimeError(
+ "Failed to import vikingbot. Make sure OpenViking/vikingbot is installed "
+ "or available on PYTHONPATH."
+ ) from exc
+
+
+class Tau2Tool(Tool):
+ """Bridge tau2 tool schema into VikingBot Tool interface."""
+
+ def __init__(self, schema: dict[str, Any], provider: Tau2BenchToolProvider):
+ self._schema = schema
+ self._provider = provider
+ function_def = schema.get("function", {}) if isinstance(schema, dict) else {}
+ self._name = function_def.get("name", "")
+ self._description = function_def.get("description", "")
+ self._parameters = function_def.get("parameters", {})
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def description(self) -> str:
+ return self._description
+
+ @property
+ def parameters(self) -> dict[str, Any]:
+ return self._parameters
+
+ async def execute(self, tool_context: ToolContext, **kwargs: Any) -> str:
+ return self._provider.call_tool(self._name, kwargs)
+
+
+def _build_agent(config_path: str | None) -> AgentLoop:
+ config = ensure_config(Path(config_path).expanduser() if config_path else None)
+ _init_bot_data(config)
+
+ bus = MessageBus()
+ session_manager = SessionManager(config.bot_data_path)
+
+ sandbox_parent_path = config.workspace_path
+ source_workspace_path = get_source_workspace_path()
+ sandbox_manager = SandboxManager(config, sandbox_parent_path, source_workspace_path)
+
+ provider = _make_provider(config)
+
+ return AgentLoop(
+ bus=bus,
+ provider=provider,
+ workspace=config.workspace_path,
+ model=config.agents.model,
+ # max_iterations=config.agents.max_tool_iterations,
+ max_iterations=30,
+ memory_window=config.agents.memory_window,
+ brave_api_key=config.tools.web.search.api_key or None,
+ exa_api_key=None,
+ gen_image_model=config.agents.gen_image_model,
+ exec_config=config.tools.exec,
+ cron_service=None,
+ session_manager=session_manager,
+ sandbox_manager=sandbox_manager,
+ config=config,
+ eval=True,
+ mcp_servers=None,
+ )
+
+
+# Stable markers around the injected memory block in VikingBot's prompt wrapper.
+# The wrapper looks like:
+# ## Current Time: <system time>\n\n---\n\n## Current Session
+# Channel: cli\n\n---\n\n<MEMORY>---\n\nReply in the same language ... User's query:
+# The current time is generated by the system at runtime and may be anything, so we
+# anchor only on the session header and the trailing reply instruction — never on the time.
+MEMORY_PROMPT_PREFIX = "## Current Session\nChannel: cli\n\n---\n\n"
+MEMORY_PROMPT_SUFFIX = (
+ "---\n\nReply in the same language as the user's query, ignoring the language of "
+ "the reference materials. User's query:"
+)
+
+
+def _extract_memory_content(content: str) -> str | None:
+ """Extract the injected memory block from VikingBot's prompt wrapper.
+
+ Time-independent: locates the block between the session header and the trailing
+ reply instruction, regardless of the (system-generated) current time.
+ """
+ start = content.find(MEMORY_PROMPT_PREFIX)
+ end = content.rfind(MEMORY_PROMPT_SUFFIX)
+ if start == -1 or end == -1:
+ return None
+ start += len(MEMORY_PROMPT_PREFIX)
+ if start > end:
+ return None
+ return content[start:end]
+
+
+
+
+async def _run_agent(
+ agent: AgentLoop,
+ system_prompt: str,
+ user_prompt: str,
+ session_key: SessionKey,
+ sender_id: str | None,
+ agent_id: str | None,
+ keep_default_tools: bool,
+ messages_output_path: Path | None,
+) -> tuple[str | None, str | None, dict[str, int], int, str | None]:
+ messages = await agent.context.build_messages(
+ history=[],
+ current_message=user_prompt,
+ session_key=session_key,
+ ov_tools_enable=keep_default_tools,
+ media=None,
+ profile_user_list=[],
+ memory_users=agent_id,
+ )
+ if system_prompt:
+ messages.insert(1, {"role": "system", "content": system_prompt})
+
+ memory_content = None
+ if len(messages) > 2 and isinstance(messages[2].get("content"), str):
+ memory_content = _extract_memory_content(messages[2]["content"])
+
+ result = await agent._run_agent_loop(
+ messages=messages,
+ session_key=session_key,
+ publish_events=False,
+ sender_id=sender_id,
+ ov_tools_enable=keep_default_tools,
+ )
+ if messages_output_path is not None:
+ messages_output_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(messages_output_path, "w", encoding="utf-8") as f:
+ json.dump(messages, f, ensure_ascii=False, indent=2)
+ print(f"Saved Full Messages: {messages_output_path}")
+ return (*result, memory_content)
+
+
+def _derive_messages_path(output_path: Path) -> Path:
+ parent_parts = [part.replace("result", "trajectory") for part in output_path.parent.parts]
+ base_path = Path(*parent_parts) / output_path.name
+
+ if base_path.name.endswith("_trajectory.json"):
+ return base_path.with_name(base_path.name.replace("_trajectory.json", "_messages.json"))
+ return base_path.with_name(f"{base_path.stem}_messages{base_path.suffix}")
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="VikingBot tau2-bench runner")
+ parser.add_argument("--data-split", required=True, help="e.g. telecom_test")
+ parser.add_argument("--task-no", type=int, required=True, help="index in split list")
+ parser.add_argument("--output", default="./result/tau2_trajectory.json")
+ parser.add_argument(
+ "--continue",
+ dest="continue_run",
+ action="store_true",
+ help="If output file exists, skip running and exit",
+ )
+ parser.add_argument("--sender", default="tau2_user")
+ parser.add_argument("--agent-id", default="", help="airline_v0 domain split workspace")
+ parser.add_argument("--session", default=None)
+ parser.add_argument("--config", default=None, help="ov.conf path (optional)")
+ parser.add_argument(
+ "--keep-default-tools",
+ action="store_true",
+ help="Keep VikingBot default tools (default: only tau2 tools)",
+ )
+ parser.add_argument(
+ "--max-iterations",
+ type=int,
+ default=None,
+ help="Override agent max iterations for this run",
+ )
+ args = parser.parse_args()
+
+ output_path = Path(args.output)
+ if output_path.exists():
+ if args.continue_run:
+ print(f"[Runner] Output exists, skip run: {output_path}")
+ return
+ print(f"[Runner] Output exists, will overwrite: {output_path}")
+
+ domain, task_id = load_task_id(args.data_split, args.task_no)
+ print("[Runner] Initializing tau2 environment...", flush=True)
+ provider = Tau2BenchToolProvider(domain, task_id)
+ provider.reset()
+
+ user_query = provider.user_query
+ policy = provider.policy
+ ground_truth = provider.ground_truth
+
+ agent = _build_agent(args.config)
+ if args.max_iterations is not None:
+ agent.max_iterations = args.max_iterations
+
+ if not args.keep_default_tools:
+ for tool_name in list(agent.tools.tool_names):
+ # print(tool_name)
+ agent.tools.unregister(tool_name)
+
+ agent.tools.unregister("openviking_memory_commit")
+ for schema in provider.list_openai_tools():
+ agent.tools.register(Tau2Tool(schema, provider))
+
+ instructions = []
+ if policy:
+ instructions.append(policy)
+ instructions.append("Use the provided tools to interact with the environment.")
+ if args.keep_default_tools:
+ instructions.append("Before you attend to customer, you MUST read relevant agent memory that stores experiences distilled from similar tasks and carefully learn them.")
+ instructions.append(
+ "If you need to communicate with the user, you MUST call tool `communicate_with_user`."
+ )
+ instructions.append("When the task is finished or terminated, call tool `done` first and output an ending content without using any tool calling for the next round to exit.")
+
+ system_prompt = "\n".join(instructions)
+ user_prompt = user_query
+
+ session_id = args.session or f"tau2_{args.data_split}_{args.task_no}"
+ session_key = SessionKey(type="cli", channel_id="tau2", chat_id=session_id)
+
+ messages_output_path = _derive_messages_path(output_path)
+
+ final_content, final_reasoning_content, tools_used, token_usage, iteration, memory_content = asyncio.run(
+ _run_agent(
+ agent,
+ system_prompt,
+ user_prompt,
+ session_key,
+ args.sender,
+ args.agent_id,
+ args.keep_default_tools,
+ messages_output_path,
+ )
+ )
+
+ reward = None
+ evaluation_result = None
+ if provider.env is not None:
+ try:
+ reward, evaluation_result = provider.env.env._get_reward()
+ except Exception:
+ pass
+
+ trajectory = {
+ "data_split": args.data_split,
+ "task_no": args.task_no,
+ "domain": domain,
+ "task_id": task_id,
+ "user_query": user_query,
+ "policy": policy,
+ "ground_truth": ground_truth,
+ "system_prompt": system_prompt,
+ "user_prompt": user_prompt,
+ "final_content": final_content,
+ "reward": reward,
+ "evaluation_result": evaluation_result,
+ "tools_used": tools_used,
+ "token_usage": token_usage,
+ "iterations": iteration,
+ "memory": memory_content,
+ }
+
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(output_path, "w", encoding="utf-8") as f:
+ json.dump(trajectory, f, ensure_ascii=False, indent=2)
+
+ print(f"Saved: {output_path}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/tau2/vikingbot/setup_env.sh b/benchmark/tau2/vikingbot/setup_env.sh
new file mode 100755
index 000000000..8d9536074
--- /dev/null
+++ b/benchmark/tau2/vikingbot/setup_env.sh
@@ -0,0 +1,198 @@
+#!/usr/bin/env bash
+# One-step environment setup for the VikingBot × tau2-bench runner.
+#
+# Does everything in a single step:
+# 1. creates a fresh .venv at the OpenViking repo root (if missing)
+# 2. clones tau2-bench into ./tau2-bench (if missing; external dependency)
+# 3. installs openviking + vikingbot (pip install -e .[bot] -> runs the Cargo build;
+# the [bot] extra provides prompt_toolkit/gradio/mcp/... needed by the runner and
+# `openviking-server --with-bot`)
+# 4. ensures the ragfs_python native binding is built (via maturin) and bundled into
+# openviking/lib/ (the editable install can skip it under pip build isolation)
+# 5. installs tau2-bench with the gym extra (pip install -e ./tau2-bench[gym] -> gymnasium)
+# 6. installs smolagents
+# 7. activates the venv and exports the runtime env vars
+#
+# Usage:
+# source setup_env.sh # install on first run, then activate + export
+# source setup_env.sh --reinstall # delete .venv and rebuild from scratch
+#
+# Safe to source repeatedly: the install phase runs only when the venv is missing
+# (or when --reinstall is passed); later sources just activate + export.
+#
+# Overridable via env vars (export before sourcing):
+# TAU2_BENCH_ROOT tau2-bench checkout location (default ./tau2-bench)
+# TAU2_BENCH_REPO git URL to clone tau2-bench (default sierra-research/tau2-bench)
+# TAU2_BENCH_REF git ref/branch/tag to check out after clone (default: repo default)
+# VIKINGBOT_ROOT vikingbot package dir (default REPO_ROOT/bot)
+# OPENVIKING_CONFIG_FILE, OPENAI_API_KEY / ARK_API_KEY, OPENAI_API_BASE
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# OpenViking repo root (this folder lives at benchmark/tau2/vikingbot/).
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
+VENV="${REPO_ROOT}/.venv"
+SETUP_MARKER="${VENV}/.tau2_setup_complete"
+
+# tau2-bench checkout (external dependency; gitignored). Resolved early so the
+# install phase can clone/install it and the exports can derive TAU2_DATA_ROOT.
+TAU2_BENCH_ROOT="${TAU2_BENCH_ROOT:-${SCRIPT_DIR}/tau2-bench}"
+TAU2_BENCH_REPO="${TAU2_BENCH_REPO:-https://github.com/sierra-research/tau2-bench}"
+
+# Detect source vs execute so errors abort cleanly without killing the user's shell.
+(return 0 2>/dev/null) && _SOURCED=1 || _SOURCED=0
+_abort() { echo "[setup_env] ERROR: $*" >&2; if [[ "${_SOURCED}" -eq 1 ]]; then return 1; else exit 1; fi; }
+
+# --- parse args ---
+REINSTALL=0
+for _arg in "$@"; do
+ case "${_arg}" in
+ --reinstall|--force) REINSTALL=1 ;;
+ -h|--help)
+ echo "Usage: source setup_env.sh [--reinstall]"
+ { [[ "${_SOURCED}" -eq 1 ]] && return 0; } 2>/dev/null || exit 0 ;;
+ esac
+done
+
+# --- install phase (one-time) ---------------------------------------------------
+_setup_install() {
+ if [[ "${REINSTALL}" -eq 1 && -d "${VENV}" ]]; then
+ echo "[setup_env] --reinstall: removing ${VENV}"
+ rm -rf "${VENV}"
+ fi
+
+ if [[ -f "${SETUP_MARKER}" && "${REINSTALL}" -eq 0 ]]; then
+ return 0 # already installed; nothing to do
+ fi
+
+ if [[ ! -f "${VENV}/bin/activate" ]]; then
+ echo "[setup_env] Creating venv at ${VENV}"
+ python3 -m venv "${VENV}" || { echo "[setup_env] failed to create venv"; return 1; }
+ fi
+
+ local PY="${VENV}/bin/python"
+
+ echo "[setup_env] Upgrading pip/setuptools/wheel"
+ "${PY}" -m pip install --upgrade pip setuptools wheel || return 1
+
+ # tau2-bench: external checkout. Clone it if it isn't present yet.
+ if [[ ! -d "${TAU2_BENCH_ROOT}" ]]; then
+ if ! command -v git >/dev/null 2>&1; then
+ echo "[setup_env] 'git' not found; cannot clone tau2-bench into ${TAU2_BENCH_ROOT}"; return 1
+ fi
+ echo "[setup_env] Cloning tau2-bench: ${TAU2_BENCH_REPO} -> ${TAU2_BENCH_ROOT}"
+ git clone "${TAU2_BENCH_REPO}" "${TAU2_BENCH_ROOT}" || { echo "[setup_env] tau2-bench clone failed"; return 1; }
+ if [[ -n "${TAU2_BENCH_REF:-}" ]]; then
+ echo "[setup_env] Checking out tau2-bench ref: ${TAU2_BENCH_REF}"
+ git -C "${TAU2_BENCH_ROOT}" checkout "${TAU2_BENCH_REF}" || { echo "[setup_env] tau2-bench checkout failed"; return 1; }
+ fi
+ fi
+
+ # openviking + vikingbot: editable install of the repo root (pyproject finds both).
+ # NOTE: this triggers a Cargo/Rust release build of the `ov` CLI.
+ if ! command -v cargo >/dev/null 2>&1; then
+ echo "[setup_env] WARNING: 'cargo' not found on PATH; the openviking Rust build may fail." >&2
+ fi
+ # Install with the [bot] extra: the vikingbot runner imports vikingbot.cli and
+ # `openviking-server --with-bot` needs the bot deps (prompt_toolkit, gradio, mcp, ...).
+ echo "[setup_env] Installing openviking + vikingbot with bot extras (pip install -e ${REPO_ROOT}[bot])"
+ "${PY}" -m pip install -e "${REPO_ROOT}[bot]" || { echo "[setup_env] openviking install failed"; return 1; }
+
+ # Ensure the ragfs_python native (PyO3) binding is bundled. The editable install
+ # can skip the maturin build under pip build isolation, leaving the server unable
+ # to load RAGFS ("ragfs_python native library is not available"). If it's not
+ # importable, build it via maturin and copy the extension into openviking/lib/.
+ if ! "${PY}" -c "from openviking.pyagfs import get_binding_client; get_binding_client()" >/dev/null 2>&1; then
+ echo "[setup_env] ragfs_python native binding missing; building via maturin..."
+ "${PY}" -m pip install "maturin>=1.0,<2.0" || { echo "[setup_env] maturin install failed"; return 1; }
+ _ragfs_out="$(mktemp -d)"
+ if "${PY}" -m maturin build --release -m "${REPO_ROOT}/crates/ragfs-python/Cargo.toml" --out "${_ragfs_out}"; then
+ "${PY}" - "${_ragfs_out}" "${REPO_ROOT}/openviking/lib" <<'PYEOF'
+import sys, os, glob, zipfile
+from pathlib import Path
+out_dir, lib_dir = sys.argv[1], Path(sys.argv[2])
+whls = sorted(glob.glob(os.path.join(out_dir, "ragfs_python-*.whl")))
+if not whls:
+ sys.exit("maturin produced no ragfs_python wheel")
+lib_dir.mkdir(parents=True, exist_ok=True)
+for pat in ("ragfs_python*.so", "ragfs_python*.pyd", "ragfs_python*.dylib"):
+ for stale in lib_dir.glob(pat):
+ stale.unlink()
+with zipfile.ZipFile(whls[0]) as zf:
+ for name in zf.namelist():
+ base = Path(name).name
+ if base == "ragfs_python.pyd" or (base.startswith("ragfs_python.abi3.") and base.endswith((".so", ".pyd"))):
+ target = lib_dir / base
+ with zf.open(name) as src, open(target, "wb") as dst:
+ dst.write(src.read())
+ if not sys.platform.startswith("win"):
+ os.chmod(target, 0o755)
+ print(f"[setup_env] ragfs_python: bundled {base} -> {target}")
+ break
+ else:
+ sys.exit("ragfs_python native extension not found in wheel")
+PYEOF
+ _ragfs_rc=$?
+ rm -rf "${_ragfs_out}"
+ [[ "${_ragfs_rc}" -eq 0 ]] || { echo "[setup_env] ragfs_python bundling failed"; return 1; }
+ else
+ rm -rf "${_ragfs_out}"
+ echo "[setup_env] maturin build of ragfs_python failed"; return 1
+ fi
+ else
+ echo "[setup_env] ragfs_python native binding present"
+ fi
+
+ # tau2-bench: install the [gym] extra so tau2.gym (gymnasium) is available to the runner.
+ echo "[setup_env] Installing tau2-bench with gym extra (pip install -e ${TAU2_BENCH_ROOT}[gym])"
+ "${PY}" -m pip install -e "${TAU2_BENCH_ROOT}[gym]" || { echo "[setup_env] tau2-bench install failed"; return 1; }
+
+ echo "[setup_env] Installing smolagents"
+ "${PY}" -m pip install smolagents || { echo "[setup_env] smolagents install failed"; return 1; }
+
+ touch "${SETUP_MARKER}"
+ echo "[setup_env] Install complete."
+ return 0
+}
+
+if ! _setup_install; then
+ _abort "environment install failed (see messages above)"
+ unset -f _setup_install _abort
+ return 1 2>/dev/null || exit 1
+fi
+unset -f _setup_install
+
+# --- activate venv --------------------------------------------------------------
+if [[ ! -f "${VENV}/bin/activate" ]]; then
+ _abort "venv not found at ${VENV} after install"
+ return 1 2>/dev/null || exit 1
+fi
+# shellcheck disable=SC1090
+source "${VENV}/bin/activate"
+echo "[setup_env] venv activated: ${VENV}"
+
+# --- runtime env vars -----------------------------------------------------------
+# openviking + vikingbot come from the editable install; REPO_ROOT + bot/ kept on
+# PYTHONPATH as a harmless import fallback.
+OPENVIKING_TAU2_ROOT="${REPO_ROOT}"
+VIKINGBOT_ROOT="${VIKINGBOT_ROOT:-${REPO_ROOT}/bot}"
+export PYTHONPATH="${OPENVIKING_TAU2_ROOT}:${VIKINGBOT_ROOT}:${PYTHONPATH:-}"
+
+# tau2 dataset root (derived from the tau2-bench checkout)
+export TAU2_DATA_ROOT="${TAU2_DATA_ROOT:-${TAU2_BENCH_ROOT}/data/tau2}"
+
+# OpenViking server config
+export OPENVIKING_CONFIG_FILE="${OPENVIKING_CONFIG_FILE:-${HOME}/.openviking/ov.conf}"
+
+# LLM for the tau2 user simulator (e.g. Doubao via volcengine ARK, OpenAI-compatible).
+# Provide your own key via ARK_API_KEY (do NOT commit real keys).
+export OPENAI_API_KEY="${OPENAI_API_KEY:-${ARK_API_KEY:-}}"
+export OPENAI_API_BASE="${OPENAI_API_BASE:-https://ark.cn-beijing.volces.com/api/v3}"
+if [[ -z "${OPENAI_API_KEY}" ]]; then
+ echo "[setup_env] WARNING: OPENAI_API_KEY/ARK_API_KEY is empty; the tau2 user simulator will fail."
+fi
+
+echo "[setup_env] PYTHONPATH includes openviking (${OPENVIKING_TAU2_ROOT}) and vikingbot (${VIKINGBOT_ROOT})"
+echo "[setup_env] TAU2_DATA_ROOT=${TAU2_DATA_ROOT}"
+echo "[setup_env] OPENAI_API_BASE=${OPENAI_API_BASE}"
+
+unset -f _abort 2>/dev/null || true
diff --git a/benchmark/tau2/vikingbot/tau2_env/tau2_environment.py b/benchmark/tau2/vikingbot/tau2_env/tau2_environment.py
new file mode 100644
index 000000000..27a89143a
--- /dev/null
+++ b/benchmark/tau2/vikingbot/tau2_env/tau2_environment.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+
+from __future__ import annotations
+
+import json
+
+from tau2.gym.gym_agent import AgentGymEnv
+
+
+class CommunicateWithUser:
+ """The agent's only channel for talking to the user.
+
+ tau2's environment has no native "speak to the user" action, so we add this
+ tool: whatever ``content`` the agent passes is delivered to the tau2 user
+ simulator, and the simulator's reply comes back as the tool result. This class
+ owns both the tool's schema (``openai_schema``, consumed by
+ ``Tau2BenchToolProvider``) and its execution (``forward``, invoked by
+ ``Tau2BenchEnv.tool_call``).
+ """
+
+ name = "communicate_with_user"
+ description = (
+ "say something to the user. Note that the customer cannot see the answer "
+ "returned in `final_answer`. You must communicate with the customer "
+ "exclusively through this tool."
+ )
+ parameters = {
+ "type": "object",
+ "properties": {
+ "content": {
+ "type": "string",
+ "description": "the content to say to the user",
+ }
+ },
+ "required": ["content"],
+ }
+
+ def __init__(self, env):
+ # ``env`` is the underlying tau2 AgentGymEnv.
+ self.env = env
+
+ def forward(self, content: str):
+ """Deliver ``content`` to the tau2 user simulator.
+
+ Returns the raw gym step tuple ``(obs, reward, terminated, truncated, info)``;
+ ``Tau2BenchEnv.tool_call`` cleans the observation and tracks termination.
+ """
+ response = self.env.tool_call(self.name, {"content": content})
+ return response
+
+ @classmethod
+ def openai_schema(cls) -> dict:
+ return {
+ "type": "function",
+ "function": {
+ "name": cls.name,
+ "description": cls.description,
+ "parameters": cls.parameters,
+ },
+ }
+
+
+class Tau2BenchEnv:
+ def __init__(self, domain: str, task_id: str):
+ self.env = AgentGymEnv(domain=domain, task_id=task_id, user_llm="openai/doubao-seed-2-0-pro-260215")
+ self.terminated = False
+
+ def reset(self):
+ user_query, info_dict = self.env.reset()
+ self.user_query = user_query.lstrip("user: ")
+ self.task = info_dict["task"]
+ self.simulation_run = info_dict["simulation_run"]
+ self.policy = info_dict["policy"]
+ # All OpenAI tool schemas exposed to the agent: tau2's native tools plus
+ # communicate_with_user (the agent's only channel to the user).
+ self.tool_schemas = [tool.openai_schema for tool in info_dict["tools"]]
+ self.tool_schemas.append(CommunicateWithUser.openai_schema())
+ self.ground_truth = str(self.task.evaluation_criteria)
+ self.user_scenario = self.task.user_scenario
+
+ def tool_call(self, tool_name: str, arguments: dict) -> str:
+ if self.terminated:
+ return "Task Terminated"
+
+ if tool_name == CommunicateWithUser.name:
+ obs, reward, terminated, truncated, info = self.env.step(arguments["content"])
+ else:
+ action = {"name": tool_name, "arguments": arguments}
+ obs, reward, terminated, truncated, info = self.env.step(json.dumps(action))
+
+ if "tool: " in obs:
+ obs = obs.lstrip("tool: ")
+ if "user: " in obs:
+ obs = obs.lstrip("user: ")
+ self.terminated = terminated
+ return obs
diff --git a/benchmark/tau2/vikingbot/tau2_env/tau2_tool_provider.py b/benchmark/tau2/vikingbot/tau2_env/tau2_tool_provider.py
new file mode 100644
index 000000000..fec8e5a26
--- /dev/null
+++ b/benchmark/tau2/vikingbot/tau2_env/tau2_tool_provider.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Tau2-bench tool provider that exposes environment tools in OpenAI schema."""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any, Iterable
+
+from .tau2_environment import Tau2BenchEnv
+
+
+class Tau2BenchToolProvider:
+ """Wrap Tau2BenchEnv tools into OpenAI tool schemas for MCP."""
+
+ def __init__(self, domain: str, task_id: str, data_root: str | None = None):
+ self.domain = domain
+ self.task_id = task_id
+ self.data_root = data_root
+ self.env: Tau2BenchEnv | None = None
+ self._openai_tools: list[dict[str, Any]] = []
+
+ def reset(self) -> None:
+ env = Tau2BenchEnv(self.domain, self.task_id)
+ env.reset()
+ self.env = env
+ # Tau2BenchEnv.tool_schemas already includes tau2's native tools plus
+ # communicate_with_user.
+ self._openai_tools = list(env.tool_schemas)
+
+ @property
+ def user_query(self) -> str:
+ if not self.env:
+ return ""
+ return self.env.user_query
+
+ @property
+ def policy(self) -> str:
+ if not self.env:
+ return ""
+ return self.env.policy
+
+ @property
+ def ground_truth(self) -> str:
+ if not self.env:
+ return ""
+ return self.env.ground_truth
+
+ def list_openai_tools(self) -> Iterable[dict[str, Any]]:
+ return self._openai_tools
+
+ def call_tool(self, name: str, arguments: dict[str, Any]) -> str:
+ if not self.env:
+ raise RuntimeError("Environment not initialized. Call reset() first.")
+ normalized = name.replace("-", "_")
+ return self.env.tool_call(normalized, arguments)
+
+
+def load_task_id(data_split: str, task_no: int) -> tuple[str, str]:
+ """Resolve task_id from split file and return (domain, task_id)."""
+ domain, split = data_split.split("_", 1)
+ data_root = os.getenv("TAU2_DATA_ROOT")
+ if not data_root:
+ raise RuntimeError(
+ "TAU2_DATA_ROOT is not set. Point it at your tau2-bench data dir, e.g. "
+ "export TAU2_DATA_ROOT=<tau2-bench>/data/tau2 (see setup_env.sh)."
+ )
+ split_path = os.path.join(data_root, "domains", domain, "split_tasks.json")
+ with open(split_path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ task_ids = data[split]
+ task_id = task_ids[task_no]
+ return domain, task_id