Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

docs(tau2): clarify fixed-user PR-B reproduction (#2223)

huangruiteng9fffdf4a5254
diff --git a/benchmark/tau2/README.md b/benchmark/tau2/README.md
index 59ec971a8..dd96b4430 100644
--- a/benchmark/tau2/README.md
+++ b/benchmark/tau2/README.md
@@ -63,6 +63,18 @@ benchmark/tau2/scripts/setup_tau2_repo.sh
source benchmark/tau2/.env.tau2
```
+For PR-B-compatible reproduction, pin the TAU-2 checkout to a ref that includes
+the confirmation-aware text-user-simulator prompt. The original PR-B evidence
+used the open TAU-2 fix PR head (`79dbf0c18ac7637aedf869cb3122babcd57aaf17`):
+
+```bash
+benchmark/tau2/scripts/setup_tau2_repo.sh \
+ --ref refs/pull/297/head
+source benchmark/tau2/.env.tau2
+```
+
+Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).
+
Plan the default benchmark without running TAU-2:
```bash
@@ -119,8 +131,64 @@ benchmark/tau2/run_full_eval.sh \
The PR-B headline and content-shape ablation use
`config/prb_content_matrix_new_prompt.yaml`. It runs the no-memory control plus
-trajectory top4, experience top2, and representative 4000-character budget
-ablation routes across `retail + airline` with 8 repeats.
+trajectory first-user top4 / pre-write top2, experience top2, and representative
+4000-character budget ablation routes across `retail + airline` with 8 repeats.
+
+### 1. Bootstrap fixed-first-user fixtures
+
+The default PR-B configs require fixed-first-user fixtures, so a fresh checkout
+needs one live-user bootstrap pass before strict reproduction. This pass uses
+the same confirmation-aware simulator policy but does not require fixed fixtures.
+
+Run one bootstrap pass per domain:
+
+```bash
+benchmark/tau2/run_full_eval.sh \
+ --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \
+ --domain retail \
+ --run-id fixed_first_user_bootstrap_retail \
+ --strict-preflight \
+ --execute
+
+benchmark/tau2/run_full_eval.sh \
+ --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \
+ --domain airline \
+ --run-id fixed_first_user_bootstrap_airline \
+ --strict-preflight \
+ --execute
+```
+
+Then convert each bootstrap `results.json` into a fixture:
+
+```bash
+RETAIL_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_retail/memory_cells/fixed_first_user_bootstrap_retail_retail_no_memory_r1/fixed_first_user_bootstrap_retail_retail_no_memory_r1.json
+AIRLINE_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_airline/memory_cells/fixed_first_user_bootstrap_airline_airline_no_memory_r1/fixed_first_user_bootstrap_airline_airline_no_memory_r1.json
+
+python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \
+ --repo "$TAU2_REPO" \
+ --results-json "$RETAIL_RESULTS" \
+ --domain retail \
+ --task-split-name test \
+ --output benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json \
+ --require-full-split
+
+python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \
+ --repo "$TAU2_REPO" \
+ --results-json "$AIRLINE_RESULTS" \
+ --domain airline \
+ --task-split-name test \
+ --output benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json \
+ --require-full-split
+```
+
+Export the generated fixture paths for subsequent strict runs:
+
+```bash
+export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json"
+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"
+```
+
+### 2. Run smoke and full PR-B matrix
First run one tiny end-to-end smoke against a clean local OpenViking service:
@@ -148,8 +216,9 @@ benchmark/tau2/run_full_eval.sh \
The main result is written to
`benchmark/tau2/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.
-Per-cell outputs live under `cell_results/`; corpus identity and generated
-memory checks live under `memory_corpora/`.
+Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON
+lives under `memory_cells/`, and corpus identity / generated memory checks live
+under `memory_corpora/`.
For a small E2E smoke, keep both the eval and train slices tiny:
diff --git a/benchmark/tau2/config/fixed_first_user_bootstrap.yaml b/benchmark/tau2/config/fixed_first_user_bootstrap.yaml
new file mode 100644
index 000000000..3bf670b14
--- /dev/null
+++ b/benchmark/tau2/config/fixed_first_user_bootstrap.yaml
@@ -0,0 +1,14 @@
+extends: no_memory.yaml
+
+benchmark:
+ name: tau2_fixed_first_user_bootstrap
+
+eval:
+ # Use live simulator turns once to collect the first user request fixture.
+ # The resulting fixture can then be reused by the strict PR-B configs.
+ protocol: live_user_bootstrap
+ require_fixed_first_user: false
+ user_simulator_policy: confirmation_aware
+ fixed_first_user_fixtures:
+ retail:
+ airline:
diff --git a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
index 8a0a18a39..55607c479 100644
--- a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
+++ b/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
@@ -27,7 +27,7 @@ strategies:
scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
- id: new_traj_fixed_prewrite_only
- label: PR-B new trajectory fixed-count prewrite top4
+ label: PR-B new trajectory fixed-count prewrite top2
memory_backend: openviking
train_required: true
corpus_id: memory_v2_operation_family_v1_success_only
@@ -38,12 +38,12 @@ strategies:
search_memory_type: trajectories
retrieval_mode: prewrite
retrieval_top_k: 4
- prewrite_retrieval_top_k: 4
- prewrite_inject_top_k: 4
+ prewrite_retrieval_top_k: 2
+ prewrite_inject_top_k: 2
scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
- id: new_traj_fixed_first_user_prewrite
- label: PR-B new trajectory fixed-count first-user + prewrite top4
+ label: PR-B new trajectory fixed-count first-user top4 + prewrite top2
memory_backend: openviking
train_required: true
corpus_id: memory_v2_operation_family_v1_success_only
@@ -55,8 +55,8 @@ strategies:
retrieval_mode: first_user_prewrite
retrieval_top_k: 4
first_user_inject_top_k: 4
- prewrite_retrieval_top_k: 4
- prewrite_inject_top_k: 4
+ prewrite_retrieval_top_k: 2
+ prewrite_inject_top_k: 2
scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
- id: new_exp_fixed_first_user
diff --git a/benchmark/tau2/config/prb_scope_fairness.yaml b/benchmark/tau2/config/prb_scope_fairness.yaml
index cc2f820c1..8d044d216 100644
--- a/benchmark/tau2/config/prb_scope_fairness.yaml
+++ b/benchmark/tau2/config/prb_scope_fairness.yaml
@@ -20,7 +20,7 @@ strategies:
scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
- id: trajectory_top4_first_user_prewrite_generic_scope
- label: Trajectory top4 first-user + pre-write with generic memory scope prompt
+ label: Trajectory top4 first-user + pre-write top2 with generic memory scope prompt
memory_backend: openviking
train_required: true
corpus_id: memory_v2_trajectory_view
@@ -29,6 +29,6 @@ strategies:
retrieval_mode: first_user_prewrite
retrieval_top_k: 4
first_user_inject_top_k: 4
- prewrite_retrieval_top_k: 4
- prewrite_inject_top_k: 4
+ prewrite_retrieval_top_k: 2
+ prewrite_inject_top_k: 2
scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/scripts/build_fixed_first_user_fixture.py b/benchmark/tau2/scripts/build_fixed_first_user_fixture.py
new file mode 100644
index 000000000..8b4c0cc49
--- /dev/null
+++ b/benchmark/tau2/scripts/build_fixed_first_user_fixture.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""Build a TAU-2 fixed-first-user fixture from a TAU-2 results.json file."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+
+def _add_tau2_to_path(repo: Path) -> None:
+ repo = repo.expanduser().resolve()
+ for candidate in (repo, repo / "src"):
+ if str(candidate) not in sys.path:
+ sys.path.insert(0, str(candidate))
+
+
+def _load_json(path: Path) -> Any:
+ return json.loads(path.expanduser().read_text(encoding="utf-8"))
+
+
+def _scenario_sha256(instructions: str) -> str:
+ return hashlib.sha256(instructions.encode("utf-8")).hexdigest()
+
+
+def _first_user(simulation: dict[str, Any]) -> str:
+ for message in simulation.get("messages") or []:
+ if message.get("role") == "user":
+ return str(message.get("content") or "")
+ return ""
+
+
+def _get_tasks(repo: Path, domain: str, task_split_name: str) -> list[Any]:
+ _add_tau2_to_path(repo)
+ try:
+ from tau2.runner.helpers import get_tasks
+
+ return list(get_tasks(domain, task_split_name=task_split_name))
+ except ModuleNotFoundError:
+ from tau2.run import get_tasks as legacy_get_tasks
+
+ return list(legacy_get_tasks(task_set_name=domain, task_split_name=task_split_name))
+
+
+def build_fixture(
+ *,
+ repo: Path,
+ results_json: Path,
+ domain: str,
+ task_split_name: str,
+) -> dict[str, Any]:
+ results = _load_json(results_json)
+ first_user_by_task_id = {
+ str(simulation.get("task_id")): _first_user(simulation)
+ for simulation in results.get("simulations") or []
+ }
+ tasks = _get_tasks(repo, domain, task_split_name)
+
+ records: list[dict[str, str]] = []
+ by_scenario_sha256: dict[str, str] = {}
+ missing_task_ids: list[str] = []
+ for task in tasks:
+ task_id = str(task.id)
+ first_user = first_user_by_task_id.get(task_id, "")
+ if not first_user:
+ missing_task_ids.append(task_id)
+ continue
+ key = _scenario_sha256(str(task.user_scenario))
+ by_scenario_sha256[key] = first_user
+ records.append(
+ {
+ "task_id": task_id,
+ "scenario_sha256": key,
+ "first_user": first_user,
+ "first_user_preview": first_user[:220],
+ }
+ )
+
+ return {
+ "fixture_type": "tau2_fixed_first_user.v0",
+ "domain": domain,
+ "task_split_name": task_split_name,
+ "source_results_json": str(results_json.expanduser().resolve()),
+ "expected_task_count": len(tasks),
+ "record_count": len(records),
+ "missing_task_ids": missing_task_ids,
+ "by_scenario_sha256": by_scenario_sha256,
+ "records": records,
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--repo", type=Path, default=Path("benchmark/tau2/.external/tau2-bench"))
+ parser.add_argument("--results-json", type=Path, required=True)
+ parser.add_argument("--domain", required=True)
+ parser.add_argument("--task-split-name", default="test")
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument(
+ "--require-full-split",
+ action="store_true",
+ help="Fail if the results file does not cover every task in the requested split.",
+ )
+ args = parser.parse_args()
+
+ fixture = build_fixture(
+ repo=args.repo,
+ results_json=args.results_json,
+ domain=args.domain,
+ task_split_name=args.task_split_name,
+ )
+ if args.require_full_split and fixture["record_count"] != fixture["expected_task_count"]:
+ raise SystemExit(
+ "fixed-first-user fixture coverage incomplete: "
+ f"{fixture['record_count']}/{fixture['expected_task_count']} records, "
+ f"missing task ids: {fixture['missing_task_ids'][:20]}"
+ )
+
+ output = args.output.expanduser()
+ output.parent.mkdir(parents=True, exist_ok=True)
+ output.write_text(
+ json.dumps(fixture, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+ encoding="utf-8",
+ )
+ print(
+ json.dumps(
+ {
+ "expected_task_count": fixture["expected_task_count"],
+ "record_count": fixture["record_count"],
+ "missing_task_count": len(fixture["missing_task_ids"]),
+ "path": str(output.resolve()),
+ },
+ ensure_ascii=False,
+ indent=2,
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/tau2/scripts/setup_tau2_repo.sh b/benchmark/tau2/scripts/setup_tau2_repo.sh
index 3cee2655a..815f9bc0f 100755
--- a/benchmark/tau2/scripts/setup_tau2_repo.sh
+++ b/benchmark/tau2/scripts/setup_tau2_repo.sh
@@ -60,7 +60,12 @@ else
fi
if [[ -n "$REF" ]]; then
- git -C "$REPO_DIR" checkout "$REF"
+ if git -C "$REPO_DIR" rev-parse --verify --quiet "$REF^{commit}" >/dev/null; then
+ git -C "$REPO_DIR" checkout "$REF"
+ else
+ git -C "$REPO_DIR" fetch "$REPO_URL" "$REF"
+ git -C "$REPO_DIR" checkout FETCH_HEAD
+ fi
fi
TAU2_CLI="tau2"