Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
docs(tau2): clarify fixed-user PR-B reproduction (#2223)
huangruiteng9fffdf4a5254
diff --git a/benchmark/tau2/README.md b/benchmark/tau2/README.mdindex 59ec971a8..dd96b4430 100644--- a/benchmark/tau2/README.md+++ b/benchmark/tau2/README.md@@ -63,6 +63,18 @@ benchmark/tau2/scripts/setup_tau2_repo.shsource benchmark/tau2/.env.tau2```+For PR-B-compatible reproduction, pin the TAU-2 checkout to a ref that includes+the confirmation-aware text-user-simulator prompt. The original PR-B evidence+used the open TAU-2 fix PR head (`79dbf0c18ac7637aedf869cb3122babcd57aaf17`):++```bash+benchmark/tau2/scripts/setup_tau2_repo.sh \+ --ref refs/pull/297/head+source benchmark/tau2/.env.tau2+```++Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).+Plan the default benchmark without running TAU-2:```bash@@ -119,8 +131,64 @@ benchmark/tau2/run_full_eval.sh \The PR-B headline and content-shape ablation use`config/prb_content_matrix_new_prompt.yaml`. It runs the no-memory control plus-trajectory top4, experience top2, and representative 4000-character budget-ablation routes across `retail + airline` with 8 repeats.+trajectory first-user top4 / pre-write top2, experience top2, and representative+4000-character budget ablation routes across `retail + airline` with 8 repeats.++### 1. Bootstrap fixed-first-user fixtures++The default PR-B configs require fixed-first-user fixtures, so a fresh checkout+needs one live-user bootstrap pass before strict reproduction. This pass uses+the same confirmation-aware simulator policy but does not require fixed fixtures.++Run one bootstrap pass per domain:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \+ --domain retail \+ --run-id fixed_first_user_bootstrap_retail \+ --strict-preflight \+ --execute++benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/fixed_first_user_bootstrap.yaml \+ --domain airline \+ --run-id fixed_first_user_bootstrap_airline \+ --strict-preflight \+ --execute+```++Then convert each bootstrap `results.json` into a fixture:++```bash+RETAIL_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_retail/memory_cells/fixed_first_user_bootstrap_retail_retail_no_memory_r1/fixed_first_user_bootstrap_retail_retail_no_memory_r1.json+AIRLINE_RESULTS=benchmark/tau2/result/fixed_first_user_bootstrap_airline/memory_cells/fixed_first_user_bootstrap_airline_airline_no_memory_r1/fixed_first_user_bootstrap_airline_airline_no_memory_r1.json++python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \+ --repo "$TAU2_REPO" \+ --results-json "$RETAIL_RESULTS" \+ --domain retail \+ --task-split-name test \+ --output benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json \+ --require-full-split++python benchmark/tau2/scripts/build_fixed_first_user_fixture.py \+ --repo "$TAU2_REPO" \+ --results-json "$AIRLINE_RESULTS" \+ --domain airline \+ --task-split-name test \+ --output benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json \+ --require-full-split+```++Export the generated fixture paths for subsequent strict runs:++```bash+export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/retail/fixed_first_user_fixture.json"+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"+```++### 2. Run smoke and full PR-B matrixFirst run one tiny end-to-end smoke against a clean local OpenViking service:@@ -148,8 +216,9 @@ benchmark/tau2/run_full_eval.sh \The main result is written to`benchmark/tau2/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.-Per-cell outputs live under `cell_results/`; corpus identity and generated-memory checks live under `memory_corpora/`.+Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON+lives under `memory_cells/`, and corpus identity / generated memory checks live+under `memory_corpora/`.For a small E2E smoke, keep both the eval and train slices tiny:diff --git a/benchmark/tau2/config/fixed_first_user_bootstrap.yaml b/benchmark/tau2/config/fixed_first_user_bootstrap.yamlnew file mode 100644index 000000000..3bf670b14--- /dev/null+++ b/benchmark/tau2/config/fixed_first_user_bootstrap.yaml@@ -0,0 +1,14 @@+extends: no_memory.yaml++benchmark:+ name: tau2_fixed_first_user_bootstrap++eval:+ # Use live simulator turns once to collect the first user request fixture.+ # The resulting fixture can then be reused by the strict PR-B configs.+ protocol: live_user_bootstrap+ require_fixed_first_user: false+ user_simulator_policy: confirmation_aware+ fixed_first_user_fixtures:+ retail:+ airline:diff --git a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/config/prb_content_matrix_new_prompt.yamlindex 8a0a18a39..55607c479 100644--- a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml+++ b/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml@@ -27,7 +27,7 @@ strategies:scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md- id: new_traj_fixed_prewrite_only- label: PR-B new trajectory fixed-count prewrite top4+ label: PR-B new trajectory fixed-count prewrite top2memory_backend: openvikingtrain_required: truecorpus_id: memory_v2_operation_family_v1_success_only@@ -38,12 +38,12 @@ strategies:search_memory_type: trajectoriesretrieval_mode: prewriteretrieval_top_k: 4- prewrite_retrieval_top_k: 4- prewrite_inject_top_k: 4+ prewrite_retrieval_top_k: 2+ prewrite_inject_top_k: 2scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md- id: new_traj_fixed_first_user_prewrite- label: PR-B new trajectory fixed-count first-user + prewrite top4+ label: PR-B new trajectory fixed-count first-user top4 + prewrite top2memory_backend: openvikingtrain_required: truecorpus_id: memory_v2_operation_family_v1_success_only@@ -55,8 +55,8 @@ strategies:retrieval_mode: first_user_prewriteretrieval_top_k: 4first_user_inject_top_k: 4- prewrite_retrieval_top_k: 4- prewrite_inject_top_k: 4+ prewrite_retrieval_top_k: 2+ prewrite_inject_top_k: 2scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md- id: new_exp_fixed_first_userdiff --git a/benchmark/tau2/config/prb_scope_fairness.yaml b/benchmark/tau2/config/prb_scope_fairness.yamlindex cc2f820c1..8d044d216 100644--- a/benchmark/tau2/config/prb_scope_fairness.yaml+++ b/benchmark/tau2/config/prb_scope_fairness.yaml@@ -20,7 +20,7 @@ strategies:scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.md- id: trajectory_top4_first_user_prewrite_generic_scope- label: Trajectory top4 first-user + pre-write with generic memory scope prompt+ label: Trajectory top4 first-user + pre-write top2 with generic memory scope promptmemory_backend: openvikingtrain_required: truecorpus_id: memory_v2_trajectory_view@@ -29,6 +29,6 @@ strategies:retrieval_mode: first_user_prewriteretrieval_top_k: 4first_user_inject_top_k: 4- prewrite_retrieval_top_k: 4- prewrite_inject_top_k: 4+ prewrite_retrieval_top_k: 2+ prewrite_inject_top_k: 2scope_prompt_file: benchmark/tau2/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/scripts/build_fixed_first_user_fixture.py b/benchmark/tau2/scripts/build_fixed_first_user_fixture.pynew file mode 100644index 000000000..8b4c0cc49--- /dev/null+++ b/benchmark/tau2/scripts/build_fixed_first_user_fixture.py@@ -0,0 +1,143 @@+#!/usr/bin/env python3+"""Build a TAU-2 fixed-first-user fixture from a TAU-2 results.json file."""++from __future__ import annotations++import argparse+import hashlib+import json+import sys+from pathlib import Path+from typing import Any+++def _add_tau2_to_path(repo: Path) -> None:+ repo = repo.expanduser().resolve()+ for candidate in (repo, repo / "src"):+ if str(candidate) not in sys.path:+ sys.path.insert(0, str(candidate))+++def _load_json(path: Path) -> Any:+ return json.loads(path.expanduser().read_text(encoding="utf-8"))+++def _scenario_sha256(instructions: str) -> str:+ return hashlib.sha256(instructions.encode("utf-8")).hexdigest()+++def _first_user(simulation: dict[str, Any]) -> str:+ for message in simulation.get("messages") or []:+ if message.get("role") == "user":+ return str(message.get("content") or "")+ return ""+++def _get_tasks(repo: Path, domain: str, task_split_name: str) -> list[Any]:+ _add_tau2_to_path(repo)+ try:+ from tau2.runner.helpers import get_tasks++ return list(get_tasks(domain, task_split_name=task_split_name))+ except ModuleNotFoundError:+ from tau2.run import get_tasks as legacy_get_tasks++ return list(legacy_get_tasks(task_set_name=domain, task_split_name=task_split_name))+++def build_fixture(+ *,+ repo: Path,+ results_json: Path,+ domain: str,+ task_split_name: str,+) -> dict[str, Any]:+ results = _load_json(results_json)+ first_user_by_task_id = {+ str(simulation.get("task_id")): _first_user(simulation)+ for simulation in results.get("simulations") or []+ }+ tasks = _get_tasks(repo, domain, task_split_name)++ records: list[dict[str, str]] = []+ by_scenario_sha256: dict[str, str] = {}+ missing_task_ids: list[str] = []+ for task in tasks:+ task_id = str(task.id)+ first_user = first_user_by_task_id.get(task_id, "")+ if not first_user:+ missing_task_ids.append(task_id)+ continue+ key = _scenario_sha256(str(task.user_scenario))+ by_scenario_sha256[key] = first_user+ records.append(+ {+ "task_id": task_id,+ "scenario_sha256": key,+ "first_user": first_user,+ "first_user_preview": first_user[:220],+ }+ )++ return {+ "fixture_type": "tau2_fixed_first_user.v0",+ "domain": domain,+ "task_split_name": task_split_name,+ "source_results_json": str(results_json.expanduser().resolve()),+ "expected_task_count": len(tasks),+ "record_count": len(records),+ "missing_task_ids": missing_task_ids,+ "by_scenario_sha256": by_scenario_sha256,+ "records": records,+ }+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--repo", type=Path, default=Path("benchmark/tau2/.external/tau2-bench"))+ parser.add_argument("--results-json", type=Path, required=True)+ parser.add_argument("--domain", required=True)+ parser.add_argument("--task-split-name", default="test")+ parser.add_argument("--output", type=Path, required=True)+ parser.add_argument(+ "--require-full-split",+ action="store_true",+ help="Fail if the results file does not cover every task in the requested split.",+ )+ args = parser.parse_args()++ fixture = build_fixture(+ repo=args.repo,+ results_json=args.results_json,+ domain=args.domain,+ task_split_name=args.task_split_name,+ )+ if args.require_full_split and fixture["record_count"] != fixture["expected_task_count"]:+ raise SystemExit(+ "fixed-first-user fixture coverage incomplete: "+ f"{fixture['record_count']}/{fixture['expected_task_count']} records, "+ f"missing task ids: {fixture['missing_task_ids'][:20]}"+ )++ output = args.output.expanduser()+ output.parent.mkdir(parents=True, exist_ok=True)+ output.write_text(+ json.dumps(fixture, ensure_ascii=False, indent=2, sort_keys=True) + "\n",+ encoding="utf-8",+ )+ print(+ json.dumps(+ {+ "expected_task_count": fixture["expected_task_count"],+ "record_count": fixture["record_count"],+ "missing_task_count": len(fixture["missing_task_ids"]),+ "path": str(output.resolve()),+ },+ ensure_ascii=False,+ indent=2,+ )+ )+++if __name__ == "__main__":+ main()diff --git a/benchmark/tau2/scripts/setup_tau2_repo.sh b/benchmark/tau2/scripts/setup_tau2_repo.shindex 3cee2655a..815f9bc0f 100755--- a/benchmark/tau2/scripts/setup_tau2_repo.sh+++ b/benchmark/tau2/scripts/setup_tau2_repo.sh@@ -60,7 +60,12 @@ elsefiif [[ -n "$REF" ]]; then- git -C "$REPO_DIR" checkout "$REF"+ if git -C "$REPO_DIR" rev-parse --verify --quiet "$REF^{commit}" >/dev/null; then+ git -C "$REPO_DIR" checkout "$REF"+ else+ git -C "$REPO_DIR" fetch "$REPO_URL" "$REF"+ git -C "$REPO_DIR" checkout FETCH_HEAD+ fifiTAU2_CLI="tau2"