Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

docs: simplify tau2 benchmark reproduction (#2267)

huangruiteng646ff735dbdb
diff --git a/benchmark/tau2/llm/README.md b/benchmark/tau2/llm/README.md
index 527d6dedf..4933a4d78 100644
--- a/benchmark/tau2/llm/README.md
+++ b/benchmark/tau2/llm/README.md
@@ -1,14 +1,23 @@
# TAU-2 Benchmark
-This directory contains a small OpenViking-style entry point for TAU-2 memory
-evaluation. The scope is intentionally narrow:
+This directory contains the OpenViking TAU-2 LLM benchmark entry point. The
+reproduction surface is intentionally narrow:
-- fresh OpenViking Memory V2 experience-only baseline;
-- Memory V2 pre-write recall treatment.
-- trajectory memory retrieval treatment for the refined extraction prompt and
- retrieval-anchor embedding text.
+- `no_memory`: same-seed TAU-2 baseline without OpenViking memory injection;
+- `template_indexed_trajectory_top4_prewrite_top2`: the current best
+ template-indexed trajectory memory treatment.
-Category rerank and other harness-only diagnostics are intentionally left out.
+The template-indexed trajectory treatment trains OpenViking Memory V2 from
+TAU-2 train conversations, retrieves generated `trajectories`, and uses the
+trajectory embedding template `{{ trajectory_name }}\n\n{{ retrieval_anchor }}`
+instead of broad procedure bodies for retrieval. It injects trajectory top4 at
+the first user turn and top2 before write-like tool calls, with the generic
+memory scope prompt enabled.
+
+Category rerank, experience-memory routes, fixed-count-only ablations,
+character-budget ablations, and official-user parity controls are intentionally
+left out of this README and config set so reproduction agents do not mistake
+diagnostic routes for current evidence.
## Layout
@@ -16,21 +25,28 @@ Category rerank and other harness-only diagnostics are intentionally left out.
benchmark/tau2/llm/
├── config/
│ ├── baseline.yaml
-│ ├── official.yaml
-│ ├── prewrite.yaml
-│ └── trajectory.yaml
+│ ├── fixed_first_user_bootstrap.yaml
+│ ├── no_memory.yaml
+│ ├── scope_prompts/
+│ │ └── generic_memory_scope.md
+│ └── template_indexed_trajectory.yaml
├── scripts/
+│ ├── build_fixed_first_user_fixture.py
│ ├── run_eval.py
│ ├── setup_tau2_repo.sh
│ └── tau2_common.py
└── run_full_eval.sh
```
+`baseline.yaml` is a shared protocol/defaults file, not a runnable evidence
+cell by itself. Use `no_memory.yaml` for the baseline-only run and
+`template_indexed_trajectory.yaml` for the paired no-memory + trajectory run.
+
Generated eval artifacts are written to `benchmark/tau2/llm/result/<run_id>/`.
Memory corpus artifacts are cached outside the run id at
`benchmark/tau2/llm/result/memory_corpora/` by default.
-## Quick Start
+## Setup
This benchmark delegates task simulation and scoring to an external TAU-2
checkout. Point the runner at that checkout and CLI explicitly when they are not
@@ -41,21 +57,6 @@ export TAU2_REPO=/path/to/tau2-bench
export TAU2_CLI=/path/to/tau2
```
-The default OpenViking TAU-2 memory evidence protocol is
-`fixed_first_user_full8`: retail + airline, 8 repeats, same seeds, confirmation
-aware user simulator, and fixed first user fixtures for both domains. Later user
-simulator turns remain live. Set the fixture paths before running the default
-configs:
-
-```bash
-export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail/fixed_first_user_fixture.json
-export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline/fixed_first_user_fixture.json
-```
-
-`--strict-preflight` fails when `eval.require_fixed_first_user=true` and either
-fixture is missing. Use `config/official.yaml` for an explicit non-fixed,
-official-live-user control.
-
For a local one-command setup, clone and install TAU-2 into ignored benchmark
directories:
@@ -64,9 +65,15 @@ benchmark/tau2/llm/scripts/setup_tau2_repo.sh
source benchmark/tau2/llm/.env.tau2
```
-For PR-B-compatible reproduction, pin the TAU-2 checkout to a ref that includes
-the confirmation-aware text-user-simulator prompt. The original PR-B evidence
-used the open TAU-2 fix PR head (`79dbf0c18ac7637aedf869cb3122babcd57aaf17`):
+The default OpenViking TAU-2 memory evidence protocol is
+`fixed_first_user_full8`: retail + airline, 8 repeats, same seeds,
+confirmation-aware user simulator, and fixed first-user fixtures for both
+domains. Later user simulator turns remain live.
+
+The confirmation-aware simulator behavior is available from
+[sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).
+Pin the local TAU-2 checkout to a ref that includes that behavior when
+reproducing these numbers:
```bash
benchmark/tau2/llm/scripts/setup_tau2_repo.sh \
@@ -74,78 +81,22 @@ benchmark/tau2/llm/scripts/setup_tau2_repo.sh \
source benchmark/tau2/llm/.env.tau2
```
-Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).
-
-Plan the default benchmark without running TAU-2:
-
-```bash
-python benchmark/tau2/llm/scripts/run_eval.py --config benchmark/tau2/llm/config/baseline.yaml --plan-only
-```
-
-Add `--preflight` or `--strict-preflight` when you want the runner to write a
-small environment/config check next to the run plan.
-
-After setup, verify the local TAU-2 link and write a one-cell run plan:
-
-```bash
-benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/baseline.yaml \
- --strict-preflight \
- --domain retail \
- --strategy-id memory_v2_experience_only \
- --task-id 5 \
- --repeat-count 1
-```
-
-Plan a one-cell Memory V2 pre-write smoke:
+When using Doubao through an OpenAI-compatible endpoint, set `OPENAI_API_KEY`
+and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.
-```bash
-benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/baseline.yaml \
- --domain retail \
- --strategy-id memory_v2_prewrite \
- --num-tasks 1 \
- --repeat-count 1
-```
+## Fixed-First-User Fixtures
-Plan a one-cell trajectory memory smoke:
+Strict reproduction requires fixed first-user fixtures:
```bash
-benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/trajectory.yaml \
- --domain retail \
- --strategy-id memory_v2_trajectory_view \
- --num-tasks 1 \
- --train-num-tasks 1 \
- --repeat-count 1
-```
-
-Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):
-
-```bash
-benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/baseline.yaml \
- --execute
+export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail/fixed_first_user_fixture.json
+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline/fixed_first_user_fixture.json
```
-## Reproduce the PR-B Evidence
-
-The PR-B headline and content-shape ablation use
-`config/prb_content_matrix_new_prompt.yaml`. It runs the no-memory control plus
-trajectory first-user top4 / pre-write top2, experience top2, and representative
-4000-character budget ablation routes across `retail + airline` with 8 repeats.
-The current trajectory prompt indexes `trajectory_name + retrieval_anchor`
-instead of the full procedure body. This keeps retrieval focused on the positive
-operation boundary and reduces broad terminal-handoff / cancellation-like memory
-matches.
-
-### 1. Bootstrap fixed-first-user fixtures
-
-The default PR-B configs require fixed-first-user fixtures, so a fresh checkout
-needs one live-user bootstrap pass before strict reproduction. This pass uses
-the same confirmation-aware simulator policy but does not require fixed fixtures.
+`--strict-preflight` fails when `eval.require_fixed_first_user=true` and either
+fixture is missing.
-Run one bootstrap pass per domain:
+For a fresh checkout, run one live-user bootstrap pass per domain:
```bash
benchmark/tau2/llm/run_full_eval.sh \
@@ -193,134 +144,146 @@ export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_f
export TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"
```
-### 2. Run smoke and full PR-B matrix
+## Run Plans And Smoke Checks
+
+Plan the no-memory baseline without running TAU-2:
+
+```bash
+python benchmark/tau2/llm/scripts/run_eval.py \
+ --config benchmark/tau2/llm/config/no_memory.yaml \
+ --plan-only
+```
+
+Plan the paired current-evidence config without running TAU-2:
+
+```bash
+python benchmark/tau2/llm/scripts/run_eval.py \
+ --config benchmark/tau2/llm/config/template_indexed_trajectory.yaml \
+ --plan-only
+```
-First run one tiny end-to-end smoke against a clean local OpenViking service:
+Run a tiny no-memory smoke:
```bash
benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \
+ --config benchmark/tau2/llm/config/no_memory.yaml \
--domain retail \
- --strategy-id new_traj_fixed_first_user_prewrite \
+ --strategy-id no_memory \
--num-tasks 1 \
- --train-num-tasks 1 \
--repeat-count 1 \
--strict-preflight \
--execute
```
-Then run the full PR-B matrix:
+Run a tiny template-indexed trajectory smoke against a clean local OpenViking
+service:
```bash
benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \
- --run-id prb_content_matrix_new_prompt_full8 \
+ --config benchmark/tau2/llm/config/template_indexed_trajectory.yaml \
+ --domain retail \
+ --strategy-id template_indexed_trajectory_top4_prewrite_top2 \
+ --num-tasks 1 \
+ --train-num-tasks 1 \
+ --repeat-count 1 \
--strict-preflight \
--execute
```
-The main result is written to
-`benchmark/tau2/llm/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.
-Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON
-lives under `memory_cells/`, and corpus identity / generated memory checks live
-under `memory_corpora/`.
+Start the OpenViking service before executing memory cells, and verify it with
+`ov status`. For trajectory memory evidence, start the service from this branch
+and inspect generated trajectory files; changing `search_uri` alone does not
+prove the template-indexed trajectory prompt was used.
-For the strongest trajectory-only treatment, inspect the
-`new_traj_fixed_first_user_prewrite` row. It uses trajectory top4 at the first
-user turn, trajectory top4 retrieval before writes, pre-write injection top2,
-fixed-first-user fixtures, and the generic scope prompt.
+## Full Reproduction
-For a small E2E smoke, keep both the eval and train slices tiny:
+Run the no-memory full8 baseline:
```bash
benchmark/tau2/llm/run_full_eval.sh \
- --config benchmark/tau2/llm/config/baseline.yaml \
- --domain retail \
- --strategy-id memory_v2_experience_only \
- --num-tasks 1 \
- --train-num-tasks 1 \
- --repeat-count 1 \
+ --config benchmark/tau2/llm/config/no_memory.yaml \
+ --run-id no_memory_full8 \
+ --strict-preflight \
--execute
```
-When using Doubao through an OpenAI-compatible endpoint, set `OPENAI_API_KEY`
-and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.
+Run the paired no-memory + current trajectory evidence config:
-Start the OpenViking service before executing memory cells, and verify it with
-`ov status`. For evidence runs, use a clean OpenViking workspace/config and set
-`OPENVIKING_URL` explicitly so local template overrides do not pollute the
-Memory V2 baseline. For trajectory memory evidence, start the service from this
-branch and inspect generated trajectory files; changing `search_uri` alone does
-not prove the new trajectory prompt was used.
+```bash
+benchmark/tau2/llm/run_full_eval.sh \
+ --config benchmark/tau2/llm/config/template_indexed_trajectory.yaml \
+ --run-id template_indexed_trajectory_full8 \
+ --strict-preflight \
+ --execute
+```
+
+The main result is written to
+`benchmark/tau2/llm/result/template_indexed_trajectory_full8/scoreboard.json`.
+Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON
+lives under `memory_cells/`, and corpus identity / generated memory checks live
+under `memory_corpora/`.
## Memory Adapter
-Memory V2 cells run through a small TAU-2 agent adapter in this directory:
+Memory cells run through a small TAU-2 agent adapter in this directory:
- train by writing TAU-2 training conversations into OpenViking sessions;
-- evaluate by retrieving OpenViking memory at the first user turn;
+- retrieve OpenViking memory at the first user turn;
- for pre-write recall, retrieve again before write-like tool calls and
- regenerate that step with the matched memories. The default benchmark
- retrieves 6 pre-write candidates and injects 2, which keeps extra candidates
- visible in traces without expanding the prompt budget;
-- optionally run an explicit generic scope-prompt treatment that keeps retrieved
- memories advisory and asks the agent to preserve the current task scope before
- write-like tool calls. The benchmark configs use a single benchmark-neutral
- `scope_prompt_file`; the runner still accepts `scope_prompt_files` for custom
- local experiments;
-- emit artifact metadata to identify the OpenViking account, agent,
- corpus, retrieval mode, and simulator policy used by each cell.
-
-For exploratory gates, prefer a bounded run with `--cell-timeout-seconds`.
-Timed-out cells are recorded with return code `124`, `timed_out=true`, and are
-excluded from scoreboard metrics, which keeps smoke runs from silently becoming
-long-running evidence jobs.
-
-The existing `train_memory_mode: experience_only` value selects the Memory V2
-session-commit path. `search_memory_type` selects which generated memory bucket
-is retrieved during eval (`experiences` by default, `trajectories` for
-`config/trajectory.yaml`). The runner prepares each distinct
-`domain + corpus_id` once and reuses it across eval run ids when the cached
-`corpus_manifest.json` is present. Different corpora may be prepared in
-parallel with `benchmark.corpus_prepare_concurrency`; session commits inside one
-corpus remain serial to preserve OpenViking write semantics.
+ regenerate that step with the matched memories;
+- optionally apply a generic scope prompt that keeps retrieved memories
+ advisory and asks the agent to preserve the current task scope before
+ write-like tool calls;
+- emit artifact metadata identifying the OpenViking account, agent, corpus,
+ retrieval mode, search memory type, and simulator policy used by each cell.
+
+The current trajectory config uses:
+
+- `train_memory_mode: experience_only`, which selects the Memory V2
+ session-commit path that writes generated memory artifacts;
+- `train_transcript_format: role_tool_blocks`, which preserves role-prefixed
+ messages plus tool-call/tool-response blocks during training;
+- `train_include_system_prompt: true`, which includes the domain policy in the
+ training session;
+- `train_skip_failed_sessions: true`, which avoids learning from failed train
+ sessions;
+- `search_memory_type: trajectories`, which retrieves generated trajectory
+ memory during eval.
+
+The runner prepares each distinct `domain + corpus_id` once and reuses it across
+eval run ids when the cached `corpus_manifest.json` is present. Different
+corpora may be prepared in parallel with `benchmark.corpus_prepare_concurrency`;
+session commits inside one corpus remain serial to preserve OpenViking write
+semantics.
By default, trajectory extraction is transcript-only: the runner replays TAU-2
messages into an OpenViking session and does not expose held-out reward or
-assertion results to the extractor. The PR-B evidence config can also use a
-structured role/tool transcript, include the domain policy in the training
-session, skip failed train sessions when building positive procedure memory, and
-cap injected memory by total character budget for content-shape ablations.
+assertion results to the extractor.
Eval cells run in parallel with `benchmark.strategy_concurrency` by default and
can be overridden with `--strategy-concurrency`. This only parallelizes read-only
TAU-2 eval cells; corpus writes inside one corpus are still serialized by the
prepare step.
+For exploratory gates, prefer a bounded run with `--cell-timeout-seconds`.
+Timed-out cells are recorded with return code `124`, `timed_out=true`, and are
+excluded from scoreboard metrics, which keeps smoke runs from silently becoming
+long-running evidence jobs.
+
## User Simulator Policy
The runner default is the official TAU-2 user simulator if
`eval.user_simulator_policy` is omitted. The bundled OpenViking memory benchmark
-config sets `confirmation_aware`, because a memory benchmark should not treat
+configs set `confirmation_aware`, because a memory benchmark should not treat
user confirmation as task completion before the backend write has happened.
`confirmation_aware` applies a small idempotent prompt patch to the configured
TAU-2 checkout before planning or running. The patch appends only the behavioral
confirmation boundary to the TAU-2 user simulator guidelines; metadata such as
the upstream PR link is kept in run artifacts, not in the simulator prompt.
-Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).
Optional fixed-first-user fixtures keep the first simulated user turn stable
-while preserving live simulator behavior after that turn:
-
-```bash
-export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail_fixture.json
-export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline_fixture.json
-```
-
-Use `config/official.yaml` with a clean TAU-2 checkout when you need an
-official-user-simulator parity run. If the checkout was already patched, the
-artifact records that boundary instead of labeling the run pure official.
+while preserving live simulator behavior after that turn.
## Evidence Boundary
diff --git a/benchmark/tau2/llm/config/baseline.yaml b/benchmark/tau2/llm/config/baseline.yaml
index 31b054dc9..f4fd0091f 100644
--- a/benchmark/tau2/llm/config/baseline.yaml
+++ b/benchmark/tau2/llm/config/baseline.yaml
@@ -34,7 +34,7 @@ eval:
# prompt; run_eval.py applies that small prompt patch idempotently when needed.
user_simulator_policy: confirmation_aware
# Fixed-first-user fixtures keep the first simulated user turn stable while
- # leaving later turns live. Main PR-B evidence requires these env vars.
+ # leaving later turns live. Strict reproduction configs require these env vars.
fixed_first_user_fixtures:
retail: ${TAU2_RETAIL_FIXED_FIRST_USER_FILE:-}
airline: ${TAU2_AIRLINE_FIXED_FIRST_USER_FILE:-}
@@ -53,19 +53,3 @@ openviking:
prewrite_retrieval_top_k: 6
prewrite_inject_top_k: 2
replay_write_policy: read_only
-
-strategies:
- - id: memory_v2_experience_only
- label: OpenViking Memory V2 experience-only
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_experience_only
- train_memory_mode: experience_only
- retrieval_mode: first_user
- - id: memory_v2_prewrite
- label: OpenViking Memory V2 pre-write recall
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_experience_only
- train_memory_mode: experience_only
- retrieval_mode: first_user_prewrite
diff --git a/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml b/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml
index 3bf670b14..92153abd8 100644
--- a/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml
+++ b/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml
@@ -5,7 +5,7 @@ benchmark:
eval:
# Use live simulator turns once to collect the first user request fixture.
- # The resulting fixture can then be reused by the strict PR-B configs.
+ # The resulting fixture can then be reused by strict fixed-first-user configs.
protocol: live_user_bootstrap
require_fixed_first_user: false
user_simulator_policy: confirmation_aware
diff --git a/benchmark/tau2/llm/config/official.yaml b/benchmark/tau2/llm/config/official.yaml
deleted file mode 100644
index 9a47661ff..000000000
--- a/benchmark/tau2/llm/config/official.yaml
+++ /dev/null
@@ -1,12 +0,0 @@
-extends: baseline.yaml
-
-benchmark:
- name: tau2_openviking_official_user_simulator
-
-eval:
- protocol: official_live_user
- require_fixed_first_user: false
- user_simulator_policy: official
- fixed_first_user_fixtures:
- retail:
- airline:
diff --git a/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml
deleted file mode 100644
index be128c15d..000000000
--- a/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml
+++ /dev/null
@@ -1,143 +0,0 @@
-extends: baseline.yaml
-
-benchmark:
- name: tau2_prb_content_matrix_new_prompt
- strategy_concurrency: 16
- task_max_concurrency: 5
- corpus_prepare_concurrency: 1
-
-strategies:
- - id: no_memory
- label: TAU-2 no-memory same-seed baseline
- memory_backend: none
-
- - id: new_traj_fixed_first_user
- label: PR-B new trajectory fixed-count first-user top4
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: trajectories
- retrieval_mode: first_user
- retrieval_top_k: 4
- first_user_inject_top_k: 4
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_traj_fixed_prewrite_only
- label: PR-B new trajectory fixed-count prewrite top2
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: trajectories
- retrieval_mode: prewrite
- retrieval_top_k: 4
- prewrite_retrieval_top_k: 2
- prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_traj_fixed_first_user_prewrite
- label: PR-B new trajectory fixed-count first-user top4 + prewrite top2
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: trajectories
- retrieval_mode: first_user_prewrite
- retrieval_top_k: 4
- first_user_inject_top_k: 4
- prewrite_retrieval_top_k: 2
- prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_exp_fixed_first_user
- label: PR-B new experience fixed-count first-user top2
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: experiences
- retrieval_mode: first_user
- retrieval_top_k: 2
- first_user_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_exp_fixed_prewrite_only
- label: PR-B new experience fixed-count prewrite top2
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: experiences
- retrieval_mode: prewrite
- retrieval_top_k: 2
- prewrite_retrieval_top_k: 2
- prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_exp_fixed_first_user_prewrite
- label: PR-B new experience fixed-count first-user + prewrite top2
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: experiences
- retrieval_mode: first_user_prewrite
- retrieval_top_k: 2
- first_user_inject_top_k: 2
- prewrite_retrieval_top_k: 2
- prewrite_inject_top_k: 2
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_traj_4000_prewrite_only
- label: PR-B new trajectory 4000-char prewrite
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: trajectories
- retrieval_mode: prewrite
- retrieval_top_k: 8
- prewrite_retrieval_top_k: 8
- prewrite_inject_top_k: 8
- memory_inject_max_chars: 4000
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: new_exp_4000_first_user_prewrite
- label: PR-B new experience 4000-char first-user + prewrite
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_operation_family_v1_success_only
- train_memory_mode: experience_only
- train_transcript_format: role_tool_blocks
- train_include_system_prompt: true
- train_skip_failed_sessions: true
- search_memory_type: experiences
- retrieval_mode: first_user_prewrite
- retrieval_top_k: 8
- first_user_inject_top_k: 8
- prewrite_retrieval_top_k: 8
- prewrite_inject_top_k: 8
- memory_inject_max_chars: 4000
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/llm/config/prewrite.yaml b/benchmark/tau2/llm/config/prewrite.yaml
deleted file mode 100644
index 834963b41..000000000
--- a/benchmark/tau2/llm/config/prewrite.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-extends: baseline.yaml
-
-benchmark:
- name: tau2_openviking_prewrite
-
-strategies:
- - id: memory_v2_prewrite
- label: OpenViking Memory V2 pre-write recall
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_experience_only
- train_memory_mode: experience_only
- retrieval_mode: first_user_prewrite
diff --git a/benchmark/tau2/llm/config/prb_scope_fairness.yaml b/benchmark/tau2/llm/config/template_indexed_trajectory.yaml
similarity index 54%
rename from benchmark/tau2/llm/config/prb_scope_fairness.yaml
rename to benchmark/tau2/llm/config/template_indexed_trajectory.yaml
index bee9ed94f..44dc0e84c 100644
--- a/benchmark/tau2/llm/config/prb_scope_fairness.yaml
+++ b/benchmark/tau2/llm/config/template_indexed_trajectory.yaml
@@ -1,30 +1,25 @@
extends: baseline.yaml
benchmark:
- name: tau2_prb_scope_fairness
+ name: tau2_template_indexed_trajectory
strategy_concurrency: 16
task_max_concurrency: 5
corpus_prepare_concurrency: 1
-openviking:
- url: ${OPENVIKING_URL:-http://localhost:1933}
-
strategies:
- id: no_memory
label: TAU-2 no-memory same-seed baseline
memory_backend: none
- - id: no_memory_generic_scope
- label: TAU-2 no-memory same-seed baseline with generic memory scope prompt
- memory_backend: none
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
-
- - id: trajectory_top4_first_user_prewrite_generic_scope
- label: Trajectory top4 first-user + pre-write top2 with generic memory scope prompt
+ - id: template_indexed_trajectory_top4_prewrite_top2
+ label: Template-indexed trajectory top4 first-user + pre-write top2
memory_backend: openviking
train_required: true
- corpus_id: memory_v2_trajectory_view
+ corpus_id: memory_v2_operation_family_v1_success_only
train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
search_memory_type: trajectories
retrieval_mode: first_user_prewrite
retrieval_top_k: 4
diff --git a/benchmark/tau2/llm/config/trajectory.yaml b/benchmark/tau2/llm/config/trajectory.yaml
deleted file mode 100644
index aa0396e12..000000000
--- a/benchmark/tau2/llm/config/trajectory.yaml
+++ /dev/null
@@ -1,57 +0,0 @@
-extends: baseline.yaml
-
-benchmark:
- name: tau2_openviking_trajectory_view
-
-strategies:
- - id: memory_v2_trajectory_view
- label: OpenViking Memory V2 trajectory first-user recall
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_trajectory_view
- train_memory_mode: experience_only
- search_memory_type: trajectories
- retrieval_mode: first_user
- - id: memory_v2_trajectory_view_scope
- label: OpenViking Memory V2 trajectory first-user recall with scope prompt
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_trajectory_view
- train_memory_mode: experience_only
- search_memory_type: trajectories
- retrieval_mode: first_user
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- - id: memory_v2_trajectory_prewrite_only
- label: OpenViking Memory V2 trajectory pre-write-only recall
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_trajectory_view
- train_memory_mode: experience_only
- search_memory_type: trajectories
- retrieval_mode: prewrite
- - id: memory_v2_trajectory_prewrite
- label: OpenViking Memory V2 trajectory first-user + pre-write recall
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_trajectory_view
- train_memory_mode: experience_only
- search_memory_type: trajectories
- retrieval_mode: first_user_prewrite
- - id: memory_v2_trajectory_prewrite_only_scope
- label: OpenViking Memory V2 trajectory pre-write-only recall with scope prompt
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_trajectory_view
- train_memory_mode: experience_only
- search_memory_type: trajectories
- retrieval_mode: prewrite
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
- - id: memory_v2_trajectory_prewrite_scope
- label: OpenViking Memory V2 trajectory first-user + pre-write recall with scope prompt
- memory_backend: openviking
- train_required: true
- corpus_id: memory_v2_trajectory_view
- train_memory_mode: experience_only
- search_memory_type: trajectories
- retrieval_mode: first_user_prewrite
- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md
diff --git a/benchmark/tau2/llm/run_full_eval.sh b/benchmark/tau2/llm/run_full_eval.sh
index a3b63ab8f..ed983bbba 100755
--- a/benchmark/tau2/llm/run_full_eval.sh
+++ b/benchmark/tau2/llm/run_full_eval.sh
@@ -4,7 +4,7 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-python3}"
-CONFIG="$SCRIPT_DIR/config/baseline.yaml"
+CONFIG="$SCRIPT_DIR/config/no_memory.yaml"
EXECUTE=false
PREFLIGHT=false
STRICT_PREFLIGHT=false
diff --git a/benchmark/tau2/llm/scripts/run_eval.py b/benchmark/tau2/llm/scripts/run_eval.py
index 4965eb465..c4aaf4a50 100755
--- a/benchmark/tau2/llm/scripts/run_eval.py
+++ b/benchmark/tau2/llm/scripts/run_eval.py
@@ -1028,7 +1028,7 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:
def main() -> int:
parser = argparse.ArgumentParser(description="Plan or run TAU-2 benchmark cells.")
parser.add_argument(
- "--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml"
+ "--config", type=Path, default=Path(__file__).parents[1] / "config" / "no_memory.yaml"
)
parser.add_argument("--run-id", default=run_id())
parser.add_argument(
diff --git a/benchmark/tau2/llm/scripts/run_memory_v2_eval.py b/benchmark/tau2/llm/scripts/run_memory_v2_eval.py
index dd9ed0633..6cb3488b1 100644
--- a/benchmark/tau2/llm/scripts/run_memory_v2_eval.py
+++ b/benchmark/tau2/llm/scripts/run_memory_v2_eval.py
@@ -967,7 +967,7 @@ def main() -> int:
"How to replay TAU-2 train messages into OpenViking sessions. "
"openviking_text preserves the compact adapter text format; role_tool_blocks "
"uses role-prefixed messages plus tool-call/tool-response blocks. "
- "custom_like is a compatibility alias for older cached PR-B corpora."
+ "custom_like is a compatibility alias for older cached custom-like corpora."
),
)
parser.add_argument(