Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
docs: simplify tau2 benchmark reproduction (#2267)
huangruiteng646ff735dbdb
diff --git a/benchmark/tau2/llm/README.md b/benchmark/tau2/llm/README.mdindex 527d6dedf..4933a4d78 100644--- a/benchmark/tau2/llm/README.md+++ b/benchmark/tau2/llm/README.md@@ -1,14 +1,23 @@# TAU-2 Benchmark-This directory contains a small OpenViking-style entry point for TAU-2 memory-evaluation. The scope is intentionally narrow:+This directory contains the OpenViking TAU-2 LLM benchmark entry point. The+reproduction surface is intentionally narrow:-- fresh OpenViking Memory V2 experience-only baseline;-- Memory V2 pre-write recall treatment.-- trajectory memory retrieval treatment for the refined extraction prompt and- retrieval-anchor embedding text.+- `no_memory`: same-seed TAU-2 baseline without OpenViking memory injection;+- `template_indexed_trajectory_top4_prewrite_top2`: the current best+ template-indexed trajectory memory treatment.-Category rerank and other harness-only diagnostics are intentionally left out.+The template-indexed trajectory treatment trains OpenViking Memory V2 from+TAU-2 train conversations, retrieves generated `trajectories`, and uses the+trajectory embedding template `{{ trajectory_name }}\n\n{{ retrieval_anchor }}`+instead of broad procedure bodies for retrieval. It injects trajectory top4 at+the first user turn and top2 before write-like tool calls, with the generic+memory scope prompt enabled.++Category rerank, experience-memory routes, fixed-count-only ablations,+character-budget ablations, and official-user parity controls are intentionally+left out of this README and config set so reproduction agents do not mistake+diagnostic routes for current evidence.## Layout@@ -16,21 +25,28 @@ Category rerank and other harness-only diagnostics are intentionally left out.benchmark/tau2/llm/├── config/│ ├── baseline.yaml-│ ├── official.yaml-│ ├── prewrite.yaml-│ └── trajectory.yaml+│ ├── fixed_first_user_bootstrap.yaml+│ ├── no_memory.yaml+│ ├── scope_prompts/+│ │ └── generic_memory_scope.md+│ └── template_indexed_trajectory.yaml├── scripts/+│ ├── build_fixed_first_user_fixture.py│ ├── run_eval.py│ ├── setup_tau2_repo.sh│ └── tau2_common.py└── run_full_eval.sh```+`baseline.yaml` is a shared protocol/defaults file, not a runnable evidence+cell by itself. Use `no_memory.yaml` for the baseline-only run and+`template_indexed_trajectory.yaml` for the paired no-memory + trajectory run.+Generated eval artifacts are written to `benchmark/tau2/llm/result/<run_id>/`.Memory corpus artifacts are cached outside the run id at`benchmark/tau2/llm/result/memory_corpora/` by default.-## Quick Start+## SetupThis benchmark delegates task simulation and scoring to an external TAU-2checkout. Point the runner at that checkout and CLI explicitly when they are not@@ -41,21 +57,6 @@ export TAU2_REPO=/path/to/tau2-benchexport TAU2_CLI=/path/to/tau2```-The default OpenViking TAU-2 memory evidence protocol is-`fixed_first_user_full8`: retail + airline, 8 repeats, same seeds, confirmation-aware user simulator, and fixed first user fixtures for both domains. Later user-simulator turns remain live. Set the fixture paths before running the default-configs:--```bash-export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail/fixed_first_user_fixture.json-export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline/fixed_first_user_fixture.json-```--`--strict-preflight` fails when `eval.require_fixed_first_user=true` and either-fixture is missing. Use `config/official.yaml` for an explicit non-fixed,-official-live-user control.-For a local one-command setup, clone and install TAU-2 into ignored benchmarkdirectories:@@ -64,9 +65,15 @@ benchmark/tau2/llm/scripts/setup_tau2_repo.shsource benchmark/tau2/llm/.env.tau2```-For PR-B-compatible reproduction, pin the TAU-2 checkout to a ref that includes-the confirmation-aware text-user-simulator prompt. The original PR-B evidence-used the open TAU-2 fix PR head (`79dbf0c18ac7637aedf869cb3122babcd57aaf17`):+The default OpenViking TAU-2 memory evidence protocol is+`fixed_first_user_full8`: retail + airline, 8 repeats, same seeds,+confirmation-aware user simulator, and fixed first-user fixtures for both+domains. Later user simulator turns remain live.++The confirmation-aware simulator behavior is available from+[sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).+Pin the local TAU-2 checkout to a ref that includes that behavior when+reproducing these numbers:```bashbenchmark/tau2/llm/scripts/setup_tau2_repo.sh \@@ -74,78 +81,22 @@ benchmark/tau2/llm/scripts/setup_tau2_repo.sh \source benchmark/tau2/llm/.env.tau2```-Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).--Plan the default benchmark without running TAU-2:--```bash-python benchmark/tau2/llm/scripts/run_eval.py --config benchmark/tau2/llm/config/baseline.yaml --plan-only-```--Add `--preflight` or `--strict-preflight` when you want the runner to write a-small environment/config check next to the run plan.--After setup, verify the local TAU-2 link and write a one-cell run plan:--```bash-benchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/baseline.yaml \- --strict-preflight \- --domain retail \- --strategy-id memory_v2_experience_only \- --task-id 5 \- --repeat-count 1-```--Plan a one-cell Memory V2 pre-write smoke:+When using Doubao through an OpenAI-compatible endpoint, set `OPENAI_API_KEY`+and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.-```bash-benchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/baseline.yaml \- --domain retail \- --strategy-id memory_v2_prewrite \- --num-tasks 1 \- --repeat-count 1-```+## Fixed-First-User Fixtures-Plan a one-cell trajectory memory smoke:+Strict reproduction requires fixed first-user fixtures:```bash-benchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/trajectory.yaml \- --domain retail \- --strategy-id memory_v2_trajectory_view \- --num-tasks 1 \- --train-num-tasks 1 \- --repeat-count 1-```--Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):--```bash-benchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/baseline.yaml \- --execute+export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail/fixed_first_user_fixture.json+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline/fixed_first_user_fixture.json```-## Reproduce the PR-B Evidence--The PR-B headline and content-shape ablation use-`config/prb_content_matrix_new_prompt.yaml`. It runs the no-memory control plus-trajectory first-user top4 / pre-write top2, experience top2, and representative-4000-character budget ablation routes across `retail + airline` with 8 repeats.-The current trajectory prompt indexes `trajectory_name + retrieval_anchor`-instead of the full procedure body. This keeps retrieval focused on the positive-operation boundary and reduces broad terminal-handoff / cancellation-like memory-matches.--### 1. Bootstrap fixed-first-user fixtures--The default PR-B configs require fixed-first-user fixtures, so a fresh checkout-needs one live-user bootstrap pass before strict reproduction. This pass uses-the same confirmation-aware simulator policy but does not require fixed fixtures.+`--strict-preflight` fails when `eval.require_fixed_first_user=true` and either+fixture is missing.-Run one bootstrap pass per domain:+For a fresh checkout, run one live-user bootstrap pass per domain:```bashbenchmark/tau2/llm/run_full_eval.sh \@@ -193,134 +144,146 @@ export TAU2_RETAIL_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_fexport TAU2_AIRLINE_FIXED_FIRST_USER_FILE="$PWD/benchmark/tau2/llm/result/fixed_first_user_fixtures/airline/fixed_first_user_fixture.json"```-### 2. Run smoke and full PR-B matrix+## Run Plans And Smoke Checks++Plan the no-memory baseline without running TAU-2:++```bash+python benchmark/tau2/llm/scripts/run_eval.py \+ --config benchmark/tau2/llm/config/no_memory.yaml \+ --plan-only+```++Plan the paired current-evidence config without running TAU-2:++```bash+python benchmark/tau2/llm/scripts/run_eval.py \+ --config benchmark/tau2/llm/config/template_indexed_trajectory.yaml \+ --plan-only+```-First run one tiny end-to-end smoke against a clean local OpenViking service:+Run a tiny no-memory smoke:```bashbenchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \+ --config benchmark/tau2/llm/config/no_memory.yaml \--domain retail \- --strategy-id new_traj_fixed_first_user_prewrite \+ --strategy-id no_memory \--num-tasks 1 \- --train-num-tasks 1 \--repeat-count 1 \--strict-preflight \--execute```-Then run the full PR-B matrix:+Run a tiny template-indexed trajectory smoke against a clean local OpenViking+service:```bashbenchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml \- --run-id prb_content_matrix_new_prompt_full8 \+ --config benchmark/tau2/llm/config/template_indexed_trajectory.yaml \+ --domain retail \+ --strategy-id template_indexed_trajectory_top4_prewrite_top2 \+ --num-tasks 1 \+ --train-num-tasks 1 \+ --repeat-count 1 \--strict-preflight \--execute```-The main result is written to-`benchmark/tau2/llm/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.-Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON-lives under `memory_cells/`, and corpus identity / generated memory checks live-under `memory_corpora/`.+Start the OpenViking service before executing memory cells, and verify it with+`ov status`. For trajectory memory evidence, start the service from this branch+and inspect generated trajectory files; changing `search_uri` alone does not+prove the template-indexed trajectory prompt was used.-For the strongest trajectory-only treatment, inspect the-`new_traj_fixed_first_user_prewrite` row. It uses trajectory top4 at the first-user turn, trajectory top4 retrieval before writes, pre-write injection top2,-fixed-first-user fixtures, and the generic scope prompt.+## Full Reproduction-For a small E2E smoke, keep both the eval and train slices tiny:+Run the no-memory full8 baseline:```bashbenchmark/tau2/llm/run_full_eval.sh \- --config benchmark/tau2/llm/config/baseline.yaml \- --domain retail \- --strategy-id memory_v2_experience_only \- --num-tasks 1 \- --train-num-tasks 1 \- --repeat-count 1 \+ --config benchmark/tau2/llm/config/no_memory.yaml \+ --run-id no_memory_full8 \+ --strict-preflight \--execute```-When using Doubao through an OpenAI-compatible endpoint, set `OPENAI_API_KEY`-and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.+Run the paired no-memory + current trajectory evidence config:-Start the OpenViking service before executing memory cells, and verify it with-`ov status`. For evidence runs, use a clean OpenViking workspace/config and set-`OPENVIKING_URL` explicitly so local template overrides do not pollute the-Memory V2 baseline. For trajectory memory evidence, start the service from this-branch and inspect generated trajectory files; changing `search_uri` alone does-not prove the new trajectory prompt was used.+```bash+benchmark/tau2/llm/run_full_eval.sh \+ --config benchmark/tau2/llm/config/template_indexed_trajectory.yaml \+ --run-id template_indexed_trajectory_full8 \+ --strict-preflight \+ --execute+```++The main result is written to+`benchmark/tau2/llm/result/template_indexed_trajectory_full8/scoreboard.json`.+Per-cell execution records live under `cell_results/`, raw TAU-2 result JSON+lives under `memory_cells/`, and corpus identity / generated memory checks live+under `memory_corpora/`.## Memory Adapter-Memory V2 cells run through a small TAU-2 agent adapter in this directory:+Memory cells run through a small TAU-2 agent adapter in this directory:- train by writing TAU-2 training conversations into OpenViking sessions;-- evaluate by retrieving OpenViking memory at the first user turn;+- retrieve OpenViking memory at the first user turn;- for pre-write recall, retrieve again before write-like tool calls and- regenerate that step with the matched memories. The default benchmark- retrieves 6 pre-write candidates and injects 2, which keeps extra candidates- visible in traces without expanding the prompt budget;-- optionally run an explicit generic scope-prompt treatment that keeps retrieved- memories advisory and asks the agent to preserve the current task scope before- write-like tool calls. The benchmark configs use a single benchmark-neutral- `scope_prompt_file`; the runner still accepts `scope_prompt_files` for custom- local experiments;-- emit artifact metadata to identify the OpenViking account, agent,- corpus, retrieval mode, and simulator policy used by each cell.--For exploratory gates, prefer a bounded run with `--cell-timeout-seconds`.-Timed-out cells are recorded with return code `124`, `timed_out=true`, and are-excluded from scoreboard metrics, which keeps smoke runs from silently becoming-long-running evidence jobs.--The existing `train_memory_mode: experience_only` value selects the Memory V2-session-commit path. `search_memory_type` selects which generated memory bucket-is retrieved during eval (`experiences` by default, `trajectories` for-`config/trajectory.yaml`). The runner prepares each distinct-`domain + corpus_id` once and reuses it across eval run ids when the cached-`corpus_manifest.json` is present. Different corpora may be prepared in-parallel with `benchmark.corpus_prepare_concurrency`; session commits inside one-corpus remain serial to preserve OpenViking write semantics.+ regenerate that step with the matched memories;+- optionally apply a generic scope prompt that keeps retrieved memories+ advisory and asks the agent to preserve the current task scope before+ write-like tool calls;+- emit artifact metadata identifying the OpenViking account, agent, corpus,+ retrieval mode, search memory type, and simulator policy used by each cell.++The current trajectory config uses:++- `train_memory_mode: experience_only`, which selects the Memory V2+ session-commit path that writes generated memory artifacts;+- `train_transcript_format: role_tool_blocks`, which preserves role-prefixed+ messages plus tool-call/tool-response blocks during training;+- `train_include_system_prompt: true`, which includes the domain policy in the+ training session;+- `train_skip_failed_sessions: true`, which avoids learning from failed train+ sessions;+- `search_memory_type: trajectories`, which retrieves generated trajectory+ memory during eval.++The runner prepares each distinct `domain + corpus_id` once and reuses it across+eval run ids when the cached `corpus_manifest.json` is present. Different+corpora may be prepared in parallel with `benchmark.corpus_prepare_concurrency`;+session commits inside one corpus remain serial to preserve OpenViking write+semantics.By default, trajectory extraction is transcript-only: the runner replays TAU-2messages into an OpenViking session and does not expose held-out reward or-assertion results to the extractor. The PR-B evidence config can also use a-structured role/tool transcript, include the domain policy in the training-session, skip failed train sessions when building positive procedure memory, and-cap injected memory by total character budget for content-shape ablations.+assertion results to the extractor.Eval cells run in parallel with `benchmark.strategy_concurrency` by default andcan be overridden with `--strategy-concurrency`. This only parallelizes read-onlyTAU-2 eval cells; corpus writes inside one corpus are still serialized by theprepare step.+For exploratory gates, prefer a bounded run with `--cell-timeout-seconds`.+Timed-out cells are recorded with return code `124`, `timed_out=true`, and are+excluded from scoreboard metrics, which keeps smoke runs from silently becoming+long-running evidence jobs.+## User Simulator PolicyThe runner default is the official TAU-2 user simulator if`eval.user_simulator_policy` is omitted. The bundled OpenViking memory benchmark-config sets `confirmation_aware`, because a memory benchmark should not treat+configs set `confirmation_aware`, because a memory benchmark should not treatuser confirmation as task completion before the backend write has happened.`confirmation_aware` applies a small idempotent prompt patch to the configuredTAU-2 checkout before planning or running. The patch appends only the behavioralconfirmation boundary to the TAU-2 user simulator guidelines; metadata such asthe upstream PR link is kept in run artifacts, not in the simulator prompt.-Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).Optional fixed-first-user fixtures keep the first simulated user turn stable-while preserving live simulator behavior after that turn:--```bash-export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail_fixture.json-export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline_fixture.json-```--Use `config/official.yaml` with a clean TAU-2 checkout when you need an-official-user-simulator parity run. If the checkout was already patched, the-artifact records that boundary instead of labeling the run pure official.+while preserving live simulator behavior after that turn.## Evidence Boundarydiff --git a/benchmark/tau2/llm/config/baseline.yaml b/benchmark/tau2/llm/config/baseline.yamlindex 31b054dc9..f4fd0091f 100644--- a/benchmark/tau2/llm/config/baseline.yaml+++ b/benchmark/tau2/llm/config/baseline.yaml@@ -34,7 +34,7 @@ eval:# prompt; run_eval.py applies that small prompt patch idempotently when needed.user_simulator_policy: confirmation_aware# Fixed-first-user fixtures keep the first simulated user turn stable while- # leaving later turns live. Main PR-B evidence requires these env vars.+ # leaving later turns live. Strict reproduction configs require these env vars.fixed_first_user_fixtures:retail: ${TAU2_RETAIL_FIXED_FIRST_USER_FILE:-}airline: ${TAU2_AIRLINE_FIXED_FIRST_USER_FILE:-}@@ -53,19 +53,3 @@ openviking:prewrite_retrieval_top_k: 6prewrite_inject_top_k: 2replay_write_policy: read_only--strategies:- - id: memory_v2_experience_only- label: OpenViking Memory V2 experience-only- memory_backend: openviking- train_required: true- corpus_id: memory_v2_experience_only- train_memory_mode: experience_only- retrieval_mode: first_user- - id: memory_v2_prewrite- label: OpenViking Memory V2 pre-write recall- memory_backend: openviking- train_required: true- corpus_id: memory_v2_experience_only- train_memory_mode: experience_only- retrieval_mode: first_user_prewritediff --git a/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml b/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yamlindex 3bf670b14..92153abd8 100644--- a/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml+++ b/benchmark/tau2/llm/config/fixed_first_user_bootstrap.yaml@@ -5,7 +5,7 @@ benchmark:eval:# Use live simulator turns once to collect the first user request fixture.- # The resulting fixture can then be reused by the strict PR-B configs.+ # The resulting fixture can then be reused by strict fixed-first-user configs.protocol: live_user_bootstraprequire_fixed_first_user: falseuser_simulator_policy: confirmation_awarediff --git a/benchmark/tau2/llm/config/official.yaml b/benchmark/tau2/llm/config/official.yamldeleted file mode 100644index 9a47661ff..000000000--- a/benchmark/tau2/llm/config/official.yaml+++ /dev/null@@ -1,12 +0,0 @@-extends: baseline.yaml--benchmark:- name: tau2_openviking_official_user_simulator--eval:- protocol: official_live_user- require_fixed_first_user: false- user_simulator_policy: official- fixed_first_user_fixtures:- retail:- airline:diff --git a/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yamldeleted file mode 100644index be128c15d..000000000--- a/benchmark/tau2/llm/config/prb_content_matrix_new_prompt.yaml+++ /dev/null@@ -1,143 +0,0 @@-extends: baseline.yaml--benchmark:- name: tau2_prb_content_matrix_new_prompt- strategy_concurrency: 16- task_max_concurrency: 5- corpus_prepare_concurrency: 1--strategies:- - id: no_memory- label: TAU-2 no-memory same-seed baseline- memory_backend: none-- - id: new_traj_fixed_first_user- label: PR-B new trajectory fixed-count first-user top4- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: trajectories- retrieval_mode: first_user- retrieval_top_k: 4- first_user_inject_top_k: 4- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_traj_fixed_prewrite_only- label: PR-B new trajectory fixed-count prewrite top2- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: trajectories- retrieval_mode: prewrite- retrieval_top_k: 4- prewrite_retrieval_top_k: 2- prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_traj_fixed_first_user_prewrite- label: PR-B new trajectory fixed-count first-user top4 + prewrite top2- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: trajectories- retrieval_mode: first_user_prewrite- retrieval_top_k: 4- first_user_inject_top_k: 4- prewrite_retrieval_top_k: 2- prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_exp_fixed_first_user- label: PR-B new experience fixed-count first-user top2- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: experiences- retrieval_mode: first_user- retrieval_top_k: 2- first_user_inject_top_k: 2- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_exp_fixed_prewrite_only- label: PR-B new experience fixed-count prewrite top2- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: experiences- retrieval_mode: prewrite- retrieval_top_k: 2- prewrite_retrieval_top_k: 2- prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_exp_fixed_first_user_prewrite- label: PR-B new experience fixed-count first-user + prewrite top2- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: experiences- retrieval_mode: first_user_prewrite- retrieval_top_k: 2- first_user_inject_top_k: 2- prewrite_retrieval_top_k: 2- prewrite_inject_top_k: 2- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_traj_4000_prewrite_only- label: PR-B new trajectory 4000-char prewrite- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: trajectories- retrieval_mode: prewrite- retrieval_top_k: 8- prewrite_retrieval_top_k: 8- prewrite_inject_top_k: 8- memory_inject_max_chars: 4000- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: new_exp_4000_first_user_prewrite- label: PR-B new experience 4000-char first-user + prewrite- memory_backend: openviking- train_required: true- corpus_id: memory_v2_operation_family_v1_success_only- train_memory_mode: experience_only- train_transcript_format: role_tool_blocks- train_include_system_prompt: true- train_skip_failed_sessions: true- search_memory_type: experiences- retrieval_mode: first_user_prewrite- retrieval_top_k: 8- first_user_inject_top_k: 8- prewrite_retrieval_top_k: 8- prewrite_inject_top_k: 8- memory_inject_max_chars: 4000- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/llm/config/prewrite.yaml b/benchmark/tau2/llm/config/prewrite.yamldeleted file mode 100644index 834963b41..000000000--- a/benchmark/tau2/llm/config/prewrite.yaml+++ /dev/null@@ -1,13 +0,0 @@-extends: baseline.yaml--benchmark:- name: tau2_openviking_prewrite--strategies:- - id: memory_v2_prewrite- label: OpenViking Memory V2 pre-write recall- memory_backend: openviking- train_required: true- corpus_id: memory_v2_experience_only- train_memory_mode: experience_only- retrieval_mode: first_user_prewritediff --git a/benchmark/tau2/llm/config/prb_scope_fairness.yaml b/benchmark/tau2/llm/config/template_indexed_trajectory.yamlsimilarity index 54%rename from benchmark/tau2/llm/config/prb_scope_fairness.yamlrename to benchmark/tau2/llm/config/template_indexed_trajectory.yamlindex bee9ed94f..44dc0e84c 100644--- a/benchmark/tau2/llm/config/prb_scope_fairness.yaml+++ b/benchmark/tau2/llm/config/template_indexed_trajectory.yaml@@ -1,30 +1,25 @@extends: baseline.yamlbenchmark:- name: tau2_prb_scope_fairness+ name: tau2_template_indexed_trajectorystrategy_concurrency: 16task_max_concurrency: 5corpus_prepare_concurrency: 1-openviking:- url: ${OPENVIKING_URL:-http://localhost:1933}-strategies:- id: no_memorylabel: TAU-2 no-memory same-seed baselinememory_backend: none- - id: no_memory_generic_scope- label: TAU-2 no-memory same-seed baseline with generic memory scope prompt- memory_backend: none- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md-- - id: trajectory_top4_first_user_prewrite_generic_scope- label: Trajectory top4 first-user + pre-write top2 with generic memory scope prompt+ - id: template_indexed_trajectory_top4_prewrite_top2+ label: Template-indexed trajectory top4 first-user + pre-write top2memory_backend: openvikingtrain_required: true- corpus_id: memory_v2_trajectory_view+ corpus_id: memory_v2_operation_family_v1_success_onlytrain_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: truesearch_memory_type: trajectoriesretrieval_mode: first_user_prewriteretrieval_top_k: 4diff --git a/benchmark/tau2/llm/config/trajectory.yaml b/benchmark/tau2/llm/config/trajectory.yamldeleted file mode 100644index aa0396e12..000000000--- a/benchmark/tau2/llm/config/trajectory.yaml+++ /dev/null@@ -1,57 +0,0 @@-extends: baseline.yaml--benchmark:- name: tau2_openviking_trajectory_view--strategies:- - id: memory_v2_trajectory_view- label: OpenViking Memory V2 trajectory first-user recall- memory_backend: openviking- train_required: true- corpus_id: memory_v2_trajectory_view- train_memory_mode: experience_only- search_memory_type: trajectories- retrieval_mode: first_user- - id: memory_v2_trajectory_view_scope- label: OpenViking Memory V2 trajectory first-user recall with scope prompt- memory_backend: openviking- train_required: true- corpus_id: memory_v2_trajectory_view- train_memory_mode: experience_only- search_memory_type: trajectories- retrieval_mode: first_user- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- - id: memory_v2_trajectory_prewrite_only- label: OpenViking Memory V2 trajectory pre-write-only recall- memory_backend: openviking- train_required: true- corpus_id: memory_v2_trajectory_view- train_memory_mode: experience_only- search_memory_type: trajectories- retrieval_mode: prewrite- - id: memory_v2_trajectory_prewrite- label: OpenViking Memory V2 trajectory first-user + pre-write recall- memory_backend: openviking- train_required: true- corpus_id: memory_v2_trajectory_view- train_memory_mode: experience_only- search_memory_type: trajectories- retrieval_mode: first_user_prewrite- - id: memory_v2_trajectory_prewrite_only_scope- label: OpenViking Memory V2 trajectory pre-write-only recall with scope prompt- memory_backend: openviking- train_required: true- corpus_id: memory_v2_trajectory_view- train_memory_mode: experience_only- search_memory_type: trajectories- retrieval_mode: prewrite- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.md- - id: memory_v2_trajectory_prewrite_scope- label: OpenViking Memory V2 trajectory first-user + pre-write recall with scope prompt- memory_backend: openviking- train_required: true- corpus_id: memory_v2_trajectory_view- train_memory_mode: experience_only- search_memory_type: trajectories- retrieval_mode: first_user_prewrite- scope_prompt_file: benchmark/tau2/llm/config/scope_prompts/generic_memory_scope.mddiff --git a/benchmark/tau2/llm/run_full_eval.sh b/benchmark/tau2/llm/run_full_eval.shindex a3b63ab8f..ed983bbba 100755--- a/benchmark/tau2/llm/run_full_eval.sh+++ b/benchmark/tau2/llm/run_full_eval.sh@@ -4,7 +4,7 @@ set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"PYTHON_BIN="${PYTHON_BIN:-python3}"-CONFIG="$SCRIPT_DIR/config/baseline.yaml"+CONFIG="$SCRIPT_DIR/config/no_memory.yaml"EXECUTE=falsePREFLIGHT=falseSTRICT_PREFLIGHT=falsediff --git a/benchmark/tau2/llm/scripts/run_eval.py b/benchmark/tau2/llm/scripts/run_eval.pyindex 4965eb465..c4aaf4a50 100755--- a/benchmark/tau2/llm/scripts/run_eval.py+++ b/benchmark/tau2/llm/scripts/run_eval.py@@ -1028,7 +1028,7 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:def main() -> int:parser = argparse.ArgumentParser(description="Plan or run TAU-2 benchmark cells.")parser.add_argument(- "--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml"+ "--config", type=Path, default=Path(__file__).parents[1] / "config" / "no_memory.yaml")parser.add_argument("--run-id", default=run_id())parser.add_argument(diff --git a/benchmark/tau2/llm/scripts/run_memory_v2_eval.py b/benchmark/tau2/llm/scripts/run_memory_v2_eval.pyindex dd9ed0633..6cb3488b1 100644--- a/benchmark/tau2/llm/scripts/run_memory_v2_eval.py+++ b/benchmark/tau2/llm/scripts/run_memory_v2_eval.py@@ -967,7 +967,7 @@ def main() -> int:"How to replay TAU-2 train messages into OpenViking sessions. ""openviking_text preserves the compact adapter text format; role_tool_blocks ""uses role-prefixed messages plus tool-call/tool-response blocks. "- "custom_like is a compatibility alias for older cached PR-B corpora."+ "custom_like is a compatibility alias for older cached custom-like corpora."),)parser.add_argument(