Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(memory): upgrade trajectory extraction to beat no-memory baseline (#2017)
huangruiteng520713c6247e
diff --git a/benchmark/tau2/README.md b/benchmark/tau2/README.mdindex 7ebdb0807..3c9e5501a 100644--- a/benchmark/tau2/README.md+++ b/benchmark/tau2/README.md@@ -1,13 +1,13 @@# TAU-2 BenchmarkThis directory contains a small OpenViking-style entry point for TAU-2 memory-evaluation. The first version is intentionally narrow:+evaluation. The scope is intentionally narrow:- fresh OpenViking Memory V2 experience-only baseline;- Memory V2 pre-write recall treatment.+- trajectory memory retrieval treatment for the refined extraction prompt.-Trajectory / procedure-view prompts, category rerank, and other harness-only-diagnostics are intentionally left out of this first PR.+Category rerank and other harness-only diagnostics are intentionally left out.## Layout@@ -16,7 +16,8 @@ benchmark/tau2/├── config/│ ├── baseline.yaml│ ├── official.yaml-│ └── prewrite.yaml+│ ├── prewrite.yaml+│ └── trajectory.yaml├── scripts/│ ├── run_eval.py│ ├── setup_tau2_repo.sh@@ -24,7 +25,9 @@ benchmark/tau2/└── run_full_eval.sh```-Generated artifacts are written to `benchmark/tau2/result/<run_id>/`.+Generated eval artifacts are written to `benchmark/tau2/result/<run_id>/`.+Memory corpus artifacts are cached outside the run id at+`benchmark/tau2/result/memory_corpora/` by default.## Quick Start@@ -37,6 +40,21 @@ export TAU2_REPO=/path/to/tau2-benchexport TAU2_CLI=/path/to/tau2```+The default OpenViking TAU-2 memory evidence protocol is+`fixed_first_user_full8`: retail + airline, 8 repeats, same seeds, confirmation+aware user simulator, and fixed first user fixtures for both domains. Later user+simulator turns remain live. Set the fixture paths before running the default+configs:++```bash+export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail/fixed_first_user_fixture.json+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline/fixed_first_user_fixture.json+```++`--strict-preflight` fails when `eval.require_fixed_first_user=true` and either+fixture is missing. Use `config/official.yaml` for an explicit non-fixed,+official-live-user control.+For a local one-command setup, clone and install TAU-2 into ignored benchmarkdirectories:@@ -77,6 +95,18 @@ benchmark/tau2/run_full_eval.sh \--repeat-count 1```+Plan a one-cell trajectory memory smoke:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/trajectory.yaml \+ --domain retail \+ --strategy-id memory_v2_trajectory_view \+ --num-tasks 1 \+ --train-num-tasks 1 \+ --repeat-count 1+```+Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):```bash@@ -85,6 +115,42 @@ benchmark/tau2/run_full_eval.sh \--execute```+## Reproduce the PR-B Evidence++The PR-B headline and content-shape ablation use+`config/prb_content_matrix_new_prompt.yaml`. It runs the no-memory control plus+trajectory top4, experience top2, and representative 4000-character budget+ablation routes across `retail + airline` with 8 repeats.++First run one tiny end-to-end smoke against a clean local OpenViking service:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \+ --domain retail \+ --strategy-id new_traj_fixed_first_user_prewrite \+ --num-tasks 1 \+ --train-num-tasks 1 \+ --repeat-count 1 \+ --strict-preflight \+ --execute+```++Then run the full PR-B matrix:++```bash+benchmark/tau2/run_full_eval.sh \+ --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \+ --run-id prb_content_matrix_new_prompt_full8 \+ --strict-preflight \+ --execute+```++The main result is written to+`benchmark/tau2/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.+Per-cell outputs live under `cell_results/`; corpus identity and generated+memory checks live under `memory_corpora/`.+For a small E2E smoke, keep both the eval and train slices tiny:```bash@@ -103,21 +169,54 @@ and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.Start the OpenViking service before executing memory cells, and verify it with`ov status`. For evidence runs, use a clean OpenViking workspace/config and set-`OPENVIKING_URL` explicitly so local custom memory templates do not pollute the-Memory V2 baseline.+`OPENVIKING_URL` explicitly so local template overrides do not pollute the+Memory V2 baseline. For trajectory memory evidence, start the service from this+branch and inspect generated trajectory files; changing `search_uri` alone does+not prove the new trajectory prompt was used.## Memory Adapter-`memory_v2_experience_only` and `memory_v2_prewrite` cells run through a small-TAU-2 agent adapter in this directory:+Memory V2 cells run through a small TAU-2 agent adapter in this directory:- train by writing TAU-2 training conversations into OpenViking sessions;-- evaluate by retrieving OpenViking experience memory at the first user turn;+- evaluate by retrieving OpenViking memory at the first user turn;- for pre-write recall, retrieve again before write-like tool calls and- regenerate that step with the matched memories;+ regenerate that step with the matched memories. The default benchmark+ retrieves 6 pre-write candidates and injects 2, which keeps extra candidates+ visible in traces without expanding the prompt budget;+- optionally run an explicit scope-prompt treatment that keeps retrieved+ memories advisory and asks the agent to preserve the current task scope before+ write-like tool calls. Configs provide per-domain files through+ `scope_prompt_files`;- emit artifact metadata to identify the OpenViking account, agent,corpus, retrieval mode, and simulator policy used by each cell.+For exploratory gates, prefer a bounded run with `--cell-timeout-seconds`.+Timed-out cells are recorded with return code `124`, `timed_out=true`, and are+excluded from scoreboard metrics, which keeps smoke runs from silently becoming+long-running evidence jobs.++The existing `train_memory_mode: experience_only` value selects the Memory V2+session-commit path. `search_memory_type` selects which generated memory bucket+is retrieved during eval (`experiences` by default, `trajectories` for+`config/trajectory.yaml`). The runner prepares each distinct+`domain + corpus_id` once and reuses it across eval run ids when the cached+`corpus_manifest.json` is present. Different corpora may be prepared in+parallel with `benchmark.corpus_prepare_concurrency`; session commits inside one+corpus remain serial to preserve OpenViking write semantics.++By default, trajectory extraction is transcript-only: the runner replays TAU-2+messages into an OpenViking session and does not expose held-out reward or+assertion results to the extractor. The PR-B evidence config can also use a+structured role/tool transcript, include the domain policy in the training+session, skip failed train sessions when building positive procedure memory, and+cap injected memory by total character budget for content-shape ablations.++Eval cells run in parallel with `benchmark.strategy_concurrency` by default and+can be overridden with `--strategy-concurrency`. This only parallelizes read-only+TAU-2 eval cells; corpus writes inside one corpus are still serialized by the+prepare step.+## User Simulator PolicyThe runner default is the official TAU-2 user simulator if@@ -131,6 +230,14 @@ confirmation boundary to the TAU-2 user simulator guidelines; metadata such asthe upstream PR link is kept in run artifacts, not in the simulator prompt.Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).+Optional fixed-first-user fixtures keep the first simulated user turn stable+while preserving live simulator behavior after that turn:++```bash+export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail_fixture.json+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline_fixture.json+```+Use `config/official.yaml` with a clean TAU-2 checkout when you need anofficial-user-simulator parity run. If the checkout was already patched, theartifact records that boundary instead of labeling the run pure official.diff --git a/benchmark/tau2/config/baseline.yaml b/benchmark/tau2/config/baseline.yamlindex 4c4a5060e..0672322bf 100644--- a/benchmark/tau2/config/baseline.yaml+++ b/benchmark/tau2/config/baseline.yaml@@ -6,7 +6,9 @@ benchmark:train_split_name: traineval_split_name: testrepeat_count: 8+ strategy_concurrency: 8task_max_concurrency: 10+ corpus_prepare_concurrency: 2max_steps: 200seed: 300agent: llm_agent@@ -17,12 +19,25 @@ paths:tau2_repo: ${TAU2_REPO:-data/external_benchmarks/tau2-bench}tau2_cli: ${TAU2_CLI:-tau2}output_dir: benchmark/tau2/result+ # Corpus writes are expensive and should be reused across eval run ids when+ # the train split and memory prompt/config did not change.+ corpus_cache_dir: benchmark/tau2/result/memory_corporaeval:+ # Default OpenViking TAU-2 memory evidence uses the fixed-first-user full8+ # protocol: retail + airline, 8 repeats, same seeds, first user turn pinned by+ # fixtures, later user simulator turns still live.+ protocol: fixed_first_user_full8+ require_fixed_first_user: true# The runner default is official if this field is omitted. The OpenViking# memory benchmark config opts into a confirmation-aware TAU-2 user simulator# prompt; run_eval.py applies that small prompt patch idempotently when needed.user_simulator_policy: confirmation_aware+ # Fixed-first-user fixtures keep the first simulated user turn stable while+ # leaving later turns live. Main PR-B evidence requires these env vars.+ fixed_first_user_fixtures:+ retail: ${TAU2_RETAIL_FIXED_FIRST_USER_FILE:-}+ airline: ${TAU2_AIRLINE_FIXED_FIRST_USER_FILE:-}model:agent_llm: ${TAU2_AGENT_LLM:-openai/doubao-seed-2-0-pro-260215}@@ -33,7 +48,10 @@ openviking:url: ${OPENVIKING_URL:-http://localhost:1933}account: ${OPENVIKING_ACCOUNT:-default}agent_id: ${OPENVIKING_AGENT_ID:-tau2-openviking-agent}+ reuse_corpus_across_runs: trueretrieval_top_k: 4+ prewrite_retrieval_top_k: 6+ prewrite_inject_top_k: 2replay_write_policy: read_onlystrategies:diff --git a/benchmark/tau2/config/no_memory.yaml b/benchmark/tau2/config/no_memory.yamlnew file mode 100644index 000000000..93f35633b--- /dev/null+++ b/benchmark/tau2/config/no_memory.yaml@@ -0,0 +1,9 @@+extends: baseline.yaml++benchmark:+ name: tau2_openviking_no_memory++strategies:+ - id: no_memory+ label: TAU-2 no-memory baseline+ memory_backend: nonediff --git a/benchmark/tau2/config/official.yaml b/benchmark/tau2/config/official.yamlindex d10bee872..9a47661ff 100644--- a/benchmark/tau2/config/official.yaml+++ b/benchmark/tau2/config/official.yaml@@ -4,4 +4,9 @@ benchmark:name: tau2_openviking_official_user_simulatoreval:+ protocol: official_live_user+ require_fixed_first_user: falseuser_simulator_policy: official+ fixed_first_user_fixtures:+ retail:+ airline:diff --git a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/config/prb_content_matrix_new_prompt.yamlnew file mode 100644index 000000000..bfb711e74--- /dev/null+++ b/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml@@ -0,0 +1,159 @@+extends: baseline.yaml++benchmark:+ name: tau2_prb_content_matrix_new_prompt+ strategy_concurrency: 16+ task_max_concurrency: 5+ corpus_prepare_concurrency: 1++strategies:+ - id: no_memory+ label: TAU-2 no-memory same-seed baseline+ memory_backend: none++ - id: new_traj_fixed_first_user+ label: PR-B new trajectory fixed-count first-user top4+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: trajectories+ retrieval_mode: first_user+ retrieval_top_k: 4+ first_user_inject_top_k: 4+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_traj_fixed_prewrite_only+ label: PR-B new trajectory fixed-count prewrite top4+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: trajectories+ retrieval_mode: prewrite+ retrieval_top_k: 4+ prewrite_retrieval_top_k: 4+ prewrite_inject_top_k: 4+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_traj_fixed_first_user_prewrite+ label: PR-B new trajectory fixed-count first-user + prewrite top4+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: trajectories+ retrieval_mode: first_user_prewrite+ retrieval_top_k: 4+ first_user_inject_top_k: 4+ prewrite_retrieval_top_k: 4+ prewrite_inject_top_k: 4+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_exp_fixed_first_user+ label: PR-B new experience fixed-count first-user top2+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: experiences+ retrieval_mode: first_user+ retrieval_top_k: 2+ first_user_inject_top_k: 2+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_exp_fixed_prewrite_only+ label: PR-B new experience fixed-count prewrite top2+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: experiences+ retrieval_mode: prewrite+ retrieval_top_k: 2+ prewrite_retrieval_top_k: 2+ prewrite_inject_top_k: 2+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_exp_fixed_first_user_prewrite+ label: PR-B new experience fixed-count first-user + prewrite top2+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: experiences+ retrieval_mode: first_user_prewrite+ retrieval_top_k: 2+ first_user_inject_top_k: 2+ prewrite_retrieval_top_k: 2+ prewrite_inject_top_k: 2+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_traj_4000_prewrite_only+ label: PR-B new trajectory 4000-char prewrite+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: trajectories+ retrieval_mode: prewrite+ retrieval_top_k: 8+ prewrite_retrieval_top_k: 8+ prewrite_inject_top_k: 8+ memory_inject_max_chars: 4000+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md++ - id: new_exp_4000_first_user_prewrite+ label: PR-B new experience 4000-char first-user + prewrite+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_operation_family_v1_success_only+ train_memory_mode: experience_only+ train_transcript_format: role_tool_blocks+ train_include_system_prompt: true+ train_skip_failed_sessions: true+ search_memory_type: experiences+ retrieval_mode: first_user_prewrite+ retrieval_top_k: 8+ first_user_inject_top_k: 8+ prewrite_retrieval_top_k: 8+ prewrite_inject_top_k: 8+ memory_inject_max_chars: 4000+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.mddiff --git a/benchmark/tau2/config/scope_prompts/airline_memory_scope.md b/benchmark/tau2/config/scope_prompts/airline_memory_scope.mdnew file mode 100644index 000000000..8847796a9--- /dev/null+++ b/benchmark/tau2/config/scope_prompts/airline_memory_scope.md@@ -0,0 +1,18 @@+<openviking_memory_scope_guard>+OpenViking memories are advisory. Use them only when their trigger, preconditions,+and applicability boundary match the current airline task.++- Do not broaden the user's requested booking, cancellation, rebooking, flight+ update, passenger update, baggage update, insurance, or payment scope because a+ retrieved memory describes a nearby workflow.+- Keep the current reservation scope explicit. Only use flights, passengers,+ baggage entries, cabin changes, insurance choices, payment IDs, dates, and+ amounts that are grounded in user input, recent tool observations, reservation+ state, profile/payment state, or an explicit search/lookup result.+- Before a write tool call, verify that the selected write action matches the+ user's requested operation. Do not mix cancellation, rebooking, upgrade,+ downgrade, baggage, or passenger-update flows unless the current task asks for+ that combined operation.+- If a memory and the current task disagree, follow the current task state and the+ domain policy.+</openviking_memory_scope_guard>diff --git a/benchmark/tau2/config/scope_prompts/retail_memory_scope.md b/benchmark/tau2/config/scope_prompts/retail_memory_scope.mdnew file mode 100644index 000000000..65a8d61ff--- /dev/null+++ b/benchmark/tau2/config/scope_prompts/retail_memory_scope.md@@ -0,0 +1,16 @@+<openviking_memory_scope_guard>+OpenViking memories are advisory. Use them only when their trigger, preconditions,+and applicability boundary match the current retail task.++- Do not broaden the user's requested replacement, return, exchange, cancellation,+ address-change, or payment scope because a retrieved memory describes a nearby+ workflow.+- If the user restricts the request to the current order, same order, observed+ order items, or a specific product variant, choose write arguments only from the+ current tool observations or an explicitly requested catalog lookup.+- Before a write tool call, order IDs, item IDs, new item IDs, payment method IDs,+ addresses, amounts, and refund/payment direction must be grounded in user input,+ recent tool observations, profile/order state, or an explicit catalog lookup.+- If a memory and the current task disagree, follow the current task state and the+ domain policy.+</openviking_memory_scope_guard>diff --git a/benchmark/tau2/config/trajectory.yaml b/benchmark/tau2/config/trajectory.yamlnew file mode 100644index 000000000..40c378542--- /dev/null+++ b/benchmark/tau2/config/trajectory.yaml@@ -0,0 +1,63 @@+extends: baseline.yaml++benchmark:+ name: tau2_openviking_trajectory_view++strategies:+ - id: memory_v2_trajectory_view+ label: OpenViking Memory V2 trajectory first-user recall+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_trajectory_view+ train_memory_mode: experience_only+ search_memory_type: trajectories+ retrieval_mode: first_user+ - id: memory_v2_trajectory_view_scope+ label: OpenViking Memory V2 trajectory first-user recall with scope prompt+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_trajectory_view+ train_memory_mode: experience_only+ search_memory_type: trajectories+ retrieval_mode: first_user+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md+ - id: memory_v2_trajectory_prewrite_only+ label: OpenViking Memory V2 trajectory pre-write-only recall+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_trajectory_view+ train_memory_mode: experience_only+ search_memory_type: trajectories+ retrieval_mode: prewrite+ - id: memory_v2_trajectory_prewrite+ label: OpenViking Memory V2 trajectory first-user + pre-write recall+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_trajectory_view+ train_memory_mode: experience_only+ search_memory_type: trajectories+ retrieval_mode: first_user_prewrite+ - id: memory_v2_trajectory_prewrite_only_scope+ label: OpenViking Memory V2 trajectory pre-write-only recall with scope prompt+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_trajectory_view+ train_memory_mode: experience_only+ search_memory_type: trajectories+ retrieval_mode: prewrite+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md+ - id: memory_v2_trajectory_prewrite_scope+ label: OpenViking Memory V2 trajectory first-user + pre-write recall with scope prompt+ memory_backend: openviking+ train_required: true+ corpus_id: memory_v2_trajectory_view+ train_memory_mode: experience_only+ search_memory_type: trajectories+ retrieval_mode: first_user_prewrite+ scope_prompt_files:+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.mddiff --git a/benchmark/tau2/run_full_eval.sh b/benchmark/tau2/run_full_eval.shindex ca69a7a32..728f3ce46 100755--- a/benchmark/tau2/run_full_eval.sh+++ b/benchmark/tau2/run_full_eval.sh@@ -33,7 +33,7 @@ while [[ $# -gt 0 ]]; doSTRICT_PREFLIGHT=trueshift;;- --domain|--repeat-count|--strategy-id|--task-id|--num-tasks|--train-num-tasks)+ --domain|--repeat-count|--strategy-id|--task-id|--num-tasks|--train-num-tasks|--strategy-concurrency)RUN_EVAL_EXTRA+=("$1" "$2")shift 2;;diff --git a/benchmark/tau2/scripts/run_eval.py b/benchmark/tau2/scripts/run_eval.pyindex 5458ba61a..ea747045a 100755--- a/benchmark/tau2/scripts/run_eval.py+++ b/benchmark/tau2/scripts/run_eval.py@@ -4,27 +4,33 @@ from __future__ import annotationsimport argparseimport importlib.utilimport json+import osimport subprocessimport sys+from concurrent.futures import ThreadPoolExecutor, as_completedfrom pathlib import Pathfrom typing import Anyfrom tau2_common import (+ assert_tau2_results_complete,domains,load_config,- output_dir,normalize_litellm_env,+ output_dir,+ resolve_path,run_id,simulator_policy_report,split_file,strategy_ids,- tau2_cli,tau2_context,tau2_repo,user_simulator_policy,write_json,)+TRAIN_TRANSCRIPT_OPENVIKING_TEXT = "openviking_text"+DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS = 5000+def _reward(sim: dict[str, Any]) -> float:info = sim.get("reward_info") or {}@@ -46,8 +52,190 @@ def _db_match(sim: dict[str, Any]) -> bool | None:return sim.get("db_match")+def _strategy_int(+ config: dict[str, Any],+ strategy: dict[str, Any],+ key: str,+ *,+ fallback_key: str | None = None,+ default: int = 4,+) -> int:+ openviking = config.get("openviking", {})+ value = strategy.get(key)+ if value is None:+ value = openviking.get(key)+ if value is None and fallback_key:+ value = strategy.get(fallback_key)+ if value is None and fallback_key:+ value = openviking.get(fallback_key)+ if value is None:+ value = default+ return int(value)+++def _strategy_optional_int(+ config: dict[str, Any],+ strategy: dict[str, Any],+ key: str,+ *,+ fallback_key: str | None = None,+) -> int | None:+ openviking = config.get("openviking", {})+ value = strategy.get(key)+ if value is None:+ value = openviking.get(key)+ if value is None and fallback_key:+ value = strategy.get(fallback_key)+ if value is None and fallback_key:+ value = openviking.get(fallback_key)+ if value is None:+ return None+ return int(value)+++def _enabled(value: Any) -> bool:+ if isinstance(value, bool):+ return value+ if value is None:+ return False+ return str(value).strip().lower() in {"1", "true", "yes", "on"}+++def _require_fixed_first_user(config: dict[str, Any]) -> bool:+ return _enabled(config.get("eval", {}).get("require_fixed_first_user"))+++def _retrieval_budget(config: dict[str, Any], strategy: dict[str, Any]) -> dict[str, int | None]:+ retrieval_top_k = _strategy_int(config, strategy, "retrieval_top_k", default=4)+ first_user_retrieval_top_k = _strategy_int(+ config,+ strategy,+ "first_user_retrieval_top_k",+ fallback_key="retrieval_top_k",+ default=retrieval_top_k,+ )+ first_user_inject_top_k = _strategy_int(+ config,+ strategy,+ "first_user_inject_top_k",+ fallback_key="first_user_retrieval_top_k",+ default=first_user_retrieval_top_k,+ )+ prewrite_retrieval_top_k = _strategy_int(+ config,+ strategy,+ "prewrite_retrieval_top_k",+ fallback_key="retrieval_top_k",+ default=retrieval_top_k,+ )+ prewrite_inject_top_k = _strategy_int(+ config,+ strategy,+ "prewrite_inject_top_k",+ fallback_key="prewrite_retrieval_top_k",+ default=prewrite_retrieval_top_k,+ )+ memory_inject_max_chars = _strategy_optional_int(+ config,+ strategy,+ "memory_inject_max_chars",+ )+ first_user_memory_inject_max_chars = _strategy_optional_int(+ config,+ strategy,+ "first_user_memory_inject_max_chars",+ fallback_key="memory_inject_max_chars",+ )+ prewrite_memory_inject_max_chars = _strategy_optional_int(+ config,+ strategy,+ "prewrite_memory_inject_max_chars",+ fallback_key="memory_inject_max_chars",+ )+ for key, value in {+ "memory_inject_max_chars": memory_inject_max_chars,+ "first_user_memory_inject_max_chars": first_user_memory_inject_max_chars,+ "prewrite_memory_inject_max_chars": prewrite_memory_inject_max_chars,+ }.items():+ if value is not None and value < 0:+ raise ValueError(f"{strategy['id']} has negative {key}: {value}")+ return {+ "retrieval_top_k": retrieval_top_k,+ "first_user_retrieval_top_k": first_user_retrieval_top_k,+ "first_user_inject_top_k": first_user_inject_top_k,+ "prewrite_retrieval_top_k": prewrite_retrieval_top_k,+ "prewrite_inject_top_k": prewrite_inject_top_k,+ "memory_inject_max_chars": memory_inject_max_chars,+ "first_user_memory_inject_max_chars": first_user_memory_inject_max_chars,+ "prewrite_memory_inject_max_chars": prewrite_memory_inject_max_chars,+ }+++def _memory_corpus_key_for(+ *,+ domain: str,+ strategy: dict[str, Any],+ train_num_tasks: int | None,+) -> str:+ corpus_id = str(strategy.get("corpus_id") or strategy["id"])+ raw_key = strategy.get("corpus_cache_key")+ if raw_key:+ key = str(raw_key).format(+ domain=domain,+ strategy_id=strategy["id"],+ corpus_id=corpus_id,+ )+ else:+ key = f"{domain}_{corpus_id}"+ if train_num_tasks is not None:+ key = f"{key}_train{train_num_tasks}"+ return key+++def _memory_corpus_dir(config: dict[str, Any], configured_run_id: str, corpus_key: str) -> Path:+ raw = config.get("paths", {}).get("corpus_cache_dir")+ if raw:+ return resolve_path(str(raw)) / corpus_key+ return output_dir(config, configured_run_id) / "memory_corpora" / corpus_key+++def _search_uri(agent_id: str, search_memory_type: str) -> str:+ return f"viking://agent/{agent_id}/memories/{search_memory_type}"+++def _train_transcript_format(strategy: dict[str, Any]) -> str:+ return str(strategy.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT)+++def _train_tool_output_max_chars(strategy: dict[str, Any]) -> int:+ raw = strategy.get("train_tool_output_max_chars")+ if raw is None:+ return DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS+ return int(raw)+++def _train_skip_failed_sessions(strategy: dict[str, Any]) -> bool:+ return _enabled(strategy.get("train_skip_failed_sessions"))+++def _manifest_openviking_identity(corpus_dir: Path) -> dict[str, str] | None:+ manifest_path = corpus_dir / "corpus_manifest.json"+ if not manifest_path.is_file():+ return None+ try:+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))+ except json.JSONDecodeError:+ return None+ openviking = manifest.get("openviking") or {}+ required = ("account", "user", "agent_id", "search_uri")+ if not all(openviking.get(key) for key in required):+ return None+ return {key: str(openviking[key]) for key in required}++def _metrics_from_tau2_results(results_path: Path) -> dict[str, Any]:data = json.loads(results_path.read_text(encoding="utf-8"))+ assert_tau2_results_complete(data, context=str(results_path))sims = data.get("simulations") or []rewards = [_reward(sim) for sim in sims]db_values = [_db_match(sim) for sim in sims]@@ -55,7 +243,9 @@ def _metrics_from_tau2_results(results_path: Path) -> dict[str, Any]:return {"simulation_count": len(sims),"avg_reward": sum(rewards) / len(rewards) if rewards else 0.0,- "db_match_rate": (sum(1 for value in db_known if value) / len(db_known)) if db_known else None,+ "db_match_rate": (sum(1 for value in db_known if value) / len(db_known))+ if db_known+ else None,}@@ -80,6 +270,8 @@ def _tau2_command(if reasoning_effort:agent_llm_args = f'{{"temperature":0.0,"reasoning_effort":"{reasoning_effort}"}}'user_llm_args = f'{{"temperature":0.0,"reasoning_effort":"{reasoning_effort}"}}'+ fixed_first_user_file = _fixed_first_user_file(config, domain)+ scope_prompt_file = _scope_prompt_file(config, strategy, domain)if (strategy.get("memory_backend") == "openviking"@@ -87,10 +279,39 @@ def _tau2_command():openviking = config["openviking"]corpus_id = str(strategy.get("corpus_id") or strategy["id"])- account = f"{openviking['account']}-{configured_run_id}-{domain}-{corpus_id}"- agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"- user = f"tau2-{domain}-{corpus_id}"- search_uri = f"viking://agent/{agent_id}/memories/experiences"+ resolved_train_num_tasks = (+ train_num_tasks if train_num_tasks is not None else strategy.get("train_num_tasks")+ )+ corpus_key = _memory_corpus_key_for(+ domain=domain,+ strategy=strategy,+ train_num_tasks=resolved_train_num_tasks,+ )+ corpus_dir = _memory_corpus_dir(config, configured_run_id, corpus_key)+ reuse_identity = _manifest_openviking_identity(corpus_dir)+ if reuse_identity is not None:+ account = reuse_identity["account"]+ agent_id = reuse_identity["agent_id"]+ user = reuse_identity["user"]+ search_uri = ""+ elif openviking.get("reuse_corpus_across_runs", False):+ account = f"{openviking['account']}-{corpus_key}"+ agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"+ user = f"tau2-{domain}-{corpus_id}"+ search_uri = ""+ else:+ account = f"{openviking['account']}-{configured_run_id}-{domain}-{corpus_id}"+ agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"+ user = f"tau2-{domain}-{corpus_id}"+ search_uri = ""+ search_memory_type = str(strategy.get("search_memory_type", "experiences"))+ if search_memory_type not in {"experiences", "trajectories"}:+ raise ValueError(+ f"Unsupported search_memory_type for {strategy['id']}: {search_memory_type}"+ )+ if not search_uri:+ search_uri = _search_uri(agent_id, search_memory_type)+ budget = _retrieval_budget(config, strategy)command = [sys.executable,str(Path(__file__).with_name("run_memory_v2_eval.py")),@@ -99,11 +320,7 @@ def _tau2_command("--run-dir",str(output_dir(config, configured_run_id) / "memory_cells" / run_label),"--corpus-dir",- str(- output_dir(config, configured_run_id)- / "memory_corpora"- / f"{domain}_{corpus_id}"- ),+ str(corpus_dir),"--run-label",run_label,"--strategy-id",@@ -137,64 +354,135 @@ def _tau2_command("--search-uri",search_uri,"--retrieval-top-k",- str(openviking.get("retrieval_top_k", 4)),+ str(budget["retrieval_top_k"]),+ "--first-user-retrieval-top-k",+ str(budget["first_user_retrieval_top_k"]),+ "--first-user-inject-top-k",+ str(budget["first_user_inject_top_k"]),+ "--prewrite-retrieval-top-k",+ str(budget["prewrite_retrieval_top_k"]),+ "--prewrite-inject-top-k",+ str(budget["prewrite_inject_top_k"]),"--retrieval-mode",str(strategy.get("retrieval_mode", "first_user")),+ "--train-transcript-format",+ _train_transcript_format(strategy),+ "--train-tool-output-max-chars",+ str(_train_tool_output_max_chars(strategy)),"--seed",str(seed),]+ if budget["memory_inject_max_chars"] is not None:+ command.extend(["--memory-inject-max-chars", str(budget["memory_inject_max_chars"])])+ if budget["first_user_memory_inject_max_chars"] is not None:+ command.extend(+ [+ "--first-user-memory-inject-max-chars",+ str(budget["first_user_memory_inject_max_chars"]),+ ]+ )+ if budget["prewrite_memory_inject_max_chars"] is not None:+ command.extend(+ [+ "--prewrite-memory-inject-max-chars",+ str(budget["prewrite_memory_inject_max_chars"]),+ ]+ )+ if _enabled(strategy.get("train_include_system_prompt")):+ command.append("--train-include-system-prompt")+ if _train_skip_failed_sessions(strategy):+ command.append("--train-skip-failed-sessions")+ if fixed_first_user_file is not None:+ command.extend(["--fixed-first-user-file", str(fixed_first_user_file)])+ if scope_prompt_file is not None:+ command.extend(["--scope-prompt-file", str(scope_prompt_file)])if task_ids:for task_id in task_ids:command.extend(["--task-id", task_id])elif num_tasks is not None:command.extend(["--num-tasks", str(num_tasks)])- train_num_tasks = train_num_tasks if train_num_tasks is not None else strategy.get("train_num_tasks")- if train_num_tasks is not None:- command.extend(["--train-num-tasks", str(train_num_tasks)])+ if resolved_train_num_tasks is not None:+ command.extend(["--train-num-tasks", str(resolved_train_num_tasks)])return commandif strategy.get("memory_backend") != "none":return Nonecommand = [- tau2_cli(config),- "run",+ sys.executable,+ str(Path(__file__).with_name("run_memory_v2_eval.py")),+ "--tau2-repo",+ str(tau2_repo(config)),+ "--run-dir",+ str(output_dir(config, configured_run_id) / "memory_cells" / run_label),+ "--run-label",+ run_label,+ "--strategy-id",+ strategy["id"],"--domain",domain,- "--agent",- str(benchmark.get("agent", "llm_agent")),- "--user",- str(benchmark.get("user", "user_simulator")),- "--task-split-name",+ "--eval-split-name",str(benchmark.get("eval_split_name", "test")),- "--num-trials",- "1","--max-steps",str(benchmark.get("max_steps", 200)),"--max-concurrency",str(benchmark.get("task_max_concurrency", 10)),+ "--base-agent",+ str(benchmark.get("agent", "llm_agent")),+ "--user",+ str(benchmark.get("user", "user_simulator")),"--agent-llm",str(model["agent_llm"]),"--user-llm",str(model["user_llm"]),- "--save-to",- run_label,+ "--agent-llm-args",+ agent_llm_args,+ "--user-llm-args",+ user_llm_args,"--seed",str(seed),+ "--no-memory",]-- command.extend(["--agent-llm-args", agent_llm_args])- command.extend(["--user-llm-args", user_llm_args])+ if fixed_first_user_file is not None:+ command.extend(["--fixed-first-user-file", str(fixed_first_user_file)])if task_ids:- command.append("--task-ids")- command.extend(task_ids)+ for task_id in task_ids:+ command.extend(["--task-id", task_id])elif num_tasks is not None:command.extend(["--num-tasks", str(num_tasks)])return command+def _fixed_first_user_file(config: dict[str, Any], domain: str) -> Path | None:+ raw = config.get("eval", {}).get("fixed_first_user_fixture")+ if raw is None:+ raw = config.get("eval", {}).get("fixed_first_user_fixtures")+ if isinstance(raw, dict):+ raw = raw.get(domain) or raw.get("default")+ if raw is None or str(raw).strip() == "":+ return None+ return resolve_path(str(raw))+++def _scope_prompt_file(+ config: dict[str, Any], strategy: dict[str, Any], domain: str+) -> Path | None:+ raw = strategy.get("scope_prompt_file")+ if raw is None:+ raw = strategy.get("scope_prompt_files")+ if raw is None:+ raw = config.get("openviking", {}).get("scope_prompt_file")+ if raw is None:+ raw = config.get("openviking", {}).get("scope_prompt_files")+ if isinstance(raw, dict):+ raw = raw.get(domain) or raw.get("default")+ if raw is None or str(raw).strip() == "":+ return None+ return resolve_path(str(raw))++def _build_plan(config: dict[str, Any],configured_run_id: str,@@ -205,16 +493,30 @@ def _build_plan(num_tasks: int | None,train_num_tasks: int | None,repeat_count_override: int | None,+ cell_concurrency_override: int | None,+ strategy_concurrency_override: int | None,) -> dict[str, Any]:repeat_count = repeat_count_override or int(config["benchmark"].get("repeat_count", 8))base_seed = int(config["benchmark"].get("seed", 300))+ cell_timeout_seconds = int(config["benchmark"].get("cell_timeout_seconds", 0) or 0)+ strategy_concurrency = strategy_concurrency_override+ if strategy_concurrency is None:+ strategy_concurrency = cell_concurrency_override+ if strategy_concurrency is None:+ strategy_concurrency = config["benchmark"].get("strategy_concurrency")+ if strategy_concurrency is None:+ strategy_concurrency = config["benchmark"].get("cell_concurrency", 1)+ strategy_concurrency = max(1, int(strategy_concurrency or 1))policy_report = simulator_policy_report(config)+ require_fixed_first_user = _require_fixed_first_user(config)strategies = config.get("strategies") or []if selected_strategy_ids:unknown = selected_strategy_ids - set(strategy_ids(config))if unknown:raise ValueError(f"unknown strategy ids: {sorted(unknown)}")- strategies = [strategy for strategy in strategies if strategy["id"] in selected_strategy_ids]+ strategies = [+ strategy for strategy in strategies if strategy["id"] in selected_strategy_ids+ ]cells = []plan_domains = domains(config)if selected_domains:@@ -222,6 +524,18 @@ def _build_plan(if unknown_domains:raise ValueError(f"unknown domains: {sorted(unknown_domains)}")plan_domains = [domain for domain in plan_domains if domain in selected_domains]+ if require_fixed_first_user:+ missing_domains = [+ domain for domain in plan_domains if _fixed_first_user_file(config, domain) is None+ ]+ if missing_domains:+ protocol = config.get("eval", {}).get("protocol", "fixed_first_user_full8")+ raise ValueError(+ f"eval protocol {protocol!r} requires fixed-first-user fixtures for "+ f"{missing_domains}; set TAU2_RETAIL_FIXED_FIRST_USER_FILE and "+ "TAU2_AIRLINE_FIXED_FIRST_USER_FILE, or use a config that explicitly "+ "sets eval.require_fixed_first_user=false"+ )for domain in plan_domains:split_path = split_file(config, domain)for strategy in strategies:@@ -239,6 +553,8 @@ def _build_plan(train_num_tasks=train_num_tasks,seed=seed,)+ fixed_first_user_file = _fixed_first_user_file(config, domain)+ scope_prompt_file = _scope_prompt_file(config, strategy, domain)non_executable_reason = Noneif command is None:non_executable_reason = (@@ -256,11 +572,47 @@ def _build_plan("train_required": bool(strategy.get("train_required")),"memory_backend": strategy.get("memory_backend"),"corpus_id": strategy.get("corpus_id", strategy["id"]),+ "corpus_key": _memory_corpus_key_for(+ domain=domain,+ strategy=strategy,+ train_num_tasks=(+ train_num_tasks+ if train_num_tasks is not None+ else strategy.get("train_num_tasks")+ ),+ ),+ "corpus_dir": str(+ _memory_corpus_dir(+ config,+ configured_run_id,+ _memory_corpus_key_for(+ domain=domain,+ strategy=strategy,+ train_num_tasks=(+ train_num_tasks+ if train_num_tasks is not None+ else strategy.get("train_num_tasks")+ ),+ ),+ )+ ),"retrieval_mode": strategy.get("retrieval_mode"),+ "train_transcript_format": _train_transcript_format(strategy),+ "train_include_system_prompt": _enabled(+ strategy.get("train_include_system_prompt")+ ),+ "train_skip_failed_sessions": _train_skip_failed_sessions(strategy),+ "train_tool_output_max_chars": _train_tool_output_max_chars(strategy),+ "retrieval_budget": _retrieval_budget(config, strategy),+ "search_memory_type": strategy.get("search_memory_type", "experiences"),"adapter_status": strategy.get("adapter_status", "ready"),"executable": command is not None,"user_simulator_policy": user_simulator_policy(config),"user_simulator_policy_supported": policy_report["supported"],+ "fixed_first_user_file": str(fixed_first_user_file)+ if fixed_first_user_file+ else None,+ "scope_prompt_file": str(scope_prompt_file) if scope_prompt_file else None,"split_file": str(split_path),"command": command,"non_executable_reason": non_executable_reason,@@ -274,35 +626,39 @@ def _build_plan("strategy_ids": strategy_ids(config),"domains": plan_domains,"tau2": tau2_context(config),+ "eval_protocol": config.get("eval", {}).get("protocol"),+ "require_fixed_first_user": require_fixed_first_user,"simulator_policy": policy_report,"cell_count": len(cells),"executable_cell_count": executable_cell_count,"pending_cell_count": len(cells) - executable_cell_count,+ "corpus_prepare_concurrency": int(config["benchmark"].get("corpus_prepare_concurrency", 1)),+ "strategy_concurrency": strategy_concurrency,+ "cell_concurrency": strategy_concurrency,+ "cell_timeout_seconds": cell_timeout_seconds or None,"cells": cells,}def _cell_artifacts(cell: dict[str, Any], repo: Path, out: Path) -> dict[str, str]:- if cell.get("memory_backend") == "openviking":+ if cell.get("memory_backend") in {"openviking", "none"}:run_dir = out / "memory_cells" / cell["run_label"]- corpus_id = str(cell.get("corpus_id") or cell["strategy_id"])- corpus_dir = out / "memory_corpora" / f"{cell['domain']}_{corpus_id}"- return {+ artifacts = {"summary": str(run_dir / f"{cell['run_label']}.summary.json"),"results": str(run_dir / f"{cell['run_label']}.json"),- "retrieval_trace": str(run_dir / f"{cell['run_label']}.retrieval_trace.jsonl"),- "corpus_manifest": str(corpus_dir / "corpus_manifest.json"),}- return {- "results": str(repo / "data" / "simulations" / f"{cell['run_label']}.json")- }+ if cell.get("memory_backend") == "none":+ return artifacts+ corpus_dir = Path(cell["corpus_dir"])+ artifacts["retrieval_trace"] = str(run_dir / f"{cell['run_label']}.retrieval_trace.jsonl")+ artifacts["corpus_manifest"] = str(corpus_dir / "corpus_manifest.json")+ return artifacts+ return {"results": str(repo / "data" / "simulations" / f"{cell['run_label']}.json")}def _cell_metrics(cell: dict[str, Any], artifacts: dict[str, str]) -> dict[str, Any] | None:- if cell.get("memory_backend") == "openviking":- summary_path = Path(artifacts["summary"])- if not summary_path.is_file():- return None+ summary_path = Path(artifacts.get("summary", ""))+ if summary_path.is_file():summary = json.loads(summary_path.read_text(encoding="utf-8"))return summary.get("metrics")@@ -312,6 +668,129 @@ def _cell_metrics(cell: dict[str, Any], artifacts: dict[str, str]) -> dict[str,return _metrics_from_tau2_results(results_path)+def _memory_corpus_key(cell: dict[str, Any]) -> str:+ return str(cell.get("corpus_key") or f"{cell['domain']}_{cell['corpus_id']}")+++def _tau2_subprocess_env(repo: Path) -> dict[str, str]:+ env = os.environ.copy()+ src = repo / "src"+ pythonpath_entry = str(src if src.is_dir() else repo)+ existing = env.get("PYTHONPATH")+ env["PYTHONPATH"] = (+ pythonpath_entry if not existing else f"{pythonpath_entry}{os.pathsep}{existing}"+ )+ return env+++def _prepare_memory_corpus(cell: dict[str, Any], repo: Path, out: Path) -> dict[str, Any]:+ key = _memory_corpus_key(cell)+ manifest_path = Path(cell["corpus_dir"]) / "corpus_manifest.json"+ if manifest_path.is_file():+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))+ cached_transcript_format = str(+ manifest.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT+ )+ requested_transcript_format = str(+ cell.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT+ )+ if cached_transcript_format != requested_transcript_format:+ raise RuntimeError(+ "cached corpus train_transcript_format mismatch for "+ f"{key}: {cached_transcript_format!r} != {requested_transcript_format!r}; "+ "use a distinct corpus_id or rebuild the corpus"+ )+ cached_include_system_prompt = bool(manifest.get("train_include_system_prompt") or False)+ requested_include_system_prompt = bool(cell.get("train_include_system_prompt") or False)+ if cached_include_system_prompt != requested_include_system_prompt:+ raise RuntimeError(+ "cached corpus train_include_system_prompt mismatch for "+ f"{key}: {cached_include_system_prompt!r} != {requested_include_system_prompt!r}; "+ "use a distinct corpus_id or rebuild the corpus"+ )+ cached_tool_output_max_chars = int(+ manifest.get("train_tool_output_max_chars") or DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS+ )+ requested_tool_output_max_chars = int(+ cell.get("train_tool_output_max_chars") or DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS+ )+ if cached_tool_output_max_chars != requested_tool_output_max_chars:+ raise RuntimeError(+ "cached corpus train_tool_output_max_chars mismatch for "+ f"{key}: {cached_tool_output_max_chars!r} != {requested_tool_output_max_chars!r}; "+ "use a distinct corpus_id or rebuild the corpus"+ )+ cached_skip_failed = bool(manifest.get("train_skip_failed_sessions") or False)+ requested_skip_failed = bool(cell.get("train_skip_failed_sessions") or False)+ if cached_skip_failed != requested_skip_failed:+ raise RuntimeError(+ "cached corpus train_skip_failed_sessions mismatch for "+ f"{key}: {cached_skip_failed!r} != {requested_skip_failed!r}; "+ "use a distinct corpus_id or rebuild the corpus"+ )+ row = {+ "domain": cell["domain"],+ "strategy_id": cell["strategy_id"],+ "corpus_id": str(cell.get("corpus_id") or cell["strategy_id"]),+ "corpus_key": key,+ "returncode": 0,+ "reused": True,+ "artifacts": {"corpus_manifest": str(manifest_path)},+ }+ write_json(out / "corpus_prepare_results" / f"{key}.json", row)+ print(f"[tau2] reusing corpus {key}", flush=True)+ return row+ command = list(cell["command"]) + ["--prepare-corpus-only"]+ print(f"[tau2] preparing corpus {key}", flush=True)+ completed = subprocess.run(+ command,+ cwd=repo,+ env=_tau2_subprocess_env(repo),+ text=True,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ check=False,+ )+ row = {+ "domain": cell["domain"],+ "strategy_id": cell["strategy_id"],+ "corpus_id": str(cell.get("corpus_id") or cell["strategy_id"]),+ "corpus_key": key,+ "returncode": completed.returncode,+ "stdout_tail": completed.stdout[-4000:],+ "stderr_tail": completed.stderr[-4000:],+ "artifacts": {"corpus_manifest": str(Path(cell["corpus_dir"]) / "corpus_manifest.json")},+ }+ write_json(out / "corpus_prepare_results" / f"{key}.json", row)+ if completed.returncode != 0:+ raise RuntimeError(f"corpus prepare failed: {key} returncode={completed.returncode}")+ return row+++def _prepare_memory_corpora(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str, Any]]:+ corpus_cells: dict[str, dict[str, Any]] = {}+ for cell in plan["cells"]:+ if cell.get("memory_backend") != "openviking" or not cell.get("train_required"):+ continue+ corpus_cells.setdefault(_memory_corpus_key(cell), cell)+ if not corpus_cells:+ return []++ worker_count = max(1, int(plan.get("corpus_prepare_concurrency") or 1))+ if worker_count == 1 or len(corpus_cells) == 1:+ return [_prepare_memory_corpus(cell, repo, out) for cell in corpus_cells.values()]++ rows: list[dict[str, Any]] = []+ with ThreadPoolExecutor(max_workers=worker_count) as executor:+ futures = {+ executor.submit(_prepare_memory_corpus, cell, repo, out): key+ for key, cell in corpus_cells.items()+ }+ for future in as_completed(futures):+ rows.append(future.result())+ return rows++def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:def weighted(rows_for_group: list[dict[str, Any]]) -> dict[str, Any]:metric_rows = [row for row in rows_for_group if row.get("metrics")]@@ -327,7 +806,9 @@ def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:if row["metrics"].get("db_match_rate") is not Noneand int(row["metrics"].get("simulation_count") or 0) > 0]- db_weight = sum(int(row["metrics"].get("simulation_count") or 0) for row in db_weighted_rows)+ db_weight = sum(+ int(row["metrics"].get("simulation_count") or 0) for row in db_weighted_rows+ )db_sum = sum(float(row["metrics"]["db_match_rate"])* int(row["metrics"].get("simulation_count") or 0)@@ -367,6 +848,64 @@ def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:}+def _execute_cell(+ cell: dict[str, Any], repo: Path, out: Path, cell_timeout: int | None+) -> dict[str, Any]:+ cell_result_path = out / "cell_results" / f"{cell['run_label']}.json"+ if cell_result_path.is_file():+ existing_row = json.loads(cell_result_path.read_text(encoding="utf-8"))+ if existing_row.get("returncode") == 0 and existing_row.get("metrics"):+ print(f"[tau2] skipping completed {cell['run_label']}", flush=True)+ return existing_row++ print(f"[tau2] running {cell['run_label']}", flush=True)+ try:+ completed = subprocess.run(+ cell["command"],+ cwd=repo,+ env=_tau2_subprocess_env(repo),+ text=True,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ check=False,+ timeout=cell_timeout,+ )+ except subprocess.TimeoutExpired as exc:+ stdout = exc.stdout or ""+ stderr = exc.stderr or ""+ if isinstance(stdout, bytes):+ stdout = stdout.decode(errors="replace")+ if isinstance(stderr, bytes):+ stderr = stderr.decode(errors="replace")+ row = {+ "run_label": cell["run_label"],+ "domain": cell["domain"],+ "strategy_id": cell["strategy_id"],+ "returncode": 124,+ "timed_out": True,+ "timeout_seconds": cell_timeout,+ "stdout_tail": stdout[-4000:],+ "stderr_tail": stderr[-4000:],+ "artifacts": _cell_artifacts(cell, repo, out),+ "metrics": None,+ }+ write_json(cell_result_path, row)+ return row++ row = {+ "run_label": cell["run_label"],+ "domain": cell["domain"],+ "strategy_id": cell["strategy_id"],+ "returncode": completed.returncode,+ "stdout_tail": completed.stdout[-4000:],+ "stderr_tail": completed.stderr[-4000:],+ }+ row["artifacts"] = _cell_artifacts(cell, repo, out)+ row["metrics"] = _cell_metrics(cell, row["artifacts"])+ write_json(cell_result_path, row)+ return row++def _execute_cells(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str, Any]]:policy_report = plan.get("simulator_policy") or {}if not policy_report.get("supported", False):@@ -374,39 +913,42 @@ def _execute_cells(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str"configured user simulator policy is not supported by this TAU-2 checkout: "f"{policy_report}")- rows = []+ _prepare_memory_corpora(plan, repo, out)+ cells = []for cell in plan["cells"]:if not cell.get("executable"):raise RuntimeError(f"cell is not executable yet: {cell['run_label']} "f"(strategy_id={cell['strategy_id']}, adapter_status={cell.get('adapter_status')})")- print(f"[tau2] running {cell['run_label']}")- completed = subprocess.run(- cell["command"],- cwd=repo,- text=True,- stdout=subprocess.PIPE,- stderr=subprocess.PIPE,- check=False,- )- row = {- "run_label": cell["run_label"],- "domain": cell["domain"],- "strategy_id": cell["strategy_id"],- "returncode": completed.returncode,- "stdout_tail": completed.stdout[-4000:],- "stderr_tail": completed.stderr[-4000:],+ cells.append(cell)++ cell_timeout = int(plan.get("cell_timeout_seconds") or 0) or None+ worker_count = max(+ 1, int(plan.get("strategy_concurrency") or plan.get("cell_concurrency") or 1)+ )+ if worker_count == 1 or len(cells) == 1:+ return [_execute_cell(cell, repo, out, cell_timeout) for cell in cells]++ print(f"[tau2] running eval cells with concurrency={worker_count}", flush=True)+ rows: list[dict[str, Any]] = []+ with ThreadPoolExecutor(max_workers=worker_count) as executor:+ futures = {+ executor.submit(_execute_cell, cell, repo, out, cell_timeout): cell for cell in cells}- row["artifacts"] = _cell_artifacts(cell, repo, out)- row["metrics"] = _cell_metrics(cell, row["artifacts"])- rows.append(row)- write_json(out / "cell_results" / f"{cell['run_label']}.json", row)- if completed.returncode != 0:- raise RuntimeError(f"cell failed: {cell['run_label']} returncode={completed.returncode}")+ for future in as_completed(futures):+ rows.append(future.result())return rows+def _execution_failures(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:+ return [+ row+ for row in rows+ if row.get("returncode") != 0 or row.get("timed_out") or not row.get("metrics")+ ]++def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:errors: list[str] = []llm_env = normalize_litellm_env()@@ -419,19 +961,36 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:if strict and not llm_env["has_api_key"]:errors.append("missing LLM API key: set OPENAI_API_KEY or ARK_API_KEY")if strict and not llm_env["has_base_url"]:- errors.append("missing OpenAI-compatible base URL: set OPENAI_API_BASE, OPENAI_BASE_URL, or ARK_BASE_URL")+ errors.append(+ "missing OpenAI-compatible base URL: set OPENAI_API_BASE, OPENAI_BASE_URL, or ARK_BASE_URL"+ )if strict and not policy_report["supported"]:errors.append("configured confirmation-aware user simulator policy requires a TAU-2 "f"checkout with the prompt fix: {policy_report['prompt_files']}")split_rows = []+ fixture_rows = []for domain in domains(config):path = split_file(config, domain)exists = path.is_file()split_rows.append({"domain": domain, "path": str(path), "exists": exists})if strict and not exists:errors.append(f"missing split file for {domain}: {path}")+ fixed_first_user_file = _fixed_first_user_file(config, domain)+ fixture_exists = fixed_first_user_file.is_file() if fixed_first_user_file else False+ fixture_rows.append(+ {+ "domain": domain,+ "path": str(fixed_first_user_file) if fixed_first_user_file else None,+ "exists": fixture_exists,+ }+ )+ if strict and _require_fixed_first_user(config) and not fixture_exists:+ errors.append(+ "missing fixed-first-user fixture for "+ f"{domain}: set TAU2_{domain.upper()}_FIXED_FIRST_USER_FILE"+ )import_rows = []for module in ("openviking", "openviking_cli", "tau2"):@@ -444,12 +1003,15 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:"status": "failed" if errors else "ok","strict": strict,"tau2": tau2_info,+ "eval_protocol": config.get("eval", {}).get("protocol"),+ "require_fixed_first_user": _require_fixed_first_user(config),"llm_env": llm_env,"simulator_policy": policy_report,"domains": domains(config),"strategies": strategy_ids(config),"imports": import_rows,"split_files": split_rows,+ "fixed_first_user_fixtures": fixture_rows,"errors": errors,}write_json(out / "preflight.json", report)@@ -463,14 +1025,38 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:def main() -> int:parser = argparse.ArgumentParser(description="Plan or run TAU-2 benchmark cells.")- parser.add_argument("--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml")+ parser.add_argument(+ "--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml"+ )parser.add_argument("--run-id", default=run_id())- parser.add_argument("--domain", action="append", help="Run only this configured domain; may be repeated.")- parser.add_argument("--repeat-count", type=int, help="Override benchmark.repeat_count for smoke runs.")- parser.add_argument("--strategy-id", action="append", help="Run only this strategy id; may be repeated.")- parser.add_argument("--task-id", action="append", help="Run only this TAU-2 task id; may be repeated.")- parser.add_argument("--num-tasks", type=int, help="Run the first N tasks from the selected split.")- parser.add_argument("--train-num-tasks", type=int, help="Train OpenViking memory on the first N train tasks.")+ parser.add_argument(+ "--domain", action="append", help="Run only this configured domain; may be repeated."+ )+ parser.add_argument(+ "--repeat-count", type=int, help="Override benchmark.repeat_count for smoke runs."+ )+ parser.add_argument(+ "--cell-concurrency",+ type=int,+ help="Deprecated alias for --strategy-concurrency.",+ )+ parser.add_argument(+ "--strategy-concurrency",+ type=int,+ help="Override benchmark.strategy_concurrency for parallel matrix cells.",+ )+ parser.add_argument(+ "--strategy-id", action="append", help="Run only this strategy id; may be repeated."+ )+ parser.add_argument(+ "--task-id", action="append", help="Run only this TAU-2 task id; may be repeated."+ )+ parser.add_argument(+ "--num-tasks", type=int, help="Run the first N tasks from the selected split."+ )+ parser.add_argument(+ "--train-num-tasks", type=int, help="Train OpenViking memory on the first N train tasks."+ )parser.add_argument("--preflight",action="store_true",@@ -488,6 +1074,10 @@ def main() -> int:if args.plan_only and args.execute:raise SystemExit("--plan-only and --execute are mutually exclusive")+ if args.cell_concurrency is not None and args.cell_concurrency < 1:+ raise SystemExit("--cell-concurrency must be >= 1")+ if args.strategy_concurrency is not None and args.strategy_concurrency < 1:+ raise SystemExit("--strategy-concurrency must be >= 1")config = load_config(args.config)out = output_dir(config, args.run_id)@@ -506,6 +1096,8 @@ def main() -> int:num_tasks=args.num_tasks,train_num_tasks=args.train_num_tasks,repeat_count_override=args.repeat_count,+ cell_concurrency_override=args.cell_concurrency,+ strategy_concurrency_override=args.strategy_concurrency,)write_json(out / "run_plan.json", plan)write_json(out / "resolved_config.json", config)@@ -514,10 +1106,15 @@ def main() -> int:if args.execute:try:rows = _execute_cells(plan, tau2_repo(config), out)- plan["status"] = "succeeded"+ failures = _execution_failures(rows)+ plan["status"] = "failed" if failures else "succeeded"plan["executed_cell_count"] = len(rows)+ plan["failed_cell_count"] = len(failures)write_json(out / "run_plan.json", plan)write_json(out / "scoreboard.json", _summarize(rows))+ if failures:+ labels = ", ".join(str(row.get("run_label")) for row in failures[:5])+ raise RuntimeError(f"{len(failures)} cell(s) failed or incomplete: {labels}")except Exception as exc:plan["status"] = "failed"plan["error"] = str(exc)diff --git a/benchmark/tau2/scripts/run_memory_v2_eval.py b/benchmark/tau2/scripts/run_memory_v2_eval.pyindex de5ef5441..26b2d69e7 100644--- a/benchmark/tau2/scripts/run_memory_v2_eval.py+++ b/benchmark/tau2/scripts/run_memory_v2_eval.py@@ -2,15 +2,17 @@from __future__ import annotationsimport argparse+import hashlib+import importlibimport jsonimport shutilimport sysimport time+from copy import deepcopyfrom pathlib import Pathfrom typing import Any-from tau2_common import normalize_litellm_env-+from tau2_common import assert_tau2_results_complete, normalize_litellm_envAGENT_NAME = "openviking_memory_agent"REPO_ROOT = Path(__file__).resolve().parents[3]@@ -29,6 +31,11 @@ WRITE_TOOL_PREFIXES = ("grant_","reboot_",)+FIXED_FIRST_USER_NAME = "openviking_fixed_first_user_simulator"+TRAIN_TRANSCRIPT_OPENVIKING_TEXT = "openviking_text"+TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS = "role_tool_blocks"+TRAIN_TRANSCRIPT_CUSTOM_LIKE = "custom_like"+DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS = 5000def _json(text: str) -> dict[str, Any]:@@ -46,6 +53,51 @@ def _add_tau2_to_path(tau2_repo: Path) -> None:sys.path.insert(0, str(src if src.is_dir() else tau2_repo))+def _load_domain_policy(tau2_repo: Path, domain: str) -> str:+ path = tau2_repo / "data" / "tau2" / "domains" / domain / "policy.md"+ if not path.is_file():+ return ""+ return path.read_text(encoding="utf-8").strip()+++def _build_system_prompt(policy: str) -> str:+ return (+ "You are a customer service representative for a benchmark evaluation.\n"+ "Help customers strictly according to the policy below.\n\n"+ "You have two kinds of tools:\n"+ "1. Memory/context tools available to the benchmark agent.\n"+ "2. Business system tools executed by the TAU-2 environment.\n\n"+ "Rules for business system tools:\n"+ "- Call ONE tool at a time. Wait for the result before calling the next tool.\n"+ "- Only pass arguments the customer explicitly provided. Do NOT infer or add optional arguments.\n"+ "- After receiving a tool result, use it to continue helping the customer.\n\n"+ "## Policy\n"+ f"{policy}\n\n"+ "Now wait for the customer to start. Follow the policy exactly."+ )+++def _patch_tau2_auxiliary_llm_defaults(llm: str, llm_args: dict[str, Any]) -> None:+ # TAU-2 exposes agent/user LLMs in TextRunConfig, but NL assertion scoring+ # still reads module defaults. Keep the evaluator on the same configured+ # model so benchmark runs do not fall back to inaccessible upstream defaults.+ patches = {+ "DEFAULT_LLM_NL_ASSERTIONS": llm,+ "DEFAULT_LLM_NL_ASSERTIONS_ARGS": deepcopy(llm_args),+ "DEFAULT_LLM_ENV_INTERFACE": llm,+ "DEFAULT_LLM_ENV_INTERFACE_ARGS": deepcopy(llm_args),+ }+ for module_name in (+ "tau2.config",+ "tau2.evaluator.evaluator_nl_assertions",+ "tau2.environment.utils.interface_agent",+ ):+ module = importlib.import_module(module_name)+ for name, value in patches.items():+ if hasattr(module, name):+ setattr(module, name, deepcopy(value))++def _save_to_arg(path: Path) -> str:# Some TAU-2 versions append ".json"; newer versions treat save_to as a# run directory and write results.json under it.@@ -77,6 +129,10 @@ def _db_match(sim: dict[str, Any]) -> bool | None:return sim.get("db_match")+def _task_success(sim: dict[str, Any]) -> bool:+ return _reward(sim) >= 1.0++def _metrics(results_path: Path) -> dict[str, Any]:data = json.loads(results_path.read_text())sims = data.get("simulations") or []@@ -86,7 +142,9 @@ def _metrics(results_path: Path) -> dict[str, Any]:return {"simulation_count": len(sims),"avg_reward": sum(rewards) / len(rewards) if rewards else 0.0,- "db_match_rate": (sum(1 for value in db_known if value) / len(db_known)) if db_known else None,+ "db_match_rate": (sum(1 for value in db_known if value) / len(db_known))+ if db_known+ else None,}@@ -118,12 +176,14 @@ def _tool_call_query(tool_calls: list[Any], state_messages: list[Any]) -> str:recent_user = [str(getattr(message, "content", "") or "")for message in state_messages[-8:]- if str(getattr(message, "role", "")) == "user" and str(getattr(message, "content", "") or "").strip()+ if str(getattr(message, "role", "")) == "user"+ and str(getattr(message, "content", "") or "").strip()]recent_observations = [str(getattr(message, "content", "") or "")[:600]for message in state_messages[-12:]- if str(getattr(message, "role", "")) == "tool" and str(getattr(message, "content", "") or "").strip()+ if str(getattr(message, "role", "")) == "tool"+ and str(getattr(message, "content", "") or "").strip()]parts = ["Before executing write-like tool call(s): " + "; ".join(rendered),@@ -134,7 +194,26 @@ def _tool_call_query(tool_calls: list[Any], state_messages: list[Any]) -> str:return "\n".join(parts)-def _message_text(message: dict[str, Any]) -> tuple[str, str]:+def _tool_call_id(tool_call: dict[str, Any]) -> str:+ return str(tool_call.get("id") or tool_call.get("tool_call_id") or "").strip()+++def _tool_result_call_id(message: dict[str, Any]) -> str:+ return str(message.get("id") or message.get("tool_call_id") or "").strip()+++def _compact_train_tool_output(content: Any, *, max_chars: int) -> str:+ text = (+ content+ if isinstance(content, str)+ else json.dumps(content, ensure_ascii=False, sort_keys=True)+ )+ if len(text) <= max_chars:+ return text+ return text[:max_chars] + f"... <truncated {len(text) - max_chars} chars>"+++def _message_text_openviking_text(message: dict[str, Any]) -> tuple[str, str]:role = str(message.get("role") or "assistant")if role == "user":return "user", str(message.get("content") or "")@@ -151,6 +230,143 @@ def _message_text(message: dict[str, Any]) -> tuple[str, str]:return "assistant", str(message.get("content") or "")+def _message_texts_role_tool_blocks(+ message: dict[str, Any],+ *,+ tool_calls_by_id: dict[str, dict[str, Any]],+ max_tool_output_chars: int,+) -> list[tuple[str, str]]:+ role = str(message.get("role") or "assistant")+ rows: list[tuple[str, str]] = []+ if role in {"user", "assistant"}:+ content = message.get("content")+ if isinstance(content, str) and content.strip():+ rows.append((role, f"{role}:\n{content}"))+ for tool_call in message.get("tool_calls") or []:+ if not isinstance(tool_call, dict):+ continue+ call_id = _tool_call_id(tool_call)+ if call_id:+ tool_calls_by_id[call_id] = tool_call+ requestor = str(tool_call.get("requestor") or role or "assistant")+ name = _tool_call_name(tool_call)+ arguments = _tool_call_arguments(tool_call)+ lines = ["tool-call:"]+ if call_id:+ lines.append(f"call_id: {call_id}")+ if name:+ lines.append(f"name: {name}")+ lines.append(+ "arguments: "+ + json.dumps(arguments, ensure_ascii=False, sort_keys=True, default=str)+ )+ rows.append((requestor, "\n".join(lines)))+ elif role == "tool":+ call_id = _tool_result_call_id(message)+ tool_call = tool_calls_by_id.get(call_id) or {}+ requestor = str(message.get("requestor") or tool_call.get("requestor") or "assistant")+ output = _compact_train_tool_output(+ message.get("content"),+ max_chars=max_tool_output_chars,+ )+ lines = ["tool-response:"]+ if call_id:+ lines.append(f"call_id: {call_id}")+ name = _tool_call_name(tool_call)+ if name:+ lines.append(f"name: {name}")+ if message.get("error"):+ lines.append("error: true")+ lines.append(f"output: {output}")+ rows.append((requestor, "\n".join(lines)))+ else:+ content = str(message.get("content") or "").strip()+ if content:+ rows.append(("assistant", f"{role}:\n{content}"))+ return rows+++def _message_texts(+ message: dict[str, Any],+ *,+ transcript_format: str,+ tool_calls_by_id: dict[str, dict[str, Any]],+ max_tool_output_chars: int,+) -> list[tuple[str, str]]:+ if transcript_format == TRAIN_TRANSCRIPT_OPENVIKING_TEXT:+ return [_message_text_openviking_text(message)]+ if transcript_format in {TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS, TRAIN_TRANSCRIPT_CUSTOM_LIKE}:+ return _message_texts_role_tool_blocks(+ message,+ tool_calls_by_id=tool_calls_by_id,+ max_tool_output_chars=max_tool_output_chars,+ )+ raise ValueError(f"Unsupported train_transcript_format: {transcript_format}")+++def _scenario_sha256(instructions: str) -> str:+ return hashlib.sha256(instructions.encode("utf-8")).hexdigest()+++def _load_fixed_first_user_fixture(path: Path) -> dict[str, str]:+ if not path.is_file():+ raise FileNotFoundError(f"fixed-first-user fixture not found: {path}")+ data = json.loads(path.read_text(encoding="utf-8"))+ mapping = data.get("by_scenario_sha256") if isinstance(data, dict) else None+ if not isinstance(mapping, dict) or not mapping:+ raise ValueError(f"fixed-first-user fixture has no by_scenario_sha256 map: {path}")+ return {str(key): str(value) for key, value in mapping.items()}+++def _has_user_message(state: Any) -> bool:+ for message in getattr(state, "messages", []) or []:+ role = getattr(message, "role", None)+ if str(getattr(role, "value", role)) == "user":+ return True+ return False+++def _append_incoming_user_context(message: Any, state: Any) -> None:+ from tau2.data_model.message import AssistantMessage, MultiToolMessage, ToolMessage++ if isinstance(message, MultiToolMessage):+ state.messages.extend(message.tool_messages)+ elif isinstance(message, ToolMessage):+ state.messages.append(message)+ elif isinstance(message, AssistantMessage) and (+ message.has_content() or message.is_tool_call()+ ):+ state.messages.append(message)+++def _register_fixed_first_user(args: argparse.Namespace) -> str:+ if not args.fixed_first_user_file:+ return args.user+ _add_tau2_to_path(args.tau2_repo)+ mapping = _load_fixed_first_user_fixture(args.fixed_first_user_file)++ from tau2.data_model.message import UserMessage+ from tau2.registry import registry+ from tau2.user.user_simulator import UserSimulator++ class FixedFirstUserSimulator(UserSimulator): # type: ignore[misc]+ def _generate_next_message(self, message: Any, state: Any) -> UserMessage: # type: ignore[override]+ if not _has_user_message(state):+ key = _scenario_sha256(str(self.instructions or ""))+ fixed = mapping.get(key)+ if fixed is None:+ raise RuntimeError(+ f"fixed-first-user fixture does not cover this TAU-2 scenario: sha256={key}"+ )+ _append_incoming_user_context(message, state)+ return UserMessage(role="user", content=fixed)+ return super()._generate_next_message(message, state)++ if FIXED_FIRST_USER_NAME not in registry.get_users():+ registry.register_user(FixedFirstUserSimulator, FIXED_FIRST_USER_NAME)+ return FIXED_FIRST_USER_NAME++def _run_tau2(*,tau2_repo: Path,@@ -171,6 +387,7 @@ def _run_tau2(save_to: Path,):_add_tau2_to_path(tau2_repo)+ _patch_tau2_auxiliary_llm_defaults(agent_llm, agent_llm_args)from tau2.data_model.simulation import RunConfig, TextRunConfigfrom tau2.run import run_domain@@ -275,51 +492,125 @@ def _probe_corpus(args: argparse.Namespace, client: Any) -> dict[str, Any]:def _train(args: argparse.Namespace, train_results: Path, corpus_manifest: Path) -> dict[str, Any]:if corpus_manifest.is_file() and not args.force_train:- return json.loads(corpus_manifest.read_text())-- _run_tau2(- tau2_repo=args.tau2_repo,- domain=args.domain,- split=args.train_split_name,- task_ids=args.train_task_ids,- num_tasks=args.train_num_tasks,- trials=1,- max_steps=args.max_steps,- max_concurrency=args.max_concurrency,- agent=args.base_agent,- user=args.user,- agent_llm=args.agent_llm,- user_llm=args.user_llm,- agent_llm_args=args.agent_llm_args,- user_llm_args=args.user_llm_args,- seed=args.seed,- save_to=train_results,- )+ manifest = json.loads(corpus_manifest.read_text())+ cached_transcript_format = str(+ manifest.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT+ )+ if cached_transcript_format != args.train_transcript_format:+ raise ValueError(+ "cached corpus train_transcript_format mismatch: "+ f"{cached_transcript_format!r} != {args.train_transcript_format!r}; "+ "use a distinct corpus_id or --force-train"+ )+ cached_include_system_prompt = bool(manifest.get("train_include_system_prompt") or False)+ if cached_include_system_prompt != bool(args.train_include_system_prompt):+ raise ValueError(+ "cached corpus train_include_system_prompt mismatch: "+ f"{cached_include_system_prompt!r} != {bool(args.train_include_system_prompt)!r}; "+ "use a distinct corpus_id or --force-train"+ )+ cached_tool_output_max_chars = int(+ manifest.get("train_tool_output_max_chars") or DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS+ )+ if cached_tool_output_max_chars != int(args.train_tool_output_max_chars):+ raise ValueError(+ "cached corpus train_tool_output_max_chars mismatch: "+ f"{cached_tool_output_max_chars!r} != {int(args.train_tool_output_max_chars)!r}; "+ "use a distinct corpus_id or --force-train"+ )+ cached_skip_failed = bool(manifest.get("train_skip_failed_sessions") or False)+ if cached_skip_failed != bool(args.train_skip_failed_sessions):+ raise ValueError(+ "cached corpus train_skip_failed_sessions mismatch: "+ f"{cached_skip_failed!r} != {bool(args.train_skip_failed_sessions)!r}; "+ "use a distinct corpus_id or --force-train"+ )+ return manifest++ if train_results.is_file() and not args.force_train:+ data = json.loads(train_results.read_text())+ assert_tau2_results_complete(data, context=f"{args.domain} cached train")+ else:+ _run_tau2(+ tau2_repo=args.tau2_repo,+ domain=args.domain,+ split=args.train_split_name,+ task_ids=args.train_task_ids,+ num_tasks=args.train_num_tasks,+ trials=1,+ max_steps=args.max_steps,+ max_concurrency=args.max_concurrency,+ agent=args.base_agent,+ user=args.user,+ agent_llm=args.agent_llm,+ user_llm=args.user_llm,+ agent_llm_args=args.agent_llm_args,+ user_llm_args=args.user_llm_args,+ seed=args.seed,+ save_to=train_results,+ )+ data = json.loads(train_results.read_text())+ assert_tau2_results_complete(data, context=f"{args.domain} train")- data = json.loads(train_results.read_text())client = _client(args)committed = []+ system_prompt_text = ""+ if args.train_include_system_prompt:+ policy = _load_domain_policy(args.tau2_repo, args.domain)+ system_prompt_text = _build_system_prompt(policy)+ skipped_failed_sessions: list[dict[str, Any]] = []try:for sim in data.get("simulations") or []:- session_id = f"tau2-{args.domain}-train-{sim.get('task_id')}-trial-{sim.get('trial', 0)}"+ if args.train_skip_failed_sessions and not _task_success(sim):+ skipped_failed_sessions.append(+ {+ "session_id": (+ f"tau2-{args.domain}-train-{sim.get('task_id')}-"+ f"trial-{sim.get('trial', 0)}"+ ),+ "task_id": sim.get("task_id"),+ "trial": sim.get("trial", 0),+ "reward": _reward(sim),+ "db_match": _db_match(sim),+ }+ )+ continue+ session_id = (+ f"tau2-{args.domain}-train-{sim.get('task_id')}-trial-{sim.get('trial', 0)}"+ )created = client.create_session(session_id=session_id)sid = created.get("session_id", session_id)- for msg in sim.get("messages") or []:- role, text = _message_text(msg)- if not text.strip():- continue+ if system_prompt_text.strip():client.add_message(sid,- role=role,- parts=[{"type": "text", "text": text}],- created_at=msg.get("timestamp"),+ role="user",+ parts=[{"type": "text", "text": f"system:\n{system_prompt_text}"}],)+ tool_calls_by_id: dict[str, dict[str, Any]] = {}+ for msg in sim.get("messages") or []:+ for role, text in _message_texts(+ msg,+ transcript_format=args.train_transcript_format,+ tool_calls_by_id=tool_calls_by_id,+ max_tool_output_chars=args.train_tool_output_max_chars,+ ):+ if not text.strip():+ continue+ client.add_message(+ sid,+ role=role,+ parts=[{"type": "text", "text": text}],+ created_at=msg.get("timestamp"),+ )result = client.commit_session(sid, telemetry=True)task = _wait_task(client, result.get("task_id"), args.openviking_wait_timeout)committed.append({"session_id": sid,"task_id": sim.get("task_id"),+ "trial": sim.get("trial", 0),+ "reward": _reward(sim),+ "db_match": _db_match(sim),"commit_status": result.get("status"),"openviking_task_id": result.get("task_id"),"openviking_task_status": task.get("status"),@@ -344,8 +635,14 @@ def _train(args: argparse.Namespace, train_results: Path, corpus_manifest: Path)"agent_id": args.openviking_agent_id,"search_uri": args.search_uri,},+ "train_transcript_format": args.train_transcript_format,+ "train_include_system_prompt": bool(args.train_include_system_prompt),+ "train_skip_failed_sessions": bool(args.train_skip_failed_sessions),+ "train_tool_output_max_chars": args.train_tool_output_max_chars,"committed_sessions": committed,"committed_session_count": len(committed),+ "skipped_failed_sessions": skipped_failed_sessions,+ "skipped_failed_session_count": len(skipped_failed_sessions),"corpus_probe": corpus_probe,}_write_json(corpus_manifest, manifest)@@ -360,36 +657,81 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:from tau2.registry import registryfrom tau2.utils.llm_utils import generate+ scope_prompt = ""+ if args.scope_prompt_file is not None:+ scope_prompt = args.scope_prompt_file.read_text(encoding="utf-8").strip()+class OpenVikingMemoryAgent(LLMAgent):def get_init_state(self, message_history=None):state = super().get_init_state(message_history)+ if scope_prompt:+ state.system_messages.append(SystemMessage(role="system", content=scope_prompt))if args.retrieval_mode in {"first_user", "first_user_prewrite"}:state.system_messages.append(SystemMessage(role="system", content="<openviking_memory_not_loaded/>"))return state- def _retrieve(self, query: str) -> tuple[str, list[dict[str, Any]]]:+ def _retrieve(+ self,+ query: str,+ *,+ search_limit: int,+ inject_limit: int,+ inject_max_chars: int | None = None,+ ) -> tuple[str, list[dict[str, Any]]]:client = _client(args)rows: list[dict[str, Any]] = []try:- result = client.search(query=query, target_uri=args.search_uri, limit=args.retrieval_top_k)+ result = client.search(query=query, target_uri=args.search_uri, limit=search_limit)memories = list(getattr(result, "memories", []) or [])blocks = []- for index, match in enumerate(memories[: args.retrieval_top_k], 1):+ injected_chars_used = 0+ for index, match in enumerate(memories[:search_limit], 1):uri = getattr(match, "uri", "")text, read_error = _read_memory_text(client, match)+ clean_text = text.strip()+ block_text = f"Memory {index} ({uri}):\n{clean_text}" if clean_text else ""+ block_chars = len(block_text)+ budget_used_before = injected_chars_used+ budget_dropped = False+ truncated = False+ injected = index <= inject_limit and bool(block_text)+ if injected and inject_max_chars is not None:+ remaining = inject_max_chars - injected_chars_used+ if remaining <= 0:+ injected = False+ budget_dropped = True+ elif block_chars > remaining:+ if not blocks:+ block_text = block_text[:remaining]+ block_chars = len(block_text)+ truncated = True+ else:+ injected = False+ budget_dropped = True+ if injected:+ injected_chars_used += block_charsrow = {"uri": uri,"score": getattr(match, "score", None),"level": getattr(match, "level", None),"text_chars": len(text),+ "block_chars": block_chars,+ "injected": injected,+ "inject_max_chars": inject_max_chars,+ "inject_budget_used_before": budget_used_before,+ "inject_budget_used_after": injected_chars_used,+ "inject_budget_dropped": budget_dropped,+ "inject_budget_truncated": truncated,}+ if budget_dropped:+ row["skipped_reason"] = "inject_char_budget_exceeded"if read_error:row["read_error"] = read_errorrows.append(row)- if text.strip():- blocks.append(f"Memory {index} ({uri}):\n{text.strip()}")+ if injected:+ blocks.append(block_text)return "\n\n".join(blocks), rowsfinally:client.close()@@ -400,11 +742,13 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:@staticmethoddef _trace_injection_fields(block: str, matches: list[dict[str, Any]]) -> dict[str, Any]:- injected_count = sum(1 for row in matches if int(row.get("text_chars") or 0) > 0)+ injected_count = sum(1 for row in matches if row.get("injected"))return {"injected": bool(block.strip()),"injected_count": injected_count if block.strip() else 0,- "retrieval_action_taken": "retrieve_and_inject" if block.strip() else "retrieve_no_injection",+ "retrieval_action_taken": "retrieve_and_inject"+ if block.strip()+ else "retrieve_no_injection",}def _generate(self, messages):@@ -476,7 +820,8 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:(ifor i, item in enumerate(state.system_messages)- if isinstance(item, SystemMessage) and item.content == "<openviking_memory_not_loaded/>"+ if isinstance(item, SystemMessage)+ and item.content == "<openviking_memory_not_loaded/>"),None,)@@ -484,11 +829,16 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:role_value = getattr(role, "value", role)if marker_index is not None and str(role_value) == "user":query = str(getattr(message, "content", "") or "")- block, matches = self._retrieve(query)+ block, matches = self._retrieve(+ query,+ search_limit=args.first_user_retrieval_top_k,+ inject_limit=args.first_user_inject_top_k,+ inject_max_chars=args.first_user_memory_inject_max_chars,+ )prompt = ("No OpenViking memory matched this user request."if not block- else "Use these OpenViking experience memories only when they match the current task:\n\n"+ else "Use these OpenViking memories only when they match the current task:\n\n"+ block)state.system_messages[marker_index] = SystemMessage(role="system", content=prompt)@@ -496,6 +846,9 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:{"decision_node": "first_user","query": query,+ "search_limit": args.first_user_retrieval_top_k,+ "inject_limit": args.first_user_inject_top_k,+ "inject_max_chars": args.first_user_memory_inject_max_chars,"match_count": len(matches),"matches": matches,**self._trace_injection_fields(block, matches),@@ -508,11 +861,19 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:write_calls = [call for call in tool_calls if _is_write_tool_call(call)]if write_calls:query = _tool_call_query(write_calls, state.messages)- block, matches = self._retrieve(query)+ block, matches = self._retrieve(+ query,+ search_limit=args.prewrite_retrieval_top_k,+ inject_limit=args.prewrite_inject_top_k,+ inject_max_chars=args.prewrite_memory_inject_max_chars,+ )self._trace({"decision_node": "before_write_tool_call","query": query,+ "search_limit": args.prewrite_retrieval_top_k,+ "inject_limit": args.prewrite_inject_top_k,+ "inject_max_chars": args.prewrite_memory_inject_max_chars,"match_count": len(matches),"matches": matches,**self._trace_injection_fields(block, matches),@@ -528,8 +889,7 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:if block:prompt = ("Before executing the pending write-like tool call, use these "- "OpenViking experience memories only when they match the current task:\n\n"- + block+ "OpenViking memories only when they match the current task:\n\n" + block)assistant_message = self._generate(state.system_messages@@ -540,6 +900,7 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:return assistant_message, stateif AGENT_NAME not in registry.get_agents():+def create_openviking_memory_agent(tools, domain_policy, **kwargs):return OpenVikingMemoryAgent(tools=tools,@@ -577,36 +938,190 @@ def main() -> int:parser.add_argument("--user-llm", required=True)parser.add_argument("--agent-llm-args", type=_json, default={})parser.add_argument("--user-llm-args", type=_json, default={})- parser.add_argument("--openviking-url", required=True)- parser.add_argument("--openviking-account", required=True)- parser.add_argument("--openviking-user", required=True)- parser.add_argument("--openviking-agent-id", required=True)+ parser.add_argument("--openviking-url")+ parser.add_argument("--openviking-account")+ parser.add_argument("--openviking-user")+ parser.add_argument("--openviking-agent-id")parser.add_argument("--openviking-timeout", type=float, default=600.0)parser.add_argument("--openviking-wait-timeout", type=int, default=600)- parser.add_argument("--search-uri", required=True)+ parser.add_argument("--search-uri")parser.add_argument("--retrieval-top-k", type=int, default=4)+ parser.add_argument("--first-user-retrieval-top-k", type=int)+ parser.add_argument("--first-user-inject-top-k", type=int)+ parser.add_argument("--prewrite-retrieval-top-k", type=int)+ parser.add_argument("--prewrite-inject-top-k", type=int)+ parser.add_argument("--memory-inject-max-chars", type=int)+ parser.add_argument("--first-user-memory-inject-max-chars", type=int)+ parser.add_argument("--prewrite-memory-inject-max-chars", type=int)+ parser.add_argument("--fixed-first-user-file", type=Path)+ parser.add_argument("--scope-prompt-file", type=Path)+ parser.add_argument(+ "--train-transcript-format",+ choices=[+ TRAIN_TRANSCRIPT_OPENVIKING_TEXT,+ TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS,+ TRAIN_TRANSCRIPT_CUSTOM_LIKE,+ ],+ default=TRAIN_TRANSCRIPT_OPENVIKING_TEXT,+ help=(+ "How to replay TAU-2 train messages into OpenViking sessions. "+ "openviking_text preserves the compact adapter text format; role_tool_blocks "+ "uses role-prefixed messages plus tool-call/tool-response blocks. "+ "custom_like is a compatibility alias for older cached PR-B corpora."+ ),+ )+ parser.add_argument(+ "--train-include-system-prompt",+ action="store_true",+ help="Prepend the domain policy as a user-visible system block during training.",+ )+ parser.add_argument(+ "--train-skip-failed-sessions",+ action="store_true",+ help="Skip reward<1 train sessions when building positive trajectory memory.",+ )+ parser.add_argument(+ "--train-tool-output-max-chars",+ type=int,+ default=DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS,+ help=(+ "Maximum characters kept for each tool-response block when "+ f"--train-transcript-format={TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS}."+ ),+ )parser.add_argument("--retrieval-mode",choices=["first_user", "prewrite", "first_user_prewrite"],default="first_user",)parser.add_argument("--force-train", action="store_true")+ parser.add_argument("--prepare-corpus-only", action="store_true")+ parser.add_argument(+ "--no-memory",+ action="store_true",+ help="Run the configured TAU-2 agent without OpenViking retrieval.",+ )args = parser.parse_args()normalize_litellm_env()+ if args.train_tool_output_max_chars <= 0:+ parser.error("--train-tool-output-max-chars must be positive")+ for name in (+ "memory_inject_max_chars",+ "first_user_memory_inject_max_chars",+ "prewrite_memory_inject_max_chars",+ ):+ value = getattr(args, name)+ if value is not None and value < 0:+ parser.error(f"--{name.replace('_', '-')} must be non-negative")+ if not args.no_memory:+ missing = [+ name+ for name in (+ "openviking_url",+ "openviking_account",+ "openviking_user",+ "openviking_agent_id",+ "search_uri",+ )+ if not getattr(args, name)+ ]+ if missing:+ parser.error(+ "OpenViking memory runs require: "+ + ", ".join("--" + name.replace("_", "-") for name in missing)+ )args.tau2_repo = args.tau2_repo.resolve()+ args.run_dir = args.run_dir.resolve()+ if args.corpus_dir is not None:+ args.corpus_dir = args.corpus_dir.resolve()args.run_dir.mkdir(parents=True, exist_ok=True)corpus_dir = args.corpus_dir or args.run_dircorpus_dir.mkdir(parents=True, exist_ok=True)+ args.first_user_retrieval_top_k = args.first_user_retrieval_top_k or args.retrieval_top_k+ args.first_user_inject_top_k = args.first_user_inject_top_k or args.first_user_retrieval_top_k+ args.prewrite_retrieval_top_k = args.prewrite_retrieval_top_k or args.retrieval_top_k+ args.prewrite_inject_top_k = args.prewrite_inject_top_k or args.prewrite_retrieval_top_k+ args.first_user_memory_inject_max_chars = (+ args.first_user_memory_inject_max_chars+ if args.first_user_memory_inject_max_chars is not None+ else args.memory_inject_max_chars+ )+ args.prewrite_memory_inject_max_chars = (+ args.prewrite_memory_inject_max_chars+ if args.prewrite_memory_inject_max_chars is not None+ else args.memory_inject_max_chars+ )+ if args.fixed_first_user_file is not None:+ args.fixed_first_user_file = args.fixed_first_user_file.expanduser().resolve()+ if args.scope_prompt_file is not None:+ args.scope_prompt_file = args.scope_prompt_file.expanduser().resolve()+ if not args.scope_prompt_file.is_file():+ parser.error(f"--scope-prompt-file does not exist: {args.scope_prompt_file}")train_results = corpus_dir / "train_results.json"corpus_manifest = corpus_dir / "corpus_manifest.json"eval_results = args.run_dir / f"{args.run_label}.json"trace_path = args.run_dir / f"{args.run_label}.retrieval_trace.jsonl"summary_path = args.run_dir / f"{args.run_label}.summary.json"+ if args.no_memory:+ user_name = _register_fixed_first_user(args)+ _run_tau2(+ tau2_repo=args.tau2_repo,+ domain=args.domain,+ split=args.eval_split_name,+ task_ids=args.task_ids,+ num_tasks=args.num_tasks,+ trials=1,+ max_steps=args.max_steps,+ max_concurrency=args.max_concurrency,+ agent=args.base_agent,+ user=user_name,+ agent_llm=args.agent_llm,+ user_llm=args.user_llm,+ agent_llm_args=args.agent_llm_args,+ user_llm_args=args.user_llm_args,+ seed=args.seed,+ save_to=eval_results,+ )+ assert_tau2_results_complete(+ json.loads(eval_results.read_text()), context=f"{args.domain} eval"+ )+ summary = {+ "run_label": args.run_label,+ "domain": args.domain,+ "strategy_id": args.strategy_id,+ "seed": args.seed,+ "fixed_first_user_file": str(args.fixed_first_user_file)+ if args.fixed_first_user_file+ else None,+ "eval_results": str(eval_results),+ "metrics": _metrics(eval_results),+ }+ _write_json(summary_path, summary)+ print(json.dumps(summary, ensure_ascii=False, sort_keys=True))+ return 0+corpus = _train(args, train_results, corpus_manifest)+ if args.prepare_corpus_only:+ print(+ json.dumps(+ {+ "run_label": args.run_label,+ "domain": args.domain,+ "strategy_id": args.strategy_id,+ "prepare_corpus_only": True,+ "corpus": corpus,+ },+ ensure_ascii=False,+ sort_keys=True,+ )+ )+ return 0+trace_path.touch()_register_memory_agent(args, trace_path)+ user_name = _register_fixed_first_user(args)_run_tau2(tau2_repo=args.tau2_repo,domain=args.domain,@@ -617,7 +1132,7 @@ def main() -> int:max_steps=args.max_steps,max_concurrency=args.max_concurrency,agent=AGENT_NAME,- user=args.user,+ user=user_name,agent_llm=args.agent_llm,user_llm=args.user_llm,agent_llm_args=args.agent_llm_args,@@ -625,12 +1140,32 @@ def main() -> int:seed=args.seed,save_to=eval_results,)+ assert_tau2_results_complete(+ json.loads(eval_results.read_text()), context=f"{args.domain} eval"+ )summary = {"run_label": args.run_label,"domain": args.domain,"strategy_id": args.strategy_id,"retrieval_mode": args.retrieval_mode,+ "retrieval": {+ "first_user_retrieval_top_k": args.first_user_retrieval_top_k,+ "first_user_inject_top_k": args.first_user_inject_top_k,+ "first_user_memory_inject_max_chars": args.first_user_memory_inject_max_chars,+ "prewrite_retrieval_top_k": args.prewrite_retrieval_top_k,+ "prewrite_inject_top_k": args.prewrite_inject_top_k,+ "prewrite_memory_inject_max_chars": args.prewrite_memory_inject_max_chars,+ "memory_inject_max_chars": args.memory_inject_max_chars,+ },+ "train_transcript_format": args.train_transcript_format,+ "train_include_system_prompt": bool(args.train_include_system_prompt),+ "train_skip_failed_sessions": bool(args.train_skip_failed_sessions),+ "train_tool_output_max_chars": args.train_tool_output_max_chars,"seed": args.seed,+ "fixed_first_user_file": str(args.fixed_first_user_file)+ if args.fixed_first_user_file+ else None,+ "scope_prompt_file": str(args.scope_prompt_file) if args.scope_prompt_file else None,"corpus": corpus,"eval_results": str(eval_results),"retrieval_trace": str(trace_path),diff --git a/benchmark/tau2/scripts/tau2_common.py b/benchmark/tau2/scripts/tau2_common.pyindex a8b5ce201..4f4505bd8 100755--- a/benchmark/tau2/scripts/tau2_common.py+++ b/benchmark/tau2/scripts/tau2_common.py@@ -5,13 +5,13 @@ import osimport reimport shutilimport subprocess+from collections import Counterfrom datetime import datetime, timezonefrom pathlib import Pathfrom typing import Anyimport yaml-TAU2_DIR = Path(__file__).resolve().parents[1]REPO_ROOT = TAU2_DIR.parents[1]CONFIRMATION_AWARE_UPSTREAM_PR = "https://github.com/sierra-research/tau2-bench/pull/297"@@ -63,6 +63,7 @@ def normalize_litellm_env() -> dict[str, Any]:def render_env(value: Any) -> Any:if isinstance(value, str):+def replace(match: re.Match[str]) -> str:name = match.group(1)default = match.group(2) or ""@@ -79,11 +80,7 @@ def render_env(value: Any) -> Any:def deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:merged = dict(base)for key, value in override.items():- if (- key in merged- and isinstance(merged[key], dict)- and isinstance(value, dict)- ):+ if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):merged[key] = deep_merge(merged[key], value)else:merged[key] = value@@ -125,6 +122,66 @@ def write_json(path: Path, payload: Any) -> None:)+def tau2_result_failures(data: dict[str, Any], *, expected_trials: int = 1) -> list[str]:+ tasks = data.get("tasks") or []+ simulations = data.get("simulations") or []+ failures: list[str] = []+ if tasks:+ expected_task_ids = {+ str(task.get("id", task.get("task_id"))) for task in tasks if isinstance(task, dict)+ }+ observed_task_ids = {str(sim.get("task_id")) for sim in simulations}+ expected = len(tasks) * expected_trials+ if len(simulations) != expected:+ failures.append(f"expected {expected} simulations, found {len(simulations)}")+ if observed_task_ids != expected_task_ids:+ missing = sorted(expected_task_ids - observed_task_ids)+ extra = sorted(observed_task_ids - expected_task_ids)+ failures.append(+ f"simulation task ids do not match tasks: missing={missing[:10]} extra={extra[:10]}"+ )+ expected_pairs = {+ (task_id, trial) for task_id in expected_task_ids for trial in range(expected_trials)+ }+ observed_pairs = [+ (str(sim.get("task_id")), int(sim.get("trial", 0))) for sim in simulations+ ]+ duplicate_pairs = sorted(+ pair for pair, count in Counter(observed_pairs).items() if count != 1+ )+ missing_pairs = sorted(expected_pairs - set(observed_pairs))+ if duplicate_pairs or missing_pairs:+ failures.append(+ "simulation task/trial coverage mismatch: "+ f"missing={missing_pairs[:10]} duplicate={duplicate_pairs[:10]}"+ )++ for sim in simulations:+ info = sim.get("info") or {}+ termination_reason = str(sim.get("termination_reason") or "")+ if info.get("failed_after_attempts") or "infrastructure_error" in termination_reason:+ failures.append(+ "task="+ f"{sim.get('task_id')} trial={sim.get('trial', 0)} "+ f"termination={termination_reason} error={info.get('error') or info.get('error_type')}"+ )+ elif not sim.get("messages"):+ failures.append(+ f"task={sim.get('task_id')} trial={sim.get('trial', 0)} has no messages"+ )+ return failures+++def assert_tau2_results_complete(+ data: dict[str, Any], *, context: str, expected_trials: int = 1+) -> None:+ failures = tau2_result_failures(data, expected_trials=expected_trials)+ if failures:+ preview = "; ".join(failures[:5])+ more = f"; ... {len(failures) - 5} more" if len(failures) > 5 else ""+ raise RuntimeError(f"{context} produced invalid TAU-2 results: {preview}{more}")++def strategy_ids(config: dict[str, Any]) -> list[str]:strategies = config.get("strategies") or []if not isinstance(strategies, list):@@ -219,9 +276,7 @@ def user_simulator_policy(config: dict[str, Any]) -> str:policy = config.get("eval", {}).get("user_simulator_policy", "official")policy = str(policy)if policy not in {"official", "confirmation_aware"}:- raise ValueError(- "eval.user_simulator_policy must be 'official' or 'confirmation_aware'"- )+ raise ValueError("eval.user_simulator_policy must be 'official' or 'confirmation_aware'")return policydiff --git a/openviking/prompts/templates/memory/trajectories.yaml b/openviking/prompts/templates/memory/trajectories.yamlindex bd894dfef..d369a93c5 100644--- a/openviking/prompts/templates/memory/trajectories.yaml+++ b/openviking/prompts/templates/memory/trajectories.yaml@@ -1,6 +1,6 @@memory_type: trajectoriesdescription: |- A record of agent execution in one business domain within a conversation.+ A compact, reusable operation contract distilled from an agent task trajectory.Extract when the agent worked through identifiable tasks involving decisions, tool calls, or multi-step actions.Skip pure chitchat or simple Q&A with no execution trace.@@ -20,7 +20,7 @@ fields:If English: lowercase snake_case, max 5 words.If Chinese: concise noun phrase, max 15 characters.Do not create near-duplicate names for the same task.- Good: "flight_booking_duplicate_resolution", "pytest_asyncio_cancel_hang_fix", "机票重复预订处理".+ Good: "duplicate_request_resolution", "pytest_asyncio_cancel_hang_fix", "重复请求处理".merge_op: immutable- name: outcome@@ -32,31 +32,48 @@ fields:- name: contenttype: stringdescription: |- Execution trace in EXACTLY this format:- - Goal: <what the user wanted to achieve — one sentence>- - Trajectory:- 1. <what the agent aimed to achieve>.- Actions: <summarized interactions, tools used, and summarized responses>.- Progress: <how this shifted the state>.- 2. <...>.- Actions: <...>.- Progress: <...>.- 3. <... continue sequentially to capture the entire execution>- - Result: <final outcome and any important side effects — one sentence>- - Fail reason: <If task succeeded, leave empty or write "None". If failed, provide a detailed analysis of why the failure occurred and suggest how the agent's approach could be improved.>+ Procedure-like operation contract in EXACTLY this format:- Rules for 'Trajectory' section:- - STANDALONE COMPLETENESS: The trajectory must be comprehensive enough that a reader can fully understand exactly how the agent performed the task, what attempts it made, and what tools it used, entirely without the raw logs.- - EXHAUSTIVE TRACKING: Record every logical interaction, tool use, and system response in chronological order as a numbered list.- - SUMMARIZE LONG TEXTS: Do NOT record exact tool responses, raw JSON payloads, or verbatim user/agent messages, especially when they are very long. You MUST use a concise, summarized version that captures the core meaning without losing any key information, constraints, or data points critical to the task.- - CAPTURE ALL MISTAKES: You MUST explicitly detail any errors, false attempts, wrong function calls, or agent misunderstandings. Do not gloss over failures, dead ends, or retry loops.- - INTENT & PROGRESS TRACKING: Every step must clearly state the current agent intention, summarize the 'Actions/Events' taken, and conclude with 'Progress' (evaluating what was achieved or blocked).- - TOOL TRACKING (under 'Actions'):- * Format (Success): "Called <tool_name(s)> -> Response: <summarized output retaining key info>."- * Format (Error/Mistake): "Called <tool_name(s)> -> Error/Issue: <summarized error> | Context: <why it failed or why it was the wrong call>."+ # <trajectory_name>+ - Domain: <task domain or product area, e.g. customer support, commerce operations, software tooling>+ - Trigger: <the reusable user intent or task situation — one sentence>+ - Operation Intent:+ - Family: <exactly one of read_verify_only, create_new_object, update_existing_object, cancel_existing_object, replace_existing_object, attribute_append_remove, compensation_or_refund, read_then_transfer, final_response_only>+ - Primary object: <the object whose state or lifecycle this record is about>+ - Target object/lifecycle: <the intended final state, lifecycle transition, or handoff target>+ - Preconditions:+ 1. <state, identity, policy, tool-observation, or confirmation prerequisite>+ 2. <...>+ - Immutable Object Boundary:+ - <object fields, lifecycle state, policy facts, permissions, or ownership facts that must be read/verified and not inferred or overwritten>+ - Procedure:+ 1. <future-agent instruction for the first decision/action step; include key tool names when useful>+ 2. <future-agent instruction for the next decision/action step>+ 3. <continue until the write/confirmation/final-response boundary is clear>+ - Write Field Provenance:+ - <write-relevant field family>: <allowed source such as latest user request, explicit confirmation, fresh tool observation, computed policy result, stable policy, or current object state>+ - Anti-patterns:+ - <mistake, missing check, wrong ordering, over-broad action, or failed attempt to avoid>+ - Applicability Boundary:+ - <when this memory applies>+ - <when this memory should not be used>+ - Negative Applicability:+ - <nearby intent, object lifecycle, policy state, permission state, or target object where this memory must not be applied>+ - Result: <final outcome and important side effects — one sentence>+ - Evidence: <brief generalized source note from this session, including success/failure/partial status>- General Rules:- - Use exactly the 4 labels (Goal, Trajectory, Result, Fail reason) in this exact order.- - Goal and Result are ONE sentence each.+ Rules:+ - Write for future agent execution. Capture the reusable contract: trigger -> prerequisites -> read/verify -> confirm if needed -> one primary write, handoff, or final response -> verify/communicate result.+ - Extract one record per primary user intent, object, and lifecycle transition. Split separable intents, pivots, follow-ups, enabling writes, and final writes; omit low-value side work.+ - Family must be exactly one listed enum value. Do not invent compound labels. Use read_then_transfer when the reusable path ends in handoff after verification.+ - Procedure must describe only the primary operation in this record. Mention other operations only as prerequisites, consequences, boundaries, or negative applicability.+ - Stop each record at its family boundary: update_existing_object stops after update verification, cancel_existing_object after cancellation verification, create_new_object after creation verification, replace_existing_object after replacement verification, read_then_transfer after handoff verification.+ - Do not include a second lifecycle-changing write in the Procedure, Target, Result, or Evidence of the same record, even when that later write happened immediately after the first one.+ - Do not let a later operation in the same session rename, narrow, or extend this record's Trigger, Target, Procedure, Result, or Evidence. Put the later operation in a separate record.+ - Put immutable facts and write/user-visible field sources in their dedicated sections. Do not copy fields from a previous/source object into a new/replacement object without current user request or explicit confirmation.+ - Keep blocked actions and failed lessons in Anti-patterns / Negative Applicability / Evidence, not as positive procedure steps. Do not invent a corrected path when the session did not demonstrate one.+ - Generalize the session: remove raw identifiers, names, contact values, exact dates, case-specific amounts, exact routes/locations, product names, and raw tool payloads. Keep tool names only when they are part of the reusable path.+ - Use exactly the title, Domain line, and 11 labels above in this exact order.+ - Trigger, Result, and Evidence are ONE sentence each.- No extra headings, free paragraphs, or closing remarks.merge_op: patchdiff --git a/openviking/session/memory/agent_trajectory_context_provider.py b/openviking/session/memory/agent_trajectory_context_provider.pyindex 2316052f7..1b8cd42cf 100644--- a/openviking/session/memory/agent_trajectory_context_provider.py+++ b/openviking/session/memory/agent_trajectory_context_provider.py@@ -14,7 +14,6 @@ from openviking.server.identity import RequestContextfrom openviking.session.memory.session_extract_context_provider import (SessionExtractContextProvider,)-from openviking.storage.viking_fs import VikingFSfrom openviking_cli.utils import get_loggerlogger = get_logger(__name__)@@ -24,18 +23,25 @@ TRAJECTORY_MEMORY_TYPE = "trajectories"class AgentTrajectoryContextProvider(SessionExtractContextProvider):- """Phase 1 provider: extract trajectory summaries from conversation."""+ """Phase 1 provider: extract reusable trajectory-view memories."""def instruction(self) -> str:output_language = self._output_language- return f"""You are a memory extraction agent. Summarize this agent session as a trajectory record.--One session = one trajectory. Always output exactly one, no exceptions.-Sub-tasks, pivots, errors, and follow-ups are numbered steps inside that one record — not separate trajectories.--Output a JSON object with a `trajectories` array containing exactly one item.-Follow field descriptions in the schema. JSON only, no explanation.-All content fields must be written in {output_language}.+ return f"""You are a memory extraction agent. Convert this session into reusable trajectory-view memories.++Each memory is a compact operation contract for a future agent, not a transcript.+Capture trigger, prerequisites, verified boundaries, procedure, provenance,+anti-patterns, and applicability. Split separate intents, pivots, enabling writes,+and final writes into separate records; omit low-value side work. Use only schema+operation-family enum values, not compound labels. Generalize away raw identifiers,+names, exact dates, amounts, routes, and tool payloads. Do not let a later+operation in the session rename, narrow, or extend an earlier operation record.+Stop each record at its family boundary; do not include a second lifecycle-+changing write in the same operation contract.++Output JSON only: {{"trajectories": [...]}}. Include items only for reusable+operation contracts supported by the session. Follow field descriptions in the+schema. All content fields must be written in {output_language}."""def get_memory_schemas(self, ctx: RequestContext) -> List[Any]: