Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(memory): upgrade trajectory extraction to beat no-memory baseline (#2017)

huangruiteng520713c6247e
diff --git a/benchmark/tau2/README.md b/benchmark/tau2/README.md
index 7ebdb0807..3c9e5501a 100644
--- a/benchmark/tau2/README.md
+++ b/benchmark/tau2/README.md
@@ -1,13 +1,13 @@
# TAU-2 Benchmark
This directory contains a small OpenViking-style entry point for TAU-2 memory
-evaluation. The first version is intentionally narrow:
+evaluation. The scope is intentionally narrow:
- fresh OpenViking Memory V2 experience-only baseline;
- Memory V2 pre-write recall treatment.
+- trajectory memory retrieval treatment for the refined extraction prompt.
-Trajectory / procedure-view prompts, category rerank, and other harness-only
-diagnostics are intentionally left out of this first PR.
+Category rerank and other harness-only diagnostics are intentionally left out.
## Layout
@@ -16,7 +16,8 @@ benchmark/tau2/
├── config/
│ ├── baseline.yaml
│ ├── official.yaml
-│ └── prewrite.yaml
+│ ├── prewrite.yaml
+│ └── trajectory.yaml
├── scripts/
│ ├── run_eval.py
│ ├── setup_tau2_repo.sh
@@ -24,7 +25,9 @@ benchmark/tau2/
└── run_full_eval.sh
```
-Generated artifacts are written to `benchmark/tau2/result/<run_id>/`.
+Generated eval artifacts are written to `benchmark/tau2/result/<run_id>/`.
+Memory corpus artifacts are cached outside the run id at
+`benchmark/tau2/result/memory_corpora/` by default.
## Quick Start
@@ -37,6 +40,21 @@ export TAU2_REPO=/path/to/tau2-bench
export TAU2_CLI=/path/to/tau2
```
+The default OpenViking TAU-2 memory evidence protocol is
+`fixed_first_user_full8`: retail + airline, 8 repeats, same seeds, confirmation
+aware user simulator, and fixed first user fixtures for both domains. Later user
+simulator turns remain live. Set the fixture paths before running the default
+configs:
+
+```bash
+export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail/fixed_first_user_fixture.json
+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline/fixed_first_user_fixture.json
+```
+
+`--strict-preflight` fails when `eval.require_fixed_first_user=true` and either
+fixture is missing. Use `config/official.yaml` for an explicit non-fixed,
+official-live-user control.
+
For a local one-command setup, clone and install TAU-2 into ignored benchmark
directories:
@@ -77,6 +95,18 @@ benchmark/tau2/run_full_eval.sh \
--repeat-count 1
```
+Plan a one-cell trajectory memory smoke:
+
+```bash
+benchmark/tau2/run_full_eval.sh \
+ --config benchmark/tau2/config/trajectory.yaml \
+ --domain retail \
+ --strategy-id memory_v2_trajectory_view \
+ --num-tasks 1 \
+ --train-num-tasks 1 \
+ --repeat-count 1
+```
+
Run the Memory V2 8-trial matrix (`retail + airline` x 2 strategies x 8 repeats):
```bash
@@ -85,6 +115,42 @@ benchmark/tau2/run_full_eval.sh \
--execute
```
+## Reproduce the PR-B Evidence
+
+The PR-B headline and content-shape ablation use
+`config/prb_content_matrix_new_prompt.yaml`. It runs the no-memory control plus
+trajectory top4, experience top2, and representative 4000-character budget
+ablation routes across `retail + airline` with 8 repeats.
+
+First run one tiny end-to-end smoke against a clean local OpenViking service:
+
+```bash
+benchmark/tau2/run_full_eval.sh \
+ --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \
+ --domain retail \
+ --strategy-id new_traj_fixed_first_user_prewrite \
+ --num-tasks 1 \
+ --train-num-tasks 1 \
+ --repeat-count 1 \
+ --strict-preflight \
+ --execute
+```
+
+Then run the full PR-B matrix:
+
+```bash
+benchmark/tau2/run_full_eval.sh \
+ --config benchmark/tau2/config/prb_content_matrix_new_prompt.yaml \
+ --run-id prb_content_matrix_new_prompt_full8 \
+ --strict-preflight \
+ --execute
+```
+
+The main result is written to
+`benchmark/tau2/result/prb_content_matrix_new_prompt_full8/scoreboard.json`.
+Per-cell outputs live under `cell_results/`; corpus identity and generated
+memory checks live under `memory_corpora/`.
+
For a small E2E smoke, keep both the eval and train slices tiny:
```bash
@@ -103,21 +169,54 @@ and `OPENAI_API_BASE` for LiteLLM before running upstream TAU-2.
Start the OpenViking service before executing memory cells, and verify it with
`ov status`. For evidence runs, use a clean OpenViking workspace/config and set
-`OPENVIKING_URL` explicitly so local custom memory templates do not pollute the
-Memory V2 baseline.
+`OPENVIKING_URL` explicitly so local template overrides do not pollute the
+Memory V2 baseline. For trajectory memory evidence, start the service from this
+branch and inspect generated trajectory files; changing `search_uri` alone does
+not prove the new trajectory prompt was used.
## Memory Adapter
-`memory_v2_experience_only` and `memory_v2_prewrite` cells run through a small
-TAU-2 agent adapter in this directory:
+Memory V2 cells run through a small TAU-2 agent adapter in this directory:
- train by writing TAU-2 training conversations into OpenViking sessions;
-- evaluate by retrieving OpenViking experience memory at the first user turn;
+- evaluate by retrieving OpenViking memory at the first user turn;
- for pre-write recall, retrieve again before write-like tool calls and
- regenerate that step with the matched memories;
+ regenerate that step with the matched memories. The default benchmark
+ retrieves 6 pre-write candidates and injects 2, which keeps extra candidates
+ visible in traces without expanding the prompt budget;
+- optionally run an explicit scope-prompt treatment that keeps retrieved
+ memories advisory and asks the agent to preserve the current task scope before
+ write-like tool calls. Configs provide per-domain files through
+ `scope_prompt_files`;
- emit artifact metadata to identify the OpenViking account, agent,
corpus, retrieval mode, and simulator policy used by each cell.
+For exploratory gates, prefer a bounded run with `--cell-timeout-seconds`.
+Timed-out cells are recorded with return code `124`, `timed_out=true`, and are
+excluded from scoreboard metrics, which keeps smoke runs from silently becoming
+long-running evidence jobs.
+
+The existing `train_memory_mode: experience_only` value selects the Memory V2
+session-commit path. `search_memory_type` selects which generated memory bucket
+is retrieved during eval (`experiences` by default, `trajectories` for
+`config/trajectory.yaml`). The runner prepares each distinct
+`domain + corpus_id` once and reuses it across eval run ids when the cached
+`corpus_manifest.json` is present. Different corpora may be prepared in
+parallel with `benchmark.corpus_prepare_concurrency`; session commits inside one
+corpus remain serial to preserve OpenViking write semantics.
+
+By default, trajectory extraction is transcript-only: the runner replays TAU-2
+messages into an OpenViking session and does not expose held-out reward or
+assertion results to the extractor. The PR-B evidence config can also use a
+structured role/tool transcript, include the domain policy in the training
+session, skip failed train sessions when building positive procedure memory, and
+cap injected memory by total character budget for content-shape ablations.
+
+Eval cells run in parallel with `benchmark.strategy_concurrency` by default and
+can be overridden with `--strategy-concurrency`. This only parallelizes read-only
+TAU-2 eval cells; corpus writes inside one corpus are still serialized by the
+prepare step.
+
## User Simulator Policy
The runner default is the official TAU-2 user simulator if
@@ -131,6 +230,14 @@ confirmation boundary to the TAU-2 user simulator guidelines; metadata such as
the upstream PR link is kept in run artifacts, not in the simulator prompt.
Reference: [sierra-research/tau2-bench#297](https://github.com/sierra-research/tau2-bench/pull/297).
+Optional fixed-first-user fixtures keep the first simulated user turn stable
+while preserving live simulator behavior after that turn:
+
+```bash
+export TAU2_RETAIL_FIXED_FIRST_USER_FILE=/path/to/retail_fixture.json
+export TAU2_AIRLINE_FIXED_FIRST_USER_FILE=/path/to/airline_fixture.json
+```
+
Use `config/official.yaml` with a clean TAU-2 checkout when you need an
official-user-simulator parity run. If the checkout was already patched, the
artifact records that boundary instead of labeling the run pure official.
diff --git a/benchmark/tau2/config/baseline.yaml b/benchmark/tau2/config/baseline.yaml
index 4c4a5060e..0672322bf 100644
--- a/benchmark/tau2/config/baseline.yaml
+++ b/benchmark/tau2/config/baseline.yaml
@@ -6,7 +6,9 @@ benchmark:
train_split_name: train
eval_split_name: test
repeat_count: 8
+ strategy_concurrency: 8
task_max_concurrency: 10
+ corpus_prepare_concurrency: 2
max_steps: 200
seed: 300
agent: llm_agent
@@ -17,12 +19,25 @@ paths:
tau2_repo: ${TAU2_REPO:-data/external_benchmarks/tau2-bench}
tau2_cli: ${TAU2_CLI:-tau2}
output_dir: benchmark/tau2/result
+ # Corpus writes are expensive and should be reused across eval run ids when
+ # the train split and memory prompt/config did not change.
+ corpus_cache_dir: benchmark/tau2/result/memory_corpora
eval:
+ # Default OpenViking TAU-2 memory evidence uses the fixed-first-user full8
+ # protocol: retail + airline, 8 repeats, same seeds, first user turn pinned by
+ # fixtures, later user simulator turns still live.
+ protocol: fixed_first_user_full8
+ require_fixed_first_user: true
# The runner default is official if this field is omitted. The OpenViking
# memory benchmark config opts into a confirmation-aware TAU-2 user simulator
# prompt; run_eval.py applies that small prompt patch idempotently when needed.
user_simulator_policy: confirmation_aware
+ # Fixed-first-user fixtures keep the first simulated user turn stable while
+ # leaving later turns live. Main PR-B evidence requires these env vars.
+ fixed_first_user_fixtures:
+ retail: ${TAU2_RETAIL_FIXED_FIRST_USER_FILE:-}
+ airline: ${TAU2_AIRLINE_FIXED_FIRST_USER_FILE:-}
model:
agent_llm: ${TAU2_AGENT_LLM:-openai/doubao-seed-2-0-pro-260215}
@@ -33,7 +48,10 @@ openviking:
url: ${OPENVIKING_URL:-http://localhost:1933}
account: ${OPENVIKING_ACCOUNT:-default}
agent_id: ${OPENVIKING_AGENT_ID:-tau2-openviking-agent}
+ reuse_corpus_across_runs: true
retrieval_top_k: 4
+ prewrite_retrieval_top_k: 6
+ prewrite_inject_top_k: 2
replay_write_policy: read_only
strategies:
diff --git a/benchmark/tau2/config/no_memory.yaml b/benchmark/tau2/config/no_memory.yaml
new file mode 100644
index 000000000..93f35633b
--- /dev/null
+++ b/benchmark/tau2/config/no_memory.yaml
@@ -0,0 +1,9 @@
+extends: baseline.yaml
+
+benchmark:
+ name: tau2_openviking_no_memory
+
+strategies:
+ - id: no_memory
+ label: TAU-2 no-memory baseline
+ memory_backend: none
diff --git a/benchmark/tau2/config/official.yaml b/benchmark/tau2/config/official.yaml
index d10bee872..9a47661ff 100644
--- a/benchmark/tau2/config/official.yaml
+++ b/benchmark/tau2/config/official.yaml
@@ -4,4 +4,9 @@ benchmark:
name: tau2_openviking_official_user_simulator
eval:
+ protocol: official_live_user
+ require_fixed_first_user: false
user_simulator_policy: official
+ fixed_first_user_fixtures:
+ retail:
+ airline:
diff --git a/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml b/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
new file mode 100644
index 000000000..bfb711e74
--- /dev/null
+++ b/benchmark/tau2/config/prb_content_matrix_new_prompt.yaml
@@ -0,0 +1,159 @@
+extends: baseline.yaml
+
+benchmark:
+ name: tau2_prb_content_matrix_new_prompt
+ strategy_concurrency: 16
+ task_max_concurrency: 5
+ corpus_prepare_concurrency: 1
+
+strategies:
+ - id: no_memory
+ label: TAU-2 no-memory same-seed baseline
+ memory_backend: none
+
+ - id: new_traj_fixed_first_user
+ label: PR-B new trajectory fixed-count first-user top4
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: trajectories
+ retrieval_mode: first_user
+ retrieval_top_k: 4
+ first_user_inject_top_k: 4
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_traj_fixed_prewrite_only
+ label: PR-B new trajectory fixed-count prewrite top4
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: trajectories
+ retrieval_mode: prewrite
+ retrieval_top_k: 4
+ prewrite_retrieval_top_k: 4
+ prewrite_inject_top_k: 4
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_traj_fixed_first_user_prewrite
+ label: PR-B new trajectory fixed-count first-user + prewrite top4
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: trajectories
+ retrieval_mode: first_user_prewrite
+ retrieval_top_k: 4
+ first_user_inject_top_k: 4
+ prewrite_retrieval_top_k: 4
+ prewrite_inject_top_k: 4
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_exp_fixed_first_user
+ label: PR-B new experience fixed-count first-user top2
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: experiences
+ retrieval_mode: first_user
+ retrieval_top_k: 2
+ first_user_inject_top_k: 2
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_exp_fixed_prewrite_only
+ label: PR-B new experience fixed-count prewrite top2
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: experiences
+ retrieval_mode: prewrite
+ retrieval_top_k: 2
+ prewrite_retrieval_top_k: 2
+ prewrite_inject_top_k: 2
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_exp_fixed_first_user_prewrite
+ label: PR-B new experience fixed-count first-user + prewrite top2
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: experiences
+ retrieval_mode: first_user_prewrite
+ retrieval_top_k: 2
+ first_user_inject_top_k: 2
+ prewrite_retrieval_top_k: 2
+ prewrite_inject_top_k: 2
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_traj_4000_prewrite_only
+ label: PR-B new trajectory 4000-char prewrite
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: trajectories
+ retrieval_mode: prewrite
+ retrieval_top_k: 8
+ prewrite_retrieval_top_k: 8
+ prewrite_inject_top_k: 8
+ memory_inject_max_chars: 4000
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+
+ - id: new_exp_4000_first_user_prewrite
+ label: PR-B new experience 4000-char first-user + prewrite
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_operation_family_v1_success_only
+ train_memory_mode: experience_only
+ train_transcript_format: role_tool_blocks
+ train_include_system_prompt: true
+ train_skip_failed_sessions: true
+ search_memory_type: experiences
+ retrieval_mode: first_user_prewrite
+ retrieval_top_k: 8
+ first_user_inject_top_k: 8
+ prewrite_retrieval_top_k: 8
+ prewrite_inject_top_k: 8
+ memory_inject_max_chars: 4000
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
diff --git a/benchmark/tau2/config/scope_prompts/airline_memory_scope.md b/benchmark/tau2/config/scope_prompts/airline_memory_scope.md
new file mode 100644
index 000000000..8847796a9
--- /dev/null
+++ b/benchmark/tau2/config/scope_prompts/airline_memory_scope.md
@@ -0,0 +1,18 @@
+<openviking_memory_scope_guard>
+OpenViking memories are advisory. Use them only when their trigger, preconditions,
+and applicability boundary match the current airline task.
+
+- Do not broaden the user's requested booking, cancellation, rebooking, flight
+ update, passenger update, baggage update, insurance, or payment scope because a
+ retrieved memory describes a nearby workflow.
+- Keep the current reservation scope explicit. Only use flights, passengers,
+ baggage entries, cabin changes, insurance choices, payment IDs, dates, and
+ amounts that are grounded in user input, recent tool observations, reservation
+ state, profile/payment state, or an explicit search/lookup result.
+- Before a write tool call, verify that the selected write action matches the
+ user's requested operation. Do not mix cancellation, rebooking, upgrade,
+ downgrade, baggage, or passenger-update flows unless the current task asks for
+ that combined operation.
+- If a memory and the current task disagree, follow the current task state and the
+ domain policy.
+</openviking_memory_scope_guard>
diff --git a/benchmark/tau2/config/scope_prompts/retail_memory_scope.md b/benchmark/tau2/config/scope_prompts/retail_memory_scope.md
new file mode 100644
index 000000000..65a8d61ff
--- /dev/null
+++ b/benchmark/tau2/config/scope_prompts/retail_memory_scope.md
@@ -0,0 +1,16 @@
+<openviking_memory_scope_guard>
+OpenViking memories are advisory. Use them only when their trigger, preconditions,
+and applicability boundary match the current retail task.
+
+- Do not broaden the user's requested replacement, return, exchange, cancellation,
+ address-change, or payment scope because a retrieved memory describes a nearby
+ workflow.
+- If the user restricts the request to the current order, same order, observed
+ order items, or a specific product variant, choose write arguments only from the
+ current tool observations or an explicitly requested catalog lookup.
+- Before a write tool call, order IDs, item IDs, new item IDs, payment method IDs,
+ addresses, amounts, and refund/payment direction must be grounded in user input,
+ recent tool observations, profile/order state, or an explicit catalog lookup.
+- If a memory and the current task disagree, follow the current task state and the
+ domain policy.
+</openviking_memory_scope_guard>
diff --git a/benchmark/tau2/config/trajectory.yaml b/benchmark/tau2/config/trajectory.yaml
new file mode 100644
index 000000000..40c378542
--- /dev/null
+++ b/benchmark/tau2/config/trajectory.yaml
@@ -0,0 +1,63 @@
+extends: baseline.yaml
+
+benchmark:
+ name: tau2_openviking_trajectory_view
+
+strategies:
+ - id: memory_v2_trajectory_view
+ label: OpenViking Memory V2 trajectory first-user recall
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_trajectory_view
+ train_memory_mode: experience_only
+ search_memory_type: trajectories
+ retrieval_mode: first_user
+ - id: memory_v2_trajectory_view_scope
+ label: OpenViking Memory V2 trajectory first-user recall with scope prompt
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_trajectory_view
+ train_memory_mode: experience_only
+ search_memory_type: trajectories
+ retrieval_mode: first_user
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+ - id: memory_v2_trajectory_prewrite_only
+ label: OpenViking Memory V2 trajectory pre-write-only recall
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_trajectory_view
+ train_memory_mode: experience_only
+ search_memory_type: trajectories
+ retrieval_mode: prewrite
+ - id: memory_v2_trajectory_prewrite
+ label: OpenViking Memory V2 trajectory first-user + pre-write recall
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_trajectory_view
+ train_memory_mode: experience_only
+ search_memory_type: trajectories
+ retrieval_mode: first_user_prewrite
+ - id: memory_v2_trajectory_prewrite_only_scope
+ label: OpenViking Memory V2 trajectory pre-write-only recall with scope prompt
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_trajectory_view
+ train_memory_mode: experience_only
+ search_memory_type: trajectories
+ retrieval_mode: prewrite
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
+ - id: memory_v2_trajectory_prewrite_scope
+ label: OpenViking Memory V2 trajectory first-user + pre-write recall with scope prompt
+ memory_backend: openviking
+ train_required: true
+ corpus_id: memory_v2_trajectory_view
+ train_memory_mode: experience_only
+ search_memory_type: trajectories
+ retrieval_mode: first_user_prewrite
+ scope_prompt_files:
+ retail: benchmark/tau2/config/scope_prompts/retail_memory_scope.md
+ airline: benchmark/tau2/config/scope_prompts/airline_memory_scope.md
diff --git a/benchmark/tau2/run_full_eval.sh b/benchmark/tau2/run_full_eval.sh
index ca69a7a32..728f3ce46 100755
--- a/benchmark/tau2/run_full_eval.sh
+++ b/benchmark/tau2/run_full_eval.sh
@@ -33,7 +33,7 @@ while [[ $# -gt 0 ]]; do
STRICT_PREFLIGHT=true
shift
;;
- --domain|--repeat-count|--strategy-id|--task-id|--num-tasks|--train-num-tasks)
+ --domain|--repeat-count|--strategy-id|--task-id|--num-tasks|--train-num-tasks|--strategy-concurrency)
RUN_EVAL_EXTRA+=("$1" "$2")
shift 2
;;
diff --git a/benchmark/tau2/scripts/run_eval.py b/benchmark/tau2/scripts/run_eval.py
index 5458ba61a..ea747045a 100755
--- a/benchmark/tau2/scripts/run_eval.py
+++ b/benchmark/tau2/scripts/run_eval.py
@@ -4,27 +4,33 @@ from __future__ import annotations
import argparse
import importlib.util
import json
+import os
import subprocess
import sys
+from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from typing import Any
from tau2_common import (
+ assert_tau2_results_complete,
domains,
load_config,
- output_dir,
normalize_litellm_env,
+ output_dir,
+ resolve_path,
run_id,
simulator_policy_report,
split_file,
strategy_ids,
- tau2_cli,
tau2_context,
tau2_repo,
user_simulator_policy,
write_json,
)
+TRAIN_TRANSCRIPT_OPENVIKING_TEXT = "openviking_text"
+DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS = 5000
+
def _reward(sim: dict[str, Any]) -> float:
info = sim.get("reward_info") or {}
@@ -46,8 +52,190 @@ def _db_match(sim: dict[str, Any]) -> bool | None:
return sim.get("db_match")
+def _strategy_int(
+ config: dict[str, Any],
+ strategy: dict[str, Any],
+ key: str,
+ *,
+ fallback_key: str | None = None,
+ default: int = 4,
+) -> int:
+ openviking = config.get("openviking", {})
+ value = strategy.get(key)
+ if value is None:
+ value = openviking.get(key)
+ if value is None and fallback_key:
+ value = strategy.get(fallback_key)
+ if value is None and fallback_key:
+ value = openviking.get(fallback_key)
+ if value is None:
+ value = default
+ return int(value)
+
+
+def _strategy_optional_int(
+ config: dict[str, Any],
+ strategy: dict[str, Any],
+ key: str,
+ *,
+ fallback_key: str | None = None,
+) -> int | None:
+ openviking = config.get("openviking", {})
+ value = strategy.get(key)
+ if value is None:
+ value = openviking.get(key)
+ if value is None and fallback_key:
+ value = strategy.get(fallback_key)
+ if value is None and fallback_key:
+ value = openviking.get(fallback_key)
+ if value is None:
+ return None
+ return int(value)
+
+
+def _enabled(value: Any) -> bool:
+ if isinstance(value, bool):
+ return value
+ if value is None:
+ return False
+ return str(value).strip().lower() in {"1", "true", "yes", "on"}
+
+
+def _require_fixed_first_user(config: dict[str, Any]) -> bool:
+ return _enabled(config.get("eval", {}).get("require_fixed_first_user"))
+
+
+def _retrieval_budget(config: dict[str, Any], strategy: dict[str, Any]) -> dict[str, int | None]:
+ retrieval_top_k = _strategy_int(config, strategy, "retrieval_top_k", default=4)
+ first_user_retrieval_top_k = _strategy_int(
+ config,
+ strategy,
+ "first_user_retrieval_top_k",
+ fallback_key="retrieval_top_k",
+ default=retrieval_top_k,
+ )
+ first_user_inject_top_k = _strategy_int(
+ config,
+ strategy,
+ "first_user_inject_top_k",
+ fallback_key="first_user_retrieval_top_k",
+ default=first_user_retrieval_top_k,
+ )
+ prewrite_retrieval_top_k = _strategy_int(
+ config,
+ strategy,
+ "prewrite_retrieval_top_k",
+ fallback_key="retrieval_top_k",
+ default=retrieval_top_k,
+ )
+ prewrite_inject_top_k = _strategy_int(
+ config,
+ strategy,
+ "prewrite_inject_top_k",
+ fallback_key="prewrite_retrieval_top_k",
+ default=prewrite_retrieval_top_k,
+ )
+ memory_inject_max_chars = _strategy_optional_int(
+ config,
+ strategy,
+ "memory_inject_max_chars",
+ )
+ first_user_memory_inject_max_chars = _strategy_optional_int(
+ config,
+ strategy,
+ "first_user_memory_inject_max_chars",
+ fallback_key="memory_inject_max_chars",
+ )
+ prewrite_memory_inject_max_chars = _strategy_optional_int(
+ config,
+ strategy,
+ "prewrite_memory_inject_max_chars",
+ fallback_key="memory_inject_max_chars",
+ )
+ for key, value in {
+ "memory_inject_max_chars": memory_inject_max_chars,
+ "first_user_memory_inject_max_chars": first_user_memory_inject_max_chars,
+ "prewrite_memory_inject_max_chars": prewrite_memory_inject_max_chars,
+ }.items():
+ if value is not None and value < 0:
+ raise ValueError(f"{strategy['id']} has negative {key}: {value}")
+ return {
+ "retrieval_top_k": retrieval_top_k,
+ "first_user_retrieval_top_k": first_user_retrieval_top_k,
+ "first_user_inject_top_k": first_user_inject_top_k,
+ "prewrite_retrieval_top_k": prewrite_retrieval_top_k,
+ "prewrite_inject_top_k": prewrite_inject_top_k,
+ "memory_inject_max_chars": memory_inject_max_chars,
+ "first_user_memory_inject_max_chars": first_user_memory_inject_max_chars,
+ "prewrite_memory_inject_max_chars": prewrite_memory_inject_max_chars,
+ }
+
+
+def _memory_corpus_key_for(
+ *,
+ domain: str,
+ strategy: dict[str, Any],
+ train_num_tasks: int | None,
+) -> str:
+ corpus_id = str(strategy.get("corpus_id") or strategy["id"])
+ raw_key = strategy.get("corpus_cache_key")
+ if raw_key:
+ key = str(raw_key).format(
+ domain=domain,
+ strategy_id=strategy["id"],
+ corpus_id=corpus_id,
+ )
+ else:
+ key = f"{domain}_{corpus_id}"
+ if train_num_tasks is not None:
+ key = f"{key}_train{train_num_tasks}"
+ return key
+
+
+def _memory_corpus_dir(config: dict[str, Any], configured_run_id: str, corpus_key: str) -> Path:
+ raw = config.get("paths", {}).get("corpus_cache_dir")
+ if raw:
+ return resolve_path(str(raw)) / corpus_key
+ return output_dir(config, configured_run_id) / "memory_corpora" / corpus_key
+
+
+def _search_uri(agent_id: str, search_memory_type: str) -> str:
+ return f"viking://agent/{agent_id}/memories/{search_memory_type}"
+
+
+def _train_transcript_format(strategy: dict[str, Any]) -> str:
+ return str(strategy.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT)
+
+
+def _train_tool_output_max_chars(strategy: dict[str, Any]) -> int:
+ raw = strategy.get("train_tool_output_max_chars")
+ if raw is None:
+ return DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS
+ return int(raw)
+
+
+def _train_skip_failed_sessions(strategy: dict[str, Any]) -> bool:
+ return _enabled(strategy.get("train_skip_failed_sessions"))
+
+
+def _manifest_openviking_identity(corpus_dir: Path) -> dict[str, str] | None:
+ manifest_path = corpus_dir / "corpus_manifest.json"
+ if not manifest_path.is_file():
+ return None
+ try:
+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+ except json.JSONDecodeError:
+ return None
+ openviking = manifest.get("openviking") or {}
+ required = ("account", "user", "agent_id", "search_uri")
+ if not all(openviking.get(key) for key in required):
+ return None
+ return {key: str(openviking[key]) for key in required}
+
+
def _metrics_from_tau2_results(results_path: Path) -> dict[str, Any]:
data = json.loads(results_path.read_text(encoding="utf-8"))
+ assert_tau2_results_complete(data, context=str(results_path))
sims = data.get("simulations") or []
rewards = [_reward(sim) for sim in sims]
db_values = [_db_match(sim) for sim in sims]
@@ -55,7 +243,9 @@ def _metrics_from_tau2_results(results_path: Path) -> dict[str, Any]:
return {
"simulation_count": len(sims),
"avg_reward": sum(rewards) / len(rewards) if rewards else 0.0,
- "db_match_rate": (sum(1 for value in db_known if value) / len(db_known)) if db_known else None,
+ "db_match_rate": (sum(1 for value in db_known if value) / len(db_known))
+ if db_known
+ else None,
}
@@ -80,6 +270,8 @@ def _tau2_command(
if reasoning_effort:
agent_llm_args = f'{{"temperature":0.0,"reasoning_effort":"{reasoning_effort}"}}'
user_llm_args = f'{{"temperature":0.0,"reasoning_effort":"{reasoning_effort}"}}'
+ fixed_first_user_file = _fixed_first_user_file(config, domain)
+ scope_prompt_file = _scope_prompt_file(config, strategy, domain)
if (
strategy.get("memory_backend") == "openviking"
@@ -87,10 +279,39 @@ def _tau2_command(
):
openviking = config["openviking"]
corpus_id = str(strategy.get("corpus_id") or strategy["id"])
- account = f"{openviking['account']}-{configured_run_id}-{domain}-{corpus_id}"
- agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"
- user = f"tau2-{domain}-{corpus_id}"
- search_uri = f"viking://agent/{agent_id}/memories/experiences"
+ resolved_train_num_tasks = (
+ train_num_tasks if train_num_tasks is not None else strategy.get("train_num_tasks")
+ )
+ corpus_key = _memory_corpus_key_for(
+ domain=domain,
+ strategy=strategy,
+ train_num_tasks=resolved_train_num_tasks,
+ )
+ corpus_dir = _memory_corpus_dir(config, configured_run_id, corpus_key)
+ reuse_identity = _manifest_openviking_identity(corpus_dir)
+ if reuse_identity is not None:
+ account = reuse_identity["account"]
+ agent_id = reuse_identity["agent_id"]
+ user = reuse_identity["user"]
+ search_uri = ""
+ elif openviking.get("reuse_corpus_across_runs", False):
+ account = f"{openviking['account']}-{corpus_key}"
+ agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"
+ user = f"tau2-{domain}-{corpus_id}"
+ search_uri = ""
+ else:
+ account = f"{openviking['account']}-{configured_run_id}-{domain}-{corpus_id}"
+ agent_id = f"{openviking['agent_id']}-{domain}-{corpus_id}"
+ user = f"tau2-{domain}-{corpus_id}"
+ search_uri = ""
+ search_memory_type = str(strategy.get("search_memory_type", "experiences"))
+ if search_memory_type not in {"experiences", "trajectories"}:
+ raise ValueError(
+ f"Unsupported search_memory_type for {strategy['id']}: {search_memory_type}"
+ )
+ if not search_uri:
+ search_uri = _search_uri(agent_id, search_memory_type)
+ budget = _retrieval_budget(config, strategy)
command = [
sys.executable,
str(Path(__file__).with_name("run_memory_v2_eval.py")),
@@ -99,11 +320,7 @@ def _tau2_command(
"--run-dir",
str(output_dir(config, configured_run_id) / "memory_cells" / run_label),
"--corpus-dir",
- str(
- output_dir(config, configured_run_id)
- / "memory_corpora"
- / f"{domain}_{corpus_id}"
- ),
+ str(corpus_dir),
"--run-label",
run_label,
"--strategy-id",
@@ -137,64 +354,135 @@ def _tau2_command(
"--search-uri",
search_uri,
"--retrieval-top-k",
- str(openviking.get("retrieval_top_k", 4)),
+ str(budget["retrieval_top_k"]),
+ "--first-user-retrieval-top-k",
+ str(budget["first_user_retrieval_top_k"]),
+ "--first-user-inject-top-k",
+ str(budget["first_user_inject_top_k"]),
+ "--prewrite-retrieval-top-k",
+ str(budget["prewrite_retrieval_top_k"]),
+ "--prewrite-inject-top-k",
+ str(budget["prewrite_inject_top_k"]),
"--retrieval-mode",
str(strategy.get("retrieval_mode", "first_user")),
+ "--train-transcript-format",
+ _train_transcript_format(strategy),
+ "--train-tool-output-max-chars",
+ str(_train_tool_output_max_chars(strategy)),
"--seed",
str(seed),
]
+ if budget["memory_inject_max_chars"] is not None:
+ command.extend(["--memory-inject-max-chars", str(budget["memory_inject_max_chars"])])
+ if budget["first_user_memory_inject_max_chars"] is not None:
+ command.extend(
+ [
+ "--first-user-memory-inject-max-chars",
+ str(budget["first_user_memory_inject_max_chars"]),
+ ]
+ )
+ if budget["prewrite_memory_inject_max_chars"] is not None:
+ command.extend(
+ [
+ "--prewrite-memory-inject-max-chars",
+ str(budget["prewrite_memory_inject_max_chars"]),
+ ]
+ )
+ if _enabled(strategy.get("train_include_system_prompt")):
+ command.append("--train-include-system-prompt")
+ if _train_skip_failed_sessions(strategy):
+ command.append("--train-skip-failed-sessions")
+ if fixed_first_user_file is not None:
+ command.extend(["--fixed-first-user-file", str(fixed_first_user_file)])
+ if scope_prompt_file is not None:
+ command.extend(["--scope-prompt-file", str(scope_prompt_file)])
if task_ids:
for task_id in task_ids:
command.extend(["--task-id", task_id])
elif num_tasks is not None:
command.extend(["--num-tasks", str(num_tasks)])
- train_num_tasks = train_num_tasks if train_num_tasks is not None else strategy.get("train_num_tasks")
- if train_num_tasks is not None:
- command.extend(["--train-num-tasks", str(train_num_tasks)])
+ if resolved_train_num_tasks is not None:
+ command.extend(["--train-num-tasks", str(resolved_train_num_tasks)])
return command
if strategy.get("memory_backend") != "none":
return None
command = [
- tau2_cli(config),
- "run",
+ sys.executable,
+ str(Path(__file__).with_name("run_memory_v2_eval.py")),
+ "--tau2-repo",
+ str(tau2_repo(config)),
+ "--run-dir",
+ str(output_dir(config, configured_run_id) / "memory_cells" / run_label),
+ "--run-label",
+ run_label,
+ "--strategy-id",
+ strategy["id"],
"--domain",
domain,
- "--agent",
- str(benchmark.get("agent", "llm_agent")),
- "--user",
- str(benchmark.get("user", "user_simulator")),
- "--task-split-name",
+ "--eval-split-name",
str(benchmark.get("eval_split_name", "test")),
- "--num-trials",
- "1",
"--max-steps",
str(benchmark.get("max_steps", 200)),
"--max-concurrency",
str(benchmark.get("task_max_concurrency", 10)),
+ "--base-agent",
+ str(benchmark.get("agent", "llm_agent")),
+ "--user",
+ str(benchmark.get("user", "user_simulator")),
"--agent-llm",
str(model["agent_llm"]),
"--user-llm",
str(model["user_llm"]),
- "--save-to",
- run_label,
+ "--agent-llm-args",
+ agent_llm_args,
+ "--user-llm-args",
+ user_llm_args,
"--seed",
str(seed),
+ "--no-memory",
]
-
- command.extend(["--agent-llm-args", agent_llm_args])
- command.extend(["--user-llm-args", user_llm_args])
+ if fixed_first_user_file is not None:
+ command.extend(["--fixed-first-user-file", str(fixed_first_user_file)])
if task_ids:
- command.append("--task-ids")
- command.extend(task_ids)
+ for task_id in task_ids:
+ command.extend(["--task-id", task_id])
elif num_tasks is not None:
command.extend(["--num-tasks", str(num_tasks)])
return command
+def _fixed_first_user_file(config: dict[str, Any], domain: str) -> Path | None:
+ raw = config.get("eval", {}).get("fixed_first_user_fixture")
+ if raw is None:
+ raw = config.get("eval", {}).get("fixed_first_user_fixtures")
+ if isinstance(raw, dict):
+ raw = raw.get(domain) or raw.get("default")
+ if raw is None or str(raw).strip() == "":
+ return None
+ return resolve_path(str(raw))
+
+
+def _scope_prompt_file(
+ config: dict[str, Any], strategy: dict[str, Any], domain: str
+) -> Path | None:
+ raw = strategy.get("scope_prompt_file")
+ if raw is None:
+ raw = strategy.get("scope_prompt_files")
+ if raw is None:
+ raw = config.get("openviking", {}).get("scope_prompt_file")
+ if raw is None:
+ raw = config.get("openviking", {}).get("scope_prompt_files")
+ if isinstance(raw, dict):
+ raw = raw.get(domain) or raw.get("default")
+ if raw is None or str(raw).strip() == "":
+ return None
+ return resolve_path(str(raw))
+
+
def _build_plan(
config: dict[str, Any],
configured_run_id: str,
@@ -205,16 +493,30 @@ def _build_plan(
num_tasks: int | None,
train_num_tasks: int | None,
repeat_count_override: int | None,
+ cell_concurrency_override: int | None,
+ strategy_concurrency_override: int | None,
) -> dict[str, Any]:
repeat_count = repeat_count_override or int(config["benchmark"].get("repeat_count", 8))
base_seed = int(config["benchmark"].get("seed", 300))
+ cell_timeout_seconds = int(config["benchmark"].get("cell_timeout_seconds", 0) or 0)
+ strategy_concurrency = strategy_concurrency_override
+ if strategy_concurrency is None:
+ strategy_concurrency = cell_concurrency_override
+ if strategy_concurrency is None:
+ strategy_concurrency = config["benchmark"].get("strategy_concurrency")
+ if strategy_concurrency is None:
+ strategy_concurrency = config["benchmark"].get("cell_concurrency", 1)
+ strategy_concurrency = max(1, int(strategy_concurrency or 1))
policy_report = simulator_policy_report(config)
+ require_fixed_first_user = _require_fixed_first_user(config)
strategies = config.get("strategies") or []
if selected_strategy_ids:
unknown = selected_strategy_ids - set(strategy_ids(config))
if unknown:
raise ValueError(f"unknown strategy ids: {sorted(unknown)}")
- strategies = [strategy for strategy in strategies if strategy["id"] in selected_strategy_ids]
+ strategies = [
+ strategy for strategy in strategies if strategy["id"] in selected_strategy_ids
+ ]
cells = []
plan_domains = domains(config)
if selected_domains:
@@ -222,6 +524,18 @@ def _build_plan(
if unknown_domains:
raise ValueError(f"unknown domains: {sorted(unknown_domains)}")
plan_domains = [domain for domain in plan_domains if domain in selected_domains]
+ if require_fixed_first_user:
+ missing_domains = [
+ domain for domain in plan_domains if _fixed_first_user_file(config, domain) is None
+ ]
+ if missing_domains:
+ protocol = config.get("eval", {}).get("protocol", "fixed_first_user_full8")
+ raise ValueError(
+ f"eval protocol {protocol!r} requires fixed-first-user fixtures for "
+ f"{missing_domains}; set TAU2_RETAIL_FIXED_FIRST_USER_FILE and "
+ "TAU2_AIRLINE_FIXED_FIRST_USER_FILE, or use a config that explicitly "
+ "sets eval.require_fixed_first_user=false"
+ )
for domain in plan_domains:
split_path = split_file(config, domain)
for strategy in strategies:
@@ -239,6 +553,8 @@ def _build_plan(
train_num_tasks=train_num_tasks,
seed=seed,
)
+ fixed_first_user_file = _fixed_first_user_file(config, domain)
+ scope_prompt_file = _scope_prompt_file(config, strategy, domain)
non_executable_reason = None
if command is None:
non_executable_reason = (
@@ -256,11 +572,47 @@ def _build_plan(
"train_required": bool(strategy.get("train_required")),
"memory_backend": strategy.get("memory_backend"),
"corpus_id": strategy.get("corpus_id", strategy["id"]),
+ "corpus_key": _memory_corpus_key_for(
+ domain=domain,
+ strategy=strategy,
+ train_num_tasks=(
+ train_num_tasks
+ if train_num_tasks is not None
+ else strategy.get("train_num_tasks")
+ ),
+ ),
+ "corpus_dir": str(
+ _memory_corpus_dir(
+ config,
+ configured_run_id,
+ _memory_corpus_key_for(
+ domain=domain,
+ strategy=strategy,
+ train_num_tasks=(
+ train_num_tasks
+ if train_num_tasks is not None
+ else strategy.get("train_num_tasks")
+ ),
+ ),
+ )
+ ),
"retrieval_mode": strategy.get("retrieval_mode"),
+ "train_transcript_format": _train_transcript_format(strategy),
+ "train_include_system_prompt": _enabled(
+ strategy.get("train_include_system_prompt")
+ ),
+ "train_skip_failed_sessions": _train_skip_failed_sessions(strategy),
+ "train_tool_output_max_chars": _train_tool_output_max_chars(strategy),
+ "retrieval_budget": _retrieval_budget(config, strategy),
+ "search_memory_type": strategy.get("search_memory_type", "experiences"),
"adapter_status": strategy.get("adapter_status", "ready"),
"executable": command is not None,
"user_simulator_policy": user_simulator_policy(config),
"user_simulator_policy_supported": policy_report["supported"],
+ "fixed_first_user_file": str(fixed_first_user_file)
+ if fixed_first_user_file
+ else None,
+ "scope_prompt_file": str(scope_prompt_file) if scope_prompt_file else None,
"split_file": str(split_path),
"command": command,
"non_executable_reason": non_executable_reason,
@@ -274,35 +626,39 @@ def _build_plan(
"strategy_ids": strategy_ids(config),
"domains": plan_domains,
"tau2": tau2_context(config),
+ "eval_protocol": config.get("eval", {}).get("protocol"),
+ "require_fixed_first_user": require_fixed_first_user,
"simulator_policy": policy_report,
"cell_count": len(cells),
"executable_cell_count": executable_cell_count,
"pending_cell_count": len(cells) - executable_cell_count,
+ "corpus_prepare_concurrency": int(config["benchmark"].get("corpus_prepare_concurrency", 1)),
+ "strategy_concurrency": strategy_concurrency,
+ "cell_concurrency": strategy_concurrency,
+ "cell_timeout_seconds": cell_timeout_seconds or None,
"cells": cells,
}
def _cell_artifacts(cell: dict[str, Any], repo: Path, out: Path) -> dict[str, str]:
- if cell.get("memory_backend") == "openviking":
+ if cell.get("memory_backend") in {"openviking", "none"}:
run_dir = out / "memory_cells" / cell["run_label"]
- corpus_id = str(cell.get("corpus_id") or cell["strategy_id"])
- corpus_dir = out / "memory_corpora" / f"{cell['domain']}_{corpus_id}"
- return {
+ artifacts = {
"summary": str(run_dir / f"{cell['run_label']}.summary.json"),
"results": str(run_dir / f"{cell['run_label']}.json"),
- "retrieval_trace": str(run_dir / f"{cell['run_label']}.retrieval_trace.jsonl"),
- "corpus_manifest": str(corpus_dir / "corpus_manifest.json"),
}
- return {
- "results": str(repo / "data" / "simulations" / f"{cell['run_label']}.json")
- }
+ if cell.get("memory_backend") == "none":
+ return artifacts
+ corpus_dir = Path(cell["corpus_dir"])
+ artifacts["retrieval_trace"] = str(run_dir / f"{cell['run_label']}.retrieval_trace.jsonl")
+ artifacts["corpus_manifest"] = str(corpus_dir / "corpus_manifest.json")
+ return artifacts
+ return {"results": str(repo / "data" / "simulations" / f"{cell['run_label']}.json")}
def _cell_metrics(cell: dict[str, Any], artifacts: dict[str, str]) -> dict[str, Any] | None:
- if cell.get("memory_backend") == "openviking":
- summary_path = Path(artifacts["summary"])
- if not summary_path.is_file():
- return None
+ summary_path = Path(artifacts.get("summary", ""))
+ if summary_path.is_file():
summary = json.loads(summary_path.read_text(encoding="utf-8"))
return summary.get("metrics")
@@ -312,6 +668,129 @@ def _cell_metrics(cell: dict[str, Any], artifacts: dict[str, str]) -> dict[str,
return _metrics_from_tau2_results(results_path)
+def _memory_corpus_key(cell: dict[str, Any]) -> str:
+ return str(cell.get("corpus_key") or f"{cell['domain']}_{cell['corpus_id']}")
+
+
+def _tau2_subprocess_env(repo: Path) -> dict[str, str]:
+ env = os.environ.copy()
+ src = repo / "src"
+ pythonpath_entry = str(src if src.is_dir() else repo)
+ existing = env.get("PYTHONPATH")
+ env["PYTHONPATH"] = (
+ pythonpath_entry if not existing else f"{pythonpath_entry}{os.pathsep}{existing}"
+ )
+ return env
+
+
+def _prepare_memory_corpus(cell: dict[str, Any], repo: Path, out: Path) -> dict[str, Any]:
+ key = _memory_corpus_key(cell)
+ manifest_path = Path(cell["corpus_dir"]) / "corpus_manifest.json"
+ if manifest_path.is_file():
+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+ cached_transcript_format = str(
+ manifest.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT
+ )
+ requested_transcript_format = str(
+ cell.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT
+ )
+ if cached_transcript_format != requested_transcript_format:
+ raise RuntimeError(
+ "cached corpus train_transcript_format mismatch for "
+ f"{key}: {cached_transcript_format!r} != {requested_transcript_format!r}; "
+ "use a distinct corpus_id or rebuild the corpus"
+ )
+ cached_include_system_prompt = bool(manifest.get("train_include_system_prompt") or False)
+ requested_include_system_prompt = bool(cell.get("train_include_system_prompt") or False)
+ if cached_include_system_prompt != requested_include_system_prompt:
+ raise RuntimeError(
+ "cached corpus train_include_system_prompt mismatch for "
+ f"{key}: {cached_include_system_prompt!r} != {requested_include_system_prompt!r}; "
+ "use a distinct corpus_id or rebuild the corpus"
+ )
+ cached_tool_output_max_chars = int(
+ manifest.get("train_tool_output_max_chars") or DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS
+ )
+ requested_tool_output_max_chars = int(
+ cell.get("train_tool_output_max_chars") or DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS
+ )
+ if cached_tool_output_max_chars != requested_tool_output_max_chars:
+ raise RuntimeError(
+ "cached corpus train_tool_output_max_chars mismatch for "
+ f"{key}: {cached_tool_output_max_chars!r} != {requested_tool_output_max_chars!r}; "
+ "use a distinct corpus_id or rebuild the corpus"
+ )
+ cached_skip_failed = bool(manifest.get("train_skip_failed_sessions") or False)
+ requested_skip_failed = bool(cell.get("train_skip_failed_sessions") or False)
+ if cached_skip_failed != requested_skip_failed:
+ raise RuntimeError(
+ "cached corpus train_skip_failed_sessions mismatch for "
+ f"{key}: {cached_skip_failed!r} != {requested_skip_failed!r}; "
+ "use a distinct corpus_id or rebuild the corpus"
+ )
+ row = {
+ "domain": cell["domain"],
+ "strategy_id": cell["strategy_id"],
+ "corpus_id": str(cell.get("corpus_id") or cell["strategy_id"]),
+ "corpus_key": key,
+ "returncode": 0,
+ "reused": True,
+ "artifacts": {"corpus_manifest": str(manifest_path)},
+ }
+ write_json(out / "corpus_prepare_results" / f"{key}.json", row)
+ print(f"[tau2] reusing corpus {key}", flush=True)
+ return row
+ command = list(cell["command"]) + ["--prepare-corpus-only"]
+ print(f"[tau2] preparing corpus {key}", flush=True)
+ completed = subprocess.run(
+ command,
+ cwd=repo,
+ env=_tau2_subprocess_env(repo),
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ check=False,
+ )
+ row = {
+ "domain": cell["domain"],
+ "strategy_id": cell["strategy_id"],
+ "corpus_id": str(cell.get("corpus_id") or cell["strategy_id"]),
+ "corpus_key": key,
+ "returncode": completed.returncode,
+ "stdout_tail": completed.stdout[-4000:],
+ "stderr_tail": completed.stderr[-4000:],
+ "artifacts": {"corpus_manifest": str(Path(cell["corpus_dir"]) / "corpus_manifest.json")},
+ }
+ write_json(out / "corpus_prepare_results" / f"{key}.json", row)
+ if completed.returncode != 0:
+ raise RuntimeError(f"corpus prepare failed: {key} returncode={completed.returncode}")
+ return row
+
+
+def _prepare_memory_corpora(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str, Any]]:
+ corpus_cells: dict[str, dict[str, Any]] = {}
+ for cell in plan["cells"]:
+ if cell.get("memory_backend") != "openviking" or not cell.get("train_required"):
+ continue
+ corpus_cells.setdefault(_memory_corpus_key(cell), cell)
+ if not corpus_cells:
+ return []
+
+ worker_count = max(1, int(plan.get("corpus_prepare_concurrency") or 1))
+ if worker_count == 1 or len(corpus_cells) == 1:
+ return [_prepare_memory_corpus(cell, repo, out) for cell in corpus_cells.values()]
+
+ rows: list[dict[str, Any]] = []
+ with ThreadPoolExecutor(max_workers=worker_count) as executor:
+ futures = {
+ executor.submit(_prepare_memory_corpus, cell, repo, out): key
+ for key, cell in corpus_cells.items()
+ }
+ for future in as_completed(futures):
+ rows.append(future.result())
+ return rows
+
+
def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:
def weighted(rows_for_group: list[dict[str, Any]]) -> dict[str, Any]:
metric_rows = [row for row in rows_for_group if row.get("metrics")]
@@ -327,7 +806,9 @@ def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:
if row["metrics"].get("db_match_rate") is not None
and int(row["metrics"].get("simulation_count") or 0) > 0
]
- db_weight = sum(int(row["metrics"].get("simulation_count") or 0) for row in db_weighted_rows)
+ db_weight = sum(
+ int(row["metrics"].get("simulation_count") or 0) for row in db_weighted_rows
+ )
db_sum = sum(
float(row["metrics"]["db_match_rate"])
* int(row["metrics"].get("simulation_count") or 0)
@@ -367,6 +848,64 @@ def _summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:
}
+def _execute_cell(
+ cell: dict[str, Any], repo: Path, out: Path, cell_timeout: int | None
+) -> dict[str, Any]:
+ cell_result_path = out / "cell_results" / f"{cell['run_label']}.json"
+ if cell_result_path.is_file():
+ existing_row = json.loads(cell_result_path.read_text(encoding="utf-8"))
+ if existing_row.get("returncode") == 0 and existing_row.get("metrics"):
+ print(f"[tau2] skipping completed {cell['run_label']}", flush=True)
+ return existing_row
+
+ print(f"[tau2] running {cell['run_label']}", flush=True)
+ try:
+ completed = subprocess.run(
+ cell["command"],
+ cwd=repo,
+ env=_tau2_subprocess_env(repo),
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ check=False,
+ timeout=cell_timeout,
+ )
+ except subprocess.TimeoutExpired as exc:
+ stdout = exc.stdout or ""
+ stderr = exc.stderr or ""
+ if isinstance(stdout, bytes):
+ stdout = stdout.decode(errors="replace")
+ if isinstance(stderr, bytes):
+ stderr = stderr.decode(errors="replace")
+ row = {
+ "run_label": cell["run_label"],
+ "domain": cell["domain"],
+ "strategy_id": cell["strategy_id"],
+ "returncode": 124,
+ "timed_out": True,
+ "timeout_seconds": cell_timeout,
+ "stdout_tail": stdout[-4000:],
+ "stderr_tail": stderr[-4000:],
+ "artifacts": _cell_artifacts(cell, repo, out),
+ "metrics": None,
+ }
+ write_json(cell_result_path, row)
+ return row
+
+ row = {
+ "run_label": cell["run_label"],
+ "domain": cell["domain"],
+ "strategy_id": cell["strategy_id"],
+ "returncode": completed.returncode,
+ "stdout_tail": completed.stdout[-4000:],
+ "stderr_tail": completed.stderr[-4000:],
+ }
+ row["artifacts"] = _cell_artifacts(cell, repo, out)
+ row["metrics"] = _cell_metrics(cell, row["artifacts"])
+ write_json(cell_result_path, row)
+ return row
+
+
def _execute_cells(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str, Any]]:
policy_report = plan.get("simulator_policy") or {}
if not policy_report.get("supported", False):
@@ -374,39 +913,42 @@ def _execute_cells(plan: dict[str, Any], repo: Path, out: Path) -> list[dict[str
"configured user simulator policy is not supported by this TAU-2 checkout: "
f"{policy_report}"
)
- rows = []
+ _prepare_memory_corpora(plan, repo, out)
+ cells = []
for cell in plan["cells"]:
if not cell.get("executable"):
raise RuntimeError(
f"cell is not executable yet: {cell['run_label']} "
f"(strategy_id={cell['strategy_id']}, adapter_status={cell.get('adapter_status')})"
)
- print(f"[tau2] running {cell['run_label']}")
- completed = subprocess.run(
- cell["command"],
- cwd=repo,
- text=True,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- check=False,
- )
- row = {
- "run_label": cell["run_label"],
- "domain": cell["domain"],
- "strategy_id": cell["strategy_id"],
- "returncode": completed.returncode,
- "stdout_tail": completed.stdout[-4000:],
- "stderr_tail": completed.stderr[-4000:],
+ cells.append(cell)
+
+ cell_timeout = int(plan.get("cell_timeout_seconds") or 0) or None
+ worker_count = max(
+ 1, int(plan.get("strategy_concurrency") or plan.get("cell_concurrency") or 1)
+ )
+ if worker_count == 1 or len(cells) == 1:
+ return [_execute_cell(cell, repo, out, cell_timeout) for cell in cells]
+
+ print(f"[tau2] running eval cells with concurrency={worker_count}", flush=True)
+ rows: list[dict[str, Any]] = []
+ with ThreadPoolExecutor(max_workers=worker_count) as executor:
+ futures = {
+ executor.submit(_execute_cell, cell, repo, out, cell_timeout): cell for cell in cells
}
- row["artifacts"] = _cell_artifacts(cell, repo, out)
- row["metrics"] = _cell_metrics(cell, row["artifacts"])
- rows.append(row)
- write_json(out / "cell_results" / f"{cell['run_label']}.json", row)
- if completed.returncode != 0:
- raise RuntimeError(f"cell failed: {cell['run_label']} returncode={completed.returncode}")
+ for future in as_completed(futures):
+ rows.append(future.result())
return rows
+def _execution_failures(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ return [
+ row
+ for row in rows
+ if row.get("returncode") != 0 or row.get("timed_out") or not row.get("metrics")
+ ]
+
+
def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:
errors: list[str] = []
llm_env = normalize_litellm_env()
@@ -419,19 +961,36 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:
if strict and not llm_env["has_api_key"]:
errors.append("missing LLM API key: set OPENAI_API_KEY or ARK_API_KEY")
if strict and not llm_env["has_base_url"]:
- errors.append("missing OpenAI-compatible base URL: set OPENAI_API_BASE, OPENAI_BASE_URL, or ARK_BASE_URL")
+ errors.append(
+ "missing OpenAI-compatible base URL: set OPENAI_API_BASE, OPENAI_BASE_URL, or ARK_BASE_URL"
+ )
if strict and not policy_report["supported"]:
errors.append(
"configured confirmation-aware user simulator policy requires a TAU-2 "
f"checkout with the prompt fix: {policy_report['prompt_files']}"
)
split_rows = []
+ fixture_rows = []
for domain in domains(config):
path = split_file(config, domain)
exists = path.is_file()
split_rows.append({"domain": domain, "path": str(path), "exists": exists})
if strict and not exists:
errors.append(f"missing split file for {domain}: {path}")
+ fixed_first_user_file = _fixed_first_user_file(config, domain)
+ fixture_exists = fixed_first_user_file.is_file() if fixed_first_user_file else False
+ fixture_rows.append(
+ {
+ "domain": domain,
+ "path": str(fixed_first_user_file) if fixed_first_user_file else None,
+ "exists": fixture_exists,
+ }
+ )
+ if strict and _require_fixed_first_user(config) and not fixture_exists:
+ errors.append(
+ "missing fixed-first-user fixture for "
+ f"{domain}: set TAU2_{domain.upper()}_FIXED_FIRST_USER_FILE"
+ )
import_rows = []
for module in ("openviking", "openviking_cli", "tau2"):
@@ -444,12 +1003,15 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:
"status": "failed" if errors else "ok",
"strict": strict,
"tau2": tau2_info,
+ "eval_protocol": config.get("eval", {}).get("protocol"),
+ "require_fixed_first_user": _require_fixed_first_user(config),
"llm_env": llm_env,
"simulator_policy": policy_report,
"domains": domains(config),
"strategies": strategy_ids(config),
"imports": import_rows,
"split_files": split_rows,
+ "fixed_first_user_fixtures": fixture_rows,
"errors": errors,
}
write_json(out / "preflight.json", report)
@@ -463,14 +1025,38 @@ def _preflight(config: dict[str, Any], out: Path, *, strict: bool) -> int:
def main() -> int:
parser = argparse.ArgumentParser(description="Plan or run TAU-2 benchmark cells.")
- parser.add_argument("--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml")
+ parser.add_argument(
+ "--config", type=Path, default=Path(__file__).parents[1] / "config" / "baseline.yaml"
+ )
parser.add_argument("--run-id", default=run_id())
- parser.add_argument("--domain", action="append", help="Run only this configured domain; may be repeated.")
- parser.add_argument("--repeat-count", type=int, help="Override benchmark.repeat_count for smoke runs.")
- parser.add_argument("--strategy-id", action="append", help="Run only this strategy id; may be repeated.")
- parser.add_argument("--task-id", action="append", help="Run only this TAU-2 task id; may be repeated.")
- parser.add_argument("--num-tasks", type=int, help="Run the first N tasks from the selected split.")
- parser.add_argument("--train-num-tasks", type=int, help="Train OpenViking memory on the first N train tasks.")
+ parser.add_argument(
+ "--domain", action="append", help="Run only this configured domain; may be repeated."
+ )
+ parser.add_argument(
+ "--repeat-count", type=int, help="Override benchmark.repeat_count for smoke runs."
+ )
+ parser.add_argument(
+ "--cell-concurrency",
+ type=int,
+ help="Deprecated alias for --strategy-concurrency.",
+ )
+ parser.add_argument(
+ "--strategy-concurrency",
+ type=int,
+ help="Override benchmark.strategy_concurrency for parallel matrix cells.",
+ )
+ parser.add_argument(
+ "--strategy-id", action="append", help="Run only this strategy id; may be repeated."
+ )
+ parser.add_argument(
+ "--task-id", action="append", help="Run only this TAU-2 task id; may be repeated."
+ )
+ parser.add_argument(
+ "--num-tasks", type=int, help="Run the first N tasks from the selected split."
+ )
+ parser.add_argument(
+ "--train-num-tasks", type=int, help="Train OpenViking memory on the first N train tasks."
+ )
parser.add_argument(
"--preflight",
action="store_true",
@@ -488,6 +1074,10 @@ def main() -> int:
if args.plan_only and args.execute:
raise SystemExit("--plan-only and --execute are mutually exclusive")
+ if args.cell_concurrency is not None and args.cell_concurrency < 1:
+ raise SystemExit("--cell-concurrency must be >= 1")
+ if args.strategy_concurrency is not None and args.strategy_concurrency < 1:
+ raise SystemExit("--strategy-concurrency must be >= 1")
config = load_config(args.config)
out = output_dir(config, args.run_id)
@@ -506,6 +1096,8 @@ def main() -> int:
num_tasks=args.num_tasks,
train_num_tasks=args.train_num_tasks,
repeat_count_override=args.repeat_count,
+ cell_concurrency_override=args.cell_concurrency,
+ strategy_concurrency_override=args.strategy_concurrency,
)
write_json(out / "run_plan.json", plan)
write_json(out / "resolved_config.json", config)
@@ -514,10 +1106,15 @@ def main() -> int:
if args.execute:
try:
rows = _execute_cells(plan, tau2_repo(config), out)
- plan["status"] = "succeeded"
+ failures = _execution_failures(rows)
+ plan["status"] = "failed" if failures else "succeeded"
plan["executed_cell_count"] = len(rows)
+ plan["failed_cell_count"] = len(failures)
write_json(out / "run_plan.json", plan)
write_json(out / "scoreboard.json", _summarize(rows))
+ if failures:
+ labels = ", ".join(str(row.get("run_label")) for row in failures[:5])
+ raise RuntimeError(f"{len(failures)} cell(s) failed or incomplete: {labels}")
except Exception as exc:
plan["status"] = "failed"
plan["error"] = str(exc)
diff --git a/benchmark/tau2/scripts/run_memory_v2_eval.py b/benchmark/tau2/scripts/run_memory_v2_eval.py
index de5ef5441..26b2d69e7 100644
--- a/benchmark/tau2/scripts/run_memory_v2_eval.py
+++ b/benchmark/tau2/scripts/run_memory_v2_eval.py
@@ -2,15 +2,17 @@
from __future__ import annotations
import argparse
+import hashlib
+import importlib
import json
import shutil
import sys
import time
+from copy import deepcopy
from pathlib import Path
from typing import Any
-from tau2_common import normalize_litellm_env
-
+from tau2_common import assert_tau2_results_complete, normalize_litellm_env
AGENT_NAME = "openviking_memory_agent"
REPO_ROOT = Path(__file__).resolve().parents[3]
@@ -29,6 +31,11 @@ WRITE_TOOL_PREFIXES = (
"grant_",
"reboot_",
)
+FIXED_FIRST_USER_NAME = "openviking_fixed_first_user_simulator"
+TRAIN_TRANSCRIPT_OPENVIKING_TEXT = "openviking_text"
+TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS = "role_tool_blocks"
+TRAIN_TRANSCRIPT_CUSTOM_LIKE = "custom_like"
+DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS = 5000
def _json(text: str) -> dict[str, Any]:
@@ -46,6 +53,51 @@ def _add_tau2_to_path(tau2_repo: Path) -> None:
sys.path.insert(0, str(src if src.is_dir() else tau2_repo))
+def _load_domain_policy(tau2_repo: Path, domain: str) -> str:
+ path = tau2_repo / "data" / "tau2" / "domains" / domain / "policy.md"
+ if not path.is_file():
+ return ""
+ return path.read_text(encoding="utf-8").strip()
+
+
+def _build_system_prompt(policy: str) -> str:
+ return (
+ "You are a customer service representative for a benchmark evaluation.\n"
+ "Help customers strictly according to the policy below.\n\n"
+ "You have two kinds of tools:\n"
+ "1. Memory/context tools available to the benchmark agent.\n"
+ "2. Business system tools executed by the TAU-2 environment.\n\n"
+ "Rules for business system tools:\n"
+ "- Call ONE tool at a time. Wait for the result before calling the next tool.\n"
+ "- Only pass arguments the customer explicitly provided. Do NOT infer or add optional arguments.\n"
+ "- After receiving a tool result, use it to continue helping the customer.\n\n"
+ "## Policy\n"
+ f"{policy}\n\n"
+ "Now wait for the customer to start. Follow the policy exactly."
+ )
+
+
+def _patch_tau2_auxiliary_llm_defaults(llm: str, llm_args: dict[str, Any]) -> None:
+ # TAU-2 exposes agent/user LLMs in TextRunConfig, but NL assertion scoring
+ # still reads module defaults. Keep the evaluator on the same configured
+ # model so benchmark runs do not fall back to inaccessible upstream defaults.
+ patches = {
+ "DEFAULT_LLM_NL_ASSERTIONS": llm,
+ "DEFAULT_LLM_NL_ASSERTIONS_ARGS": deepcopy(llm_args),
+ "DEFAULT_LLM_ENV_INTERFACE": llm,
+ "DEFAULT_LLM_ENV_INTERFACE_ARGS": deepcopy(llm_args),
+ }
+ for module_name in (
+ "tau2.config",
+ "tau2.evaluator.evaluator_nl_assertions",
+ "tau2.environment.utils.interface_agent",
+ ):
+ module = importlib.import_module(module_name)
+ for name, value in patches.items():
+ if hasattr(module, name):
+ setattr(module, name, deepcopy(value))
+
+
def _save_to_arg(path: Path) -> str:
# Some TAU-2 versions append ".json"; newer versions treat save_to as a
# run directory and write results.json under it.
@@ -77,6 +129,10 @@ def _db_match(sim: dict[str, Any]) -> bool | None:
return sim.get("db_match")
+def _task_success(sim: dict[str, Any]) -> bool:
+ return _reward(sim) >= 1.0
+
+
def _metrics(results_path: Path) -> dict[str, Any]:
data = json.loads(results_path.read_text())
sims = data.get("simulations") or []
@@ -86,7 +142,9 @@ def _metrics(results_path: Path) -> dict[str, Any]:
return {
"simulation_count": len(sims),
"avg_reward": sum(rewards) / len(rewards) if rewards else 0.0,
- "db_match_rate": (sum(1 for value in db_known if value) / len(db_known)) if db_known else None,
+ "db_match_rate": (sum(1 for value in db_known if value) / len(db_known))
+ if db_known
+ else None,
}
@@ -118,12 +176,14 @@ def _tool_call_query(tool_calls: list[Any], state_messages: list[Any]) -> str:
recent_user = [
str(getattr(message, "content", "") or "")
for message in state_messages[-8:]
- if str(getattr(message, "role", "")) == "user" and str(getattr(message, "content", "") or "").strip()
+ if str(getattr(message, "role", "")) == "user"
+ and str(getattr(message, "content", "") or "").strip()
]
recent_observations = [
str(getattr(message, "content", "") or "")[:600]
for message in state_messages[-12:]
- if str(getattr(message, "role", "")) == "tool" and str(getattr(message, "content", "") or "").strip()
+ if str(getattr(message, "role", "")) == "tool"
+ and str(getattr(message, "content", "") or "").strip()
]
parts = [
"Before executing write-like tool call(s): " + "; ".join(rendered),
@@ -134,7 +194,26 @@ def _tool_call_query(tool_calls: list[Any], state_messages: list[Any]) -> str:
return "\n".join(parts)
-def _message_text(message: dict[str, Any]) -> tuple[str, str]:
+def _tool_call_id(tool_call: dict[str, Any]) -> str:
+ return str(tool_call.get("id") or tool_call.get("tool_call_id") or "").strip()
+
+
+def _tool_result_call_id(message: dict[str, Any]) -> str:
+ return str(message.get("id") or message.get("tool_call_id") or "").strip()
+
+
+def _compact_train_tool_output(content: Any, *, max_chars: int) -> str:
+ text = (
+ content
+ if isinstance(content, str)
+ else json.dumps(content, ensure_ascii=False, sort_keys=True)
+ )
+ if len(text) <= max_chars:
+ return text
+ return text[:max_chars] + f"... <truncated {len(text) - max_chars} chars>"
+
+
+def _message_text_openviking_text(message: dict[str, Any]) -> tuple[str, str]:
role = str(message.get("role") or "assistant")
if role == "user":
return "user", str(message.get("content") or "")
@@ -151,6 +230,143 @@ def _message_text(message: dict[str, Any]) -> tuple[str, str]:
return "assistant", str(message.get("content") or "")
+def _message_texts_role_tool_blocks(
+ message: dict[str, Any],
+ *,
+ tool_calls_by_id: dict[str, dict[str, Any]],
+ max_tool_output_chars: int,
+) -> list[tuple[str, str]]:
+ role = str(message.get("role") or "assistant")
+ rows: list[tuple[str, str]] = []
+ if role in {"user", "assistant"}:
+ content = message.get("content")
+ if isinstance(content, str) and content.strip():
+ rows.append((role, f"{role}:\n{content}"))
+ for tool_call in message.get("tool_calls") or []:
+ if not isinstance(tool_call, dict):
+ continue
+ call_id = _tool_call_id(tool_call)
+ if call_id:
+ tool_calls_by_id[call_id] = tool_call
+ requestor = str(tool_call.get("requestor") or role or "assistant")
+ name = _tool_call_name(tool_call)
+ arguments = _tool_call_arguments(tool_call)
+ lines = ["tool-call:"]
+ if call_id:
+ lines.append(f"call_id: {call_id}")
+ if name:
+ lines.append(f"name: {name}")
+ lines.append(
+ "arguments: "
+ + json.dumps(arguments, ensure_ascii=False, sort_keys=True, default=str)
+ )
+ rows.append((requestor, "\n".join(lines)))
+ elif role == "tool":
+ call_id = _tool_result_call_id(message)
+ tool_call = tool_calls_by_id.get(call_id) or {}
+ requestor = str(message.get("requestor") or tool_call.get("requestor") or "assistant")
+ output = _compact_train_tool_output(
+ message.get("content"),
+ max_chars=max_tool_output_chars,
+ )
+ lines = ["tool-response:"]
+ if call_id:
+ lines.append(f"call_id: {call_id}")
+ name = _tool_call_name(tool_call)
+ if name:
+ lines.append(f"name: {name}")
+ if message.get("error"):
+ lines.append("error: true")
+ lines.append(f"output: {output}")
+ rows.append((requestor, "\n".join(lines)))
+ else:
+ content = str(message.get("content") or "").strip()
+ if content:
+ rows.append(("assistant", f"{role}:\n{content}"))
+ return rows
+
+
+def _message_texts(
+ message: dict[str, Any],
+ *,
+ transcript_format: str,
+ tool_calls_by_id: dict[str, dict[str, Any]],
+ max_tool_output_chars: int,
+) -> list[tuple[str, str]]:
+ if transcript_format == TRAIN_TRANSCRIPT_OPENVIKING_TEXT:
+ return [_message_text_openviking_text(message)]
+ if transcript_format in {TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS, TRAIN_TRANSCRIPT_CUSTOM_LIKE}:
+ return _message_texts_role_tool_blocks(
+ message,
+ tool_calls_by_id=tool_calls_by_id,
+ max_tool_output_chars=max_tool_output_chars,
+ )
+ raise ValueError(f"Unsupported train_transcript_format: {transcript_format}")
+
+
+def _scenario_sha256(instructions: str) -> str:
+ return hashlib.sha256(instructions.encode("utf-8")).hexdigest()
+
+
+def _load_fixed_first_user_fixture(path: Path) -> dict[str, str]:
+ if not path.is_file():
+ raise FileNotFoundError(f"fixed-first-user fixture not found: {path}")
+ data = json.loads(path.read_text(encoding="utf-8"))
+ mapping = data.get("by_scenario_sha256") if isinstance(data, dict) else None
+ if not isinstance(mapping, dict) or not mapping:
+ raise ValueError(f"fixed-first-user fixture has no by_scenario_sha256 map: {path}")
+ return {str(key): str(value) for key, value in mapping.items()}
+
+
+def _has_user_message(state: Any) -> bool:
+ for message in getattr(state, "messages", []) or []:
+ role = getattr(message, "role", None)
+ if str(getattr(role, "value", role)) == "user":
+ return True
+ return False
+
+
+def _append_incoming_user_context(message: Any, state: Any) -> None:
+ from tau2.data_model.message import AssistantMessage, MultiToolMessage, ToolMessage
+
+ if isinstance(message, MultiToolMessage):
+ state.messages.extend(message.tool_messages)
+ elif isinstance(message, ToolMessage):
+ state.messages.append(message)
+ elif isinstance(message, AssistantMessage) and (
+ message.has_content() or message.is_tool_call()
+ ):
+ state.messages.append(message)
+
+
+def _register_fixed_first_user(args: argparse.Namespace) -> str:
+ if not args.fixed_first_user_file:
+ return args.user
+ _add_tau2_to_path(args.tau2_repo)
+ mapping = _load_fixed_first_user_fixture(args.fixed_first_user_file)
+
+ from tau2.data_model.message import UserMessage
+ from tau2.registry import registry
+ from tau2.user.user_simulator import UserSimulator
+
+ class FixedFirstUserSimulator(UserSimulator): # type: ignore[misc]
+ def _generate_next_message(self, message: Any, state: Any) -> UserMessage: # type: ignore[override]
+ if not _has_user_message(state):
+ key = _scenario_sha256(str(self.instructions or ""))
+ fixed = mapping.get(key)
+ if fixed is None:
+ raise RuntimeError(
+ f"fixed-first-user fixture does not cover this TAU-2 scenario: sha256={key}"
+ )
+ _append_incoming_user_context(message, state)
+ return UserMessage(role="user", content=fixed)
+ return super()._generate_next_message(message, state)
+
+ if FIXED_FIRST_USER_NAME not in registry.get_users():
+ registry.register_user(FixedFirstUserSimulator, FIXED_FIRST_USER_NAME)
+ return FIXED_FIRST_USER_NAME
+
+
def _run_tau2(
*,
tau2_repo: Path,
@@ -171,6 +387,7 @@ def _run_tau2(
save_to: Path,
):
_add_tau2_to_path(tau2_repo)
+ _patch_tau2_auxiliary_llm_defaults(agent_llm, agent_llm_args)
from tau2.data_model.simulation import RunConfig, TextRunConfig
from tau2.run import run_domain
@@ -275,51 +492,125 @@ def _probe_corpus(args: argparse.Namespace, client: Any) -> dict[str, Any]:
def _train(args: argparse.Namespace, train_results: Path, corpus_manifest: Path) -> dict[str, Any]:
if corpus_manifest.is_file() and not args.force_train:
- return json.loads(corpus_manifest.read_text())
-
- _run_tau2(
- tau2_repo=args.tau2_repo,
- domain=args.domain,
- split=args.train_split_name,
- task_ids=args.train_task_ids,
- num_tasks=args.train_num_tasks,
- trials=1,
- max_steps=args.max_steps,
- max_concurrency=args.max_concurrency,
- agent=args.base_agent,
- user=args.user,
- agent_llm=args.agent_llm,
- user_llm=args.user_llm,
- agent_llm_args=args.agent_llm_args,
- user_llm_args=args.user_llm_args,
- seed=args.seed,
- save_to=train_results,
- )
+ manifest = json.loads(corpus_manifest.read_text())
+ cached_transcript_format = str(
+ manifest.get("train_transcript_format") or TRAIN_TRANSCRIPT_OPENVIKING_TEXT
+ )
+ if cached_transcript_format != args.train_transcript_format:
+ raise ValueError(
+ "cached corpus train_transcript_format mismatch: "
+ f"{cached_transcript_format!r} != {args.train_transcript_format!r}; "
+ "use a distinct corpus_id or --force-train"
+ )
+ cached_include_system_prompt = bool(manifest.get("train_include_system_prompt") or False)
+ if cached_include_system_prompt != bool(args.train_include_system_prompt):
+ raise ValueError(
+ "cached corpus train_include_system_prompt mismatch: "
+ f"{cached_include_system_prompt!r} != {bool(args.train_include_system_prompt)!r}; "
+ "use a distinct corpus_id or --force-train"
+ )
+ cached_tool_output_max_chars = int(
+ manifest.get("train_tool_output_max_chars") or DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS
+ )
+ if cached_tool_output_max_chars != int(args.train_tool_output_max_chars):
+ raise ValueError(
+ "cached corpus train_tool_output_max_chars mismatch: "
+ f"{cached_tool_output_max_chars!r} != {int(args.train_tool_output_max_chars)!r}; "
+ "use a distinct corpus_id or --force-train"
+ )
+ cached_skip_failed = bool(manifest.get("train_skip_failed_sessions") or False)
+ if cached_skip_failed != bool(args.train_skip_failed_sessions):
+ raise ValueError(
+ "cached corpus train_skip_failed_sessions mismatch: "
+ f"{cached_skip_failed!r} != {bool(args.train_skip_failed_sessions)!r}; "
+ "use a distinct corpus_id or --force-train"
+ )
+ return manifest
+
+ if train_results.is_file() and not args.force_train:
+ data = json.loads(train_results.read_text())
+ assert_tau2_results_complete(data, context=f"{args.domain} cached train")
+ else:
+ _run_tau2(
+ tau2_repo=args.tau2_repo,
+ domain=args.domain,
+ split=args.train_split_name,
+ task_ids=args.train_task_ids,
+ num_tasks=args.train_num_tasks,
+ trials=1,
+ max_steps=args.max_steps,
+ max_concurrency=args.max_concurrency,
+ agent=args.base_agent,
+ user=args.user,
+ agent_llm=args.agent_llm,
+ user_llm=args.user_llm,
+ agent_llm_args=args.agent_llm_args,
+ user_llm_args=args.user_llm_args,
+ seed=args.seed,
+ save_to=train_results,
+ )
+ data = json.loads(train_results.read_text())
+ assert_tau2_results_complete(data, context=f"{args.domain} train")
- data = json.loads(train_results.read_text())
client = _client(args)
committed = []
+ system_prompt_text = ""
+ if args.train_include_system_prompt:
+ policy = _load_domain_policy(args.tau2_repo, args.domain)
+ system_prompt_text = _build_system_prompt(policy)
+ skipped_failed_sessions: list[dict[str, Any]] = []
try:
for sim in data.get("simulations") or []:
- session_id = f"tau2-{args.domain}-train-{sim.get('task_id')}-trial-{sim.get('trial', 0)}"
+ if args.train_skip_failed_sessions and not _task_success(sim):
+ skipped_failed_sessions.append(
+ {
+ "session_id": (
+ f"tau2-{args.domain}-train-{sim.get('task_id')}-"
+ f"trial-{sim.get('trial', 0)}"
+ ),
+ "task_id": sim.get("task_id"),
+ "trial": sim.get("trial", 0),
+ "reward": _reward(sim),
+ "db_match": _db_match(sim),
+ }
+ )
+ continue
+ session_id = (
+ f"tau2-{args.domain}-train-{sim.get('task_id')}-trial-{sim.get('trial', 0)}"
+ )
created = client.create_session(session_id=session_id)
sid = created.get("session_id", session_id)
- for msg in sim.get("messages") or []:
- role, text = _message_text(msg)
- if not text.strip():
- continue
+ if system_prompt_text.strip():
client.add_message(
sid,
- role=role,
- parts=[{"type": "text", "text": text}],
- created_at=msg.get("timestamp"),
+ role="user",
+ parts=[{"type": "text", "text": f"system:\n{system_prompt_text}"}],
)
+ tool_calls_by_id: dict[str, dict[str, Any]] = {}
+ for msg in sim.get("messages") or []:
+ for role, text in _message_texts(
+ msg,
+ transcript_format=args.train_transcript_format,
+ tool_calls_by_id=tool_calls_by_id,
+ max_tool_output_chars=args.train_tool_output_max_chars,
+ ):
+ if not text.strip():
+ continue
+ client.add_message(
+ sid,
+ role=role,
+ parts=[{"type": "text", "text": text}],
+ created_at=msg.get("timestamp"),
+ )
result = client.commit_session(sid, telemetry=True)
task = _wait_task(client, result.get("task_id"), args.openviking_wait_timeout)
committed.append(
{
"session_id": sid,
"task_id": sim.get("task_id"),
+ "trial": sim.get("trial", 0),
+ "reward": _reward(sim),
+ "db_match": _db_match(sim),
"commit_status": result.get("status"),
"openviking_task_id": result.get("task_id"),
"openviking_task_status": task.get("status"),
@@ -344,8 +635,14 @@ def _train(args: argparse.Namespace, train_results: Path, corpus_manifest: Path)
"agent_id": args.openviking_agent_id,
"search_uri": args.search_uri,
},
+ "train_transcript_format": args.train_transcript_format,
+ "train_include_system_prompt": bool(args.train_include_system_prompt),
+ "train_skip_failed_sessions": bool(args.train_skip_failed_sessions),
+ "train_tool_output_max_chars": args.train_tool_output_max_chars,
"committed_sessions": committed,
"committed_session_count": len(committed),
+ "skipped_failed_sessions": skipped_failed_sessions,
+ "skipped_failed_session_count": len(skipped_failed_sessions),
"corpus_probe": corpus_probe,
}
_write_json(corpus_manifest, manifest)
@@ -360,36 +657,81 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
from tau2.registry import registry
from tau2.utils.llm_utils import generate
+ scope_prompt = ""
+ if args.scope_prompt_file is not None:
+ scope_prompt = args.scope_prompt_file.read_text(encoding="utf-8").strip()
+
class OpenVikingMemoryAgent(LLMAgent):
def get_init_state(self, message_history=None):
state = super().get_init_state(message_history)
+ if scope_prompt:
+ state.system_messages.append(SystemMessage(role="system", content=scope_prompt))
if args.retrieval_mode in {"first_user", "first_user_prewrite"}:
state.system_messages.append(
SystemMessage(role="system", content="<openviking_memory_not_loaded/>")
)
return state
- def _retrieve(self, query: str) -> tuple[str, list[dict[str, Any]]]:
+ def _retrieve(
+ self,
+ query: str,
+ *,
+ search_limit: int,
+ inject_limit: int,
+ inject_max_chars: int | None = None,
+ ) -> tuple[str, list[dict[str, Any]]]:
client = _client(args)
rows: list[dict[str, Any]] = []
try:
- result = client.search(query=query, target_uri=args.search_uri, limit=args.retrieval_top_k)
+ result = client.search(query=query, target_uri=args.search_uri, limit=search_limit)
memories = list(getattr(result, "memories", []) or [])
blocks = []
- for index, match in enumerate(memories[: args.retrieval_top_k], 1):
+ injected_chars_used = 0
+ for index, match in enumerate(memories[:search_limit], 1):
uri = getattr(match, "uri", "")
text, read_error = _read_memory_text(client, match)
+ clean_text = text.strip()
+ block_text = f"Memory {index} ({uri}):\n{clean_text}" if clean_text else ""
+ block_chars = len(block_text)
+ budget_used_before = injected_chars_used
+ budget_dropped = False
+ truncated = False
+ injected = index <= inject_limit and bool(block_text)
+ if injected and inject_max_chars is not None:
+ remaining = inject_max_chars - injected_chars_used
+ if remaining <= 0:
+ injected = False
+ budget_dropped = True
+ elif block_chars > remaining:
+ if not blocks:
+ block_text = block_text[:remaining]
+ block_chars = len(block_text)
+ truncated = True
+ else:
+ injected = False
+ budget_dropped = True
+ if injected:
+ injected_chars_used += block_chars
row = {
"uri": uri,
"score": getattr(match, "score", None),
"level": getattr(match, "level", None),
"text_chars": len(text),
+ "block_chars": block_chars,
+ "injected": injected,
+ "inject_max_chars": inject_max_chars,
+ "inject_budget_used_before": budget_used_before,
+ "inject_budget_used_after": injected_chars_used,
+ "inject_budget_dropped": budget_dropped,
+ "inject_budget_truncated": truncated,
}
+ if budget_dropped:
+ row["skipped_reason"] = "inject_char_budget_exceeded"
if read_error:
row["read_error"] = read_error
rows.append(row)
- if text.strip():
- blocks.append(f"Memory {index} ({uri}):\n{text.strip()}")
+ if injected:
+ blocks.append(block_text)
return "\n\n".join(blocks), rows
finally:
client.close()
@@ -400,11 +742,13 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
@staticmethod
def _trace_injection_fields(block: str, matches: list[dict[str, Any]]) -> dict[str, Any]:
- injected_count = sum(1 for row in matches if int(row.get("text_chars") or 0) > 0)
+ injected_count = sum(1 for row in matches if row.get("injected"))
return {
"injected": bool(block.strip()),
"injected_count": injected_count if block.strip() else 0,
- "retrieval_action_taken": "retrieve_and_inject" if block.strip() else "retrieve_no_injection",
+ "retrieval_action_taken": "retrieve_and_inject"
+ if block.strip()
+ else "retrieve_no_injection",
}
def _generate(self, messages):
@@ -476,7 +820,8 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
(
i
for i, item in enumerate(state.system_messages)
- if isinstance(item, SystemMessage) and item.content == "<openviking_memory_not_loaded/>"
+ if isinstance(item, SystemMessage)
+ and item.content == "<openviking_memory_not_loaded/>"
),
None,
)
@@ -484,11 +829,16 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
role_value = getattr(role, "value", role)
if marker_index is not None and str(role_value) == "user":
query = str(getattr(message, "content", "") or "")
- block, matches = self._retrieve(query)
+ block, matches = self._retrieve(
+ query,
+ search_limit=args.first_user_retrieval_top_k,
+ inject_limit=args.first_user_inject_top_k,
+ inject_max_chars=args.first_user_memory_inject_max_chars,
+ )
prompt = (
"No OpenViking memory matched this user request."
if not block
- else "Use these OpenViking experience memories only when they match the current task:\n\n"
+ else "Use these OpenViking memories only when they match the current task:\n\n"
+ block
)
state.system_messages[marker_index] = SystemMessage(role="system", content=prompt)
@@ -496,6 +846,9 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
{
"decision_node": "first_user",
"query": query,
+ "search_limit": args.first_user_retrieval_top_k,
+ "inject_limit": args.first_user_inject_top_k,
+ "inject_max_chars": args.first_user_memory_inject_max_chars,
"match_count": len(matches),
"matches": matches,
**self._trace_injection_fields(block, matches),
@@ -508,11 +861,19 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
write_calls = [call for call in tool_calls if _is_write_tool_call(call)]
if write_calls:
query = _tool_call_query(write_calls, state.messages)
- block, matches = self._retrieve(query)
+ block, matches = self._retrieve(
+ query,
+ search_limit=args.prewrite_retrieval_top_k,
+ inject_limit=args.prewrite_inject_top_k,
+ inject_max_chars=args.prewrite_memory_inject_max_chars,
+ )
self._trace(
{
"decision_node": "before_write_tool_call",
"query": query,
+ "search_limit": args.prewrite_retrieval_top_k,
+ "inject_limit": args.prewrite_inject_top_k,
+ "inject_max_chars": args.prewrite_memory_inject_max_chars,
"match_count": len(matches),
"matches": matches,
**self._trace_injection_fields(block, matches),
@@ -528,8 +889,7 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
if block:
prompt = (
"Before executing the pending write-like tool call, use these "
- "OpenViking experience memories only when they match the current task:\n\n"
- + block
+ "OpenViking memories only when they match the current task:\n\n" + block
)
assistant_message = self._generate(
state.system_messages
@@ -540,6 +900,7 @@ def _register_memory_agent(args: argparse.Namespace, trace_path: Path) -> None:
return assistant_message, state
if AGENT_NAME not in registry.get_agents():
+
def create_openviking_memory_agent(tools, domain_policy, **kwargs):
return OpenVikingMemoryAgent(
tools=tools,
@@ -577,36 +938,190 @@ def main() -> int:
parser.add_argument("--user-llm", required=True)
parser.add_argument("--agent-llm-args", type=_json, default={})
parser.add_argument("--user-llm-args", type=_json, default={})
- parser.add_argument("--openviking-url", required=True)
- parser.add_argument("--openviking-account", required=True)
- parser.add_argument("--openviking-user", required=True)
- parser.add_argument("--openviking-agent-id", required=True)
+ parser.add_argument("--openviking-url")
+ parser.add_argument("--openviking-account")
+ parser.add_argument("--openviking-user")
+ parser.add_argument("--openviking-agent-id")
parser.add_argument("--openviking-timeout", type=float, default=600.0)
parser.add_argument("--openviking-wait-timeout", type=int, default=600)
- parser.add_argument("--search-uri", required=True)
+ parser.add_argument("--search-uri")
parser.add_argument("--retrieval-top-k", type=int, default=4)
+ parser.add_argument("--first-user-retrieval-top-k", type=int)
+ parser.add_argument("--first-user-inject-top-k", type=int)
+ parser.add_argument("--prewrite-retrieval-top-k", type=int)
+ parser.add_argument("--prewrite-inject-top-k", type=int)
+ parser.add_argument("--memory-inject-max-chars", type=int)
+ parser.add_argument("--first-user-memory-inject-max-chars", type=int)
+ parser.add_argument("--prewrite-memory-inject-max-chars", type=int)
+ parser.add_argument("--fixed-first-user-file", type=Path)
+ parser.add_argument("--scope-prompt-file", type=Path)
+ parser.add_argument(
+ "--train-transcript-format",
+ choices=[
+ TRAIN_TRANSCRIPT_OPENVIKING_TEXT,
+ TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS,
+ TRAIN_TRANSCRIPT_CUSTOM_LIKE,
+ ],
+ default=TRAIN_TRANSCRIPT_OPENVIKING_TEXT,
+ help=(
+ "How to replay TAU-2 train messages into OpenViking sessions. "
+ "openviking_text preserves the compact adapter text format; role_tool_blocks "
+ "uses role-prefixed messages plus tool-call/tool-response blocks. "
+ "custom_like is a compatibility alias for older cached PR-B corpora."
+ ),
+ )
+ parser.add_argument(
+ "--train-include-system-prompt",
+ action="store_true",
+ help="Prepend the domain policy as a user-visible system block during training.",
+ )
+ parser.add_argument(
+ "--train-skip-failed-sessions",
+ action="store_true",
+ help="Skip reward<1 train sessions when building positive trajectory memory.",
+ )
+ parser.add_argument(
+ "--train-tool-output-max-chars",
+ type=int,
+ default=DEFAULT_TRAIN_TOOL_OUTPUT_MAX_CHARS,
+ help=(
+ "Maximum characters kept for each tool-response block when "
+ f"--train-transcript-format={TRAIN_TRANSCRIPT_ROLE_TOOL_BLOCKS}."
+ ),
+ )
parser.add_argument(
"--retrieval-mode",
choices=["first_user", "prewrite", "first_user_prewrite"],
default="first_user",
)
parser.add_argument("--force-train", action="store_true")
+ parser.add_argument("--prepare-corpus-only", action="store_true")
+ parser.add_argument(
+ "--no-memory",
+ action="store_true",
+ help="Run the configured TAU-2 agent without OpenViking retrieval.",
+ )
args = parser.parse_args()
normalize_litellm_env()
+ if args.train_tool_output_max_chars <= 0:
+ parser.error("--train-tool-output-max-chars must be positive")
+ for name in (
+ "memory_inject_max_chars",
+ "first_user_memory_inject_max_chars",
+ "prewrite_memory_inject_max_chars",
+ ):
+ value = getattr(args, name)
+ if value is not None and value < 0:
+ parser.error(f"--{name.replace('_', '-')} must be non-negative")
+ if not args.no_memory:
+ missing = [
+ name
+ for name in (
+ "openviking_url",
+ "openviking_account",
+ "openviking_user",
+ "openviking_agent_id",
+ "search_uri",
+ )
+ if not getattr(args, name)
+ ]
+ if missing:
+ parser.error(
+ "OpenViking memory runs require: "
+ + ", ".join("--" + name.replace("_", "-") for name in missing)
+ )
args.tau2_repo = args.tau2_repo.resolve()
+ args.run_dir = args.run_dir.resolve()
+ if args.corpus_dir is not None:
+ args.corpus_dir = args.corpus_dir.resolve()
args.run_dir.mkdir(parents=True, exist_ok=True)
corpus_dir = args.corpus_dir or args.run_dir
corpus_dir.mkdir(parents=True, exist_ok=True)
+ args.first_user_retrieval_top_k = args.first_user_retrieval_top_k or args.retrieval_top_k
+ args.first_user_inject_top_k = args.first_user_inject_top_k or args.first_user_retrieval_top_k
+ args.prewrite_retrieval_top_k = args.prewrite_retrieval_top_k or args.retrieval_top_k
+ args.prewrite_inject_top_k = args.prewrite_inject_top_k or args.prewrite_retrieval_top_k
+ args.first_user_memory_inject_max_chars = (
+ args.first_user_memory_inject_max_chars
+ if args.first_user_memory_inject_max_chars is not None
+ else args.memory_inject_max_chars
+ )
+ args.prewrite_memory_inject_max_chars = (
+ args.prewrite_memory_inject_max_chars
+ if args.prewrite_memory_inject_max_chars is not None
+ else args.memory_inject_max_chars
+ )
+ if args.fixed_first_user_file is not None:
+ args.fixed_first_user_file = args.fixed_first_user_file.expanduser().resolve()
+ if args.scope_prompt_file is not None:
+ args.scope_prompt_file = args.scope_prompt_file.expanduser().resolve()
+ if not args.scope_prompt_file.is_file():
+ parser.error(f"--scope-prompt-file does not exist: {args.scope_prompt_file}")
train_results = corpus_dir / "train_results.json"
corpus_manifest = corpus_dir / "corpus_manifest.json"
eval_results = args.run_dir / f"{args.run_label}.json"
trace_path = args.run_dir / f"{args.run_label}.retrieval_trace.jsonl"
summary_path = args.run_dir / f"{args.run_label}.summary.json"
+ if args.no_memory:
+ user_name = _register_fixed_first_user(args)
+ _run_tau2(
+ tau2_repo=args.tau2_repo,
+ domain=args.domain,
+ split=args.eval_split_name,
+ task_ids=args.task_ids,
+ num_tasks=args.num_tasks,
+ trials=1,
+ max_steps=args.max_steps,
+ max_concurrency=args.max_concurrency,
+ agent=args.base_agent,
+ user=user_name,
+ agent_llm=args.agent_llm,
+ user_llm=args.user_llm,
+ agent_llm_args=args.agent_llm_args,
+ user_llm_args=args.user_llm_args,
+ seed=args.seed,
+ save_to=eval_results,
+ )
+ assert_tau2_results_complete(
+ json.loads(eval_results.read_text()), context=f"{args.domain} eval"
+ )
+ summary = {
+ "run_label": args.run_label,
+ "domain": args.domain,
+ "strategy_id": args.strategy_id,
+ "seed": args.seed,
+ "fixed_first_user_file": str(args.fixed_first_user_file)
+ if args.fixed_first_user_file
+ else None,
+ "eval_results": str(eval_results),
+ "metrics": _metrics(eval_results),
+ }
+ _write_json(summary_path, summary)
+ print(json.dumps(summary, ensure_ascii=False, sort_keys=True))
+ return 0
+
corpus = _train(args, train_results, corpus_manifest)
+ if args.prepare_corpus_only:
+ print(
+ json.dumps(
+ {
+ "run_label": args.run_label,
+ "domain": args.domain,
+ "strategy_id": args.strategy_id,
+ "prepare_corpus_only": True,
+ "corpus": corpus,
+ },
+ ensure_ascii=False,
+ sort_keys=True,
+ )
+ )
+ return 0
+
trace_path.touch()
_register_memory_agent(args, trace_path)
+ user_name = _register_fixed_first_user(args)
_run_tau2(
tau2_repo=args.tau2_repo,
domain=args.domain,
@@ -617,7 +1132,7 @@ def main() -> int:
max_steps=args.max_steps,
max_concurrency=args.max_concurrency,
agent=AGENT_NAME,
- user=args.user,
+ user=user_name,
agent_llm=args.agent_llm,
user_llm=args.user_llm,
agent_llm_args=args.agent_llm_args,
@@ -625,12 +1140,32 @@ def main() -> int:
seed=args.seed,
save_to=eval_results,
)
+ assert_tau2_results_complete(
+ json.loads(eval_results.read_text()), context=f"{args.domain} eval"
+ )
summary = {
"run_label": args.run_label,
"domain": args.domain,
"strategy_id": args.strategy_id,
"retrieval_mode": args.retrieval_mode,
+ "retrieval": {
+ "first_user_retrieval_top_k": args.first_user_retrieval_top_k,
+ "first_user_inject_top_k": args.first_user_inject_top_k,
+ "first_user_memory_inject_max_chars": args.first_user_memory_inject_max_chars,
+ "prewrite_retrieval_top_k": args.prewrite_retrieval_top_k,
+ "prewrite_inject_top_k": args.prewrite_inject_top_k,
+ "prewrite_memory_inject_max_chars": args.prewrite_memory_inject_max_chars,
+ "memory_inject_max_chars": args.memory_inject_max_chars,
+ },
+ "train_transcript_format": args.train_transcript_format,
+ "train_include_system_prompt": bool(args.train_include_system_prompt),
+ "train_skip_failed_sessions": bool(args.train_skip_failed_sessions),
+ "train_tool_output_max_chars": args.train_tool_output_max_chars,
"seed": args.seed,
+ "fixed_first_user_file": str(args.fixed_first_user_file)
+ if args.fixed_first_user_file
+ else None,
+ "scope_prompt_file": str(args.scope_prompt_file) if args.scope_prompt_file else None,
"corpus": corpus,
"eval_results": str(eval_results),
"retrieval_trace": str(trace_path),
diff --git a/benchmark/tau2/scripts/tau2_common.py b/benchmark/tau2/scripts/tau2_common.py
index a8b5ce201..4f4505bd8 100755
--- a/benchmark/tau2/scripts/tau2_common.py
+++ b/benchmark/tau2/scripts/tau2_common.py
@@ -5,13 +5,13 @@ import os
import re
import shutil
import subprocess
+from collections import Counter
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import yaml
-
TAU2_DIR = Path(__file__).resolve().parents[1]
REPO_ROOT = TAU2_DIR.parents[1]
CONFIRMATION_AWARE_UPSTREAM_PR = "https://github.com/sierra-research/tau2-bench/pull/297"
@@ -63,6 +63,7 @@ def normalize_litellm_env() -> dict[str, Any]:
def render_env(value: Any) -> Any:
if isinstance(value, str):
+
def replace(match: re.Match[str]) -> str:
name = match.group(1)
default = match.group(2) or ""
@@ -79,11 +80,7 @@ def render_env(value: Any) -> Any:
def deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
merged = dict(base)
for key, value in override.items():
- if (
- key in merged
- and isinstance(merged[key], dict)
- and isinstance(value, dict)
- ):
+ if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):
merged[key] = deep_merge(merged[key], value)
else:
merged[key] = value
@@ -125,6 +122,66 @@ def write_json(path: Path, payload: Any) -> None:
)
+def tau2_result_failures(data: dict[str, Any], *, expected_trials: int = 1) -> list[str]:
+ tasks = data.get("tasks") or []
+ simulations = data.get("simulations") or []
+ failures: list[str] = []
+ if tasks:
+ expected_task_ids = {
+ str(task.get("id", task.get("task_id"))) for task in tasks if isinstance(task, dict)
+ }
+ observed_task_ids = {str(sim.get("task_id")) for sim in simulations}
+ expected = len(tasks) * expected_trials
+ if len(simulations) != expected:
+ failures.append(f"expected {expected} simulations, found {len(simulations)}")
+ if observed_task_ids != expected_task_ids:
+ missing = sorted(expected_task_ids - observed_task_ids)
+ extra = sorted(observed_task_ids - expected_task_ids)
+ failures.append(
+ f"simulation task ids do not match tasks: missing={missing[:10]} extra={extra[:10]}"
+ )
+ expected_pairs = {
+ (task_id, trial) for task_id in expected_task_ids for trial in range(expected_trials)
+ }
+ observed_pairs = [
+ (str(sim.get("task_id")), int(sim.get("trial", 0))) for sim in simulations
+ ]
+ duplicate_pairs = sorted(
+ pair for pair, count in Counter(observed_pairs).items() if count != 1
+ )
+ missing_pairs = sorted(expected_pairs - set(observed_pairs))
+ if duplicate_pairs or missing_pairs:
+ failures.append(
+ "simulation task/trial coverage mismatch: "
+ f"missing={missing_pairs[:10]} duplicate={duplicate_pairs[:10]}"
+ )
+
+ for sim in simulations:
+ info = sim.get("info") or {}
+ termination_reason = str(sim.get("termination_reason") or "")
+ if info.get("failed_after_attempts") or "infrastructure_error" in termination_reason:
+ failures.append(
+ "task="
+ f"{sim.get('task_id')} trial={sim.get('trial', 0)} "
+ f"termination={termination_reason} error={info.get('error') or info.get('error_type')}"
+ )
+ elif not sim.get("messages"):
+ failures.append(
+ f"task={sim.get('task_id')} trial={sim.get('trial', 0)} has no messages"
+ )
+ return failures
+
+
+def assert_tau2_results_complete(
+ data: dict[str, Any], *, context: str, expected_trials: int = 1
+) -> None:
+ failures = tau2_result_failures(data, expected_trials=expected_trials)
+ if failures:
+ preview = "; ".join(failures[:5])
+ more = f"; ... {len(failures) - 5} more" if len(failures) > 5 else ""
+ raise RuntimeError(f"{context} produced invalid TAU-2 results: {preview}{more}")
+
+
def strategy_ids(config: dict[str, Any]) -> list[str]:
strategies = config.get("strategies") or []
if not isinstance(strategies, list):
@@ -219,9 +276,7 @@ def user_simulator_policy(config: dict[str, Any]) -> str:
policy = config.get("eval", {}).get("user_simulator_policy", "official")
policy = str(policy)
if policy not in {"official", "confirmation_aware"}:
- raise ValueError(
- "eval.user_simulator_policy must be 'official' or 'confirmation_aware'"
- )
+ raise ValueError("eval.user_simulator_policy must be 'official' or 'confirmation_aware'")
return policy
diff --git a/openviking/prompts/templates/memory/trajectories.yaml b/openviking/prompts/templates/memory/trajectories.yaml
index bd894dfef..d369a93c5 100644
--- a/openviking/prompts/templates/memory/trajectories.yaml
+++ b/openviking/prompts/templates/memory/trajectories.yaml
@@ -1,6 +1,6 @@
memory_type: trajectories
description: |
- A record of agent execution in one business domain within a conversation.
+ A compact, reusable operation contract distilled from an agent task trajectory.
Extract when the agent worked through identifiable tasks involving decisions, tool calls, or multi-step actions.
Skip pure chitchat or simple Q&A with no execution trace.
@@ -20,7 +20,7 @@ fields:
If English: lowercase snake_case, max 5 words.
If Chinese: concise noun phrase, max 15 characters.
Do not create near-duplicate names for the same task.
- Good: "flight_booking_duplicate_resolution", "pytest_asyncio_cancel_hang_fix", "机票重复预订处理".
+ Good: "duplicate_request_resolution", "pytest_asyncio_cancel_hang_fix", "重复请求处理".
merge_op: immutable
- name: outcome
@@ -32,31 +32,48 @@ fields:
- name: content
type: string
description: |
- Execution trace in EXACTLY this format:
- - Goal: <what the user wanted to achieve — one sentence>
- - Trajectory:
- 1. <what the agent aimed to achieve>.
- Actions: <summarized interactions, tools used, and summarized responses>.
- Progress: <how this shifted the state>.
- 2. <...>.
- Actions: <...>.
- Progress: <...>.
- 3. <... continue sequentially to capture the entire execution>
- - Result: <final outcome and any important side effects — one sentence>
- - Fail reason: <If task succeeded, leave empty or write "None". If failed, provide a detailed analysis of why the failure occurred and suggest how the agent's approach could be improved.>
+ Procedure-like operation contract in EXACTLY this format:
- Rules for 'Trajectory' section:
- - STANDALONE COMPLETENESS: The trajectory must be comprehensive enough that a reader can fully understand exactly how the agent performed the task, what attempts it made, and what tools it used, entirely without the raw logs.
- - EXHAUSTIVE TRACKING: Record every logical interaction, tool use, and system response in chronological order as a numbered list.
- - SUMMARIZE LONG TEXTS: Do NOT record exact tool responses, raw JSON payloads, or verbatim user/agent messages, especially when they are very long. You MUST use a concise, summarized version that captures the core meaning without losing any key information, constraints, or data points critical to the task.
- - CAPTURE ALL MISTAKES: You MUST explicitly detail any errors, false attempts, wrong function calls, or agent misunderstandings. Do not gloss over failures, dead ends, or retry loops.
- - INTENT & PROGRESS TRACKING: Every step must clearly state the current agent intention, summarize the 'Actions/Events' taken, and conclude with 'Progress' (evaluating what was achieved or blocked).
- - TOOL TRACKING (under 'Actions'):
- * Format (Success): "Called <tool_name(s)> -> Response: <summarized output retaining key info>."
- * Format (Error/Mistake): "Called <tool_name(s)> -> Error/Issue: <summarized error> | Context: <why it failed or why it was the wrong call>."
+ # <trajectory_name>
+ - Domain: <task domain or product area, e.g. customer support, commerce operations, software tooling>
+ - Trigger: <the reusable user intent or task situation — one sentence>
+ - Operation Intent:
+ - Family: <exactly one of read_verify_only, create_new_object, update_existing_object, cancel_existing_object, replace_existing_object, attribute_append_remove, compensation_or_refund, read_then_transfer, final_response_only>
+ - Primary object: <the object whose state or lifecycle this record is about>
+ - Target object/lifecycle: <the intended final state, lifecycle transition, or handoff target>
+ - Preconditions:
+ 1. <state, identity, policy, tool-observation, or confirmation prerequisite>
+ 2. <...>
+ - Immutable Object Boundary:
+ - <object fields, lifecycle state, policy facts, permissions, or ownership facts that must be read/verified and not inferred or overwritten>
+ - Procedure:
+ 1. <future-agent instruction for the first decision/action step; include key tool names when useful>
+ 2. <future-agent instruction for the next decision/action step>
+ 3. <continue until the write/confirmation/final-response boundary is clear>
+ - Write Field Provenance:
+ - <write-relevant field family>: <allowed source such as latest user request, explicit confirmation, fresh tool observation, computed policy result, stable policy, or current object state>
+ - Anti-patterns:
+ - <mistake, missing check, wrong ordering, over-broad action, or failed attempt to avoid>
+ - Applicability Boundary:
+ - <when this memory applies>
+ - <when this memory should not be used>
+ - Negative Applicability:
+ - <nearby intent, object lifecycle, policy state, permission state, or target object where this memory must not be applied>
+ - Result: <final outcome and important side effects — one sentence>
+ - Evidence: <brief generalized source note from this session, including success/failure/partial status>
- General Rules:
- - Use exactly the 4 labels (Goal, Trajectory, Result, Fail reason) in this exact order.
- - Goal and Result are ONE sentence each.
+ Rules:
+ - Write for future agent execution. Capture the reusable contract: trigger -> prerequisites -> read/verify -> confirm if needed -> one primary write, handoff, or final response -> verify/communicate result.
+ - Extract one record per primary user intent, object, and lifecycle transition. Split separable intents, pivots, follow-ups, enabling writes, and final writes; omit low-value side work.
+ - Family must be exactly one listed enum value. Do not invent compound labels. Use read_then_transfer when the reusable path ends in handoff after verification.
+ - Procedure must describe only the primary operation in this record. Mention other operations only as prerequisites, consequences, boundaries, or negative applicability.
+ - Stop each record at its family boundary: update_existing_object stops after update verification, cancel_existing_object after cancellation verification, create_new_object after creation verification, replace_existing_object after replacement verification, read_then_transfer after handoff verification.
+ - Do not include a second lifecycle-changing write in the Procedure, Target, Result, or Evidence of the same record, even when that later write happened immediately after the first one.
+ - Do not let a later operation in the same session rename, narrow, or extend this record's Trigger, Target, Procedure, Result, or Evidence. Put the later operation in a separate record.
+ - Put immutable facts and write/user-visible field sources in their dedicated sections. Do not copy fields from a previous/source object into a new/replacement object without current user request or explicit confirmation.
+ - Keep blocked actions and failed lessons in Anti-patterns / Negative Applicability / Evidence, not as positive procedure steps. Do not invent a corrected path when the session did not demonstrate one.
+ - Generalize the session: remove raw identifiers, names, contact values, exact dates, case-specific amounts, exact routes/locations, product names, and raw tool payloads. Keep tool names only when they are part of the reusable path.
+ - Use exactly the title, Domain line, and 11 labels above in this exact order.
+ - Trigger, Result, and Evidence are ONE sentence each.
- No extra headings, free paragraphs, or closing remarks.
merge_op: patch
diff --git a/openviking/session/memory/agent_trajectory_context_provider.py b/openviking/session/memory/agent_trajectory_context_provider.py
index 2316052f7..1b8cd42cf 100644
--- a/openviking/session/memory/agent_trajectory_context_provider.py
+++ b/openviking/session/memory/agent_trajectory_context_provider.py
@@ -14,7 +14,6 @@ from openviking.server.identity import RequestContext
from openviking.session.memory.session_extract_context_provider import (
SessionExtractContextProvider,
)
-from openviking.storage.viking_fs import VikingFS
from openviking_cli.utils import get_logger
logger = get_logger(__name__)
@@ -24,18 +23,25 @@ TRAJECTORY_MEMORY_TYPE = "trajectories"
class AgentTrajectoryContextProvider(SessionExtractContextProvider):
- """Phase 1 provider: extract trajectory summaries from conversation."""
+ """Phase 1 provider: extract reusable trajectory-view memories."""
def instruction(self) -> str:
output_language = self._output_language
- return f"""You are a memory extraction agent. Summarize this agent session as a trajectory record.
-
-One session = one trajectory. Always output exactly one, no exceptions.
-Sub-tasks, pivots, errors, and follow-ups are numbered steps inside that one record — not separate trajectories.
-
-Output a JSON object with a `trajectories` array containing exactly one item.
-Follow field descriptions in the schema. JSON only, no explanation.
-All content fields must be written in {output_language}.
+ return f"""You are a memory extraction agent. Convert this session into reusable trajectory-view memories.
+
+Each memory is a compact operation contract for a future agent, not a transcript.
+Capture trigger, prerequisites, verified boundaries, procedure, provenance,
+anti-patterns, and applicability. Split separate intents, pivots, enabling writes,
+and final writes into separate records; omit low-value side work. Use only schema
+operation-family enum values, not compound labels. Generalize away raw identifiers,
+names, exact dates, amounts, routes, and tool payloads. Do not let a later
+operation in the session rename, narrow, or extend an earlier operation record.
+Stop each record at its family boundary; do not include a second lifecycle-
+changing write in the same operation contract.
+
+Output JSON only: {{"trajectories": [...]}}. Include items only for reusable
+operation contracts supported by the session. Follow field descriptions in the
+schema. All content fields must be written in {output_language}.
"""
def get_memory_schemas(self, ctx: RequestContext) -> List[Any]: