Skip to content
Python

fix(sdk): qualify provider flows and graceful budget summaries

Rohit P0f488afa6265
diff --git a/.github/workflows/rust-sdk.yml b/.github/workflows/rust-sdk.yml
index 5e58b7c9..9df4a294 100644
--- a/.github/workflows/rust-sdk.yml
+++ b/.github/workflows/rust-sdk.yml
@@ -227,8 +227,7 @@ jobs:
npm --prefix typescript-sdk test
npm --prefix typescript-sdk run test:integration
npm --prefix typescript-sdk run test:dynamic-agents
- node --test --test-reporter=tap typescript-sdk/test/chat-usage.test.mjs typescript-sdk/test/openrouter-streaming.test.mjs
- npm --prefix typescript-sdk run test:session-options
+ node --test --test-concurrency=2 --test-reporter=tap typescript-sdk/test/chat-usage.test.mjs typescript-sdk/test/openrouter-streaming.test.mjs typescript-sdk/test/anthropic-caching.test.mjs typescript-sdk/test/gemini-compat.test.mjs typescript-sdk/test/session-options.test.mjs typescript-sdk/test/stream-events.test.mjs
node --test --test-reporter=tap typescript-sdk/test/desktop-api.test.mjs typescript-sdk/test/agent-team.test.mjs
node --test --test-reporter=tap typescript-sdk/test/coding.test.mjs typescript-sdk/test/assistant.test.mjs
- name: Starter and partner packaging
@@ -285,5 +284,6 @@ jobs:
run: |
cargo test --locked -p openworker-runtime -- --test-threads=1
cargo test --locked -p openworker-telemetry --features otlp
+ cargo test --locked -p openworker-store-postgres --test store imported_history_and_turn_context_ -- --ignored --test-threads=1
- name: Shared PostgreSQL team contract
run: cargo test --locked -p openworker-store-postgres --features test-support --test team_contract --test agent_team_contract --test agent_team_worker -- --ignored --test-threads=1
diff --git a/docs/harness/guides/releases.md b/docs/harness/guides/releases.md
index f43338dd..195d3d7f 100644
--- a/docs/harness/guides/releases.md
+++ b/docs/harness/guides/releases.md
@@ -32,5 +32,18 @@ Protocol negotiation does not make arbitrary SDK and host releases interchangeab
The application controls when it upgrades; the harness does not replace its running
executable automatically.
+## Session and streaming API updates
+
+The new turn options, history import, built-in tool controls and provider settings
+require rebuilding the native harness alongside the SDK. They are not enabled by
+updating a TypeScript declaration file alone. Existing sessions and omitted
+configuration keep their previous tool selection and per-call limits.
+
+Rust applications using explicit `TurnLimits` or `TurnOptions` literals should
+include the new fields or use `..Default::default()`. Tool event variants now
+carry call and turn IDs, and there are new attempt/reasoning/diagnostic variants;
+update exhaustive matches, or use `..` when ignoring additional fields. See
+[session options](sessions.md) and [streaming and retries](streaming.md) for examples.
+
See [troubleshooting](troubleshooting.md) for runtime discovery, version mismatch
and storage errors.
diff --git a/docs/harness/guides/sessions.md b/docs/harness/guides/sessions.md
index 0cb8e45c..6a430f99 100644
--- a/docs/harness/guides/sessions.md
+++ b/docs/harness/guides/sessions.md
@@ -14,7 +14,8 @@ unknown, inspect its status before retrying it.
Use `instructionAddendum` in TypeScript, or `TurnOptions.instruction_addendum` in
Rust, for application-owned preferences that change between turns. The addendum
accompanies every model call and retry in that turn, survives recovery, and is
-checked by model-input guardrails/hooks. It is stored with the turn rather than
+included in the assembled request inspected by before-model hooks. User-input
+guardrails apply to the user message, not this application-owned field. It is stored with the turn rather than
as user transcript text. It expires for the next turn. Keep user-supplied text in
the user message; an addendum does not grant tools, approvals or sandbox access.
@@ -69,6 +70,11 @@ provider requires signed reasoning metadata to replay an old tool call, resume
the original harness session or import a text-only conversation summary instead.
An import never restores permission grants or an in-progress operation.
+Import validates structure and bounds; it does not retroactively run user-input
+guardrails or tool-result hooks over the old transcript. The application must
+sanitize migrated content before import. Before-model hooks can inspect the
+assembled request, including imported history, before the next model call.
+
## Approvals and questions
Render durable approval/question interruptions in your UI, then send the user's
diff --git a/docs/harness/guides/streaming.md b/docs/harness/guides/streaming.md
index c247407e..075fdac2 100644
--- a/docs/harness/guides/streaming.md
+++ b/docs/harness/guides/streaming.md
@@ -78,11 +78,21 @@ await harness.setModelProvider({
Rust uses `ModelProvider::anthropic("Claude").thinking(AnthropicThinking::Adaptive)`
or `AnthropicThinking::Enabled { budget_tokens: 2048 }`. An enabled budget must
-be at least 1,024 tokens and smaller than the current call's output limit. These
-settings apply to main agent calls, not short connection probes, compactors or
-reviewers. Omitted settings preserve an existing choice; explicitly use
-`{ type: "disabled" }` to disable thinking generation. Provider configuration
-persists across restart and credential rotation.
+be at least 1,024 tokens. The configured value is a ceiling: the harness clamps
+it below the current call's output limit. If the remaining allowance cannot fit
+the minimum thinking budget plus an answer token, the request omits that manual
+thinking configuration. Final turn-limit summaries also omit the agent's thinking
+configuration so the small summary reserve is not rejected by a larger manual
+budget. Prior thinking/signature blocks are still replayed unchanged.
+
+These settings apply to working-agent calls, not connection probes, compactors,
+reviewers or final turn-limit summaries. Omitting a request setting uses the
+provider/model default; it does not guarantee thinking is disabled on models where
+thinking is always on. Output limits remain hard ceilings. Omitted settings during
+provider registration preserve an existing choice. Use `{ type: "disabled" }`
+only for models that support it; some adaptive-only models reject that mode.
+Provider configuration persists across restart and credential rotation. See
+[Anthropic's model-specific thinking rules](https://platform.claude.com/docs/en/build-with-claude/thinking-troubleshooting).
Output guardrails and blocking output/tool hooks suppress candidate reasoning as
well as candidate answer text, including for agents without tools. Display
diff --git a/harness/runtime/src/anthropic.rs b/harness/runtime/src/anthropic.rs
index 4d30a010..37e86be4 100644
--- a/harness/runtime/src/anthropic.rs
+++ b/harness/runtime/src/anthropic.rs
@@ -29,18 +29,22 @@ impl Thinking {
}
Ok(())
}
- pub fn request(&self, max_tokens: usize) -> Result<Value, Error> {
+ /// Output limits take precedence over manual thinking allowances. Return
+ /// None below the manual minimum rather than sending an invalid request or
+ /// forcing `disabled`, which some adaptive-only models reject.
+ pub fn request(&self, max_tokens: usize) -> Result<Option<Value>, Error> {
self.validate().map_err(|_| config())?;
- Ok(match self {
+ Ok(Some(match self {
Self::Disabled => json!({"type":"disabled"}),
Self::Adaptive => json!({"type":"adaptive"}),
Self::Enabled { budget_tokens } => {
- if *budget_tokens as usize >= max_tokens {
- return Err(config());
+ let budget_tokens = (*budget_tokens as usize).min(max_tokens.saturating_sub(1));
+ if budget_tokens < 1024 {
+ return Ok(None);
}
json!({"type":"enabled","budget_tokens":budget_tokens})
}
- })
+ }))
}
}
diff --git a/harness/runtime/src/model.rs b/harness/runtime/src/model.rs
index 6f0151bf..966e8bd9 100644
--- a/harness/runtime/src/model.rs
+++ b/harness/runtime/src/model.rs
@@ -12,6 +12,9 @@ pub struct Model {
pub content_withheld: bool,
/// Opt-in forwarding of provider-supported display reasoning, never signatures.
pub stream_reasoning: bool,
+ /// A final turn-limit summary uses its small reserved output allowance,
+ /// without inheriting the working agent's thinking-generation configuration.
+ pub budget_summary: bool,
pub connection: Option<(
std::sync::Arc<crate::connections::Manager>,
crate::connections::Selection,
@@ -98,6 +101,7 @@ impl Model {
observation: None,
content_withheld: false,
stream_reasoning: false,
+ budget_summary: false,
connection: None,
#[cfg(test)]
sample_hook: None,
@@ -421,15 +425,19 @@ impl Model {
}
// Short probes, reviewers and compactors have separate small
// budgets; do not apply the agent's thinking generation budget.
- if self
- .observation
- .as_ref()
- .is_some_and(|c| c.purpose == crate::telemetry::ModelPurpose::Main)
+ if !self.budget_summary
+ && self
+ .observation
+ .as_ref()
+ .is_some_and(|c| c.purpose == crate::telemetry::ModelPurpose::Main)
{
if let Some((manager, selection)) = &self.connection {
if let Some(thinking) = manager.thinking(selection)? {
- body["thinking"] =
- thinking.request(max_tokens).map_err(|e| e.with_api(api))?;
+ if let Some(request) =
+ thinking.request(max_tokens).map_err(|e| e.with_api(api))?
+ {
+ body["thinking"] = request;
+ }
}
}
}
diff --git a/harness/runtime/src/runtime.rs b/harness/runtime/src/runtime.rs
index 17deb9b0..b5ee452c 100644
--- a/harness/runtime/src/runtime.rs
+++ b/harness/runtime/src/runtime.rs
@@ -570,7 +570,8 @@ impl Runtime {
feedback: Option<&str>,
) -> Result<crate::provider::Completion> {
let started = tokio::time::Instant::now();
- for number in 1..=if summary { 1 } else { 3 } {
+ let max_attempts = if summary { 1 } else { 3 };
+ for number in 1..=max_attempts {
if started.elapsed().as_secs() >= 240 {
return Err(
crate::provider::Error::new(crate::provider::Failure::Transport).into(),
@@ -596,7 +597,8 @@ impl Runtime {
.any(|h| h.when == crate::safety::HookPhase::BeforeTool);
model.content_withheld = guarded_output;
let output_options = &s.turn.as_ref().context("No current turn")?.output;
- model.stream_reasoning = output_options.reasoning && !guarded_output;
+ model.budget_summary = summary;
+ model.stream_reasoning = output_options.reasoning && !guarded_output && !summary;
let buffer_attempts = output_options.buffer_attempts;
let show_reasoning = model.stream_reasoning;
let mut instructions = self.bundle.instructions_for(&s.skills, &s.loaded_skills);
@@ -911,7 +913,7 @@ impl Runtime {
},
)?;
}
- if number == 3
+ if number == max_attempts
|| !failure.is_some_and(|e| e.retryable())
|| started.elapsed().as_secs() >= 240
{
diff --git a/harness/runtime/src/runtime_budget_tests.rs b/harness/runtime/src/runtime_budget_tests.rs
index c2aceac5..5a44c440 100644
--- a/harness/runtime/src/runtime_budget_tests.rs
+++ b/harness/runtime/src/runtime_budget_tests.rs
@@ -67,6 +67,170 @@ fn model(f: impl Fn(usize, String) -> Value + Send + Sync + 'static, output: u64
m
}
+#[tokio::test]
+async fn budget_summary_transport_failure_returns_error_without_retry_or_panic() {
+ let calls = Arc::new(AtomicUsize::new(0));
+ let seen = calls.clone();
+ let mut m = Model::new("summary-error".into(), true);
+ m.sample_hook = Some(Arc::new(move |_, _, _| {
+ seen.fetch_add(1, Ordering::SeqCst);
+ Box::pin(async {
+ Err(crate::provider::Error::new(crate::provider::Failure::Transport).into())
+ })
+ }));
+ let (_tmp, rt, sid) = fixture(m, None).await;
+ let history = rt.store.lock().unwrap().get(&sid).unwrap().history;
+ let error = rt.sample_mode(&sid, &history, &[], true).await.unwrap_err();
+ assert_eq!(
+ error.downcast_ref::<crate::provider::Error>().unwrap().kind,
+ crate::provider::Failure::Transport
+ );
+ assert_eq!(calls.load(Ordering::SeqCst), 1);
+ let events = rt.store.lock().unwrap().events(&sid, 0).unwrap();
+ assert!(!events.iter().any(|e| e.kind == "model_retrying"));
+ assert_eq!(
+ events
+ .iter()
+ .filter(|e| e.kind == "model_attempt_finished" && e.data["status"] == "failed")
+ .count(),
+ 1
+ );
+}
+
+#[tokio::test]
+async fn anthropic_work_and_summary_requests_respect_remaining_budget_and_keep_replay_blocks() {
+ use tokio::io::{AsyncReadExt, AsyncWriteExt};
+ for total in [2048, 1024] {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let endpoint = format!("http://{}/v1", listener.local_addr().unwrap());
+ let server = tokio::spawn(async move {
+ for index in 0..2 {
+ let (mut socket, _) = listener.accept().await.unwrap();
+ let mut raw = vec![];
+ let mut chunk = [0; 4096];
+ let body: Value = loop {
+ let n = socket.read(&mut chunk).await.unwrap();
+ assert!(n > 0);
+ raw.extend_from_slice(&chunk[..n]);
+ assert!(raw.len() < 1_000_000);
+ if let Some(end) = raw.windows(4).position(|w| w == b"\r\n\r\n") {
+ let headers = String::from_utf8_lossy(&raw[..end]).to_ascii_lowercase();
+ let length: usize = headers
+ .lines()
+ .find_map(|l| l.strip_prefix("content-length: "))
+ .unwrap()
+ .parse()
+ .unwrap();
+ if raw.len() >= end + 4 + length {
+ break serde_json::from_slice(&raw[end + 4..end + 4 + length]).unwrap();
+ }
+ }
+ };
+ if index == 0 {
+ assert_eq!(body["max_tokens"], total - 512);
+ if total == 2048 {
+ assert_eq!(
+ body["thinking"],
+ json!({"type":"enabled","budget_tokens":1535})
+ );
+ } else {
+ assert!(body.get("thinking").is_none());
+ }
+ } else {
+ assert_eq!(body["max_tokens"], 512);
+ assert!(
+ body.get("thinking").is_none(),
+ "Reserved summary must not inherit manual thinking budget"
+ );
+ assert!(body
+ .get("tools")
+ .is_none_or(|v| v.as_array().is_some_and(Vec::is_empty)));
+ let prior = body["messages"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .flat_map(|m| m["content"].as_array().into_iter().flatten())
+ .find(|b| b["type"] == "thinking")
+ .unwrap();
+ assert_eq!(
+ prior,
+ &json!({"type":"thinking","thinking":"PRIVATE_REPLAY","signature":"signature-retained"})
+ );
+ }
+ let mut events = vec![
+ json!({"type":"message_start","message":{"id":format!("r-{index}"),"role":"assistant","content":[],"usage":{"input_tokens":20,"output_tokens":0}}}),
+ ];
+ if index == 0 {
+ events.push(json!({"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":"PRIVATE_REPLAY","signature":"signature-retained"}}));
+ events.push(json!({"type":"content_block_stop","index":0}));
+ events.push(json!({"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"bounded-plan","name":"update_plan","input":{"expected_revision":0,"explanation":"Plan","steps":[{"id":"work","description":"Find a name","status":"in_progress"}]}}}));
+ events.push(json!({"type":"content_block_stop","index":1}));
+ } else {
+ events.push(json!({"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}));
+ events.push(json!({"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Stopped at the configured limit; name lookup is incomplete."}}));
+ events.push(json!({"type":"content_block_stop","index":0}));
+ }
+ events.push(json!({"type":"message_delta","delta":{"stop_reason":if index==0 {"tool_use"} else {"end_turn"}},"usage":{"output_tokens":10}}));
+ events.push(json!({"type":"message_stop"}));
+ let stream = events
+ .iter()
+ .map(|e| format!("data: {e}\n\n"))
+ .collect::<String>();
+ socket.write_all(format!("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{stream}",stream.len()).as_bytes()).await.unwrap();
+ }
+ });
+ let (_tmp, rt, sid) = fixture(
+ Model::new("claude-fixture".into(), false)
+ .with_provider(crate::provider::Kind::Anthropic),
+ Some(Limits {
+ max_model_calls: 1,
+ max_output_tokens: total as u64,
+ summary_tokens: 512,
+ ..Default::default()
+ }),
+ )
+ .await;
+ rt.connections.set_provider(json!({"name":"Fixture","api":"anthropic","base_url":endpoint,"credentials":{"api_key":"fixture-key","storage":"memory"},"thinking":{"type":"enabled","budgetTokens":4096}})).unwrap();
+ let selection = rt
+ .connections
+ .resolve(&crate::connections::Choice::new(
+ "Fixture",
+ "claude-fixture",
+ ))
+ .unwrap();
+ rt.update(&sid, "model_bound", json!({}), |s| {
+ s.model_selection = Some(selection);
+ Ok(())
+ })
+ .unwrap();
+ tokio::time::timeout(
+ std::time::Duration::from_secs(10),
+ rt.drive(&sid, Default::default(), vec![crate::planning::schema()]),
+ )
+ .await
+ .unwrap()
+ .unwrap();
+ tokio::time::timeout(std::time::Duration::from_secs(10), server)
+ .await
+ .unwrap()
+ .unwrap();
+ let s = rt.store.lock().unwrap().get(&sid).unwrap();
+ assert_eq!(s.turn.as_ref().unwrap().status, "completed");
+ assert_eq!(
+ s.turn.as_ref().unwrap().budget.stop_reason.as_deref(),
+ Some("model_call_limit")
+ );
+ assert_eq!(
+ s.history.last().unwrap()["content"][0]["text"],
+ "Stopped at the configured limit; name lookup is incomplete."
+ );
+ assert!(
+ s.plan.is_none(),
+ "The work call stopped before its queued tool executed"
+ );
+ }
+}
+
#[tokio::test]
async fn instruction_addendum_applies_to_retries_and_tool_followups_only_in_its_turn() {
let count = Arc::new(AtomicUsize::new(0));
diff --git a/harness/runtime/src/runtime_safety.rs b/harness/runtime/src/runtime_safety.rs
index 9ca2e994..307e039d 100644
--- a/harness/runtime/src/runtime_safety.rs
+++ b/harness/runtime/src/runtime_safety.rs
@@ -152,12 +152,13 @@ impl Runtime {
.check_content(sid, &config, Phase::ModelOutput, text.clone(), None, None)
.await?;
let mut rejected_calls = vec![];
- // Google may sign only the first call in a parallel batch. Treat
- // every call in that candidate as part of the signed response.
- let signed = completion
- .output
- .iter()
- .any(|item| crate::chat_completions::signed_tool_id(item).is_some());
+ // Google may sign the first text part or only the first call in a
+ // parallel batch. Every call in either candidate belongs to that
+ // signed response; no sibling call may be silently rewritten.
+ let signed = completion.output.iter().any(|item| {
+ crate::chat_completions::signed_tool_id(item).is_some()
+ || crate::chat_completions::message_signature(item)
+ });
if blocked.is_none() {
for call in completion
.output
diff --git a/harness/runtime/src/runtime_safety_tests.rs b/harness/runtime/src/runtime_safety_tests.rs
index dde00cf3..a51710e5 100644
--- a/harness/runtime/src/runtime_safety_tests.rs
+++ b/harness/runtime/src/runtime_safety_tests.rs
@@ -93,7 +93,9 @@ fn signed_tool_config(action: Action) -> Config {
#[tokio::test]
async fn signed_tool_redaction_and_denial_discard_candidate_with_bounded_retry() {
for action in [Action::redact(), Action::Block] {
- for recovers in [true, false] {
+ for (recovers, message_level) in
+ [(true, false), (false, false), (true, true), (false, true)]
+ {
let seen = Arc::new(AtomicUsize::new(0));
let count = seen.clone();
let mut model = Model::new("test".into(), true);
@@ -111,6 +113,16 @@ async fn signed_tool_redaction_and_denial_discard_candidate_with_bounded_retry()
// Only the first call is signed; an unsafe parallel sibling
// must also cause the entire candidate to be discarded.
let mut batch = signed_write("{\"path\":\"safe.txt\",\"content\":\"safe\"}");
+ if message_level {
+ // Some Google models sign the first text part instead
+ // of a function call. The following tools remain part
+ // of the same signed candidate.
+ batch[0]["block"]["target"] = json!("message");
+ batch[0]["block"]["id"] = json!("signed-message");
+ let mut text = message("I will write both files.");
+ text["id"] = json!("signed-message");
+ batch.insert(1, text);
+ }
batch.push(json!({"type":"function_call","name":"write_file","call_id":"parallel-call","arguments":"{\"path\":\"no-write.txt\",\"content\":\"SECRET_SENTINEL\"}"}));
batch
};
diff --git a/harness/runtime/tests/anthropic.rs b/harness/runtime/tests/anthropic.rs
index c3b0ee49..b990ccd8 100644
--- a/harness/runtime/tests/anthropic.rs
+++ b/harness/runtime/tests/anthropic.rs
@@ -164,7 +164,7 @@ fn thinking_generation_configuration_is_explicit_and_budgeted() {
use openworker_runtime::anthropic::Thinking;
assert_eq!(
Thinking::Adaptive.request(4096).unwrap(),
- json!({"type":"adaptive"})
+ Some(json!({"type":"adaptive"}))
);
assert_eq!(
Thinking::Enabled {
@@ -172,13 +172,32 @@ fn thinking_generation_configuration_is_explicit_and_budgeted() {
}
.request(4096)
.unwrap(),
- json!({"type":"enabled","budget_tokens":1024})
+ Some(json!({"type":"enabled","budget_tokens":1024}))
+ );
+ assert_eq!(
+ Thinking::Enabled {
+ budget_tokens: 1024
+ }
+ .request(1024)
+ .unwrap(),
+ None
+ );
+ assert_eq!(
+ Thinking::Enabled {
+ budget_tokens: 4096
+ }
+ .request(2048)
+ .unwrap(),
+ Some(json!({"type":"enabled","budget_tokens":2047}))
+ );
+ assert_eq!(
+ Thinking::Enabled {
+ budget_tokens: 4096
+ }
+ .request(1025)
+ .unwrap(),
+ Some(json!({"type":"enabled","budget_tokens":1024}))
);
- assert!(Thinking::Enabled {
- budget_tokens: 1024
- }
- .request(1024)
- .is_err());
assert!(Thinking::Enabled { budget_tokens: 5 }.validate().is_err());
}
#[test]
diff --git a/harness/store-postgres/tests/store.rs b/harness/store-postgres/tests/store.rs
index 59171a3e..4774c85c 100644
--- a/harness/store-postgres/tests/store.rs
+++ b/harness/store-postgres/tests/store.rs
@@ -126,6 +126,12 @@ fn imported_history_and_turn_context_survive_postgres_atomicity_and_takeover() {
"DROP TRIGGER reject_import_event ON events; DROP FUNCTION reject_import_event();",
)
.unwrap();
+ // A database fault deliberately fences this adapter. Reconcile rollback
+ // through the independent admin connection above, then reconnect rather
+ // than reusing a writer whose outcome was uncertain.
+ assert!(owner.get(&s.id).is_err());
+ drop(owner);
+ let mut owner = PostgresStore::open(config.clone()).unwrap();
owner.save(&s, "session_created", json!({})).unwrap();
assert!(owner.get(&s.id).unwrap().turn.is_none());
let context = Context {
diff --git a/harness/typescript-sdk/test/live-tool-conversation.mjs b/harness/typescript-sdk/test/live-tool-conversation.mjs
new file mode 100644
index 00000000..444d11b7
--- /dev/null
+++ b/harness/typescript-sdk/test/live-tool-conversation.mjs
@@ -0,0 +1,85 @@
+// Opt-in paid smoke test: streamed tool use, accounting, and durable replay.
+// Only synthetic data and a local no-side-effect lookup are used.
+import assert from "node:assert/strict";
+import { randomUUID } from "node:crypto";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { Harness } from "../dist/index.js";
+
+if (process.env.OW_RUN_LIVE !== "1") throw new Error("Set OW_RUN_LIVE=1 for paid qualification");
+for (const key of ["OW_TS_TEST_HOST", "OW_LIVE_MODEL", "OW_LIVE_API", "OW_LIVE_KEY_ENV"])
+ if (!process.env[key]) throw new Error(`${key} is required`);
+assert.ok(["responses", "chat-completions", "anthropic"].includes(process.env.OW_LIVE_API));
+assert.ok(process.env[process.env.OW_LIVE_KEY_ENV], "Provider key is missing");
+if (process.env.OW_LIVE_API === "chat-completions")
+ assert.ok(process.env.OW_LIVE_BASE_URL, "Compatible providers require an explicit endpoint");
+
+const stateDir = await mkdtemp(join(tmpdir(), "ow-live-tools-"));
+const receipt = `check-${randomUUID()}`;
+let harness;
+let toolCalls = 0;
+async function start() {
+ harness = await Harness.start({ executable: process.env.OW_TS_TEST_HOST, stateDir, sandbox: "off" });
+ await harness.setModelProvider({
+ name: "LiveQualification", api: process.env.OW_LIVE_API,
+ ...(process.env.OW_LIVE_BASE_URL ? { baseUrl: process.env.OW_LIVE_BASE_URL } : {}),
+ credentials: { apiKeyEnv: process.env.OW_LIVE_KEY_ENV, storage: "memory" },
+ ...(process.env.OW_LIVE_API === "anthropic" ? { thinking: { type: "adaptive" } } : {}),
+ });
+ await harness.registerTools({ namespace: "app", tools: { lookup: {
+ description: "Read a synthetic price and receipt for atlas.example. This performs no external action.",
+ inputSchema: { type: "object", properties: { name: { type: "string", enum: ["atlas.example"] } }, required: ["name"], additionalProperties: false },
+ execute(args) {
+ assert.equal(args.name, "atlas.example");
+ toolCalls++;
+ assert.ok(toolCalls <= 1, "The previously completed lookup must not replay");
+ return { name: args.name, priceUsd: 12.34, receipt };
+ },
+ } } });
+ return harness.createAgent({
+ name: "tool-qualification", instructions: "Follow the test request precisely. Use app.lookup only when requested. Copy the tool receipt exactly into your final answer. Keep final answers under 60 words.",
+ tools: ["app.lookup"], builtinTools: { askUser: false, readHistory: false },
+ planning: false, memory: false, scheduling: false,
+ approvals: { mode: "only", tools: [] },
+ limits: { maxModelCalls: 4, maxOutputTokens: 12288, maxOutputTokensPerCall: 4096, summaryTokens: 256 },
+ defaultModel: { provider: "LiveQualification", name: process.env.OW_LIVE_MODEL },
+ });
+}
+async function turn(session, input) {
+ const run = await session.send(input, { output: { reasoning: true } });
+ const totals = { input: 0, output: 0, usageEvents: 0, reasoningCharacters: 0, textCharacters: 0, attempts: 0 };
+ for await (const item of run.events()) {
+ if (item.summary?.type === "reasoning_delta") totals.reasoningCharacters += item.summary.text.length;
+ if (item.summary?.type === "assistant_delta") totals.textCharacters += item.summary.text.length;
+ if (item.summary?.type === "model_attempt_started") totals.attempts++;
+ if (item.event.type === "model_usage") {
+ totals.usageEvents++;
+ totals.input += item.event.data.input_tokens;
+ totals.output += item.event.data.output_tokens;
+ }
+ }
+ const result = await run.result();
+ if (result.status !== "completed") {
+ console.log(JSON.stringify({ status: result.status, providerError: result.providerError, totals, toolCalls }));
+ throw new Error("Live conversation did not complete");
+ }
+ assert.ok(result.finalOutput.includes(receipt), "Final answer must use the actual tool receipt");
+ assert.ok(totals.usageEvents > 0 && totals.input > 0 && totals.output > 0, "Provider usage must be reported");
+ return totals;
+}
+try {
+ const agent = await start();
+ const session = await agent.newSession();
+ const first = await turn(session, "Call app.lookup once for atlas.example. Then give its price and exact receipt.");
+ assert.equal(toolCalls, 1);
+ const reference = session.reference();
+ await harness.shutdown();
+ const restarted = await start();
+ const second = await turn(await restarted.resumeSession(reference), "Repeat the exact receipt from the previous lookup. Do not call any tool.");
+ assert.equal(toolCalls, 1, "Restart/history replay must not execute the old call");
+ console.log(JSON.stringify({ model: process.env.OW_LIVE_MODEL, api: process.env.OW_LIVE_API, first, afterRestart: second, toolCalls }));
+} finally {
+ try { await harness?.shutdown(); }
+ finally { await rm(stateDir, { recursive: true, force: true }); }
+}