Skip to content
Python

feat(sdk): expose safe attempt streams and provider diagnostics

Rohit P1dcaeaa4b93d
diff --git a/docs/harness/guides/providers.md b/docs/harness/guides/providers.md
index 22839167..83e3539e 100644
--- a/docs/harness/guides/providers.md
+++ b/docs/harness/guides/providers.md
@@ -27,6 +27,9 @@ and [guardrails guide](guardrails.md).
A compatible endpoint does not guarantee equivalent tool or image behavior;
qualify the specific model/configuration used by your application.
+See [Streaming and retries](streaming.md) for provider reasoning display, explicit
+Anthropic thinking settings and structured failure diagnostics.
+
## Streaming token usage
Chat Completions providers request `stream_options: { include_usage: true }` by
diff --git a/docs/harness/guides/streaming.md b/docs/harness/guides/streaming.md
new file mode 100644
index 00000000..c247407e
--- /dev/null
+++ b/docs/harness/guides/streaming.md
@@ -0,0 +1,115 @@
+# Streaming, retries and provider diagnostics
+
+An agent turn can make several model calls, and a failed call can be retried. Build
+the UI around attempt IDs rather than concatenating every text delta into one
+permanent message. A successful model attempt is still provisional until the
+turn completes; it can request tools or be followed by another model call.
+
+```typescript
+import { AttemptOutput } from "@openworkerai/harness-sdk";
+
+const display = new AttemptOutput(); // Create one projection for this run.
+const run = await session.send("Explain the trade-offs", {
+ output: { reasoning: true, bufferAttempts: false },
+});
+for await (const item of run.events()) {
+ display.apply(item);
+ // Render as escaped text; never inject model text as HTML.
+ answerElement.textContent = display.text;
+ thinkingElement.textContent = display.reasoning;
+}
+const result = await run.result();
+if (result.status !== "completed") display.interrupt();
+// Save result.finalOutput as the final answer, not the accumulated stream.
+```
+
+The helper removes failed candidate text and reasoning as soon as the attempt
+fails. Replayed event IDs are ignored. It limits retained attempts and characters
+for display; these limits do not truncate the durable transcript. After cancelling
+a run, call `interrupt()` to discard any remaining provisional output. To avoid
+showing failed candidates at all, set `bufferAttempts: true`: the runtime releases
+display deltas only after that provider attempt completes successfully. This
+trades incremental output for a quieter UI during retries.
+
+Rust exposes the same options through `TurnOptions.output` / `OutputOptions` and
+typed `RunEvent::AttemptStarted`, `AttemptFinished`, `TextDelta`, `ReasoningDelta`,
+`ModelActivity` and `ProviderError`. Keep separate buffers per
+`(session_id, turn_id, attempt_id)` and discard a failed or interrupted buffer.
+TypeScript summaries use `attemptId`; the raw journal retains `attempt_id`.
+
+| Event | Meaning |
+| --- | --- |
+| `model_attempt_started` | A provider attempt began. |
+| `assistant_delta` | Candidate answer text for that attempt. |
+| `reasoning_delta` | Opt-in provider-designated display reasoning, separate from the answer. |
+| `model_activity` | The request is still pending; includes elapsed milliseconds, at most once every five seconds. |
+| `model_attempt_finished` | That attempt completed or failed. |
+| `provider_error` | Safe structured diagnostics for a failed attempt. |
+
+An activity event is a liveness signal, not a claim that the model has completed
+a particular task. It does not extend the request timeout or add provider calls.
+
+## Enable reasoning display
+
+Reasoning display is off by default. `output.reasoning: true` exposes only the
+provider's supported display text. It never decodes encrypted reasoning, exposes
+signatures or reveals redacted thinking. Display is bounded to 64 KiB per attempt;
+`reasoning_display_limited` records when that display bound is reached.
+
+- **OpenAI Responses:** requests `reasoning.summary: "auto"` and forwards
+ `response.reasoning_summary_text.delta`. Use a model that supports summaries.
+- **Anthropic:** forwards `thinking_delta`. Explicitly configure thinking
+ generation for the selected model; display alone does not enable generation.
+- **Chat Completions:** forwards `reasoning_content` or `reasoning` text when the
+ selected endpoint/model supplies it. The harness does not guess a vendor's
+ generation settings or extract hidden reasoning from answer text.
+
+For Anthropic, choose a mode supported by your exact model:
+
+```typescript
+await harness.setModelProvider({
+ name: "Claude", api: "anthropic",
+ credentials: { apiKeyEnv: "ANTHROPIC_API_KEY" },
+ thinking: { type: "adaptive" },
+ // For a model using an explicit thinking budget instead:
+ // thinking: { type: "enabled", budgetTokens: 2048 },
+});
+```
+
+Rust uses `ModelProvider::anthropic("Claude").thinking(AnthropicThinking::Adaptive)`
+or `AnthropicThinking::Enabled { budget_tokens: 2048 }`. An enabled budget must
+be at least 1,024 tokens and smaller than the current call's output limit. These
+settings apply to main agent calls, not short connection probes, compactors or
+reviewers. Omitted settings preserve an existing choice; explicitly use
+`{ type: "disabled" }` to disable thinking generation. Provider configuration
+persists across restart and credential rotation.
+
+Output guardrails and blocking output/tool hooks suppress candidate reasoning as
+well as candidate answer text, including for agents without tools. Display
+opt-in does not bypass inspection or allow withheld content into telemetry.
+
+See the provider contracts for [OpenAI summaries](https://developers.openai.com/api/docs/guides/reasoning#reasoning-summaries),
+[Anthropic thinking streams](https://platform.claude.com/docs/en/build-with-claude/streaming#thinking-delta)
+and [Together reasoning fields](https://docs.together.ai/docs/inference/chat/reasoning).
+
+## Diagnose provider failures
+
+Failed run results expose `providerError` in TypeScript or `provider_error` in
+Rust. The diagnostic contains a category, retryability, and available HTTP status,
+API, request ID and fixed parser/allowlisted upstream code. Probe failures expose
+the same data through `HarnessError.context.providerError` or Rust
+`Error::Provider`. An `invalid_tool_call_index` code, for example, identifies a
+malformed streamed tool-call index without logging the tool arguments.
+
+```typescript
+const result = await run.result();
+if (result.providerError) {
+ const { category, httpStatus, requestId, code } = result.providerError;
+ console.error({ category, httpStatus, requestId, code });
+}
+```
+
+Upstream response bodies, prompts, keys and arbitrary error messages are not
+included in these diagnostics. Use the request ID with the provider when more
+detail is needed. A retryable provider error does not authorize an application to
+replay tool actions: retain the SDK's run reference and reconcile unknown outcomes.
diff --git a/harness/harness-sdk/src/agents/session.rs b/harness/harness-sdk/src/agents/session.rs
index 2c9c4def..5422ed3d 100644
--- a/harness/harness-sdk/src/agents/session.rs
+++ b/harness/harness-sdk/src/agents/session.rs
@@ -123,6 +123,7 @@ pub struct RunResult {
pub cursor: i64,
/// Team board at the end of the run, for sessions with subagents.
pub board: Option<Value>,
+ pub provider_error: Option<openworker_runtime::provider::Diagnostic>,
}
impl RunResult {
fn from_wire(v: &Value) -> Self {
@@ -144,6 +145,7 @@ impl RunResult {
submission_id: v["submission_id"].as_str().map(str::to_owned),
cursor: v["cursor"].as_i64().unwrap_or(0),
board: v.get("board").filter(|b| !b.is_null()).cloned(),
+ provider_error: serde_json::from_value(v["provider_error"].clone()).ok(),
}
}
}
@@ -151,6 +153,12 @@ impl RunResult {
/// Progress while a run is streaming.
#[derive(Clone, Debug)]
pub enum RunEvent {
+ AttemptStarted { session_id: String, turn_id: Option<String>, attempt_id: String, attempt: u32 },
+ AttemptFinished { session_id: String, turn_id: Option<String>, attempt_id: String, status: String, failure: Option<openworker_runtime::provider::Failure> },
+ TextDelta { session_id: String, turn_id: Option<String>, attempt_id: String, text: String },
+ ReasoningDelta { session_id: String, turn_id: Option<String>, attempt_id: String, kind: openworker_runtime::provider::ReasoningKind, text: String, block_index: u64 },
+ ModelActivity { session_id: String, turn_id: Option<String>, attempt_id: String, elapsed_ms: u64 },
+ ProviderError { session_id: String, turn_id: Option<String>, attempt_id: String, diagnostic: openworker_runtime::provider::Diagnostic },
/// Named-team events retain both instance and definition identities.
TeamActivity {
agent_id: String,
@@ -205,6 +213,9 @@ impl RunEvent {
}
let agent = item["agent"].as_str().unwrap_or_default().to_owned();
let summary = &item["summary"];
+ let session_id = item["event"]["session_id"].as_str().unwrap_or_default().to_owned();
+ let turn_id = item["event"]["turn_id"].as_str().map(str::to_owned);
+ let attempt_id = || summary["attempt_id"].as_str().unwrap_or_default().to_owned();
let tool = || summary["tool"].as_str().unwrap_or_default().to_owned();
// Older hosts omit this from summaries, but retain it in the journal.
let call_id = || {
@@ -215,6 +226,12 @@ impl RunEvent {
.to_owned()
};
Some(match summary["type"].as_str() {
+ Some("model_attempt_started") => RunEvent::AttemptStarted { session_id, turn_id, attempt_id:attempt_id(), attempt:summary["attempt"].as_u64()? as u32 },
+ Some("model_attempt_finished") => RunEvent::AttemptFinished { session_id, turn_id, attempt_id:attempt_id(), status:summary["status"].as_str()?.into(), failure:serde_json::from_value(summary["kind"].clone()).ok() },
+ Some("assistant_delta") => RunEvent::TextDelta { session_id, turn_id, attempt_id:attempt_id(), text:summary["text"].as_str()?.into() },
+ Some("reasoning_delta") => RunEvent::ReasoningDelta { session_id, turn_id, attempt_id:attempt_id(), kind:serde_json::from_value(summary["kind"].clone()).ok()?, text:summary["text"].as_str()?.into(), block_index:summary["block_index"].as_u64()? },
+ Some("model_activity") => RunEvent::ModelActivity { session_id, turn_id, attempt_id:attempt_id(), elapsed_ms:summary["elapsed_ms"].as_u64()? },
+ Some("provider_error") => RunEvent::ProviderError { session_id, turn_id, attempt_id:attempt_id(), diagnostic:serde_json::from_value(summary["diagnostic"].clone()).ok()? },
Some("message") => RunEvent::Message {
agent,
text: summary["text"].as_str().unwrap_or_default().to_owned(),
@@ -843,4 +860,17 @@ mod tests {
assert!(keys[2].0.is_none());
}
}
+
+ #[test]
+ fn attempt_reasoning_and_provider_diagnostics_are_typed() {
+ let event=json!({"session_id":"s","turn_id":"t","data":{}});
+ let item=|summary:Value|json!({"agent":"a","event":event,"summary":summary});
+ assert!(matches!(RunEvent::from_wire(&item(json!({"type":"model_attempt_started","attempt_id":"a1","attempt":1}))),Some(RunEvent::AttemptStarted{attempt:1,attempt_id,..}) if attempt_id=="a1"));
+ assert!(matches!(RunEvent::from_wire(&item(json!({"type":"model_attempt_finished","attempt_id":"a1","status":"failed","kind":"incomplete_stream"}))),Some(RunEvent::AttemptFinished{failure:Some(openworker_runtime::provider::Failure::IncompleteStream),..})));
+ assert!(matches!(RunEvent::from_wire(&item(json!({"type":"reasoning_delta","attempt_id":"a1","kind":"summary","text":"display","block_index":0}))),Some(RunEvent::ReasoningDelta{kind:openworker_runtime::provider::ReasoningKind::Summary,text,..}) if text=="display"));
+ let diagnostic=json!({"category":"authentication","retryable":false,"http_status":401,"request_id":"req_fixture","api":"responses","code":"invalid_api_key"});
+ assert!(matches!(RunEvent::from_wire(&item(json!({"type":"provider_error","attempt_id":"a1","diagnostic":diagnostic}))),Some(RunEvent::ProviderError{diagnostic,..}) if diagnostic.http_status==Some(401)));
+ let result=RunResult::from_wire(&json!({"status":"failed","provider_error":diagnostic}));
+ assert_eq!(result.provider_error.unwrap().request_id.as_deref(),Some("req_fixture"));
+ }
}
diff --git a/harness/harness-sdk/src/host_runs.rs b/harness/harness-sdk/src/host_runs.rs
index 20ec59ed..b0154657 100644
--- a/harness/harness-sdk/src/host_runs.rs
+++ b/harness/harness-sdk/src/host_runs.rs
@@ -985,6 +985,7 @@ impl Host {
}));
if status == "failed" && cached.is_none() {
out["failure"] = lead_turn["status"].clone();
+ out["provider_error"] = lead_turn["provider_error"].clone();
}
if cached.is_none() && status != "running" && status != "interrupted" {
if let Some(r) = record.as_mut() {
@@ -1084,6 +1085,12 @@ fn final_output(snapshot: &Value) -> Value {
/// A journal event plus a small typed summary SDKs can map directly.
fn stream_item(agent: &str, e: Event) -> Value {
let summary = match e.kind.as_str() {
+ "model_attempt_started" | "model_attempt_finished" | "assistant_delta"
+ | "reasoning_delta" | "model_activity" | "provider_error" => {
+ let mut summary = e.data.clone();
+ summary["type"] = json!(e.kind);
+ Some(summary)
+ }
"interaction_resolved" => (e.data["origin"] == "app_policy")
.then(|| json!({"type": "auto_approved", "tool": e.data["tool"]})),
"assistant_message" => {
diff --git a/harness/harness-sdk/src/lib.rs b/harness/harness-sdk/src/lib.rs
index cba78732..383237bd 100644
--- a/harness/harness-sdk/src/lib.rs
+++ b/harness/harness-sdk/src/lib.rs
@@ -36,6 +36,8 @@ pub use openworker_telemetry::otlp::{
/// Opt-in prompt/tool content in telemetry (off by default).
pub use openworker_telemetry::{Capture, ContentPolicy};
pub use providers::{Credentials, Model, ModelApi, ModelProvider};
+pub use openworker_runtime::anthropic::Thinking as AnthropicThinking;
+pub use openworker_runtime::provider::{Diagnostic as ProviderDiagnostic, Failure as ProviderFailure, ReasoningKind};
pub enum EmbeddedConfig {
Local(RuntimeConfig),
@@ -81,6 +83,7 @@ pub type Result<T> = std::result::Result<T, Error>;
pub enum Error {
InvalidInput(String),
Remote(String),
+ Provider(openworker_runtime::provider::Diagnostic),
Setup(String),
Unavailable,
/// May have committed or started. Reconcile durable IDs; do not blindly retry.
@@ -91,6 +94,7 @@ impl std::fmt::Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::InvalidInput(s) | Self::Remote(s) | Self::Setup(s) => f.write_str(s),
+ Self::Provider(diagnostic) => std::fmt::Display::fmt(diagnostic, f),
Self::Unavailable => f.write_str("Harness unavailable or recovering; request not sent"),
Self::OutcomeUnknown => f.write_str("Harness response lost; execution outcome unknown. Reconcile saved session before retrying."),
Self::Closed => f.write_str("Harness client closed"),
diff --git a/harness/harness-sdk/src/providers.rs b/harness/harness-sdk/src/providers.rs
index 03e78f7d..cfef6a8b 100644
--- a/harness/harness-sdk/src/providers.rs
+++ b/harness/harness-sdk/src/providers.rs
@@ -50,6 +50,7 @@ pub struct ModelProvider {
base_url: Option<String>,
stream_usage: Option<bool>,
prompt_caching: Option<bool>,
+ thinking: Option<openworker_runtime::anthropic::Thinking>,
credentials: Option<Credentials>,
pub(crate) access_auth: bool,
pub(crate) refresh: Option<openworker_runtime::provider_auth::Refresh>,
@@ -71,6 +72,7 @@ impl ModelProvider {
base_url: None,
stream_usage: None,
prompt_caching: None,
+ thinking: None,
credentials: None,
access_auth: false,
refresh: None,
@@ -93,6 +95,11 @@ impl ModelProvider {
self.prompt_caching = Some(enabled);
self
}
+ /// Explicit Anthropic generation mode; choose one supported by your model.
+ pub fn thinking(mut self, thinking: openworker_runtime::anthropic::Thinking) -> Self {
+ self.thinking = Some(thinking);
+ self
+ }
pub fn credentials(mut self, value: Credentials) -> Self {
self.credentials = Some(value);
self
@@ -128,6 +135,10 @@ impl ModelProvider {
"prompt_caching is only supported for Anthropic Messages".into(),
));
}
+ if let Some(thinking) = &self.thinking {
+ if self.api != ModelApi::Anthropic { return Err(Error::InvalidInput("thinking is only supported for Anthropic".into())); }
+ thinking.validate().map_err(|e|Error::InvalidInput(e.to_string()))?;
+ }
if self.stream_usage.is_some() && self.api != ModelApi::ChatCompletions {
return Err(Error::InvalidInput(
"stream_usage is only supported for Chat Completions".into(),
@@ -145,6 +156,7 @@ impl ModelProvider {
if let Some(enabled) = self.prompt_caching {
wire["prompt_caching"] = json!(enabled);
}
+ if let Some(thinking) = self.thinking { wire["thinking"] = json!(thinking); }
if self.access_auth {
wire["access_token_auth"] = json!(true);
}
diff --git a/harness/harness-sdk/src/transport.rs b/harness/harness-sdk/src/transport.rs
index c93b1757..4fcd2da3 100644
--- a/harness/harness-sdk/src/transport.rs
+++ b/harness/harness-sdk/src/transport.rs
@@ -6,6 +6,13 @@ use tokio::{
process::{Child, ChildStdin},
};
+fn provider_error(error: anyhow::Error) -> Error {
+ match error.downcast_ref::<openworker_runtime::provider::Error>() {
+ Some(error) => Error::Provider(error.diagnostic()),
+ None => Error::Remote(error.to_string()),
+ }
+}
+
pub(crate) enum Backend {
Host {
host: Arc<crate::host::Host>,
@@ -342,7 +349,7 @@ impl Backend {
tokio::pin!(request);
loop {
tokio::select! {
- result=&mut request=>break result.map_err(|e|Error::Remote(e.to_string())),
+ result=&mut request=>break result.map_err(provider_error),
value=application.recv()=>{let value=value.map_err(|_|Error::OutcomeUnknown)?;let _=sink.send(Notification::Application(value));},
}
}
@@ -350,7 +357,7 @@ impl Backend {
Self::Embedded { runtime, .. } => runtime
.dispatch(method, params)
.await
- .map_err(|e| Error::Remote(e.to_string())),
+ .map_err(provider_error),
Self::Process(p) => {
p.next_id += 1;
let id = p.next_id;
@@ -380,6 +387,9 @@ impl Backend {
return Err(Error::OutcomeUnknown);
}
if let Some(message) = v["error"]["message"].as_str() {
+ if let Ok(diagnostic) = serde_json::from_value::<openworker_runtime::provider::Diagnostic>(v["error"]["provider_error"].clone()) {
+ return Err(Error::Provider(diagnostic));
+ }
return Err(Error::Remote(message.into()));
}
return v.get("result").cloned().ok_or(Error::OutcomeUnknown);
diff --git a/harness/runtime/examples/provider_probe.rs b/harness/runtime/examples/provider_probe.rs
index 7313222f..f16cd33d 100644
--- a/harness/runtime/examples/provider_probe.rs
+++ b/harness/runtime/examples/provider_probe.rs
@@ -44,7 +44,7 @@ async fn main() -> Result<()> {
p=rx.recv(),if !closed=>match p {
Some(Progress::Delta(_))=>delta_count+=1,
Some(Progress::Complete(c))=>complete=Some(c),
- Some(Progress::OutputStarted|Progress::Usage(_))=>{},
+ Some(Progress::Reasoning(_)|Progress::OutputStarted|Progress::Usage(_))=>{},
None=>closed=true,
}
}
diff --git a/harness/runtime/src/anthropic.rs b/harness/runtime/src/anthropic.rs
index 2205745c..a407ed22 100644
--- a/harness/runtime/src/anthropic.rs
+++ b/harness/runtime/src/anthropic.rs
@@ -8,11 +8,39 @@ use serde_json::{json, Value};
use std::collections::{BTreeMap, BTreeSet};
use tokio::sync::mpsc;
+/// Explicit generation configuration; display is controlled separately per turn.
+#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+#[serde(tag="type",rename_all="snake_case",deny_unknown_fields)]
+pub enum Thinking {
+ Disabled,
+ Adaptive,
+ Enabled { #[serde(rename="budgetTokens")] budget_tokens: u32 },
+}
+impl Thinking {
+ pub fn validate(&self) -> anyhow::Result<()> {
+ if let Self::Enabled { budget_tokens } = self {
+ anyhow::ensure!((1024..=1_000_000).contains(budget_tokens), "Thinking budget must be between 1024 and 1000000 tokens");
+ }
+ Ok(())
+ }
+ pub fn request(&self, max_tokens: usize) -> Result<Value, Error> {
+ self.validate().map_err(|_| config())?;
+ Ok(match self {
+ Self::Disabled => json!({"type":"disabled"}),
+ Self::Adaptive => json!({"type":"adaptive"}),
+ Self::Enabled { budget_tokens } => {
+ if *budget_tokens as usize >= max_tokens { return Err(config()); }
+ json!({"type":"enabled","budget_tokens":budget_tokens})
+ }
+ })
+ }
+}
+
fn invalid() -> Error {
- Error::new(Failure::InvalidResponse)
+ Error::new(Failure::InvalidResponse).with_code("invalid_anthropic_message")
}
fn config() -> Error {
- Error::new(Failure::Configuration)
+ Error::new(Failure::Configuration).with_code("invalid_anthropic_request")
}
fn append(messages: &mut Vec<Value>, role: &str, blocks: Vec<Value>) {
@@ -171,8 +199,10 @@ pub struct Decoder {
usage: Usage,
output_known: bool,
input_known: bool,
+ reasoning: Option<crate::provider::ReasoningDelta>,
}
impl Decoder {
+ pub fn take_reasoning(&mut self) -> Option<crate::provider::ReasoningDelta> { self.reasoning.take() }
fn usage_report(&self) -> Option<crate::telemetry::TokenUsage> {
self.input_known.then(|| {
let mut u = self.usage.reported();
@@ -296,6 +326,13 @@ impl Decoder {
if field == "text" {
return Ok(Some(fragment.into()));
}
+ if field == "thinking" {
+ self.reasoning = Some(crate::provider::ReasoningDelta {
+ kind: crate::provider::ReasoningKind::Thinking,
+ text: fragment.into(),
+ block_index: e["index"].as_u64().ok_or_else(invalid)?,
+ });
+ }
}
Some("content_block_stop") => {
let block = self
@@ -422,6 +459,7 @@ pub async fn request(
.await
.map_err(|_| Error::new(Failure::Transport))?;
let status = response.status().as_u16();
+ let request_id = crate::provider::request_id(response.headers(), key);
let retry_after = response
.headers()
.get("retry-after")
@@ -431,7 +469,7 @@ pub async fn request(
if !(200..300).contains(&status) {
let mut bytes = vec![];
while let Some(chunk) = stream.next().await {
- let chunk = chunk.map_err(|_| Error::new(classify(status, &Value::Null)))?;
+ let chunk = chunk.map_err(|_| Error {kind:classify(status,&Value::Null),retry_after,http_status:Some(status),request_id:request_id.clone(),api:Some(crate::model_api::Api::Anthropic),code:Some("http_error_body_unavailable")})?;
if bytes.len() + chunk.len() > 16384 {
break;
}
@@ -440,6 +478,9 @@ pub async fn request(
let error: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
return Err(Error {
http_status: Some(status),
+ request_id,
+ api: Some(crate::model_api::Api::Anthropic),
+ code: crate::provider::upstream_code(&error["error"]),
kind: classify(status, &error["error"]),
retry_after,
});
@@ -472,6 +513,9 @@ pub async fn request(
previous_usage = Some(usage);
}
let decoded = decoded?;
+ if let Some(delta) = decoder.take_reasoning() {
+ tx.send(Progress::Reasoning(delta)).await.map_err(|_| Error::new(Failure::Transport))?;
+ }
if !first_sent && decoder.output_started() {
tx.send(Progress::OutputStarted)
.await
diff --git a/harness/runtime/src/chat_completions.rs b/harness/runtime/src/chat_completions.rs
index 61aab61c..3e98fcee 100644
--- a/harness/runtime/src/chat_completions.rs
+++ b/harness/runtime/src/chat_completions.rs
@@ -4,10 +4,10 @@ use crate::provider::{Completion, Error, Failure, StreamDecoder, Usage};
use serde_json::{json, Value};
use std::collections::{BTreeMap, BTreeSet};
fn invalid() -> Error {
- Error::new(Failure::InvalidResponse)
+ Error::new(Failure::InvalidResponse).with_code("invalid_chat_completion")
}
fn config() -> Error {
- Error::new(Failure::Configuration)
+ Error::new(Failure::Configuration).with_code("invalid_chat_request_history")
}
const GOOGLE_PRIVATE: &str = "google-chat-completions";
@@ -227,6 +227,7 @@ pub struct Decoder {
accounting_tail: bool,
usage: Option<Usage>,
bytes: usize,
+ reasoning: Option<crate::provider::ReasoningDelta>,
binding: Option<String>,
google_compat: bool,
message_signature: Option<String>,
@@ -347,6 +348,16 @@ impl Decoder {
}
None => None,
};
+ let reasoning_content = delta.get("reasoning_content").filter(|v| !v.is_null());
+ let reasoning = delta.get("reasoning").filter(|v| !v.is_null());
+ if reasoning_content.zip(reasoning).is_some_and(|(a,b)|a != b) { return Err(invalid()); }
+ if let Some(text) = reasoning_content.or(reasoning) {
+ self.reasoning = Some(crate::provider::ReasoningDelta {
+ kind: crate::provider::ReasoningKind::Thinking,
+ text: text.as_str().ok_or_else(invalid)?.to_owned(),
+ block_index: 0,
+ });
+ }
if let Some(calls) = delta.get("tool_calls") {
for c in calls.as_array().ok_or_else(invalid)? {
let signature = google_signature(c)?;
@@ -378,7 +389,7 @@ impl Decoder {
}
self.calls.keys().next_back().map_or(0, |index| index + 1)
} else {
- c["index"].as_u64().ok_or_else(invalid)?
+ c["index"].as_u64().ok_or_else(|| invalid().with_code("invalid_tool_call_index"))?
};
if index >= 64 || c.get("type").is_some_and(|v| v != "function") {
return Err(invalid());
@@ -429,6 +440,7 @@ impl Decoder {
}
}
impl StreamDecoder for Decoder {
+ fn take_reasoning(&mut self) -> Option<crate::provider::ReasoningDelta> { self.reasoning.take() }
fn usage_report(&self) -> Option<crate::telemetry::TokenUsage> {
self.usage.as_ref().map(Usage::reported)
}
diff --git a/harness/runtime/src/connections.rs b/harness/runtime/src/connections.rs
index b8e39034..49ce08c0 100644
--- a/harness/runtime/src/connections.rs
+++ b/harness/runtime/src/connections.rs
@@ -74,6 +74,8 @@ struct Set {
stream_usage: Option<bool>,
#[serde(default)]
prompt_caching: Option<bool>,
+ #[serde(default)]
+ thinking: Option<crate::anthropic::Thinking>,
}
impl Manager {
/// Trusted server bootstrap only; not exposed by the model or stdio protocol.
@@ -173,6 +175,7 @@ impl Manager {
)?;
}
db.execute_batch("CREATE TABLE IF NOT EXISTS endpoints(id TEXT PRIMARY KEY, api TEXT NOT NULL, base_url TEXT NOT NULL);")?;
+ db.execute_batch("CREATE TABLE IF NOT EXISTS thinking_options(id TEXT PRIMARY KEY, value TEXT NOT NULL);")?;
let has_stream_usage = db
.prepare("PRAGMA table_info(endpoints)")?
.query_map([], |r| r.get::<_, String>(1))?
@@ -254,6 +257,8 @@ impl Manager {
}
if api == "anthropic" {
connections.last_mut().unwrap()["prompt_caching"] = json!(prompt_caching);
+ let thinking: Option<String> = state.db.query_row("SELECT value FROM thinking_options WHERE id=?1", [&id], |r|r.get(0)).optional()?;
+ if let Some(value) = thinking { connections.last_mut().unwrap()["thinking"] = serde_json::from_str(&value)?; }
}
}
Ok(
@@ -281,6 +286,10 @@ impl Manager {
if input.prompt_caching.is_some() && api != crate::model_api::Api::Anthropic {
bail!("prompt_caching is only supported for Anthropic Messages");
}
+ if let Some(thinking) = &input.thinking {
+ anyhow::ensure!(api == crate::model_api::Api::Anthropic, "thinking is only supported for Anthropic");
+ thinking.validate()?;
+ }
let base_url =
crate::model_api::base_url(input.base_url.as_deref().unwrap_or(api.default_url()))?;
let api_json = serde_json::to_string(&api)?;
@@ -375,6 +384,9 @@ impl Manager {
"INSERT INTO endpoints(id,api,base_url,stream_usage,prompt_caching) VALUES(?1,?2,?3,COALESCE(?4,1),COALESCE(?5,1)) ON CONFLICT(id) DO UPDATE SET stream_usage=COALESCE(?4,endpoints.stream_usage),prompt_caching=COALESCE(?5,endpoints.prompt_caching)",
params![input.id, api_json, base_url, input.stream_usage, input.prompt_caching],
)?;
+ if let Some(thinking) = input.thinking {
+ transaction.execute("INSERT INTO thinking_options(id,value) VALUES(?1,?2) ON CONFLICT(id) DO UPDATE SET value=excluded.value", params![input.id,serde_json::to_string(&thinking)?])?;
+ }
transaction.commit()?;
#[cfg(target_os = "macos")]
if was_secure && input.persistence != "secure" {
@@ -410,6 +422,7 @@ impl Manager {
state
.db
.execute("DELETE FROM endpoints WHERE id=?1", [id])?;
+ state.db.execute("DELETE FROM thinking_options WHERE id=?1", [id])?;
Ok(json!({"removed":true}))
}
pub fn defaults(&self) -> Result<Defaults> {
@@ -635,6 +648,12 @@ impl Manager {
|r| r.get(0),
).context("Selected model provider is missing or its identity changed")
}
+ pub fn thinking(&self, selected: &Selection) -> Result<Option<crate::anthropic::Thinking>> {
+ let value: Option<String> = self.state.lock().unwrap().db.query_row(
+ "SELECT t.value FROM connections c LEFT JOIN thinking_options t ON c.id=t.id WHERE c.id=?1 AND c.identity=?2",
+ params![selected.connection, selected.identity], |r|r.get(0))?;
+ value.map(|v|serde_json::from_str(&v).map_err(Into::into)).transpose()
+ }
/// Harness-owned provider registration; never journal this secret-bearing request.
pub fn set_provider(&self, value: Value) -> Result<Value> {
#[derive(Deserialize)]
@@ -649,6 +668,8 @@ impl Manager {
#[serde(default)]
prompt_caching: Option<bool>,
#[serde(default)]
+ thinking: Option<crate::anthropic::Thinking>,
+ #[serde(default)]
access_token_auth: bool,
}
#[derive(Deserialize)]
@@ -666,6 +687,10 @@ impl Manager {
if input.prompt_caching.is_some() && input.api != crate::model_api::Api::Anthropic {
bail!("prompt_caching is only supported for Anthropic Messages");
}
+ if let Some(thinking) = &input.thinking {
+ anyhow::ensure!(input.api == crate::model_api::Api::Anthropic, "thinking is only supported for Anthropic");
+ thinking.validate()?;
+ }
anyhow::ensure!(
!input.access_token_auth
|| (input.credentials.is_none()
@@ -730,6 +755,9 @@ impl Manager {
params![input.name, serde_json::to_string(&input.api)?, base_url, enabled],
)?;
}
+ if let Some(thinking) = input.thinking {
+ self.state.lock().unwrap().db.execute("INSERT INTO thinking_options(id,value) VALUES(?1,?2) ON CONFLICT(id) DO UPDATE SET value=excluded.value", params![input.name,serde_json::to_string(&thinking)?])?;
+ }
return Ok(json!({"saved":true,"credentials_unchanged":true}));
}
}
@@ -751,7 +779,7 @@ impl Manager {
.into(),
),
};
- self.set(json!({"id":input.name,"label":input.name,"provider":input.api.kind(),"api":input.api,"base_url":base_url,"stream_usage":input.stream_usage,"prompt_caching":input.prompt_caching,"api_key":key,"persistence":storage}))
+ self.set(json!({"id":input.name,"label":input.name,"provider":input.api.kind(),"api":input.api,"base_url":base_url,"stream_usage":input.stream_usage,"prompt_caching":input.prompt_caching,"thinking":input.thinking,"api_key":key,"persistence":storage}))
}
}
#[cfg(not(any(windows, target_os = "macos")))]
diff --git a/harness/runtime/src/model.rs b/harness/runtime/src/model.rs
index 6a191b05..566919ea 100644
--- a/harness/runtime/src/model.rs
+++ b/harness/runtime/src/model.rs
@@ -10,6 +10,8 @@ pub struct Model {
pub observation: Option<CallContext>,
/// Withhold candidate bodies from exporters until runtime safety checks finish.
pub content_withheld: bool,
+ /// Opt-in forwarding of provider-supported display reasoning, never signatures.
+ pub stream_reasoning: bool,
pub connection: Option<(
std::sync::Arc<crate::connections::Manager>,
crate::connections::Selection,
@@ -31,6 +33,7 @@ pub type SampleHook = std::sync::Arc<
pub type SummaryHook = std::sync::Arc<dyn Fn(String) -> TestFuture<String> + Send + Sync>;
pub enum Progress {
Delta(String),
+ Reasoning(crate::provider::ReasoningDelta),
/// Internal measurement only; never authorizes partial provider output.
OutputStarted,
/// Latest cumulative provider snapshot, not a token increment.
@@ -94,6 +97,7 @@ impl Model {
scripted,
observation: None,
content_withheld: false,
+ stream_reasoning: false,
connection: None,
#[cfg(test)]
sample_hook: None,
@@ -277,6 +281,7 @@ impl Model {
};
match &progress {
Progress::Delta(text) if !text.is_empty() => span.first_output(),
+ Progress::Reasoning(delta) if !delta.text.is_empty() => span.first_output(),
Progress::OutputStarted => span.first_output(),
Progress::Usage(usage) => span.reported_usage(*usage),
Progress::Complete(completion) => {
@@ -396,7 +401,7 @@ impl Model {
tx,
crate::chat_completions::Decoder::with_binding(binding, google_compat),
)
- .await?;
+ .await.map_err(|e| e.with_api(api))?;
return Ok(());
}
if api == crate::model_api::Api::Anthropic {
@@ -413,7 +418,16 @@ impl Model {
body.as_object_mut().unwrap().remove("cache_control");
}
}
- crate::anthropic::request(&client, &endpoint, key, body, tx).await?;
+ // Short probes, reviewers and compactors have separate small
+ // budgets; do not apply the agent's thinking generation budget.
+ if self.observation.as_ref().is_some_and(|c| c.purpose == crate::telemetry::ModelPurpose::Main) {
+ if let Some((manager, selection)) = &self.connection {
+ if let Some(thinking) = manager.thinking(selection)? {
+ body["thinking"] = thinking.request(max_tokens).map_err(|e|e.with_api(api))?;
+ }
+ }
+ }
+ crate::anthropic::request(&client, &endpoint, key, body, tx).await.map_err(|e| e.with_api(api))?;
return Ok(());
}
if history.iter().any(|v| v.get("provider").is_some()) {
@@ -431,10 +445,11 @@ impl Model {
})
.collect::<Vec<_>>();
let mut body = json!({"model":self.name,"instructions":instructions,"input":input,"tools":tools,"stream":true,"store":false,"include":["reasoning.encrypted_content"],"parallel_tool_calls":false,"max_output_tokens":max_tokens});
+ if self.stream_reasoning { body["reasoning"] = json!({"summary":"auto"}); }
if let Some(format) = format {
body["text"] = json!({"format":format});
}
- crate::provider::request(&client, &endpoint, key, body, tx).await?;
+ crate::provider::request(&client, &endpoint, key, body, tx).await.map_err(|e| e.with_api(api))?;
Ok(())
}
pub async fn summarize(
@@ -546,7 +561,7 @@ impl Model {
r=&mut request,if !finished=>{r?;finished=true;},
p=rx.recv(),if !closed=>match p {
Some(Progress::Complete(r))=>response=Some(r),
- Some(Progress::Delta(_)|Progress::OutputStarted|Progress::Usage(_))=>{},None=>closed=true
+ Some(Progress::Delta(_)|Progress::Reasoning(_)|Progress::OutputStarted|Progress::Usage(_))=>{},None=>closed=true
}
}
}
diff --git a/harness/runtime/src/protocol.rs b/harness/runtime/src/protocol.rs
index 5bb22c99..1d6b3e2a 100644
--- a/harness/runtime/src/protocol.rs
+++ b/harness/runtime/src/protocol.rs
@@ -132,7 +132,7 @@ pub async fn serve_endpoint(rt: impl Endpoint + 'static) -> Result<()> {
if method=="session/attach"{subscribed.insert(p["session_id"].as_str().unwrap().to_owned());}
Ok::<_,anyhow::Error>(result)
}.await;
- match result{Ok(result)=>json!({"jsonrpc":"2.0","id":id,"result":result}),Err(e)=>json!({"jsonrpc":"2.0","id":id,"error":{"code":-32000,"message":e.to_string()}})}
+ match result{Ok(result)=>json!({"jsonrpc":"2.0","id":id,"result":result}),Err(e)=>json!({"jsonrpc":"2.0","id":id,"error":{"code":-32000,"message":e.to_string(),"provider_error":e.downcast_ref::<crate::provider::Error>().map(|e|e.diagnostic())}})}
}
}
},
diff --git a/harness/runtime/src/provider.rs b/harness/runtime/src/provider.rs
index f839a62d..a798ddd6 100644
--- a/harness/runtime/src/provider.rs
+++ b/harness/runtime/src/provider.rs
@@ -41,6 +41,18 @@ pub enum Failure {
Configuration,
SafetyBlocked,
}
+#[derive(Clone, Debug)]
+pub struct Error {
+ pub kind: Failure,
+ pub retry_after: Option<u64>,
+ /// Set only on an HTTP rejection, never on an error inside a stream.
+ pub http_status: Option<u16>,
+ pub request_id: Option<String>,
+ pub api: Option<crate::model_api::Api>,
+ /// Fixed parser code or an allowlisted upstream code; never arbitrary text.
+ pub code: Option<&'static str>,
+}
+
/// Safe structured context. Never contains request/response bodies or credentials.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
@@ -56,20 +68,30 @@ pub struct Diagnostic {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub code: Option<String>,
}
-#[derive(Clone, Debug)]
-pub struct Error {
- pub kind: Failure,
- pub retry_after: Option<u64>,
- /// Set only on an HTTP rejection, never on an error inside a stream.
- pub http_status: Option<u16>,
+
+impl std::fmt::Display for Diagnostic {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "Provider {:?}", self.category)?;
+ if let Some(status) = self.http_status { write!(f, " (HTTP {status})")?; }
+ if let Some(id) = &self.request_id { write!(f, "; request {id}")?; }
+ if let Some(code) = &self.code { write!(f, "; {code}")?; }
+ Ok(())
+ }
+}
+
+/// Request IDs are identifiers, not arbitrary upstream diagnostic text.
+pub(crate) fn request_id(headers: &reqwest::header::HeaderMap, key: &str) -> Option<String> {
+ ["x-request-id", "request-id"].into_iter().find_map(|name| {
+ let id = headers.get(name)?.to_str().ok()?;
+ (!id.is_empty() && id.len() <= 128
+ && id.bytes().all(|b| b.is_ascii_alphanumeric() || b"-_:".contains(&b))
+ && !id.starts_with("sk-") && (key.is_empty() || !id.contains(key)))
+ .then(|| id.to_owned())
+ })
}
impl std::fmt::Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- write!(
- f,
- "Provider {:?}; no tool actions accepted from this attempt",
- self.kind
- )
+ write!(f, "{}; no tool actions accepted from this attempt", self.diagnostic())
}
}
impl std::error::Error for Error {}
@@ -79,8 +101,20 @@ impl Error {
kind,
retry_after: None,
http_status: None,
+ request_id: None,
+ api: None,
+ code: match kind { Failure::InvalidResponse=>Some("invalid_response"), Failure::IncompleteStream=>Some("incomplete_stream"), _=>None },
}
}
+ pub fn diagnostic(&self) -> Diagnostic {
+ Diagnostic { category:self.kind, retryable:self.retryable(), http_status:self.http_status,
+ request_id:self.request_id.clone(), api:self.api, code:self.code.map(str::to_owned) }
+ }
+ pub fn with_api(mut self, api: crate::model_api::Api) -> Self {
+ self.api = Some(api);
+ self
+ }
+ pub fn with_code(mut self, code: &'static str) -> Self { self.code=Some(code);self }
pub fn retryable(&self) -> bool {
matches!(
self.kind,
@@ -91,6 +125,55 @@ impl Error {
) && self.retry_after.is_none_or(|s| s <= 30)
}
}
+
+pub(crate) fn upstream_code(value: &Value) -> Option<&'static str> {
+ const CODES:&[&str]=&["invalid_api_key","authentication_error","permission_error","invalid_request_error","rate_limit_exceeded","rate_limit_error","insufficient_quota","context_length_exceeded","server_error","api_error","overloaded_error","misalignment_policy_violation","RESOURCE_EXHAUSTED","INVALID_ARGUMENT","UNAUTHENTICATED","PERMISSION_DENIED"];
+ CODES.iter().copied().find(|code| value["code"]==*code || value["type"]==*code || value["status"]==*code)
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all="snake_case")]
+pub enum ReasoningKind { Summary, Thinking }
+
+/// Provider-designated display text only. Opaque signatures never enter this type.
+#[derive(Clone, Debug)]
+pub struct ReasoningDelta {
+ pub kind: ReasoningKind,
+ pub text: String,
+ pub block_index: u64,
+}
+
+#[cfg(test)]
+mod diagnostics_tests {
+ use super::*;
+ use serde_json::json;
+ #[test]
+ fn safe_diagnostics_exclude_credentials_and_unbounded_header_text() {
+ let mut headers = reqwest::header::HeaderMap::new();
+ for value in ["sk-secret-token", "Bearer token", "arbitrary\ttext", "credential-sentinel"] {
+ headers.insert("x-request-id",value.parse().unwrap());
+ assert!(request_id(&headers,"credential-sentinel").is_none());
+ }
+ headers.insert("x-request-id","req_fixture-123".parse().unwrap());
+ assert_eq!(request_id(&headers,"credential-sentinel").as_deref(),Some("req_fixture-123"));
+ let mut e=Error::new(Failure::RateLimit).with_api(crate::model_api::Api::Responses);
+ e.http_status=Some(429);e.request_id=request_id(&headers,"");
+ assert_eq!(serde_json::to_value(e.diagnostic()).unwrap(),json!({"category":"rate_limit","retryable":true,"http_status":429,"request_id":"req_fixture-123","api":"responses"}));
+ }
+ #[test]
+ fn responses_only_exposes_designated_summary_deltas() {
+ let mut d=Decoder::default();
+ for event in [json!({"type":"response.reasoning_text.delta","delta":"hidden"}),json!({"type":"response.output_item.added","item":{"type":"reasoning","encrypted_content":"opaque"}})] {
+ assert!(d.event(&event).unwrap().is_none());
+ assert!(d.take_reasoning().is_none());
+ }
+ d.event(&json!({"type":"response.reasoning_summary_text.delta","delta":"Display summary","output_index":0})).unwrap();
+ let delta=d.take_reasoning().unwrap();
+ assert_eq!(delta.kind,ReasoningKind::Summary);
+ assert_eq!(delta.text,"Display summary");
+ assert!(d.take_reasoning().is_none());
+ }
+}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub struct Usage {
pub input_tokens: u64,
@@ -175,6 +258,7 @@ pub struct Decoder {
completion: Option<Completion>,
reported: Option<Usage>,
output_started: bool,
+ reasoning: Option<ReasoningDelta>,
}
impl Decoder {
pub fn event(&mut self, event: &Value) -> Result<Option<String>, Error> {
@@ -201,6 +285,14 @@ impl Decoder {
return Err(Error::new(Failure::InvalidResponse));
}
match event["type"].as_str().unwrap_or("") {
+ "response.reasoning_summary_text.delta" => {
+ self.reasoning = Some(ReasoningDelta {
+ kind: ReasoningKind::Summary,
+ text: event["delta"].as_str().ok_or_else(|| Error::new(Failure::InvalidResponse))?.to_owned(),
+ block_index: event["output_index"].as_u64().ok_or_else(|| Error::new(Failure::InvalidResponse))?,
+ });
+ Ok(None)
+ }
"response.output_text.delta" => Ok(Some(
event["delta"]
.as_str()
@@ -287,8 +379,10 @@ pub(crate) trait StreamDecoder {
fn output_started(&self) -> bool {
false
}
+ fn take_reasoning(&mut self) -> Option<ReasoningDelta> { None }
}
impl StreamDecoder for Decoder {
+ fn take_reasoning(&mut self) -> Option<ReasoningDelta> { self.reasoning.take() }
fn usage_report(&self) -> Option<crate::telemetry::TokenUsage> {
self.reported.as_ref().map(Usage::reported)
}
@@ -323,6 +417,7 @@ pub(crate) async fn request_decoded<D: StreamDecoder>(
.map_err(|_| Error::new(Failure::Transport))?;
if !response.status().is_success() {
let status = response.status().as_u16();
+ let request_id = request_id(response.headers(), key);
let retry_after = response
.headers()
.get("retry-after")
@@ -331,7 +426,7 @@ pub(crate) async fn request_decoded<D: StreamDecoder>(
let mut stream = response.bytes_stream();
let mut bytes = vec![];
while let Some(chunk) = stream.next().await {
- let chunk = chunk.map_err(|_| Error::new(classify(status, "")))?;
+ let chunk = chunk.map_err(|_| Error { kind:classify(status,""),retry_after,http_status:Some(status),request_id:request_id.clone(),api:None,code:Some("http_error_body_unavailable") })?;
if bytes.len() + chunk.len() > 16384 {
break;
}
@@ -342,6 +437,9 @@ pub(crate) async fn request_decoded<D: StreamDecoder>(
kind: classify(status, error["error"]["code"].as_str().unwrap_or("")),
retry_after,
http_status: Some(status),
+ request_id,
+ api: None,
+ code: upstream_code(&error["error"]),
});
}
let mut stream = response.bytes_stream();
@@ -369,7 +467,7 @@ pub(crate) async fn request_decoded<D: StreamDecoder>(
continue;
}
let event: Value =
- serde_json::from_str(data).map_err(|_| Error::new(Failure::InvalidResponse))?;
+ serde_json::from_str(data).map_err(|_| Error::new(Failure::InvalidResponse).with_code("invalid_stream_json"))?;
let decoded = decoder.event(&event);
if let Some(usage) = decoder
.usage_report()
@@ -381,6 +479,9 @@ pub(crate) async fn request_decoded<D: StreamDecoder>(
previous_usage = Some(usage);
}
let decoded = decoded?;
+ if let Some(delta) = decoder.take_reasoning() {
+ tx.send(Progress::Reasoning(delta)).await.map_err(|_| Error::new(Failure::Transport))?;
+ }
if !first_sent && decoder.output_started() {
tx.send(Progress::OutputStarted)
.await
diff --git a/harness/runtime/src/runtime.rs b/harness/runtime/src/runtime.rs
index 54c0c18e..0bf80c13 100644
--- a/harness/runtime/src/runtime.rs
+++ b/harness/runtime/src/runtime.rs
@@ -595,6 +595,10 @@ impl Runtime {
.iter()
.any(|h| h.when == crate::safety::HookPhase::BeforeTool);
model.content_withheld = guarded_output;
+ let output_options = &s.turn.as_ref().context("No current turn")?.output;
+ model.stream_reasoning = output_options.reasoning && !guarded_output;
+ let buffer_attempts = output_options.buffer_attempts;
+ let show_reasoning = model.stream_reasoning;
let mut instructions = self.bundle.instructions_for(&s.skills, &s.loaded_skills);
if let Some(addendum) = s
.turn
@@ -777,12 +781,36 @@ impl Runtime {
let mut result = None;
let mut closed = false;
let mut reported_output = None;
+ let attempt_started = tokio::time::Instant::now();
+ let mut heartbeat = tokio::time::interval_at(
+ attempt_started + std::time::Duration::from_secs(5),
+ std::time::Duration::from_secs(5),
+ );
+ heartbeat.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
+ let mut reasoning = Vec::<crate::provider::ReasoningDelta>::new();
+ let mut reasoning_bytes = 0usize;
+ let mut reasoning_limited = false;
while result.is_none() || !closed {
tokio::select! {
r=&mut request,if result.is_none()=>{result=Some(r);},
+ _=heartbeat.tick(),if result.is_none()=>{
+ self.progress(sid,"model_activity",json!({"attempt_id":id,"phase":"waiting_for_model","elapsed_ms":attempt_started.elapsed().as_millis() as u64}))?;
+ },
p=rx.recv(),if !closed=>match p {
Some(Progress::Delta(text))=>{
- if !guarded_output {let e=self.store.lock().unwrap().attempt_delta(sid,&id,&text)?;self.publish(e);}
+ if !guarded_output && !buffer_attempts {let e=self.store.lock().unwrap().attempt_delta(sid,&id,&text)?;self.publish(e);}
+ },
+ Some(Progress::Reasoning(delta))=>{
+ if show_reasoning && !delta.text.is_empty() {
+ if reasoning_bytes.saturating_add(delta.text.len()) <= 65536 {
+ reasoning_bytes += delta.text.len();
+ if buffer_attempts { reasoning.push(delta); }
+ else { self.progress(sid,"reasoning_delta",json!({"attempt_id":id,"kind":delta.kind,"text":delta.text,"block_index":delta.block_index}))?; }
+ } else if !reasoning_limited {
+ reasoning_limited = true;
+ self.progress(sid,"reasoning_display_limited",json!({"attempt_id":id,"max_bytes":65536}))?;
+ }
+ }
},
Some(Progress::Usage(u))=>{reported_output=u.output;},
Some(Progress::OutputStarted)=>{},
@@ -829,6 +857,21 @@ impl Runtime {
match result.unwrap() {
Ok(()) => {
let response = completed.context("Missing completed provider output")?;
+ if buffer_attempts && !guarded_output {
+ for delta in reasoning {
+ self.progress(sid,"reasoning_delta",json!({"attempt_id":id,"kind":delta.kind,"text":delta.text,"block_index":delta.block_index}))?;
+ }
+ for item in &response.output {
+ if item["type"] == "message" {
+ for part in item["content"].as_array().into_iter().flatten() {
+ if let Some(text) = part["text"].as_str() {
+ let e = self.store.lock().unwrap().attempt_delta(sid,&id,text)?;
+ self.publish(e);
+ }
+ }
+ }
+ }
+ }
let event =
self.store
.lock()
@@ -839,6 +882,9 @@ impl Runtime {
}
Err(error) => {
let failure = error.downcast_ref::<crate::provider::Error>();
+ if let Some(failure) = failure {
+ self.progress(sid,"provider_error",json!({"attempt_id":id,"diagnostic":failure.diagnostic()}))?;
+ }
let event = self.store.lock().unwrap().finish_attempt(
sid,
&id,
@@ -1756,6 +1802,7 @@ impl Runtime {
}
}
if let Err(e) = result {
+ let provider_error = e.downcast_ref::<crate::provider::Error>().map(|e| e.diagnostic());
let status = if e.is::<safety::Unavailable>() {
"waiting_safety"
} else if e.is::<safety::Blocked>() {
@@ -1775,10 +1822,11 @@ impl Runtime {
} else {
"turn_completed"
},
- json!({"status":status,"error":e.to_string()}),
+ json!({"status":status,"error":e.to_string(),"provider_error":provider_error}),
|s| {
if let Some(t) = &mut s.turn {
t.status = status.into();
+ t.provider_error = provider_error.clone();
}
Ok(())
},
diff --git a/harness/runtime/src/runtime_m2_tests.rs b/harness/runtime/src/runtime_m2_tests.rs
index ca4128b9..a32d5c6f 100644
--- a/harness/runtime/src/runtime_m2_tests.rs
+++ b/harness/runtime/src/runtime_m2_tests.rs
@@ -2,6 +2,79 @@ use super::*;
use crate::provider::{Completion, Error, Failure};
use std::sync::atomic::AtomicUsize;
+#[tokio::test]
+async fn reasoning_is_opt_in_and_protected_output_never_streams_candidates() {
+ for (enabled, protected) in [(false,false),(true,false),(true,true)] {
+ let mut model = Model::new("fixture".into(), true);
+ model.sample_hook = Some(Arc::new(|_,_,tx|Box::pin(async move {
+ tx.send(Progress::Reasoning(crate::provider::ReasoningDelta { kind:crate::provider::ReasoningKind::Summary,text:"DISPLAY_SENTINEL".into(),block_index:0 })).await.unwrap();
+ tx.send(Progress::Delta("answer".into())).await.unwrap();
+ tx.send(Progress::Complete(Completion { output:vec![json!({"type":"message","role":"assistant","content":[{"type":"output_text","text":"answer"}]})],usage:None })).await.unwrap();
+ Ok(())
+ })));
+ let (_tmp,rt,sid) = fixture(model,true).await;
+ rt.update(&sid,"configure",json!({}),|s| {
+ s.turn.as_mut().unwrap().output.reasoning = enabled;
+ if protected { s.turn.as_mut().unwrap().safety.guardrails.model_output.push(crate::safety::Rule::regex("check","forbidden",crate::safety::Action::Block)); }
+ Ok(())
+ }).unwrap();
+ rt.sample(&sid,&[],&[]).await.unwrap();
+ let events = rt.store.lock().unwrap().events(&sid,0).unwrap();
+ assert_eq!(events.iter().any(|e|e.kind=="reasoning_delta"),enabled && !protected);
+ if !enabled || protected { assert!(!serde_json::to_string(&events).unwrap().contains("DISPLAY_SENTINEL")); }
+ if protected { assert!(!events.iter().any(|e|e.kind=="assistant_delta")); }
+ }
+}
+
+#[tokio::test]
+async fn buffered_attempts_discard_failed_text_and_reasoning() {
+ let count = Arc::new(AtomicUsize::new(0));
+ let seen = count.clone();
+ let mut model = Model::new("fixture".into(),true);
+ model.sample_hook = Some(Arc::new(move |_,_,tx| {
+ let failed = seen.fetch_add(1,Ordering::SeqCst)==0;
+ Box::pin(async move {
+ let text = if failed {"FAILED_CANDIDATE"} else {"accepted"};
+ tx.send(Progress::Delta(text.into())).await.unwrap();
+ tx.send(Progress::Reasoning(crate::provider::ReasoningDelta {kind:crate::provider::ReasoningKind::Summary,text:text.into(),block_index:0})).await.unwrap();
+ if failed { return Err(Error::new(Failure::IncompleteStream).into()); }
+ tx.send(Progress::Complete(Completion {output:vec![json!({"type":"message","role":"assistant","content":[{"type":"output_text","text":text}]})],usage:None})).await.unwrap();
+ Ok(())
+ })
+ }));
+ let (_tmp,rt,sid)=fixture(model,true).await;
+ rt.update(&sid,"configure",json!({}),|s| { let o=&mut s.turn.as_mut().unwrap().output;o.reasoning=true;o.buffer_attempts=true;Ok(()) }).unwrap();
+ rt.sample(&sid,&[],&[]).await.unwrap();
+ let events=rt.store.lock().unwrap().events(&sid,0).unwrap();
+ assert!(!serde_json::to_string(&events).unwrap().contains("FAILED_CANDIDATE"));
+ assert_eq!(events.iter().filter(|e|e.kind=="assistant_delta").count(),1);
+ assert_eq!(events.iter().filter(|e|e.kind=="reasoning_delta").count(),1);
+ let finished:Vec<_>=events.iter().filter(|e|e.kind=="model_attempt_finished").collect();
+ assert_eq!(finished.len(),2);
+ assert_eq!(finished[0].data["status"],"failed");
+ assert_eq!(finished[1].data["status"],"completed");
+ assert_ne!(finished[0].data["attempt_id"],finished[1].data["attempt_id"]);
+}
+
+#[tokio::test]
+async fn long_model_call_emits_bounded_truthful_activity() {
+ let mut model=Model::new("fixture".into(),true);
+ model.sample_hook=Some(Arc::new(|_,_,tx|Box::pin(async move {
+ tokio::time::sleep(std::time::Duration::from_millis(5100)).await;
+ tx.send(Progress::Complete(Completion {output:vec![json!({"type":"message","role":"assistant","content":[{"type":"output_text","text":"ready"}]})],usage:None})).await.unwrap();
+ Ok(())
+ })));
+ let (_tmp,rt,sid)=fixture(model,true).await;
+ tokio::time::timeout(std::time::Duration::from_secs(10),rt.sample(&sid,&[],&[])).await.unwrap().unwrap();
+ let events=rt.store.lock().unwrap().events(&sid,0).unwrap();
+ let ticks:Vec<_>=events.iter().filter(|e|e.kind=="model_activity").collect();
+ assert_eq!(ticks.len(),1);
+ assert_eq!(ticks[0].data["phase"],"waiting_for_model");
+ assert!(ticks[0].data["elapsed_ms"].as_u64().unwrap()>=5000);
+ let end=events.iter().find(|e|e.kind=="model_attempt_finished").unwrap();
+ assert!(ticks[0].event_id<end.event_id);
+}
+
#[tokio::test]
async fn live_tool_access_reaches_model_and_refreshes_on_retry_without_entering_history() {
use openworker_tools::{Tool, ToolContext, ToolDescriptor, ToolFuture};
@@ -427,6 +500,9 @@ async fn retries_keep_partial_attempts_separate_and_do_not_touch_tools_or_histor
kind: Failure::RateLimit,
retry_after: Some(0),
http_status: None,
+ request_id: None,
+ api: None,
+ code: None,
}
.into());
}
@@ -545,6 +621,9 @@ async fn repeated_transient_failure_stops_after_three_attempts() {
kind: Failure::Transient,
retry_after: Some(0),
http_status: None,
+ request_id: None,
+ api: None,
+ code: None,
}
.into())
})
@@ -577,6 +656,9 @@ async fn retry_backoff_is_cancellable_without_another_request() {
kind: Failure::RateLimit,
retry_after: Some(30),
http_status: None,
+ request_id: None,
+ api: None,
+ code: None,
}
.into())
})
@@ -637,6 +719,9 @@ async fn compaction_retry_and_pending_approval_survive_the_same_restart_scenario
kind: Failure::Transient,
retry_after: Some(0),
http_status: None,
+ request_id: None,
+ api: None,
+ code: None,
}
.into());
}
diff --git a/harness/runtime/tests/anthropic.rs b/harness/runtime/tests/anthropic.rs
index fdadf6a0..466f7b94 100644
--- a/harness/runtime/tests/anthropic.rs
+++ b/harness/runtime/tests/anthropic.rs
@@ -142,6 +142,28 @@ fn private_thinking_is_retained_but_not_streamed_and_foreign_history_is_rejected
history[1] = json!({"type":"reasoning","encrypted_content":"OPENAI_ONLY"});
assert!(anthropic::body("test", "system", &history, &[], 1024, None).is_err());
}
+
+#[test]
+fn display_thinking_is_separate_from_answer_and_never_includes_signatures() {
+ let mut d=Decoder::default();d.event(&start()).unwrap();
+ d.event(&json!({"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}})).unwrap();
+ assert!(d.event(&json!({"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"Display text"}})).unwrap().is_none());
+ let text=d.take_reasoning().unwrap();assert_eq!(text.text,"Display text");
+ d.event(&json!({"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"SECRET_SIGNATURE"}})).unwrap();
+ assert!(d.take_reasoning().is_none());
+ d.event(&json!({"type":"content_block_stop","index":0})).unwrap();
+ d.event(&json!({"type":"content_block_start","index":1,"content_block":{"type":"redacted_thinking","data":"SECRET_REDACTED"}})).unwrap();
+ assert!(d.take_reasoning().is_none());
+}
+
+#[test]
+fn thinking_generation_configuration_is_explicit_and_budgeted() {
+ use openworker_runtime::anthropic::Thinking;
+ assert_eq!(Thinking::Adaptive.request(4096).unwrap(),json!({"type":"adaptive"}));
+ assert_eq!(Thinking::Enabled{budget_tokens:1024}.request(4096).unwrap(),json!({"type":"enabled","budget_tokens":1024}));
+ assert!(Thinking::Enabled{budget_tokens:1024}.request(1024).is_err());
+ assert!(Thinking::Enabled{budget_tokens:5}.validate().is_err());
+}
#[test]
fn input_pairing_images_denials_and_structured_reviewer_format() {
let history = vec![
diff --git a/harness/runtime/tests/connections.rs b/harness/runtime/tests/connections.rs
index 482a2e40..a6da131a 100644
--- a/harness/runtime/tests/connections.rs
+++ b/harness/runtime/tests/connections.rs
@@ -290,6 +290,28 @@ fn chat_stream_usage_defaults_on_and_opt_out_survives_rotation_and_restart() {
assert!(m.stream_usage(&selected).is_err());
}
+#[test]
+fn anthropic_thinking_is_explicit_validated_and_persisted() {
+ use openworker_runtime::anthropic::Thinking;
+ let root=tempfile::tempdir().unwrap();
+ let m=Manager::open(root.path()).unwrap();
+ let config=json!({"name":"Claude","api":"anthropic","base_url":"https://api.anthropic.com/v1"});
+ m.set_provider(config.clone()).unwrap();
+ let selected=m.resolve(&Choice::new("Claude","fixture")).unwrap();
+ assert!(m.thinking(&selected).unwrap().is_none());
+ let mut enabled=config.clone();enabled["thinking"]=json!({"type":"enabled","budgetTokens":2048});
+ m.set_provider(enabled).unwrap();m.set_provider(config.clone()).unwrap();
+ assert_eq!(m.thinking(&selected).unwrap(),Some(Thinking::Enabled{budget_tokens:2048}));
+ let mut invalid=config.clone();invalid["thinking"]=json!({"type":"enabled","budgetTokens":2});
+ assert!(m.set_provider(invalid).is_err());
+ drop(m);let m=Manager::open(root.path()).unwrap();
+ assert_eq!(m.thinking(&selected).unwrap(),Some(Thinking::Enabled{budget_tokens:2048}));
+ assert!(m.set_provider(json!({"name":"Wrong","api":"responses","thinking":{"type":"adaptive"}})).is_err());
+ let mut disabled=config;disabled["thinking"]=json!({"type":"disabled"});m.set_provider(disabled).unwrap();
+ assert_eq!(m.thinking(&selected).unwrap(),Some(Thinking::Disabled));
+ m.remove("Claude").unwrap();assert!(m.thinking(&selected).is_err());
+}
+
#[test]
fn chat_stream_usage_migrates_existing_endpoint_rows() {
let root = tempfile::tempdir().unwrap();
diff --git a/harness/runtime/tests/provider.rs b/harness/runtime/tests/provider.rs
index 4584a72f..0ff44726 100644
--- a/harness/runtime/tests/provider.rs
+++ b/harness/runtime/tests/provider.rs
@@ -128,3 +128,30 @@ async fn real_http_parser_handles_rate_limit_and_truncated_stream() {
assert!(matches!(rx.recv().await,Some(Progress::Delta(t)) if t=="partial"));
assert!(rx.recv().await.is_none());
}
+
+#[tokio::test]
+async fn http_diagnostics_retain_status_and_request_id_without_upstream_body() {
+ let body=r#"{"error":{"code":"invalid_api_key","message":"secret=BODY_SENTINEL"}}"#;
+ let (url,server)=http_once(&format!("HTTP/1.1 401 Unauthorized\r\nX-Request-ID: req_fixture\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",body.len(),body)).await;
+ let (tx,_)=mpsc::channel(10);
+ let e=provider::request(&reqwest::Client::new(),&url,"KEY_SENTINEL",json!({}),tx).await.unwrap_err();
+ server.await.unwrap();
+ assert_eq!(e.http_status,Some(401));
+ assert_eq!(e.request_id.as_deref(),Some("req_fixture"));
+ assert!(!e.retryable());
+ for text in [e.to_string(),serde_json::to_string(&e.diagnostic()).unwrap()] {
+ assert!(!text.contains("BODY_SENTINEL"));assert!(!text.contains("KEY_SENTINEL"));
+ }
+}
+
+#[tokio::test]
+async fn broken_http_error_body_keeps_known_status_and_request_id() {
+ let (url,server)=http_once("HTTP/1.1 401 Unauthorized\r\nX-Request-ID: req_broken\r\nContent-Length: 999\r\nConnection: close\r\n\r\npartial").await;
+ let (tx,_)=mpsc::channel(10);
+ let error=provider::request(&reqwest::Client::new(),&url,"test-only",json!({}),tx).await.unwrap_err();
+ server.await.unwrap();
+ assert_eq!(error.http_status,Some(401));
+ assert_eq!(error.request_id.as_deref(),Some("req_broken"));
+ assert_eq!(error.code,Some("http_error_body_unavailable"));
+ assert_eq!(error.kind,Failure::Authentication);
+}
diff --git a/harness/typescript-sdk/README.md b/harness/typescript-sdk/README.md
index 479d13bc..da7450d7 100644
--- a/harness/typescript-sdk/README.md
+++ b/harness/typescript-sdk/README.md
@@ -159,6 +159,10 @@ attempts separately; do not blindly concatenate a failed attempt into its retry.
The game-builder starter's existing UI displays progress and completed messages;
incremental-text rendering is a separate UI improvement.
+For typed attempt events, opt-in reasoning display, optional attempt buffering and
+safe provider diagnostics, see [Streaming and retries](../../docs/harness/guides/streaming.md).
+`AttemptOutput` provides a bounded UI projection that removes failed candidates.
+
Application tools receive `context.callId`. Tool-start and tool-finish summaries
expose that same ID as `item.summary.callId`; the raw journal uses
`item.event.data.call_id`. Use it to correlate calls, including repeated calls to
diff --git a/harness/typescript-sdk/package.json b/harness/typescript-sdk/package.json
index d5e870da..b492dbaf 100644
--- a/harness/typescript-sdk/package.json
+++ b/harness/typescript-sdk/package.json
@@ -24,7 +24,8 @@
"test:team-placement": "npm run build && node --test test/team-placement.test.mjs",
"test:coding-presets:integration": "npm run build && node --test --test-concurrency=1 test/coding-presets-live.test.mjs",
"build": "tsc -p tsconfig.json",
- "test": "npm run build && node --test test/config.test.mjs test/transport.test.mjs",
+ "test": "npm run build && node --test test/config.test.mjs test/transport.test.mjs test/attempt-output.test.mjs",
+ "test:stream-events": "npm run build && node --test --test-concurrency=1 test/stream-events.test.mjs",
"test:integration": "npm run build && node --test --test-concurrency=1 test/integration.test.mjs",
"test:dynamic-agents": "npm run build && node --test --test-concurrency=1 test/dynamic-agents.test.mjs",
"test:tool-free": "npm run build && node --test --test-concurrency=1 test/tool-free.test.mjs",
diff --git a/harness/typescript-sdk/src/attempt-output.ts b/harness/typescript-sdk/src/attempt-output.ts
new file mode 100644
index 00000000..0b7cacd6
--- /dev/null
+++ b/harness/typescript-sdk/src/attempt-output.ts
@@ -0,0 +1,55 @@
+import type { RunEvent } from "./types.js";
+
+export interface AttemptView {
+ sessionId: string;
+ turnId: string | null;
+ attemptId: string;
+ status: "running" | "completed" | "failed" | "interrupted";
+ text: string;
+ reasoning: string;
+}
+
+/** UI projection for one run. Failed candidates disappear when their attempt
+ * fails; replayed events are ignored. This never changes the durable transcript.
+ * Render text with textContent/escaped text, never as HTML. */
+export class AttemptOutput {
+ private readonly rows = new Map<string, AttemptView>();
+ private readonly cursors = new Map<string, number>();
+ constructor(private readonly maxAttempts = 128, private readonly maxChars = 262144) {
+ if (!Number.isSafeInteger(maxAttempts) || maxAttempts < 1 || maxAttempts > 1024 ||
+ !Number.isSafeInteger(maxChars) || maxChars < 1 || maxChars > 1048576)
+ throw new Error("Invalid attempt display limits");
+ }
+ apply(item: RunEvent): void {
+ const { event, sessionId } = item;
+ if (event.event_id <= (this.cursors.get(sessionId) ?? -1)) return;
+ this.cursors.set(sessionId, event.event_id);
+ const summary = item.summary;
+ if (!summary || !("attemptId" in summary)) return;
+ const key = JSON.stringify([sessionId, event.turn_id, summary.attemptId]);
+ let row = this.rows.get(key);
+ if (!row) {
+ row = { sessionId, turnId:event.turn_id, attemptId:summary.attemptId,
+ status:"running", text:"", reasoning:"" };
+ this.rows.set(key, row);
+ while (this.rows.size > this.maxAttempts) this.rows.delete(this.rows.keys().next().value!);
+ }
+ if (summary.type === "model_attempt_finished") {
+ row.status = summary.status;
+ if (row.status !== "completed") { row.text = ""; row.reasoning = ""; }
+ } else if (row.status === "running") {
+ if (summary.type === "assistant_delta") row.text = (row.text + summary.text).slice(0, this.maxChars);
+ if (summary.type === "reasoning_delta") row.reasoning = (row.reasoning + summary.text).slice(0, this.maxChars);
+ }
+ // A cancelled/interrupted turn may end before an attempt-finished event.
+ // Call interrupt() when the run terminates without completing.
+ }
+ interrupt(): void {
+ for (const row of this.rows.values()) if (row.status === "running") {
+ row.status = "interrupted"; row.text = ""; row.reasoning = "";
+ }
+ }
+ get attempts(): readonly AttemptView[] { return [...this.rows.values()].map((row) => ({...row})); }
+ get text(): string { return [...this.rows.values()].filter((row) => row.status === "running" || row.status === "completed").map((row) => row.text).join(""); }
+ get reasoning(): string { return [...this.rows.values()].filter((row) => row.status === "running" || row.status === "completed").map((row) => row.reasoning).join(""); }
+}
diff --git a/harness/typescript-sdk/src/index.ts b/harness/typescript-sdk/src/index.ts
index ac261868..f8fe7e6c 100644
--- a/harness/typescript-sdk/src/index.ts
+++ b/harness/typescript-sdk/src/index.ts
@@ -3,6 +3,7 @@ export * from "./safety.js";
import { Service, type ServiceOptions } from "./service.js";
export * from "./service.js";
export * from "./coding.js";
+export * from "./attempt-output.js";
import { randomUUID } from "node:crypto";
import { isAbsolute } from "node:path";
import { Transport } from "./transport.js";
@@ -12,6 +13,7 @@ import {
ToolImageResult,
checkMemberErrors,
record,
+ providerDiagnostic,
HarnessError,
type HarnessOptions,
type SessionRef,
@@ -70,6 +72,7 @@ function resultOf(value: unknown): RunResult {
interruptions: v.interruptions.map(interruptionOf),
submissionId: v.submission_id as string | null,
failure: typeof v.failure === "string" ? v.failure : undefined,
+ providerError: providerDiagnostic(v.provider_error),
cursor: Number(v.cursor),
board: (v.board ?? null) as Json,
};
@@ -393,6 +396,7 @@ export class Harness {
base_url: provider.baseUrl,
stream_usage: provider.streamUsage,
prompt_caching: provider.promptCaching,
+ thinking: provider.thinking,
access_token_auth: provider.auth ? true : undefined,
credentials: credentials
? { api_key: apiKey, storage: credentials.storage ?? "secure" }
@@ -1061,6 +1065,22 @@ export class RunStream implements AsyncIterable<RunEvent> {
const callId = wireSummary.call_id ?? record(event.data).call_id;
summary = { ...summary, callId: typeof callId === "string" ? callId : "" };
}
+ if (summary && ["model_attempt_started", "model_attempt_finished", "assistant_delta", "reasoning_delta", "model_activity", "provider_error"].includes(summary.type)) {
+ const wire = record(item.summary);
+ const attemptId = String(wire.attempt_id ?? "");
+ switch (summary.type) {
+ case "model_attempt_started": summary = { type:summary.type, attemptId, attempt:Number(wire.attempt) }; break;
+ case "model_attempt_finished": summary = { type:summary.type, attemptId, status:wire.status as "completed" | "failed" | "interrupted", failure:typeof wire.kind === "string" ? wire.kind as import("./types.js").ProviderFailure : undefined }; break;
+ case "assistant_delta": summary = { type:summary.type, attemptId, text:String(wire.text ?? "") }; break;
+ case "reasoning_delta": summary = { type:summary.type, attemptId, kind:wire.kind as "summary" | "thinking", text:String(wire.text ?? ""), blockIndex:Number(wire.block_index) }; break;
+ case "model_activity": summary = { type:summary.type, attemptId, elapsedMs:Number(wire.elapsed_ms) }; break;
+ case "provider_error": {
+ const diagnostic = providerDiagnostic(wire.diagnostic);
+ summary = diagnostic ? { type:summary.type, attemptId, diagnostic } : null;
+ break;
+ }
+ }
+ }
yield {
agent: String(item.agent),
...(typeof item.agent_id === "string"
diff --git a/harness/typescript-sdk/src/transport.ts b/harness/typescript-sdk/src/transport.ts
index 0e6ab92e..294ef5c8 100644
--- a/harness/typescript-sdk/src/transport.ts
+++ b/harness/typescript-sdk/src/transport.ts
@@ -10,6 +10,7 @@ import {
HarnessOptions,
ResolvedHarnessOptions,
record,
+ providerDiagnostic,
type ConnectionState,
} from "./types.js";
@@ -322,10 +323,12 @@ export class Transport {
clearTimeout(p.timer);
if (frame.error) {
// Do not reflect untrusted diagnostics, tool arguments or credentials in errors.
+ const diagnostic = providerDiagnostic(frame.error.provider_error);
p.reject(
new HarnessError(
"Remote",
- "Harness rejected the request; inspect its configuration or durable session state",
+ diagnostic ? `Provider ${diagnostic.category}${diagnostic.httpStatus ? ` (HTTP ${diagnostic.httpStatus})` : ""}` : "Harness rejected the request; inspect its configuration or durable session state",
+ diagnostic ? { providerError: diagnostic } : undefined,
),
);
} else p.resolve(frame.result);
diff --git a/harness/typescript-sdk/src/types.ts b/harness/typescript-sdk/src/types.ts
index 9170db7a..95833aa1 100644
--- a/harness/typescript-sdk/src/types.ts
+++ b/harness/typescript-sdk/src/types.ts
@@ -49,6 +49,9 @@ export interface ModelProvider {
streamUsage?: boolean;
/** Anthropic automatic prompt caching, enabled by default. False omits cache_control. */
promptCaching?: boolean;
+ /** Anthropic generation mode. Choose a mode supported by the configured model.
+ * Display requires per-turn output.reasoning; omission preserves existing settings. */
+ thinking?: { type: "disabled" } | { type: "adaptive" } | { type: "enabled"; budgetTokens: number };
/** Memory-only bearer access tokens; mutually exclusive with credentials. */
auth?: ProviderAuth;
credentials?: (
@@ -242,6 +245,7 @@ export interface RunResult {
interruptions: Interruption[];
submissionId: string | null;
failure?: string;
+ providerError?: ProviderDiagnostic;
cursor: number;
board: Json | null;
}
@@ -309,6 +313,12 @@ export interface RunEvent {
summary:
| (
| { type: "message"; text: string }
+ | { type: "model_attempt_started"; attemptId: string; attempt: number }
+ | { type: "model_attempt_finished"; attemptId: string; status: "completed" | "failed" | "interrupted"; failure?: ProviderFailure }
+ | { type: "assistant_delta"; attemptId: string; text: string }
+ | { type: "reasoning_delta"; attemptId: string; kind: "summary" | "thinking"; text: string; blockIndex: number }
+ | { type: "model_activity"; attemptId: string; elapsedMs: number }
+ | { type: "provider_error"; attemptId: string; diagnostic: ProviderDiagnostic }
| {
type: "tool_started" | "tool_finished";
tool: string;
@@ -377,6 +387,7 @@ export class HarnessError extends Error {
sessionId?: string;
submissionId?: string;
requestId?: string;
+ providerError?: ProviderDiagnostic;
},
) {
super(message);
@@ -384,6 +395,31 @@ export class HarnessError extends Error {
}
}
+export type ProviderFailure = "rate_limit" | "transient" | "transport" | "incomplete_stream" | "invalid_response" | "context_overflow" | "truncated" | "authentication" | "configuration" | "safety_blocked";
+export interface ProviderDiagnostic {
+ category: ProviderFailure;
+ retryable: boolean;
+ httpStatus?: number;
+ requestId?: string;
+ api?: "responses" | "chat-completions" | "anthropic";
+ /** Fixed parser code or allowlisted upstream code, not upstream message text. */
+ code?: string;
+}
+/** Parse only the structured safe fields; never reflect arbitrary remote text. */
+export function providerDiagnostic(value: unknown): ProviderDiagnostic | undefined {
+ if (!value || typeof value !== "object" || Array.isArray(value)) return undefined;
+ const v = value as Record<string, unknown>;
+ const categories = ["rate_limit", "transient", "transport", "incomplete_stream", "invalid_response", "context_overflow", "truncated", "authentication", "configuration", "safety_blocked"];
+ if (!categories.includes(String(v.category)) || typeof v.retryable !== "boolean") return undefined;
+ return {
+ category: v.category as ProviderFailure, retryable: v.retryable,
+ ...(Number.isInteger(v.http_status) && Number(v.http_status) >= 100 && Number(v.http_status) <= 599 ? { httpStatus: Number(v.http_status) } : {}),
+ ...(typeof v.request_id === "string" && /^[a-zA-Z0-9_:-]{1,128}$/.test(v.request_id) && !v.request_id.startsWith("sk-") ? { requestId: v.request_id } : {}),
+ ...(["responses", "chat-completions", "anthropic"].includes(String(v.api)) ? { api: v.api as ProviderDiagnostic["api"] } : {}),
+ ...(["invalid_api_key","authentication_error","permission_error","invalid_request_error","rate_limit_exceeded","rate_limit_error","insufficient_quota","context_length_exceeded","server_error","api_error","overloaded_error","misalignment_policy_violation","RESOURCE_EXHAUSTED","INVALID_ARGUMENT","UNAUTHENTICATED","PERMISSION_DENIED","http_error_body_unavailable","invalid_response","incomplete_stream","invalid_stream_json","invalid_anthropic_message","invalid_anthropic_request","invalid_chat_completion","invalid_chat_request_history","invalid_tool_call_index"].includes(String(v.code)) ? {code:String(v.code)} : {}),
+ };
+}
+
export function record(v: unknown): Record<string, unknown> {
if (!v || typeof v !== "object" || Array.isArray(v))
throw new HarnessError("Protocol", "Expected a protocol object");
diff --git a/harness/typescript-sdk/test/attempt-output.test.mjs b/harness/typescript-sdk/test/attempt-output.test.mjs
new file mode 100644
index 00000000..bddd5a48
--- /dev/null
+++ b/harness/typescript-sdk/test/attempt-output.test.mjs
@@ -0,0 +1,35 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+import { AttemptOutput, providerDiagnostic } from "../dist/index.js";
+function item(id, type, data = {}, session = "s") {
+ return { sessionId:session, agent:"guide", event:{event_id:id,session_id:session,turn_id:"t",type,data:{}}, summary:{type,attemptId:"a",...data} };
+}
+test("attempt projection replaces failed candidates and deduplicates replay",()=>{
+ const view=new AttemptOutput();
+ view.apply(item(1,"model_attempt_started",{attempt:1}));
+ view.apply(item(2,"assistant_delta",{text:"partial"}));
+ view.apply(item(3,"reasoning_delta",{text:"candidate",kind:"summary",blockIndex:0}));
+ assert.equal(view.text,"partial");
+ view.apply(item(4,"model_attempt_finished",{status:"failed"}));
+ assert.equal(view.text,"");assert.equal(view.reasoning,"");
+ view.apply(item(5,"model_attempt_started",{attemptId:"b",attempt:2}));
+ const delta=item(6,"assistant_delta",{attemptId:"b",text:"accepted"});
+ view.apply(delta);view.apply(delta);
+ view.apply(item(7,"model_attempt_finished",{attemptId:"b",status:"completed"}));
+ assert.equal(view.text,"accepted");
+ view.interrupt();assert.equal(view.text,"accepted");
+});
+test("attempt projection scopes sessions and drops cancelled provisional output",()=>{
+ const view=new AttemptOutput(2,8);
+ view.apply(item(1,"assistant_delta",{text:"abcdefghijk"},"one"));
+ view.apply(item(1,"assistant_delta",{text:"second"},"two"));
+ assert.equal(view.text,"abcdefghsecond");
+ view.interrupt();assert.equal(view.text,"");
+ assert.throws(()=>new AttemptOutput(0));
+});
+test("diagnostics expose only safe structured values",()=>{
+ assert.deepEqual(providerDiagnostic({category:"authentication",retryable:false,http_status:401,request_id:"req_fixture",api:"anthropic",message:"SECRET",body:"SECRET"}),
+ {category:"authentication",retryable:false,httpStatus:401,requestId:"req_fixture",api:"anthropic"});
+ assert.equal(providerDiagnostic({category:"arbitrary secret",retryable:true}),undefined);
+ assert.equal(providerDiagnostic({category:"transport",retryable:true,request_id:"sk-secret"}).requestId,undefined);
+});
diff --git a/harness/typescript-sdk/test/stream-events.test.mjs b/harness/typescript-sdk/test/stream-events.test.mjs
new file mode 100644
index 00000000..e2a85a81
--- /dev/null
+++ b/harness/typescript-sdk/test/stream-events.test.mjs
@@ -0,0 +1,94 @@
+// Actual managed Rust host, synthetic loopback streams, no provider credentials.
+import test from "node:test";
+import assert from "node:assert/strict";
+import { createServer } from "node:http";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { Harness, AttemptOutput, guard } from "../dist/index.js";
+
+const options={skip:!process.env.OW_TS_TEST_HOST,timeout:90000};
+const frame=value=>`data: ${JSON.stringify(value)}\n\n`;
+function stream(api, text, reasoning, complete=true) {
+ if(api==="responses") return frame({type:"response.reasoning_summary_text.delta",delta:reasoning,output_index:0,summary_index:0})+
+ frame({type:"response.output_text.delta",delta:text})+
+ (complete?frame({type:"response.completed",response:{status:"completed",output:[{type:"reasoning",encrypted_content:"PRIVATE_SIGNATURE"},{type:"message",role:"assistant",content:[{type:"output_text",text}]}],usage:{input_tokens:10,output_tokens:8}}}):"");
+ if(api==="chat-completions") {
+ const chunk=(delta,finish_reason=null)=>frame({id:"fixture",choices:[{index:0,delta,finish_reason}]});
+ return chunk({reasoning_content:reasoning})+chunk({content:text})+(complete?chunk({},"stop")+"data: [DONE]\n\n":"");
+ }
+ const events=[
+ {type:"message_start",message:{id:"fixture",role:"assistant",content:[],usage:{input_tokens:10,output_tokens:0}}},
+ {type:"content_block_start",index:0,content_block:{type:"thinking",thinking:""}},
+ {type:"content_block_delta",index:0,delta:{type:"thinking_delta",thinking:reasoning}},
+ {type:"content_block_delta",index:0,delta:{type:"signature_delta",signature:"PRIVATE_SIGNATURE"}},
+ {type:"content_block_stop",index:0},
+ {type:"content_block_start",index:1,content_block:{type:"text",text:""}},
+ {type:"content_block_delta",index:1,delta:{type:"text_delta",text}},
+ ];
+ if(complete)events.push({type:"content_block_stop",index:1},{type:"message_delta",delta:{stop_reason:"end_turn"},usage:{output_tokens:8}},{type:"message_stop"});
+ return events.map(frame).join("");
+}
+async function setup(t,api,mode="normal") {
+ const root=await mkdtemp(join(tmpdir(),"ow-stream-events-"));
+ const requests=[];let serverError;
+ const server=createServer(async(req,res)=>{
+ try{
+ let body="";for await(const part of req){body+=part;assert.ok(body.length<1_000_000);}
+ const input=JSON.parse(body);requests.push(input);
+ assert.deepEqual(input.tools??[],[]);
+ if(mode==="error") {res.writeHead(401,{"x-request-id":"req_fixture"});res.end(JSON.stringify({error:{code:"invalid_api_key",type:"authentication_error",message:"SECRET_UPSTREAM_BODY"}}));return;}
+ const failed=mode==="retry"&&requests.length===1;
+ res.writeHead(200,{"content-type":"text/event-stream"});
+ res.end(stream(api,failed?"FAILED_TEXT":"Accepted answer",failed?"FAILED_REASON":"Display reasoning",!failed));
+ }catch(error){serverError=error;res.destroy();}
+ });
+ await new Promise(resolve=>server.listen(0,"127.0.0.1",resolve));
+ let h;t.after(async()=>{await h?.shutdown();server.closeAllConnections();await new Promise(resolve=>server.close(resolve));await rm(root,{recursive:true,force:true});assert.equal(serverError,undefined);});
+ h=await Harness.start({executable:process.env.OW_TS_TEST_HOST,stateDir:root,sandbox:"off"});
+ await h.setModelProvider({name:"Fixture",api,baseUrl:`http://127.0.0.1:${server.address().port}/v1`,credentials:{apiKey:"SYNTHETIC_KEY_SENTINEL",storage:"memory"},...(api==="anthropic"?{thinking:{type:"adaptive"}}:{})});
+ const agent=h.createAgent({name:"guide",instructions:"Give a hint.",tools:[],memory:false,planning:false,scheduling:false,defaultModel:{provider:"Fixture",name:"fixture"}});
+ return {h,agent,requests};
+}
+for(const api of ["responses","chat-completions","anthropic"]){
+ test(`reasoning is separate, opt-in and suppressed by output guardrails (${api})`,options,async t=>{
+ const {agent,requests}=await setup(t,api);
+ for(const [enabled,protectedOutput] of [[false,false],[true,false],[true,true]]){
+ const session=await agent.newSession();
+ const run=await session.send("Explain",{output:{reasoning:enabled},...(protectedOutput?{guardrails:{modelOutput:[guard.regex({name:"protect",pattern:"forbidden"})]}}:{})});
+ const events=[];for await(const item of run.events())events.push(item);
+ assert.equal((await run.result()).status,"completed");
+ const reasoning=events.filter(e=>e.summary?.type==="reasoning_delta");
+ assert.equal(reasoning.length,enabled&&!protectedOutput?1:0);
+ if(reasoning.length){assert.equal(reasoning[0].summary.text,"Display reasoning");assert.ok(reasoning[0].summary.attemptId);}
+ assert.ok(!JSON.stringify(events).includes("PRIVATE_SIGNATURE"));
+ if(!enabled||protectedOutput)assert.ok(!JSON.stringify(events).includes("Display reasoning"));
+ const body=requests.at(-1);
+ if(api==="responses")assert.deepEqual(body.reasoning,enabled&&!protectedOutput?{summary:"auto"}:undefined);
+ if(api==="anthropic")assert.deepEqual(body.thinking,{type:"adaptive"});
+ }
+ });
+ for(const buffered of [false,true])test(`failed attempts are replaceable and optionally buffered (${api}, ${buffered})`,options,async t=>{
+ const {agent,requests}=await setup(t,api,"retry");
+ const run=await (await agent.newSession()).send("Explain",{output:{reasoning:true,bufferAttempts:buffered}});
+ const projection=new AttemptOutput();const events=[];
+ for await(const item of run.events()){projection.apply(item);events.push(item);}
+ assert.equal((await run.result()).status,"completed");assert.equal(requests.length,2);
+ assert.equal(projection.text,"Accepted answer");assert.equal(projection.reasoning,"Display reasoning");
+ assert.equal(events.filter(e=>e.summary?.type==="model_attempt_started").length,2);
+ assert.deepEqual(events.filter(e=>e.summary?.type==="model_attempt_finished").map(e=>e.summary.status),["failed","completed"]);
+ assert.equal(JSON.stringify(events).includes("FAILED_TEXT"),!buffered);
+ assert.equal(JSON.stringify(events).includes("FAILED_REASON"),!buffered);
+ });
+ test(`safe provider diagnostics survive result and probe errors (${api})`,options,async t=>{
+ const {h,agent,requests}=await setup(t,api,"error");
+ const run=await (await agent.newSession()).send("Explain");
+ const events=[];for await(const item of run.events())events.push(item);
+ const result=await run.result();assert.equal(result.status,"failed");
+ assert.deepEqual(result.providerError,{category:"authentication",retryable:false,httpStatus:401,requestId:"req_fixture",api,code:"invalid_api_key"});
+ const error=events.find(e=>e.summary?.type==="provider_error");assert.deepEqual(error.summary.diagnostic,result.providerError);
+ await assert.rejects(h.testModelProvider({provider:"Fixture",name:"fixture"}),e=>e.code==="Remote"&&e.context?.providerError?.httpStatus===401);
+ assert.equal(requests.length,2);
+ assert.ok(!JSON.stringify([events,result]).includes("SECRET_UPSTREAM_BODY"));assert.ok(!JSON.stringify([events,result]).includes("SYNTHETIC_KEY_SENTINEL"));
+ });
+}