World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
Keep a tool call written as text out of the chat answer in every turn, and send the turn back once (#946)
Maya Kidf2e0d5bd67d9
diff --git a/crates/utopia-server/src/api/agent.rs b/crates/utopia-server/src/api/agent.rsindex a0e460ae..739ca676 100644--- a/crates/utopia-server/src/api/agent.rs+++ b/crates/utopia-server/src/api/agent.rs@@ -59,78 +59,156 @@ pub(crate) const EMPTY_REPLY_RETRY: &str = "(system) Your previous reply was empthe user now: answer from the evidence gathered above, or call a tool if you still need \evidence.";+/// 模型把工具调用写进了正文时退回去的那句(#937)。+///+/// 端点偶尔不走协议里的 `tool_calls`,而是把调用的标记原样写成正文:DeepSeek 的+/// `<|tool▁call▁begin|>function…`(台子上对话那一路 35 题里错了 2 题),或者 DSML。+/// 什么都没执行,那段也不是答案。退回时保留那一回合,模型看得见自己想调什么;+/// 只给一次,再写一次就以 [`CONTROL_TEXT`] 报错+pub(crate) const MARKUP_RETRY: &str = "(system) Your previous reply wrote a tool call out as \+ text, so nothing ran. Make the call as a tool call, or answer the user in plain text.";+/// Bounded buffering applies only to the tool-free terminal call. This is a byte/// limit, independent of the provider's token accounting.pub(crate) const MAX_FINAL_ANSWER_BYTES: usize = 1024 * 1024;const FINAL_TOOL_CALL: &str = "Model attempted a tool call after the tool budget was exhausted";+pub(crate) const CONTROL_TEXT: &str = "Model returned tool-control text instead of a final answer";-/// Deliberately scoped to budget finalization and bare control output. Explanations-/// and fenced examples are prose, and an explicit request about DSML may legitimately-/// ask for the raw encoding. Never interpret this text as an executable tool call.-pub(crate) fn finalization_error(- text: &str,- has_calls: bool,- question: &str,-) -> Option<&'static str> {- if has_calls {- return Some(FINAL_TOOL_CALL);- }- let text = text.trim();- if text.is_empty() {- return Some("Model returned an empty answer");- }+/// Line starts that open a tool call written as text, as [`control_key`] spells+/// them: DSML, and DeepSeek's native markup (#937).+const CONTROL_MARKERS: [&str; 5] = [+ "<DSMLcalls>",+ "<DSMLtool_calls>",+ "<DSMLinvokename=",+ "<tool▁calls▁begin>",+ "<tool▁call▁begin>",+];++/// Accommodate the known ASCII/full-width and doubled-pipe spellings.+fn control_key(candidate: &str) -> String {+ candidate+ .chars()+ .take(80)+ .filter(|c| !c.is_whitespace() && *c != '|' && *c != '|')+ .collect()+}++fn is_control(candidate: &str) -> bool {+ let key = control_key(candidate);+ CONTROL_MARKERS.iter().any(|marker| key.starts_with(marker))+}++/// A line still being written that could yet turn out to be a marker.+fn may_become_control(candidate: &str) -> bool {+ let key = control_key(candidate);+ CONTROL_MARKERS+ .iter()+ .any(|marker| marker.starts_with(key.as_str()))+}++/// Whether a reply is a tool call written out as text (#845, #937). Explanations+/// and fenced examples are prose, and an explicit request about the encoding may+/// legitimately ask for the raw markup. This is a publication guard: never+/// interpret this text as an executable tool call.+pub(crate) fn tool_call_text(text: &str, question: &str) -> bool {let request = question.to_ascii_lowercase();- let asks_for_encoding = request.contains("dsml")+ let asks_for_encoding = (request.contains("dsml") || request.contains("tool▁call"))&& ["example", "verbatim", "示例", "原样"].iter().any(|term| request.contains(term))&& !request.contains("business")&& !request.contains("业务");- if !asks_for_encoding {- // Accommodate the known ASCII/full-width and doubled-pipe spellings.- // Inspect the assembled turn, so SSE chunk boundaries do not matter.- let is_control = |candidate: &str| {- let prefix: String = candidate- .chars()- .take(80)- .filter(|c| !c.is_whitespace() && *c != '|' && *c != '|')- .collect();- ["<DSMLcalls>", "<DSMLtool_calls>", "<DSMLinvokename="]- .iter()- .any(|marker| prefix.starts_with(marker))- };- // A real endpoint prefixed its final control block with "Let me examine- // it." in the SAME turn. Check bare line starts too, without treating- // inline mentions, block quotes, or Markdown fenced examples as calls.- let mut fence: Option<(char, usize)> = None;- let mut bare_control = is_control(text);- for line in text.lines() {- let line = line.trim_start();- if let Some(marker @ ('`' | '~')) = line.chars().next() {- let len = line.chars().take_while(|c| *c == marker).count();- if len >= 3 {- match fence {- None => fence = Some((marker, len)),- Some((open, size))- if marker == open && len >= size && line[len..].trim().is_empty() =>- {- fence = None;- }- _ => {}+ if asks_for_encoding {+ return false;+ }+ // Inspect the assembled turn, so SSE chunk boundaries do not matter.+ let text = text.trim();+ if is_control(text) {+ return true;+ }+ // A real endpoint prefixed its final control block with "Let me examine+ // it." in the SAME turn. Check bare line starts too, without treating+ // inline mentions, block quotes, or Markdown fenced examples as calls.+ let mut fence: Option<(char, usize)> = None;+ for line in text.lines() {+ let line = line.trim_start();+ if let Some(marker @ ('`' | '~')) = line.chars().next() {+ let len = line.chars().take_while(|c| *c == marker).count();+ if len >= 3 {+ match fence {+ None => fence = Some((marker, len)),+ Some((open, size))+ if marker == open && len >= size && line[len..].trim().is_empty() =>+ {+ fence = None;}- continue;+ _ => {}}+ continue;}- if fence.is_none() && is_control(line) {- bare_control = true;- break;- }}- if bare_control {- return Some("Model returned tool-control text instead of a final answer");+ if fence.is_none() && is_control(line) {+ return true;+ }+ }+ false+}++/// How much of a tool round's text can go out while it is still streaming (#937).+///+/// Text goes out line by line. A line is held while it could still become a+/// tool-call marker; once one has, the rest of the turn is held with it, to be+/// sent back or dropped. `from` is how much has gone out already, and the line+/// it ends in was judged then. Fences are not tracked here: a marker in a fenced+/// example is held until the turn ends, when [`tool_call_text`] releases it.+pub(crate) fn publishable(text: &str, from: usize, holding: &mut bool) -> usize {+ if *holding {+ return from;+ }+ let mut at = if from == 0 || text[..from].ends_with('\n') {+ from+ } else {+ match text[from..].find('\n') {+ Some(i) => from + i + 1,+ None => return text.len(),+ }+ };+ while at < text.len() {+ let rest = &text[at..];+ let (line, finished) = match rest.find('\n') {+ Some(i) => (&rest[..i], true),+ None => (rest, false),+ };+ if is_control(line) {+ *holding = true;+ return at;+ }+ if !finished {+ return if may_become_control(line) {+ at+ } else {+ text.len()+ };}+ at += line.len() + 1;}- None+ at+}++/// What keeps a budget-boundary candidate from being published: a structured+/// call, blank text, or a tool call written as text.+pub(crate) fn finalization_error(+ text: &str,+ has_calls: bool,+ question: &str,+) -> Option<&'static str> {+ if has_calls {+ return Some(FINAL_TOOL_CALL);+ }+ if text.trim().is_empty() {+ return Some("Model returned an empty answer");+ }+ tool_call_text(text, question).then_some(CONTROL_TEXT)}/// 一场对话里工具共用的东西:库、权限、引用清单,以及给界面的轨迹。@@ -161,6 +239,8 @@ pub struct Shared {nudged: AtomicBool,/// 空回复的重问也只给一次(见 `EMPTY_REPLY_RETRY`)asked_again: AtomicBool,+ /// 把工具调用写成正文的退回也只给一次(见 `MARKUP_RETRY`)+ markup_retried: AtomicBool,/// Set before the final request so the route can withhold unvalidated text.finalizing: AtomicBool,/// Only a rejected model candidate authorizes the one-shot recovery, not an@@ -196,6 +276,7 @@ impl Shared {gate_passed: AtomicBool::new(false),nudged: AtomicBool::new(false),asked_again: AtomicBool::new(false),+ markup_retried: AtomicBool::new(false),finalizing: AtomicBool::new(false),answer_requested: AtomicBool::new(false),})@@ -359,6 +440,17 @@ impl AgentHook for Policy {.content.iter().any(|c| matches!(c, AssistantContent::Text(t) if !t.text.trim().is_empty()));+ let written_call = !has_tool_call && {+ let text: String = event+ .content+ .iter()+ .filter_map(|c| match c {+ AssistantContent::Text(t) => Some(t.text.as_str()),+ _ => None,+ })+ .collect();+ tool_call_text(&text, &self.shared.question)+ };let turn = event.turn;let shared = self.shared.clone();let max_rounds = self.max_rounds;@@ -374,6 +466,25 @@ impl AgentHook for Policy {}return ModelTurnAction::Continue;}+ // 工具调用写成了正文:什么都没执行,这一回合也不是答案(#937)。哪一轮都+ // 一样,不只是首轮。退回一次;再这样就放它结束,`chat` 那边扣着没发的+ // 标记,以 `CONTROL_TEXT` 收尾,不存+ if written_call {+ if !shared.markup_retried.swap(true, Ordering::Relaxed) {+ tracing::warn!(+ model = shared.model,+ turn,+ "模型把工具调用写成了正文,退回一次"+ );+ return ModelTurnAction::Retry(RetryRequest::Feedback(MARKUP_RETRY.into()));+ }+ tracing::warn!(+ model = shared.model,+ turn,+ "退回一次后仍把工具调用写成正文,按错误收尾"+ );+ return ModelTurnAction::Continue;+ }if has_tool_call || shared.gate_passed.load(Ordering::Relaxed) || turn > max_rounds {return ModelTurnAction::Continue;}@@ -560,6 +671,77 @@ mod tests {assert!(finalization_error(after_example, false, "What happened?").is_some());}+ /// DeepSeek 的原生标记也是写成正文的工具调用(#937):两种竖线、叙述之后的行首都认;+ /// 行内提及、引用、围栏里的示例、明确要原样示例的问题都不算+ #[test]+ fn deepseek_markup_is_a_tool_call_written_as_text() {+ let call =+ "<|tool▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>search_chunks\n\+ ```json\n{\"query\":\"Acme\"}\n```<|tool▁call▁end|><|tool▁calls▁end|>";+ let question = "What changed at Acme?";+ assert!(tool_call_text(call, question));+ assert!(tool_call_text(&call.replace('|', "|"), question));+ assert!(tool_call_text(+ &format!("Let me search.\n\n{call}"),+ question+ ));+ assert!(tool_call_text(+ "<|tool▁call▁begin|>function<|tool▁sep|>search_chunks",+ question+ ));+ for prose in [+ "The reply contained <|tool▁calls▁begin|> in the middle of a sentence.",+ "> <|tool▁calls▁begin|>\nThat line quotes the markup.",+ "It looks like this:\n~~~\n<|tool▁calls▁begin|><|tool▁call▁begin|>\n~~~",+ "Acme's revenue rose 5% [1].",+ ] {+ assert!(!tool_call_text(prose, question), "{prose}");+ }+ assert!(!tool_call_text(+ call,+ "Show a <|tool▁calls▁begin|> example verbatim"+ ));+ assert_eq!(+ finalization_error(call, false, question),+ Some(CONTROL_TEXT)+ );+ }++ /// 流式外发按行:可能长成标记的那一行扣着,成了标记就连同这一回合剩下的都扣着;+ /// 其余照常外发,已经发出去一半的那一行不再判断+ #[test]+ fn a_line_is_held_while_it_may_become_a_tool_call() {+ // (本回合正文, 已发出的字节数, 应当发到哪里为止, 之后是否整回合扣住)+ for (text, from, out, held) in [+ ("Let me search.\n<|to", 0, "Let me search.\n", false),+ (+ "Let me search.\n<|tool▁calls▁begin|><|tool▁call▁begin|>",+ 15,+ "Let me search.\n",+ true,+ ),+ ("One.\n\nTwo.\n ", 0, "One.\n\nTwo.\n", false),+ ("<", 0, "", false),+ ("<b>Acme</b> grew", 0, "<b>Acme</b> grew", false),+ ("Acme grew\n<b>", 9, "Acme grew\n<b>", false),+ (+ "Acme grew <|tool▁calls▁begin|> inline",+ 9,+ "Acme grew <|tool▁calls▁begin|> inline",+ false,+ ),+ ("Acme grew\n<|tool▁call▁begin|>", 9, "Acme grew\n", true),+ ("<|DSML|calls>\nMore.", 0, "", true),+ ] {+ let mut holding = false;+ let upto = publishable(text, from, &mut holding);+ assert_eq!(&text[..upto], out, "{text:?} from {from}");+ assert_eq!(holding, held, "{text:?} from {from}");+ }+ let mut holding = true;+ assert_eq!(publishable("Held.\nStill held.", 0, &mut holding), 0);+ }+/// 上一轮的工具往返插在它的结论之前,tool 消息找回自己的工具名#[test]fn the_last_exchange_sits_before_its_conclusion() {diff --git a/crates/utopia-server/src/api/chat.rs b/crates/utopia-server/src/api/chat.rsindex e0fdd3b9..bd62e974 100644--- a/crates/utopia-server/src/api/chat.rs+++ b/crates/utopia-server/src/api/chat.rs@@ -758,6 +758,9 @@ pub async fn chat(// 当前模型回合里说的话与发出的调用;回合的结果一到,攒成一条 assistant 消息let mut turn_text = String::new();let mut turn_calls: Vec<serde_json::Value> = Vec::new();+ // 这一回合的正文发出去了多少;后面的扣着,因为那可能是写成正文的工具调用(#937)+ let mut turn_published = 0usize;+ let mut turn_holding = false;let mut finished = false;let mut published_sources = 0;let mut answer_requested = false;@@ -765,8 +768,10 @@ pub async fn chat(while let Some(item) = run.next().await {match item {Ok(MultiTurnStreamItem::StreamAssistantItem(StreamedAssistantContent::Text(t))) => {- // Tool-round narration stays live. Withhold only the final call:- // validation after streaming cannot retract protocol garbage.+ // Tool-round narration stays live, line by line; a line that is or+ // may become a tool call written as text is held (#937). Withhold the+ // final call whole: validation after streaming cannot retract+ // protocol garbage.if shared.finalizing() {if turn_text.len().saturating_add(t.text.len()) > agent::MAX_FINAL_ANSWER_BYTES {yield ProducerEvent::Outcome(Err("Model final answer exceeded the size limit".into()));@@ -774,9 +779,14 @@ pub async fn chat(}turn_text.push_str(&t.text);} else {- answer_acc.push_str(&t.text);turn_text.push_str(&t.text);- yield ProducerEvent::Progress(delta_event(&t.text));+ let upto = agent::publishable(&turn_text, turn_published, &mut turn_holding);+ if upto > turn_published {+ let fresh = turn_text[turn_published..upto].to_string();+ turn_published = upto;+ answer_acc.push_str(&fresh);+ yield ProducerEvent::Progress(delta_event(&fresh));+ }}}Ok(MultiTurnStreamItem::StreamAssistantItem(StreamedAssistantContent::ToolCall {@@ -796,6 +806,17 @@ pub async fn chat(internal_call_id,})) => {if !turn_calls.is_empty() {+ // 这一回合真的调了工具。扣着的是标记就丢掉,也不回放给模型;+ // 不是(例如围栏里的示例)就补发,赶在这一步之前,`at` 才对得上+ if turn_published < turn_text.len() {+ if agent::tool_call_text(&turn_text, &query) {+ turn_text.truncate(turn_published);+ } else {+ let rest = turn_text[turn_published..].to_string();+ answer_acc.push_str(&rest);+ yield ProducerEvent::Progress(delta_event(&rest));+ }+ }// 工具轮带了叙述文本:与后续轮次的正文之间补一个段落分隔if !turn_text.is_empty() {answer_acc.push_str("\n\n");@@ -811,6 +832,8 @@ pub async fn chat("tool_calls": std::mem::take(&mut turn_calls),}));turn_text.clear();+ turn_published = 0;+ turn_holding = false;}let text = rig_model::tool_result_text(&tool_result.content);// 闸门工具不留轨迹:「你好」下面挂一条「声明不用查」是噪音@@ -855,15 +878,17 @@ pub async fn chat(if let Some(failed) = is_error { recorded["is_error"] = json!(failed); }exchange_acc.push(recorded);}- // 钩子把一个只说不查的回合退了回去:那段话已经流给用户,收不回来;- // 接下来的正文另起一段+ // 钩子把这一回合退了回去。已经流给用户的话收不回来,接下来的正文+ // 另起一段;还扣着的(写成正文的工具调用)随这一回合丢掉Ok(MultiTurnStreamItem::ModelTurnRetried { .. }) => {- if !turn_text.is_empty() {+ if turn_published > 0 {answer_acc.push_str("\n\n");yield ProducerEvent::Progress(delta_event("\n\n"));}turn_text.clear();turn_calls.clear();+ turn_published = 0;+ turn_holding = false;}Ok(MultiTurnStreamItem::FinalResponse(_)) => finished = true,Ok(_) => {}@@ -927,9 +952,22 @@ pub async fn chat(return;}answer_acc.push_str(&turn_text);- } else if turn_text.trim().is_empty() {- yield ProducerEvent::Outcome(Err("Model returned an empty answer".into()));- return;+ } else {+ // 纯文字的最后一回合还扣着东西。是写成正文的工具调用,钩子已经退回过+ // 一次,这是第二次:报错,不存(#937)。不是(例如围栏里的示例),补发+ if turn_published < turn_text.len() {+ if agent::tool_call_text(&turn_text, &query) {+ yield ProducerEvent::Outcome(Err(agent::CONTROL_TEXT.into()));+ return;+ }+ let rest = turn_text[turn_published..].to_string();+ answer_acc.push_str(&rest);+ yield ProducerEvent::Progress(delta_event(&rest));+ }+ if turn_text.trim().is_empty() {+ yield ProducerEvent::Outcome(Err("Model returned an empty answer".into()));+ return;+ }}let (sources, resolved) = {let sink = shared.sink.lock().await;diff --git a/crates/utopia-server/src/api/chat_empty_reply_tests.rs b/crates/utopia-server/src/api/chat_empty_reply_tests.rsindex b66f51ae..bc2c6a35 100644--- a/crates/utopia-server/src/api/chat_empty_reply_tests.rs+++ b/crates/utopia-server/src/api/chat_empty_reply_tests.rs@@ -33,6 +33,8 @@ pub(super) enum Reply {Text(&'static str),SplitText(&'static [&'static str]),NarratedTool,+ /// 一段正文,同一回合里还真的调了 find_entities+ TextWithTool(&'static str),ParallelTools,OversizedText,Finished(&'static str, &'static str),@@ -109,6 +111,12 @@ impl Respond for Scripted {"function": { "name": "find_entities", "arguments": "{\"name\":\"Acme\"}" }}]} }] })),+ Reply::TextWithTool(text) => Some(serde_json::json!({ "choices": [{ "delta": {+ "content": text,+ "tool_calls": [{ "index": 0, "id": format!("call_{n}"),+ "function": { "name": "find_entities", "arguments": "{\"name\":\"Acme\"}" }+ }]+ } }] })),Reply::Document(id) => Some(serde_json::json!({ "choices": [{ "delta": {"tool_calls": [{ "index": 0, "id": format!("call_{n}"),"function": { "name": "get_document", "arguments":@@ -368,6 +376,8 @@ mod fallback_tests;mod history_tests;#[path = "chat_lookup_tests.rs"]mod lookup_tests;+#[path = "chat_markup_tests.rs"]+mod markup_tests;#[path = "chat_persistence_tests.rs"]mod persistence_tests;#[path = "chat_registry_tests.rs"]diff --git a/crates/utopia-server/src/api/chat_markup_tests.rs b/crates/utopia-server/src/api/chat_markup_tests.rsnew file mode 100644index 00000000..dc109f05--- /dev/null+++ b/crates/utopia-server/src/api/chat_markup_tests.rs@@ -0,0 +1,273 @@+//! 模型把工具调用写成了正文(#937)。+//!+//! 端点偶尔不走协议里的 `tool_calls`,而是把调用的标记原样写进 `delta.content`:+//! DeepSeek 的 `<|tool▁calls▁begin|>…`,或者 DSML。什么都没执行,那段也不是答案。+//! 从前只有预算用完后的收尾那一轮会查它;普通回合里它照常流给用户、存成答案、+//! 以 `done` 收尾。这里钉的是现在的样子:+//!+//! 1. 标记不外发:流里、存下的答案里、回放给下一轮的工具往返里都没有;+//! 2. 只有文字的那一回合退回一次(`MARKUP_RETRY`),模型接着调工具或作答;+//! 3. 再写一次就以 error 收尾,什么都不存;+//! 4. 叙述照常流出,只扣可能是标记的那几行。只是以 `<` 开头的正文、围栏里的示例,+//! 分辨出来之后原样放行,也不会被退回。+use super::*;++/// DeepSeek 的原生写法:一次工具调用被原样写成了正文+const CALL: &str = "<|tool▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>search_chunks\n\+ ```json\n{\"query\":\"Acme revenue\"}\n```<|tool▁call▁end|><|tool▁calls▁end|>";+const ANSWER: &str = "Acme's revenue rose 5% last quarter.";+const TOOL: Reply = Reply::Tool("find_entities", r#"{"name":"Acme"}"#);+const QUESTION: &str = "What was Acme's revenue last quarter?";++/// 流给用户的全部正文,按帧拼起来+fn streamed(sse: &str) -> String {+ sse.split("\n\n")+ .filter_map(|frame| frame.strip_prefix("event: delta\ndata: "))+ .map(|data| {+ serde_json::from_str::<serde_json::Value>(data).expect("delta is JSON")["text"]+ .as_str()+ .expect("delta has text")+ .to_string()+ })+ .collect()+}++/// 有几次请求以退回的那句话结尾+fn sent_back(f: &Fx) -> usize {+ f.requests()+ .iter()+ .filter(|r| last_message(r) == agent::MARKUP_RETRY)+ .count()+}++/// 存下的这一轮工具往返,下一轮回放给模型的就是它+async fn stored_exchange(f: &Fx) -> anyhow::Result<serde_json::Value> {+ Ok(sqlx::query_scalar(+ "SELECT m.tool_exchange FROM conversation_messages m+ JOIN conversations c ON c.id = m.conversation_id+ WHERE c.kb_id = $1 AND m.role = 'assistant'",+ )+ .bind(f.kb)+ .fetch_one(&f.pool)+ .await?)+}++#[tokio::test]+async fn a_call_written_as_text_after_a_tool_is_sent_back_once() -> anyhow::Result<()> {+ let Some(f) = fixture(Scripted::new(vec![+ TOOL,+ Reply::Text(CALL),+ Reply::Text(ANSWER),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f.ask(QUESTION).await?;++ assert!(sse.contains("event: done"), "{sse}");+ assert!(!sse.contains("event: error"), "{sse}");+ assert_eq!(+ streamed(&sse),+ ANSWER,+ "the markup never reaches the reader"+ );+ assert_eq!(f.stored_answer().await?.as_deref(), Some(ANSWER));+ let requests = f.requests();+ assert_eq!(requests.len(), 3, "tool, markup, then one retry");+ assert_eq!(last_message(&requests[2]), agent::MARKUP_RETRY);+ assert_eq!(sent_back(&f), 1);+ // 退回时保留那一回合:模型看得见自己写了什么+ let messages = requests[2]["messages"].as_array().expect("messages");+ assert_eq!(messages[messages.len() - 2]["role"], "assistant");+ assert_eq!(messages[messages.len() - 2]["content"], CALL);+ f.cleanup().await+}++#[tokio::test]+async fn a_call_written_as_text_in_the_first_turn_is_left_out_of_the_answer() -> anyhow::Result<()>+{+ let Some(f) = fixture(Scripted::new(vec![+ Reply::Text(CALL),+ TOOL,+ Reply::Text(ANSWER),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f.ask(QUESTION).await?;++ assert!(sse.contains("event: done"), "{sse}");+ assert!(+ sse.contains("event: step"),+ "the retry made the call for real:\n{sse}"+ );+ assert_eq!(streamed(&sse), ANSWER);+ assert_eq!(f.stored_answer().await?.as_deref(), Some(ANSWER));+ let requests = f.requests();+ assert_eq!(requests.len(), 3);+ assert_eq!(+ last_message(&requests[1]),+ agent::MARKUP_RETRY,+ "sent back as markup, not as a turn that skipped the tools"+ );+ assert_eq!(sent_back(&f), 1);+ f.cleanup().await+}++#[tokio::test]+async fn a_call_written_as_text_twice_is_an_error_and_nothing_is_stored() -> anyhow::Result<()> {+ let Some(f) = fixture(Scripted::new(vec![+ TOOL,+ Reply::Text(CALL),+ Reply::Text(CALL),+ Reply::Text("Never requested"),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f.ask(QUESTION).await?;++ assert!(sse.contains("event: error"), "{sse}");+ assert!(!sse.contains("event: done"), "{sse}");+ assert!(sse.contains(agent::CONTROL_TEXT), "{sse}");+ assert_eq!(streamed(&sse), "", "neither turn reached the reader");+ assert!(f.stored_answer().await?.is_none());+ assert_eq!(f.requests().len(), 3, "one retry, not a loop");+ assert_eq!(sent_back(&f), 1);+ f.cleanup().await+}++#[tokio::test]+async fn narration_streams_while_a_split_marker_is_held() -> anyhow::Result<()> {+ let Some(f) = fixture(Scripted::new(vec![+ TOOL,+ Reply::SplitText(&[+ "I will search the filings.\n",+ "<|tool",+ "▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>search_chunks\n",+ "```json\n{\"query\":\"Acme filings\"}\n```<|tool▁call▁end|><|tool▁calls▁end|>",+ ]),+ Reply::Text(ANSWER),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f.ask(QUESTION).await?;++ assert!(sse.contains("event: done"), "{sse}");+ let shown = streamed(&sse);+ assert!(+ shown.starts_with("I will search the filings.\n"),+ "{shown:?}"+ );+ assert!(shown.ends_with(ANSWER), "{shown:?}");+ assert!(+ !shown.contains("<|tool") && !shown.contains("```json"),+ "no part of the split marker went out: {shown:?}"+ );+ assert_eq!(f.stored_answer().await?.as_deref(), Some(shown.as_str()));+ assert_eq!(sent_back(&f), 1);+ f.cleanup().await+}++#[tokio::test]+async fn markup_beside_a_real_call_is_dropped_and_not_replayed() -> anyhow::Result<()> {+ const SAID: &str = "Checking the ledger.\n<|tool▁calls▁begin|><|tool▁call▁begin|>function\+ <|tool▁sep|>find_entities\n```json\n{\"name\":\"Acme\"}\n```\+ <|tool▁call▁end|><|tool▁calls▁end|>";+ let Some(f) = fixture(Scripted::new(vec![+ Reply::TextWithTool(SAID),+ Reply::Text(ANSWER),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f.ask(QUESTION).await?;++ assert!(sse.contains("event: done"), "{sse}");+ let shown = streamed(&sse);+ assert!(shown.starts_with("Checking the ledger.\n"), "{shown:?}");+ assert!(shown.ends_with(ANSWER), "{shown:?}");+ assert!(!shown.contains("tool▁"), "{shown:?}");+ assert_eq!(f.stored_answer().await?.as_deref(), Some(shown.as_str()));+ assert_eq!(+ f.requests().len(),+ 2,+ "the real call ran; nothing is sent back"+ );+ assert_eq!(sent_back(&f), 0);+ // 这一步落在叙述之后、答案之前,不在扣掉的标记之后+ let step: serde_json::Value = sse+ .split("\n\n")+ .find_map(|frame| frame.strip_prefix("event: step\ndata: "))+ .map(serde_json::from_str)+ .expect("a step")?;+ let before_answer = shown.len() - ANSWER.len();+ assert_eq!(step["at"], serde_json::json!(before_answer));+ let exchange = stored_exchange(&f).await?;+ assert_eq!(exchange[0]["content"], "Checking the ledger.\n");+ assert!(!exchange.to_string().contains("tool▁"), "{exchange}");+ f.cleanup().await+}++#[tokio::test]+async fn text_that_only_starts_like_a_marker_goes_out_whole_and_once() -> anyhow::Result<()> {+ const TEXT: &str = "<b>Acme</b> grew 5% [1].\n<DSML> is the name of that markup, not a call.";+ let Some(f) = fixture(Scripted::new(vec![+ TOOL,+ Reply::SplitText(&[+ "<",+ "b>Acme</b> grew 5% [1].",+ "\n<DS",+ "ML> is the name of that markup, not a call.",+ ]),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f.ask(QUESTION).await?;++ assert!(sse.contains("event: done"), "{sse}");+ assert_eq!(streamed(&sse), TEXT);+ assert_eq!(f.stored_answer().await?.as_deref(), Some(TEXT));+ assert_eq!(f.requests().len(), 2);+ assert_eq!(sent_back(&f), 0);+ f.cleanup().await+}++#[tokio::test]+async fn a_fenced_example_of_the_markup_is_released_when_the_turn_ends() -> anyhow::Result<()> {+ const EXAMPLE: &str = "It looks like this:\n~~~\n<|tool▁calls▁begin|><|tool▁call▁begin|>\+ function<|tool▁sep|>search_chunks\n~~~\nThat is the raw form of one call.";+ let Some(f) = fixture(Scripted::new(vec![+ Reply::Tool("no_evidence_needed", r#"{"reason":"about the format"}"#),+ Reply::Text(EXAMPLE),+ ]))+ .await?+ else {+ return Ok(());+ };++ let sse = f+ .ask("What does DeepSeek's raw tool-call format look like?")+ .await?;++ assert!(sse.contains("event: done"), "{sse}");+ assert_eq!(streamed(&sse), EXAMPLE);+ assert_eq!(f.stored_answer().await?.as_deref(), Some(EXAMPLE));+ assert_eq!(f.requests().len(), 2);+ assert_eq!(sent_back(&f), 0);+ f.cleanup().await+}diff --git a/crates/utopia-server/src/api/chat_terminal_tests.rs b/crates/utopia-server/src/api/chat_terminal_tests.rsindex b183bfa0..ced0edd5 100644--- a/crates/utopia-server/src/api/chat_terminal_tests.rs+++ b/crates/utopia-server/src/api/chat_terminal_tests.rs@@ -101,9 +101,10 @@ fn assert_one_earned_terminal(what: &str, ends: Ends, sse: &str) {/// 一个工具调用,让这一轮走完整的取证路径再收尾。const TOOL: Reply = Reply::Tool("find_entities", r#"{"name":"Acme"}"#);-// #845 guards the exhausted gathering boundary, not every ordinary early answer.-// Reach that boundary before injecting a final candidate; do not widen the policy-// just to make a one-tool fixture exercise a six-turn handoff.+// The #845 rows test the exhausted gathering boundary, so they reach it before+// injecting a final candidate; a one-tool fixture would exercise an ordinary turn+// instead. Since #937 an ordinary turn is checked for tool-control text as well,+// in a row of its own.fn at_budget(candidate: Reply) -> Vec<Reply> {let mut replies = vec![TOOL; 6];replies.push(candidate);@@ -151,6 +152,17 @@ fn table() -> Vec<Case> {ends: Ends::Error,fallback: false,},+ // #937:普通回合把工具调用写成正文,退回一次之后又写了一次+ Case {+ what: "普通回合连续两次把工具调用写成正文",+ replies: vec![+ TOOL,+ Reply::Text("<DSMLcalls><DSMLinvoke name=\"search\"></DSMLinvoke></DSMLcalls>"),+ Reply::Text("<DSMLcalls><DSMLinvoke name=\"search\"></DSMLinvoke></DSMLcalls>"),+ ],+ ends: Ends::Error,+ fallback: false,+ },]}diff --git a/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md b/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.mdindex 5170ce4b..d3bac4f2 100644--- a/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md+++ b/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md@@ -2,7 +2,8 @@- **Status**: Implemented (#548) · the loop is rig's runner (`rig-core` / `rig-agent` 0.42, nodefault features) · policy is one `AgentHook` in `api/agent.rs` · the wire stays `LlmClient`- behind `api/rig_model.rs`+ behind `api/rig_model.rs` · revised 2026-09-26 (#937: tool-control text is checked in every+ turn, not only the boundary answer)- **Written**: 2026-09-13 (conventions in the [README](README.md))- **Related**: #546 (the issue and its findings), #509 / #543 (the stall and the guard thisreplaces), #547 (the mark on answers that cite nothing), #631 (the empty-reply retry, moved@@ -35,7 +36,7 @@ a branch in a loop body:| `on_completion_call` | `tool_choice: required` until a tool has run; at the budget boundary, stop before provider I/O and hand evidence to the answer call || `on_tool_call` | `check_call` refuses a malformed call, and the model gets the same message as before || `on_tool_result` | the tool's UI step goes to the stream |-| `on_model_turn_finished` | an empty turn is asked again once (#631); a text-only first turn from an endpoint that ignored `required` is sent back once |+| `on_model_turn_finished` | an empty turn is asked again once (#631); a text-only first turn from an endpoint that ignored `required` is sent back once; a turn that writes its tool call as text is sent back once (#937) |rig was chosen over swiftide-agents because it is maintained and does not bring its own contextmodel. Neither has a provider we want: see decision 2.@@ -125,6 +126,21 @@ model or database I/O. The assistant INSERT must succeed before the buffered ansterminal event. Already-streamed early narration cannot be retracted. Disconnect/reattachcontinues through the existing background producer and persisted body/source mapping.+> **Revised 2026-09-26 (#937): tool-control text is checked in every turn.** The two paragraphs+> above kept the check to the boundary answer. An ordinary turn could still write its call as+> text, with no structured call: DeepSeek's native `<|tool▁call▁begin|>` markup (2 of the 35+> questions in the bench's chat run failed this way) or DSML. Nothing ran, yet the markup+> streamed to the reader, was stored as the answer and ended in `done`; in a first turn the+> required-tool nudge got a real call, but the markup stayed at the head of the stored answer.+> The check now runs on every turn and also knows DeepSeek's two opening markers. While a tool+> round streams, a line that is or may still become a marker is held back, and after a marker+> the rest of the turn is held with it; other narration streams as before. A text-only turn+> that is tool-control text is sent back once (`MARKUP_RETRY`, keeping the turn so the model+> sees what it wrote); a second one is an error and nothing is stored. Held text beside a+> structured call is dropped if it is markup and released otherwise, and a fenced example is+> released when its turn ends. It is still a publication guard, not a parser, and Qwen-style+> `<tool_call>` tags are not recognised.+This boundary is not a factuality oracle. The evaluation separately records required factslots, citation syntax/mapping, additional unsupported statements and false insufficiency.A clean result on one frozen set is not a zero-failure guarantee.diff --git a/docs/decisions/README.md b/docs/decisions/README.mdindex 6eff5539..9fd0a958 100644--- a/docs/decisions/README.md+++ b/docs/decisions/README.md@@ -67,7 +67,7 @@ The test for writing one: if someone (including us) looks at a piece of code in| 0039 | [A chunk is what extraction sees](0039-a-chunk-is-what-extraction-sees.md) | Implemented, cut 1 · the parser's Markdown is read into blocks and packed by a token budget (1000, cl100k) · a table travels with its caption and header and a continuation repeats both · headings become a breadcrumb and fill `chunks.heading` · a page-break rule is not a boundary and a table it split is joined back · not measured on its own: the budget · cut 2: an external parser behind the block model, evidence that names a cell or a region || 0040 | [A chunk says where its words came from](0040-a-chunk-says-where-its-words-came-from.md) | Cuts 1–3 implemented (the ledger shape, and a file that needs a reader degrades with an alert instead of becoming garbage text; scans and images read by a workspace's MinerU service, one segment per page, waiting on the service without spending retries; recordings read by a diarizing transcription model, speakers in the text and times in the anchor, an unlabelled transcript degrades) · scans and images through a MinerU service, recordings only with speaker labels (revised 2026-09-15) · a chunk carries an **origin** (`stated`, `ocr`, `transcribed`, `described`) and the model that produced it, and an **anchor** back into the original bytes (a page and region, a time range, an image inside the file) — decided before any media reader, because a transcript stored without its times can never be tied to the recording again · the packer never mixes origins in a chunk · facts from a description enter below `AUTO_CLOSE_MIN_CONFIDENCE`, so a misread chart opens a conflict instead of closing a correct fact · per-modality model settings, media reading as a resumable job, origin and anchor in the API, MCP and RDF · cuts: ledger shape, scans via MinerU, recordings, descriptions, video || 0041 | [A name is a claim about an entity](0041-a-name-is-a-claim-about-an-entity.md) | Cut 0 built (identity bench) · cut 1 implemented (#670): names are value facts on `known_as`, the extractor reports other names, a shared name goes to the adjudicator; forward/reverse F1 0.43/0.54 → 0.68/0.68 · cuts 2–4 (name vectors and neighbours, evidence decides, re-evaluation) not started |-| 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented |+| 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented · revised 2026-09-26 (#937: a tool call written as text is held back and sent back once, in any turn) || 0043 | [Every review queue is governed](0043-every-review-queue-is-governed.md) | Decided 2026-09-14 · no code on `dev` · 0025 gave one queue — duplicate pairs — to an agent that reads the ledger before it decides, and the rest went on waiting for a person who never came: temporal conflicts, low-confidence facts, facts a new document version left stale. Every queue is governed **except nods**, which stay with the person who said the sentence. Each queue keeps its own actions and they are the people's own store calls, so an agent's decision and a person's leave the graph in the same shape; the model decides and the server checks what can be checked (a close date must appear in the evidence, a stale fact is confirmed by quoting the current version), and a failed check turns a confident verdict into a proposal that says why it was held. Every applied action records its undo. Cut 1 was built in PR #699 and that PR was **closed 2026-09-17** to re-land on the open graph after 0044 cut 2; violations, ontology defects and concept mappings are cut 2 || 0044 | [The ontology is a view over what documents say](0044-the-ontology-is-a-view-over-what-documents-say.md) | Accepted · cut 1 built (#731, #735, #736, #741, #743–#745): extraction writes open statements, memory documents take the same path, the typed path is deleted, kind words bind to classes · cut 2 (alignment producing typed facts, identity profiles, the errata agent) not built · three layers: extraction writes an open graph in the documents' words (0 of 333 statements unstated in the prototype, against 4–9% of facts bound at write time), a small ontology proposed by an agent and approved by people, and a typed graph computed from the open graph on cached signatures · time mentions resolved against a document time context by code · identity across documents on deterministic evidence before the adjudicator · an errata agent reviews the typed graph || 0045 | [A time mention is resolved against its document](0045-a-time-mention-is-resolved-against-its-document.md) | Accepted · cuts 1 and 2 built (#740): a document is dated from its own text, each mention is interpreted by the model and computed by code, upload time is used nowhere · cuts 3 and 4 (grades replace the confidence gate, re-resolution and the anchor queue) not built · a time expression is a mention with its words and place; the model returns shape, anchor, offset and granularity and code computes the interval; a document carries its own date, calendars and anchors across chunks, never its upload time; unresolved mentions wait for an anchor; timelines close on resolution grade instead of confidence |diff --git a/docs/design/chat-and-mcp.md b/docs/design/chat-and-mcp.mdindex d77f059d..2ae95d49 100644--- a/docs/design/chat-and-mcp.md+++ b/docs/design/chat-and-mcp.md@@ -10,9 +10,11 @@ Records: [0042] (the loop), [0014] (MCP tools and scope), [0015] (`remember` andpolicy is one `AgentHook` in `api/agent.rs`: `tool_choice: required` until a tool has run, then thebudget withdraws the tools and orders an answer; a malformed call is refused with the same messageas before; a tool's UI step goes to the stream; an empty turn is asked again once; a text-only first-turn from an endpoint that ignored `required` is sent back once [0042]. A turn cannot end before a-tool has run; `no_evidence_needed` is the exit for a greeting or "make it shorter"; `STALL_NUDGE`-and `DONE` are gone [0042 d3]. The wire stays `LlmClient` behind `RigModel`: the read timeout,+turn from an endpoint that ignored `required` is sent back once; a turn that writes its tool call as+text is held back from the stream and sent back once, and a second one is an error [0042]. A turn+cannot end before a tool has run; `no_evidence_needed` is the exit for a greeting or "make it+shorter"; `STALL_NUDGE` and `DONE` are gone [0042 d3]. The wire stays `LlmClient` behind+`RigModel`: the read timeout,error bodies, the classification of a failure as out of credit, rate limited, unavailable or nothingof the kind, and cache logging; earlierentities become a `system` message right before the question; degradation to one-shot RAG happens