Skip to content

World's first open-source enterprise world model.

Keep a tool call written as text out of the chat answer in every turn, and send the turn back once (#946)

Maya Kidf2e0d5bd67d9
diff --git a/crates/utopia-server/src/api/agent.rs b/crates/utopia-server/src/api/agent.rs
index a0e460ae..739ca676 100644
--- a/crates/utopia-server/src/api/agent.rs
+++ b/crates/utopia-server/src/api/agent.rs
@@ -59,78 +59,156 @@ pub(crate) const EMPTY_REPLY_RETRY: &str = "(system) Your previous reply was emp
the user now: answer from the evidence gathered above, or call a tool if you still need \
evidence.";
+/// 模型把工具调用写进了正文时退回去的那句(#937)。
+///
+/// 端点偶尔不走协议里的 `tool_calls`,而是把调用的标记原样写成正文:DeepSeek 的
+/// `<|tool▁call▁begin|>function…`(台子上对话那一路 35 题里错了 2 题),或者 DSML。
+/// 什么都没执行,那段也不是答案。退回时保留那一回合,模型看得见自己想调什么;
+/// 只给一次,再写一次就以 [`CONTROL_TEXT`] 报错
+pub(crate) const MARKUP_RETRY: &str = "(system) Your previous reply wrote a tool call out as \
+ text, so nothing ran. Make the call as a tool call, or answer the user in plain text.";
+
/// Bounded buffering applies only to the tool-free terminal call. This is a byte
/// limit, independent of the provider's token accounting.
pub(crate) const MAX_FINAL_ANSWER_BYTES: usize = 1024 * 1024;
const FINAL_TOOL_CALL: &str = "Model attempted a tool call after the tool budget was exhausted";
+pub(crate) const CONTROL_TEXT: &str = "Model returned tool-control text instead of a final answer";
-/// Deliberately scoped to budget finalization and bare control output. Explanations
-/// and fenced examples are prose, and an explicit request about DSML may legitimately
-/// ask for the raw encoding. Never interpret this text as an executable tool call.
-pub(crate) fn finalization_error(
- text: &str,
- has_calls: bool,
- question: &str,
-) -> Option<&'static str> {
- if has_calls {
- return Some(FINAL_TOOL_CALL);
- }
- let text = text.trim();
- if text.is_empty() {
- return Some("Model returned an empty answer");
- }
+/// Line starts that open a tool call written as text, as [`control_key`] spells
+/// them: DSML, and DeepSeek's native markup (#937).
+const CONTROL_MARKERS: [&str; 5] = [
+ "<DSMLcalls>",
+ "<DSMLtool_calls>",
+ "<DSMLinvokename=",
+ "<tool▁calls▁begin>",
+ "<tool▁call▁begin>",
+];
+
+/// Accommodate the known ASCII/full-width and doubled-pipe spellings.
+fn control_key(candidate: &str) -> String {
+ candidate
+ .chars()
+ .take(80)
+ .filter(|c| !c.is_whitespace() && *c != '|' && *c != '|')
+ .collect()
+}
+
+fn is_control(candidate: &str) -> bool {
+ let key = control_key(candidate);
+ CONTROL_MARKERS.iter().any(|marker| key.starts_with(marker))
+}
+
+/// A line still being written that could yet turn out to be a marker.
+fn may_become_control(candidate: &str) -> bool {
+ let key = control_key(candidate);
+ CONTROL_MARKERS
+ .iter()
+ .any(|marker| marker.starts_with(key.as_str()))
+}
+
+/// Whether a reply is a tool call written out as text (#845, #937). Explanations
+/// and fenced examples are prose, and an explicit request about the encoding may
+/// legitimately ask for the raw markup. This is a publication guard: never
+/// interpret this text as an executable tool call.
+pub(crate) fn tool_call_text(text: &str, question: &str) -> bool {
let request = question.to_ascii_lowercase();
- let asks_for_encoding = request.contains("dsml")
+ let asks_for_encoding = (request.contains("dsml") || request.contains("tool▁call"))
&& ["example", "verbatim", "示例", "原样"]
.iter()
.any(|term| request.contains(term))
&& !request.contains("business")
&& !request.contains("业务");
- if !asks_for_encoding {
- // Accommodate the known ASCII/full-width and doubled-pipe spellings.
- // Inspect the assembled turn, so SSE chunk boundaries do not matter.
- let is_control = |candidate: &str| {
- let prefix: String = candidate
- .chars()
- .take(80)
- .filter(|c| !c.is_whitespace() && *c != '|' && *c != '|')
- .collect();
- ["<DSMLcalls>", "<DSMLtool_calls>", "<DSMLinvokename="]
- .iter()
- .any(|marker| prefix.starts_with(marker))
- };
- // A real endpoint prefixed its final control block with "Let me examine
- // it." in the SAME turn. Check bare line starts too, without treating
- // inline mentions, block quotes, or Markdown fenced examples as calls.
- let mut fence: Option<(char, usize)> = None;
- let mut bare_control = is_control(text);
- for line in text.lines() {
- let line = line.trim_start();
- if let Some(marker @ ('`' | '~')) = line.chars().next() {
- let len = line.chars().take_while(|c| *c == marker).count();
- if len >= 3 {
- match fence {
- None => fence = Some((marker, len)),
- Some((open, size))
- if marker == open && len >= size && line[len..].trim().is_empty() =>
- {
- fence = None;
- }
- _ => {}
+ if asks_for_encoding {
+ return false;
+ }
+ // Inspect the assembled turn, so SSE chunk boundaries do not matter.
+ let text = text.trim();
+ if is_control(text) {
+ return true;
+ }
+ // A real endpoint prefixed its final control block with "Let me examine
+ // it." in the SAME turn. Check bare line starts too, without treating
+ // inline mentions, block quotes, or Markdown fenced examples as calls.
+ let mut fence: Option<(char, usize)> = None;
+ for line in text.lines() {
+ let line = line.trim_start();
+ if let Some(marker @ ('`' | '~')) = line.chars().next() {
+ let len = line.chars().take_while(|c| *c == marker).count();
+ if len >= 3 {
+ match fence {
+ None => fence = Some((marker, len)),
+ Some((open, size))
+ if marker == open && len >= size && line[len..].trim().is_empty() =>
+ {
+ fence = None;
}
- continue;
+ _ => {}
}
+ continue;
}
- if fence.is_none() && is_control(line) {
- bare_control = true;
- break;
- }
}
- if bare_control {
- return Some("Model returned tool-control text instead of a final answer");
+ if fence.is_none() && is_control(line) {
+ return true;
+ }
+ }
+ false
+}
+
+/// How much of a tool round's text can go out while it is still streaming (#937).
+///
+/// Text goes out line by line. A line is held while it could still become a
+/// tool-call marker; once one has, the rest of the turn is held with it, to be
+/// sent back or dropped. `from` is how much has gone out already, and the line
+/// it ends in was judged then. Fences are not tracked here: a marker in a fenced
+/// example is held until the turn ends, when [`tool_call_text`] releases it.
+pub(crate) fn publishable(text: &str, from: usize, holding: &mut bool) -> usize {
+ if *holding {
+ return from;
+ }
+ let mut at = if from == 0 || text[..from].ends_with('\n') {
+ from
+ } else {
+ match text[from..].find('\n') {
+ Some(i) => from + i + 1,
+ None => return text.len(),
+ }
+ };
+ while at < text.len() {
+ let rest = &text[at..];
+ let (line, finished) = match rest.find('\n') {
+ Some(i) => (&rest[..i], true),
+ None => (rest, false),
+ };
+ if is_control(line) {
+ *holding = true;
+ return at;
+ }
+ if !finished {
+ return if may_become_control(line) {
+ at
+ } else {
+ text.len()
+ };
}
+ at += line.len() + 1;
}
- None
+ at
+}
+
+/// What keeps a budget-boundary candidate from being published: a structured
+/// call, blank text, or a tool call written as text.
+pub(crate) fn finalization_error(
+ text: &str,
+ has_calls: bool,
+ question: &str,
+) -> Option<&'static str> {
+ if has_calls {
+ return Some(FINAL_TOOL_CALL);
+ }
+ if text.trim().is_empty() {
+ return Some("Model returned an empty answer");
+ }
+ tool_call_text(text, question).then_some(CONTROL_TEXT)
}
/// 一场对话里工具共用的东西:库、权限、引用清单,以及给界面的轨迹。
@@ -161,6 +239,8 @@ pub struct Shared {
nudged: AtomicBool,
/// 空回复的重问也只给一次(见 `EMPTY_REPLY_RETRY`)
asked_again: AtomicBool,
+ /// 把工具调用写成正文的退回也只给一次(见 `MARKUP_RETRY`)
+ markup_retried: AtomicBool,
/// Set before the final request so the route can withhold unvalidated text.
finalizing: AtomicBool,
/// Only a rejected model candidate authorizes the one-shot recovery, not an
@@ -196,6 +276,7 @@ impl Shared {
gate_passed: AtomicBool::new(false),
nudged: AtomicBool::new(false),
asked_again: AtomicBool::new(false),
+ markup_retried: AtomicBool::new(false),
finalizing: AtomicBool::new(false),
answer_requested: AtomicBool::new(false),
})
@@ -359,6 +440,17 @@ impl AgentHook for Policy {
.content
.iter()
.any(|c| matches!(c, AssistantContent::Text(t) if !t.text.trim().is_empty()));
+ let written_call = !has_tool_call && {
+ let text: String = event
+ .content
+ .iter()
+ .filter_map(|c| match c {
+ AssistantContent::Text(t) => Some(t.text.as_str()),
+ _ => None,
+ })
+ .collect();
+ tool_call_text(&text, &self.shared.question)
+ };
let turn = event.turn;
let shared = self.shared.clone();
let max_rounds = self.max_rounds;
@@ -374,6 +466,25 @@ impl AgentHook for Policy {
}
return ModelTurnAction::Continue;
}
+ // 工具调用写成了正文:什么都没执行,这一回合也不是答案(#937)。哪一轮都
+ // 一样,不只是首轮。退回一次;再这样就放它结束,`chat` 那边扣着没发的
+ // 标记,以 `CONTROL_TEXT` 收尾,不存
+ if written_call {
+ if !shared.markup_retried.swap(true, Ordering::Relaxed) {
+ tracing::warn!(
+ model = shared.model,
+ turn,
+ "模型把工具调用写成了正文,退回一次"
+ );
+ return ModelTurnAction::Retry(RetryRequest::Feedback(MARKUP_RETRY.into()));
+ }
+ tracing::warn!(
+ model = shared.model,
+ turn,
+ "退回一次后仍把工具调用写成正文,按错误收尾"
+ );
+ return ModelTurnAction::Continue;
+ }
if has_tool_call || shared.gate_passed.load(Ordering::Relaxed) || turn > max_rounds {
return ModelTurnAction::Continue;
}
@@ -560,6 +671,77 @@ mod tests {
assert!(finalization_error(after_example, false, "What happened?").is_some());
}
+ /// DeepSeek 的原生标记也是写成正文的工具调用(#937):两种竖线、叙述之后的行首都认;
+ /// 行内提及、引用、围栏里的示例、明确要原样示例的问题都不算
+ #[test]
+ fn deepseek_markup_is_a_tool_call_written_as_text() {
+ let call =
+ "<|tool▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>search_chunks\n\
+ ```json\n{\"query\":\"Acme\"}\n```<|tool▁call▁end|><|tool▁calls▁end|>";
+ let question = "What changed at Acme?";
+ assert!(tool_call_text(call, question));
+ assert!(tool_call_text(&call.replace('|', "|"), question));
+ assert!(tool_call_text(
+ &format!("Let me search.\n\n{call}"),
+ question
+ ));
+ assert!(tool_call_text(
+ "<|tool▁call▁begin|>function<|tool▁sep|>search_chunks",
+ question
+ ));
+ for prose in [
+ "The reply contained <|tool▁calls▁begin|> in the middle of a sentence.",
+ "> <|tool▁calls▁begin|>\nThat line quotes the markup.",
+ "It looks like this:\n~~~\n<|tool▁calls▁begin|><|tool▁call▁begin|>\n~~~",
+ "Acme's revenue rose 5% [1].",
+ ] {
+ assert!(!tool_call_text(prose, question), "{prose}");
+ }
+ assert!(!tool_call_text(
+ call,
+ "Show a <|tool▁calls▁begin|> example verbatim"
+ ));
+ assert_eq!(
+ finalization_error(call, false, question),
+ Some(CONTROL_TEXT)
+ );
+ }
+
+ /// 流式外发按行:可能长成标记的那一行扣着,成了标记就连同这一回合剩下的都扣着;
+ /// 其余照常外发,已经发出去一半的那一行不再判断
+ #[test]
+ fn a_line_is_held_while_it_may_become_a_tool_call() {
+ // (本回合正文, 已发出的字节数, 应当发到哪里为止, 之后是否整回合扣住)
+ for (text, from, out, held) in [
+ ("Let me search.\n<|to", 0, "Let me search.\n", false),
+ (
+ "Let me search.\n<|tool▁calls▁begin|><|tool▁call▁begin|>",
+ 15,
+ "Let me search.\n",
+ true,
+ ),
+ ("One.\n\nTwo.\n ", 0, "One.\n\nTwo.\n", false),
+ ("<", 0, "", false),
+ ("<b>Acme</b> grew", 0, "<b>Acme</b> grew", false),
+ ("Acme grew\n<b>", 9, "Acme grew\n<b>", false),
+ (
+ "Acme grew <|tool▁calls▁begin|> inline",
+ 9,
+ "Acme grew <|tool▁calls▁begin|> inline",
+ false,
+ ),
+ ("Acme grew\n<|tool▁call▁begin|>", 9, "Acme grew\n", true),
+ ("<|DSML|calls>\nMore.", 0, "", true),
+ ] {
+ let mut holding = false;
+ let upto = publishable(text, from, &mut holding);
+ assert_eq!(&text[..upto], out, "{text:?} from {from}");
+ assert_eq!(holding, held, "{text:?} from {from}");
+ }
+ let mut holding = true;
+ assert_eq!(publishable("Held.\nStill held.", 0, &mut holding), 0);
+ }
+
/// 上一轮的工具往返插在它的结论之前,tool 消息找回自己的工具名
#[test]
fn the_last_exchange_sits_before_its_conclusion() {
diff --git a/crates/utopia-server/src/api/chat.rs b/crates/utopia-server/src/api/chat.rs
index e0fdd3b9..bd62e974 100644
--- a/crates/utopia-server/src/api/chat.rs
+++ b/crates/utopia-server/src/api/chat.rs
@@ -758,6 +758,9 @@ pub async fn chat(
// 当前模型回合里说的话与发出的调用;回合的结果一到,攒成一条 assistant 消息
let mut turn_text = String::new();
let mut turn_calls: Vec<serde_json::Value> = Vec::new();
+ // 这一回合的正文发出去了多少;后面的扣着,因为那可能是写成正文的工具调用(#937)
+ let mut turn_published = 0usize;
+ let mut turn_holding = false;
let mut finished = false;
let mut published_sources = 0;
let mut answer_requested = false;
@@ -765,8 +768,10 @@ pub async fn chat(
while let Some(item) = run.next().await {
match item {
Ok(MultiTurnStreamItem::StreamAssistantItem(StreamedAssistantContent::Text(t))) => {
- // Tool-round narration stays live. Withhold only the final call:
- // validation after streaming cannot retract protocol garbage.
+ // Tool-round narration stays live, line by line; a line that is or
+ // may become a tool call written as text is held (#937). Withhold the
+ // final call whole: validation after streaming cannot retract
+ // protocol garbage.
if shared.finalizing() {
if turn_text.len().saturating_add(t.text.len()) > agent::MAX_FINAL_ANSWER_BYTES {
yield ProducerEvent::Outcome(Err("Model final answer exceeded the size limit".into()));
@@ -774,9 +779,14 @@ pub async fn chat(
}
turn_text.push_str(&t.text);
} else {
- answer_acc.push_str(&t.text);
turn_text.push_str(&t.text);
- yield ProducerEvent::Progress(delta_event(&t.text));
+ let upto = agent::publishable(&turn_text, turn_published, &mut turn_holding);
+ if upto > turn_published {
+ let fresh = turn_text[turn_published..upto].to_string();
+ turn_published = upto;
+ answer_acc.push_str(&fresh);
+ yield ProducerEvent::Progress(delta_event(&fresh));
+ }
}
}
Ok(MultiTurnStreamItem::StreamAssistantItem(StreamedAssistantContent::ToolCall {
@@ -796,6 +806,17 @@ pub async fn chat(
internal_call_id,
})) => {
if !turn_calls.is_empty() {
+ // 这一回合真的调了工具。扣着的是标记就丢掉,也不回放给模型;
+ // 不是(例如围栏里的示例)就补发,赶在这一步之前,`at` 才对得上
+ if turn_published < turn_text.len() {
+ if agent::tool_call_text(&turn_text, &query) {
+ turn_text.truncate(turn_published);
+ } else {
+ let rest = turn_text[turn_published..].to_string();
+ answer_acc.push_str(&rest);
+ yield ProducerEvent::Progress(delta_event(&rest));
+ }
+ }
// 工具轮带了叙述文本:与后续轮次的正文之间补一个段落分隔
if !turn_text.is_empty() {
answer_acc.push_str("\n\n");
@@ -811,6 +832,8 @@ pub async fn chat(
"tool_calls": std::mem::take(&mut turn_calls),
}));
turn_text.clear();
+ turn_published = 0;
+ turn_holding = false;
}
let text = rig_model::tool_result_text(&tool_result.content);
// 闸门工具不留轨迹:「你好」下面挂一条「声明不用查」是噪音
@@ -855,15 +878,17 @@ pub async fn chat(
if let Some(failed) = is_error { recorded["is_error"] = json!(failed); }
exchange_acc.push(recorded);
}
- // 钩子把一个只说不查的回合退了回去:那段话已经流给用户,收不回来;
- // 接下来的正文另起一段
+ // 钩子把这一回合退了回去。已经流给用户的话收不回来,接下来的正文
+ // 另起一段;还扣着的(写成正文的工具调用)随这一回合丢掉
Ok(MultiTurnStreamItem::ModelTurnRetried { .. }) => {
- if !turn_text.is_empty() {
+ if turn_published > 0 {
answer_acc.push_str("\n\n");
yield ProducerEvent::Progress(delta_event("\n\n"));
}
turn_text.clear();
turn_calls.clear();
+ turn_published = 0;
+ turn_holding = false;
}
Ok(MultiTurnStreamItem::FinalResponse(_)) => finished = true,
Ok(_) => {}
@@ -927,9 +952,22 @@ pub async fn chat(
return;
}
answer_acc.push_str(&turn_text);
- } else if turn_text.trim().is_empty() {
- yield ProducerEvent::Outcome(Err("Model returned an empty answer".into()));
- return;
+ } else {
+ // 纯文字的最后一回合还扣着东西。是写成正文的工具调用,钩子已经退回过
+ // 一次,这是第二次:报错,不存(#937)。不是(例如围栏里的示例),补发
+ if turn_published < turn_text.len() {
+ if agent::tool_call_text(&turn_text, &query) {
+ yield ProducerEvent::Outcome(Err(agent::CONTROL_TEXT.into()));
+ return;
+ }
+ let rest = turn_text[turn_published..].to_string();
+ answer_acc.push_str(&rest);
+ yield ProducerEvent::Progress(delta_event(&rest));
+ }
+ if turn_text.trim().is_empty() {
+ yield ProducerEvent::Outcome(Err("Model returned an empty answer".into()));
+ return;
+ }
}
let (sources, resolved) = {
let sink = shared.sink.lock().await;
diff --git a/crates/utopia-server/src/api/chat_empty_reply_tests.rs b/crates/utopia-server/src/api/chat_empty_reply_tests.rs
index b66f51ae..bc2c6a35 100644
--- a/crates/utopia-server/src/api/chat_empty_reply_tests.rs
+++ b/crates/utopia-server/src/api/chat_empty_reply_tests.rs
@@ -33,6 +33,8 @@ pub(super) enum Reply {
Text(&'static str),
SplitText(&'static [&'static str]),
NarratedTool,
+ /// 一段正文,同一回合里还真的调了 find_entities
+ TextWithTool(&'static str),
ParallelTools,
OversizedText,
Finished(&'static str, &'static str),
@@ -109,6 +111,12 @@ impl Respond for Scripted {
"function": { "name": "find_entities", "arguments": "{\"name\":\"Acme\"}" }
}]
} }] })),
+ Reply::TextWithTool(text) => Some(serde_json::json!({ "choices": [{ "delta": {
+ "content": text,
+ "tool_calls": [{ "index": 0, "id": format!("call_{n}"),
+ "function": { "name": "find_entities", "arguments": "{\"name\":\"Acme\"}" }
+ }]
+ } }] })),
Reply::Document(id) => Some(serde_json::json!({ "choices": [{ "delta": {
"tool_calls": [{ "index": 0, "id": format!("call_{n}"),
"function": { "name": "get_document", "arguments":
@@ -368,6 +376,8 @@ mod fallback_tests;
mod history_tests;
#[path = "chat_lookup_tests.rs"]
mod lookup_tests;
+#[path = "chat_markup_tests.rs"]
+mod markup_tests;
#[path = "chat_persistence_tests.rs"]
mod persistence_tests;
#[path = "chat_registry_tests.rs"]
diff --git a/crates/utopia-server/src/api/chat_markup_tests.rs b/crates/utopia-server/src/api/chat_markup_tests.rs
new file mode 100644
index 00000000..dc109f05
--- /dev/null
+++ b/crates/utopia-server/src/api/chat_markup_tests.rs
@@ -0,0 +1,273 @@
+//! 模型把工具调用写成了正文(#937)。
+//!
+//! 端点偶尔不走协议里的 `tool_calls`,而是把调用的标记原样写进 `delta.content`:
+//! DeepSeek 的 `<|tool▁calls▁begin|>…`,或者 DSML。什么都没执行,那段也不是答案。
+//! 从前只有预算用完后的收尾那一轮会查它;普通回合里它照常流给用户、存成答案、
+//! 以 `done` 收尾。这里钉的是现在的样子:
+//!
+//! 1. 标记不外发:流里、存下的答案里、回放给下一轮的工具往返里都没有;
+//! 2. 只有文字的那一回合退回一次(`MARKUP_RETRY`),模型接着调工具或作答;
+//! 3. 再写一次就以 error 收尾,什么都不存;
+//! 4. 叙述照常流出,只扣可能是标记的那几行。只是以 `<` 开头的正文、围栏里的示例,
+//! 分辨出来之后原样放行,也不会被退回。
+use super::*;
+
+/// DeepSeek 的原生写法:一次工具调用被原样写成了正文
+const CALL: &str = "<|tool▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>search_chunks\n\
+ ```json\n{\"query\":\"Acme revenue\"}\n```<|tool▁call▁end|><|tool▁calls▁end|>";
+const ANSWER: &str = "Acme's revenue rose 5% last quarter.";
+const TOOL: Reply = Reply::Tool("find_entities", r#"{"name":"Acme"}"#);
+const QUESTION: &str = "What was Acme's revenue last quarter?";
+
+/// 流给用户的全部正文,按帧拼起来
+fn streamed(sse: &str) -> String {
+ sse.split("\n\n")
+ .filter_map(|frame| frame.strip_prefix("event: delta\ndata: "))
+ .map(|data| {
+ serde_json::from_str::<serde_json::Value>(data).expect("delta is JSON")["text"]
+ .as_str()
+ .expect("delta has text")
+ .to_string()
+ })
+ .collect()
+}
+
+/// 有几次请求以退回的那句话结尾
+fn sent_back(f: &Fx) -> usize {
+ f.requests()
+ .iter()
+ .filter(|r| last_message(r) == agent::MARKUP_RETRY)
+ .count()
+}
+
+/// 存下的这一轮工具往返,下一轮回放给模型的就是它
+async fn stored_exchange(f: &Fx) -> anyhow::Result<serde_json::Value> {
+ Ok(sqlx::query_scalar(
+ "SELECT m.tool_exchange FROM conversation_messages m
+ JOIN conversations c ON c.id = m.conversation_id
+ WHERE c.kb_id = $1 AND m.role = 'assistant'",
+ )
+ .bind(f.kb)
+ .fetch_one(&f.pool)
+ .await?)
+}
+
+#[tokio::test]
+async fn a_call_written_as_text_after_a_tool_is_sent_back_once() -> anyhow::Result<()> {
+ let Some(f) = fixture(Scripted::new(vec![
+ TOOL,
+ Reply::Text(CALL),
+ Reply::Text(ANSWER),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f.ask(QUESTION).await?;
+
+ assert!(sse.contains("event: done"), "{sse}");
+ assert!(!sse.contains("event: error"), "{sse}");
+ assert_eq!(
+ streamed(&sse),
+ ANSWER,
+ "the markup never reaches the reader"
+ );
+ assert_eq!(f.stored_answer().await?.as_deref(), Some(ANSWER));
+ let requests = f.requests();
+ assert_eq!(requests.len(), 3, "tool, markup, then one retry");
+ assert_eq!(last_message(&requests[2]), agent::MARKUP_RETRY);
+ assert_eq!(sent_back(&f), 1);
+ // 退回时保留那一回合:模型看得见自己写了什么
+ let messages = requests[2]["messages"].as_array().expect("messages");
+ assert_eq!(messages[messages.len() - 2]["role"], "assistant");
+ assert_eq!(messages[messages.len() - 2]["content"], CALL);
+ f.cleanup().await
+}
+
+#[tokio::test]
+async fn a_call_written_as_text_in_the_first_turn_is_left_out_of_the_answer() -> anyhow::Result<()>
+{
+ let Some(f) = fixture(Scripted::new(vec![
+ Reply::Text(CALL),
+ TOOL,
+ Reply::Text(ANSWER),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f.ask(QUESTION).await?;
+
+ assert!(sse.contains("event: done"), "{sse}");
+ assert!(
+ sse.contains("event: step"),
+ "the retry made the call for real:\n{sse}"
+ );
+ assert_eq!(streamed(&sse), ANSWER);
+ assert_eq!(f.stored_answer().await?.as_deref(), Some(ANSWER));
+ let requests = f.requests();
+ assert_eq!(requests.len(), 3);
+ assert_eq!(
+ last_message(&requests[1]),
+ agent::MARKUP_RETRY,
+ "sent back as markup, not as a turn that skipped the tools"
+ );
+ assert_eq!(sent_back(&f), 1);
+ f.cleanup().await
+}
+
+#[tokio::test]
+async fn a_call_written_as_text_twice_is_an_error_and_nothing_is_stored() -> anyhow::Result<()> {
+ let Some(f) = fixture(Scripted::new(vec![
+ TOOL,
+ Reply::Text(CALL),
+ Reply::Text(CALL),
+ Reply::Text("Never requested"),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f.ask(QUESTION).await?;
+
+ assert!(sse.contains("event: error"), "{sse}");
+ assert!(!sse.contains("event: done"), "{sse}");
+ assert!(sse.contains(agent::CONTROL_TEXT), "{sse}");
+ assert_eq!(streamed(&sse), "", "neither turn reached the reader");
+ assert!(f.stored_answer().await?.is_none());
+ assert_eq!(f.requests().len(), 3, "one retry, not a loop");
+ assert_eq!(sent_back(&f), 1);
+ f.cleanup().await
+}
+
+#[tokio::test]
+async fn narration_streams_while_a_split_marker_is_held() -> anyhow::Result<()> {
+ let Some(f) = fixture(Scripted::new(vec![
+ TOOL,
+ Reply::SplitText(&[
+ "I will search the filings.\n",
+ "<|tool",
+ "▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>search_chunks\n",
+ "```json\n{\"query\":\"Acme filings\"}\n```<|tool▁call▁end|><|tool▁calls▁end|>",
+ ]),
+ Reply::Text(ANSWER),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f.ask(QUESTION).await?;
+
+ assert!(sse.contains("event: done"), "{sse}");
+ let shown = streamed(&sse);
+ assert!(
+ shown.starts_with("I will search the filings.\n"),
+ "{shown:?}"
+ );
+ assert!(shown.ends_with(ANSWER), "{shown:?}");
+ assert!(
+ !shown.contains("<|tool") && !shown.contains("```json"),
+ "no part of the split marker went out: {shown:?}"
+ );
+ assert_eq!(f.stored_answer().await?.as_deref(), Some(shown.as_str()));
+ assert_eq!(sent_back(&f), 1);
+ f.cleanup().await
+}
+
+#[tokio::test]
+async fn markup_beside_a_real_call_is_dropped_and_not_replayed() -> anyhow::Result<()> {
+ const SAID: &str = "Checking the ledger.\n<|tool▁calls▁begin|><|tool▁call▁begin|>function\
+ <|tool▁sep|>find_entities\n```json\n{\"name\":\"Acme\"}\n```\
+ <|tool▁call▁end|><|tool▁calls▁end|>";
+ let Some(f) = fixture(Scripted::new(vec![
+ Reply::TextWithTool(SAID),
+ Reply::Text(ANSWER),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f.ask(QUESTION).await?;
+
+ assert!(sse.contains("event: done"), "{sse}");
+ let shown = streamed(&sse);
+ assert!(shown.starts_with("Checking the ledger.\n"), "{shown:?}");
+ assert!(shown.ends_with(ANSWER), "{shown:?}");
+ assert!(!shown.contains("tool▁"), "{shown:?}");
+ assert_eq!(f.stored_answer().await?.as_deref(), Some(shown.as_str()));
+ assert_eq!(
+ f.requests().len(),
+ 2,
+ "the real call ran; nothing is sent back"
+ );
+ assert_eq!(sent_back(&f), 0);
+ // 这一步落在叙述之后、答案之前,不在扣掉的标记之后
+ let step: serde_json::Value = sse
+ .split("\n\n")
+ .find_map(|frame| frame.strip_prefix("event: step\ndata: "))
+ .map(serde_json::from_str)
+ .expect("a step")?;
+ let before_answer = shown.len() - ANSWER.len();
+ assert_eq!(step["at"], serde_json::json!(before_answer));
+ let exchange = stored_exchange(&f).await?;
+ assert_eq!(exchange[0]["content"], "Checking the ledger.\n");
+ assert!(!exchange.to_string().contains("tool▁"), "{exchange}");
+ f.cleanup().await
+}
+
+#[tokio::test]
+async fn text_that_only_starts_like_a_marker_goes_out_whole_and_once() -> anyhow::Result<()> {
+ const TEXT: &str = "<b>Acme</b> grew 5% [1].\n<DSML> is the name of that markup, not a call.";
+ let Some(f) = fixture(Scripted::new(vec![
+ TOOL,
+ Reply::SplitText(&[
+ "<",
+ "b>Acme</b> grew 5% [1].",
+ "\n<DS",
+ "ML> is the name of that markup, not a call.",
+ ]),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f.ask(QUESTION).await?;
+
+ assert!(sse.contains("event: done"), "{sse}");
+ assert_eq!(streamed(&sse), TEXT);
+ assert_eq!(f.stored_answer().await?.as_deref(), Some(TEXT));
+ assert_eq!(f.requests().len(), 2);
+ assert_eq!(sent_back(&f), 0);
+ f.cleanup().await
+}
+
+#[tokio::test]
+async fn a_fenced_example_of_the_markup_is_released_when_the_turn_ends() -> anyhow::Result<()> {
+ const EXAMPLE: &str = "It looks like this:\n~~~\n<|tool▁calls▁begin|><|tool▁call▁begin|>\
+ function<|tool▁sep|>search_chunks\n~~~\nThat is the raw form of one call.";
+ let Some(f) = fixture(Scripted::new(vec![
+ Reply::Tool("no_evidence_needed", r#"{"reason":"about the format"}"#),
+ Reply::Text(EXAMPLE),
+ ]))
+ .await?
+ else {
+ return Ok(());
+ };
+
+ let sse = f
+ .ask("What does DeepSeek's raw tool-call format look like?")
+ .await?;
+
+ assert!(sse.contains("event: done"), "{sse}");
+ assert_eq!(streamed(&sse), EXAMPLE);
+ assert_eq!(f.stored_answer().await?.as_deref(), Some(EXAMPLE));
+ assert_eq!(f.requests().len(), 2);
+ assert_eq!(sent_back(&f), 0);
+ f.cleanup().await
+}
diff --git a/crates/utopia-server/src/api/chat_terminal_tests.rs b/crates/utopia-server/src/api/chat_terminal_tests.rs
index b183bfa0..ced0edd5 100644
--- a/crates/utopia-server/src/api/chat_terminal_tests.rs
+++ b/crates/utopia-server/src/api/chat_terminal_tests.rs
@@ -101,9 +101,10 @@ fn assert_one_earned_terminal(what: &str, ends: Ends, sse: &str) {
/// 一个工具调用,让这一轮走完整的取证路径再收尾。
const TOOL: Reply = Reply::Tool("find_entities", r#"{"name":"Acme"}"#);
-// #845 guards the exhausted gathering boundary, not every ordinary early answer.
-// Reach that boundary before injecting a final candidate; do not widen the policy
-// just to make a one-tool fixture exercise a six-turn handoff.
+// The #845 rows test the exhausted gathering boundary, so they reach it before
+// injecting a final candidate; a one-tool fixture would exercise an ordinary turn
+// instead. Since #937 an ordinary turn is checked for tool-control text as well,
+// in a row of its own.
fn at_budget(candidate: Reply) -> Vec<Reply> {
let mut replies = vec![TOOL; 6];
replies.push(candidate);
@@ -151,6 +152,17 @@ fn table() -> Vec<Case> {
ends: Ends::Error,
fallback: false,
},
+ // #937:普通回合把工具调用写成正文,退回一次之后又写了一次
+ Case {
+ what: "普通回合连续两次把工具调用写成正文",
+ replies: vec![
+ TOOL,
+ Reply::Text("<DSMLcalls><DSMLinvoke name=\"search\"></DSMLinvoke></DSMLcalls>"),
+ Reply::Text("<DSMLcalls><DSMLinvoke name=\"search\"></DSMLinvoke></DSMLcalls>"),
+ ],
+ ends: Ends::Error,
+ fallback: false,
+ },
]
}
diff --git a/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md b/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md
index 5170ce4b..d3bac4f2 100644
--- a/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md
+++ b/docs/decisions/0042-the-chat-loop-is-a-runner-with-hooks.md
@@ -2,7 +2,8 @@
- **Status**: Implemented (#548) · the loop is rig's runner (`rig-core` / `rig-agent` 0.42, no
default features) · policy is one `AgentHook` in `api/agent.rs` · the wire stays `LlmClient`
- behind `api/rig_model.rs`
+ behind `api/rig_model.rs` · revised 2026-09-26 (#937: tool-control text is checked in every
+ turn, not only the boundary answer)
- **Written**: 2026-09-13 (conventions in the [README](README.md))
- **Related**: #546 (the issue and its findings), #509 / #543 (the stall and the guard this
replaces), #547 (the mark on answers that cite nothing), #631 (the empty-reply retry, moved
@@ -35,7 +36,7 @@ a branch in a loop body:
| `on_completion_call` | `tool_choice: required` until a tool has run; at the budget boundary, stop before provider I/O and hand evidence to the answer call |
| `on_tool_call` | `check_call` refuses a malformed call, and the model gets the same message as before |
| `on_tool_result` | the tool's UI step goes to the stream |
-| `on_model_turn_finished` | an empty turn is asked again once (#631); a text-only first turn from an endpoint that ignored `required` is sent back once |
+| `on_model_turn_finished` | an empty turn is asked again once (#631); a text-only first turn from an endpoint that ignored `required` is sent back once; a turn that writes its tool call as text is sent back once (#937) |
rig was chosen over swiftide-agents because it is maintained and does not bring its own context
model. Neither has a provider we want: see decision 2.
@@ -125,6 +126,21 @@ model or database I/O. The assistant INSERT must succeed before the buffered ans
terminal event. Already-streamed early narration cannot be retracted. Disconnect/reattach
continues through the existing background producer and persisted body/source mapping.
+> **Revised 2026-09-26 (#937): tool-control text is checked in every turn.** The two paragraphs
+> above kept the check to the boundary answer. An ordinary turn could still write its call as
+> text, with no structured call: DeepSeek's native `<|tool▁call▁begin|>` markup (2 of the 35
+> questions in the bench's chat run failed this way) or DSML. Nothing ran, yet the markup
+> streamed to the reader, was stored as the answer and ended in `done`; in a first turn the
+> required-tool nudge got a real call, but the markup stayed at the head of the stored answer.
+> The check now runs on every turn and also knows DeepSeek's two opening markers. While a tool
+> round streams, a line that is or may still become a marker is held back, and after a marker
+> the rest of the turn is held with it; other narration streams as before. A text-only turn
+> that is tool-control text is sent back once (`MARKUP_RETRY`, keeping the turn so the model
+> sees what it wrote); a second one is an error and nothing is stored. Held text beside a
+> structured call is dropped if it is markup and released otherwise, and a fenced example is
+> released when its turn ends. It is still a publication guard, not a parser, and Qwen-style
+> `<tool_call>` tags are not recognised.
+
This boundary is not a factuality oracle. The evaluation separately records required fact
slots, citation syntax/mapping, additional unsupported statements and false insufficiency.
A clean result on one frozen set is not a zero-failure guarantee.
diff --git a/docs/decisions/README.md b/docs/decisions/README.md
index 6eff5539..9fd0a958 100644
--- a/docs/decisions/README.md
+++ b/docs/decisions/README.md
@@ -67,7 +67,7 @@ The test for writing one: if someone (including us) looks at a piece of code in
| 0039 | [A chunk is what extraction sees](0039-a-chunk-is-what-extraction-sees.md) | Implemented, cut 1 · the parser's Markdown is read into blocks and packed by a token budget (1000, cl100k) · a table travels with its caption and header and a continuation repeats both · headings become a breadcrumb and fill `chunks.heading` · a page-break rule is not a boundary and a table it split is joined back · not measured on its own: the budget · cut 2: an external parser behind the block model, evidence that names a cell or a region |
| 0040 | [A chunk says where its words came from](0040-a-chunk-says-where-its-words-came-from.md) | Cuts 1–3 implemented (the ledger shape, and a file that needs a reader degrades with an alert instead of becoming garbage text; scans and images read by a workspace's MinerU service, one segment per page, waiting on the service without spending retries; recordings read by a diarizing transcription model, speakers in the text and times in the anchor, an unlabelled transcript degrades) · scans and images through a MinerU service, recordings only with speaker labels (revised 2026-09-15) · a chunk carries an **origin** (`stated`, `ocr`, `transcribed`, `described`) and the model that produced it, and an **anchor** back into the original bytes (a page and region, a time range, an image inside the file) — decided before any media reader, because a transcript stored without its times can never be tied to the recording again · the packer never mixes origins in a chunk · facts from a description enter below `AUTO_CLOSE_MIN_CONFIDENCE`, so a misread chart opens a conflict instead of closing a correct fact · per-modality model settings, media reading as a resumable job, origin and anchor in the API, MCP and RDF · cuts: ledger shape, scans via MinerU, recordings, descriptions, video |
| 0041 | [A name is a claim about an entity](0041-a-name-is-a-claim-about-an-entity.md) | Cut 0 built (identity bench) · cut 1 implemented (#670): names are value facts on `known_as`, the extractor reports other names, a shared name goes to the adjudicator; forward/reverse F1 0.43/0.54 → 0.68/0.68 · cuts 2–4 (name vectors and neighbours, evidence decides, re-evaluation) not started |
-| 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented |
+| 0042 | [The chat loop is a runner with hooks](0042-the-chat-loop-is-a-runner-with-hooks.md) | Implemented (#548) · the loop is rig's runner and every policy is a hook with a typed result · the wire stays `LlmClient` behind `RigModel` · a turn cannot end before a tool has run, `no_evidence_needed` is the exit for questions not about the base · RAG fallback only on a 400/422 to the first request with tools · an empty reply is asked again once · the skip rate is the model's (DeepSeek-V3 1–3 of 12, Qwen2.5-72B 0) and recorded, not prevented · revised 2026-09-26 (#937: a tool call written as text is held back and sent back once, in any turn) |
| 0043 | [Every review queue is governed](0043-every-review-queue-is-governed.md) | Decided 2026-09-14 · no code on `dev` · 0025 gave one queue — duplicate pairs — to an agent that reads the ledger before it decides, and the rest went on waiting for a person who never came: temporal conflicts, low-confidence facts, facts a new document version left stale. Every queue is governed **except nods**, which stay with the person who said the sentence. Each queue keeps its own actions and they are the people's own store calls, so an agent's decision and a person's leave the graph in the same shape; the model decides and the server checks what can be checked (a close date must appear in the evidence, a stale fact is confirmed by quoting the current version), and a failed check turns a confident verdict into a proposal that says why it was held. Every applied action records its undo. Cut 1 was built in PR #699 and that PR was **closed 2026-09-17** to re-land on the open graph after 0044 cut 2; violations, ontology defects and concept mappings are cut 2 |
| 0044 | [The ontology is a view over what documents say](0044-the-ontology-is-a-view-over-what-documents-say.md) | Accepted · cut 1 built (#731, #735, #736, #741, #743–#745): extraction writes open statements, memory documents take the same path, the typed path is deleted, kind words bind to classes · cut 2 (alignment producing typed facts, identity profiles, the errata agent) not built · three layers: extraction writes an open graph in the documents' words (0 of 333 statements unstated in the prototype, against 4–9% of facts bound at write time), a small ontology proposed by an agent and approved by people, and a typed graph computed from the open graph on cached signatures · time mentions resolved against a document time context by code · identity across documents on deterministic evidence before the adjudicator · an errata agent reviews the typed graph |
| 0045 | [A time mention is resolved against its document](0045-a-time-mention-is-resolved-against-its-document.md) | Accepted · cuts 1 and 2 built (#740): a document is dated from its own text, each mention is interpreted by the model and computed by code, upload time is used nowhere · cuts 3 and 4 (grades replace the confidence gate, re-resolution and the anchor queue) not built · a time expression is a mention with its words and place; the model returns shape, anchor, offset and granularity and code computes the interval; a document carries its own date, calendars and anchors across chunks, never its upload time; unresolved mentions wait for an anchor; timelines close on resolution grade instead of confidence |
diff --git a/docs/design/chat-and-mcp.md b/docs/design/chat-and-mcp.md
index d77f059d..2ae95d49 100644
--- a/docs/design/chat-and-mcp.md
+++ b/docs/design/chat-and-mcp.md
@@ -10,9 +10,11 @@ Records: [0042] (the loop), [0014] (MCP tools and scope), [0015] (`remember` and
policy is one `AgentHook` in `api/agent.rs`: `tool_choice: required` until a tool has run, then the
budget withdraws the tools and orders an answer; a malformed call is refused with the same message
as before; a tool's UI step goes to the stream; an empty turn is asked again once; a text-only first
-turn from an endpoint that ignored `required` is sent back once [0042]. A turn cannot end before a
-tool has run; `no_evidence_needed` is the exit for a greeting or "make it shorter"; `STALL_NUDGE`
-and `DONE` are gone [0042 d3]. The wire stays `LlmClient` behind `RigModel`: the read timeout,
+turn from an endpoint that ignored `required` is sent back once; a turn that writes its tool call as
+text is held back from the stream and sent back once, and a second one is an error [0042]. A turn
+cannot end before a tool has run; `no_evidence_needed` is the exit for a greeting or "make it
+shorter"; `STALL_NUDGE` and `DONE` are gone [0042 d3]. The wire stays `LlmClient` behind
+`RigModel`: the read timeout,
error bodies, the classification of a failure as out of credit, rate limited, unavailable or nothing
of the kind, and cache logging; earlier
entities become a `system` message right before the question; degradation to one-shot RAG happens