Skip to content

World's first open-source enterprise world model.

A reply that keeps repeating itself comes back cut instead of running to the token ceiling (#1306)

Quintin Botesfa6cd6cc13e3
diff --git a/crates/utopia-llm/src/lib.rs b/crates/utopia-llm/src/lib.rs
index c046e93a..c311f72f 100644
--- a/crates/utopia-llm/src/lib.rs
+++ b/crates/utopia-llm/src/lib.rs
@@ -386,6 +386,9 @@ pub struct Reply {
pub finish_reason: Option<String>,
/// 端点报的用量(最后一帧)。不报就是 `None`——账上不编数字
pub usage: Option<Usage>,
+ /// 回复在复读,是我们停了读(#1301),不是端点收的尾。与 `finish_reason` 分开记:
+ /// 记成 `length` 的话丢弃行会说「撞上了 token 上限」,看的人就去查那个上限了
+ pub repeating: bool,
}
/// 一次调用的 token 用量,端点自己报的
@@ -510,6 +513,28 @@ fn strip_reasoning(reply: &str) -> &str {
}
}
+/// 回复的尾巴是不是同一段在复读(#1301)。
+///
+/// 温度 0 的小模型会掉进复读:同一行陈述一遍遍写下去,直到撞上
+/// [`MAX_COMPLETION_TOKENS`]。本地 qwen3-8b 上约 3% 的抽取调用如此,每次占一个槽
+/// 一个多小时,吃掉六成的生成量。上限本身不能降(开推理的端点上它是思考预算),
+/// 所以看尾巴:同一段(至少 30 字节)一字不差连着出现五次,就不是在答题了——
+/// 真实的陈述行彼此不同。按字节比,多字节字符切在哪都不影响。
+///
+/// 只算**最短周期**也有 30 字节的尾巴:每条陈述以原文引文开头,一段 150 字节以上的
+/// 短周期串照样能拆成 30 字节以上的段——目录行 200 个点的引导线、一排二十个空
+/// `<td></td>`,不加这一条就可能被当成复读截断(看每 256 字节的检查落在哪),引文之后
+/// 的全丢了。真复读是整行在重复;短于 30 字节的行复读时这里不截(一夜 57 次截断里有
+/// 4 次是这种),宁可让它跑到上限,也不截断引文
+fn repeats_at_tail(text: &[u8]) -> bool {
+ const REPEATS: usize = 5;
+ (30..=600.min(text.len() / REPEATS)).any(|span| {
+ let tail = &text[text.len() - span * REPEATS..];
+ tail.chunks(span).all(|piece| piece == &tail[..span])
+ && !(1..30).any(|shift| tail[shift..] == tail[..tail.len() - shift])
+ })
+}
+
impl LlmClient {
pub fn new(base_url: &str, api_key: Option<&str>, model: &str) -> Self {
Self::with_timeouts(base_url, api_key, model, CONNECT_TIMEOUT, READ_TIMEOUT)
@@ -698,7 +723,8 @@ impl LlmClient {
let (mut saw_frame, mut ended) = (false, false);
let mut finish_reason: Option<String> = None;
let mut usage: Option<Usage> = None;
- while let Some(part) = bytes.next().await {
+ let (mut checked, mut repeating) = (0, false);
+ 'read: while let Some(part) = bytes.next().await {
let part = part.map_err(Unreachable)?;
// 网络片段可能断在 UTF-8 字符中间,等完整 SSE 帧到齐再解码。
buf.push(&part);
@@ -712,6 +738,24 @@ impl LlmClient {
&mut finish_reason,
&mut usage,
);
+ // 复读了就不再读,回复记成 `repeating`:抽取当它是半截的,前面完整的陈述照收,
+ // 不再等它写到 65,536 个 token。丢掉响应流,连接一关,端点那边也就停了。按帧看
+ // 而不是按网络片段看:一个片段可能装着整段复读。每多 256 字节看一次
+ if !ended && answer.len() >= checked + 256 {
+ checked = answer.len();
+ if repeats_at_tail(answer.as_bytes()) {
+ tracing::warn!(
+ model = %self.model,
+ call = %call_label(messages),
+ chars = answer.len(),
+ "回复在复读,按截断收下"
+ );
+ repeating = true;
+ ended = true;
+ buf = SseBuffer::default();
+ break 'read;
+ }
+ }
}
}
// **收尾那一帧也算**:末尾不跟空行的实现有的是,丢掉它就是丢掉答案的尾巴
@@ -749,6 +793,7 @@ impl LlmClient {
text: strip_reasoning(&answer).to_string(),
finish_reason,
usage,
+ repeating,
})
}
@@ -1998,6 +2043,55 @@ mod tests {
}
}
+ fn content_frame(text: &str) -> String {
+ format!(
+ "data: {}\n\n",
+ json!({"choices": [{"delta": {"content": text}}]})
+ )
+ }
+
+ async fn streamed_reply(sse: &str) -> Reply {
+ let (addr, server) = an_http_response("200 OK", "text/event-stream", sse).await;
+ let user = ChatMessage {
+ role: "user".into(),
+ content: "hi".into(),
+ };
+ let reply = client_at(addr)
+ .chat_at_streaming(&[user], Some(0.0))
+ .await
+ .unwrap();
+ server.await.unwrap();
+ reply
+ }
+
+ /// 复读的回复按截断收下(#1301):这条流没有收尾,没有护栏就是「流断在半路」
+ #[tokio::test]
+ async fn a_reply_that_keeps_repeating_comes_back_cut() {
+ let row = r#"[null,"a","is on","b",null,{},null,null],"#;
+ let sse =
+ content_frame(r#"{"e":[["a","thing",true]],"s":["#) + &content_frame(row).repeat(200);
+ let reply = streamed_reply(&sse).await;
+ assert!(reply.repeating);
+ assert!(!reply.hit_token_ceiling(), "the endpoint never said length");
+ assert!(
+ reply.text.len() < row.len() * 20,
+ "read {} bytes past the loop",
+ reply.text.len()
+ );
+ }
+
+ /// 引文里一条 200 个点的目录引导线不是复读:周期是一个点,不是一整行
+ #[tokio::test]
+ async fn a_quoted_dot_leader_comes_back_whole() {
+ let sse = content_frame(r#"{"e":[["Introduction","section",true]],"s":[["Introduction "#)
+ + &content_frame(&".".repeat(200))
+ + &content_frame(r#" 3","Introduction","is on page",null,"3",{},null,null]],"n":[]}"#)
+ + "data: [DONE]\n\n";
+ let reply = streamed_reply(&sse).await;
+ assert!(!reply.repeating);
+ assert!(reply.text.ends_with(r#""n":[]}"#), "{}", reply.text);
+ }
+
#[tokio::test]
async fn a_raw_stream_cut_before_its_finish_is_an_error() {
use futures_util::TryStreamExt;
diff --git a/crates/utopia-server/src/extraction_open.rs b/crates/utopia-server/src/extraction_open.rs
index 6dc1aee3..bac9f8ea 100644
--- a/crates/utopia-server/src/extraction_open.rs
+++ b/crates/utopia-server/src/extraction_open.rs
@@ -45,6 +45,14 @@ struct ChunkParsed {
name_vecs: HashMap<String, Vec<f32>>,
}
+/// 回复为什么是半截的:端点撞上了 token 上限(#760),或护栏看见它在复读、停了读(#1301)。
+/// 两种都当半截处理,丢弃行要说清是哪一种——复读的去查上限,是找错了地方
+#[derive(Debug, Clone, Copy)]
+enum Cut {
+ Ceiling,
+ Repeating,
+}
+
/// 一个分块的前半步:调模型、解析回复、算名字向量(#588 cut 1)。这半步不写图,
/// 所以分块之间可以不按顺序做——并发(cut 2)动的就是它;后半步 `apply_open_extraction`
/// 要按分块顺序。`pushed`(0054)的块本身就是契约,不调模型,直接解析。
@@ -72,8 +80,8 @@ async fn call_and_parse_one_chunk(
unextracted: &mut Vec<(i32, String)>,
) -> anyhow::Result<Option<ChunkParsed>> {
// 推送来的陈述:块就是契约,解析它而不是问模型(0054)。下面从解析起一步不变
- let (reply_text, cut_by_ceiling) = if pushed {
- (chunk.text.clone(), false)
+ let (reply_text, cut) = if pushed {
+ (chunk.text.clone(), None)
} else {
let (settings, client) = match (settings, client) {
(Some(s), Some(c)) => (s, c),
@@ -109,27 +117,35 @@ async fn call_and_parse_one_chunk(
}
};
tracing::debug!(%document_id, seq = chunk.seq, reply = %reply.text, "开放抽取的原始回复");
- // 端点说它是撞上 token 上限停的。解析器只看得见 JSON 少了尾巴,看不见
- // 少的原因,所以这句话得从回复里带过来(#760)
- let cut_by_ceiling = reply.hit_token_ceiling();
- (reply.text, cut_by_ceiling)
+ // 解析器只看得见 JSON 少了尾巴,看不见少的原因,所以原因得从回复里带过来(#760)
+ let cut = if reply.repeating {
+ Some(Cut::Repeating)
+ } else if reply.hit_token_ceiling() {
+ Some(Cut::Ceiling)
+ } else {
+ None
+ };
+ (reply.text, cut)
};
let mut extraction = match utopia_extract::open::parse_open_response(&reply_text) {
Ok(x) => x,
Err(e) => {
- tracing::warn!(%document_id, seq = chunk.seq, error = %e, hit_token_ceiling = cut_by_ceiling, "开放抽取回复解析失败,跳过该分块");
- // 一整块没进图,而原因不同:撞上上限是「答案太长」,改得动;
- // 别的解析失败是「回复不合结构」。两种都写成同一句话就分不开了
- let (why, detail) = if cut_by_ceiling {
- (
+ tracing::warn!(%document_id, seq = chunk.seq, error = %e, hit_token_ceiling = matches!(cut, Some(Cut::Ceiling)), repeating = matches!(cut, Some(Cut::Repeating)), "开放抽取回复解析失败,跳过该分块");
+ // 一整块没进图,而原因不同:撞上上限是「答案太长」,改得动;复读是模型卡在一行上;
+ // 别的解析失败是「回复不合结构」。写成同一句话就分不开了
+ let (why, detail) = match cut {
+ Some(Cut::Ceiling) => (
"response hit the token ceiling and could not be parsed; this chunk did not enter the graph",
format!("#{}: hit the token ceiling; {e}", chunk.seq),
- )
- } else {
- (
+ ),
+ Some(Cut::Repeating) => (
+ "response started repeating itself and could not be parsed; this chunk did not enter the graph",
+ format!("#{}: started repeating itself; {e}", chunk.seq),
+ ),
+ None => (
"response could not be parsed; this chunk did not enter the graph",
format!("#{}: {e}", chunk.seq),
- )
+ ),
};
drop_signal(
state,
@@ -164,12 +180,16 @@ async fn call_and_parse_one_chunk(
kb_id,
document_id,
reason::TRUNCATED_REPLY,
- if cut_by_ceiling {
- "the open reply hit the token ceiling; kept up to the last complete item"
- } else {
+ match cut {
+ Some(Cut::Ceiling) => {
+ "the open reply hit the token ceiling; kept up to the last complete item"
+ }
+ Some(Cut::Repeating) => {
+ "the reply started repeating itself; kept up to the last complete item"
+ }
// 回复正常结束而整体不合法:模型写坏了括号。逐条读出来的都收了,写坏的
// 条数在下面那条信号里
- "the open reply was not valid JSON as a whole; read item by item"
+ None => "the open reply was not valid JSON as a whole; read item by item",
},
None,
)
diff --git a/crates/utopia-server/src/phrase_alignment.rs b/crates/utopia-server/src/phrase_alignment.rs
index 04902722..81fb8b45 100644
--- a/crates/utopia-server/src/phrase_alignment.rs
+++ b/crates/utopia-server/src/phrase_alignment.rs
@@ -964,6 +964,7 @@ async fn align_phrases_locked(
item_ids = ?items.iter().map(|item| item.id).collect::<Vec<_>>(),
malformed,
finish_reason = ?reply.finish_reason,
+ repeating = reply.repeating,
chars = reply.text.chars().count(),
reply = %snippet(&reply.text),
"短语对齐回复读不出一条,这一批留到下次"
diff --git a/crates/utopia-server/src/type_alignment.rs b/crates/utopia-server/src/type_alignment.rs
index 4d0da08a..7052acff 100644
--- a/crates/utopia-server/src/type_alignment.rs
+++ b/crates/utopia-server/src/type_alignment.rs
@@ -319,6 +319,7 @@ async fn align_types_locked(
items = items.len(),
malformed,
finish_reason = ?reply.finish_reason,
+ repeating = reply.repeating,
chars = reply.text.chars().count(),
reply = %snippet(&reply.text),
"类别词对齐回复读不出一项,这一批留到下次"
diff --git a/docs/design/extraction.md b/docs/design/extraction.md
index 1371e02e..0928cb38 100644
--- a/docs/design/extraction.md
+++ b/docs/design/extraction.md
@@ -105,6 +105,20 @@ token ceiling* instead of only *could not be parsed* — and the two causes stay
is too long is a number to change, a reply that does not fit the shape is not. Across all bases in
the e2e database `truncated_reply` has fired 10 times, and those rows keep every complete item.
+**Where a looping answer stops** [#1301]. At temperature 0 a small local model sometimes falls into
+a loop and writes the same row until the ceiling: on qwen3-8b about 3% of extraction calls, each
+holding a slot for over an hour and together about 60% of the tokens generated. The ceiling stays
+where it is, for the reasons above. Instead the streamed reply is read until the same span of 30 to
+600 bytes fills its tail five times in a row, and only if that run's shortest period is also at
+least 30 bytes: every statement starts with a verbatim quote, and a quoted dot leader or a row of
+empty table cells repeats with a short period. A loop of rows shorter than 30 bytes is left to run
+to the ceiling (4 of 57 cuts in one night of a local deployment) rather than risk cutting a quote.
+The reply comes back marked `repeating`, apart from `finish_reason`, so it is never counted as the
+ceiling. Its drop row reads *started repeating itself* under the same `truncated_reply` code
+(`chunk_unextracted` when nothing before the loop parses), and it keeps every complete item before
+the loop. The guard sits in the streaming call, so the alignment, errata, implication, time and
+ontology calls that stream get it too.
+
**Server checks, each a drop reason in `extraction_drops`.** Every quote must occur in the chunk
(`quote_not_in_chunk`); every name in its quote (`name_not_in_text`); a time mention only when its
words occur in the statement's own quote (`time_not_in_quote`), because the model otherwise attaches