World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A reply that keeps repeating itself comes back cut instead of running to the token ceiling (#1306)
Quintin Botesfa6cd6cc13e3
diff --git a/crates/utopia-llm/src/lib.rs b/crates/utopia-llm/src/lib.rsindex c046e93a..c311f72f 100644--- a/crates/utopia-llm/src/lib.rs+++ b/crates/utopia-llm/src/lib.rs@@ -386,6 +386,9 @@ pub struct Reply {pub finish_reason: Option<String>,/// 端点报的用量(最后一帧)。不报就是 `None`——账上不编数字pub usage: Option<Usage>,+ /// 回复在复读,是我们停了读(#1301),不是端点收的尾。与 `finish_reason` 分开记:+ /// 记成 `length` 的话丢弃行会说「撞上了 token 上限」,看的人就去查那个上限了+ pub repeating: bool,}/// 一次调用的 token 用量,端点自己报的@@ -510,6 +513,28 @@ fn strip_reasoning(reply: &str) -> &str {}}+/// 回复的尾巴是不是同一段在复读(#1301)。+///+/// 温度 0 的小模型会掉进复读:同一行陈述一遍遍写下去,直到撞上+/// [`MAX_COMPLETION_TOKENS`]。本地 qwen3-8b 上约 3% 的抽取调用如此,每次占一个槽+/// 一个多小时,吃掉六成的生成量。上限本身不能降(开推理的端点上它是思考预算),+/// 所以看尾巴:同一段(至少 30 字节)一字不差连着出现五次,就不是在答题了——+/// 真实的陈述行彼此不同。按字节比,多字节字符切在哪都不影响。+///+/// 只算**最短周期**也有 30 字节的尾巴:每条陈述以原文引文开头,一段 150 字节以上的+/// 短周期串照样能拆成 30 字节以上的段——目录行 200 个点的引导线、一排二十个空+/// `<td></td>`,不加这一条就可能被当成复读截断(看每 256 字节的检查落在哪),引文之后+/// 的全丢了。真复读是整行在重复;短于 30 字节的行复读时这里不截(一夜 57 次截断里有+/// 4 次是这种),宁可让它跑到上限,也不截断引文+fn repeats_at_tail(text: &[u8]) -> bool {+ const REPEATS: usize = 5;+ (30..=600.min(text.len() / REPEATS)).any(|span| {+ let tail = &text[text.len() - span * REPEATS..];+ tail.chunks(span).all(|piece| piece == &tail[..span])+ && !(1..30).any(|shift| tail[shift..] == tail[..tail.len() - shift])+ })+}+impl LlmClient {pub fn new(base_url: &str, api_key: Option<&str>, model: &str) -> Self {Self::with_timeouts(base_url, api_key, model, CONNECT_TIMEOUT, READ_TIMEOUT)@@ -698,7 +723,8 @@ impl LlmClient {let (mut saw_frame, mut ended) = (false, false);let mut finish_reason: Option<String> = None;let mut usage: Option<Usage> = None;- while let Some(part) = bytes.next().await {+ let (mut checked, mut repeating) = (0, false);+ 'read: while let Some(part) = bytes.next().await {let part = part.map_err(Unreachable)?;// 网络片段可能断在 UTF-8 字符中间,等完整 SSE 帧到齐再解码。buf.push(&part);@@ -712,6 +738,24 @@ impl LlmClient {&mut finish_reason,&mut usage,);+ // 复读了就不再读,回复记成 `repeating`:抽取当它是半截的,前面完整的陈述照收,+ // 不再等它写到 65,536 个 token。丢掉响应流,连接一关,端点那边也就停了。按帧看+ // 而不是按网络片段看:一个片段可能装着整段复读。每多 256 字节看一次+ if !ended && answer.len() >= checked + 256 {+ checked = answer.len();+ if repeats_at_tail(answer.as_bytes()) {+ tracing::warn!(+ model = %self.model,+ call = %call_label(messages),+ chars = answer.len(),+ "回复在复读,按截断收下"+ );+ repeating = true;+ ended = true;+ buf = SseBuffer::default();+ break 'read;+ }+ }}}// **收尾那一帧也算**:末尾不跟空行的实现有的是,丢掉它就是丢掉答案的尾巴@@ -749,6 +793,7 @@ impl LlmClient {text: strip_reasoning(&answer).to_string(),finish_reason,usage,+ repeating,})}@@ -1998,6 +2043,55 @@ mod tests {}}+ fn content_frame(text: &str) -> String {+ format!(+ "data: {}\n\n",+ json!({"choices": [{"delta": {"content": text}}]})+ )+ }++ async fn streamed_reply(sse: &str) -> Reply {+ let (addr, server) = an_http_response("200 OK", "text/event-stream", sse).await;+ let user = ChatMessage {+ role: "user".into(),+ content: "hi".into(),+ };+ let reply = client_at(addr)+ .chat_at_streaming(&[user], Some(0.0))+ .await+ .unwrap();+ server.await.unwrap();+ reply+ }++ /// 复读的回复按截断收下(#1301):这条流没有收尾,没有护栏就是「流断在半路」+ #[tokio::test]+ async fn a_reply_that_keeps_repeating_comes_back_cut() {+ let row = r#"[null,"a","is on","b",null,{},null,null],"#;+ let sse =+ content_frame(r#"{"e":[["a","thing",true]],"s":["#) + &content_frame(row).repeat(200);+ let reply = streamed_reply(&sse).await;+ assert!(reply.repeating);+ assert!(!reply.hit_token_ceiling(), "the endpoint never said length");+ assert!(+ reply.text.len() < row.len() * 20,+ "read {} bytes past the loop",+ reply.text.len()+ );+ }++ /// 引文里一条 200 个点的目录引导线不是复读:周期是一个点,不是一整行+ #[tokio::test]+ async fn a_quoted_dot_leader_comes_back_whole() {+ let sse = content_frame(r#"{"e":[["Introduction","section",true]],"s":[["Introduction "#)+ + &content_frame(&".".repeat(200))+ + &content_frame(r#" 3","Introduction","is on page",null,"3",{},null,null]],"n":[]}"#)+ + "data: [DONE]\n\n";+ let reply = streamed_reply(&sse).await;+ assert!(!reply.repeating);+ assert!(reply.text.ends_with(r#""n":[]}"#), "{}", reply.text);+ }+#[tokio::test]async fn a_raw_stream_cut_before_its_finish_is_an_error() {use futures_util::TryStreamExt;diff --git a/crates/utopia-server/src/extraction_open.rs b/crates/utopia-server/src/extraction_open.rsindex 6dc1aee3..bac9f8ea 100644--- a/crates/utopia-server/src/extraction_open.rs+++ b/crates/utopia-server/src/extraction_open.rs@@ -45,6 +45,14 @@ struct ChunkParsed {name_vecs: HashMap<String, Vec<f32>>,}+/// 回复为什么是半截的:端点撞上了 token 上限(#760),或护栏看见它在复读、停了读(#1301)。+/// 两种都当半截处理,丢弃行要说清是哪一种——复读的去查上限,是找错了地方+#[derive(Debug, Clone, Copy)]+enum Cut {+ Ceiling,+ Repeating,+}+/// 一个分块的前半步:调模型、解析回复、算名字向量(#588 cut 1)。这半步不写图,/// 所以分块之间可以不按顺序做——并发(cut 2)动的就是它;后半步 `apply_open_extraction`/// 要按分块顺序。`pushed`(0054)的块本身就是契约,不调模型,直接解析。@@ -72,8 +80,8 @@ async fn call_and_parse_one_chunk(unextracted: &mut Vec<(i32, String)>,) -> anyhow::Result<Option<ChunkParsed>> {// 推送来的陈述:块就是契约,解析它而不是问模型(0054)。下面从解析起一步不变- let (reply_text, cut_by_ceiling) = if pushed {- (chunk.text.clone(), false)+ let (reply_text, cut) = if pushed {+ (chunk.text.clone(), None)} else {let (settings, client) = match (settings, client) {(Some(s), Some(c)) => (s, c),@@ -109,27 +117,35 @@ async fn call_and_parse_one_chunk(}};tracing::debug!(%document_id, seq = chunk.seq, reply = %reply.text, "开放抽取的原始回复");- // 端点说它是撞上 token 上限停的。解析器只看得见 JSON 少了尾巴,看不见- // 少的原因,所以这句话得从回复里带过来(#760)- let cut_by_ceiling = reply.hit_token_ceiling();- (reply.text, cut_by_ceiling)+ // 解析器只看得见 JSON 少了尾巴,看不见少的原因,所以原因得从回复里带过来(#760)+ let cut = if reply.repeating {+ Some(Cut::Repeating)+ } else if reply.hit_token_ceiling() {+ Some(Cut::Ceiling)+ } else {+ None+ };+ (reply.text, cut)};let mut extraction = match utopia_extract::open::parse_open_response(&reply_text) {Ok(x) => x,Err(e) => {- tracing::warn!(%document_id, seq = chunk.seq, error = %e, hit_token_ceiling = cut_by_ceiling, "开放抽取回复解析失败,跳过该分块");- // 一整块没进图,而原因不同:撞上上限是「答案太长」,改得动;- // 别的解析失败是「回复不合结构」。两种都写成同一句话就分不开了- let (why, detail) = if cut_by_ceiling {- (+ tracing::warn!(%document_id, seq = chunk.seq, error = %e, hit_token_ceiling = matches!(cut, Some(Cut::Ceiling)), repeating = matches!(cut, Some(Cut::Repeating)), "开放抽取回复解析失败,跳过该分块");+ // 一整块没进图,而原因不同:撞上上限是「答案太长」,改得动;复读是模型卡在一行上;+ // 别的解析失败是「回复不合结构」。写成同一句话就分不开了+ let (why, detail) = match cut {+ Some(Cut::Ceiling) => ("response hit the token ceiling and could not be parsed; this chunk did not enter the graph",format!("#{}: hit the token ceiling; {e}", chunk.seq),- )- } else {- (+ ),+ Some(Cut::Repeating) => (+ "response started repeating itself and could not be parsed; this chunk did not enter the graph",+ format!("#{}: started repeating itself; {e}", chunk.seq),+ ),+ None => ("response could not be parsed; this chunk did not enter the graph",format!("#{}: {e}", chunk.seq),- )+ ),};drop_signal(state,@@ -164,12 +180,16 @@ async fn call_and_parse_one_chunk(kb_id,document_id,reason::TRUNCATED_REPLY,- if cut_by_ceiling {- "the open reply hit the token ceiling; kept up to the last complete item"- } else {+ match cut {+ Some(Cut::Ceiling) => {+ "the open reply hit the token ceiling; kept up to the last complete item"+ }+ Some(Cut::Repeating) => {+ "the reply started repeating itself; kept up to the last complete item"+ }// 回复正常结束而整体不合法:模型写坏了括号。逐条读出来的都收了,写坏的// 条数在下面那条信号里- "the open reply was not valid JSON as a whole; read item by item"+ None => "the open reply was not valid JSON as a whole; read item by item",},None,)diff --git a/crates/utopia-server/src/phrase_alignment.rs b/crates/utopia-server/src/phrase_alignment.rsindex 04902722..81fb8b45 100644--- a/crates/utopia-server/src/phrase_alignment.rs+++ b/crates/utopia-server/src/phrase_alignment.rs@@ -964,6 +964,7 @@ async fn align_phrases_locked(item_ids = ?items.iter().map(|item| item.id).collect::<Vec<_>>(),malformed,finish_reason = ?reply.finish_reason,+ repeating = reply.repeating,chars = reply.text.chars().count(),reply = %snippet(&reply.text),"短语对齐回复读不出一条,这一批留到下次"diff --git a/crates/utopia-server/src/type_alignment.rs b/crates/utopia-server/src/type_alignment.rsindex 4d0da08a..7052acff 100644--- a/crates/utopia-server/src/type_alignment.rs+++ b/crates/utopia-server/src/type_alignment.rs@@ -319,6 +319,7 @@ async fn align_types_locked(items = items.len(),malformed,finish_reason = ?reply.finish_reason,+ repeating = reply.repeating,chars = reply.text.chars().count(),reply = %snippet(&reply.text),"类别词对齐回复读不出一项,这一批留到下次"diff --git a/docs/design/extraction.md b/docs/design/extraction.mdindex 1371e02e..0928cb38 100644--- a/docs/design/extraction.md+++ b/docs/design/extraction.md@@ -105,6 +105,20 @@ token ceiling* instead of only *could not be parsed* — and the two causes stayis too long is a number to change, a reply that does not fit the shape is not. Across all bases inthe e2e database `truncated_reply` has fired 10 times, and those rows keep every complete item.+**Where a looping answer stops** [#1301]. At temperature 0 a small local model sometimes falls into+a loop and writes the same row until the ceiling: on qwen3-8b about 3% of extraction calls, each+holding a slot for over an hour and together about 60% of the tokens generated. The ceiling stays+where it is, for the reasons above. Instead the streamed reply is read until the same span of 30 to+600 bytes fills its tail five times in a row, and only if that run's shortest period is also at+least 30 bytes: every statement starts with a verbatim quote, and a quoted dot leader or a row of+empty table cells repeats with a short period. A loop of rows shorter than 30 bytes is left to run+to the ceiling (4 of 57 cuts in one night of a local deployment) rather than risk cutting a quote.+The reply comes back marked `repeating`, apart from `finish_reason`, so it is never counted as the+ceiling. Its drop row reads *started repeating itself* under the same `truncated_reply` code+(`chunk_unextracted` when nothing before the loop parses), and it keeps every complete item before+the loop. The guard sits in the streaming call, so the alignment, errata, implication, time and+ontology calls that stream get it too.+**Server checks, each a drop reason in `extraction_drops`.** Every quote must occur in the chunk(`quote_not_in_chunk`); every name in its quote (`name_not_in_text`); a time mention only when itswords occur in the statement's own quote (`time_not_in_quote`), because the model otherwise attaches