World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A chunk reads the opening of its document, and a date keeps the contract format (#680)
WaylandYang785ce6a348c7
diff --git a/crates/utopia-extract/src/lib.rs b/crates/utopia-extract/src/lib.rsindex f21707a4..5d1ff432 100644--- a/crates/utopia-extract/src/lib.rs+++ b/crates/utopia-extract/src/lib.rs@@ -8,7 +8,7 @@ use utopia_llm::ChatMessage;pub mod governor;pub mod normalize;-pub use normalize::{normalize_facts, Normalization};+pub use normalize::{drop_quotes_from_opening, normalize_facts, Normalization};#[derive(Debug, Deserialize)]pub struct Extraction {@@ -140,6 +140,34 @@ pub fn build_messages(// 第一块为空——那时还没有"前面"known: &[KnownEntity],chunk_text: &str,+) -> Vec<ChatMessage> {+ build_messages_with_opening(+ types, relations, attributes, doc_time, filename, known, None, chunk_text,+ )+}++/// 文件开头进提示词的字符预算。一份补充协议的标题、生效日、当事方和「修订的是哪份+/// 协议」通常在头一千字符里;新闻稿的电头与导语也是+pub const OPENING_BUDGET_CHARS: usize = 1500;++/// 同 [`build_messages`],另带**这份文件的开头**(第一块的原文),给第二块往后用。+///+/// **一块是孤立抽取的,它看不见自己属于什么。** 补充协议把截止日写在第三块的表格里,+/// 那一块只说「Article 13 的日期延至……」:改的是哪份租约、从哪天起改,都写在第一块。+/// 模型拿不到,就只能把「Phase 2 Exercise Deadline」本身当主语(服务端按主语未声明丢掉),+/// 或者抽出一个没有起点的日期(时态引擎没法据此关闭旧值)——Blackbaud 总部租约链+/// 上五次改期丢了两次,抽到的三次一次都没关上旧值。开头只作背景,不从里面抽事实:+/// 它自己那一块会抽,重复抽只会多出重复的事实+#[allow(clippy::too_many_arguments)]+pub fn build_messages_with_opening(+ types: &[(String, String, String)],+ relations: &[PromptRelation],+ attributes: &[String],+ doc_time: Option<&str>,+ filename: &str,+ known: &[KnownEntity],+ opening: Option<&str>,+ chunk_text: &str,) -> Vec<ChatMessage> {// **有描述时不送 label**。label 是给人看的显示名,而且它跟界面无关、// 跟这个库的语料语言走——中文库里 person 的 label 是"人物"。@@ -265,7 +293,9 @@ pub fn build_messages(} else {"\n10. Attribute facts carry \"value\" (no \"object\"): number = the figure **as the text writes it, magnitude and currency included** \n (\"86亿元\", \"$5 billion\", \"4,300 人\") — never reduce it to a bare number, the server converts; date = \"YYYY[-MM[-DD]]\" (a zoned clock time only when the text gives one); bool = true/false; \text = a short string. Only attach an attribute to a subject of its listed class. \- valid_from = when this value took effect, if the text says so."+ valid_from = when this value took effect, if the text or the opening of the document says so. \+ A document that changes a value set earlier — amends, extends or replaces it — makes the new value \+ hold from the date the change takes effect, which is the document's own effective date unless the text gives another.".to_string()};let system = format!(@@ -296,7 +326,10 @@ pub fn build_messages(list each entity once. Text introduces a full name and then shortens it — \\"星云科技上海研究院\" becomes \"上海研究院\", \"Nebula Technologies Inc.\" becomes \\"Nebula\" — and both forms mean one entity, listed once under the fuller form. \- Two names are two entities only when the text is talking about two things.\n\+ Two names are two entities only when the text is talking about two things. \+ A name identifies the thing; it is not a description of its history. When the text \+ names something and then describes what happened to it, the name ends where the \+ description begins.\n\1b. Every other name the text gives an entity goes into \"names\", once per name: the \shortened form it introduces or uses (\"上海研究院\" for \"星云科技上海研究院\"), a \former name, the name in another language. \"ref\" is the entity's local_id or its \@@ -328,7 +361,7 @@ pub fn build_messages(of values that hold in that period.\n\{temporal_note}\n\4. {time_ctx}\n\- 5. quote must be a contiguous excerpt from the source text; every fact needs one.\n\+ 5. quote must be a contiguous excerpt from the Text block; never quote the opening of the document. Every fact needs one.\n\6. confidence in 0~1: 0.9 explicitly stated, 0.7 inferred, 0.5 uncertain.\n\7. If nothing can be extracted, output {{\"entities\":[],\"facts\":[]}}.\n\8. If no listed relation fits, do not force the nearest one — write the predicate the \@@ -341,10 +374,14 @@ pub fn build_messages(\"quote\":\"...\"}}. Name the predicate after the text when no listed attribute \fits — \"purchase_price\", \"job_title\", \"generation_capacity\", \"record_date\". \Attach it to the entity the text attaches it to, and keep the literal as written, \- units and all. **A stated figure left out is the loss that costs most**: the reader \+ units and all — except a date, which is always written in the format of rule 3 \+ (\"June 23, 2020\" is \"2020-06-23\"): the server keeps a date only in that \+ format, and a date written any other way is lost. A deadline or a period stated \+ relative to an event, with no calendar date, is not a date. \+ **A stated figure left out is the loss that costs most**: the reader \came for those numbers, and no later step can recover one that was never written \down.\n\- 8c. A listed relation followed by {{…}} can carry those **qualifiers on the edge**: when the same sentence gives both the other entity and a figure for it — an amount, a stake, a price, a share count — write the relation with its \"object\" and put the figure in \"qualifiers\" keyed exactly as listed, **as written in the text, currency and all** (\"€30 million\", \"15亿元人民币\", never a bare number): {{\"subject\":\"Vega Capital\",\"predicate\":\"invested_in\",\"object\":\"Northwind\", \"qualifiers\":{{\"amount\":\"$5 billion\"}},…}}. Never invent a key that is not listed for that relation, and never drop the figure to keep the edge — a relation without its amount is half the sentence. A relation you name after the text (rule 8) carries its figure the same way — keyed by the listed attribute that fits it, or by the plainest word for it (\"amount\", \"stake\", \"price\") when none does.+ 8c. A listed relation followed by {{…}} can carry those **qualifiers on the edge**: when the same sentence gives both the other entity and a figure for it — an amount, a stake, a price, a share count — write the relation with its \"object\" and put the figure in \"qualifiers\" keyed exactly as listed, **as written in the text, currency and all** (\"€30 million\", \"15亿元人民币\", never a bare number) — except a date, which takes the format of rule 3: {{\"subject\":\"Vega Capital\",\"predicate\":\"invested_in\",\"object\":\"Northwind\", \"qualifiers\":{{\"amount\":\"$5 billion\"}},…}}. Never invent a key that is not listed for that relation, and never drop the figure to keep the edge — a relation without its amount is half the sentence. A relation you name after the text (rule 8) carries its figure the same way — keyed by the listed attribute that fits it, or by the plainest word for it (\"amount\", \"stake\", \"price\") when none does.8b. A **listed** relation also takes \"value\" when what the text gives is a \string rather than another entity — a job title, a designation, a ticker, a \model number. Never invent an entity for a string. And when the text introduces \@@ -367,6 +404,10 @@ pub fn build_messages(Copy them; never paraphrase. When the words that do the thing are a description \rather than a name — \"former X employees\", \"companies using X\" — the span \is that description, whatever you wrote in subject.\n\+ 8e. An obligation, a deadline or a right belongs to the agreement, law or decision \+ that imposes it, even when it concerns another agreement or thing. A lease that \+ sets the last day to sign a second lease gives that deadline to the first lease; \+ the second lease is only what the deadline is about.\n\9. The same holds for entity types: if none of the listed types fits, write the type \the text implies, in snake_case (e.g. \"model\", \"technology\"). Do not fall back \to a broad listed type such as \"thing\" or \"creative_work\" merely because \@@ -381,7 +422,8 @@ pub fn build_messages(// 已知实体紧挨着正文:服从性靠位置,理由见 known_block 的注释let user = format!(- "Source file: \"{filename}\"\n{}\nText:\n{chunk_text}",+ "Source file: \"{filename}\"\n{}{}\nText:\n{chunk_text}",+ opening_block(opening),known_block(known));@@ -397,6 +439,29 @@ pub fn build_messages(]}+/// 文件开头排版成提示词里的一段。开头为空(或只有空白)时返回空串;+/// 「这一块就是开头本身」由调用方判断,那时它传 `None`。+///+/// 按字符截:不会截断一个字符,但会截在词中间——英文的最后一个词可能只剩半个+fn opening_block(opening: Option<&str>) -> String {+ let Some(text) = opening.map(str::trim).filter(|t| !t.is_empty()) else {+ return String::new();+ };+ let cut: String = text.chars().take(OPENING_BUDGET_CHARS).collect();+ let more = if cut.chars().count() < text.chars().count() {+ " …"+ } else {+ ""+ };+ format!(+ "\nOpening of this document, for context only (do not extract facts from it; they are \+ extracted from that part separately). Use it to know what the text below belongs to — \+ which agreement, company or event it concerns, who the parties are, and the date it \+ takes effect — so that facts in the text below attach to the right entity and carry \+ the right dates:\n\"\"\"\n{cut}{more}\n\"\"\"\n"+ )+}+/// 清单里给关系带的标记:事件 `[event]`、恒常 `[eternal]`;状态不标。/// 认不出的值当状态——数据库的 CHECK 只放这三个进来,这里不再报错fn temporal_mark(temporal: &str) -> Option<&'static str> {@@ -1214,6 +1279,63 @@ mod prompt_shape_tests {);}+ #[test]+ fn an_obligation_belongs_to_the_agreement_that_imposes_it() {+ // 主租约里写着「签二期租约的截止日」,模型时而把截止日挂到二期租约上:+ // 主租约的时间线上就少了这次改期(#681 §3)+ let msgs = build_messages(&[], &[], &[], None, "a.txt", &[], "text");+ assert!(msgs[0]+ .content+ .contains("belongs to the agreement, law or decision that imposes it"));+ }++ #[test]+ fn a_literal_keeps_its_units_but_a_date_takes_the_contract_format() {+ // 8a 从前说「字面值按原文写」并把日期列在字面值里,而规则 3 与属性规则要求+ // YYYY-MM-DD:两条互相打架,模型写出「June 23, 2020」,服务端按格式不合整条丢掉。+ // Blackbaud 总部租约链上各轮累计丢了二十多次+ let msgs = build_messages(&[], &[], &[], None, "a.txt", &[], "text");+ let system = &msgs[0].content;+ assert!(system.contains("except a date, which is always written in the format of rule 3"));+ }++ #[test]+ fn a_later_chunk_reads_the_opening_of_its_document() {+ let opening = "FIFTH AMENDMENT TO LEASE AGREEMENT entered into as of February 18, 2020";+ let msgs = build_messages_with_opening(+ &[],+ &[],+ &[],+ None,+ "a.html",+ &[],+ Some(opening),+ "The Existing Dates are extended to March 17, 2020.",+ );+ let user = &msgs[1].content;+ let at_opening = user.find(opening).expect("opening is in the user message");+ let at_text = user+ .find("The Existing Dates")+ .expect("text is in the user message");+ assert!(+ at_opening < at_text,+ "the opening comes before the text it frames"+ );+ // 没有开头时,提示词与从前一字不差+ let plain = build_messages(&[], &[], &[], None, "a.html", &[], "t");+ let framed = build_messages_with_opening(&[], &[], &[], None, "a.html", &[], None, "t");+ assert_eq!(plain[1].content, framed[1].content);+ }++ #[test]+ fn a_long_opening_is_cut_on_a_character_boundary() {+ let long = "租".repeat(OPENING_BUDGET_CHARS + 10);+ let block = opening_block(Some(&long));+ assert_eq!(block.matches('租').count(), OPENING_BUDGET_CHARS);+ assert!(block.contains(" …"));+ assert_eq!(opening_block(Some(" ")), "");+ }+/// 已知实体必须落在 **user** 消息里、紧挨着正文。////// 理由是服从性不是缓存:抽象规则打不过挨着它的具体块。清单放进 system 的diff --git a/crates/utopia-extract/src/normalize.rs b/crates/utopia-extract/src/normalize.rsindex f0a7aa11..399f374e 100644--- a/crates/utopia-extract/src/normalize.rs+++ b/crates/utopia-extract/src/normalize.rs@@ -9,7 +9,7 @@//! 每条规则做了什么都返回给服务端记进丢弃表:违约多常见、出在哪个模型,量得出来,//! 契约该怎么改看数说话。-use crate::{parse_time, ExtractedFact, Extraction};+use crate::{parse_time, ExtractedEntity, ExtractedFact, Extraction};use std::collections::HashSet;/// 形状检查做了什么;服务端按条记信号@@ -49,6 +49,9 @@ pub enum Normalization {},/// 上面几条去掉事实之后,没有任何事实再引用的声明:不建,否则就是一个孤点OrphanDeclaration { name: String },+ /// 引文抄自提示词里附的文件开头,而不是这一块:开头只作背景,它自己那一块会抽到。+ /// 照落的话,证据挂在这一块上,引的却是第一块的话——引错了出处+ QuoteFromOpening { predicate: String, quote: String },}/// 比对用的形态:空白折叠、小写@@ -230,6 +233,84 @@ fn is_contract_time(s: &str) -> bool {parse_time(s.trim()).is_some()}+/// 引文不在这一块、却在附上的文件开头里的事实:丢掉,连同因此没人引用的声明。+///+/// 只比「在不在」(空白、大小写不论),不比像不像:两边都找不到的引文不归这里管——+/// 那是模型改写了原文,与开头无关+pub fn drop_quotes_from_opening(+ x: &mut Extraction,+ chunk_text: &str,+ opening: &str,+) -> Vec<Normalization> {+ let (chunk, opening) = (norm(chunk_text), norm(opening));+ if opening.is_empty() {+ return Vec::new();+ }+ let before = referenced_names(&x.entities, &x.facts);+ let mut out = Vec::new();+ x.facts.retain(|f| {+ let q = norm(f.quote.as_deref().unwrap_or(""));+ let from_opening = !q.is_empty() && !chunk.contains(&q) && opening.contains(&q);+ if from_opening {+ out.push(Normalization::QuoteFromOpening {+ predicate: f.predicate.clone(),+ quote: f.quote.clone().unwrap_or_default(),+ });+ }+ !from_opening+ });+ if out.is_empty() {+ return out;+ }+ let mut after = referenced_names(&x.entities, &x.facts);+ // 与 normalize_facts 同一条:模型给它报了别的名字的声明不算孤点(0041)+ after.extend(x.names.iter().filter_map(|n| {+ x.entities+ .iter()+ .find(|e| e.local_id.as_deref().map(str::trim) == Some(n.entity_ref.trim()))+ .map(|e| e.name.trim().to_lowercase())+ }));+ let mut orphans = Vec::new();+ x.entities.retain(|e| {+ let n = e.name.trim().to_lowercase();+ let orphan = before.contains(&n) && !after.contains(&n);+ if orphan {+ orphans.push(e.name.trim().to_string());+ }+ !orphan+ });+ out.extend(+ orphans+ .into_iter()+ .map(|name| Normalization::OrphanDeclaration { name }),+ );+ out+}++/// 事实两侧绑到的声明名(小写):有句柄按句柄,没有按写出来的名字+fn referenced_names(entities: &[ExtractedEntity], facts: &[ExtractedFact]) -> HashSet<String> {+ let handle_name = |h: Option<&String>| {+ h.and_then(|h| {+ entities+ .iter()+ .find(|e| e.local_id.as_deref().map(str::trim) == Some(h.trim()))+ .map(|e| e.name.trim().to_string())+ })+ };+ facts+ .iter()+ .flat_map(|f| {+ [+ handle_name(f.subject_ref.as_ref()).or_else(|| Some(f.subject.trim().to_string())),+ handle_name(f.object_ref.as_ref())+ .or_else(|| f.object.as_deref().map(|o| o.trim().to_string())),+ ]+ })+ .flatten()+ .map(|n| n.to_lowercase())+ .collect()+}+pub fn normalize_facts(x: &mut Extraction) -> Vec<Normalization> {let mut out = Vec::new();let entities = std::mem::take(&mut x.entities);@@ -243,22 +324,7 @@ pub fn normalize_facts(x: &mut Extraction) -> Vec<Normalization> {.map(|e| e.name.trim().to_string())})};- let referenced = |facts: &[ExtractedFact]| -> HashSet<String> {- facts- .iter()- .flat_map(|f| {- [- handle_name(f.subject_ref.as_ref())- .or_else(|| Some(f.subject.trim().to_string())),- handle_name(f.object_ref.as_ref())- .or_else(|| f.object.as_deref().map(|o| o.trim().to_string())),- ]- })- .flatten()- .map(|n| n.to_lowercase())- .collect()- };- let before = referenced(&x.facts);+ let before = referenced_names(&entities, &x.facts);let mut facts: Vec<ExtractedFact> = Vec::with_capacity(x.facts.len());for mut f in std::mem::take(&mut x.facts) {@@ -423,7 +489,7 @@ pub fn normalize_facts(x: &mut Extraction) -> Vec<Normalization> {// ---- 被上面几条弄成孤点的声明 ----// 只去掉「原来有事实引用、现在没有了」的:模型一开始就只声明不连边的,不归这里管- let mut after = referenced(&x.facts);+ let mut after = referenced_names(&entities, &x.facts);// 模型给它报了别的名字的声明也不算孤点:那些名字要绑在它身上(0041)after.extend(x.names@@ -491,6 +557,63 @@ mod tests {.map(|(k, v)| (k.to_string(), serde_json::Value::String(v.to_string()))).collect()}+ /// 第五份补充协议的第三块:模型从附上的开头里抄了「as of February 18, 2020」那句当引文。+ /// 那条事实丢掉,只为它而声明的实体跟着不建;引文在这一块里的照落+ #[test]+ fn a_fact_quoting_the_opening_is_dropped_with_its_orphan() {+ let opening = "THIS FIFTH AMENDMENT TO LEASE AGREEMENT is entered into as of February 18, 2020 by and between HPBB1, LLC and BLACKBAUD, INC.";+ let chunk = "The Phase 2 Exercise Deadline is hereby extended to March 17, 2020.";+ let mut x = Extraction {+ entities: vec![entity("e1", "Lease"), entity("e2", "HPBB1, LLC")],+ facts: vec![+ fact(+ "Lease",+ "landlord",+ Some("HPBB1, LLC"),+ "entered into as of February 18, 2020 by and between HPBB1, LLC",+ ),+ fact(+ "Lease",+ "expansion_option_deadline",+ None,+ "the phase 2 exercise deadline is hereby extended to March 17, 2020",+ ),+ ],+ names: Vec::new(),+ skipped_entities: 0,+ skipped_facts: 0,+ truncated: false,+ };+ let n = drop_quotes_from_opening(&mut x, chunk, opening);+ let kept: Vec<&str> = x.facts.iter().map(|f| f.predicate.as_str()).collect();+ assert_eq!(+ kept,+ ["expansion_option_deadline"],+ "大小写与空白不论,引文在这一块里的留下"+ );+ let names: Vec<&str> = x.entities.iter().map(|e| e.name.as_str()).collect();+ assert_eq!(names, ["Lease"], "只为被丢那条声明的 HPBB1 不建");+ assert!(n+ .iter()+ .any(|v| matches!(v, Normalization::QuoteFromOpening { .. })));+ assert!(n+ .iter()+ .any(|v| matches!(v, Normalization::OrphanDeclaration { .. })));++ // 两边都找不到的引文不归这里管;没有开头时什么都不做+ let mut y = Extraction {+ entities: vec![entity("e1", "Lease")],+ facts: vec![fact("Lease", "note", None, "a paraphrase of neither")],+ names: Vec::new(),+ skipped_entities: 0,+ skipped_facts: 0,+ truncated: false,+ };+ assert!(drop_quotes_from_opening(&mut y, chunk, opening).is_empty());+ assert!(drop_quotes_from_opening(&mut y, chunk, " ").is_empty());+ assert_eq!(y.facts.len(), 1);+ }+fn run(entities: Vec<ExtractedEntity>,facts: Vec<ExtractedFact>,diff --git a/crates/utopia-server/src/extraction.rs b/crates/utopia-server/src/extraction.rsindex fc52b6d4..e2b4c3ac 100644--- a/crates/utopia-server/src/extraction.rs+++ b/crates/utopia-server/src/extraction.rs@@ -1054,6 +1054,13 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:let doc_time = doc.doc_time.map(|t| t.format("%Y-%m-%d").to_string());let chunks = utopia_store::documents::chunks_for_extraction(&state.pool, document_id).await?;+ // 文件开头:序号最小的现存分块(不是还没抽的第一块)。备忘文件一个片段一块,+ // 前一段不是后一段的开头,不附+ let opening_chunk = if await_nod {+ None+ } else {+ utopia_store::documents::opening_chunk(&state.pool, document_id).await?+ };let mut doc_cache: HashMap<(Option<Uuid>, String), Uuid> = HashMap::new();// Identities introduced through handles, grouped only for detecting document-local@@ -1108,13 +1115,19 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:name: name.clone(),}).collect();- let messages = utopia_extract::build_messages(+ // 这一块就是开头本身时不再重复一遍+ let opening = opening_chunk+ .as_ref()+ .filter(|(id, _)| *id != chunk.id)+ .map(|(_, text)| text.as_str());+ let messages = utopia_extract::build_messages_with_opening(&lists.types,&lists.relations,&lists.attributes,doc_time.as_deref(),&doc.filename,&known,+ opening,&chunk.text,);// 这两处 continue 跳过的是**整个分块**——它一条事实都没产出。@@ -1177,7 +1190,16 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:// **落库前先查形状**(utopia_extract::normalize):只看结构、不看词——引文里有没有// 这段字、值是不是只有标点、一侧是不是契约的日期、同句有没有另一条边。读懂时间// 归模型(提示词 3c),这里只核对它照没照契约写,做了什么都记进丢弃表- for n in utopia_extract::normalize_facts(&mut extraction) {+ let from_opening = match opening {+ Some(text) => {+ utopia_extract::drop_quotes_from_opening(&mut extraction, &chunk.text, text)+ }+ None => Vec::new(),+ };+ for n in from_opening+ .into_iter()+ .chain(utopia_extract::normalize_facts(&mut extraction))+ {use utopia_extract::Normalization as N;use utopia_store::extraction_drops::reason;let (r, detail, example) = match n {@@ -1224,6 +1246,9 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:N::OrphanDeclaration { name } => {(reason::ORPHAN_DECLARATION, "entity".to_string(), name)}+ N::QuoteFromOpening { predicate, quote } => {+ (reason::QUOTE_FROM_OPENING, predicate, quote)+ }};drop_signal(state, doc.kb_id, document_id, r, &detail, Some(&example)).await;}diff --git a/crates/utopia-store/src/documents.rs b/crates/utopia-store/src/documents.rsindex c8ab0647..97b1d62f 100644--- a/crates/utopia-store/src/documents.rs+++ b/crates/utopia-store/src/documents.rs@@ -1236,6 +1236,22 @@ pub async fn chunks_for_extraction(Ok(rows)}+/// 文件的**开头**:序号最小的现存分块,不论抽没抽过。+///+/// 不能拿 [`chunks_for_extraction`] 的第一条代替:那只是还没抽的第一块——改过的文件+/// 重抽时第 7 块会拿到第 3 块,失败重试时后面的块会拿到失败的那块+pub async fn opening_chunk(pool: &PgPool, document_id: Uuid) -> AppResult<Option<(Uuid, String)>> {+ let row = sqlx::query_as(+ "SELECT id, text FROM chunks+ WHERE document_id = $1 AND superseded_at IS NULL+ ORDER BY seq LIMIT 1",+ )+ .bind(document_id)+ .fetch_optional(pool)+ .await?;+ Ok(row)+}+/// 推进抽取状态(顺带清空上一轮的失败原因——重跑即翻篇)。pub async fn set_graph_status(pool: &PgPool, id: Uuid, status: &str) -> AppResult<()> {sqlx::query(diff --git a/crates/utopia-store/src/extraction_drops.rs b/crates/utopia-store/src/extraction_drops.rsindex 5ea10c53..54e7a037 100644--- a/crates/utopia-store/src/extraction_drops.rs+++ b/crates/utopia-store/src/extraction_drops.rs@@ -95,6 +95,8 @@ pub mod reason {pub const OBJECT_DESCRIBES_DECLARED: &str = "object_describes_declared";/// 上面几条去掉事实后没人引用的声明:不建pub const ORPHAN_DECLARATION: &str = "orphan_declaration";+ /// 引文抄自提示词里附的文件开头、不在这一块:证据会挂错出处,不落+ pub const QUOTE_FROM_OPENING: &str = "quote_from_opening";}pub async fn record(diff --git a/crates/utopia-store/tests/a_document_opening_is_its_first_live_chunk.rs b/crates/utopia-store/tests/a_document_opening_is_its_first_live_chunk.rsnew file mode 100644index 00000000..372a40b8--- /dev/null+++ b/crates/utopia-store/tests/a_document_opening_is_its_first_live_chunk.rs@@ -0,0 +1,102 @@+//! 提示词里附的「文件开头」是序号最小的现存分块,不是还没抽的第一块。+//!+//! 抽取只取 `extracted_at IS NULL` 的分块。从前开头取的是那张清单的第一条:改过的文件+//! 重抽第 3、7 块时,第 7 块把第 3 块当成开头;失败重试时,后面的块把失败的那块当成开头。+//!+//! 没有 `UTOPIA_DATABASE_URL` 时跳过而不是失败。自建自拆,绝不碰已有的库。++use sqlx::PgPool;+use uuid::Uuid;++#[tokio::test]+async fn the_opening_is_the_first_live_chunk_whether_or_not_it_was_extracted() -> anyhow::Result<()>+{+ let Some(url) = utopia_store::test_db::url() else {+ return Ok(());+ };+ let pool = PgPool::connect(&url).await?;+ let (org, ws, kb, doc) = (+ Uuid::now_v7(),+ Uuid::now_v7(),+ Uuid::now_v7(),+ Uuid::now_v7(),+ );+ sqlx::query("INSERT INTO organizations (id, name) VALUES ($1, 'opening-test')")+ .bind(org)+ .execute(&pool)+ .await?;+ sqlx::query("INSERT INTO workspaces (id, org_id, name) VALUES ($1, $2, 'opening-test')")+ .bind(ws)+ .bind(org)+ .execute(&pool)+ .await?;+ sqlx::query(+ "INSERT INTO knowledge_bases (id, workspace_id, name) VALUES ($1, $2, 'opening-test')",+ )+ .bind(kb)+ .bind(ws)+ .execute(&pool)+ .await?;++ let run = async {+ sqlx::query(+ "INSERT INTO documents (id, kb_id, filename, sha256) VALUES ($1, $2, 'a.html', 'a')",+ )+ .bind(doc)+ .bind(kb)+ .execute(&pool)+ .await?;+ let superseded = Uuid::now_v7();+ let first = Uuid::now_v7();+ // 第 0 块是上一版留下、已被取代的;第 1 块抽过了;第 3、7 块等着抽+ for (id, seq, extracted, gone) in [+ (superseded, 0, true, true),+ (first, 1, true, false),+ (Uuid::now_v7(), 3, false, false),+ (Uuid::now_v7(), 7, false, false),+ ] {+ sqlx::query(+ "INSERT INTO chunks (id, kb_id, document_id, seq, text, extracted_at, superseded_at)+ VALUES ($1, $2, $3, $4, $5,+ CASE WHEN $6 THEN now() END, CASE WHEN $7 THEN now() END)",+ )+ .bind(id)+ .bind(kb)+ .bind(doc)+ .bind(seq)+ .bind(format!("chunk {seq}"))+ .bind(extracted)+ .bind(gone)+ .execute(&pool)+ .await?;+ }+ let pending = utopia_store::documents::chunks_for_extraction(&pool, doc).await?;+ assert_eq!(+ pending.first().map(|c| c.seq),+ Some(3),+ "等着抽的第一块是第 3 块"+ );+ let opening = utopia_store::documents::opening_chunk(&pool, doc).await?;+ assert_eq!(+ opening.map(|(id, _)| id),+ Some(first),+ "开头是现存的第 1 块,抽没抽过都一样"+ );+ Ok::<_, anyhow::Error>(())+ }+ .await;++ sqlx::query("DELETE FROM knowledge_bases WHERE id = $1")+ .bind(kb)+ .execute(&pool)+ .await?;+ sqlx::query("DELETE FROM workspaces WHERE id = $1")+ .bind(ws)+ .execute(&pool)+ .await?;+ sqlx::query("DELETE FROM organizations WHERE id = $1")+ .bind(org)+ .execute(&pool)+ .await?;+ run+}