Skip to content

World's first open-source enterprise world model.

A chunk reads the opening of its document, and a date keeps the contract format (#680)

WaylandYang785ce6a348c7
diff --git a/crates/utopia-extract/src/lib.rs b/crates/utopia-extract/src/lib.rs
index f21707a4..5d1ff432 100644
--- a/crates/utopia-extract/src/lib.rs
+++ b/crates/utopia-extract/src/lib.rs
@@ -8,7 +8,7 @@ use utopia_llm::ChatMessage;
pub mod governor;
pub mod normalize;
-pub use normalize::{normalize_facts, Normalization};
+pub use normalize::{drop_quotes_from_opening, normalize_facts, Normalization};
#[derive(Debug, Deserialize)]
pub struct Extraction {
@@ -140,6 +140,34 @@ pub fn build_messages(
// 第一块为空——那时还没有"前面"
known: &[KnownEntity],
chunk_text: &str,
+) -> Vec<ChatMessage> {
+ build_messages_with_opening(
+ types, relations, attributes, doc_time, filename, known, None, chunk_text,
+ )
+}
+
+/// 文件开头进提示词的字符预算。一份补充协议的标题、生效日、当事方和「修订的是哪份
+/// 协议」通常在头一千字符里;新闻稿的电头与导语也是
+pub const OPENING_BUDGET_CHARS: usize = 1500;
+
+/// 同 [`build_messages`],另带**这份文件的开头**(第一块的原文),给第二块往后用。
+///
+/// **一块是孤立抽取的,它看不见自己属于什么。** 补充协议把截止日写在第三块的表格里,
+/// 那一块只说「Article 13 的日期延至……」:改的是哪份租约、从哪天起改,都写在第一块。
+/// 模型拿不到,就只能把「Phase 2 Exercise Deadline」本身当主语(服务端按主语未声明丢掉),
+/// 或者抽出一个没有起点的日期(时态引擎没法据此关闭旧值)——Blackbaud 总部租约链
+/// 上五次改期丢了两次,抽到的三次一次都没关上旧值。开头只作背景,不从里面抽事实:
+/// 它自己那一块会抽,重复抽只会多出重复的事实
+#[allow(clippy::too_many_arguments)]
+pub fn build_messages_with_opening(
+ types: &[(String, String, String)],
+ relations: &[PromptRelation],
+ attributes: &[String],
+ doc_time: Option<&str>,
+ filename: &str,
+ known: &[KnownEntity],
+ opening: Option<&str>,
+ chunk_text: &str,
) -> Vec<ChatMessage> {
// **有描述时不送 label**。label 是给人看的显示名,而且它跟界面无关、
// 跟这个库的语料语言走——中文库里 person 的 label 是"人物"。
@@ -265,7 +293,9 @@ pub fn build_messages(
} else {
"\n10. Attribute facts carry \"value\" (no \"object\"): number = the figure **as the text writes it, magnitude and currency included** \n (\"86亿元\", \"$5 billion\", \"4,300 人\") — never reduce it to a bare number, the server converts; date = \"YYYY[-MM[-DD]]\" (a zoned clock time only when the text gives one); bool = true/false; \
text = a short string. Only attach an attribute to a subject of its listed class. \
- valid_from = when this value took effect, if the text says so."
+ valid_from = when this value took effect, if the text or the opening of the document says so. \
+ A document that changes a value set earlier — amends, extends or replaces it — makes the new value \
+ hold from the date the change takes effect, which is the document's own effective date unless the text gives another."
.to_string()
};
let system = format!(
@@ -296,7 +326,10 @@ pub fn build_messages(
list each entity once. Text introduces a full name and then shortens it — \
\"星云科技上海研究院\" becomes \"上海研究院\", \"Nebula Technologies Inc.\" becomes \
\"Nebula\" — and both forms mean one entity, listed once under the fuller form. \
- Two names are two entities only when the text is talking about two things.\n\
+ Two names are two entities only when the text is talking about two things. \
+ A name identifies the thing; it is not a description of its history. When the text \
+ names something and then describes what happened to it, the name ends where the \
+ description begins.\n\
1b. Every other name the text gives an entity goes into \"names\", once per name: the \
shortened form it introduces or uses (\"上海研究院\" for \"星云科技上海研究院\"), a \
former name, the name in another language. \"ref\" is the entity's local_id or its \
@@ -328,7 +361,7 @@ pub fn build_messages(
of values that hold in that period.\n\
{temporal_note}\n\
4. {time_ctx}\n\
- 5. quote must be a contiguous excerpt from the source text; every fact needs one.\n\
+ 5. quote must be a contiguous excerpt from the Text block; never quote the opening of the document. Every fact needs one.\n\
6. confidence in 0~1: 0.9 explicitly stated, 0.7 inferred, 0.5 uncertain.\n\
7. If nothing can be extracted, output {{\"entities\":[],\"facts\":[]}}.\n\
8. If no listed relation fits, do not force the nearest one — write the predicate the \
@@ -341,10 +374,14 @@ pub fn build_messages(
\"quote\":\"...\"}}. Name the predicate after the text when no listed attribute \
fits — \"purchase_price\", \"job_title\", \"generation_capacity\", \"record_date\". \
Attach it to the entity the text attaches it to, and keep the literal as written, \
- units and all. **A stated figure left out is the loss that costs most**: the reader \
+ units and all — except a date, which is always written in the format of rule 3 \
+ (\"June 23, 2020\" is \"2020-06-23\"): the server keeps a date only in that \
+ format, and a date written any other way is lost. A deadline or a period stated \
+ relative to an event, with no calendar date, is not a date. \
+ **A stated figure left out is the loss that costs most**: the reader \
came for those numbers, and no later step can recover one that was never written \
down.\n\
- 8c. A listed relation followed by {{…}} can carry those **qualifiers on the edge**: when the same sentence gives both the other entity and a figure for it — an amount, a stake, a price, a share count — write the relation with its \"object\" and put the figure in \"qualifiers\" keyed exactly as listed, **as written in the text, currency and all** (\"€30 million\", \"15亿元人民币\", never a bare number): {{\"subject\":\"Vega Capital\",\"predicate\":\"invested_in\",\"object\":\"Northwind\", \"qualifiers\":{{\"amount\":\"$5 billion\"}},…}}. Never invent a key that is not listed for that relation, and never drop the figure to keep the edge — a relation without its amount is half the sentence. A relation you name after the text (rule 8) carries its figure the same way — keyed by the listed attribute that fits it, or by the plainest word for it (\"amount\", \"stake\", \"price\") when none does.
+ 8c. A listed relation followed by {{…}} can carry those **qualifiers on the edge**: when the same sentence gives both the other entity and a figure for it — an amount, a stake, a price, a share count — write the relation with its \"object\" and put the figure in \"qualifiers\" keyed exactly as listed, **as written in the text, currency and all** (\"€30 million\", \"15亿元人民币\", never a bare number) — except a date, which takes the format of rule 3: {{\"subject\":\"Vega Capital\",\"predicate\":\"invested_in\",\"object\":\"Northwind\", \"qualifiers\":{{\"amount\":\"$5 billion\"}},…}}. Never invent a key that is not listed for that relation, and never drop the figure to keep the edge — a relation without its amount is half the sentence. A relation you name after the text (rule 8) carries its figure the same way — keyed by the listed attribute that fits it, or by the plainest word for it (\"amount\", \"stake\", \"price\") when none does.
8b. A **listed** relation also takes \"value\" when what the text gives is a \
string rather than another entity — a job title, a designation, a ticker, a \
model number. Never invent an entity for a string. And when the text introduces \
@@ -367,6 +404,10 @@ pub fn build_messages(
Copy them; never paraphrase. When the words that do the thing are a description \
rather than a name — \"former X employees\", \"companies using X\" — the span \
is that description, whatever you wrote in subject.\n\
+ 8e. An obligation, a deadline or a right belongs to the agreement, law or decision \
+ that imposes it, even when it concerns another agreement or thing. A lease that \
+ sets the last day to sign a second lease gives that deadline to the first lease; \
+ the second lease is only what the deadline is about.\n\
9. The same holds for entity types: if none of the listed types fits, write the type \
the text implies, in snake_case (e.g. \"model\", \"technology\"). Do not fall back \
to a broad listed type such as \"thing\" or \"creative_work\" merely because \
@@ -381,7 +422,8 @@ pub fn build_messages(
// 已知实体紧挨着正文:服从性靠位置,理由见 known_block 的注释
let user = format!(
- "Source file: \"{filename}\"\n{}\nText:\n{chunk_text}",
+ "Source file: \"{filename}\"\n{}{}\nText:\n{chunk_text}",
+ opening_block(opening),
known_block(known)
);
@@ -397,6 +439,29 @@ pub fn build_messages(
]
}
+/// 文件开头排版成提示词里的一段。开头为空(或只有空白)时返回空串;
+/// 「这一块就是开头本身」由调用方判断,那时它传 `None`。
+///
+/// 按字符截:不会截断一个字符,但会截在词中间——英文的最后一个词可能只剩半个
+fn opening_block(opening: Option<&str>) -> String {
+ let Some(text) = opening.map(str::trim).filter(|t| !t.is_empty()) else {
+ return String::new();
+ };
+ let cut: String = text.chars().take(OPENING_BUDGET_CHARS).collect();
+ let more = if cut.chars().count() < text.chars().count() {
+ " …"
+ } else {
+ ""
+ };
+ format!(
+ "\nOpening of this document, for context only (do not extract facts from it; they are \
+ extracted from that part separately). Use it to know what the text below belongs to — \
+ which agreement, company or event it concerns, who the parties are, and the date it \
+ takes effect — so that facts in the text below attach to the right entity and carry \
+ the right dates:\n\"\"\"\n{cut}{more}\n\"\"\"\n"
+ )
+}
+
/// 清单里给关系带的标记:事件 `[event]`、恒常 `[eternal]`;状态不标。
/// 认不出的值当状态——数据库的 CHECK 只放这三个进来,这里不再报错
fn temporal_mark(temporal: &str) -> Option<&'static str> {
@@ -1214,6 +1279,63 @@ mod prompt_shape_tests {
);
}
+ #[test]
+ fn an_obligation_belongs_to_the_agreement_that_imposes_it() {
+ // 主租约里写着「签二期租约的截止日」,模型时而把截止日挂到二期租约上:
+ // 主租约的时间线上就少了这次改期(#681 §3)
+ let msgs = build_messages(&[], &[], &[], None, "a.txt", &[], "text");
+ assert!(msgs[0]
+ .content
+ .contains("belongs to the agreement, law or decision that imposes it"));
+ }
+
+ #[test]
+ fn a_literal_keeps_its_units_but_a_date_takes_the_contract_format() {
+ // 8a 从前说「字面值按原文写」并把日期列在字面值里,而规则 3 与属性规则要求
+ // YYYY-MM-DD:两条互相打架,模型写出「June 23, 2020」,服务端按格式不合整条丢掉。
+ // Blackbaud 总部租约链上各轮累计丢了二十多次
+ let msgs = build_messages(&[], &[], &[], None, "a.txt", &[], "text");
+ let system = &msgs[0].content;
+ assert!(system.contains("except a date, which is always written in the format of rule 3"));
+ }
+
+ #[test]
+ fn a_later_chunk_reads_the_opening_of_its_document() {
+ let opening = "FIFTH AMENDMENT TO LEASE AGREEMENT entered into as of February 18, 2020";
+ let msgs = build_messages_with_opening(
+ &[],
+ &[],
+ &[],
+ None,
+ "a.html",
+ &[],
+ Some(opening),
+ "The Existing Dates are extended to March 17, 2020.",
+ );
+ let user = &msgs[1].content;
+ let at_opening = user.find(opening).expect("opening is in the user message");
+ let at_text = user
+ .find("The Existing Dates")
+ .expect("text is in the user message");
+ assert!(
+ at_opening < at_text,
+ "the opening comes before the text it frames"
+ );
+ // 没有开头时,提示词与从前一字不差
+ let plain = build_messages(&[], &[], &[], None, "a.html", &[], "t");
+ let framed = build_messages_with_opening(&[], &[], &[], None, "a.html", &[], None, "t");
+ assert_eq!(plain[1].content, framed[1].content);
+ }
+
+ #[test]
+ fn a_long_opening_is_cut_on_a_character_boundary() {
+ let long = "租".repeat(OPENING_BUDGET_CHARS + 10);
+ let block = opening_block(Some(&long));
+ assert_eq!(block.matches('租').count(), OPENING_BUDGET_CHARS);
+ assert!(block.contains(" …"));
+ assert_eq!(opening_block(Some(" ")), "");
+ }
+
/// 已知实体必须落在 **user** 消息里、紧挨着正文。
///
/// 理由是服从性不是缓存:抽象规则打不过挨着它的具体块。清单放进 system 的
diff --git a/crates/utopia-extract/src/normalize.rs b/crates/utopia-extract/src/normalize.rs
index f0a7aa11..399f374e 100644
--- a/crates/utopia-extract/src/normalize.rs
+++ b/crates/utopia-extract/src/normalize.rs
@@ -9,7 +9,7 @@
//! 每条规则做了什么都返回给服务端记进丢弃表:违约多常见、出在哪个模型,量得出来,
//! 契约该怎么改看数说话。
-use crate::{parse_time, ExtractedFact, Extraction};
+use crate::{parse_time, ExtractedEntity, ExtractedFact, Extraction};
use std::collections::HashSet;
/// 形状检查做了什么;服务端按条记信号
@@ -49,6 +49,9 @@ pub enum Normalization {
},
/// 上面几条去掉事实之后,没有任何事实再引用的声明:不建,否则就是一个孤点
OrphanDeclaration { name: String },
+ /// 引文抄自提示词里附的文件开头,而不是这一块:开头只作背景,它自己那一块会抽到。
+ /// 照落的话,证据挂在这一块上,引的却是第一块的话——引错了出处
+ QuoteFromOpening { predicate: String, quote: String },
}
/// 比对用的形态:空白折叠、小写
@@ -230,6 +233,84 @@ fn is_contract_time(s: &str) -> bool {
parse_time(s.trim()).is_some()
}
+/// 引文不在这一块、却在附上的文件开头里的事实:丢掉,连同因此没人引用的声明。
+///
+/// 只比「在不在」(空白、大小写不论),不比像不像:两边都找不到的引文不归这里管——
+/// 那是模型改写了原文,与开头无关
+pub fn drop_quotes_from_opening(
+ x: &mut Extraction,
+ chunk_text: &str,
+ opening: &str,
+) -> Vec<Normalization> {
+ let (chunk, opening) = (norm(chunk_text), norm(opening));
+ if opening.is_empty() {
+ return Vec::new();
+ }
+ let before = referenced_names(&x.entities, &x.facts);
+ let mut out = Vec::new();
+ x.facts.retain(|f| {
+ let q = norm(f.quote.as_deref().unwrap_or(""));
+ let from_opening = !q.is_empty() && !chunk.contains(&q) && opening.contains(&q);
+ if from_opening {
+ out.push(Normalization::QuoteFromOpening {
+ predicate: f.predicate.clone(),
+ quote: f.quote.clone().unwrap_or_default(),
+ });
+ }
+ !from_opening
+ });
+ if out.is_empty() {
+ return out;
+ }
+ let mut after = referenced_names(&x.entities, &x.facts);
+ // 与 normalize_facts 同一条:模型给它报了别的名字的声明不算孤点(0041)
+ after.extend(x.names.iter().filter_map(|n| {
+ x.entities
+ .iter()
+ .find(|e| e.local_id.as_deref().map(str::trim) == Some(n.entity_ref.trim()))
+ .map(|e| e.name.trim().to_lowercase())
+ }));
+ let mut orphans = Vec::new();
+ x.entities.retain(|e| {
+ let n = e.name.trim().to_lowercase();
+ let orphan = before.contains(&n) && !after.contains(&n);
+ if orphan {
+ orphans.push(e.name.trim().to_string());
+ }
+ !orphan
+ });
+ out.extend(
+ orphans
+ .into_iter()
+ .map(|name| Normalization::OrphanDeclaration { name }),
+ );
+ out
+}
+
+/// 事实两侧绑到的声明名(小写):有句柄按句柄,没有按写出来的名字
+fn referenced_names(entities: &[ExtractedEntity], facts: &[ExtractedFact]) -> HashSet<String> {
+ let handle_name = |h: Option<&String>| {
+ h.and_then(|h| {
+ entities
+ .iter()
+ .find(|e| e.local_id.as_deref().map(str::trim) == Some(h.trim()))
+ .map(|e| e.name.trim().to_string())
+ })
+ };
+ facts
+ .iter()
+ .flat_map(|f| {
+ [
+ handle_name(f.subject_ref.as_ref()).or_else(|| Some(f.subject.trim().to_string())),
+ handle_name(f.object_ref.as_ref())
+ .or_else(|| f.object.as_deref().map(|o| o.trim().to_string())),
+ ]
+ })
+ .flatten()
+ .map(|n| n.to_lowercase())
+ .collect()
+}
+
pub fn normalize_facts(x: &mut Extraction) -> Vec<Normalization> {
let mut out = Vec::new();
let entities = std::mem::take(&mut x.entities);
@@ -243,22 +324,7 @@ pub fn normalize_facts(x: &mut Extraction) -> Vec<Normalization> {
.map(|e| e.name.trim().to_string())
})
};
- let referenced = |facts: &[ExtractedFact]| -> HashSet<String> {
- facts
- .iter()
- .flat_map(|f| {
- [
- handle_name(f.subject_ref.as_ref())
- .or_else(|| Some(f.subject.trim().to_string())),
- handle_name(f.object_ref.as_ref())
- .or_else(|| f.object.as_deref().map(|o| o.trim().to_string())),
- ]
- })
- .flatten()
- .map(|n| n.to_lowercase())
- .collect()
- };
- let before = referenced(&x.facts);
+ let before = referenced_names(&entities, &x.facts);
let mut facts: Vec<ExtractedFact> = Vec::with_capacity(x.facts.len());
for mut f in std::mem::take(&mut x.facts) {
@@ -423,7 +489,7 @@ pub fn normalize_facts(x: &mut Extraction) -> Vec<Normalization> {
// ---- 被上面几条弄成孤点的声明 ----
// 只去掉「原来有事实引用、现在没有了」的:模型一开始就只声明不连边的,不归这里管
- let mut after = referenced(&x.facts);
+ let mut after = referenced_names(&entities, &x.facts);
// 模型给它报了别的名字的声明也不算孤点:那些名字要绑在它身上(0041)
after.extend(
x.names
@@ -491,6 +557,63 @@ mod tests {
.map(|(k, v)| (k.to_string(), serde_json::Value::String(v.to_string())))
.collect()
}
+ /// 第五份补充协议的第三块:模型从附上的开头里抄了「as of February 18, 2020」那句当引文。
+ /// 那条事实丢掉,只为它而声明的实体跟着不建;引文在这一块里的照落
+ #[test]
+ fn a_fact_quoting_the_opening_is_dropped_with_its_orphan() {
+ let opening = "THIS FIFTH AMENDMENT TO LEASE AGREEMENT is entered into as of February 18, 2020 by and between HPBB1, LLC and BLACKBAUD, INC.";
+ let chunk = "The Phase 2 Exercise Deadline is hereby extended to March 17, 2020.";
+ let mut x = Extraction {
+ entities: vec![entity("e1", "Lease"), entity("e2", "HPBB1, LLC")],
+ facts: vec![
+ fact(
+ "Lease",
+ "landlord",
+ Some("HPBB1, LLC"),
+ "entered into as of February 18, 2020 by and between HPBB1, LLC",
+ ),
+ fact(
+ "Lease",
+ "expansion_option_deadline",
+ None,
+ "the phase 2 exercise deadline is hereby extended to March 17, 2020",
+ ),
+ ],
+ names: Vec::new(),
+ skipped_entities: 0,
+ skipped_facts: 0,
+ truncated: false,
+ };
+ let n = drop_quotes_from_opening(&mut x, chunk, opening);
+ let kept: Vec<&str> = x.facts.iter().map(|f| f.predicate.as_str()).collect();
+ assert_eq!(
+ kept,
+ ["expansion_option_deadline"],
+ "大小写与空白不论,引文在这一块里的留下"
+ );
+ let names: Vec<&str> = x.entities.iter().map(|e| e.name.as_str()).collect();
+ assert_eq!(names, ["Lease"], "只为被丢那条声明的 HPBB1 不建");
+ assert!(n
+ .iter()
+ .any(|v| matches!(v, Normalization::QuoteFromOpening { .. })));
+ assert!(n
+ .iter()
+ .any(|v| matches!(v, Normalization::OrphanDeclaration { .. })));
+
+ // 两边都找不到的引文不归这里管;没有开头时什么都不做
+ let mut y = Extraction {
+ entities: vec![entity("e1", "Lease")],
+ facts: vec![fact("Lease", "note", None, "a paraphrase of neither")],
+ names: Vec::new(),
+ skipped_entities: 0,
+ skipped_facts: 0,
+ truncated: false,
+ };
+ assert!(drop_quotes_from_opening(&mut y, chunk, opening).is_empty());
+ assert!(drop_quotes_from_opening(&mut y, chunk, " ").is_empty());
+ assert_eq!(y.facts.len(), 1);
+ }
+
fn run(
entities: Vec<ExtractedEntity>,
facts: Vec<ExtractedFact>,
diff --git a/crates/utopia-server/src/extraction.rs b/crates/utopia-server/src/extraction.rs
index fc52b6d4..e2b4c3ac 100644
--- a/crates/utopia-server/src/extraction.rs
+++ b/crates/utopia-server/src/extraction.rs
@@ -1054,6 +1054,13 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
let doc_time = doc.doc_time.map(|t| t.format("%Y-%m-%d").to_string());
let chunks = utopia_store::documents::chunks_for_extraction(&state.pool, document_id).await?;
+ // 文件开头:序号最小的现存分块(不是还没抽的第一块)。备忘文件一个片段一块,
+ // 前一段不是后一段的开头,不附
+ let opening_chunk = if await_nod {
+ None
+ } else {
+ utopia_store::documents::opening_chunk(&state.pool, document_id).await?
+ };
let mut doc_cache: HashMap<(Option<Uuid>, String), Uuid> = HashMap::new();
// Identities introduced through handles, grouped only for detecting document-local
@@ -1108,13 +1115,19 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
name: name.clone(),
})
.collect();
- let messages = utopia_extract::build_messages(
+ // 这一块就是开头本身时不再重复一遍
+ let opening = opening_chunk
+ .as_ref()
+ .filter(|(id, _)| *id != chunk.id)
+ .map(|(_, text)| text.as_str());
+ let messages = utopia_extract::build_messages_with_opening(
&lists.types,
&lists.relations,
&lists.attributes,
doc_time.as_deref(),
&doc.filename,
&known,
+ opening,
&chunk.text,
);
// 这两处 continue 跳过的是**整个分块**——它一条事实都没产出。
@@ -1177,7 +1190,16 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
// **落库前先查形状**(utopia_extract::normalize):只看结构、不看词——引文里有没有
// 这段字、值是不是只有标点、一侧是不是契约的日期、同句有没有另一条边。读懂时间
// 归模型(提示词 3c),这里只核对它照没照契约写,做了什么都记进丢弃表
- for n in utopia_extract::normalize_facts(&mut extraction) {
+ let from_opening = match opening {
+ Some(text) => {
+ utopia_extract::drop_quotes_from_opening(&mut extraction, &chunk.text, text)
+ }
+ None => Vec::new(),
+ };
+ for n in from_opening
+ .into_iter()
+ .chain(utopia_extract::normalize_facts(&mut extraction))
+ {
use utopia_extract::Normalization as N;
use utopia_store::extraction_drops::reason;
let (r, detail, example) = match n {
@@ -1224,6 +1246,9 @@ async fn run(state: &AppState, document_id: Uuid, proposer: Proposer) -> anyhow:
N::OrphanDeclaration { name } => {
(reason::ORPHAN_DECLARATION, "entity".to_string(), name)
}
+ N::QuoteFromOpening { predicate, quote } => {
+ (reason::QUOTE_FROM_OPENING, predicate, quote)
+ }
};
drop_signal(state, doc.kb_id, document_id, r, &detail, Some(&example)).await;
}
diff --git a/crates/utopia-store/src/documents.rs b/crates/utopia-store/src/documents.rs
index c8ab0647..97b1d62f 100644
--- a/crates/utopia-store/src/documents.rs
+++ b/crates/utopia-store/src/documents.rs
@@ -1236,6 +1236,22 @@ pub async fn chunks_for_extraction(
Ok(rows)
}
+/// 文件的**开头**:序号最小的现存分块,不论抽没抽过。
+///
+/// 不能拿 [`chunks_for_extraction`] 的第一条代替:那只是还没抽的第一块——改过的文件
+/// 重抽时第 7 块会拿到第 3 块,失败重试时后面的块会拿到失败的那块
+pub async fn opening_chunk(pool: &PgPool, document_id: Uuid) -> AppResult<Option<(Uuid, String)>> {
+ let row = sqlx::query_as(
+ "SELECT id, text FROM chunks
+ WHERE document_id = $1 AND superseded_at IS NULL
+ ORDER BY seq LIMIT 1",
+ )
+ .bind(document_id)
+ .fetch_optional(pool)
+ .await?;
+ Ok(row)
+}
+
/// 推进抽取状态(顺带清空上一轮的失败原因——重跑即翻篇)。
pub async fn set_graph_status(pool: &PgPool, id: Uuid, status: &str) -> AppResult<()> {
sqlx::query(
diff --git a/crates/utopia-store/src/extraction_drops.rs b/crates/utopia-store/src/extraction_drops.rs
index 5ea10c53..54e7a037 100644
--- a/crates/utopia-store/src/extraction_drops.rs
+++ b/crates/utopia-store/src/extraction_drops.rs
@@ -95,6 +95,8 @@ pub mod reason {
pub const OBJECT_DESCRIBES_DECLARED: &str = "object_describes_declared";
/// 上面几条去掉事实后没人引用的声明:不建
pub const ORPHAN_DECLARATION: &str = "orphan_declaration";
+ /// 引文抄自提示词里附的文件开头、不在这一块:证据会挂错出处,不落
+ pub const QUOTE_FROM_OPENING: &str = "quote_from_opening";
}
pub async fn record(
diff --git a/crates/utopia-store/tests/a_document_opening_is_its_first_live_chunk.rs b/crates/utopia-store/tests/a_document_opening_is_its_first_live_chunk.rs
new file mode 100644
index 00000000..372a40b8
--- /dev/null
+++ b/crates/utopia-store/tests/a_document_opening_is_its_first_live_chunk.rs
@@ -0,0 +1,102 @@
+//! 提示词里附的「文件开头」是序号最小的现存分块,不是还没抽的第一块。
+//!
+//! 抽取只取 `extracted_at IS NULL` 的分块。从前开头取的是那张清单的第一条:改过的文件
+//! 重抽第 3、7 块时,第 7 块把第 3 块当成开头;失败重试时,后面的块把失败的那块当成开头。
+//!
+//! 没有 `UTOPIA_DATABASE_URL` 时跳过而不是失败。自建自拆,绝不碰已有的库。
+
+use sqlx::PgPool;
+use uuid::Uuid;
+
+#[tokio::test]
+async fn the_opening_is_the_first_live_chunk_whether_or_not_it_was_extracted() -> anyhow::Result<()>
+{
+ let Some(url) = utopia_store::test_db::url() else {
+ return Ok(());
+ };
+ let pool = PgPool::connect(&url).await?;
+ let (org, ws, kb, doc) = (
+ Uuid::now_v7(),
+ Uuid::now_v7(),
+ Uuid::now_v7(),
+ Uuid::now_v7(),
+ );
+ sqlx::query("INSERT INTO organizations (id, name) VALUES ($1, 'opening-test')")
+ .bind(org)
+ .execute(&pool)
+ .await?;
+ sqlx::query("INSERT INTO workspaces (id, org_id, name) VALUES ($1, $2, 'opening-test')")
+ .bind(ws)
+ .bind(org)
+ .execute(&pool)
+ .await?;
+ sqlx::query(
+ "INSERT INTO knowledge_bases (id, workspace_id, name) VALUES ($1, $2, 'opening-test')",
+ )
+ .bind(kb)
+ .bind(ws)
+ .execute(&pool)
+ .await?;
+
+ let run = async {
+ sqlx::query(
+ "INSERT INTO documents (id, kb_id, filename, sha256) VALUES ($1, $2, 'a.html', 'a')",
+ )
+ .bind(doc)
+ .bind(kb)
+ .execute(&pool)
+ .await?;
+ let superseded = Uuid::now_v7();
+ let first = Uuid::now_v7();
+ // 第 0 块是上一版留下、已被取代的;第 1 块抽过了;第 3、7 块等着抽
+ for (id, seq, extracted, gone) in [
+ (superseded, 0, true, true),
+ (first, 1, true, false),
+ (Uuid::now_v7(), 3, false, false),
+ (Uuid::now_v7(), 7, false, false),
+ ] {
+ sqlx::query(
+ "INSERT INTO chunks (id, kb_id, document_id, seq, text, extracted_at, superseded_at)
+ VALUES ($1, $2, $3, $4, $5,
+ CASE WHEN $6 THEN now() END, CASE WHEN $7 THEN now() END)",
+ )
+ .bind(id)
+ .bind(kb)
+ .bind(doc)
+ .bind(seq)
+ .bind(format!("chunk {seq}"))
+ .bind(extracted)
+ .bind(gone)
+ .execute(&pool)
+ .await?;
+ }
+ let pending = utopia_store::documents::chunks_for_extraction(&pool, doc).await?;
+ assert_eq!(
+ pending.first().map(|c| c.seq),
+ Some(3),
+ "等着抽的第一块是第 3 块"
+ );
+ let opening = utopia_store::documents::opening_chunk(&pool, doc).await?;
+ assert_eq!(
+ opening.map(|(id, _)| id),
+ Some(first),
+ "开头是现存的第 1 块,抽没抽过都一样"
+ );
+ Ok::<_, anyhow::Error>(())
+ }
+ .await;
+
+ sqlx::query("DELETE FROM knowledge_bases WHERE id = $1")
+ .bind(kb)
+ .execute(&pool)
+ .await?;
+ sqlx::query("DELETE FROM workspaces WHERE id = $1")
+ .bind(ws)
+ .execute(&pool)
+ .await?;
+ sqlx::query("DELETE FROM organizations WHERE id = $1")
+ .bind(org)
+ .execute(&pool)
+ .await?;
+ run
+}