World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
The quote comes first (#748)
WaylandYangda0e124395cf
diff --git a/crates/utopia-extract/src/open.rs b/crates/utopia-extract/src/open.rsindex d1ead1d8..df5b92c3 100644--- a/crates/utopia-extract/src/open.rs+++ b/crates/utopia-extract/src/open.rs@@ -15,7 +15,7 @@//!//! ```json//! {"e": [["Harbor Bridge", "bridge", 1], ["farmland", "land", 0]],-//! "s": [["city council of Westbrook", "awarded", "paving contract", null, {"to": "Brightway Builders", "amount": "$2 million"}, "March 4, 2011", null, "The city council of Westbrook awarded the paving contract for the Harbor Bridge to Brightway Builders for $2 million on March 4, 2011."]],+//! "s": [["The city council of Westbrook awarded the paving contract for the Harbor Bridge to Brightway Builders for $2 million on March 4, 2011.", "city council of Westbrook", "awarded", "paving contract", null, {"to": "Brightway Builders", "amount": "$2 million"}, "March 4, 2011", null]],//! "n": [["Brightway Builders", "Brightway", "Brightway Builders, known locally as Brightway, is based in Port Ellen."]]}//! ```//!@@ -25,8 +25,8 @@//! ("the Harbor Treaty signed on May 3, 1998" 叫 "Harbor Treaty"),也不带数字//! ("about 40 hectares of farmland" 是 "farmland" 加一条 `area` = "about 40 hectares")。//! 没有 id。-//! - `s`:`[主语名, 短语, 宾语名或 null, 字面值或 null, 限定词对象或 null, 何时或 null,-//! 何时终止或 null, 引文]`。主语与宾语是 `e` 里(或提示词已知清单里)的名字,拼写一模+//! - `s`:`[引文, 主语名, 短语, 宾语名或 null, 字面值或 null, 限定词对象或 null, 何时或 null,+//! 何时终止或 null]`。引文在第一格:模型先抄下原句,再从那句写陈述(先引后述)。主语与宾语是 `e` 里(或提示词已知清单里)的名字,拼写一模//! 一样。宾语与字面值恰有一个(解析层不裁,两个都给调用方看)。一句话里超过两方参与、//! 或者链接带着数额、头衔、条件、比较时,主要的一对进主宾,其余进限定词,键是原文里//! 说明其角色的一两个词,值是列出的东西的名字或原文的话。何时/终止照原文的字抄,@@ -106,7 +106,7 @@ You read one passage of a document and write down what it states, in the passageOutput one JSON object and nothing else, shaped like this:\n\\n\{\"e\": [[\"name or description\", \"kind word\", 1]],\n\- \"s\": [[\"subject name\", \"relation phrase as written\", \"object name\", null, {\"to\": \"name of a listed thing\", \"amount\": \"$2 million\"}, \"when it holds or happened, as written\", \"when it ended, as written\", \"the sentence that states it, verbatim\"]],\n\+ \"s\": [[\"the sentence that states it, verbatim\", \"subject name\", \"relation phrase as written\", \"object name\", null, {\"to\": \"name of a listed thing\", \"amount\": \"$2 million\"}, \"when it holds or happened, as written\", \"when it ended, as written\"]],\n\\"n\": [[\"name as listed in e\", \"another name\", \"the sentence that uses it, verbatim\"]]}\n\\n\1. \"e\" lists the things the passage talks about, one entry each: [name, kind, named]. Every \@@ -120,9 +120,11 @@ particular it is here: \"the company\", \"the enterprises visited\", \"patients\\"the northern wing\" are described things, named by the words that say what they are. kind is \what the thing is, in two or three words, as the passage says it. A name never carries a figure: \"about 40 hectares of farmland\" is the thing \\"farmland\" with a statement \"area\" = \"about 40 hectares\". List each thing once.\n\-2. \"s\" lists the statements, one entry each: [subject, phrase, object, value, qualifiers, \-when, ended, quote]. subject and object are names, written exactly as listed in \"e\" or in \-the list of things already recorded, with the same spelling every time. Exactly one of object \+2. \"s\" lists the statements, one entry each: [quote, subject, phrase, object, value, \+qualifiers, when, ended]. quote comes first: the sentence of the passage that states it, copied \+verbatim, one sentence, and when two sentences are needed, the one that carries the link; the \+rest of the entry is written from that sentence. subject and object are names, written exactly \+as listed in \"e\" or in the list of things already recorded, with the same spelling every time. Exactly one of object \and value is set; the other is null.\n\A statement with an object links two things. phrase is how the passage says the link, \lowercase, in the passage's own words; do not translate it into any vocabulary of your own. It \@@ -178,9 +180,8 @@ the coating are part of the description.\n\words that say when it stopped, each copied exactly as written (\"March 4, 2011\", \"去年冬天\", \\"by the end of next season\"). Never compute, convert or normalise a date, and never write one \the passage does not. Each is null when the passage gives none.\n\-6. quote is the sentence of the passage that states it, copied verbatim: one sentence, and when \-two sentences are needed, the one that carries the link. Every \"s\" and \"n\" entry carries its \-own quote.\n\+6. Every \"s\" and \"n\" entry carries its own quote, copied verbatim from the passage: the first \+slot of an \"s\" entry, the last slot of an \"n\" entry.\n\7. \"n\" lists other names, one entry each: [name as listed, other name, quote] — a short form, \a former name, a spelling in another script that this passage uses for a thing in \"e\" or a \thing already recorded. Only names actually written in the passage; never a pronoun or a \@@ -193,7 +194,7 @@ passage states nothing, output {\"e\":[],\"s\":[],\"n\":[]}.\n\Example. The passage \"The city council awarded the paving contract to Brightway Builders for \$2 million on March 4, 2011.\" gives:\n\{\"e\": [[\"city council\", \"council\", 0], [\"paving contract\", \"contract\", 0], [\"Brightway Builders\", \"builders\", 1]],\n\- \"s\": [[\"city council\", \"awarded\", \"paving contract\", null, {\"to\": \"Brightway Builders\", \"amount\": \"$2 million\"}, \"March 4, 2011\", null, \"The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011.\"]],\n\+ \"s\": [[\"The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011.\", \"city council\", \"awarded\", \"paving contract\", null, {\"to\": \"Brightway Builders\", \"amount\": \"$2 million\"}, \"March 4, 2011\", null]],\n\\"n\": []}";/// 构造开放抽取的两条消息:常量系统消息 + `Document:` / 开头 / 已知实体 / `Passage:`。@@ -366,24 +367,24 @@ fn parse_entity(arr: &[Value]) -> Option<OpenEntity> {Some(OpenEntity { name, kind, named })}-/// `[subject, phrase, object, value, qualifiers, when, ended, quote]`。八格都得在:短了的+/// `[quote, subject, phrase, object, value, qualifiers, when, ended]`。八格都得在:短了的/// 多半是截断修补切出来的半条,引文都没有,收下也没法核对。主语与短语缺不得;其余格/// 放错了类型只是那一格为 None。宾语与字面值同时有则两个都留给调用方fn parse_statement(arr: &[Value]) -> Option<OpenStatement> {if arr.len() < 8 {return None;}- let subject = text_at(arr, 0)?.to_string();- let phrase = text_at(arr, 1)?.to_string();+ let subject = text_at(arr, 1)?.to_string();+ let phrase = text_at(arr, 2)?.to_string();Some(OpenStatement {subject,phrase,- object: opt_text(&arr[2]),- value: literal(&arr[3]),- qualifiers: qualifiers(&arr[4]),- when: opt_text(&arr[5]),- ended: opt_text(&arr[6]),- quote: opt_text(&arr[7]),+ object: opt_text(&arr[3]),+ value: literal(&arr[4]),+ qualifiers: qualifiers(&arr[5]),+ when: opt_text(&arr[6]),+ ended: opt_text(&arr[7]),+ quote: opt_text(&arr[0]),})}@@ -459,12 +460,12 @@ mod tests {/// 一份完整的紧凑回复:本块列出的与已知清单里的名字、两种限定词、起止时间、别名const FULL: &str = r#"{"e": [["city council", "council", 0], ["paving contract", "contract", 0],["Brightway Builders", "builders", 1], ["northern wing", "wing", 0]],- "s": [["city council", "awarded", "paving contract", null, {"to": "Brightway Builders", "amount": "$2 million"}, "March 4, 2011", null,- "The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011."],- ["Nebula Technologies Inc.", "leased", "northern wing", null, {"from": "Harbor Estates", "term": "ten years"}, "from 2010", "until the end of 2019",- "Nebula (formerly Starlight Labs) leased the northern wing from Harbor Estates from 2010 until the end of 2019."],- ["paving contract", "worth", null, "$2 million", null, null, null,- "The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011."]],+ "s": [["The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011.",+ "city council", "awarded", "paving contract", null, {"to": "Brightway Builders", "amount": "$2 million"}, "March 4, 2011", null],+ ["Nebula (formerly Starlight Labs) leased the northern wing from Harbor Estates from 2010 until the end of 2019.",+ "Nebula Technologies Inc.", "leased", "northern wing", null, {"from": "Harbor Estates", "term": "ten years"}, "from 2010", "until the end of 2019"],+ ["The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011.",+ "paving contract", "worth", null, "$2 million", null, null, null]],"n": [["Nebula Technologies Inc.", "Starlight Labs", "Nebula (formerly Starlight Labs) leased the northern wing from Harbor Estates from 2010 until the end of 2019."]]}"#;/// serde_json 的对象按键排序(没开 preserve_order),限定词的顺序不承载意义@@ -551,11 +552,11 @@ mod tests {}/// 截在一条陈述的限定词之后:修补退到那个 `}`,留下的半条不够八格,计入 skipped- /// 而不是收成一条没引文的陈述+ /// 而不是收成一条只有引文的陈述#[test]fn a_half_statement_left_by_the_repair_is_counted() {let raw = r#"{"e": [["A", "thing", 1]],- "s": [["A", "is", null, "b", null, null, null, "A is b."], ["A", "was", null, "c", {"at": "home"}, "in 20"#;+ "s": [["A is b.", "A", "is", null, "b", null, null, null], ["A was c.", "A", "was", null, "c", {"at": "home"}, "in 20"#;let x = parse_open_response(raw).unwrap();assert!(x.truncated);assert_eq!(x.statements.len(), 1);@@ -567,7 +568,7 @@ mod tests {#[test]fn a_cut_off_reply_without_any_closing_brace_is_still_repaired() {let raw = r#"{"e": [["A", "thing", 1]],- "s": [["A", "is", null, "b", null, null, null, "A is b."], ["A", "was", null, "c", nu"#;+ "s": [["A is b.", "A", "is", null, "b", null, null, null], ["A was c.", "A", "was", null, "c", nu"#;let x = parse_open_response(raw).unwrap();assert!(x.truncated);assert_eq!(x.entities.len(), 1);@@ -586,8 +587,8 @@ mod tests {#[test]fn a_statement_with_seven_slots_is_skipped_and_counted() {let raw = r#"{"e": [["A", "thing", 1]],- "s": [["A", "is", null, "b", null, null, null],- ["A", "is", null, "b", null, null, null, "A is b."]]}"#;+ "s": [["A is b.", "A", "is", null, "b", null, null],+ ["A is b.", "A", "is", null, "b", null, null, null]]}"#;let x = parse_open_response(raw).unwrap();assert_eq!(x.statements.len(), 1);assert_eq!(x.statements[0].quote.as_deref(), Some("A is b."));@@ -599,8 +600,8 @@ mod tests {#[test]fn a_number_in_the_value_slot_becomes_text() {let raw = r#"{"e": [["a", "k", 1]],- "s": [["a", "cut water use", null, 12, null, null, null, "q"],- ["a", "ratio", null, 1.5, null, null, null, "q"]]}"#;+ "s": [["q", "a", "cut water use", null, 12, null, null, null],+ ["q", "a", "ratio", null, 1.5, null, null, null]]}"#;let x = parse_open_response(raw).unwrap();assert_eq!(x.statements[0].value.as_deref(), Some("12"));assert_eq!(x.statements[1].value.as_deref(), Some("1.5"));@@ -626,10 +627,10 @@ mod tests {#[test]fn malformed_items_are_counted_not_fatal() {let raw = r#"{"e": [["A", "thing", 1], [""], [5, "x"], "not an array", []],- "s": [["A", "is", null, "b", null, null, null, "A is b."],- [null, "is", null, "b", null, null, null, "A is b."],- ["A", " ", null, "b", null, null, null, "A is b."],- ["A", "is", null, "b", null, null, null],+ "s": [["A is b.", "A", "is", null, "b", null, null, null],+ ["A is b.", null, "is", null, "b", null, null, null],+ ["A is b.", "A", " ", null, "b", null, null, null],+ ["A is b.", "A", "is", null, "b", null, null],{"subject": "A"}],"n": [["A", "Ay", "A is b."], ["A"], [{}, "Ay", "q"], ["", "Ay", "q"]]}"#;let x = parse_open_response(raw).unwrap();@@ -645,7 +646,7 @@ mod tests {#[test]fn a_wrong_type_in_an_optional_slot_is_null_not_malformed() {let raw = r#"{"e": [["A", "thing", 1]],- "s": [["A", "is", 3, true, "not an object", [2019], {"y": 2020}, 7]]}"#;+ "s": [[7, "A", "is", 3, true, "not an object", [2019], {"y": 2020}]]}"#;let x = parse_open_response(raw).unwrap();assert_eq!(x.statements.len(), 1);let s = &x.statements[0];@@ -662,7 +663,7 @@ mod tests {#[test]fn qualifiers_take_strings_and_numbers_and_drop_the_rest() {let raw = r#"{"e": [["a", "k", 1]],- "s": [["a", "cut water use", null, "12%", {"compared to": "2019", "at": "Harbor Estates", "by": 3, "ratio": 1.5, "flag": true, "who": null, "list": ["x"], "": "x", "blank": " "}, null, null, "q"]]}"#;+ "s": [["q", "a", "cut water use", null, "12%", {"compared to": "2019", "at": "Harbor Estates", "by": 3, "ratio": 1.5, "flag": true, "who": null, "list": ["x"], "": "x", "blank": " "}, null, null]]}"#;let x = parse_open_response(raw).unwrap();assert_eq!(by_key(x.statements[0].qualifiers.clone()),@@ -680,7 +681,7 @@ mod tests {#[test]fn every_string_is_trimmed() {let raw = r#"{"e": [[" A ", " thing ", 1]],- "s": [[" A ", " is ", " B ", null, {" at ": " home "}, " in 2019 ", " until 2020 ", " A is B. "]],+ "s": [[" A is B. ", " A ", " is ", " B ", null, {" at ": " home "}, " in 2019 ", " until 2020 "]],"n": [[" A ", " Ay ", " A is B. "]]}"#;let x = parse_open_response(raw).unwrap();assert_eq!(x.entities[0].name, "A");@@ -702,7 +703,7 @@ mod tests {#[test]fn a_statement_with_both_object_and_value_is_passed_through() {let raw = r#"{"e": [["A", "thing", 1], ["B", "thing", 1]],- "s": [["A", "is", "B", "b", null, null, null, "A is b."]]}"#;+ "s": [["A is b.", "A", "is", "B", "b", null, null, null]]}"#;let x = parse_open_response(raw).unwrap();assert_eq!(x.statements.len(), 1);assert_eq!(x.statements[0].object.as_deref(), Some("B"));@@ -773,7 +774,7 @@ mod tests {assert!(system.contains("copied verbatim"));assert!(system.contains("write nothing about them"));assert!(system.contains(- "\"s\": [[\"city council\", \"awarded\", \"paving contract\", null, {\"to\": \"Brightway Builders\""+ "\"s\": [[\"The city council awarded the paving contract to Brightway Builders for $2 million on March 4, 2011.\", \"city council\", \"awarded\", \"paving contract\", null, {\"to\": \"Brightway Builders\""));let doc = user.find("Document: annual-report.txt").unwrap();diff --git a/docs/design/extraction.md b/docs/design/extraction.mdindex c2dc91d6..77bec2a3 100644--- a/docs/design/extraction.md+++ b/docs/design/extraction.md@@ -16,9 +16,9 @@ are extraction's rules [0039].**Output, the compact contract** (`utopia_extract::open`). `e`: things, each a name or adescription, the document's kind word, and whether it is named or described. `s`: statements, each-naming its subject and object in words, the relation phrase as written, an object name or a literal-value, qualifiers keyed by the document's role words, `when` / `ended` time words, and its own-verbatim quote. `n`: other names the document gives a thing. Statements name their sides in words+opening with its own verbatim quote and then naming its subject and object in words, the relation+phrase as written, an object name or a literal value, qualifiers keyed by the document's role+words, and `when` / `ended` time words. `n`: other names the document gives a thing. Statements name their sides in wordsand carry their own quote because a numbered contract mixed the ids up on dense passages (misworded10 to 14% with ids, 2 to 7% with names) [#731]. Parsing is per item: a malformed item is counted anda truncated reply is repaired to its last complete item.@@ -91,7 +91,9 @@ qualifiers, time words and quote span, and a nod writes an open statement [0015,a list says it is a name; the same rule for quotes and time words [0041, 0044 d8].- **A described thing is an entity** and gets no name fact, so a description never bridges twodocuments [#731].-- **Named sides and per-statement quotes** cost tokens and buy the not-stated rate [#731].+- **Named sides and per-statement quotes** cost tokens and buy the not-stated rate [#731]; the+ quote comes first so the statement is written from the copied sentence [Huang 2024,+ prior-work item 9, #748].- **Every drop is a row** because an append-only ledger with evidence on every fact cannot dropsilently [0001].- **Temperature 0** because the endpoint default swung one paragraph between 8 and 31 statementsdiff --git a/docs/design/prior-work.md b/docs/design/prior-work.mdindex 54f0b137..83be539e 100644--- a/docs/design/prior-work.md+++ b/docs/design/prior-work.md@@ -123,7 +123,7 @@ built.matcher. The fact-checking side reached the same place: fully atomic facts are not the rightunit, decontextuality and minimality pull against each other [Gunjal 2024, Choi 2021], and moredecomposition is not monotonically better [Hu 2025]. Our rule: one statement is one complete- proposition; coordinated objects split; a verb chain sharing one object does not.+ proposition; coordinated objects split; a verb chain sharing one object does not (#743).4. **The decomposition is itself a model output.** Change the decomposition prompt and everydownstream number moves [Wanner 2024]; a segmentation pre-pass can drop entities for strongmodels [Pommeret 2026]. The contract is versioned in code and each bench round names its@@ -134,8 +134,10 @@ built.inside the extractor is what moved OpenIE6 [Kolluru 2020]; it can be done without a parser[Wang 2023b]; split-and-rephrase is the general task [Narayan 2017]; irrelevant context is anamed failure of LLM OpenIE [Ling 2023]. The judge and the coverage script measure the dense- NVDA sentences separately (#731). *Open:* the figures dropped from dense sentences get a- second-look pass.+ NVDA sentences separately (#731). A structural count of the money and percentage figures that+ no statement of their chunk carries (`scripts/bench/figures.mjs`, #748) found 2 of 83 in prose+ on the 25-document batch, 2 of 35 on the NVDA releases and 0 of 451 in their tables, with reasoning+ on at the endpoint; a second-look pass waits for a corpus that loses more.6. **Common nouns as entities.** On Wikipedia text only 42% of open-extraction arguments are namedentities [Gashteovski 2019]; canonicalization meets the same string with different referentsfirst [Galárraga 2014]; the opposite failure, one referent under many strings, is the sparsity@@ -165,8 +167,11 @@ built.9. **The contract's shape has a cost.** Format restrictions degrade reasoning [Tam 2024]; guidelineprompts are not reliably obeyed without training [Sainz 2024]; quoting before stating improvesgrounding [Huang 2024]; forcing citations finer than a sentence hurts [Wang 2026]. The compact- contract was chosen by measurement (#731). *Open:* the quote comes before the statement in `s`;- clause-level quotes measured against minimal ones.+ contract was chosen by measurement (#731); the quote is the first slot of a statement, so the+ model copies the sentence before it writes from it (#748: on the four NVDA releases 909 statements+ against 901, misworded 5% in both, prose misworded 6% against 9% and prose that does not read+ alone 5% against 11%). *Open:* clause-level quotes measured against minimal ones; a run-to-run+ stability number.10. **Numbers need their own patterns.** Open numerical extraction is its own problem[Saha 2017]; the dense-sentence figures we drop are that class. *Open:* the second-look pass ofitem 5.diff --git a/scripts/bench/figures.mjs b/scripts/bench/figures.mjsnew file mode 100644index 00000000..fad79b99--- /dev/null+++ b/scripts/bench/figures.mjs@@ -0,0 +1,50 @@+// 数字覆盖:一个库里,段落写了的钱、百分比、带单位或带千分位的数,有多少没进它那块的任何一条+// 陈述(宾语、字面值、限定词、时间词都算)。只看形状,不看词;年份不算数。表格行(以 | 起头)+// 与散文分开数。用法:node scripts/bench/figures.mjs --kb <id>;库用 BENCH_PSQL 指定,同 judge_open。+import { execFileSync } from "node:child_process";+const KB = process.argv[process.argv.indexOf("--kb") + 1];+if (!KB || KB.startsWith("--")) { console.error("要一个 --kb <id>"); process.exit(1); }+function psql(sql) {+ const cmd = process.env.BENCH_PSQL || "docker exec -e PGPASSWORD=utopia landscapebi-db-1 psql -U utopia -d utopia -tAc";+ const parts = cmd.split(" ");+ return execFileSync(parts[0], [...parts.slice(1), sql], { encoding: "utf8", maxBuffer: 64 << 20 }).trim();+}+const SEP = String.fromCharCode(31);+const chunks = psql(`SELECT concat_ws(chr(31), c.id, d.filename, replace(c.text, chr(10), chr(30))) FROM chunks c JOIN documents d ON d.id=c.document_id WHERE d.kb_id='${KB}' ORDER BY d.filename, c.id`)+ .split("\n").filter(Boolean).map(l => { const [id, file, t] = l.split(SEP); return { id, file, text: t.replace(/\x1e/g, "\n") }; });+const stmts = psql(`SELECT concat_ws(chr(31), fe.chunk_id, s.canonical_name, f.phrase, coalesce(o.canonical_name, f.object_value #>> '{value}', ''),+ coalesce((SELECT string_agg(coalesce(sq.value #>> '{}', qe.canonical_name, ''), ' ') FROM statement_qualifiers sq LEFT JOIN entities qe ON qe.id=sq.entity_id WHERE sq.fact_id=f.id), ''),+ coalesce((SELECT string_agg(tm.text, ' ') FROM time_mentions tm WHERE tm.fact_id=f.id), ''))+ FROM facts f JOIN fact_evidence fe ON fe.fact_id=f.id JOIN entities s ON s.id=f.subject_id LEFT JOIN entities o ON o.id=f.object_id+ WHERE f.kb_id='${KB}' AND f.layer='open' AND f.invalidated_at IS NULL`)+ .split("\n").filter(Boolean).map(l => { const [chunk, ...rest] = l.split(SEP); return { chunk, text: rest.join(" ") }; });+const byChunk = new Map();+for (const s of stmts) byChunk.set(s.chunk, (byChunk.get(s.chunk) || "") + " " + s.text);+const norm = (s) => s.replace(/[,\s]/g, "").toLowerCase();+// 一个「数字」:可带 $ / 百分号 / billion|million 单位;两位以上或带符号+const FIG = /(?:\$\s?)?\d[\d,]*(?:\.\d+)?(?:\s?(?:billion|million|thousand|%|percent|bps|亿元|万元|亿美元|万美元|亿|万|元|个百分点|吨|人次|人|户|家|件|亩|公顷|例|种|项|台|辆))?/gi;+const isFigure = (m) => /[$%]|billion|million|thousand|percent|bps|亿|万|元|个百分点|吨|人|户|家|件|亩|公顷|例|种|项|台|辆/i.test(m) || /\d,\d{3}/.test(m);+const perFile = {};+const samples = [];+for (const c of chunks) {+ const have = norm(byChunk.get(c.id) || "");+ for (const line of c.text.split("\n")) {+ const table = line.trim().startsWith("|");+ const sentences = table ? [line] : line.split(/(?<=[.;。;])\s+/);+ for (const sent of sentences) {+ for (const m of sent.match(FIG) || []) {+ if (!isFigure(m)) continue;+ if (/^(19|20)\d\d$/.test(m)) continue; // 年份不算数字+ const key = norm(m).replace(/^\$/, "");+ const pf = perFile[c.file] ||= { prose: [0, 0], table: [0, 0] };+ const slot = pf[table ? "table" : "prose"];+ slot[0]++;+ if (!have.includes(key)) { slot[1]++; if (!table && samples.length < 40) samples.push([c.file, m, sent.trim().slice(0, 160)]); }+ }+ }+ }+}+let T = [0, 0], P = [0, 0];+for (const [f, v] of Object.entries(perFile)) { console.log(`${f}: prose ${v.prose[1]}/${v.prose[0]} missing, table ${v.table[1]}/${v.table[0]} missing`); P[0]+=v.prose[0]; P[1]+=v.prose[1]; T[0]+=v.table[0]; T[1]+=v.table[1]; }+console.log(`ALL: prose ${P[1]}/${P[0]} (${(100*P[1]/P[0]).toFixed(0)}%) missing, table ${T[1]}/${T[0]} (${(100*T[1]/T[0]).toFixed(0)}%) missing`);+console.log("-- prose samples:"); for (const [f, m, s] of samples) console.log(` [${f.replace("nvda-","")}] ${m} ⇐ ${s}`);