World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A table is a table in every format (#750)
WaylandYang23a227c14942
diff --git a/crates/utopia-ingest/examples/render_doc.rs b/crates/utopia-ingest/examples/render_doc.rsnew file mode 100644index 00000000..370e1fff--- /dev/null+++ b/crates/utopia-ingest/examples/render_doc.rs@@ -0,0 +1,13 @@+//! 把一个文件按摄入层的读法打印成文本:`cargo run -p utopia-ingest --example render_doc -- <文件>`。+//! 看一张表、一份 docx 被读成什么样,比等一轮抽取快得多(#744 的教训)。+fn main() -> anyhow::Result<()> {+ let path = std::env::args().nth(1).expect("要一个文件路径");+ let bytes = std::fs::read(&path)?;+ let name = std::path::Path::new(&path)+ .file_name()+ .and_then(|n| n.to_str())+ .unwrap_or("file");+ let parsed = utopia_ingest::parse(name, &bytes)?;+ print!("{}", parsed.text);+ Ok(())+}diff --git a/crates/utopia-ingest/src/parsers.rs b/crates/utopia-ingest/src/parsers.rsindex 73401b60..b481a90c 100644--- a/crates/utopia-ingest/src/parsers.rs+++ b/crates/utopia-ingest/src/parsers.rs@@ -19,10 +19,110 @@ pub fn pdf(bytes: &[u8]) -> anyhow::Result<String> {pdf_extract::extract_text_from_mem(bytes).context("PDF text-layer extraction failed")}-/// docx: 解压 word/document.xml,取 w:t 文本、w:p 分段。+/// docx:解压 word/document.xml。正文 w:t 取字、w:p 分段;表格(w:tbl)收成网格交给+/// `table::render_grid`,和 HTML 表走同一套渲染:w:gridSpan 跨列,上下合并(w:vMerge)的+/// 续格留空,格内段落的左缩进 w:ind 当内边距(小节行靠它折进标签)。套在格子里的表按格子+/// 文字处理。pub fn docx(bytes: &[u8]) -> anyhow::Result<String> {let xml = read_zip_entry(bytes, "word/document.xml").context("Malformed docx structure")?;- extract_xml_text(&xml, "w:t", "w:p")+ docx_xml_to_text(&xml)+}++type GridRow = Vec<(String, usize, u32)>;++pub(crate) fn docx_xml_to_text(xml: &str) -> anyhow::Result<String> {+ let mut reader = Reader::from_str(xml);+ let mut out = String::new();+ let mut in_text = false;+ // 套了几层 w:tbl;只有最外层收成网格+ let mut depth = 0usize;+ let mut rows: Vec<GridRow> = Vec::new();+ let mut row: GridRow = Vec::new();+ let mut cell: Option<(String, usize, u32)> = None;+ let attr = |e: &quick_xml::events::BytesStart<'_>, name: &str| -> Option<String> {+ e.attributes()+ .flatten()+ .find(|a| a.key.as_ref() == name)+ .map(|a| a.value.to_string())+ };+ loop {+ match reader.read_event() {+ Ok(Event::Start(e)) => match e.name().as_ref() {+ "w:t" => in_text = true,+ "w:tbl" => {+ depth += 1;+ if depth == 1 {+ rows.clear();+ }+ }+ "w:tr" if depth == 1 => row = Vec::new(),+ "w:tc" if depth == 1 => cell = Some((String::new(), 1, 0)),+ _ => {}+ },+ Ok(Event::Empty(e)) => match e.name().as_ref() {+ "w:gridSpan" => {+ if let Some(c) = cell.as_mut() {+ if let Some(v) = attr(&e, "w:val").and_then(|v| v.parse::<usize>().ok()) {+ c.1 = v.max(1);+ }+ }+ }+ // 格子里第一段的左缩进(twips,1/20 pt)+ "w:ind" => {+ if let Some(c) = cell.as_mut().filter(|c| c.0.is_empty()) {+ if let Some(v) = attr(&e, "w:left")+ .or_else(|| attr(&e, "w:start"))+ .and_then(|v| v.parse::<u32>().ok())+ {+ c.2 = v / 20;+ }+ }+ }+ "w:tab" => match cell.as_mut() {+ Some(c) => c.0.push(' '),+ None => out.push(' '),+ },+ "w:br" | "w:cr" if cell.is_none() => out.push('\n'),+ _ => {}+ },+ Ok(Event::End(e)) => match e.name().as_ref() {+ "w:t" => in_text = false,+ "w:p" => match cell.as_mut() {+ Some(c) => c.0.push(' '),+ None => out.push('\n'),+ },+ "w:tc" if depth == 1 => {+ if let Some(c) = cell.take() {+ row.push(c);+ }+ }+ "w:tr" if depth == 1 => rows.push(std::mem::take(&mut row)),+ "w:tbl" => {+ depth = depth.saturating_sub(1);+ if depth == 0 {+ if let Some(md) = crate::table::render_grid(&rows, false) {+ out.push('\n');+ out.push_str(&md);+ out.push_str("\n\n");+ }+ rows.clear();+ }+ }+ _ => {}+ },+ Ok(Event::Text(t)) if in_text => {+ let s = t.xml_content(quick_xml::XmlVersion::Implicit1_0);+ match cell.as_mut() {+ Some(c) => c.0.push_str(&s),+ None => out.push_str(&s),+ }+ }+ Ok(Event::Eof) => break,+ Err(e) => anyhow::bail!("XML parse error: {e}"),+ _ => {}+ }+ }+ Ok(out)}/// pptx: 按页码顺序解析 ppt/slides/slideN.xml,取 a:t 文本。@@ -68,19 +168,36 @@ pub fn spreadsheet(bytes: &[u8]) -> anyhow::Result<String> {if range.is_empty() {continue;}- out.push_str(&format!("\n# Sheet: {sheet_name}\n"));- for row in range.rows().take(2000) {- let line: Vec<String> = row- .iter()- .map(|c| match c {- Data::Empty => String::new(),- other => other.to_string(),- })- .collect();- if line.iter().any(|s| !s.is_empty()) {- out.push_str(&line.join("\t"));+ out.push_str(&format!("\n# Sheet: {sheet_name}\n\n"));+ // 每张表按网格渲染成带列头的 Markdown 表;一格字都没有的表退回制表符分隔+ let rows: Vec<GridRow> = range+ .rows()+ .take(2000)+ .map(|row| {+ row.iter()+ .map(|c| {+ let text = match c {+ Data::Empty => String::new(),+ other => other.to_string(),+ };+ (text, 1, 0)+ })+ .collect::<GridRow>()+ })+ .filter(|line| line.iter().any(|(s, _, _)| !s.is_empty()))+ .collect();+ match crate::table::render_grid(&rows, true) {+ Some(md) => {+ out.push_str(&md);out.push('\n');}+ None => {+ for line in &rows {+ let cells: Vec<&str> = line.iter().map(|(s, _, _)| s.as_str()).collect();+ out.push_str(&cells.join("\t"));+ out.push('\n');+ }+ }}}Ok(out)@@ -98,16 +215,30 @@ pub fn csv_text(bytes: &[u8], tsv: bool) -> anyhow::Result<String> {.flexible(true).has_headers(false).from_reader(decoded.as_bytes());- let mut out = String::new();+ let mut rows: Vec<GridRow> = Vec::new();for (i, record) in reader.records().enumerate() {if i >= 10_000 {break;}let record = record?;- out.push_str(&record.iter().collect::<Vec<_>>().join(" | "));- out.push('\n');+ rows.push(record.iter().map(|s| (s.to_string(), 1, 0)).collect());}- Ok(out)+ // 第一条记录是列头(csv 的惯例);渲染不出表时退回竖线分隔的行+ Ok(match crate::table::render_grid(&rows, true) {+ Some(md) => md + "\n",+ None => {+ rows.iter()+ .map(|r| {+ r.iter()+ .map(|(s, _, _)| s.as_str())+ .collect::<Vec<_>>()+ .join(" | ")+ })+ .collect::<Vec<_>>()+ .join("\n")+ + "\n"+ }+ })}// ---- 工具 ----@@ -146,3 +277,77 @@ fn extract_xml_text(xml: &str, text_tag: &str, para_tag: &str) -> anyhow::Result}Ok(out)}++#[cfg(test)]+mod tests {+ use super::*;++ const DOC: &str = r#"<?xml version="1.0"?><w:document xmlns:w="x"><w:body>+<w:p><w:r><w:t>Segment results</w:t></w:r></w:p>+<w:tbl><w:tr><w:tc><w:p><w:r><w:t>Segment</w:t></w:r></w:p></w:tc><w:tc><w:tcPr><w:gridSpan w:val="2"/></w:tcPr><w:p><w:r><w:t>Revenue</w:t></w:r></w:p></w:tc></w:tr>+<w:tr><w:tc><w:p><w:r><w:t>Cloud</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>$</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>1,200</w:t></w:r></w:p></w:tc></w:tr>+<w:tr><w:tc><w:p><w:r><w:t>Devices</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>$</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>300</w:t></w:r></w:p></w:tc></w:tr></w:tbl>+<w:p><w:r><w:t>After the table.</w:t></w:r></w:p></w:body></w:document>"#;++ /// docx 的表:跨列的列头按列拼,"$" 并回数字,表前后的段落照旧+ #[test]+ fn a_docx_table_is_rendered_under_its_headings() {+ let text = docx_xml_to_text(DOC).unwrap();+ assert!(text.starts_with("Segment results\n"), "{text}");+ assert!(+ text.contains(+ "| Segment | Revenue |\n| --- | --- |\n| Cloud | $1,200 |\n| Devices | $300 |"+ ),+ "{text}"+ );+ assert!(text.ends_with("After the table.\n"), "{text}");+ }++ /// 套在格子里的表不单独成表:它的字算外层格子的字+ #[test]+ fn a_nested_docx_table_is_its_cells_text() {+ let xml = r#"<w:document xmlns:w="x"><w:body><w:tbl>+<w:tr><w:tc><w:p><w:r><w:t>Region</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>Notes</w:t></w:r></w:p></w:tc></w:tr>+<w:tr><w:tc><w:p><w:r><w:t>North</w:t></w:r></w:p></w:tc><w:tc><w:tbl><w:tr><w:tc><w:p><w:r><w:t>inner</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>7</w:t></w:r></w:p></w:tc></w:tr></w:tbl></w:tc></w:tr>+</w:tbl></w:body></w:document>"#;+ let text = docx_xml_to_text(xml).unwrap();+ assert_eq!(text.matches("| --- |").count(), 1, "{text}");+ assert!(text.contains("| North | inner 7 |"), "{text}");+ }++ /// csv 的第一条记录是列头,哪怕列头是年份+ #[test]+ fn a_csv_is_a_table_whose_first_record_is_the_header() {+ let text = csv_text(b"Item,2025,2024\nRevenue,10,8\nCost,4,3\n", false).unwrap();+ assert_eq!(+ text,+ "| Item | 2025 | 2024 |\n| --- | --- | --- |\n| Revenue | 10 | 8 |\n| Cost | 4 | 3 |\n"+ );+ }++ /// 列头之后只填了一个序号的行也是记录,不是列头的第二行+ #[test]+ fn a_lone_number_row_after_the_header_is_a_record() {+ let text = csv_text(+ b"no,name,score+1,,+2,Ada,9+",+ false,+ )+ .unwrap();+ assert!(text.starts_with("| no | name | score |"), "{text}");+ assert!(text.contains("| 1 | | |"), "{text}");+ assert!(text.contains("| 2 | Ada | 9 |"), "{text}");+ }++ /// 一格数字都没有的 csv 也是表:标签列是最左那格+ #[test]+ fn a_csv_of_words_is_still_a_table() {+ let text = csv_text(b"name,role\nAda,engineer\nGrace,admiral\n", false).unwrap();+ assert!(+ text.starts_with("| name | role |\n| --- | --- |\n| Ada | engineer |"),+ "{text}"+ );+ }+}diff --git a/crates/utopia-ingest/src/table.rs b/crates/utopia-ingest/src/table.rsindex 97f82fd7..dc03b3ab 100644--- a/crates/utopia-ingest/src/table.rs+++ b/crates/utopia-ingest/src/table.rs@@ -14,6 +14,10 @@//! `blocks` / `chunker`,它们本来就让说明句和表头跟着每一块走。//!//! 只处理最外层且不含内层表的表;套着的表留给 htmd 原路。+//!+//! docx、电子表格和 csv 的表不经过 DOM:解析器把格子收成网格交给 [`render_grid`],之后的+//! 分类与渲染和 HTML 表一模一样。电子表格和 csv 的第一排(至少两格有字的那排)按惯例是+//! 列头,哪怕列头是年份这种数字。use dom_query::{Document, Selection};@@ -317,7 +321,50 @@ fn depth_of(cell: &Cell) -> u32 {/// 一张表 → 说明句(可选)+ 带真表头的 Markdown 表,连同它用的列头(给续表接着用)。/// 表里什么都没有时返回 None。fn render_table(tbl: &Selection<'_>, inherited: &[String]) -> Option<(String, Vec<String>)> {- let mut rows = grid(tbl);+ render_rows(grid(tbl), inherited, None)+}++/// 别的解析器用的入口:一行是若干 (文字, 跨几列, 左内边距 pt)。`first_is_header` 为真时,+/// 第一排至少两格有字的行按列头算(电子表格、csv 的惯例),其余行照常分类。+pub(crate) fn render_grid(+ rows: &[Vec<(String, usize, u32)>],+ first_is_header: bool,+) -> Option<String> {+ let rows: Vec<Row> = rows+ .iter()+ .map(|cells| {+ let mut col = 0usize;+ let mut out = Vec::with_capacity(cells.len());+ for (text, span, pad) in cells {+ let span = (*span).max(1);+ out.push(Cell {+ text: clean(text),+ span,+ pad: *pad,+ col,+ });+ col += span;+ }+ Row {+ cells: out,+ width: col,+ }+ })+ .collect();+ let forced = first_is_header+ .then(|| {+ rows.iter()+ .position(|r| r.cells.iter().filter(|c| !c.text.is_empty()).count() >= 2)+ })+ .flatten();+ render_rows(rows, &[], forced).map(|(md, _)| md)+}++fn render_rows(+ mut rows: Vec<Row>,+ inherited: &[String],+ forced_header: Option<usize>,+) -> Option<(String, Vec<String>)> {let width = rows.iter().map(|r| r.width).max().unwrap_or(0);if width == 0 {return None;@@ -333,8 +380,26 @@ fn render_table(tbl: &Selection<'_>, inherited: &[String]) -> Option<(String, Ve// (标签路径, 该行)let mut data: Vec<(String, Row)> = Vec::new();let (mut headers_seen, mut data_seen) = (false, false);- for row in rows.iter_mut() {- match kind(row, headers_seen, data_seen, width) {+ for (i, row) in rows.iter_mut().enumerate() {+ let filled = row.cells.iter().filter(|c| !c.text.is_empty()).count();+ let k = match forced_header {+ Some(h) if i == h => Kind::Header,+ // 列头之后的行是记录,哪怕一格数字都没有;只填了一个数字的行(序号列)也是记录,+ // 只填了一个词的行才按小节算+ Some(h)+ if i > h+ && (filled >= 2+ || row+ .cells+ .iter()+ .find(|c| !c.text.is_empty())+ .is_some_and(|c| is_numeric(&c.text))) =>+ {+ Kind::Data+ }+ _ => kind(row, headers_seen, data_seen, width),+ };+ match k {Kind::Skip => {}Kind::Caption => {if let Some(c) = row.cells.iter().find(|c| !c.text.is_empty()) {@@ -381,6 +446,11 @@ fn render_table(tbl: &Selection<'_>, inherited: &[String]) -> Option<(String, Ve}}}+ // 全是列头没有一行数据的表不存在:一格数字都没有的表,第一排是列头,其余是数据行+ if data.is_empty() && headers.len() >= 2 {+ let rest = headers.split_off(1);+ data.extend(rest.into_iter().map(|r| (String::new(), r)));+ }if data.is_empty() && headers.is_empty() {return None;}diff --git a/docs/design/prior-work.md b/docs/design/prior-work.mdindex fda4d04f..9ad31333 100644--- a/docs/design/prior-work.md+++ b/docs/design/prior-work.md@@ -314,8 +314,12 @@ built.DOM by structure, one row per line under its real headings, the section path folded into thelabel, the caption run kept with every chunk ([sources](sources.md), #744); a heading thatnames a time is `when`, a heading that names a change between periods stays the phrase; a- phrase equal to the value or to the subject is kept and counted. *Open:*- the DOCX, PDF and spreadsheet parsers render their tables the same way; the paragraph units of+ phrase equal to the value or to the subject is kept and counted. DOCX tables, spreadsheets and+ CSV files are laid on the same grid by their parsers (#750): a DOCX cell keeps its span and+ the indent of its first paragraph, a sheet's or a file's first row of two or more cells is its+ header even when the headings are years, and a table with no figure in it is a header row and+ records. *Open:* the text layer of a PDF carries no table structure (a scanned PDF's tables+ arrive from MinerU as pipe tables already); the paragraph units ofthe NVDA cross-domain corpus are recut from rendered rows with their headings, and the referencefacts carry the period; a check that a cell statement under a period column carries `when`.27. **Precision is not confidence, and vague is not coarse.** An indeterminate instant is a set ofdiff --git a/docs/design/sources.md b/docs/design/sources.mdindex 8a118a73..272c2dab 100644--- a/docs/design/sources.md+++ b/docs/design/sources.md@@ -35,7 +35,7 @@ every table that follows it; a row is never split; headings become a breadcrumboverlap; chunk text is verbatim slices so a quote is found by offset; a document is re-chunked onits next reprocess [0039].-**Tables** [#744, prior-work item 26]. An HTML table is rendered from the DOM before the Markdown+**Tables** [#744, #750, prior-work item 26]. An HTML table is rendered from the DOM before the Markdownconverter sees it, by structure alone: cells hidden by style are skipped, spanning cells are laid ona grid, columns empty in every row go, a cell holding only a symbol ("$", ")") rejoins the numberbeside it. Rows are classified by what they hold: a lone cell before any header or data is a@@ -49,7 +49,11 @@ does what it already did: the caption and the header travel with every piece, anfive short paragraphs before a table is its caption, so a statement's chunk names the report, theunit and the period. Nested tables and the tables of other parsers still take the converter'spath with the first row promoted to header. Measured on the NVDA earnings release: 52 chunks-became 32, none of them a table without its header.+became 32, none of them a table without its header. DOCX tables, spreadsheets and CSV files are laid on the same grid by their parsers (#750): a DOCX+cell keeps its column span and the indent of its first paragraph, a sheet's or a CSV file's first+row with two or more cells is its header even when the headings are years, and a table with no+figure in it is read as a header row followed by records; the text layer of a PDF carries no table+structure.**Origin and anchor.** `chunks.origin` (stated, ocr, transcribed, described), `origin_model`, and an`anchor` whose shape is checked per origin (page and box; start, end and speakers; page and image or