Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: provenance — he info --sources, he feed --source/--refeed, he remove --document/--strategy
yifanfeng9795fdeb07ab32
diff --git a/docs/en/cli/commands/feed.md b/docs/en/cli/commands/feed.mdindex 93caf2f..68061b5 100644--- a/docs/en/cli/commands/feed.md+++ b/docs/en/cli/commands/feed.md@@ -92,6 +92,28 @@ The merge process handles:---+## Source Attribution++Pass `--source` with a document id to attribute the document you are feeding:++```bash+he feed ./ka/ doc1.md --source doc-1+```++Attribution records the document in the KA's **source ledger** together with a content hash of the input, which enables:++- **Change detection** — feeding the same source again with unchanged content prints `unchanged (content hash matches) — nothing to do` and skips entirely: **zero LLM calls**. Use `--refeed` to force re-ingestion:++ ```bash+ he feed ./ka/ doc1.md --source doc-1 --refeed+ ```++- **Per-document rollback** — everything this document contributed can later be rolled back with [`he remove --document doc-1`](remove.md)++Inspect the ledger with [`he info --sources`](info.md#source-ledger) or `ka.sources()` in Python. See the [Source Attribution & Provenance](../../python/guides/provenance.md) guide for the full picture.++---+## Workflow Example### Building a Research Knowledge Abstractdiff --git a/docs/en/cli/commands/info.md b/docs/en/cli/commands/info.mdindex 3de2f3c..1821a92 100644--- a/docs/en/cli/commands/info.md+++ b/docs/en/cli/commands/info.md@@ -28,6 +28,7 @@ Shows metadata and statistics for a knowledge abstract:- **Timestamps** — Creation and last update times- **Statistics** — Node count, edge count- **Index status** — Whether search index is built+- **Source ledger** — Contributing documents, shown with `--sources`---@@ -85,6 +86,39 @@ he info ./ka/---+## Source Ledger++Pass `--sources` to also show which documents contributed to the knowledge abstract:++```bash+he info ./ka/ --sources+```++**Output:**+```+Knowledge Abstract Info++Path ./ka/+Template general/biography_graph+Language en+Created 2024-01-15 10:30:00+Updated 2024-01-15 10:35:22+Nodes 25+Edges 32+Index Built++ Source Ledger+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓+┃ Source ID ┃ Raw Items ┃ Content Hash ┃+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │+└────────────┴───────────┴────────────────────────────────────┘+```++The ledger is read from `sources_nodes.json` / `sources_edges.json` in the KA directory and is written when documents are ingested with `--source` (see [`he feed`](feed.md)). If no ledger exists, the command prints a `No source ledger found` hint instead of the table.++---+## Use Cases### Verify Extractiondiff --git a/docs/en/cli/commands/parse.md b/docs/en/cli/commands/parse.mdindex 1cb19ba..00e536e 100644--- a/docs/en/cli/commands/parse.md+++ b/docs/en/cli/commands/parse.md@@ -40,6 +40,8 @@ Extract from a single file:he parse document.md -t general/biography_graph -o ./output/ -l en```+Adding `--source` records the document's source attribution, which enables per-document rollback later via [`he remove --document`](remove.md).+### Interactive Template SelectionOmit `-t` to select from available templates:diff --git a/docs/en/cli/commands/remove.md b/docs/en/cli/commands/remove.mdindex 74c78de..04ddeff 100644--- a/docs/en/cli/commands/remove.md+++ b/docs/en/cli/commands/remove.md@@ -19,6 +19,7 @@ he remove KA_PATH [OPTIONS]| `--edit-node TEXT` | — | Node key to **soft-edit** (LLM rewrites it minus `--fact`) || `--edit-edge TEXT` | — | Edge key to **soft-edit** || `--document TEXT` | — | **Roll back** all knowledge contributed by a source document (requires the KA to have been fed with `--source`) |+| `--strategy` | `exact` | Rollback strategy for `--document`: `exact` or `touched` || `--fact TEXT` | — | The fact to remove from the `--edit-node` / `--edit-edge` item || `--instruction TEXT` | — | Free-form edit instruction (alternative to `--fact`) || `--dry-run` | off | Preview the change without persisting it |@@ -92,7 +93,7 @@ he build-index ./ka/## Document Rollback: remove a whole source document-When documents were ingested with a source attribution, their entire contribution can be rolled back — including keys that other documents also contributed to (those are re-merged from the surviving sources).+When documents were ingested with a source attribution, their entire contribution can be rolled back:```bash# Ingest with attribution@@ -101,9 +102,19 @@ he feed ./ka/ doc2.md --source doc-2# Later: roll back everything doc-1 contributed ("b" remains via doc-2 if shared)he remove ./ka/ --document doc-1++# Or delete every key the document touched, even when other documents share it+he remove ./ka/ --document doc-1 --strategy touched```-Requires the KA's memories to track sources (graph-family KAs do by default). The rollback re-merges affected keys from the surviving sources' raw results — deterministic with classic merge strategies; LLM strategies preserve semantics with approximate wording. If the document has no recorded contributions, the command reports `Nothing matched` and exits successfully.+### Rollback Strategies (`--strategy`)++| Strategy | Behavior |+|----------|----------|+| `exact` (default) | Keys contributed solely by the removed document are deleted; keys **shared** with other documents are re-merged from the surviving sources' raw results — deterministic with classic merge strategies (LLM strategies preserve semantics with approximate wording) |+| `touched` | Every key the document touched is deleted outright, including keys other documents also contributed to |++Requires the KA's memories to track sources (graph-family KAs do by default). If the document has no recorded contributions, the command reports `Nothing matched` and exits successfully.---diff --git a/docs/en/cli/index.md b/docs/en/cli/index.mdindex fdc7b03..c7d8a06 100644--- a/docs/en/cli/index.md+++ b/docs/en/cli/index.md@@ -30,18 +30,18 @@ he --version| Command | Purpose | Common Flags ||---------|---------|--------------|-| `he parse` | Extract knowledge from documents | `-t` template, `-o` output, `-l` language |+| `he parse` | Extract knowledge from documents | `-t` template, `-o` output, `-l` language, `--source` attribution || `he show` | Visualize knowledge graph | — || `he export obsidian` | Export to an Obsidian vault | `-o` output, `--name`, `-f` force || `he export graphml` | Export a pairwise graph to GraphML | `-o` output file || `he export csv` | Export nodes/edges as CSV tables | `-o` directory, `-f` force || `he search` | Semantic search in knowledge abstract | `-n` top-k results || `he talk` | Chat with knowledge abstract | `-i` interactive, `-q` query |-| `he feed` | Add documents incrementally | — |+| `he feed` | Add documents incrementally | `--source` attribution || `he info` | Show knowledge abstract statistics | — || `he build-index` | Build/rebuild search index | `-f` force rebuild || `he clean` | Remove a KA's index (or the whole KA) | `-a` all, `-y` yes |-| `he remove` | Delete nodes/edges by key, or soft-remove a single fact | `--node`, `--edge`, `--edit-node`, `--fact`, `--dry-run` |+| `he remove` | Delete nodes/edges by key, or soft-remove a single fact | `--node`, `--edge`, `--edit-node`, `--fact`, `--dry-run`, `--document`, `--strategy` || `he list` | List templates and methods | `template` or `method` || `he template validate` | Validate a template YAML file | `--json`, `--all` || `he config` | Manage configuration | `init`, `show`, `llm`, `embedder` |diff --git a/docs/en/python/guides/provenance.md b/docs/en/python/guides/provenance.mdnew file mode 100644index 0000000..b287de9--- /dev/null+++ b/docs/en/python/guides/provenance.md@@ -0,0 +1,168 @@+# Source Attribution & Provenance++Track which documents contributed to a knowledge abstract, skip unchanged re-feeds, and roll back any document's contribution without touching the rest.++---++## Overview++Ingesting a document **with attribution** (`--source` on the CLI, `source_id=` in Python) makes the knowledge abstract remember that document: its raw extraction results are kept in a **source ledger**, together with a SHA-256 content hash of the input text.++Provenance gives you three capabilities:++| Capability | CLI | Python |+|------------|-----|--------|+| **Per-document rollback** | `he remove --document ID` | `ka.remove_source(ID)` |+| **Change detection** | `he feed --source` skips unchanged documents | `ka.source_content_hash(ID)` |+| **Audit** | `he info --sources` | `ka.sources()` |++Source tracking is on by default for graph-family KAs (graph, hypergraph, temporal/spatial graphs).++---++## Ingesting with Attribution++```bash+# Attribute at parse time (first document)+he parse doc1.md -t general/graph -o ./ka/ -l en --source doc-1++# Attribute when feeding into an existing KA+he feed ./ka/ doc2.md --source doc-2+```++```python+from hyperextract import Template++ka = Template.create("general/graph", language="en")++with open("doc1.md") as f:+ result = ka.parse(f.read(), source_id="doc-1")++result.feed_text(doc2_text, source_id="doc-2")+result.dump("./ka/")+```++### Change Detection++If the same source is fed again with unchanged content, `he feed` compares the content hash and skips the document entirely — **zero LLM calls**:++```bash+he feed ./ka/ doc2.md --source doc-2+# Source 'doc-2' is unchanged (content hash matches) — nothing to do.+# Use --refeed to re-ingest anyway.+```++```bash+# Force re-ingestion even when the hash matches+he feed ./ka/ doc2.md --source doc-2 --refeed+```++---++## Inspecting the Ledger++```bash+he info ./ka/ --sources+```++**Output:**+```+ Source Ledger+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓+┃ Source ID ┃ Raw Items ┃ Content Hash ┃+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │+│ doc-2 │ 8 │ 2c26b44babbfb4c6a5cc6b3f2a4b2f1d… │+└────────────┴───────────┴────────────────────────────────────┘+```++In Python:++```python+ka.sources()+# {+# "doc-1": {"raw_items": 12, "content_hash": "9f86d0…"},+# "doc-2": {"raw_items": 8, "content_hash": "2c26b4…"},+# }+```++---++## Rolling Back a Document++Remove everything a document contributed:++```bash+he remove ./ka/ --document doc-1++# Explicit strategy+he remove ./ka/ --document doc-1 --strategy touched+```++```python+report = ka.remove_source("doc-1") # exact (default)+report = ka.remove_source("doc-1", strategy="touched")++result.dump("./ka/") # persist+```++The strategy decides what happens to keys the document **shared** with other documents:++| Strategy | Shared keys | Sole keys |+|----------|-------------|-----------|+| `exact` (default) | Re-merged from the surviving sources' raw results (deterministic with classic merge strategies) | Deleted |+| `touched` | Deleted outright | Deleted |++`exact` keeps every surviving document's contribution intact; `touched` guarantees none of the document's influence remains, at the cost of also removing facts other documents support. With LLM merge strategies, the `exact` re-merge preserves semantics with approximate wording. Unknown or empty sources report `Nothing matched` and exit successfully. Either way, the search index is patched in place when built.++!!! note+ Hash-based change detection is built into `he feed --source`. Python callers get the same effect by comparing `ka.source_content_hash(source_id)` against their own hash of the text before calling `feed_text`.++---++## Fact-Level Editing++For surgical edits — keep a node but drop one wrong claim — use fact-level editing instead of a full rollback:++```python+report = ka.edit_node("Apple", remove_fact="founded by Steve Jobs", dry_run=True)+ka.edit_node("Apple", remove_fact="founded by Steve Jobs") # applies+```++```bash+he remove ./ka/ --edit-node Apple --fact "founded by Steve Jobs" -y+```++See [`he remove`](../../cli/commands/remove.md) for all guardrails (key invariance, dry run, backups).++---++## Persistence++`ka.dump()` — and every `he parse` / `he feed` — writes the ledger next to `data.json`:++```+./ka/+├── data.json # Extracted knowledge+├── metadata.json # Extraction metadata+├── sources_nodes.json # Node ledger: raw items per source+└── sources_edges.json # Edge ledger: raw items per source+```++- Keep these files together with the KA (treat them like `data.json` in version control): the raw per-source results live only there, so deleting the ledger disables rollback and change detection for that KA.+- Before any write, `he remove` backs up `data.json` as `data.json.bak.<timestamp>` (unless `--no-backup`), so even a rollback can itself be undone.++---++## Roadmap++Content-hash change detection ships today: `he feed --source` hashes the input and skips unchanged documents before any LLM call. Deeper lineage — mapping individual facts back to the exact chunk (passage) they were extracted from — is planned.++---++## See Also++- [`he feed`](../../cli/commands/feed.md) — `--source` and `--refeed` flags+- [`he info`](../../cli/commands/info.md) — source ledger inspection+- [`he remove`](../../cli/commands/remove.md) — document rollback and fact-level editing reference+- [Incremental Updates](incremental-updates.md) — merge behavior when feedingdiff --git a/docs/en/python/index.md b/docs/en/python/index.mdindex 8b7dadb..2a602a9 100644--- a/docs/en/python/index.md+++ b/docs/en/python/index.md@@ -94,6 +94,9 @@ ka = Template.create(template_path, language="en")# Extractresult = ka.parse(text) # New extractionresult.feed_text(text) # Add to existing+result.feed_text(text, source_id="doc-1") # attribute the source+...+result.remove_source("doc-1") # roll back this document's facts# Delete (graph-family KAs)result.remove_nodes("Apple") # Hard-delete a node + its orphan edgesdiff --git a/docs/zh/cli/commands/feed.md b/docs/zh/cli/commands/feed.mdindex 276ec41..672fd97 100644--- a/docs/zh/cli/commands/feed.md+++ b/docs/zh/cli/commands/feed.md@@ -92,6 +92,28 @@ cat new_content.md | he feed ./ka/ ----+## 来源标注++传入 `--source` 指定文档 id,为正在喂养的文档标注来源:++```bash+he feed ./ka/ doc1.md --source doc-1+```++标注会将该文档连同输入内容的哈希值一起记入知识库的**来源台账**,从而支持:++- **变更检测** — 同一来源再次喂养且内容未变化时,命令会输出 `unchanged (content hash matches) — nothing to do` 并直接跳过:**零 LLM 调用**。使用 `--refeed` 强制重新摄取:++ ```bash+ he feed ./ka/ doc1.md --source doc-1 --refeed+ ```++- **按文档回滚** — 该文档贡献的全部知识之后可用 [`he remove --document doc-1`](remove.md) 回滚++使用 [`he info --sources`](info.md) 或 Python 的 `ka.sources()` 查看台账。完整介绍见[来源标注与溯源](../../python/guides/provenance.md)指南。++---+## 工作流程示例### 构建研究知识库diff --git a/docs/zh/cli/commands/info.md b/docs/zh/cli/commands/info.mdindex e72458b..d46cf9e 100644--- a/docs/zh/cli/commands/info.md+++ b/docs/zh/cli/commands/info.md@@ -28,6 +28,7 @@ he info KA_PATH- **时间戳** — 创建和最后更新时间- **统计信息** — 节点数、边数- **索引状态** — 搜索索引是否已构建+- **来源台账** — 使用 `--sources` 显示贡献过知识的文档---@@ -85,6 +86,39 @@ he info ./ka/---+## 来源台账++传入 `--sources` 可以同时显示哪些文档为知识库做出过贡献:++```bash+he info ./ka/ --sources+```++**输出:**+```+Knowledge Abstract Info++Path ./ka/+Template general/biography_graph+Language en+Created 2024-01-15 10:30:00+Updated 2024-01-15 10:35:22+Nodes 25+Edges 32+Index Built++ Source Ledger+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓+┃ Source ID ┃ Raw Items ┃ Content Hash ┃+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │+└────────────┴───────────┴────────────────────────────────────┘+```++台账读取自知识库目录中的 `sources_nodes.json` / `sources_edges.json`,在使用 `--source` 摄取文档时写入(参见 [`he feed`](feed.md))。如果没有台账文件,命令会输出 `No source ledger found`(未找到来源台账)提示,而不是表格。++---+## 用例### 验证提取diff --git a/docs/zh/cli/commands/parse.md b/docs/zh/cli/commands/parse.mdindex 42d20e0..ec8c25b 100644--- a/docs/zh/cli/commands/parse.md+++ b/docs/zh/cli/commands/parse.md@@ -40,6 +40,8 @@ he parse INPUT [OPTIONS]he parse document.md -t general/biography_graph -o ./output/ -l zh```+使用 `--source` 标注来源,之后可用 [`he remove --document`](remove.md) 按文档回滚。+### 交互式模板选择省略 `-t` 以从可用模板中选择:diff --git a/docs/zh/cli/commands/remove.md b/docs/zh/cli/commands/remove.mdindex 19e63fe..153f7ec 100644--- a/docs/zh/cli/commands/remove.md+++ b/docs/zh/cli/commands/remove.md@@ -19,6 +19,7 @@ he remove KA_PATH [选项]| `--edit-node TEXT` | — | **软编辑**指定节点(大模型重写,移除 `--fact`) || `--edit-edge TEXT` | — | **软编辑**指定边 || `--document TEXT` | — | **回滚**某个来源文档贡献的全部知识(要求 KA 在摄取时使用了 `--source`) |+| `--strategy` | `exact` | `--document` 的回滚策略:`exact` 或 `touched` || `--fact TEXT` | — | 要从 `--edit-node` / `--edit-edge` 条目中移除的事实 || `--instruction TEXT` | — | 自由格式的编辑指令(替代 `--fact`) || `--dry-run` | 关闭 | 仅预览变更,不落盘 |@@ -92,18 +93,28 @@ he build-index ./ka/## 文档级回滚:删除整个来源文档的贡献-当文档以来源标注方式摄取后,可以整体回滚其贡献——包括与其他文档共享的 key(这些 key 会从幸存来源的原始结果重新合并)。+当文档以来源标注方式摄取后,可以整体回滚其贡献:```bash# 带来源标注摄取he feed ./ka/ doc1.md --source doc-1he feed ./ka/ doc2.md --source doc-2-# 之后:回滚 doc-1 贡献的全部知识+# 之后:回滚 doc-1 贡献的全部知识(共享 key 会通过 doc-2 保留)he remove ./ka/ --document doc-1++# 或删除该文档触及的所有 key,即使其他文档也共享这些 key+he remove ./ka/ --document doc-1 --strategy touched```-要求 KA 的记忆开启来源追踪(图谱类 KA 默认开启)。回滚会从幸存来源的原始结果重新合并受影响的 key——经典合并策略下完全确定;LLM 策略保留语义、措辞近似。若该文档没有已记录的贡献,命令会报告 `Nothing matched` 并正常退出。+### 回滚策略(`--strategy`)++| 策略 | 行为 |+|------|------|+| `exact`(默认) | 仅由被删文档贡献的 key 会被删除;与其他文档**共享**的 key 会从幸存来源的原始结果重新合并——经典合并策略下完全确定(LLM 策略保留语义、措辞近似) |+| `touched` | 该文档触及的每个 key 都会直接删除,包括其他文档也贡献过的 key |++要求 KA 的记忆开启来源追踪(图谱类 KA 默认开启)。若该文档没有已记录的贡献,命令会报告 `Nothing matched` 并正常退出。---diff --git a/docs/zh/cli/index.md b/docs/zh/cli/index.mdindex 478fdbf..ef2fb31 100644--- a/docs/zh/cli/index.md+++ b/docs/zh/cli/index.md@@ -30,18 +30,18 @@ he --version| 命令 | 用途 | 常用参数 ||---------|---------|--------------|-| `he parse` | 从文档提取知识 | `-t` 模板, `-o` 输出, `-l` 语言 |+| `he parse` | 从文档提取知识 | `-t` 模板, `-o` 输出, `-l` 语言, `--source` 来源标注 || `he show` | 可视化知识图谱 | — || `he export obsidian` | 导出为 Obsidian 知识库 | `-o` 输出, `--name`, `-f` 强制 || `he export graphml` | 将二元图导出为 GraphML | `-o` 输出文件 || `he export csv` | 将节点/边导出为 CSV 表 | `-o` 目录, `-f` 强制 || `he search` | 知识库语义搜索 | `-n` top-k 结果数 || `he talk` | 与知识库对话 | `-i` 交互模式, `-q` 查询 |-| `he feed` | 增量添加文档 | — |+| `he feed` | 增量添加文档 | `--source` 来源标注 || `he info` | 显示知识库统计信息 | — || `he build-index` | 构建/重建搜索索引 | `-f` 强制重建 || `he clean` | 删除 KA 的索引(或整个 KA) | `-a` all, `-y` yes |-| `he remove` | 按键删除节点/边,或软删除单条事实 | `--node`、`--edge`、`--edit-node`、`--fact`、`--dry-run` |+| `he remove` | 按键删除节点/边,或软删除单条事实 | `--node`、`--edge`、`--edit-node`、`--fact`、`--dry-run`、`--document`、`--strategy` || `he list` | 列出模板和方法 | `template` 或 `method` || `he template validate` | 校验模板 YAML | `--json`、`--all` || `he config` | 管理配置 | `init`, `show`, `llm`, `embedder` |diff --git a/docs/zh/python/guides/provenance.md b/docs/zh/python/guides/provenance.mdnew file mode 100644index 0000000..5c04adc--- /dev/null+++ b/docs/zh/python/guides/provenance.md@@ -0,0 +1,168 @@+# 来源标注与溯源++追踪哪些文档为知识摘要做出过贡献,跳过未变更的重复喂养,并按文档回滚其贡献,而不影响其余知识。++---++## 概述++**带来源标注**摄取文档(CLI 用 `--source`,Python 用 `source_id=`)会让知识摘要记住该文档:其原始提取结果会保存在**来源台账**中,并记录输入文本的 SHA-256 内容哈希。++溯源提供三项能力:++| 能力 | CLI | Python |+|------|-----|--------|+| **按文档回滚** | `he remove --document ID` | `ka.remove_source(ID)` |+| **变更检测** | `he feed --source` 跳过未变更文档 | `ka.source_content_hash(ID)` |+| **审计** | `he info --sources` | `ka.sources()` |++图谱类知识摘要(graph、hypergraph、时序/空间图谱)默认开启来源追踪。++---++## 带来源标注的摄取++```bash+# 解析时标注(首个文档)+he parse doc1.md -t general/graph -o ./ka/ -l zh --source doc-1++# 喂养进现有知识库时标注+he feed ./ka/ doc2.md --source doc-2+```++```python+from hyperextract import Template++ka = Template.create("general/graph", language="zh")++with open("doc1.md") as f:+ result = ka.parse(f.read(), source_id="doc-1")++result.feed_text(doc2_text, source_id="doc-2")+result.dump("./ka/")+```++### 变更检测++同一来源再次喂养且内容未变化时,`he feed` 会比对内容哈希并完全跳过该文档——**零 LLM 调用**:++```bash+he feed ./ka/ doc2.md --source doc-2+# Source 'doc-2' is unchanged (content hash matches) — nothing to do.+# Use --refeed to re-ingest anyway.+```++```bash+# 即使哈希一致也强制重新摄取+he feed ./ka/ doc2.md --source doc-2 --refeed+```++---++## 查看来源台账++```bash+he info ./ka/ --sources+```++**输出:**+```+ Source Ledger+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓+┃ Source ID ┃ Raw Items ┃ Content Hash ┃+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │+│ doc-2 │ 8 │ 2c26b44babbfb4c6a5cc6b3f2a4b2f1d… │+└────────────┴───────────┴────────────────────────────────────┘+```++在 Python 中:++```python+ka.sources()+# {+# "doc-1": {"raw_items": 12, "content_hash": "9f86d0…"},+# "doc-2": {"raw_items": 8, "content_hash": "2c26b4…"},+# }+```++---++## 回滚整个文档++删除某个文档贡献的全部知识:++```bash+he remove ./ka/ --document doc-1++# 显式指定策略+he remove ./ka/ --document doc-1 --strategy touched+```++```python+report = ka.remove_source("doc-1") # exact(默认)+report = ka.remove_source("doc-1", strategy="touched")++result.dump("./ka/") # 持久化+```++策略决定与其他文档**共享**的 key 如何处理:++| 策略 | 共享 key | 独占 key |+|------|----------|----------|+| `exact`(默认) | 从幸存来源的原始结果重新合并(经典合并策略下完全确定) | 删除 |+| `touched` | 直接删除 | 删除 |++`exact` 保证其余每个文档的贡献完整保留;`touched` 保证被删文档的影响不留痕迹,代价是其他文档也支持的事实会被一并删除。使用 LLM 合并策略时,`exact` 的重新合并会保留语义、措辞近似。未知或无贡献的来源会报告 `Nothing matched` 并正常退出。无论哪种情况,已构建的搜索索引都会原地修补。++!!! note+ 基于哈希的变更检测内置于 `he feed --source`。Python 调用者可以在调用 `feed_text` 前,将 `ka.source_content_hash(source_id)` 与自行计算的文本哈希比对,实现同样效果。++---++## 事实级编辑++若只需保留节点但移除其中一条错误描述,请使用事实级编辑而非整体回滚:++```python+report = ka.edit_node("Apple", remove_fact="founded by Steve Jobs", dry_run=True)+ka.edit_node("Apple", remove_fact="founded by Steve Jobs") # 应用+```++```bash+he remove ./ka/ --edit-node Apple --fact "founded by Steve Jobs" -y+```++完整安全护栏(key 不变性、预览、备份)见 [`he remove`](../../cli/commands/remove.md)。++---++## 持久化++`ka.dump()`——以及每次 `he parse` / `he feed`——会把台账写在 `data.json` 旁边:++```+./ka/+├── data.json # 提取的知识+├── metadata.json # 提取元数据+├── sources_nodes.json # 节点台账:每个来源的原始条目+└── sources_edges.json # 边台账:每个来源的原始条目+```++- 请将这些文件与知识摘要放在一起保存(像 `data.json` 一样纳入版本控制):每个来源的原始结果只存在于这里,删除台账文件会导致该知识摘要无法回滚、也无法做变更检测。+- 每次写入前,`he remove` 会将 `data.json` 备份为 `data.json.bak.<时间戳>`(除非传入 `--no-backup`),因此回滚本身也可以撤销。++---++## 路线图++基于内容哈希的变更检测已随本次发布落地:`he feed --source` 会在任何 LLM 调用之前计算输入哈希并跳过未变更文档。更细粒度的溯源——把单条事实映射回它被提取自的确切文本块(chunk)——已在规划中。++---++## 另请参阅++- [`he feed`](../../cli/commands/feed.md) — `--source` 与 `--refeed` 参数+- [`he info`](../../cli/commands/info.md) — 查看来源台账+- [`he remove`](../../cli/commands/remove.md) — 文档级回滚与事实级编辑参考+- [增量更新](incremental-updates.md) — 喂养时的合并行为diff --git a/docs/zh/python/index.md b/docs/zh/python/index.mdindex ad7834d..f344083 100644--- a/docs/zh/python/index.md+++ b/docs/zh/python/index.md@@ -94,6 +94,9 @@ ka = Template.create(template_path, language="zh")# 提取result = ka.parse(text) # 新提取result.feed_text(text) # 添加到现有+result.feed_text(text, source_id="doc-1") # 标注来源+...+result.remove_source("doc-1") # 回滚该文档贡献的事实# 删除(图谱类 KA)result.remove_nodes("Apple") # 硬删除节点及其孤儿边diff --git a/hyperextract/cli/cli.py b/hyperextract/cli/cli.pyindex 5425010..52ffb1d 100644--- a/hyperextract/cli/cli.py+++ b/hyperextract/cli/cli.py@@ -1,5 +1,6 @@"""CLI entry point for Hyper-Extract."""+import hashlibfrom pathlib import Pathimport typer@@ -687,9 +688,16 @@ app.add_typer(export_app, name="export")@app.command(name="info")-def info(ka_path: str = typer.Argument(..., help="Knowledge Abstract directory")):+def info(+ ka_path: str = typer.Argument(..., help="Knowledge Abstract directory"),+ sources: bool = typer.Option(+ False,+ "--sources",+ help="Show the source ledger (documents that contributed to this KA)",+ ),+):"""View Knowledge Abstract information and statistics."""- logger.info("command=info ka_path=%s", ka_path)+ logger.info("command=info ka_path=%s sources=%s", ka_path, sources)import jsonpath = validate_ka_with_data(ka_path)@@ -735,6 +743,48 @@ def info(ka_path: str = typer.Argument(..., help="Knowledge Abstract directory")console.print(table)+ if sources:+ sources_table = Table(title="Source Ledger", show_header=True)+ sources_table.add_column("Source ID", style="cyan")+ sources_table.add_column("Raw Items", justify="right")+ sources_table.add_column("Content Hash")++ ledger_files = [+ (path / "sources_nodes.json", "nodes"),+ (path / "sources_edges.json", "edges"),+ ]+ combined: dict = {}+ for ledger_path, _kind in ledger_files:+ if not ledger_path.exists():+ continue+ try:+ entries = json.loads(ledger_path.read_text(encoding="utf-8"))+ except Exception as e:+ logger.warning(+ "sources_ledger_read_failed path=%s error=%s", ledger_path, e+ )+ continue+ for entry in entries:+ sid = entry.get("source_id")+ if sid is None:+ continue+ record = combined.setdefault(+ sid, {"raw_items": 0, "content_hash": entry.get("content_hash")}+ )+ record["raw_items"] += len(entry.get("raw_items", []))++ if combined:+ for sid, info_row in sorted(combined.items()):+ sources_table.add_row(+ sid, str(info_row["raw_items"]), info_row["content_hash"] or "—"+ )+ console.print(sources_table)+ else:+ console.print(+ "[yellow]No source ledger found.[/yellow] "+ "Feed documents with --source to enable per-document rollback."+ )+@app.command(name="search")def search(@@ -940,6 +990,11 @@ def feed("--source",help="Source attribution (document id) for per-document rollback later (he remove --document)",),+ refeed: bool = typer.Option(+ False,+ "--refeed",+ help="Re-ingest even if this source's content hash is unchanged",+ ),):"""Append knowledge to an existing Knowledge Abstract."""logger.info("command=feed ka_path=%s input=%s", ka_path, input)@@ -985,9 +1040,24 @@ def feed(text = read_input(input)console.print(f"[dim]Input text: {len(text)} characters[/dim]")+ # Change detection: skip unchanged sources without any LLM calls.+ text_hash_to_record = None+ if source:+ text_hash_to_record = hashlib.sha256(text.encode("utf-8")).hexdigest()+ if not refeed:+ recorded = ka.source_content_hash(source)+ if recorded == text_hash_to_record:+ logger.info("stage=source_unchanged source=%s", source)+ console.print(+ f"[yellow]Source '{source}' is unchanged (content hash "+ "matches) — nothing to do.[/yellow] "+ "Use --refeed to re-ingest anyway."+ )+ raise typer.Exit(0)+progress.update(task, description="Appending knowledge...")logger.debug("stage=feed_text_invoked")- ka.feed_text(text, source_id=source)+ ka.feed_text(text, source_id=source, content_hash=text_hash_to_record)logger.info("stage=knowledge_appended chars=%d", len(text))progress.update(task, description="Saving data...")@@ -1151,6 +1221,12 @@ def remove_items(help="Remove all knowledge contributed by this source document ""(requires the KA to have been fed with --source)",),+ strategy: str = typer.Option(+ "exact",+ "--strategy",+ help="Rollback strategy for --document: exact (re-merge surviving "+ "sources) or touched (delete all affected keys)",+ ),):"""Delete nodes/edges by key, remove a fact (LLM-assisted), or roll back a whole document.@@ -1210,8 +1286,14 @@ def remove_items(try:if document:+ if strategy not in ("exact", "touched"):+ console.print(+ f"[red]Error:[/red] Unknown strategy: {strategy!r} "+ "(use exact or touched)."+ )+ raise typer.Exit(1)try:- report = ka.remove_source(document, strategy="exact")+ report = ka.remove_source(document, strategy=strategy)except KeyError:console.print(f"[yellow]Nothing matched — no recorded contributions "diff --git a/hyperextract/types/base.py b/hyperextract/types/base.pyindex 8f54ce1..ae64cb4 100644--- a/hyperextract/types/base.py+++ b/hyperextract/types/base.py@@ -452,7 +452,11 @@ class BaseAutoType(ABC, Generic[T]):return new_instancedef feed_text(- self, text: str, *, source_id: str | None = None+ self,+ text: str,+ *,+ source_id: str | None = None,+ content_hash: str | None = None,) -> "BaseAutoType[T]":"""Ingests text into the CURRENT knowledge abstract instance.@@ -466,12 +470,15 @@ class BaseAutoType(ABC, Generic[T]):``track_sources``): the raw extraction results are recordedunder this id so the document's contributions can later berolled back via ``remove_source``.+ content_hash: Optional hash of the source content (recorded in+ the source ledger for change detection).Returns:Self (the current instance)."""logger.debug("stage=feed_text_start input_chars=%d", len(text))self._pending_source_id = source_id+ self._pending_content_hash = content_hashtry:extracted_data = self._extract_data(text)logger.debug("stage=extract_done")@@ -481,6 +488,7 @@ class BaseAutoType(ABC, Generic[T]):logger.debug("stage=data_merged")finally:self._pending_source_id = None+ self._pending_content_hash = Noneself.metadata["updated_at"] = datetime.now()diff --git a/hyperextract/types/graph.py b/hyperextract/types/graph.pyindex d622d43..f9daa86 100644--- a/hyperextract/types/graph.py+++ b/hyperextract/types/graph.py@@ -98,11 +98,16 @@ class GraphEditMixin:"""if not source_id or not self._node_memory.track_sources:return+ content_hash = self._pending_content_hashself._node_memory.record_source(- source_id, [node.model_dump() for node in nodes]+ source_id,+ [node.model_dump() for node in nodes],+ content_hash=content_hash,)self._edge_memory.record_source(- source_id, [edge.model_dump() for edge in edges]+ source_id,+ [edge.model_dump() for edge in edges],+ content_hash=content_hash,)def _adopt_source_ledger(self, other, source_id: str) -> None:@@ -163,6 +168,32 @@ class GraphEditMixin:and edge_report["index_patched"],}+ # ==================== Source Ledger (read) ====================++ def sources(self) -> dict[str, dict[str, Any]]:+ """Summarize the source ledger (node and edge memories combined).++ Returns:+ Dict mapping source_id to {raw_items, content_hash}. Raw-item+ counts are from the node ledger; edges share the same sources.+ """+ summary: dict[str, dict[str, Any]] = {}+ for source_id, info in self._node_memory.sources().items():+ summary[source_id] = dict(info)+ # Edge memories may know sources the node ledger lacks (empty-node+ # extractions); surface them too.+ for source_id, info in self._edge_memory.sources().items():+ if source_id not in summary:+ summary[source_id] = dict(info)+ return summary++ def source_content_hash(self, source_id: str) -> str | None:+ """Return the recorded content hash of a source (None if unknown)."""+ record = self._node_memory._sources.get(+ source_id+ ) or self._edge_memory._sources.get(source_id)+ return record.content_hash if record else None+# ==================== Removal ====================def remove_nodes(self, *keys: str) -> dict:diff --git a/hyperextract/types/hypergraph.py b/hyperextract/types/hypergraph.pyindex 54f616d..7978ca2 100644--- a/hyperextract/types/hypergraph.py+++ b/hyperextract/types/hypergraph.py@@ -403,15 +403,29 @@ class AutoHypergraph(self.clear_index()def _update_data_state(self, incoming_data: AutoHypergraphSchema) -> None:- """Merge incoming hypergraph data into current state."""+ """Merge incoming hypergraph data into current state.++ The vector index is patched in place: only the incoming nodes/edges+ are re-embedded, instead of dropping the whole index.+ """if self.empty():self._set_data_state(incoming_data)- else:+ return++ # Suspend index invalidation while merging; re-embed only the+ # affected keys afterwards.+ with self._node_memory.suspended_index(), self._edge_memory.suspended_index():if incoming_data.nodes:self._node_memory.add(incoming_data.nodes)if incoming_data.edges:self._edge_memory.add(incoming_data.edges)- self.clear_index()++ if incoming_data.nodes:+ node_keys = {self.key_extractor(n) for n in incoming_data.nodes}+ self._node_memory.sync_index(upserted_keys=node_keys)+ if incoming_data.edges:+ edge_keys = {self.key_extractor(e) for e in incoming_data.edges}+ self._edge_memory.sync_index(upserted_keys=edge_keys)@propertydef data(self) -> AutoHypergraphSchema:diff --git a/mkdocs.yml b/mkdocs.ymlindex c39f95c..8025ddf 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -95,6 +95,7 @@ plugins:- python/guides/using-methods.md- python/guides/working-with-autotypes.md- python/guides/provider-configuration.md+ - python/guides/provenance.md- python/guides/custom-templates.md- python/guides/search-and-chat.md- python/guides/incremental-updates.md@@ -191,6 +192,7 @@ plugins:- zh/python/guides/using-methods.md- zh/python/guides/working-with-autotypes.md- zh/python/guides/provider-configuration.md+ - zh/python/guides/provenance.md- zh/python/guides/custom-templates.md- zh/python/guides/search-and-chat.md- zh/python/guides/incremental-updates.mddiff --git a/tests/cli/test_info.py b/tests/cli/test_info.pynew file mode 100644index 0000000..fb9545c--- /dev/null+++ b/tests/cli/test_info.py@@ -0,0 +1,65 @@+"""Tests for the `he info` command (--sources source ledger view)."""++import json++from typer.testing import CliRunner++from hyperextract.cli.cli import app++runner = CliRunner()+++def _make_ka_with_sources(tmp_path, *, with_ledger: bool = True):+ """A minimal KA directory on disk, optionally with source ledger files."""+ ka = tmp_path / "ka"+ ka.mkdir(parents=True)+ (ka / "data.json").write_text(+ json.dumps(+ {+ "nodes": [{"name": "mock_name", "type": "ENTITY", "description": ""}],+ "edges": [],+ }+ ),+ encoding="utf-8",+ )+ (ka / "metadata.json").write_text(+ json.dumps({"template": "general/graph", "lang": "en"}),+ encoding="utf-8",+ )+ if with_ledger:+ (ka / "sources_nodes.json").write_text(+ json.dumps(+ [+ {+ "source_id": "doc-1",+ "content_hash": "abc",+ "raw_items": [+ {"name": "mock_name", "type": "ENTITY", "description": ""}+ ],+ }+ ]+ ),+ encoding="utf-8",+ )+ (ka / "sources_edges.json").write_text(json.dumps([]), encoding="utf-8")+ return ka+++class TestInfoSourcesCli:+ def test_sources_renders_source_ledger(self, tmp_path):+ ka = _make_ka_with_sources(tmp_path)++ result = runner.invoke(app, ["info", str(ka), "--sources"])++ assert result.exit_code == 0, result.output+ assert "Source Ledger" in result.output+ assert "doc-1" in result.output+ assert "abc" in result.output++ def test_sources_without_ledger_prints_hint(self, tmp_path):+ ka = _make_ka_with_sources(tmp_path, with_ledger=False)++ result = runner.invoke(app, ["info", str(ka), "--sources"])++ assert result.exit_code == 0, result.output+ assert "No source ledger" in result.outputdiff --git a/tests/cli/test_remove.py b/tests/cli/test_remove.pyindex 1eed241..4119758 100644--- a/tests/cli/test_remove.py+++ b/tests/cli/test_remove.py@@ -246,6 +246,76 @@ class TestDocumentRollbackCli:assert "Nothing matched" in result.output+class TestDocumentRollbackStrategies:+ @staticmethod+ def _strategy_ka(tmp_path, monkeypatch):+ """A KA with two recorded sources sharing the node "Shared", plus+ s1's own node "FromS1" — all seeded deterministically (no extraction+ LLM): ledger entries via record_source, storage via direct add()."""+ g = _real_ka()+ g.metadata["template"] = "general/graph"+ g.metadata["lang"] = "en"+ g._node_memory.add(+ [E(name="FromS1", type="X"), E(name="Shared", description="shared")]+ )+ g._node_memory.record_source(+ "s1",+ [E(name="FromS1", type="X").model_dump(), E(name="Shared").model_dump()],+ )+ g._node_memory.record_source("s2", [E(name="Shared").model_dump()])+ # The edge ledger needs the source too, or remove_source bails out+ # with KeyError before any node is touched.+ g._edge_memory.record_source("s1", [])+ g._edge_memory.record_source("s2", [])+ ka = tmp_path / "ka"+ g.dump(ka)+ monkeypatch.setattr(climod.Template, "create", staticmethod(lambda *a, **k: g))+ return ka, g++ def test_remove_document_strategy_touched(self, tmp_path, monkeypatch):+ ka, _ = self._strategy_ka(tmp_path, monkeypatch)++ result = runner.invoke(+ app,+ ["remove", str(ka), "--document", "s1", "--strategy", "touched", "-y"],+ )++ assert result.exit_code == 0, result.output+ names = [n["name"] for n in _data(ka)["nodes"]]+ # Touched deletes every key s1 touched outright — "Shared" is gone+ # even though s2 also contributed it (no re-merge step).+ assert "FromS1" not in names+ assert "Shared" not in names+ assert "Apple" in names and "Google" in names+ assert "Document Rollback Report" in result.output+ assert "FromS1" in result.output+ assert "Shared" in result.output++ def test_remove_document_strategy_exact_remerges(self, tmp_path, monkeypatch):+ from ontomem.merger import MergeStrategy, create_merger++ ka, g = self._strategy_ka(tmp_path, monkeypatch)+ # Deterministic re-merge: the default LLM.BALANCED merger would need+ # a real LLM; MERGE_FIELD keeps this offline.+ g._node_memory._merger = create_merger(+ MergeStrategy.MERGE_FIELD, key_extractor=lambda x: x.name+ )++ result = runner.invoke(+ app,+ ["remove", str(ka), "--document", "s1", "--strategy", "exact", "-y"],+ )++ assert result.exit_code == 0, result.output+ names = [n["name"] for n in _data(ka)["nodes"]]+ # "Shared" survives: re-merged from s2's raw contribution. "FromS1"+ # (s1's sole contribution) is gone.+ assert "Shared" in names+ assert "FromS1" not in names+ assert "Document Rollback Report" in result.output+ assert "Shared" in result.output++class TestArgumentValidation:def test_hard_and_soft_delete_are_mutually_exclusive(self, ka_env):ka, _ = ka_env