Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

feat: provenance — he info --sources, he feed --source/--refeed, he remove --document/--strategy

yifanfeng9795fdeb07ab32
diff --git a/docs/en/cli/commands/feed.md b/docs/en/cli/commands/feed.md
index 93caf2f..68061b5 100644
--- a/docs/en/cli/commands/feed.md
+++ b/docs/en/cli/commands/feed.md
@@ -92,6 +92,28 @@ The merge process handles:
---
+## Source Attribution
+
+Pass `--source` with a document id to attribute the document you are feeding:
+
+```bash
+he feed ./ka/ doc1.md --source doc-1
+```
+
+Attribution records the document in the KA's **source ledger** together with a content hash of the input, which enables:
+
+- **Change detection** — feeding the same source again with unchanged content prints `unchanged (content hash matches) — nothing to do` and skips entirely: **zero LLM calls**. Use `--refeed` to force re-ingestion:
+
+ ```bash
+ he feed ./ka/ doc1.md --source doc-1 --refeed
+ ```
+
+- **Per-document rollback** — everything this document contributed can later be rolled back with [`he remove --document doc-1`](remove.md)
+
+Inspect the ledger with [`he info --sources`](info.md#source-ledger) or `ka.sources()` in Python. See the [Source Attribution & Provenance](../../python/guides/provenance.md) guide for the full picture.
+
+---
+
## Workflow Example
### Building a Research Knowledge Abstract
diff --git a/docs/en/cli/commands/info.md b/docs/en/cli/commands/info.md
index 3de2f3c..1821a92 100644
--- a/docs/en/cli/commands/info.md
+++ b/docs/en/cli/commands/info.md
@@ -28,6 +28,7 @@ Shows metadata and statistics for a knowledge abstract:
- **Timestamps** — Creation and last update times
- **Statistics** — Node count, edge count
- **Index status** — Whether search index is built
+- **Source ledger** — Contributing documents, shown with `--sources`
---
@@ -85,6 +86,39 @@ he info ./ka/
---
+## Source Ledger
+
+Pass `--sources` to also show which documents contributed to the knowledge abstract:
+
+```bash
+he info ./ka/ --sources
+```
+
+**Output:**
+```
+Knowledge Abstract Info
+
+Path ./ka/
+Template general/biography_graph
+Language en
+Created 2024-01-15 10:30:00
+Updated 2024-01-15 10:35:22
+Nodes 25
+Edges 32
+Index Built
+
+ Source Ledger
+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Source ID ┃ Raw Items ┃ Content Hash ┃
+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │
+└────────────┴───────────┴────────────────────────────────────┘
+```
+
+The ledger is read from `sources_nodes.json` / `sources_edges.json` in the KA directory and is written when documents are ingested with `--source` (see [`he feed`](feed.md)). If no ledger exists, the command prints a `No source ledger found` hint instead of the table.
+
+---
+
## Use Cases
### Verify Extraction
diff --git a/docs/en/cli/commands/parse.md b/docs/en/cli/commands/parse.md
index 1cb19ba..00e536e 100644
--- a/docs/en/cli/commands/parse.md
+++ b/docs/en/cli/commands/parse.md
@@ -40,6 +40,8 @@ Extract from a single file:
he parse document.md -t general/biography_graph -o ./output/ -l en
```
+Adding `--source` records the document's source attribution, which enables per-document rollback later via [`he remove --document`](remove.md).
+
### Interactive Template Selection
Omit `-t` to select from available templates:
diff --git a/docs/en/cli/commands/remove.md b/docs/en/cli/commands/remove.md
index 74c78de..04ddeff 100644
--- a/docs/en/cli/commands/remove.md
+++ b/docs/en/cli/commands/remove.md
@@ -19,6 +19,7 @@ he remove KA_PATH [OPTIONS]
| `--edit-node TEXT` | — | Node key to **soft-edit** (LLM rewrites it minus `--fact`) |
| `--edit-edge TEXT` | — | Edge key to **soft-edit** |
| `--document TEXT` | — | **Roll back** all knowledge contributed by a source document (requires the KA to have been fed with `--source`) |
+| `--strategy` | `exact` | Rollback strategy for `--document`: `exact` or `touched` |
| `--fact TEXT` | — | The fact to remove from the `--edit-node` / `--edit-edge` item |
| `--instruction TEXT` | — | Free-form edit instruction (alternative to `--fact`) |
| `--dry-run` | off | Preview the change without persisting it |
@@ -92,7 +93,7 @@ he build-index ./ka/
## Document Rollback: remove a whole source document
-When documents were ingested with a source attribution, their entire contribution can be rolled back — including keys that other documents also contributed to (those are re-merged from the surviving sources).
+When documents were ingested with a source attribution, their entire contribution can be rolled back:
```bash
# Ingest with attribution
@@ -101,9 +102,19 @@ he feed ./ka/ doc2.md --source doc-2
# Later: roll back everything doc-1 contributed ("b" remains via doc-2 if shared)
he remove ./ka/ --document doc-1
+
+# Or delete every key the document touched, even when other documents share it
+he remove ./ka/ --document doc-1 --strategy touched
```
-Requires the KA's memories to track sources (graph-family KAs do by default). The rollback re-merges affected keys from the surviving sources' raw results — deterministic with classic merge strategies; LLM strategies preserve semantics with approximate wording. If the document has no recorded contributions, the command reports `Nothing matched` and exits successfully.
+### Rollback Strategies (`--strategy`)
+
+| Strategy | Behavior |
+|----------|----------|
+| `exact` (default) | Keys contributed solely by the removed document are deleted; keys **shared** with other documents are re-merged from the surviving sources' raw results — deterministic with classic merge strategies (LLM strategies preserve semantics with approximate wording) |
+| `touched` | Every key the document touched is deleted outright, including keys other documents also contributed to |
+
+Requires the KA's memories to track sources (graph-family KAs do by default). If the document has no recorded contributions, the command reports `Nothing matched` and exits successfully.
---
diff --git a/docs/en/cli/index.md b/docs/en/cli/index.md
index fdc7b03..c7d8a06 100644
--- a/docs/en/cli/index.md
+++ b/docs/en/cli/index.md
@@ -30,18 +30,18 @@ he --version
| Command | Purpose | Common Flags |
|---------|---------|--------------|
-| `he parse` | Extract knowledge from documents | `-t` template, `-o` output, `-l` language |
+| `he parse` | Extract knowledge from documents | `-t` template, `-o` output, `-l` language, `--source` attribution |
| `he show` | Visualize knowledge graph | — |
| `he export obsidian` | Export to an Obsidian vault | `-o` output, `--name`, `-f` force |
| `he export graphml` | Export a pairwise graph to GraphML | `-o` output file |
| `he export csv` | Export nodes/edges as CSV tables | `-o` directory, `-f` force |
| `he search` | Semantic search in knowledge abstract | `-n` top-k results |
| `he talk` | Chat with knowledge abstract | `-i` interactive, `-q` query |
-| `he feed` | Add documents incrementally | — |
+| `he feed` | Add documents incrementally | `--source` attribution |
| `he info` | Show knowledge abstract statistics | — |
| `he build-index` | Build/rebuild search index | `-f` force rebuild |
| `he clean` | Remove a KA's index (or the whole KA) | `-a` all, `-y` yes |
-| `he remove` | Delete nodes/edges by key, or soft-remove a single fact | `--node`, `--edge`, `--edit-node`, `--fact`, `--dry-run` |
+| `he remove` | Delete nodes/edges by key, or soft-remove a single fact | `--node`, `--edge`, `--edit-node`, `--fact`, `--dry-run`, `--document`, `--strategy` |
| `he list` | List templates and methods | `template` or `method` |
| `he template validate` | Validate a template YAML file | `--json`, `--all` |
| `he config` | Manage configuration | `init`, `show`, `llm`, `embedder` |
diff --git a/docs/en/python/guides/provenance.md b/docs/en/python/guides/provenance.md
new file mode 100644
index 0000000..b287de9
--- /dev/null
+++ b/docs/en/python/guides/provenance.md
@@ -0,0 +1,168 @@
+# Source Attribution & Provenance
+
+Track which documents contributed to a knowledge abstract, skip unchanged re-feeds, and roll back any document's contribution without touching the rest.
+
+---
+
+## Overview
+
+Ingesting a document **with attribution** (`--source` on the CLI, `source_id=` in Python) makes the knowledge abstract remember that document: its raw extraction results are kept in a **source ledger**, together with a SHA-256 content hash of the input text.
+
+Provenance gives you three capabilities:
+
+| Capability | CLI | Python |
+|------------|-----|--------|
+| **Per-document rollback** | `he remove --document ID` | `ka.remove_source(ID)` |
+| **Change detection** | `he feed --source` skips unchanged documents | `ka.source_content_hash(ID)` |
+| **Audit** | `he info --sources` | `ka.sources()` |
+
+Source tracking is on by default for graph-family KAs (graph, hypergraph, temporal/spatial graphs).
+
+---
+
+## Ingesting with Attribution
+
+```bash
+# Attribute at parse time (first document)
+he parse doc1.md -t general/graph -o ./ka/ -l en --source doc-1
+
+# Attribute when feeding into an existing KA
+he feed ./ka/ doc2.md --source doc-2
+```
+
+```python
+from hyperextract import Template
+
+ka = Template.create("general/graph", language="en")
+
+with open("doc1.md") as f:
+ result = ka.parse(f.read(), source_id="doc-1")
+
+result.feed_text(doc2_text, source_id="doc-2")
+result.dump("./ka/")
+```
+
+### Change Detection
+
+If the same source is fed again with unchanged content, `he feed` compares the content hash and skips the document entirely — **zero LLM calls**:
+
+```bash
+he feed ./ka/ doc2.md --source doc-2
+# Source 'doc-2' is unchanged (content hash matches) — nothing to do.
+# Use --refeed to re-ingest anyway.
+```
+
+```bash
+# Force re-ingestion even when the hash matches
+he feed ./ka/ doc2.md --source doc-2 --refeed
+```
+
+---
+
+## Inspecting the Ledger
+
+```bash
+he info ./ka/ --sources
+```
+
+**Output:**
+```
+ Source Ledger
+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Source ID ┃ Raw Items ┃ Content Hash ┃
+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │
+│ doc-2 │ 8 │ 2c26b44babbfb4c6a5cc6b3f2a4b2f1d… │
+└────────────┴───────────┴────────────────────────────────────┘
+```
+
+In Python:
+
+```python
+ka.sources()
+# {
+# "doc-1": {"raw_items": 12, "content_hash": "9f86d0…"},
+# "doc-2": {"raw_items": 8, "content_hash": "2c26b4…"},
+# }
+```
+
+---
+
+## Rolling Back a Document
+
+Remove everything a document contributed:
+
+```bash
+he remove ./ka/ --document doc-1
+
+# Explicit strategy
+he remove ./ka/ --document doc-1 --strategy touched
+```
+
+```python
+report = ka.remove_source("doc-1") # exact (default)
+report = ka.remove_source("doc-1", strategy="touched")
+
+result.dump("./ka/") # persist
+```
+
+The strategy decides what happens to keys the document **shared** with other documents:
+
+| Strategy | Shared keys | Sole keys |
+|----------|-------------|-----------|
+| `exact` (default) | Re-merged from the surviving sources' raw results (deterministic with classic merge strategies) | Deleted |
+| `touched` | Deleted outright | Deleted |
+
+`exact` keeps every surviving document's contribution intact; `touched` guarantees none of the document's influence remains, at the cost of also removing facts other documents support. With LLM merge strategies, the `exact` re-merge preserves semantics with approximate wording. Unknown or empty sources report `Nothing matched` and exit successfully. Either way, the search index is patched in place when built.
+
+!!! note
+ Hash-based change detection is built into `he feed --source`. Python callers get the same effect by comparing `ka.source_content_hash(source_id)` against their own hash of the text before calling `feed_text`.
+
+---
+
+## Fact-Level Editing
+
+For surgical edits — keep a node but drop one wrong claim — use fact-level editing instead of a full rollback:
+
+```python
+report = ka.edit_node("Apple", remove_fact="founded by Steve Jobs", dry_run=True)
+ka.edit_node("Apple", remove_fact="founded by Steve Jobs") # applies
+```
+
+```bash
+he remove ./ka/ --edit-node Apple --fact "founded by Steve Jobs" -y
+```
+
+See [`he remove`](../../cli/commands/remove.md) for all guardrails (key invariance, dry run, backups).
+
+---
+
+## Persistence
+
+`ka.dump()` — and every `he parse` / `he feed` — writes the ledger next to `data.json`:
+
+```
+./ka/
+├── data.json # Extracted knowledge
+├── metadata.json # Extraction metadata
+├── sources_nodes.json # Node ledger: raw items per source
+└── sources_edges.json # Edge ledger: raw items per source
+```
+
+- Keep these files together with the KA (treat them like `data.json` in version control): the raw per-source results live only there, so deleting the ledger disables rollback and change detection for that KA.
+- Before any write, `he remove` backs up `data.json` as `data.json.bak.<timestamp>` (unless `--no-backup`), so even a rollback can itself be undone.
+
+---
+
+## Roadmap
+
+Content-hash change detection ships today: `he feed --source` hashes the input and skips unchanged documents before any LLM call. Deeper lineage — mapping individual facts back to the exact chunk (passage) they were extracted from — is planned.
+
+---
+
+## See Also
+
+- [`he feed`](../../cli/commands/feed.md) — `--source` and `--refeed` flags
+- [`he info`](../../cli/commands/info.md) — source ledger inspection
+- [`he remove`](../../cli/commands/remove.md) — document rollback and fact-level editing reference
+- [Incremental Updates](incremental-updates.md) — merge behavior when feeding
diff --git a/docs/en/python/index.md b/docs/en/python/index.md
index 8b7dadb..2a602a9 100644
--- a/docs/en/python/index.md
+++ b/docs/en/python/index.md
@@ -94,6 +94,9 @@ ka = Template.create(template_path, language="en")
# Extract
result = ka.parse(text) # New extraction
result.feed_text(text) # Add to existing
+result.feed_text(text, source_id="doc-1") # attribute the source
+...
+result.remove_source("doc-1") # roll back this document's facts
# Delete (graph-family KAs)
result.remove_nodes("Apple") # Hard-delete a node + its orphan edges
diff --git a/docs/zh/cli/commands/feed.md b/docs/zh/cli/commands/feed.md
index 276ec41..672fd97 100644
--- a/docs/zh/cli/commands/feed.md
+++ b/docs/zh/cli/commands/feed.md
@@ -92,6 +92,28 @@ cat new_content.md | he feed ./ka/ -
---
+## 来源标注
+
+传入 `--source` 指定文档 id,为正在喂养的文档标注来源:
+
+```bash
+he feed ./ka/ doc1.md --source doc-1
+```
+
+标注会将该文档连同输入内容的哈希值一起记入知识库的**来源台账**,从而支持:
+
+- **变更检测** — 同一来源再次喂养且内容未变化时,命令会输出 `unchanged (content hash matches) — nothing to do` 并直接跳过:**零 LLM 调用**。使用 `--refeed` 强制重新摄取:
+
+ ```bash
+ he feed ./ka/ doc1.md --source doc-1 --refeed
+ ```
+
+- **按文档回滚** — 该文档贡献的全部知识之后可用 [`he remove --document doc-1`](remove.md) 回滚
+
+使用 [`he info --sources`](info.md) 或 Python 的 `ka.sources()` 查看台账。完整介绍见[来源标注与溯源](../../python/guides/provenance.md)指南。
+
+---
+
## 工作流程示例
### 构建研究知识库
diff --git a/docs/zh/cli/commands/info.md b/docs/zh/cli/commands/info.md
index e72458b..d46cf9e 100644
--- a/docs/zh/cli/commands/info.md
+++ b/docs/zh/cli/commands/info.md
@@ -28,6 +28,7 @@ he info KA_PATH
- **时间戳** — 创建和最后更新时间
- **统计信息** — 节点数、边数
- **索引状态** — 搜索索引是否已构建
+- **来源台账** — 使用 `--sources` 显示贡献过知识的文档
---
@@ -85,6 +86,39 @@ he info ./ka/
---
+## 来源台账
+
+传入 `--sources` 可以同时显示哪些文档为知识库做出过贡献:
+
+```bash
+he info ./ka/ --sources
+```
+
+**输出:**
+```
+Knowledge Abstract Info
+
+Path ./ka/
+Template general/biography_graph
+Language en
+Created 2024-01-15 10:30:00
+Updated 2024-01-15 10:35:22
+Nodes 25
+Edges 32
+Index Built
+
+ Source Ledger
+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Source ID ┃ Raw Items ┃ Content Hash ┃
+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │
+└────────────┴───────────┴────────────────────────────────────┘
+```
+
+台账读取自知识库目录中的 `sources_nodes.json` / `sources_edges.json`,在使用 `--source` 摄取文档时写入(参见 [`he feed`](feed.md))。如果没有台账文件,命令会输出 `No source ledger found`(未找到来源台账)提示,而不是表格。
+
+---
+
## 用例
### 验证提取
diff --git a/docs/zh/cli/commands/parse.md b/docs/zh/cli/commands/parse.md
index 42d20e0..ec8c25b 100644
--- a/docs/zh/cli/commands/parse.md
+++ b/docs/zh/cli/commands/parse.md
@@ -40,6 +40,8 @@ he parse INPUT [OPTIONS]
he parse document.md -t general/biography_graph -o ./output/ -l zh
```
+使用 `--source` 标注来源,之后可用 [`he remove --document`](remove.md) 按文档回滚。
+
### 交互式模板选择
省略 `-t` 以从可用模板中选择:
diff --git a/docs/zh/cli/commands/remove.md b/docs/zh/cli/commands/remove.md
index 19e63fe..153f7ec 100644
--- a/docs/zh/cli/commands/remove.md
+++ b/docs/zh/cli/commands/remove.md
@@ -19,6 +19,7 @@ he remove KA_PATH [选项]
| `--edit-node TEXT` | — | **软编辑**指定节点(大模型重写,移除 `--fact`) |
| `--edit-edge TEXT` | — | **软编辑**指定边 |
| `--document TEXT` | — | **回滚**某个来源文档贡献的全部知识(要求 KA 在摄取时使用了 `--source`) |
+| `--strategy` | `exact` | `--document` 的回滚策略:`exact` 或 `touched` |
| `--fact TEXT` | — | 要从 `--edit-node` / `--edit-edge` 条目中移除的事实 |
| `--instruction TEXT` | — | 自由格式的编辑指令(替代 `--fact`) |
| `--dry-run` | 关闭 | 仅预览变更,不落盘 |
@@ -92,18 +93,28 @@ he build-index ./ka/
## 文档级回滚:删除整个来源文档的贡献
-当文档以来源标注方式摄取后,可以整体回滚其贡献——包括与其他文档共享的 key(这些 key 会从幸存来源的原始结果重新合并)。
+当文档以来源标注方式摄取后,可以整体回滚其贡献:
```bash
# 带来源标注摄取
he feed ./ka/ doc1.md --source doc-1
he feed ./ka/ doc2.md --source doc-2
-# 之后:回滚 doc-1 贡献的全部知识
+# 之后:回滚 doc-1 贡献的全部知识(共享 key 会通过 doc-2 保留)
he remove ./ka/ --document doc-1
+
+# 或删除该文档触及的所有 key,即使其他文档也共享这些 key
+he remove ./ka/ --document doc-1 --strategy touched
```
-要求 KA 的记忆开启来源追踪(图谱类 KA 默认开启)。回滚会从幸存来源的原始结果重新合并受影响的 key——经典合并策略下完全确定;LLM 策略保留语义、措辞近似。若该文档没有已记录的贡献,命令会报告 `Nothing matched` 并正常退出。
+### 回滚策略(`--strategy`)
+
+| 策略 | 行为 |
+|------|------|
+| `exact`(默认) | 仅由被删文档贡献的 key 会被删除;与其他文档**共享**的 key 会从幸存来源的原始结果重新合并——经典合并策略下完全确定(LLM 策略保留语义、措辞近似) |
+| `touched` | 该文档触及的每个 key 都会直接删除,包括其他文档也贡献过的 key |
+
+要求 KA 的记忆开启来源追踪(图谱类 KA 默认开启)。若该文档没有已记录的贡献,命令会报告 `Nothing matched` 并正常退出。
---
diff --git a/docs/zh/cli/index.md b/docs/zh/cli/index.md
index 478fdbf..ef2fb31 100644
--- a/docs/zh/cli/index.md
+++ b/docs/zh/cli/index.md
@@ -30,18 +30,18 @@ he --version
| 命令 | 用途 | 常用参数 |
|---------|---------|--------------|
-| `he parse` | 从文档提取知识 | `-t` 模板, `-o` 输出, `-l` 语言 |
+| `he parse` | 从文档提取知识 | `-t` 模板, `-o` 输出, `-l` 语言, `--source` 来源标注 |
| `he show` | 可视化知识图谱 | — |
| `he export obsidian` | 导出为 Obsidian 知识库 | `-o` 输出, `--name`, `-f` 强制 |
| `he export graphml` | 将二元图导出为 GraphML | `-o` 输出文件 |
| `he export csv` | 将节点/边导出为 CSV 表 | `-o` 目录, `-f` 强制 |
| `he search` | 知识库语义搜索 | `-n` top-k 结果数 |
| `he talk` | 与知识库对话 | `-i` 交互模式, `-q` 查询 |
-| `he feed` | 增量添加文档 | — |
+| `he feed` | 增量添加文档 | `--source` 来源标注 |
| `he info` | 显示知识库统计信息 | — |
| `he build-index` | 构建/重建搜索索引 | `-f` 强制重建 |
| `he clean` | 删除 KA 的索引(或整个 KA) | `-a` all, `-y` yes |
-| `he remove` | 按键删除节点/边,或软删除单条事实 | `--node`、`--edge`、`--edit-node`、`--fact`、`--dry-run` |
+| `he remove` | 按键删除节点/边,或软删除单条事实 | `--node`、`--edge`、`--edit-node`、`--fact`、`--dry-run`、`--document`、`--strategy` |
| `he list` | 列出模板和方法 | `template` 或 `method` |
| `he template validate` | 校验模板 YAML | `--json`、`--all` |
| `he config` | 管理配置 | `init`, `show`, `llm`, `embedder` |
diff --git a/docs/zh/python/guides/provenance.md b/docs/zh/python/guides/provenance.md
new file mode 100644
index 0000000..5c04adc
--- /dev/null
+++ b/docs/zh/python/guides/provenance.md
@@ -0,0 +1,168 @@
+# 来源标注与溯源
+
+追踪哪些文档为知识摘要做出过贡献,跳过未变更的重复喂养,并按文档回滚其贡献,而不影响其余知识。
+
+---
+
+## 概述
+
+**带来源标注**摄取文档(CLI 用 `--source`,Python 用 `source_id=`)会让知识摘要记住该文档:其原始提取结果会保存在**来源台账**中,并记录输入文本的 SHA-256 内容哈希。
+
+溯源提供三项能力:
+
+| 能力 | CLI | Python |
+|------|-----|--------|
+| **按文档回滚** | `he remove --document ID` | `ka.remove_source(ID)` |
+| **变更检测** | `he feed --source` 跳过未变更文档 | `ka.source_content_hash(ID)` |
+| **审计** | `he info --sources` | `ka.sources()` |
+
+图谱类知识摘要(graph、hypergraph、时序/空间图谱)默认开启来源追踪。
+
+---
+
+## 带来源标注的摄取
+
+```bash
+# 解析时标注(首个文档)
+he parse doc1.md -t general/graph -o ./ka/ -l zh --source doc-1
+
+# 喂养进现有知识库时标注
+he feed ./ka/ doc2.md --source doc-2
+```
+
+```python
+from hyperextract import Template
+
+ka = Template.create("general/graph", language="zh")
+
+with open("doc1.md") as f:
+ result = ka.parse(f.read(), source_id="doc-1")
+
+result.feed_text(doc2_text, source_id="doc-2")
+result.dump("./ka/")
+```
+
+### 变更检测
+
+同一来源再次喂养且内容未变化时,`he feed` 会比对内容哈希并完全跳过该文档——**零 LLM 调用**:
+
+```bash
+he feed ./ka/ doc2.md --source doc-2
+# Source 'doc-2' is unchanged (content hash matches) — nothing to do.
+# Use --refeed to re-ingest anyway.
+```
+
+```bash
+# 即使哈希一致也强制重新摄取
+he feed ./ka/ doc2.md --source doc-2 --refeed
+```
+
+---
+
+## 查看来源台账
+
+```bash
+he info ./ka/ --sources
+```
+
+**输出:**
+```
+ Source Ledger
+┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Source ID ┃ Raw Items ┃ Content Hash ┃
+┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ doc-1 │ 12 │ 9f86d081884c7d659a2febb0c0413a6e… │
+│ doc-2 │ 8 │ 2c26b44babbfb4c6a5cc6b3f2a4b2f1d… │
+└────────────┴───────────┴────────────────────────────────────┘
+```
+
+在 Python 中:
+
+```python
+ka.sources()
+# {
+# "doc-1": {"raw_items": 12, "content_hash": "9f86d0…"},
+# "doc-2": {"raw_items": 8, "content_hash": "2c26b4…"},
+# }
+```
+
+---
+
+## 回滚整个文档
+
+删除某个文档贡献的全部知识:
+
+```bash
+he remove ./ka/ --document doc-1
+
+# 显式指定策略
+he remove ./ka/ --document doc-1 --strategy touched
+```
+
+```python
+report = ka.remove_source("doc-1") # exact(默认)
+report = ka.remove_source("doc-1", strategy="touched")
+
+result.dump("./ka/") # 持久化
+```
+
+策略决定与其他文档**共享**的 key 如何处理:
+
+| 策略 | 共享 key | 独占 key |
+|------|----------|----------|
+| `exact`(默认) | 从幸存来源的原始结果重新合并(经典合并策略下完全确定) | 删除 |
+| `touched` | 直接删除 | 删除 |
+
+`exact` 保证其余每个文档的贡献完整保留;`touched` 保证被删文档的影响不留痕迹,代价是其他文档也支持的事实会被一并删除。使用 LLM 合并策略时,`exact` 的重新合并会保留语义、措辞近似。未知或无贡献的来源会报告 `Nothing matched` 并正常退出。无论哪种情况,已构建的搜索索引都会原地修补。
+
+!!! note
+ 基于哈希的变更检测内置于 `he feed --source`。Python 调用者可以在调用 `feed_text` 前,将 `ka.source_content_hash(source_id)` 与自行计算的文本哈希比对,实现同样效果。
+
+---
+
+## 事实级编辑
+
+若只需保留节点但移除其中一条错误描述,请使用事实级编辑而非整体回滚:
+
+```python
+report = ka.edit_node("Apple", remove_fact="founded by Steve Jobs", dry_run=True)
+ka.edit_node("Apple", remove_fact="founded by Steve Jobs") # 应用
+```
+
+```bash
+he remove ./ka/ --edit-node Apple --fact "founded by Steve Jobs" -y
+```
+
+完整安全护栏(key 不变性、预览、备份)见 [`he remove`](../../cli/commands/remove.md)。
+
+---
+
+## 持久化
+
+`ka.dump()`——以及每次 `he parse` / `he feed`——会把台账写在 `data.json` 旁边:
+
+```
+./ka/
+├── data.json # 提取的知识
+├── metadata.json # 提取元数据
+├── sources_nodes.json # 节点台账:每个来源的原始条目
+└── sources_edges.json # 边台账:每个来源的原始条目
+```
+
+- 请将这些文件与知识摘要放在一起保存(像 `data.json` 一样纳入版本控制):每个来源的原始结果只存在于这里,删除台账文件会导致该知识摘要无法回滚、也无法做变更检测。
+- 每次写入前,`he remove` 会将 `data.json` 备份为 `data.json.bak.<时间戳>`(除非传入 `--no-backup`),因此回滚本身也可以撤销。
+
+---
+
+## 路线图
+
+基于内容哈希的变更检测已随本次发布落地:`he feed --source` 会在任何 LLM 调用之前计算输入哈希并跳过未变更文档。更细粒度的溯源——把单条事实映射回它被提取自的确切文本块(chunk)——已在规划中。
+
+---
+
+## 另请参阅
+
+- [`he feed`](../../cli/commands/feed.md) — `--source` 与 `--refeed` 参数
+- [`he info`](../../cli/commands/info.md) — 查看来源台账
+- [`he remove`](../../cli/commands/remove.md) — 文档级回滚与事实级编辑参考
+- [增量更新](incremental-updates.md) — 喂养时的合并行为
diff --git a/docs/zh/python/index.md b/docs/zh/python/index.md
index ad7834d..f344083 100644
--- a/docs/zh/python/index.md
+++ b/docs/zh/python/index.md
@@ -94,6 +94,9 @@ ka = Template.create(template_path, language="zh")
# 提取
result = ka.parse(text) # 新提取
result.feed_text(text) # 添加到现有
+result.feed_text(text, source_id="doc-1") # 标注来源
+...
+result.remove_source("doc-1") # 回滚该文档贡献的事实
# 删除(图谱类 KA)
result.remove_nodes("Apple") # 硬删除节点及其孤儿边
diff --git a/hyperextract/cli/cli.py b/hyperextract/cli/cli.py
index 5425010..52ffb1d 100644
--- a/hyperextract/cli/cli.py
+++ b/hyperextract/cli/cli.py
@@ -1,5 +1,6 @@
"""CLI entry point for Hyper-Extract."""
+import hashlib
from pathlib import Path
import typer
@@ -687,9 +688,16 @@ app.add_typer(export_app, name="export")
@app.command(name="info")
-def info(ka_path: str = typer.Argument(..., help="Knowledge Abstract directory")):
+def info(
+ ka_path: str = typer.Argument(..., help="Knowledge Abstract directory"),
+ sources: bool = typer.Option(
+ False,
+ "--sources",
+ help="Show the source ledger (documents that contributed to this KA)",
+ ),
+):
"""View Knowledge Abstract information and statistics."""
- logger.info("command=info ka_path=%s", ka_path)
+ logger.info("command=info ka_path=%s sources=%s", ka_path, sources)
import json
path = validate_ka_with_data(ka_path)
@@ -735,6 +743,48 @@ def info(ka_path: str = typer.Argument(..., help="Knowledge Abstract directory")
console.print(table)
+ if sources:
+ sources_table = Table(title="Source Ledger", show_header=True)
+ sources_table.add_column("Source ID", style="cyan")
+ sources_table.add_column("Raw Items", justify="right")
+ sources_table.add_column("Content Hash")
+
+ ledger_files = [
+ (path / "sources_nodes.json", "nodes"),
+ (path / "sources_edges.json", "edges"),
+ ]
+ combined: dict = {}
+ for ledger_path, _kind in ledger_files:
+ if not ledger_path.exists():
+ continue
+ try:
+ entries = json.loads(ledger_path.read_text(encoding="utf-8"))
+ except Exception as e:
+ logger.warning(
+ "sources_ledger_read_failed path=%s error=%s", ledger_path, e
+ )
+ continue
+ for entry in entries:
+ sid = entry.get("source_id")
+ if sid is None:
+ continue
+ record = combined.setdefault(
+ sid, {"raw_items": 0, "content_hash": entry.get("content_hash")}
+ )
+ record["raw_items"] += len(entry.get("raw_items", []))
+
+ if combined:
+ for sid, info_row in sorted(combined.items()):
+ sources_table.add_row(
+ sid, str(info_row["raw_items"]), info_row["content_hash"] or "—"
+ )
+ console.print(sources_table)
+ else:
+ console.print(
+ "[yellow]No source ledger found.[/yellow] "
+ "Feed documents with --source to enable per-document rollback."
+ )
+
@app.command(name="search")
def search(
@@ -940,6 +990,11 @@ def feed(
"--source",
help="Source attribution (document id) for per-document rollback later (he remove --document)",
),
+ refeed: bool = typer.Option(
+ False,
+ "--refeed",
+ help="Re-ingest even if this source's content hash is unchanged",
+ ),
):
"""Append knowledge to an existing Knowledge Abstract."""
logger.info("command=feed ka_path=%s input=%s", ka_path, input)
@@ -985,9 +1040,24 @@ def feed(
text = read_input(input)
console.print(f"[dim]Input text: {len(text)} characters[/dim]")
+ # Change detection: skip unchanged sources without any LLM calls.
+ text_hash_to_record = None
+ if source:
+ text_hash_to_record = hashlib.sha256(text.encode("utf-8")).hexdigest()
+ if not refeed:
+ recorded = ka.source_content_hash(source)
+ if recorded == text_hash_to_record:
+ logger.info("stage=source_unchanged source=%s", source)
+ console.print(
+ f"[yellow]Source '{source}' is unchanged (content hash "
+ "matches) — nothing to do.[/yellow] "
+ "Use --refeed to re-ingest anyway."
+ )
+ raise typer.Exit(0)
+
progress.update(task, description="Appending knowledge...")
logger.debug("stage=feed_text_invoked")
- ka.feed_text(text, source_id=source)
+ ka.feed_text(text, source_id=source, content_hash=text_hash_to_record)
logger.info("stage=knowledge_appended chars=%d", len(text))
progress.update(task, description="Saving data...")
@@ -1151,6 +1221,12 @@ def remove_items(
help="Remove all knowledge contributed by this source document "
"(requires the KA to have been fed with --source)",
),
+ strategy: str = typer.Option(
+ "exact",
+ "--strategy",
+ help="Rollback strategy for --document: exact (re-merge surviving "
+ "sources) or touched (delete all affected keys)",
+ ),
):
"""Delete nodes/edges by key, remove a fact (LLM-assisted), or roll back a whole document.
@@ -1210,8 +1286,14 @@ def remove_items(
try:
if document:
+ if strategy not in ("exact", "touched"):
+ console.print(
+ f"[red]Error:[/red] Unknown strategy: {strategy!r} "
+ "(use exact or touched)."
+ )
+ raise typer.Exit(1)
try:
- report = ka.remove_source(document, strategy="exact")
+ report = ka.remove_source(document, strategy=strategy)
except KeyError:
console.print(
f"[yellow]Nothing matched — no recorded contributions "
diff --git a/hyperextract/types/base.py b/hyperextract/types/base.py
index 8f54ce1..ae64cb4 100644
--- a/hyperextract/types/base.py
+++ b/hyperextract/types/base.py
@@ -452,7 +452,11 @@ class BaseAutoType(ABC, Generic[T]):
return new_instance
def feed_text(
- self, text: str, *, source_id: str | None = None
+ self,
+ text: str,
+ *,
+ source_id: str | None = None,
+ content_hash: str | None = None,
) -> "BaseAutoType[T]":
"""
Ingests text into the CURRENT knowledge abstract instance.
@@ -466,12 +470,15 @@ class BaseAutoType(ABC, Generic[T]):
``track_sources``): the raw extraction results are recorded
under this id so the document's contributions can later be
rolled back via ``remove_source``.
+ content_hash: Optional hash of the source content (recorded in
+ the source ledger for change detection).
Returns:
Self (the current instance).
"""
logger.debug("stage=feed_text_start input_chars=%d", len(text))
self._pending_source_id = source_id
+ self._pending_content_hash = content_hash
try:
extracted_data = self._extract_data(text)
logger.debug("stage=extract_done")
@@ -481,6 +488,7 @@ class BaseAutoType(ABC, Generic[T]):
logger.debug("stage=data_merged")
finally:
self._pending_source_id = None
+ self._pending_content_hash = None
self.metadata["updated_at"] = datetime.now()
diff --git a/hyperextract/types/graph.py b/hyperextract/types/graph.py
index d622d43..f9daa86 100644
--- a/hyperextract/types/graph.py
+++ b/hyperextract/types/graph.py
@@ -98,11 +98,16 @@ class GraphEditMixin:
"""
if not source_id or not self._node_memory.track_sources:
return
+ content_hash = self._pending_content_hash
self._node_memory.record_source(
- source_id, [node.model_dump() for node in nodes]
+ source_id,
+ [node.model_dump() for node in nodes],
+ content_hash=content_hash,
)
self._edge_memory.record_source(
- source_id, [edge.model_dump() for edge in edges]
+ source_id,
+ [edge.model_dump() for edge in edges],
+ content_hash=content_hash,
)
def _adopt_source_ledger(self, other, source_id: str) -> None:
@@ -163,6 +168,32 @@ class GraphEditMixin:
and edge_report["index_patched"],
}
+ # ==================== Source Ledger (read) ====================
+
+ def sources(self) -> dict[str, dict[str, Any]]:
+ """Summarize the source ledger (node and edge memories combined).
+
+ Returns:
+ Dict mapping source_id to {raw_items, content_hash}. Raw-item
+ counts are from the node ledger; edges share the same sources.
+ """
+ summary: dict[str, dict[str, Any]] = {}
+ for source_id, info in self._node_memory.sources().items():
+ summary[source_id] = dict(info)
+ # Edge memories may know sources the node ledger lacks (empty-node
+ # extractions); surface them too.
+ for source_id, info in self._edge_memory.sources().items():
+ if source_id not in summary:
+ summary[source_id] = dict(info)
+ return summary
+
+ def source_content_hash(self, source_id: str) -> str | None:
+ """Return the recorded content hash of a source (None if unknown)."""
+ record = self._node_memory._sources.get(
+ source_id
+ ) or self._edge_memory._sources.get(source_id)
+ return record.content_hash if record else None
+
# ==================== Removal ====================
def remove_nodes(self, *keys: str) -> dict:
diff --git a/hyperextract/types/hypergraph.py b/hyperextract/types/hypergraph.py
index 54f616d..7978ca2 100644
--- a/hyperextract/types/hypergraph.py
+++ b/hyperextract/types/hypergraph.py
@@ -403,15 +403,29 @@ class AutoHypergraph(
self.clear_index()
def _update_data_state(self, incoming_data: AutoHypergraphSchema) -> None:
- """Merge incoming hypergraph data into current state."""
+ """Merge incoming hypergraph data into current state.
+
+ The vector index is patched in place: only the incoming nodes/edges
+ are re-embedded, instead of dropping the whole index.
+ """
if self.empty():
self._set_data_state(incoming_data)
- else:
+ return
+
+ # Suspend index invalidation while merging; re-embed only the
+ # affected keys afterwards.
+ with self._node_memory.suspended_index(), self._edge_memory.suspended_index():
if incoming_data.nodes:
self._node_memory.add(incoming_data.nodes)
if incoming_data.edges:
self._edge_memory.add(incoming_data.edges)
- self.clear_index()
+
+ if incoming_data.nodes:
+ node_keys = {self.key_extractor(n) for n in incoming_data.nodes}
+ self._node_memory.sync_index(upserted_keys=node_keys)
+ if incoming_data.edges:
+ edge_keys = {self.key_extractor(e) for e in incoming_data.edges}
+ self._edge_memory.sync_index(upserted_keys=edge_keys)
@property
def data(self) -> AutoHypergraphSchema:
diff --git a/mkdocs.yml b/mkdocs.yml
index c39f95c..8025ddf 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -95,6 +95,7 @@ plugins:
- python/guides/using-methods.md
- python/guides/working-with-autotypes.md
- python/guides/provider-configuration.md
+ - python/guides/provenance.md
- python/guides/custom-templates.md
- python/guides/search-and-chat.md
- python/guides/incremental-updates.md
@@ -191,6 +192,7 @@ plugins:
- zh/python/guides/using-methods.md
- zh/python/guides/working-with-autotypes.md
- zh/python/guides/provider-configuration.md
+ - zh/python/guides/provenance.md
- zh/python/guides/custom-templates.md
- zh/python/guides/search-and-chat.md
- zh/python/guides/incremental-updates.md
diff --git a/tests/cli/test_info.py b/tests/cli/test_info.py
new file mode 100644
index 0000000..fb9545c
--- /dev/null
+++ b/tests/cli/test_info.py
@@ -0,0 +1,65 @@
+"""Tests for the `he info` command (--sources source ledger view)."""
+
+import json
+
+from typer.testing import CliRunner
+
+from hyperextract.cli.cli import app
+
+runner = CliRunner()
+
+
+def _make_ka_with_sources(tmp_path, *, with_ledger: bool = True):
+ """A minimal KA directory on disk, optionally with source ledger files."""
+ ka = tmp_path / "ka"
+ ka.mkdir(parents=True)
+ (ka / "data.json").write_text(
+ json.dumps(
+ {
+ "nodes": [{"name": "mock_name", "type": "ENTITY", "description": ""}],
+ "edges": [],
+ }
+ ),
+ encoding="utf-8",
+ )
+ (ka / "metadata.json").write_text(
+ json.dumps({"template": "general/graph", "lang": "en"}),
+ encoding="utf-8",
+ )
+ if with_ledger:
+ (ka / "sources_nodes.json").write_text(
+ json.dumps(
+ [
+ {
+ "source_id": "doc-1",
+ "content_hash": "abc",
+ "raw_items": [
+ {"name": "mock_name", "type": "ENTITY", "description": ""}
+ ],
+ }
+ ]
+ ),
+ encoding="utf-8",
+ )
+ (ka / "sources_edges.json").write_text(json.dumps([]), encoding="utf-8")
+ return ka
+
+
+class TestInfoSourcesCli:
+ def test_sources_renders_source_ledger(self, tmp_path):
+ ka = _make_ka_with_sources(tmp_path)
+
+ result = runner.invoke(app, ["info", str(ka), "--sources"])
+
+ assert result.exit_code == 0, result.output
+ assert "Source Ledger" in result.output
+ assert "doc-1" in result.output
+ assert "abc" in result.output
+
+ def test_sources_without_ledger_prints_hint(self, tmp_path):
+ ka = _make_ka_with_sources(tmp_path, with_ledger=False)
+
+ result = runner.invoke(app, ["info", str(ka), "--sources"])
+
+ assert result.exit_code == 0, result.output
+ assert "No source ledger" in result.output
diff --git a/tests/cli/test_remove.py b/tests/cli/test_remove.py
index 1eed241..4119758 100644
--- a/tests/cli/test_remove.py
+++ b/tests/cli/test_remove.py
@@ -246,6 +246,76 @@ class TestDocumentRollbackCli:
assert "Nothing matched" in result.output
+class TestDocumentRollbackStrategies:
+ @staticmethod
+ def _strategy_ka(tmp_path, monkeypatch):
+ """A KA with two recorded sources sharing the node "Shared", plus
+ s1's own node "FromS1" — all seeded deterministically (no extraction
+ LLM): ledger entries via record_source, storage via direct add()."""
+ g = _real_ka()
+ g.metadata["template"] = "general/graph"
+ g.metadata["lang"] = "en"
+ g._node_memory.add(
+ [E(name="FromS1", type="X"), E(name="Shared", description="shared")]
+ )
+ g._node_memory.record_source(
+ "s1",
+ [E(name="FromS1", type="X").model_dump(), E(name="Shared").model_dump()],
+ )
+ g._node_memory.record_source("s2", [E(name="Shared").model_dump()])
+ # The edge ledger needs the source too, or remove_source bails out
+ # with KeyError before any node is touched.
+ g._edge_memory.record_source("s1", [])
+ g._edge_memory.record_source("s2", [])
+ ka = tmp_path / "ka"
+ g.dump(ka)
+ monkeypatch.setattr(climod.Template, "create", staticmethod(lambda *a, **k: g))
+ return ka, g
+
+ def test_remove_document_strategy_touched(self, tmp_path, monkeypatch):
+ ka, _ = self._strategy_ka(tmp_path, monkeypatch)
+
+ result = runner.invoke(
+ app,
+ ["remove", str(ka), "--document", "s1", "--strategy", "touched", "-y"],
+ )
+
+ assert result.exit_code == 0, result.output
+ names = [n["name"] for n in _data(ka)["nodes"]]
+ # Touched deletes every key s1 touched outright — "Shared" is gone
+ # even though s2 also contributed it (no re-merge step).
+ assert "FromS1" not in names
+ assert "Shared" not in names
+ assert "Apple" in names and "Google" in names
+ assert "Document Rollback Report" in result.output
+ assert "FromS1" in result.output
+ assert "Shared" in result.output
+
+ def test_remove_document_strategy_exact_remerges(self, tmp_path, monkeypatch):
+ from ontomem.merger import MergeStrategy, create_merger
+
+ ka, g = self._strategy_ka(tmp_path, monkeypatch)
+ # Deterministic re-merge: the default LLM.BALANCED merger would need
+ # a real LLM; MERGE_FIELD keeps this offline.
+ g._node_memory._merger = create_merger(
+ MergeStrategy.MERGE_FIELD, key_extractor=lambda x: x.name
+ )
+
+ result = runner.invoke(
+ app,
+ ["remove", str(ka), "--document", "s1", "--strategy", "exact", "-y"],
+ )
+
+ assert result.exit_code == 0, result.output
+ names = [n["name"] for n in _data(ka)["nodes"]]
+ # "Shared" survives: re-merged from s2's raw contribution. "FromS1"
+ # (s1's sole contribution) is gone.
+ assert "Shared" in names
+ assert "FromS1" not in names
+ assert "Document Rollback Report" in result.output
+ assert "Shared" in result.output
+
+
class TestArgumentValidation:
def test_hard_and_soft_delete_are_mutually_exclusive(self, ka_env):
ka, _ = ka_env