Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
docs: add Managing Documents Over Time workflow guide (EN+ZH)
yifanfeng97e7aad38423e3
diff --git a/docs/en/cli/commands/feed.md b/docs/en/cli/commands/feed.mdindex d0e9fed..f72e5d9 100644--- a/docs/en/cli/commands/feed.md+++ b/docs/en/cli/commands/feed.md@@ -203,3 +203,9 @@ he feed ./ka/ doc.md -t general/biography_graph- [`he parse`](parse.md) — Create new knowledge abstract- [`he info`](info.md) — View knowledge abstract statistics- [`he build-index`](build-index.md) — Rebuild search index after feeding++## See Also++- [`he remove --document`](remove.md) — roll back a whole source document+- [`he tag`](tag.md) — tag sources for scoped search+- [Managing Documents Over Time](managing-documents.md) — the complete lifecycle guidediff --git a/docs/en/cli/commands/managing-documents.md b/docs/en/cli/commands/managing-documents.mdnew file mode 100644index 0000000..1b3a5fa--- /dev/null+++ b/docs/en/cli/commands/managing-documents.md@@ -0,0 +1,189 @@+# Managing Documents Over Time++A knowledge abstract is not a one-shot artifact — it evolves as your source documents change. This guide walks through the complete lifecycle: **ingest → attribute → tag → search within scope → update → roll back → audit**.++---++## Prerequisites++- A graph-family knowledge abstract (graph / hypergraph / temporal / spatial)+- hyperextract installed (`pip install hyperextract`)+- An LLM provider configured (`he config init ...`)++---++## The Workflow++```mermaid+flowchart LR+ A[📄 Feed + Attribute] --> B[🏷️ Tag]+ B --> C[🔎 Scoped Search]+ C --> D[🔄 Update / Upsert]+ D --> E[↩️ Roll Back]+ E --> F[👁️ Audit]++ style A fill:#e8f5e9+ style E fill:#ffebee+ style F fill:#e3f2fd+```++### 1. Ingest with Attribution++Every document you feed can carry a `source_id` — the foundation for per-document rollback and change detection:++```bash+he feed ./ka/ contract-acme.md --source contract-acme+he feed ./ka/ contract-globex.md --source contract-globex+he parse ./more-docs/ -o ./ka/ -t legal/contract_obligation -l en --source more-docs+```++```python+ka.feed_text(text, source_id="contract-acme")+```++!!! tip+ When the input is a **directory**, `he parse` automatically attributes each file by its stem — no explicit `--source` needed.++### 2. Tag Sources++Group documents with tags so you can scope searches and batch-identify groups later:++```bash+he tag ./ka/ --source contract-acme --add legal --add acme --add reviewed+he tag ./ka/ --source contract-globex --add legal --add globex+he tag ./ka/ --list+```++### 3. Search Within Scope++Restrict retrieval to a subset of documents by source or tag:++```bash+# Only knowledge from legal-tagged documents+he search ./ka/ "termination clause" --tag legal++# Only knowledge from a specific document+he search ./ka/ "payment terms" --source contract-acme++# Union: match either filter+he search ./ka/ "liability" --tag legal --source contract-globex+```++### 4. Update a Document (Upsert)++When a source document changes, simply re-feed it under the same source id. The previous version's contributions are rolled back automatically — facts removed from the updated document disappear, and shared keys re-merge from surviving sources:++```bash+he feed ./ka/ contract-acme-v2.md --source contract-acme+```++Facts removed from the updated document **do not linger**.++### 5. Roll Back a Document++Remove every contribution of one document without touching the rest:++```bash+he remove ./ka/ --document contract-globex+```++Keys shared with surviving documents are re-merged from their raw results — deterministic with classic merge strategies. For a scorched-earth approach, use `--strategy touched` to delete every key the source touched.++You can also remove individual items:++```bash+he remove ./ka/ --node Apple+he remove ./ka/ --edit-node Apple --fact "founded by Steve Jobs"+```++### 6. Audit the Ledger++See which documents contributed to your knowledge abstract and how much:++```bash+he info ./ka/ --sources+```++```+┌─────────────────────── Source Ledger ───────────────────────┐+│ Source ID │ Raw Items │ Content Hash │ Tags │+├───────────────────┼───────────┼──────────────┼─────────────┤+│ contract-acme │ 3 │ a1b2c3d4... │ legal,acme │+│ contract-globex │ 2 │ e5f6a7b8... │ legal │+└───────────────────┴───────────┴──────────────┴─────────────┘+```++---++## Persistence++The KA directory is **self-contained**:++```+ka/+├── data.json # merged knowledge (the derived view)+├── metadata.json # template, lang, timestamps+├── sources_nodes.json # node source ledger (raw items + hashes + tags)+├── sources_edges.json # edge source ledger+├── documents/ # archived source documents+│ ├── a1b2c3d4e5f6-contract-acme.md+│ └── e5f6a7b8c9d0-contract-globex.md+└── index/ # vector index (patched in place)+```++Copy the directory and you have the complete knowledge **plus** the full evidence trail.++---++## Change Detection++`he feed --source` computes a SHA-256 content hash and records it in the ledger. Re-feeding an unchanged document is skipped automatically:++```+Source 'contract-acme' is unchanged (content hash matches) — nothing to do.+Use --refeed to re-ingest anyway.+```++Use `--refeed` to force re-ingestion (e.g. after upgrading the extraction template or LLM model).++---++## Python API++```python+from hyperextract.types import AutoGraph++ka = AutoGraph(node_schema=..., edge_schema=..., ...)++# Attributed ingestion+ka.feed_text(text, source_id="contract-acme")++# Tag+ka.tag_source("contract-acme", add=["legal", "acme"])++# Scoped search+nodes, edges = ka.search("termination", tags=["legal"])++# Document upsert (roll back old + merge new)+ka.feed_text(v2_text, source_id="contract-acme")++# Roll back a document+report = ka.remove_source("contract-globex")++# Audit+print(ka.sources())++# Persist (ledger + documents included)+ka.dump("./ka/")+```++---++## See Also++- [`he feed`](feed.md) — incremental ingestion+- [`he remove`](remove.md) — all deletion modes (key / fact / document)+- [`he tag`](tag.md) — tag management+- [`he search`](search.md) — scoped search options+- [`he info --sources`](info.md) — ledger audit+- [Provenance Guide (Python SDK)](../../python/guides/provenance.md) — the same lifecycle via the Python APIdiff --git a/docs/en/cli/commands/remove.md b/docs/en/cli/commands/remove.mdindex 04ddeff..b1310c2 100644--- a/docs/en/cli/commands/remove.md+++ b/docs/en/cli/commands/remove.md@@ -140,3 +140,9 @@ Works for graph, hypergraph, and temporal/spatial graph knowledge abstracts.- [`he clean`](clean.md) — remove the search index or the whole KA- [`he show`](show.md) — inspect keys before deleting- [`he build-index`](build-index.md) — rebuild the search index after removal++## See Also++- [`he feed --source`](feed.md) — attributed ingestion (the other half of the lifecycle)+- [`he tag`](tag.md) — tag sources for scoped search+- [Managing Documents Over Time](managing-documents.md) — the complete lifecycle guidediff --git a/docs/en/getting-started/cli-quickstart.md b/docs/en/getting-started/cli-quickstart.mdindex 88d42fb..38f629a 100644--- a/docs/en/getting-started/cli-quickstart.md+++ b/docs/en/getting-started/cli-quickstart.md@@ -198,6 +198,17 @@ he show ./output/---+## What's Next?++Your knowledge abstract is a living artifact — as your source documents change:++- **Update** a document: `he feed ./ka/ updated-doc.md --source doc-1` (re-feed under the same source to upsert)+- **Roll back** a document: `he remove ./ka/ --document doc-1`+- **Tag and scope** your searches: `he tag ./ka/ --source doc-1 --add legal`+- **Audit** the ledger: `he info ./ka/ --sources`++→ [Managing Documents Over Time](../cli/commands/managing-documents.md) — the complete guide+## Complete WorkflowHere's the typical workflow:diff --git a/docs/zh/cli/commands/feed.md b/docs/zh/cli/commands/feed.mdindex 0a432e1..2e2397c 100644--- a/docs/zh/cli/commands/feed.md+++ b/docs/zh/cli/commands/feed.md@@ -203,3 +203,9 @@ he feed ./ka/ doc.md -t general/biography_graph- [`he parse`](parse.md) — 创建新知识库- [`he info`](info.md) — 查看知识库统计信息- [`he build-index`](build-index.md) — Feed 后重建搜索索引++## 另请参阅++- [`he remove --document`](remove.md) — 回滚整份来源文档+- [`he tag`](tag.md) — 为来源打标签以启用范围检索+- [管理文档的生命周期](managing-documents.md) — 完整生命周期指南diff --git a/docs/zh/cli/commands/managing-documents.md b/docs/zh/cli/commands/managing-documents.mdnew file mode 100644index 0000000..9d0220e--- /dev/null+++ b/docs/zh/cli/commands/managing-documents.md@@ -0,0 +1,189 @@+# 管理文档的生命周期++知识摘要不是一次性产物——它随来源文档的变化而演化。本指南完整介绍文档的生命周期:**摄取 → 标注 → 打标签 → 范围检索 → 更新 → 回滚 → 审计**。++---++## 前置条件++- 图谱类知识摘要(graph / hypergraph / temporal / spatial)+- 已安装 hyperextract(`pip install hyperextract`)+- 已配置 LLM 提供商(`he config init ...`)++---++## 完整工作流++```mermaid+flowchart LR+ A[📄 喂入 + 标注] --> B[🏷️ 打标签]+ B --> C[🔎 范围检索]+ C --> D[🔄 更新 / Upsert]+ D --> E[↩️ 回滚]+ E --> F[👁️ 审计]++ style A fill:#e8f5e9+ style E fill:#ffebee+ style F fill:#e3f2fd+```++### 1. 带来源标注摄取++每份文档都可以携带 `source_id`——这是按文档回滚和变更检测的基础:++```bash+he feed ./ka/ contract-acme.md --source contract-acme+he feed ./ka/ contract-globex.md --source contract-globex+he parse ./more-docs/ -o ./ka/ -t legal/contract_obligation -l en --source more-docs+```++```python+ka.feed_text(text, source_id="contract-acme")+```++!!! tip+ 当输入是**目录**时,`he parse` 会自动按文件名标注每个文件——无需显式 `--source`。++### 2. 打标签++用标签对文档分组,以便后续范围检索和批量识别:++```bash+he tag ./ka/ --source contract-acme --add legal --add acme --add reviewed+he tag ./ka/ --source contract-globex --add legal --add globex+he tag ./ka/ --list+```++### 3. 范围检索++按来源或标签将检索限定在部分文档范围内:++```bash+# 只检索 legal 标签文档中的知识+he search ./ka/ "termination clause" --tag legal++# 只检索特定文档贡献的知识+he search ./ka/ "payment terms" --source contract-acme++# 并集:命中任一过滤条件即保留+he search ./ka/ "liability" --tag legal --source contract-globex+```++### 4. 更新文档(Upsert)++来源文档变化后,用同一个 source id 重新喂入即可。旧版本的贡献会自动回滚——更新后删掉的事实不再残留,共享 key 从幸存来源重新合并:++```bash+he feed ./ka/ contract-acme-v2.md --source contract-acme+```++更新后删掉的事实**不会残留**。++### 5. 回滚整份文档++删除某份文档的全部贡献,不影响其余知识:++```bash+he remove ./ka/ --document contract-globex+```++与其他文档共享的 key 会从幸存来源的原始结果重新合并——经典合并策略下完全确定。如需"全部删除"可用 `--strategy touched`。++也可以删除单个条目:++```bash+he remove ./ka/ --node Apple+he remove ./ka/ --edit-node Apple --fact "founded by Steve Jobs"+```++### 6. 审计来源账本++查看哪些文档贡献了知识以及贡献量:++```bash+he info ./ka/ --sources+```++```+┌─────────────────────── 来源账本 ───────────────────────────┐+│ 来源 ID │ 原始条目 │ 内容哈希 │ 标签 │+├───────────────────┼───────────┼──────────────┼─────────────┤+│ contract-acme │ 3 │ a1b2c3d4... │ legal,acme │+│ contract-globex │ 2 │ e5f6a7b8... │ legal │+└───────────────────┴───────────┴──────────────┴─────────────┘+```++---++## 持久化++KA 目录是**自包含**的:++```+ka/+├── data.json # 合并后的知识(派生视图)+├── metadata.json # 模板、语言、时间戳+├── sources_nodes.json # 节点来源账本(原始结果 + 哈希 + 标签)+├── sources_edges.json # 边来源账本+├── documents/ # 归档的来源文档+│ ├── a1b2c3d4e5f6-contract-acme.md+│ └── e5f6a7b8c9d0-contract-globex.md+└── index/ # 向量索引(原地修补)+```++复制整个目录,即可带走完整的知识**和**完整的证据链。++---++## 变更检测++`he feed --source` 会计算输入文本的 SHA-256 哈希并记入账本。重复喂入内容未变的文档会自动跳过:++```+Source 'contract-acme' is unchanged (content hash matches) — nothing to do.+Use --refeed to re-ingest anyway.+```++使用 `--refeed` 可强制重新摄取(例如升级抽取模板或 LLM 模型后)。++---++## Python API++```python+from hyperextract.types import AutoGraph++ka = AutoGraph(node_schema=..., edge_schema=..., ...)++# 带来源标注摄取+ka.feed_text(text, source_id="contract-acme")++# 打标签+ka.tag_source("contract-acme", add=["legal", "acme"])++# 范围检索+nodes, edges = ka.search("termination", tags=["legal"])++# 文档 upsert(回滚旧版 + 合并新版)+ka.feed_text(v2_text, source_id="contract-acme")++# 回滚整份文档+report = ka.remove_source("contract-globex")++# 审计+print(ka.sources())++# 持久化(含账本和文档)+ka.dump("./ka/")+```++---++## 另请参阅++- [`he feed`](feed.md) — 增量摄取+- [`he remove`](remove.md) — 所有删除模式(按键 / 按事实 / 按文档)+- [`he tag`](tag.md) — 标签管理+- [`he search`](search.md) — 范围检索选项+- [`he info --sources`](info.md) — 来源账本审计+- [溯源指南(Python SDK)](../../python/guides/provenance.md) — 同一生命周期的 Python API 视角diff --git a/docs/zh/cli/commands/remove.md b/docs/zh/cli/commands/remove.mdindex 153f7ec..4283a8f 100644--- a/docs/zh/cli/commands/remove.md+++ b/docs/zh/cli/commands/remove.md@@ -140,3 +140,9 @@ ka.dump("./ka/") # 持久化- [`he clean`](clean.md) — 删除搜索索引或整个 KA- [`he show`](show.md) — 删除前查看 key- [`he build-index`](build-index.md) — 删除后重建搜索索引++## 另请参阅++- [`he feed --source`](feed.md) — 来源标注摄取(生命周期的另一半)+- [`he tag`](tag.md) — 为来源打标签以启用范围检索+- [管理文档的生命周期](managing-documents.md) — 完整生命周期指南diff --git a/docs/zh/getting-started/cli-quickstart.md b/docs/zh/getting-started/cli-quickstart.mdindex 79af322..0714a33 100644--- a/docs/zh/getting-started/cli-quickstart.md+++ b/docs/zh/getting-started/cli-quickstart.md@@ -193,6 +193,17 @@ he show ./output/---+## 下一步?++你的知识摘要是活的——随来源文档的变化持续演化:++- **更新**文档:`he feed ./ka/ updated-doc.md --source doc-1`(同 source 重喂即 upsert)+- **回滚**文档:`he remove ./ka/ --document doc-1`+- **打标签并范围检索**:`he tag ./ka/ --source doc-1 --add legal`+- **审计**来源账本:`he info ./ka/ --sources`++→ [管理文档的生命周期](../cli/commands/managing-documents.md) — 完整指南+## 完整工作流程以下是典型的工作流程:diff --git a/mkdocs.yml b/mkdocs.ymlindex e052843..ad62ff9 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -78,6 +78,7 @@ plugins:- cli/commands/build-index.md- cli/commands/clean.md- cli/commands/remove.md+ - cli/commands/managing-documents.md- cli/commands/search.md- cli/commands/tag.md- cli/commands/talk.md@@ -176,6 +177,7 @@ plugins:- zh/cli/commands/build-index.md- zh/cli/commands/clean.md- zh/cli/commands/remove.md+ - zh/cli/commands/managing-documents.md- zh/cli/commands/search.md- zh/cli/commands/tag.md- zh/cli/commands/talk.md