Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

feat: per-document rollback — source threading, he remove --document, feed-path index preservation

yifanfeng971170ddfa30c9
diff --git a/docs/en/cli/commands/feed.md b/docs/en/cli/commands/feed.md
index 8bdc72e..93caf2f 100644
--- a/docs/en/cli/commands/feed.md
+++ b/docs/en/cli/commands/feed.md
@@ -23,6 +23,7 @@ he feed KA_PATH INPUT [OPTIONS]
|--------|-------|-------------|
| `--template` | `-t` | Override template (uses metadata if omitted) |
| `--lang` | `-l` | Override language (uses metadata if omitted) |
+| `--source` | — | Source attribution (document id) — enables per-document rollback via `he remove --document` |
---
diff --git a/docs/en/cli/commands/parse.md b/docs/en/cli/commands/parse.md
index 9f171c5..1cb19ba 100644
--- a/docs/en/cli/commands/parse.md
+++ b/docs/en/cli/commands/parse.md
@@ -24,6 +24,7 @@ he parse INPUT [OPTIONS]
| `--template` | `-t` | Template to use (omit for interactive selection) |
| `--method` | `-m` | Method template (e.g., `light_rag`, `graph_rag`) |
| `--lang` | `-l` | Language: `zh` or `en` (required for knowledge templates) |
+| `--source` | — | Source attribution (document id) — enables per-document rollback via `he remove --document` |
| `--force` | `-f` | Force overwrite existing output |
| `--no-index` | — | Skip building search index |
diff --git a/docs/en/cli/commands/remove.md b/docs/en/cli/commands/remove.md
index 7d12e40..74c78de 100644
--- a/docs/en/cli/commands/remove.md
+++ b/docs/en/cli/commands/remove.md
@@ -18,6 +18,7 @@ he remove KA_PATH [OPTIONS]
| `--edge TEXT` | — | Edge key(s) to **hard-delete** |
| `--edit-node TEXT` | — | Node key to **soft-edit** (LLM rewrites it minus `--fact`) |
| `--edit-edge TEXT` | — | Edge key to **soft-edit** |
+| `--document TEXT` | — | **Roll back** all knowledge contributed by a source document (requires the KA to have been fed with `--source`) |
| `--fact TEXT` | — | The fact to remove from the `--edit-node` / `--edit-edge` item |
| `--instruction TEXT` | — | Free-form edit instruction (alternative to `--fact`) |
| `--dry-run` | off | Preview the change without persisting it |
@@ -89,6 +90,23 @@ he build-index ./ka/
---
+## Document Rollback: remove a whole source document
+
+When documents were ingested with a source attribution, their entire contribution can be rolled back — including keys that other documents also contributed to (those are re-merged from the surviving sources).
+
+```bash
+# Ingest with attribution
+he feed ./ka/ doc1.md --source doc-1
+he feed ./ka/ doc2.md --source doc-2
+
+# Later: roll back everything doc-1 contributed ("b" remains via doc-2 if shared)
+he remove ./ka/ --document doc-1
+```
+
+Requires the KA's memories to track sources (graph-family KAs do by default). The rollback re-merges affected keys from the surviving sources' raw results — deterministic with classic merge strategies; LLM strategies preserve semantics with approximate wording. If the document has no recorded contributions, the command reports `Nothing matched` and exits successfully.
+
+---
+
## Python API
```python
diff --git a/docs/zh/cli/commands/feed.md b/docs/zh/cli/commands/feed.md
index b8cbed8..276ec41 100644
--- a/docs/zh/cli/commands/feed.md
+++ b/docs/zh/cli/commands/feed.md
@@ -23,6 +23,7 @@ he feed KA_PATH INPUT [OPTIONS]
|--------|-------|-------------|
| `--template` | `-t` | 覆盖模板(如果省略则使用元数据) |
| `--lang` | `-l` | 覆盖语言(如果省略则使用元数据) |
+| `--source` | — | 来源标注(文档 id)——之后可通过 `he remove --document` 按文档回滚 |
---
diff --git a/docs/zh/cli/commands/parse.md b/docs/zh/cli/commands/parse.md
index 15c36e8..42d20e0 100644
--- a/docs/zh/cli/commands/parse.md
+++ b/docs/zh/cli/commands/parse.md
@@ -24,6 +24,7 @@ he parse INPUT [OPTIONS]
| `--template` | `-t` | 要使用的模板(省略以进行交互式选择) |
| `--method` | `-m` | 方法模板(例如 `light_rag`、`graph_rag`) |
| `--lang` | `-l` | 语言:`zh` 或 `en`(知识模板必填) |
+| `--source` | — | 来源标注(文档 id)——之后可通过 `he remove --document` 按文档回滚 |
| `--force` | `-f` | 强制覆盖现有输出 |
| `--no-index` | — | 跳过构建搜索索引 |
diff --git a/docs/zh/cli/commands/remove.md b/docs/zh/cli/commands/remove.md
index d8011d2..19e63fe 100644
--- a/docs/zh/cli/commands/remove.md
+++ b/docs/zh/cli/commands/remove.md
@@ -18,6 +18,7 @@ he remove KA_PATH [选项]
| `--edge TEXT` | — | **硬删除**指定边 |
| `--edit-node TEXT` | — | **软编辑**指定节点(大模型重写,移除 `--fact`) |
| `--edit-edge TEXT` | — | **软编辑**指定边 |
+| `--document TEXT` | — | **回滚**某个来源文档贡献的全部知识(要求 KA 在摄取时使用了 `--source`) |
| `--fact TEXT` | — | 要从 `--edit-node` / `--edit-edge` 条目中移除的事实 |
| `--instruction TEXT` | — | 自由格式的编辑指令(替代 `--fact`) |
| `--dry-run` | 关闭 | 仅预览变更,不落盘 |
@@ -89,6 +90,23 @@ he build-index ./ka/
---
+## 文档级回滚:删除整个来源文档的贡献
+
+当文档以来源标注方式摄取后,可以整体回滚其贡献——包括与其他文档共享的 key(这些 key 会从幸存来源的原始结果重新合并)。
+
+```bash
+# 带来源标注摄取
+he feed ./ka/ doc1.md --source doc-1
+he feed ./ka/ doc2.md --source doc-2
+
+# 之后:回滚 doc-1 贡献的全部知识
+he remove ./ka/ --document doc-1
+```
+
+要求 KA 的记忆开启来源追踪(图谱类 KA 默认开启)。回滚会从幸存来源的原始结果重新合并受影响的 key——经典合并策略下完全确定;LLM 策略保留语义、措辞近似。若该文档没有已记录的贡献,命令会报告 `Nothing matched` 并正常退出。
+
+---
+
## Python API
```python
diff --git a/hyperextract/cli/cli.py b/hyperextract/cli/cli.py
index db023aa..5425010 100644
--- a/hyperextract/cli/cli.py
+++ b/hyperextract/cli/cli.py
@@ -247,6 +247,11 @@ def parse(
no_index: bool = typer.Option(
False, "--no-index", help="Skip building search index"
),
+ source: str | None = typer.Option(
+ None,
+ "--source",
+ help="Source attribution (document id) for per-document rollback later (he remove --document)",
+ ),
):
"""Extract knowledge from text to a new directory."""
logger.info(
@@ -341,7 +346,7 @@ def parse(
progress.update(task, description="Extracting knowledge...")
logger.debug("stage=feed_text_invoked")
- ka.feed_text(combined_text)
+ ka.feed_text(combined_text, source_id=source)
logger.info("stage=knowledge_extracted chars=%d", len(combined_text))
else:
progress.update(task, description="Reading input...")
@@ -350,7 +355,7 @@ def parse(
progress.update(task, description="Extracting knowledge...")
logger.debug("stage=feed_text_invoked")
- ka.feed_text(text)
+ ka.feed_text(text, source_id=source)
logger.info("stage=knowledge_extracted chars=%d", len(text))
progress.update(task, description="Saving data...")
@@ -930,6 +935,11 @@ def feed(
input: str = typer.Argument(..., help="Input file path or '-' for stdin"),
template: str | None = typer.Option(None, "--template", "-t", help="Template"),
lang: str | None = typer.Option(None, "--lang", "-l", help="Language"),
+ source: str | None = typer.Option(
+ None,
+ "--source",
+ help="Source attribution (document id) for per-document rollback later (he remove --document)",
+ ),
):
"""Append knowledge to an existing Knowledge Abstract."""
logger.info("command=feed ka_path=%s input=%s", ka_path, input)
@@ -977,7 +987,7 @@ def feed(
progress.update(task, description="Appending knowledge...")
logger.debug("stage=feed_text_invoked")
- ka.feed_text(text)
+ ka.feed_text(text, source_id=source)
logger.info("stage=knowledge_appended chars=%d", len(text))
progress.update(task, description="Saving data...")
@@ -1135,33 +1145,51 @@ def remove_items(
True, "--backup/--no-backup", help="Back up data.json before writing"
),
yes: bool = typer.Option(False, "--yes", "-y", help="Skip the confirmation prompt"),
+ document: str | None = typer.Option(
+ None,
+ "--document",
+ help="Remove all knowledge contributed by this source document "
+ "(requires the KA to have been fed with --source)",
+ ),
):
- """Delete nodes/edges by key, or soft-remove a single fact (LLM-assisted).
+ """Delete nodes/edges by key, remove a fact (LLM-assisted), or roll back a whole document.
- Hard delete: he remove ./ka --node Apple --edge "Apple-partner-Google"
- Soft delete: he remove ./ka --edit-node Apple --fact "founded by Steve Jobs"
+ Hard delete: he remove ./ka --node Apple --edge "Apple-partner-Google"
+ Soft delete: he remove ./ka --edit-node Apple --fact "founded by Steve Jobs"
+ Document rollback: he remove ./ka --document ./docs/old-paper.md
"""
import shutil
from datetime import datetime
- logger.info("command=remove ka_path=%s dry_run=%s", ka_path, dry_run)
+ logger.info(
+ "command=remove ka_path=%s dry_run=%s document=%s",
+ ka_path,
+ dry_run,
+ document,
+ )
hard_targets = list(node or []) + list(edge or [])
soft_target = edit_node_key or edit_edge_key
- if hard_targets and soft_target:
+ if (hard_targets or soft_target or document) and sum(
+ bool(x) for x in (hard_targets, soft_target, document)
+ ) > 1:
console.print(
- "[red]Error:[/red] Combine either --node/--edge (hard delete) "
- "or --edit-node/--edit-edge (soft delete), not both."
+ "[red]Error:[/red] Choose ONE of: --node/--edge (hard delete), "
+ "--edit-node/--edit-edge (soft delete), or --document "
+ "(whole-document rollback)."
)
raise typer.Exit(1)
if soft_target and not fact and not instruction:
console.print("[red]Error:[/red] Soft delete needs --fact or --instruction.")
raise typer.Exit(1)
- if not hard_targets and not soft_target:
+ if document and not (Path(document).name or document):
+ pass # unreachable; keeps option semantics explicit
+ if not hard_targets and not soft_target and not document:
console.print(
"[yellow]Nothing to do.[/yellow] "
- "Use --node/--edge to delete items, or --edit-node/--edit-edge "
- "with --fact to remove a single fact."
+ "Use --node/--edge to delete items, --edit-node/--edit-edge "
+ "with --fact to remove a single fact, or --document to roll back "
+ "a whole source document."
)
raise typer.Exit(0)
@@ -1181,7 +1209,17 @@ def remove_items(
raise typer.Exit(1)
try:
- if hard_targets:
+ if document:
+ try:
+ report = ka.remove_source(document, strategy="exact")
+ except KeyError:
+ console.print(
+ f"[yellow]Nothing matched — no recorded contributions "
+ f"for source '{document}'. Was the KA fed with "
+ f"--source?[/yellow]"
+ )
+ raise typer.Exit(0)
+ elif hard_targets:
report = {}
if node:
report.update(ka.remove_nodes(*node))
@@ -1218,7 +1256,31 @@ def remove_items(
raise typer.Exit(1)
console.print()
- if hard_targets:
+ if document:
+ table = Table(title=f"Document Rollback Report — {document}")
+ table.add_column("Field")
+ table.add_column("Items")
+ for field, items in report.items():
+ if field in ("index_patched", "source_id", "strategy"):
+ continue
+ table.add_row(field, "\n".join(items) if items else "—")
+ console.print(table)
+ changed_any = any(
+ report.get(key)
+ for key in (
+ "removed_nodes",
+ "remerged_nodes",
+ "removed_edges",
+ "remerged_edges",
+ )
+ )
+ if not changed_any:
+ console.print(
+ "[yellow]Nothing matched — this document has no recorded "
+ "contributions. Was the KA fed with --source?[/yellow]"
+ )
+ raise typer.Exit(0)
+ elif hard_targets:
table = Table(title="Removal Report")
table.add_column("Field")
table.add_column("Items")
diff --git a/hyperextract/types/base.py b/hyperextract/types/base.py
index 52711fb..8f54ce1 100644
--- a/hyperextract/types/base.py
+++ b/hyperextract/types/base.py
@@ -97,6 +97,9 @@ class BaseAutoType(ABC, Generic[T]):
"created_at": datetime.now(),
"updated_at": datetime.now(),
}
+ # Set by parse()/feed_text() while extraction runs; graph-family
+ # subclasses record raw extraction results under it (provenance).
+ self._pending_source_id: str | None = None
def _create_empty_instance(self) -> "BaseAutoType[T]":
"""Creates a new empty instance with the same configuration as this one.
@@ -242,7 +245,12 @@ class BaseAutoType(ABC, Generic[T]):
# ==================== Extraction & Merge ====================
def _extract_data(self, text: str) -> T:
- """Internal: Unified extraction logic (Chunking -> LLM -> Merge)."""
+ """Internal: Unified extraction logic (Chunking -> LLM -> Merge).
+
+ Subclasses that support source attribution read
+ ``self._pending_source_id`` (set by parse/feed_text) and record
+ their raw, pre-merge results via ``_record_extraction_source``.
+ """
logger.debug(
"stage=extract_start input_chars=%d chunk_size=%d",
len(text),
@@ -377,6 +385,20 @@ class BaseAutoType(ABC, Generic[T]):
results.append(r)
return results
+ def _adopt_source_ledger(self, other: "BaseAutoType[T]", source_id: str) -> None:
+ """Hook: transfer ledger entries for ``source_id`` from ``other`` to
+ ``self`` after parse(). No-op by default; graph-family memories with
+ ``track_sources`` override this."""
+ return
+
+ def _dump_provenance(self, root: Path) -> None:
+ """Hook: persist the source ledger alongside the KA. No-op by default."""
+ return
+
+ def _load_provenance(self, root: Path) -> None:
+ """Hook: restore the source ledger alongside the KA. No-op by default."""
+ return
+
def _summarize_extracted(self, data: T) -> str:
"""Return a concise summary of extracted data for debug logging."""
try:
@@ -397,7 +419,7 @@ class BaseAutoType(ABC, Generic[T]):
except Exception:
return repr(data)[:100]
- def parse(self, text: str) -> "BaseAutoType[T]":
+ def parse(self, text: str, *, source_id: str | None = None) -> "BaseAutoType[T]":
"""
Parses knowledge into a NEW instance without modifying the current one.
@@ -405,21 +427,33 @@ class BaseAutoType(ABC, Generic[T]):
Args:
text: Input text.
+ source_id: Optional source attribution (graph-family memories with
+ ``track_sources``): the raw extraction results are recorded
+ under this id so the document's contributions can later be
+ rolled back via ``remove_source``.
Returns:
A new knowledge instance containing only the parsed data.
"""
- parsed_data = self._extract_data(text)
+ self._pending_source_id = source_id
+ try:
+ parsed_data = self._extract_data(text)
+ finally:
+ self._pending_source_id = None
new_instance = self._create_empty_instance()
new_instance._set_data_state(parsed_data)
+ if source_id:
+ new_instance._adopt_source_ledger(self, source_id)
new_instance.metadata["created_at"] = datetime.now()
new_instance.metadata["updated_at"] = datetime.now()
return new_instance
- def feed_text(self, text: str) -> "BaseAutoType[T]":
+ def feed_text(
+ self, text: str, *, source_id: str | None = None
+ ) -> "BaseAutoType[T]":
"""
Ingests text into the CURRENT knowledge abstract instance.
@@ -428,17 +462,25 @@ class BaseAutoType(ABC, Generic[T]):
Args:
text: Input text.
+ source_id: Optional source attribution (graph-family memories with
+ ``track_sources``): the raw extraction results are recorded
+ under this id so the document's contributions can later be
+ rolled back via ``remove_source``.
Returns:
Self (the current instance).
"""
logger.debug("stage=feed_text_start input_chars=%d", len(text))
- extracted_data = self._extract_data(text)
- logger.debug("stage=extract_done")
+ self._pending_source_id = source_id
+ try:
+ extracted_data = self._extract_data(text)
+ logger.debug("stage=extract_done")
- # Use UPDATE hook instead of manual merge+set
- self._update_data_state(extracted_data)
- logger.debug("stage=data_merged")
+ # Use UPDATE hook instead of manual merge+set
+ self._update_data_state(extracted_data)
+ logger.debug("stage=data_merged")
+ finally:
+ self._pending_source_id = None
self.metadata["updated_at"] = datetime.now()
@@ -587,6 +629,9 @@ class BaseAutoType(ABC, Generic[T]):
# user can always rebuild_index()
print(f"Warning: Failed to save vector index: {e}")
+ # 4. Provenance (Optional — graph-family with track_sources)
+ self._dump_provenance(root)
+
def load(self, folder_path: str | Path) -> None:
"""Loads the entire knowledge abstract from a directory.
@@ -615,6 +660,9 @@ class BaseAutoType(ABC, Generic[T]):
f"Warning: Failed to load vector index: {e}. You may need to rebuild_index()."
)
+ # 4. Provenance (Optional — graph-family with track_sources)
+ self._load_provenance(root)
+
# ==================== Serialization: Components ====================
def dump_data(self, file_path: str | Path) -> None:
diff --git a/hyperextract/types/graph.py b/hyperextract/types/graph.py
index eed515c..d622d43 100644
--- a/hyperextract/types/graph.py
+++ b/hyperextract/types/graph.py
@@ -83,6 +83,86 @@ class GraphEditMixin:
OMem's `sync_index` / `edit` (ontomem >= 0.3.0).
"""
+ # ==================== Source Provenance (v0.4.0+) ====================
+
+ def _record_extraction_source(
+ self,
+ source_id: str | None,
+ nodes: list[NodeSchema],
+ edges: list[EdgeSchema],
+ ) -> None:
+ """Record raw (pre-merge) extraction results in the source ledgers.
+
+ Reads ``self._pending_source_id`` (set by parse/feed_text); no-op when
+ no source was attributed or the memories don't track sources.
+ """
+ if not source_id or not self._node_memory.track_sources:
+ return
+ self._node_memory.record_source(
+ source_id, [node.model_dump() for node in nodes]
+ )
+ self._edge_memory.record_source(
+ source_id, [edge.model_dump() for edge in edges]
+ )
+
+ def _adopt_source_ledger(self, other, source_id: str) -> None:
+ """Transfer ledger entries for ``source_id`` from ``other`` to ``self``
+ after parse() (the extraction ran on ``other``'s memories)."""
+ for src_memory, dst_memory in (
+ (other._node_memory, self._node_memory),
+ (other._edge_memory, self._edge_memory),
+ ):
+ if source_id in src_memory._sources:
+ dst_memory._sources[source_id] = src_memory._sources[source_id]
+
+ def _dump_provenance(self, root: Path) -> None:
+ """Persist the node/edge source ledgers alongside the KA."""
+ try:
+ self._node_memory.dump_sources(Path(root) / "sources_nodes.json")
+ self._edge_memory.dump_sources(Path(root) / "sources_edges.json")
+ except Exception as e:
+ logger.warning("provenance_dump_failed", error=str(e))
+
+ def _load_provenance(self, root: Path) -> None:
+ """Restore the node/edge source ledgers alongside the KA."""
+ try:
+ node_path = Path(root) / "sources_nodes.json"
+ edge_path = Path(root) / "sources_edges.json"
+ if node_path.exists():
+ self._node_memory.load_sources(node_path)
+ if edge_path.exists():
+ self._edge_memory.load_sources(edge_path)
+ except Exception as e:
+ logger.warning("provenance_load_failed", error=str(e))
+
+ def remove_source(self, source_id: str, *, strategy: str = "exact") -> dict:
+ """Remove every contribution of one source document (exact rollback).
+
+ Keys contributed solely by the removed document are deleted; keys
+ shared with other documents are re-merged from the surviving sources'
+ raw results. The search index is patched in place when built.
+
+ Args:
+ source_id: Identifier used when the document was fed
+ (``feed_text(text, source_id=...)`` / ``he feed --source``).
+ strategy: "exact" (re-merge survivors) or "touched" (delete all
+ affected keys).
+
+ Returns:
+ Combined node/edge report dict.
+ """
+ node_report = self._node_memory.remove_source(source_id, strategy=strategy)
+ edge_report = self._edge_memory.remove_source(source_id, strategy=strategy)
+ return {
+ "source_id": source_id,
+ "removed_nodes": node_report["removed_keys"],
+ "remerged_nodes": node_report["remerged_keys"],
+ "removed_edges": edge_report["removed_keys"],
+ "remerged_edges": edge_report["remerged_keys"],
+ "index_patched": node_report["index_patched"]
+ and edge_report["index_patched"],
+ }
+
# ==================== Removal ====================
def remove_nodes(self, *keys: str) -> dict:
@@ -270,8 +350,8 @@ class EdgeListSchema(BaseModel, Generic[EdgeSchema]):
class AutoGraph(
- BaseAutoType[AutoGraphSchema[NodeSchema, EdgeSchema]],
GraphEditMixin,
+ BaseAutoType[AutoGraphSchema[NodeSchema, EdgeSchema]],
Generic[NodeSchema, EdgeSchema],
):
"""AutoGraph - extracts knowledge graphs with nodes and edges from text.
@@ -446,6 +526,7 @@ class AutoGraph(
strategy_or_merger=self.node_merger,
verbose=verbose,
fields_for_index=node_fields_for_index, # Pass node field selection to OMem
+ track_sources=True, # source ledger for per-document rollback (#84)
)
self._edge_memory = OMem(
@@ -456,6 +537,7 @@ class AutoGraph(
strategy_or_merger=self.edge_merger,
verbose=verbose,
fields_for_index=edge_fields_for_index, # Pass edge field selection to OMem
+ track_sources=True, # source ledger for per-document rollback (#84)
)
# Store label extractors for visualization
@@ -602,17 +684,30 @@ class AutoGraph(
) -> None:
"""Merge incoming graph data into current state.
+ The vector index is patched in place: only the incoming nodes/edges
+ are re-embedded, instead of dropping the whole index.
+
Args:
incoming_data: Incremental graph data to merge.
"""
if self.empty():
self._set_data_state(incoming_data)
- else:
+ return
+
+ # Suspend index invalidation while merging; re-embed only the
+ # affected keys afterwards.
+ with self._node_memory.suspended_index(), self._edge_memory.suspended_index():
if incoming_data.nodes:
self._node_memory.add(incoming_data.nodes)
if incoming_data.edges:
self._edge_memory.add(incoming_data.edges)
- self.clear_index()
+
+ if incoming_data.nodes:
+ node_keys = {self.node_key_extractor(n) for n in incoming_data.nodes}
+ self._node_memory.sync_index(upserted_keys=node_keys)
+ if incoming_data.edges:
+ edge_keys = {self.edge_key_extractor(e) for e in incoming_data.edges}
+ self._edge_memory.sync_index(upserted_keys=edge_keys)
# ==================== Extraction Pipeline ====================
@@ -674,6 +769,13 @@ class AutoGraph(
logger.debug("stage=one_stage_batch_complete graphs=%d", len(graph_list))
logger.debug("stage=one_stage_merge_start")
+ # Provenance: record the raw (pre-merge) chunk results under the
+ # pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for graph in graph_list for node in graph.nodes],
+ [edge for graph in graph_list for edge in graph.edges],
+ )
result = self.merge_batch_data(graph_list)
logger.debug(
"stage=one_stage_merge_complete nodes=%d edges=%d",
@@ -751,6 +853,14 @@ class AutoGraph(
[edge_list.items for edge_list in chunk_edge_lists],
)
+ # Provenance: record the raw (pre-merge) node/edge results under the
+ # pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for node_list in chunk_node_lists for node in node_list.items],
+ [edge for edge_list in chunk_edge_lists for edge in edge_list.items],
+ )
+
# 5. Global Merge (passes tuples to merge_batch_data)
logger.debug("stage=two_stage_merge_start")
result = self.merge_batch_data(partial_graphs)
diff --git a/hyperextract/types/hypergraph.py b/hyperextract/types/hypergraph.py
index 9f08305..54f616d 100644
--- a/hyperextract/types/hypergraph.py
+++ b/hyperextract/types/hypergraph.py
@@ -107,8 +107,8 @@ class EdgeListSchema(BaseModel, Generic[EdgeSchema]):
class AutoHypergraph(
- BaseAutoType[AutoHypergraphSchema[NodeSchema, EdgeSchema]],
GraphEditMixin,
+ BaseAutoType[AutoHypergraphSchema[NodeSchema, EdgeSchema]],
Generic[NodeSchema, EdgeSchema],
):
"""AutoHypergraph - Node-First extraction: Extract entities first, then relationships.
@@ -286,6 +286,7 @@ class AutoHypergraph(
strategy_or_merger=self.node_merger,
verbose=verbose,
fields_for_index=node_fields_for_index, # Pass node field selection to OMem
+ track_sources=True, # source ledger for per-document rollback (#84)
)
self._edge_memory = OMem(
@@ -296,6 +297,7 @@ class AutoHypergraph(
strategy_or_merger=self.edge_merger,
verbose=verbose,
fields_for_index=edge_fields_for_index, # Pass edge field selection to OMem
+ track_sources=True, # source ledger for per-document rollback (#84)
)
# Store label extractors for visualization
@@ -475,6 +477,14 @@ class AutoHypergraph(
default_factory=lambda: self.graph_schema(nodes=[], edges=[]),
)
+ # Provenance: record the raw (pre-merge) chunk results under the
+ # pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for graph in graph_list for node in graph.nodes],
+ [edge for graph in graph_list for edge in graph.edges],
+ )
+
# Merge multiple hypergraphs
return self.merge_batch_data(graph_list)
@@ -519,6 +529,13 @@ class AutoHypergraph(
]
# 4. Construct Partial Graphs (Tuple format for merge optimization)
+ # Provenance: record the raw (pre-merge) node/hyperedge results under
+ # the pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for node_list in chunk_node_lists for node in node_list.items],
+ [edge for edge_list in chunk_edge_lists for edge in edge_list.items],
+ )
partial_hypergraphs = (
[node_list.items for node_list in chunk_node_lists],
[edge_list.items for edge_list in chunk_edge_lists],
diff --git a/hyperextract/types/set.py b/hyperextract/types/set.py
index 744388f..959a6ca 100644
--- a/hyperextract/types/set.py
+++ b/hyperextract/types/set.py
@@ -469,9 +469,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
Args:
item: The item to add.
"""
- # OMem handles deduplication and merging automatically
- self._data_memory.add([item])
- self.clear_index()
+ # OMem handles deduplication and merging automatically; the vector
+ # index is patched in place (only the added key is re-embedded).
+ with self._data_memory.suspended_index():
+ self._data_memory.add([item])
+ self._data_memory.sync_index(upserted_keys=[self.key_extractor(item)])
self.metadata["updated_at"] = datetime.now()
def remove(self, key: Any) -> ItemSchema | None:
@@ -488,10 +490,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
if item is None:
return None
- # Remove from OMem
- removed = self._data_memory.remove(key)
- if removed:
- self.clear_index()
+ # Remove from OMem; sync_index patches the vector index in place
+ # (only the removed key's vector is dropped — no full rebuild).
+ removed_keys, _ = self._data_memory.remove_many([key])
+ if removed_keys:
+ self._data_memory.sync_index(removed_keys=removed_keys)
self.metadata["updated_at"] = datetime.now()
logger.debug(f"Removed item with key '{key}'")
return item
@@ -528,8 +531,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
"""
# add() takes a single item and would wrap the whole list as one entry;
# pass the list straight to the backing store so each item is added.
- self._data_memory.add(items)
- self.clear_index()
+ # The vector index is patched in place afterwards (no full rebuild).
+ with self._data_memory.suspended_index():
+ self._data_memory.add(items)
+ added_keys = {self.key_extractor(item) for item in items}
+ self._data_memory.sync_index(upserted_keys=added_keys)
self.metadata["updated_at"] = datetime.now()
def discard(self, key: Any) -> None:
@@ -544,13 +550,14 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
>>> skills.discard("Python") # No error if not found
Side Effects:
- - Clears the vector index (needs rebuild)
+ - Patches the vector index in place (no rebuild needed)
- Updates metadata timestamp
"""
try:
if self._data_memory.get(key) is not None:
- self._data_memory.remove(key)
- self.clear_index()
+ removed_keys, _ = self._data_memory.remove_many([key])
+ if removed_keys:
+ self._data_memory.sync_index(removed_keys=removed_keys)
self.metadata["updated_at"] = datetime.now()
except (KeyError, Exception):
pass
@@ -569,7 +576,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
>>> print(f"Removed: {skill.name}")
Side Effects:
- - Clears the vector index (needs rebuild)
+ - Patches the vector index in place (no rebuild needed)
- Updates metadata timestamp
"""
if not self._data_memory.items:
@@ -577,9 +584,10 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
item = self._data_memory.items[0]
key = self.key_extractor(item)
- self._data_memory.remove(key)
+ removed_keys, _ = self._data_memory.remove_many([key])
+ if removed_keys:
+ self._data_memory.sync_index(removed_keys=removed_keys)
- self.clear_index()
self.metadata["updated_at"] = datetime.now()
return item
diff --git a/hyperextract/types/spatial_graph.py b/hyperextract/types/spatial_graph.py
index 4c34125..b351631 100644
--- a/hyperextract/types/spatial_graph.py
+++ b/hyperextract/types/spatial_graph.py
@@ -307,6 +307,14 @@ class AutoSpatialGraph(AutoGraph[NodeSchema, EdgeSchema]):
default_factory=lambda: self.graph_schema(nodes=[], edges=[]),
)
+ # Provenance: record the raw (pre-merge) chunk results under the
+ # pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for graph in graph_list for node in graph.nodes],
+ [edge for graph in graph_list for edge in graph.edges],
+ )
+
return self.merge_batch_data(graph_list)
def _create_empty_instance(self) -> "AutoSpatialGraph[NodeSchema, EdgeSchema]":
diff --git a/hyperextract/types/spatio_temporal_graph.py b/hyperextract/types/spatio_temporal_graph.py
index 6fca415..08b09f8 100644
--- a/hyperextract/types/spatio_temporal_graph.py
+++ b/hyperextract/types/spatio_temporal_graph.py
@@ -326,6 +326,14 @@ class AutoSpatioTemporalGraph(AutoGraph[NodeSchema, EdgeSchema]):
default_factory=lambda: self.graph_schema(nodes=[], edges=[]),
)
+ # Provenance: record the raw (pre-merge) chunk results under the
+ # pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for graph in graph_list for node in graph.nodes],
+ [edge for graph in graph_list for edge in graph.edges],
+ )
+
return self.merge_batch_data(graph_list)
def _create_empty_instance(
diff --git a/hyperextract/types/temporal_graph.py b/hyperextract/types/temporal_graph.py
index b5bd97f..cce4ac0 100644
--- a/hyperextract/types/temporal_graph.py
+++ b/hyperextract/types/temporal_graph.py
@@ -318,6 +318,14 @@ class AutoTemporalGraph(AutoGraph[NodeSchema, EdgeSchema]):
default_factory=lambda: self.graph_schema(nodes=[], edges=[]),
)
+ # Provenance: record the raw (pre-merge) chunk results under the
+ # pending source id (no-op when parse/feed_text had no source_id).
+ self._record_extraction_source(
+ self._pending_source_id,
+ [node for graph in graph_list for node in graph.nodes],
+ [edge for graph in graph_list for edge in graph.edges],
+ )
+
return self.merge_batch_data(graph_list)
def _create_empty_instance(self) -> "AutoTemporalGraph[NodeSchema, EdgeSchema]":
diff --git a/pyproject.toml b/pyproject.toml
index 8dd8282..bbc4884 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -33,7 +33,7 @@ dependencies = [
"langchain-community>=0.4.1",
"langchain-openai>=1.1.7",
"structlog>=25.5.0",
- "ontomem>=0.3.0",
+ "ontomem>=0.5.0",
"ontosight>=0.1.8",
"python-dotenv>=1.2.1",
"semhash>=0.4.1",
diff --git a/tests/cli/test_remove.py b/tests/cli/test_remove.py
index dbe19b5..1eed241 100644
--- a/tests/cli/test_remove.py
+++ b/tests/cli/test_remove.py
@@ -64,9 +64,7 @@ def _real_ka():
E(name="Google"),
]
)
- g._edge_memory.add(
- [R(source="Apple", target="Google", relation_type="partner")]
- )
+ g._edge_memory.add([R(source="Apple", target="Google", relation_type="partner")])
return g
@@ -106,9 +104,7 @@ class TestHardDeleteCli:
def test_dry_run_persists_nothing(self, ka_env):
ka, _ = ka_env
- result = runner.invoke(
- app, ["remove", str(ka), "--node", "Apple", "--dry-run"]
- )
+ result = runner.invoke(app, ["remove", str(ka), "--node", "Apple", "--dry-run"])
assert result.exit_code == 0, result.output
assert [n["name"] for n in _data(ka)["nodes"]] == ["Apple", "Google"]
@@ -210,12 +206,61 @@ class TestSoftDeleteCli:
assert [n["name"] for n in _data(ka)["nodes"]] == ["Apple", "Google"]
+class TestDocumentRollbackCli:
+ def test_document_rollback_removes_only_that_document(self, tmp_path, monkeypatch):
+ g = _real_ka()
+ g.metadata["template"] = "general/graph"
+ g.metadata["lang"] = "en"
+ g._node_memory.add([E(name="Apple"), E(name="Google")])
+ g.feed_text("Apple partners with DeepMind on AI research.", source_id="doc-1")
+ ka = tmp_path / "ka"
+ g.dump(ka)
+ monkeypatch.setattr(climod.Template, "create", staticmethod(lambda *a, **k: g))
+
+ result = runner.invoke(app, ["remove", str(ka), "--document", "doc-1", "-y"])
+
+ assert result.exit_code == 0, result.output
+ names = [n["name"] for n in _data(ka)["nodes"]]
+ # Direct-added nodes stay; everything doc-1 contributed is rolled back.
+ assert "Apple" in names and "Google" in names
+ assert "mock_name" not in names
+ # Provenance ledger persisted alongside the KA.
+ assert (ka / "sources_nodes.json").exists()
+
+ def test_document_without_contributions_reports_nothing(
+ self, tmp_path, monkeypatch
+ ):
+ g = _real_ka()
+ g.metadata["template"] = "general/graph"
+ g.metadata["lang"] = "en"
+ g._node_memory.add([E(name="Apple")])
+ ka = tmp_path / "ka"
+ g.dump(ka)
+ monkeypatch.setattr(climod.Template, "create", staticmethod(lambda *a, **k: g))
+
+ result = runner.invoke(
+ app, ["remove", str(ka), "--document", "ghost-doc", "-y"]
+ )
+
+ assert result.exit_code == 0
+ assert "Nothing matched" in result.output
+
+
class TestArgumentValidation:
def test_hard_and_soft_delete_are_mutually_exclusive(self, ka_env):
ka, _ = ka_env
result = runner.invoke(
app,
- ["remove", str(ka), "--node", "Apple", "--edit-node", "Apple", "--fact", "x"],
+ [
+ "remove",
+ str(ka),
+ "--node",
+ "Apple",
+ "--edit-node",
+ "Apple",
+ "--fact",
+ "x",
+ ],
)
assert result.exit_code == 1
diff --git a/tests/types/test_graph_removal.py b/tests/types/test_graph_removal.py
index f06ae23..3654e42 100644
--- a/tests/types/test_graph_removal.py
+++ b/tests/types/test_graph_removal.py
@@ -225,7 +225,9 @@ class TestIndexPatching:
[Entity(name="A", description="A was founded by X. A is in Y.")]
)
g.build_index()
- g._node_memory.llm_client = FakeChatModel(Entity(name="A", description="A is in Y."))
+ g._node_memory.llm_client = FakeChatModel(
+ Entity(name="A", description="A is in Y.")
+ )
report = g.edit_node("A", remove_fact="founded by X")
@@ -301,7 +303,9 @@ class TestSoftDelete:
g._node_memory.add(
[Entity(name="A", description="A was founded by X. A is in Y.")]
)
- g._node_memory.llm_client = FakeChatModel(Entity(name="A", description="A is in Y."))
+ g._node_memory.llm_client = FakeChatModel(
+ Entity(name="A", description="A is in Y.")
+ )
report = g.edit_node("A", remove_fact="founded by X", dry_run=True)
@@ -315,7 +319,9 @@ class TestSoftDelete:
g._node_memory.add(
[Entity(name="A", description="A was founded by X. A is in Y.")]
)
- g._node_memory.llm_client = FakeChatModel(Entity(name="A", description="A is in Y."))
+ g._node_memory.llm_client = FakeChatModel(
+ Entity(name="A", description="A is in Y.")
+ )
report = g.edit_node("A", remove_fact="founded by X")
@@ -341,7 +347,9 @@ class TestSoftDelete:
def test_key_change_is_rejected(self):
g = _graph()
g._node_memory.add([Entity(name="A")])
- g._node_memory.llm_client = FakeChatModel(Entity(name="Renamed")) # identity break
+ g._node_memory.llm_client = FakeChatModel(
+ Entity(name="Renamed")
+ ) # identity break
with pytest.raises(ValueError, match="key changed"):
g.edit_node("A", remove_fact="whatever")
diff --git a/uv.lock b/uv.lock
index ba0d483..35ef944 100644
--- a/uv.lock
+++ b/uv.lock
@@ -957,7 +957,7 @@ requires-dist = [
{ name = "langchain-google-genai", marker = "extra == 'google'", specifier = ">=2.1.0" },
{ name = "langchain-openai", specifier = ">=1.1.7" },
{ name = "mcp", marker = "extra == 'mcp'", specifier = ">=1.2.0,<3" },
- { name = "ontomem", specifier = ">=0.3.0" },
+ { name = "ontomem", specifier = ">=0.5.0" },
{ name = "ontosight", specifier = ">=0.1.8" },
{ name = "python-dotenv", specifier = ">=1.2.1" },
{ name = "rich", specifier = ">=13.7.0" },
@@ -1879,7 +1879,7 @@ wheels = [
[[package]]
name = "ontomem"
-version = "0.3.0"
+version = "0.5.0"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
dependencies = [
{ name = "faiss-cpu" },
@@ -1891,9 +1891,9 @@ dependencies = [
{ name = "python-dotenv" },
{ name = "structlog" },
]
-sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/99/b4/a76747d5dda4bcd0e107ee59aae4b89a952a35761a82408318eebf029b5b/ontomem-0.3.0.tar.gz", hash = "sha256:725405a54ea03380c7fa6a8c9a1badb561a23f971efbad5914f2e934bd9f511e", size = 2540504, upload-time = "2026-09-04T02:13:44.824Z" }
+sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/1e/5e1e484d8d1ee7caafebf59273a5df87d9bfbc1d5f75ff49a702e8e69372/ontomem-0.5.0.tar.gz", hash = "sha256:25dd2c9676e40e1e11fc519a5b3547c79e263bcb64c17505f0e26560e10f10ee", size = 2549972, upload-time = "2026-09-04T03:55:53.486Z" }
wheels = [
- { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/5e/0930f49541d5fa61f6420b08fd2f3d10af98811a7630e815508af59be8a0/ontomem-0.3.0-py3-none-any.whl", hash = "sha256:a03520bca3db0ff87db992c4aeff5c1df121ac3a25f1111519538e7148d8b1e5", size = 41883, upload-time = "2026-09-04T02:13:42.991Z" },
+ { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cc/42/adf7dcfc94e9e255fc52a7d38713746ad2fc83f374d157c99d7ca4ecbf82/ontomem-0.5.0-py3-none-any.whl", hash = "sha256:04b2d058cb7b0617d3f3d46fdc469282ed77cee42ccba133041807b775c42b9c", size = 47040, upload-time = "2026-09-04T03:55:52.236Z" },
]
[[package]]