Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: per-document rollback — source threading, he remove --document, feed-path index preservation
yifanfeng971170ddfa30c9
diff --git a/docs/en/cli/commands/feed.md b/docs/en/cli/commands/feed.mdindex 8bdc72e..93caf2f 100644--- a/docs/en/cli/commands/feed.md+++ b/docs/en/cli/commands/feed.md@@ -23,6 +23,7 @@ he feed KA_PATH INPUT [OPTIONS]|--------|-------|-------------|| `--template` | `-t` | Override template (uses metadata if omitted) || `--lang` | `-l` | Override language (uses metadata if omitted) |+| `--source` | — | Source attribution (document id) — enables per-document rollback via `he remove --document` |---diff --git a/docs/en/cli/commands/parse.md b/docs/en/cli/commands/parse.mdindex 9f171c5..1cb19ba 100644--- a/docs/en/cli/commands/parse.md+++ b/docs/en/cli/commands/parse.md@@ -24,6 +24,7 @@ he parse INPUT [OPTIONS]| `--template` | `-t` | Template to use (omit for interactive selection) || `--method` | `-m` | Method template (e.g., `light_rag`, `graph_rag`) || `--lang` | `-l` | Language: `zh` or `en` (required for knowledge templates) |+| `--source` | — | Source attribution (document id) — enables per-document rollback via `he remove --document` || `--force` | `-f` | Force overwrite existing output || `--no-index` | — | Skip building search index |diff --git a/docs/en/cli/commands/remove.md b/docs/en/cli/commands/remove.mdindex 7d12e40..74c78de 100644--- a/docs/en/cli/commands/remove.md+++ b/docs/en/cli/commands/remove.md@@ -18,6 +18,7 @@ he remove KA_PATH [OPTIONS]| `--edge TEXT` | — | Edge key(s) to **hard-delete** || `--edit-node TEXT` | — | Node key to **soft-edit** (LLM rewrites it minus `--fact`) || `--edit-edge TEXT` | — | Edge key to **soft-edit** |+| `--document TEXT` | — | **Roll back** all knowledge contributed by a source document (requires the KA to have been fed with `--source`) || `--fact TEXT` | — | The fact to remove from the `--edit-node` / `--edit-edge` item || `--instruction TEXT` | — | Free-form edit instruction (alternative to `--fact`) || `--dry-run` | off | Preview the change without persisting it |@@ -89,6 +90,23 @@ he build-index ./ka/---+## Document Rollback: remove a whole source document++When documents were ingested with a source attribution, their entire contribution can be rolled back — including keys that other documents also contributed to (those are re-merged from the surviving sources).++```bash+# Ingest with attribution+he feed ./ka/ doc1.md --source doc-1+he feed ./ka/ doc2.md --source doc-2++# Later: roll back everything doc-1 contributed ("b" remains via doc-2 if shared)+he remove ./ka/ --document doc-1+```++Requires the KA's memories to track sources (graph-family KAs do by default). The rollback re-merges affected keys from the surviving sources' raw results — deterministic with classic merge strategies; LLM strategies preserve semantics with approximate wording. If the document has no recorded contributions, the command reports `Nothing matched` and exits successfully.++---+## Python API```pythondiff --git a/docs/zh/cli/commands/feed.md b/docs/zh/cli/commands/feed.mdindex b8cbed8..276ec41 100644--- a/docs/zh/cli/commands/feed.md+++ b/docs/zh/cli/commands/feed.md@@ -23,6 +23,7 @@ he feed KA_PATH INPUT [OPTIONS]|--------|-------|-------------|| `--template` | `-t` | 覆盖模板(如果省略则使用元数据) || `--lang` | `-l` | 覆盖语言(如果省略则使用元数据) |+| `--source` | — | 来源标注(文档 id)——之后可通过 `he remove --document` 按文档回滚 |---diff --git a/docs/zh/cli/commands/parse.md b/docs/zh/cli/commands/parse.mdindex 15c36e8..42d20e0 100644--- a/docs/zh/cli/commands/parse.md+++ b/docs/zh/cli/commands/parse.md@@ -24,6 +24,7 @@ he parse INPUT [OPTIONS]| `--template` | `-t` | 要使用的模板(省略以进行交互式选择) || `--method` | `-m` | 方法模板(例如 `light_rag`、`graph_rag`) || `--lang` | `-l` | 语言:`zh` 或 `en`(知识模板必填) |+| `--source` | — | 来源标注(文档 id)——之后可通过 `he remove --document` 按文档回滚 || `--force` | `-f` | 强制覆盖现有输出 || `--no-index` | — | 跳过构建搜索索引 |diff --git a/docs/zh/cli/commands/remove.md b/docs/zh/cli/commands/remove.mdindex d8011d2..19e63fe 100644--- a/docs/zh/cli/commands/remove.md+++ b/docs/zh/cli/commands/remove.md@@ -18,6 +18,7 @@ he remove KA_PATH [选项]| `--edge TEXT` | — | **硬删除**指定边 || `--edit-node TEXT` | — | **软编辑**指定节点(大模型重写,移除 `--fact`) || `--edit-edge TEXT` | — | **软编辑**指定边 |+| `--document TEXT` | — | **回滚**某个来源文档贡献的全部知识(要求 KA 在摄取时使用了 `--source`) || `--fact TEXT` | — | 要从 `--edit-node` / `--edit-edge` 条目中移除的事实 || `--instruction TEXT` | — | 自由格式的编辑指令(替代 `--fact`) || `--dry-run` | 关闭 | 仅预览变更,不落盘 |@@ -89,6 +90,23 @@ he build-index ./ka/---+## 文档级回滚:删除整个来源文档的贡献++当文档以来源标注方式摄取后,可以整体回滚其贡献——包括与其他文档共享的 key(这些 key 会从幸存来源的原始结果重新合并)。++```bash+# 带来源标注摄取+he feed ./ka/ doc1.md --source doc-1+he feed ./ka/ doc2.md --source doc-2++# 之后:回滚 doc-1 贡献的全部知识+he remove ./ka/ --document doc-1+```++要求 KA 的记忆开启来源追踪(图谱类 KA 默认开启)。回滚会从幸存来源的原始结果重新合并受影响的 key——经典合并策略下完全确定;LLM 策略保留语义、措辞近似。若该文档没有已记录的贡献,命令会报告 `Nothing matched` 并正常退出。++---+## Python API```pythondiff --git a/hyperextract/cli/cli.py b/hyperextract/cli/cli.pyindex db023aa..5425010 100644--- a/hyperextract/cli/cli.py+++ b/hyperextract/cli/cli.py@@ -247,6 +247,11 @@ def parse(no_index: bool = typer.Option(False, "--no-index", help="Skip building search index"),+ source: str | None = typer.Option(+ None,+ "--source",+ help="Source attribution (document id) for per-document rollback later (he remove --document)",+ ),):"""Extract knowledge from text to a new directory."""logger.info(@@ -341,7 +346,7 @@ def parse(progress.update(task, description="Extracting knowledge...")logger.debug("stage=feed_text_invoked")- ka.feed_text(combined_text)+ ka.feed_text(combined_text, source_id=source)logger.info("stage=knowledge_extracted chars=%d", len(combined_text))else:progress.update(task, description="Reading input...")@@ -350,7 +355,7 @@ def parse(progress.update(task, description="Extracting knowledge...")logger.debug("stage=feed_text_invoked")- ka.feed_text(text)+ ka.feed_text(text, source_id=source)logger.info("stage=knowledge_extracted chars=%d", len(text))progress.update(task, description="Saving data...")@@ -930,6 +935,11 @@ def feed(input: str = typer.Argument(..., help="Input file path or '-' for stdin"),template: str | None = typer.Option(None, "--template", "-t", help="Template"),lang: str | None = typer.Option(None, "--lang", "-l", help="Language"),+ source: str | None = typer.Option(+ None,+ "--source",+ help="Source attribution (document id) for per-document rollback later (he remove --document)",+ ),):"""Append knowledge to an existing Knowledge Abstract."""logger.info("command=feed ka_path=%s input=%s", ka_path, input)@@ -977,7 +987,7 @@ def feed(progress.update(task, description="Appending knowledge...")logger.debug("stage=feed_text_invoked")- ka.feed_text(text)+ ka.feed_text(text, source_id=source)logger.info("stage=knowledge_appended chars=%d", len(text))progress.update(task, description="Saving data...")@@ -1135,33 +1145,51 @@ def remove_items(True, "--backup/--no-backup", help="Back up data.json before writing"),yes: bool = typer.Option(False, "--yes", "-y", help="Skip the confirmation prompt"),+ document: str | None = typer.Option(+ None,+ "--document",+ help="Remove all knowledge contributed by this source document "+ "(requires the KA to have been fed with --source)",+ ),):- """Delete nodes/edges by key, or soft-remove a single fact (LLM-assisted).+ """Delete nodes/edges by key, remove a fact (LLM-assisted), or roll back a whole document.- Hard delete: he remove ./ka --node Apple --edge "Apple-partner-Google"- Soft delete: he remove ./ka --edit-node Apple --fact "founded by Steve Jobs"+ Hard delete: he remove ./ka --node Apple --edge "Apple-partner-Google"+ Soft delete: he remove ./ka --edit-node Apple --fact "founded by Steve Jobs"+ Document rollback: he remove ./ka --document ./docs/old-paper.md"""import shutilfrom datetime import datetime- logger.info("command=remove ka_path=%s dry_run=%s", ka_path, dry_run)+ logger.info(+ "command=remove ka_path=%s dry_run=%s document=%s",+ ka_path,+ dry_run,+ document,+ )hard_targets = list(node or []) + list(edge or [])soft_target = edit_node_key or edit_edge_key- if hard_targets and soft_target:+ if (hard_targets or soft_target or document) and sum(+ bool(x) for x in (hard_targets, soft_target, document)+ ) > 1:console.print(- "[red]Error:[/red] Combine either --node/--edge (hard delete) "- "or --edit-node/--edit-edge (soft delete), not both."+ "[red]Error:[/red] Choose ONE of: --node/--edge (hard delete), "+ "--edit-node/--edit-edge (soft delete), or --document "+ "(whole-document rollback).")raise typer.Exit(1)if soft_target and not fact and not instruction:console.print("[red]Error:[/red] Soft delete needs --fact or --instruction.")raise typer.Exit(1)- if not hard_targets and not soft_target:+ if document and not (Path(document).name or document):+ pass # unreachable; keeps option semantics explicit+ if not hard_targets and not soft_target and not document:console.print("[yellow]Nothing to do.[/yellow] "- "Use --node/--edge to delete items, or --edit-node/--edit-edge "- "with --fact to remove a single fact."+ "Use --node/--edge to delete items, --edit-node/--edit-edge "+ "with --fact to remove a single fact, or --document to roll back "+ "a whole source document.")raise typer.Exit(0)@@ -1181,7 +1209,17 @@ def remove_items(raise typer.Exit(1)try:- if hard_targets:+ if document:+ try:+ report = ka.remove_source(document, strategy="exact")+ except KeyError:+ console.print(+ f"[yellow]Nothing matched — no recorded contributions "+ f"for source '{document}'. Was the KA fed with "+ f"--source?[/yellow]"+ )+ raise typer.Exit(0)+ elif hard_targets:report = {}if node:report.update(ka.remove_nodes(*node))@@ -1218,7 +1256,31 @@ def remove_items(raise typer.Exit(1)console.print()- if hard_targets:+ if document:+ table = Table(title=f"Document Rollback Report — {document}")+ table.add_column("Field")+ table.add_column("Items")+ for field, items in report.items():+ if field in ("index_patched", "source_id", "strategy"):+ continue+ table.add_row(field, "\n".join(items) if items else "—")+ console.print(table)+ changed_any = any(+ report.get(key)+ for key in (+ "removed_nodes",+ "remerged_nodes",+ "removed_edges",+ "remerged_edges",+ )+ )+ if not changed_any:+ console.print(+ "[yellow]Nothing matched — this document has no recorded "+ "contributions. Was the KA fed with --source?[/yellow]"+ )+ raise typer.Exit(0)+ elif hard_targets:table = Table(title="Removal Report")table.add_column("Field")table.add_column("Items")diff --git a/hyperextract/types/base.py b/hyperextract/types/base.pyindex 52711fb..8f54ce1 100644--- a/hyperextract/types/base.py+++ b/hyperextract/types/base.py@@ -97,6 +97,9 @@ class BaseAutoType(ABC, Generic[T]):"created_at": datetime.now(),"updated_at": datetime.now(),}+ # Set by parse()/feed_text() while extraction runs; graph-family+ # subclasses record raw extraction results under it (provenance).+ self._pending_source_id: str | None = Nonedef _create_empty_instance(self) -> "BaseAutoType[T]":"""Creates a new empty instance with the same configuration as this one.@@ -242,7 +245,12 @@ class BaseAutoType(ABC, Generic[T]):# ==================== Extraction & Merge ====================def _extract_data(self, text: str) -> T:- """Internal: Unified extraction logic (Chunking -> LLM -> Merge)."""+ """Internal: Unified extraction logic (Chunking -> LLM -> Merge).++ Subclasses that support source attribution read+ ``self._pending_source_id`` (set by parse/feed_text) and record+ their raw, pre-merge results via ``_record_extraction_source``.+ """logger.debug("stage=extract_start input_chars=%d chunk_size=%d",len(text),@@ -377,6 +385,20 @@ class BaseAutoType(ABC, Generic[T]):results.append(r)return results+ def _adopt_source_ledger(self, other: "BaseAutoType[T]", source_id: str) -> None:+ """Hook: transfer ledger entries for ``source_id`` from ``other`` to+ ``self`` after parse(). No-op by default; graph-family memories with+ ``track_sources`` override this."""+ return++ def _dump_provenance(self, root: Path) -> None:+ """Hook: persist the source ledger alongside the KA. No-op by default."""+ return++ def _load_provenance(self, root: Path) -> None:+ """Hook: restore the source ledger alongside the KA. No-op by default."""+ return+def _summarize_extracted(self, data: T) -> str:"""Return a concise summary of extracted data for debug logging."""try:@@ -397,7 +419,7 @@ class BaseAutoType(ABC, Generic[T]):except Exception:return repr(data)[:100]- def parse(self, text: str) -> "BaseAutoType[T]":+ def parse(self, text: str, *, source_id: str | None = None) -> "BaseAutoType[T]":"""Parses knowledge into a NEW instance without modifying the current one.@@ -405,21 +427,33 @@ class BaseAutoType(ABC, Generic[T]):Args:text: Input text.+ source_id: Optional source attribution (graph-family memories with+ ``track_sources``): the raw extraction results are recorded+ under this id so the document's contributions can later be+ rolled back via ``remove_source``.Returns:A new knowledge instance containing only the parsed data."""- parsed_data = self._extract_data(text)+ self._pending_source_id = source_id+ try:+ parsed_data = self._extract_data(text)+ finally:+ self._pending_source_id = Nonenew_instance = self._create_empty_instance()new_instance._set_data_state(parsed_data)+ if source_id:+ new_instance._adopt_source_ledger(self, source_id)new_instance.metadata["created_at"] = datetime.now()new_instance.metadata["updated_at"] = datetime.now()return new_instance- def feed_text(self, text: str) -> "BaseAutoType[T]":+ def feed_text(+ self, text: str, *, source_id: str | None = None+ ) -> "BaseAutoType[T]":"""Ingests text into the CURRENT knowledge abstract instance.@@ -428,17 +462,25 @@ class BaseAutoType(ABC, Generic[T]):Args:text: Input text.+ source_id: Optional source attribution (graph-family memories with+ ``track_sources``): the raw extraction results are recorded+ under this id so the document's contributions can later be+ rolled back via ``remove_source``.Returns:Self (the current instance)."""logger.debug("stage=feed_text_start input_chars=%d", len(text))- extracted_data = self._extract_data(text)- logger.debug("stage=extract_done")+ self._pending_source_id = source_id+ try:+ extracted_data = self._extract_data(text)+ logger.debug("stage=extract_done")- # Use UPDATE hook instead of manual merge+set- self._update_data_state(extracted_data)- logger.debug("stage=data_merged")+ # Use UPDATE hook instead of manual merge+set+ self._update_data_state(extracted_data)+ logger.debug("stage=data_merged")+ finally:+ self._pending_source_id = Noneself.metadata["updated_at"] = datetime.now()@@ -587,6 +629,9 @@ class BaseAutoType(ABC, Generic[T]):# user can always rebuild_index()print(f"Warning: Failed to save vector index: {e}")+ # 4. Provenance (Optional — graph-family with track_sources)+ self._dump_provenance(root)+def load(self, folder_path: str | Path) -> None:"""Loads the entire knowledge abstract from a directory.@@ -615,6 +660,9 @@ class BaseAutoType(ABC, Generic[T]):f"Warning: Failed to load vector index: {e}. You may need to rebuild_index().")+ # 4. Provenance (Optional — graph-family with track_sources)+ self._load_provenance(root)+# ==================== Serialization: Components ====================def dump_data(self, file_path: str | Path) -> None:diff --git a/hyperextract/types/graph.py b/hyperextract/types/graph.pyindex eed515c..d622d43 100644--- a/hyperextract/types/graph.py+++ b/hyperextract/types/graph.py@@ -83,6 +83,86 @@ class GraphEditMixin:OMem's `sync_index` / `edit` (ontomem >= 0.3.0)."""+ # ==================== Source Provenance (v0.4.0+) ====================++ def _record_extraction_source(+ self,+ source_id: str | None,+ nodes: list[NodeSchema],+ edges: list[EdgeSchema],+ ) -> None:+ """Record raw (pre-merge) extraction results in the source ledgers.++ Reads ``self._pending_source_id`` (set by parse/feed_text); no-op when+ no source was attributed or the memories don't track sources.+ """+ if not source_id or not self._node_memory.track_sources:+ return+ self._node_memory.record_source(+ source_id, [node.model_dump() for node in nodes]+ )+ self._edge_memory.record_source(+ source_id, [edge.model_dump() for edge in edges]+ )++ def _adopt_source_ledger(self, other, source_id: str) -> None:+ """Transfer ledger entries for ``source_id`` from ``other`` to ``self``+ after parse() (the extraction ran on ``other``'s memories)."""+ for src_memory, dst_memory in (+ (other._node_memory, self._node_memory),+ (other._edge_memory, self._edge_memory),+ ):+ if source_id in src_memory._sources:+ dst_memory._sources[source_id] = src_memory._sources[source_id]++ def _dump_provenance(self, root: Path) -> None:+ """Persist the node/edge source ledgers alongside the KA."""+ try:+ self._node_memory.dump_sources(Path(root) / "sources_nodes.json")+ self._edge_memory.dump_sources(Path(root) / "sources_edges.json")+ except Exception as e:+ logger.warning("provenance_dump_failed", error=str(e))++ def _load_provenance(self, root: Path) -> None:+ """Restore the node/edge source ledgers alongside the KA."""+ try:+ node_path = Path(root) / "sources_nodes.json"+ edge_path = Path(root) / "sources_edges.json"+ if node_path.exists():+ self._node_memory.load_sources(node_path)+ if edge_path.exists():+ self._edge_memory.load_sources(edge_path)+ except Exception as e:+ logger.warning("provenance_load_failed", error=str(e))++ def remove_source(self, source_id: str, *, strategy: str = "exact") -> dict:+ """Remove every contribution of one source document (exact rollback).++ Keys contributed solely by the removed document are deleted; keys+ shared with other documents are re-merged from the surviving sources'+ raw results. The search index is patched in place when built.++ Args:+ source_id: Identifier used when the document was fed+ (``feed_text(text, source_id=...)`` / ``he feed --source``).+ strategy: "exact" (re-merge survivors) or "touched" (delete all+ affected keys).++ Returns:+ Combined node/edge report dict.+ """+ node_report = self._node_memory.remove_source(source_id, strategy=strategy)+ edge_report = self._edge_memory.remove_source(source_id, strategy=strategy)+ return {+ "source_id": source_id,+ "removed_nodes": node_report["removed_keys"],+ "remerged_nodes": node_report["remerged_keys"],+ "removed_edges": edge_report["removed_keys"],+ "remerged_edges": edge_report["remerged_keys"],+ "index_patched": node_report["index_patched"]+ and edge_report["index_patched"],+ }+# ==================== Removal ====================def remove_nodes(self, *keys: str) -> dict:@@ -270,8 +350,8 @@ class EdgeListSchema(BaseModel, Generic[EdgeSchema]):class AutoGraph(- BaseAutoType[AutoGraphSchema[NodeSchema, EdgeSchema]],GraphEditMixin,+ BaseAutoType[AutoGraphSchema[NodeSchema, EdgeSchema]],Generic[NodeSchema, EdgeSchema],):"""AutoGraph - extracts knowledge graphs with nodes and edges from text.@@ -446,6 +526,7 @@ class AutoGraph(strategy_or_merger=self.node_merger,verbose=verbose,fields_for_index=node_fields_for_index, # Pass node field selection to OMem+ track_sources=True, # source ledger for per-document rollback (#84))self._edge_memory = OMem(@@ -456,6 +537,7 @@ class AutoGraph(strategy_or_merger=self.edge_merger,verbose=verbose,fields_for_index=edge_fields_for_index, # Pass edge field selection to OMem+ track_sources=True, # source ledger for per-document rollback (#84))# Store label extractors for visualization@@ -602,17 +684,30 @@ class AutoGraph() -> None:"""Merge incoming graph data into current state.+ The vector index is patched in place: only the incoming nodes/edges+ are re-embedded, instead of dropping the whole index.+Args:incoming_data: Incremental graph data to merge."""if self.empty():self._set_data_state(incoming_data)- else:+ return++ # Suspend index invalidation while merging; re-embed only the+ # affected keys afterwards.+ with self._node_memory.suspended_index(), self._edge_memory.suspended_index():if incoming_data.nodes:self._node_memory.add(incoming_data.nodes)if incoming_data.edges:self._edge_memory.add(incoming_data.edges)- self.clear_index()++ if incoming_data.nodes:+ node_keys = {self.node_key_extractor(n) for n in incoming_data.nodes}+ self._node_memory.sync_index(upserted_keys=node_keys)+ if incoming_data.edges:+ edge_keys = {self.edge_key_extractor(e) for e in incoming_data.edges}+ self._edge_memory.sync_index(upserted_keys=edge_keys)# ==================== Extraction Pipeline ====================@@ -674,6 +769,13 @@ class AutoGraph(logger.debug("stage=one_stage_batch_complete graphs=%d", len(graph_list))logger.debug("stage=one_stage_merge_start")+ # Provenance: record the raw (pre-merge) chunk results under the+ # pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for graph in graph_list for node in graph.nodes],+ [edge for graph in graph_list for edge in graph.edges],+ )result = self.merge_batch_data(graph_list)logger.debug("stage=one_stage_merge_complete nodes=%d edges=%d",@@ -751,6 +853,14 @@ class AutoGraph([edge_list.items for edge_list in chunk_edge_lists],)+ # Provenance: record the raw (pre-merge) node/edge results under the+ # pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for node_list in chunk_node_lists for node in node_list.items],+ [edge for edge_list in chunk_edge_lists for edge in edge_list.items],+ )+# 5. Global Merge (passes tuples to merge_batch_data)logger.debug("stage=two_stage_merge_start")result = self.merge_batch_data(partial_graphs)diff --git a/hyperextract/types/hypergraph.py b/hyperextract/types/hypergraph.pyindex 9f08305..54f616d 100644--- a/hyperextract/types/hypergraph.py+++ b/hyperextract/types/hypergraph.py@@ -107,8 +107,8 @@ class EdgeListSchema(BaseModel, Generic[EdgeSchema]):class AutoHypergraph(- BaseAutoType[AutoHypergraphSchema[NodeSchema, EdgeSchema]],GraphEditMixin,+ BaseAutoType[AutoHypergraphSchema[NodeSchema, EdgeSchema]],Generic[NodeSchema, EdgeSchema],):"""AutoHypergraph - Node-First extraction: Extract entities first, then relationships.@@ -286,6 +286,7 @@ class AutoHypergraph(strategy_or_merger=self.node_merger,verbose=verbose,fields_for_index=node_fields_for_index, # Pass node field selection to OMem+ track_sources=True, # source ledger for per-document rollback (#84))self._edge_memory = OMem(@@ -296,6 +297,7 @@ class AutoHypergraph(strategy_or_merger=self.edge_merger,verbose=verbose,fields_for_index=edge_fields_for_index, # Pass edge field selection to OMem+ track_sources=True, # source ledger for per-document rollback (#84))# Store label extractors for visualization@@ -475,6 +477,14 @@ class AutoHypergraph(default_factory=lambda: self.graph_schema(nodes=[], edges=[]),)+ # Provenance: record the raw (pre-merge) chunk results under the+ # pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for graph in graph_list for node in graph.nodes],+ [edge for graph in graph_list for edge in graph.edges],+ )+# Merge multiple hypergraphsreturn self.merge_batch_data(graph_list)@@ -519,6 +529,13 @@ class AutoHypergraph(]# 4. Construct Partial Graphs (Tuple format for merge optimization)+ # Provenance: record the raw (pre-merge) node/hyperedge results under+ # the pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for node_list in chunk_node_lists for node in node_list.items],+ [edge for edge_list in chunk_edge_lists for edge in edge_list.items],+ )partial_hypergraphs = ([node_list.items for node_list in chunk_node_lists],[edge_list.items for edge_list in chunk_edge_lists],diff --git a/hyperextract/types/set.py b/hyperextract/types/set.pyindex 744388f..959a6ca 100644--- a/hyperextract/types/set.py+++ b/hyperextract/types/set.py@@ -469,9 +469,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):Args:item: The item to add."""- # OMem handles deduplication and merging automatically- self._data_memory.add([item])- self.clear_index()+ # OMem handles deduplication and merging automatically; the vector+ # index is patched in place (only the added key is re-embedded).+ with self._data_memory.suspended_index():+ self._data_memory.add([item])+ self._data_memory.sync_index(upserted_keys=[self.key_extractor(item)])self.metadata["updated_at"] = datetime.now()def remove(self, key: Any) -> ItemSchema | None:@@ -488,10 +490,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):if item is None:return None- # Remove from OMem- removed = self._data_memory.remove(key)- if removed:- self.clear_index()+ # Remove from OMem; sync_index patches the vector index in place+ # (only the removed key's vector is dropped — no full rebuild).+ removed_keys, _ = self._data_memory.remove_many([key])+ if removed_keys:+ self._data_memory.sync_index(removed_keys=removed_keys)self.metadata["updated_at"] = datetime.now()logger.debug(f"Removed item with key '{key}'")return item@@ -528,8 +531,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):"""# add() takes a single item and would wrap the whole list as one entry;# pass the list straight to the backing store so each item is added.- self._data_memory.add(items)- self.clear_index()+ # The vector index is patched in place afterwards (no full rebuild).+ with self._data_memory.suspended_index():+ self._data_memory.add(items)+ added_keys = {self.key_extractor(item) for item in items}+ self._data_memory.sync_index(upserted_keys=added_keys)self.metadata["updated_at"] = datetime.now()def discard(self, key: Any) -> None:@@ -544,13 +550,14 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):>>> skills.discard("Python") # No error if not foundSide Effects:- - Clears the vector index (needs rebuild)+ - Patches the vector index in place (no rebuild needed)- Updates metadata timestamp"""try:if self._data_memory.get(key) is not None:- self._data_memory.remove(key)- self.clear_index()+ removed_keys, _ = self._data_memory.remove_many([key])+ if removed_keys:+ self._data_memory.sync_index(removed_keys=removed_keys)self.metadata["updated_at"] = datetime.now()except (KeyError, Exception):pass@@ -569,7 +576,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):>>> print(f"Removed: {skill.name}")Side Effects:- - Clears the vector index (needs rebuild)+ - Patches the vector index in place (no rebuild needed)- Updates metadata timestamp"""if not self._data_memory.items:@@ -577,9 +584,10 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):item = self._data_memory.items[0]key = self.key_extractor(item)- self._data_memory.remove(key)+ removed_keys, _ = self._data_memory.remove_many([key])+ if removed_keys:+ self._data_memory.sync_index(removed_keys=removed_keys)- self.clear_index()self.metadata["updated_at"] = datetime.now()return itemdiff --git a/hyperextract/types/spatial_graph.py b/hyperextract/types/spatial_graph.pyindex 4c34125..b351631 100644--- a/hyperextract/types/spatial_graph.py+++ b/hyperextract/types/spatial_graph.py@@ -307,6 +307,14 @@ class AutoSpatialGraph(AutoGraph[NodeSchema, EdgeSchema]):default_factory=lambda: self.graph_schema(nodes=[], edges=[]),)+ # Provenance: record the raw (pre-merge) chunk results under the+ # pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for graph in graph_list for node in graph.nodes],+ [edge for graph in graph_list for edge in graph.edges],+ )+return self.merge_batch_data(graph_list)def _create_empty_instance(self) -> "AutoSpatialGraph[NodeSchema, EdgeSchema]":diff --git a/hyperextract/types/spatio_temporal_graph.py b/hyperextract/types/spatio_temporal_graph.pyindex 6fca415..08b09f8 100644--- a/hyperextract/types/spatio_temporal_graph.py+++ b/hyperextract/types/spatio_temporal_graph.py@@ -326,6 +326,14 @@ class AutoSpatioTemporalGraph(AutoGraph[NodeSchema, EdgeSchema]):default_factory=lambda: self.graph_schema(nodes=[], edges=[]),)+ # Provenance: record the raw (pre-merge) chunk results under the+ # pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for graph in graph_list for node in graph.nodes],+ [edge for graph in graph_list for edge in graph.edges],+ )+return self.merge_batch_data(graph_list)def _create_empty_instance(diff --git a/hyperextract/types/temporal_graph.py b/hyperextract/types/temporal_graph.pyindex b5bd97f..cce4ac0 100644--- a/hyperextract/types/temporal_graph.py+++ b/hyperextract/types/temporal_graph.py@@ -318,6 +318,14 @@ class AutoTemporalGraph(AutoGraph[NodeSchema, EdgeSchema]):default_factory=lambda: self.graph_schema(nodes=[], edges=[]),)+ # Provenance: record the raw (pre-merge) chunk results under the+ # pending source id (no-op when parse/feed_text had no source_id).+ self._record_extraction_source(+ self._pending_source_id,+ [node for graph in graph_list for node in graph.nodes],+ [edge for graph in graph_list for edge in graph.edges],+ )+return self.merge_batch_data(graph_list)def _create_empty_instance(self) -> "AutoTemporalGraph[NodeSchema, EdgeSchema]":diff --git a/pyproject.toml b/pyproject.tomlindex 8dd8282..bbc4884 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -33,7 +33,7 @@ dependencies = ["langchain-community>=0.4.1","langchain-openai>=1.1.7","structlog>=25.5.0",- "ontomem>=0.3.0",+ "ontomem>=0.5.0","ontosight>=0.1.8","python-dotenv>=1.2.1","semhash>=0.4.1",diff --git a/tests/cli/test_remove.py b/tests/cli/test_remove.pyindex dbe19b5..1eed241 100644--- a/tests/cli/test_remove.py+++ b/tests/cli/test_remove.py@@ -64,9 +64,7 @@ def _real_ka():E(name="Google"),])- g._edge_memory.add(- [R(source="Apple", target="Google", relation_type="partner")]- )+ g._edge_memory.add([R(source="Apple", target="Google", relation_type="partner")])return g@@ -106,9 +104,7 @@ class TestHardDeleteCli:def test_dry_run_persists_nothing(self, ka_env):ka, _ = ka_env- result = runner.invoke(- app, ["remove", str(ka), "--node", "Apple", "--dry-run"]- )+ result = runner.invoke(app, ["remove", str(ka), "--node", "Apple", "--dry-run"])assert result.exit_code == 0, result.outputassert [n["name"] for n in _data(ka)["nodes"]] == ["Apple", "Google"]@@ -210,12 +206,61 @@ class TestSoftDeleteCli:assert [n["name"] for n in _data(ka)["nodes"]] == ["Apple", "Google"]+class TestDocumentRollbackCli:+ def test_document_rollback_removes_only_that_document(self, tmp_path, monkeypatch):+ g = _real_ka()+ g.metadata["template"] = "general/graph"+ g.metadata["lang"] = "en"+ g._node_memory.add([E(name="Apple"), E(name="Google")])+ g.feed_text("Apple partners with DeepMind on AI research.", source_id="doc-1")+ ka = tmp_path / "ka"+ g.dump(ka)+ monkeypatch.setattr(climod.Template, "create", staticmethod(lambda *a, **k: g))++ result = runner.invoke(app, ["remove", str(ka), "--document", "doc-1", "-y"])++ assert result.exit_code == 0, result.output+ names = [n["name"] for n in _data(ka)["nodes"]]+ # Direct-added nodes stay; everything doc-1 contributed is rolled back.+ assert "Apple" in names and "Google" in names+ assert "mock_name" not in names+ # Provenance ledger persisted alongside the KA.+ assert (ka / "sources_nodes.json").exists()++ def test_document_without_contributions_reports_nothing(+ self, tmp_path, monkeypatch+ ):+ g = _real_ka()+ g.metadata["template"] = "general/graph"+ g.metadata["lang"] = "en"+ g._node_memory.add([E(name="Apple")])+ ka = tmp_path / "ka"+ g.dump(ka)+ monkeypatch.setattr(climod.Template, "create", staticmethod(lambda *a, **k: g))++ result = runner.invoke(+ app, ["remove", str(ka), "--document", "ghost-doc", "-y"]+ )++ assert result.exit_code == 0+ assert "Nothing matched" in result.output++class TestArgumentValidation:def test_hard_and_soft_delete_are_mutually_exclusive(self, ka_env):ka, _ = ka_envresult = runner.invoke(app,- ["remove", str(ka), "--node", "Apple", "--edit-node", "Apple", "--fact", "x"],+ [+ "remove",+ str(ka),+ "--node",+ "Apple",+ "--edit-node",+ "Apple",+ "--fact",+ "x",+ ],)assert result.exit_code == 1diff --git a/tests/types/test_graph_removal.py b/tests/types/test_graph_removal.pyindex f06ae23..3654e42 100644--- a/tests/types/test_graph_removal.py+++ b/tests/types/test_graph_removal.py@@ -225,7 +225,9 @@ class TestIndexPatching:[Entity(name="A", description="A was founded by X. A is in Y.")])g.build_index()- g._node_memory.llm_client = FakeChatModel(Entity(name="A", description="A is in Y."))+ g._node_memory.llm_client = FakeChatModel(+ Entity(name="A", description="A is in Y.")+ )report = g.edit_node("A", remove_fact="founded by X")@@ -301,7 +303,9 @@ class TestSoftDelete:g._node_memory.add([Entity(name="A", description="A was founded by X. A is in Y.")])- g._node_memory.llm_client = FakeChatModel(Entity(name="A", description="A is in Y."))+ g._node_memory.llm_client = FakeChatModel(+ Entity(name="A", description="A is in Y.")+ )report = g.edit_node("A", remove_fact="founded by X", dry_run=True)@@ -315,7 +319,9 @@ class TestSoftDelete:g._node_memory.add([Entity(name="A", description="A was founded by X. A is in Y.")])- g._node_memory.llm_client = FakeChatModel(Entity(name="A", description="A is in Y."))+ g._node_memory.llm_client = FakeChatModel(+ Entity(name="A", description="A is in Y.")+ )report = g.edit_node("A", remove_fact="founded by X")@@ -341,7 +347,9 @@ class TestSoftDelete:def test_key_change_is_rejected(self):g = _graph()g._node_memory.add([Entity(name="A")])- g._node_memory.llm_client = FakeChatModel(Entity(name="Renamed")) # identity break+ g._node_memory.llm_client = FakeChatModel(+ Entity(name="Renamed")+ ) # identity breakwith pytest.raises(ValueError, match="key changed"):g.edit_node("A", remove_fact="whatever")diff --git a/uv.lock b/uv.lockindex ba0d483..35ef944 100644--- a/uv.lock+++ b/uv.lock@@ -957,7 +957,7 @@ requires-dist = [{ name = "langchain-google-genai", marker = "extra == 'google'", specifier = ">=2.1.0" },{ name = "langchain-openai", specifier = ">=1.1.7" },{ name = "mcp", marker = "extra == 'mcp'", specifier = ">=1.2.0,<3" },- { name = "ontomem", specifier = ">=0.3.0" },+ { name = "ontomem", specifier = ">=0.5.0" },{ name = "ontosight", specifier = ">=0.1.8" },{ name = "python-dotenv", specifier = ">=1.2.1" },{ name = "rich", specifier = ">=13.7.0" },@@ -1879,7 +1879,7 @@ wheels = [[[package]]name = "ontomem"-version = "0.3.0"+version = "0.5.0"source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }dependencies = [{ name = "faiss-cpu" },@@ -1891,9 +1891,9 @@ dependencies = [{ name = "python-dotenv" },{ name = "structlog" },]-sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/99/b4/a76747d5dda4bcd0e107ee59aae4b89a952a35761a82408318eebf029b5b/ontomem-0.3.0.tar.gz", hash = "sha256:725405a54ea03380c7fa6a8c9a1badb561a23f971efbad5914f2e934bd9f511e", size = 2540504, upload-time = "2026-09-04T02:13:44.824Z" }+sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/1e/5e1e484d8d1ee7caafebf59273a5df87d9bfbc1d5f75ff49a702e8e69372/ontomem-0.5.0.tar.gz", hash = "sha256:25dd2c9676e40e1e11fc519a5b3547c79e263bcb64c17505f0e26560e10f10ee", size = 2549972, upload-time = "2026-09-04T03:55:53.486Z" }wheels = [- { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/5e/0930f49541d5fa61f6420b08fd2f3d10af98811a7630e815508af59be8a0/ontomem-0.3.0-py3-none-any.whl", hash = "sha256:a03520bca3db0ff87db992c4aeff5c1df121ac3a25f1111519538e7148d8b1e5", size = 41883, upload-time = "2026-09-04T02:13:42.991Z" },+ { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cc/42/adf7dcfc94e9e255fc52a7d38713746ad2fc83f374d157c99d7ca4ecbf82/ontomem-0.5.0-py3-none-any.whl", hash = "sha256:04b2d058cb7b0617d3f3d46fdc469282ed77cee42ccba133041807b775c42b9c", size = 47040, upload-time = "2026-09-04T03:55:52.236Z" },][[package]]