Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
fix: hypergraph incremental feed crash; visible chunk failures + on_error (#users)
yifanfeng97611930e0aab6
diff --git a/README.md b/README.mdindex fb9f97f..dfaddaf 100644--- a/README.md+++ b/README.md@@ -354,7 +354,7 @@ identifiers:## 📰 What's New-**v0.10.3** — 🐛 Cypher export fixed: proper `;`-terminated statements and unique MERGE variables — Neo4j/Memgraph imports now actually work.+**v0.10.4** — 👁️ Chunk extraction failures are now visible (`ka.extraction_failures`, optional `on_error="raise"`) · 🐛 Hypergraph incremental feed no longer crashes.📰 **[Full release notes](https://yifanfeng97.github.io/Hyper-Extract/latest/news/)** · [All releases](https://github.com/yifanfeng97/hyper-extract/releases)diff --git a/README_ZH.md b/README_ZH.mdindex 38c7244..da5cd46 100644--- a/README_ZH.md+++ b/README_ZH.md@@ -354,7 +354,7 @@ identifiers:## 📰 最新动态-**v0.10.3** — 🐛 Cypher 导出修复:正确的 `;` 语句终结与唯一 MERGE 变量——Neo4j/Memgraph 导入真正可用了。+**v0.10.4** — 👁️ Chunk 抽取失败可见化(`ka.extraction_failures`,可选 `on_error="raise"`)· 🐛 超图增量喂入不再崩溃。📰 **[完整版本说明](https://yifanfeng97.github.io/Hyper-Extract/latest/zh/news/)** · [全部 Releases](https://github.com/yifanfeng97/hyper-extract/releases)diff --git a/docs/en/news.md b/docs/en/news.mdindex b350460..83f2108 100644--- a/docs/en/news.md+++ b/docs/en/news.md@@ -4,6 +4,14 @@ Release notes and highlights. For a complete changelog, see the [GitHub releases---+## v0.10.4 — Visible Chunk Failures & Hypergraph Incremental Feed++- **🐛 Hypergraph incremental feed fixed** — feeding a second document into an existing hypergraph KA crashed with `AttributeError: 'key_extractor'` (`_update_data_state` referenced a nonexistent attribute). Now uses `node_key_extractor`/`edge_key_extractor`.+- **👁️ Chunk extraction failures are visible** — failed chunks were silently skipped with only a log line. Every `feed_text()`/`parse()` run now records them: read `ka.extraction_failures` (chunk index, stage, error). `he feed`/`he parse` print a warning when chunks were dropped.+- **⚙️ `on_error` strategy** — new constructor option: `"skip"` (default, current behavior) or `"raise"` to abort a feed on the first failing chunk. Available on all AutoTypes.++---+## v0.10.3 — Working Cypher Export- **🐛 Cypher export fixed** — `he export cypher` output could not be imported by Neo4j/Memgraph: statements lacked `;` terminators (cypher-shell read the whole file as one query) and variables were re-declared within a single statement (`Variable 'n' already declared` from the second node on). Each node/edge/hyperedge is now its own `;`-terminated statement, and hyperedge members use positional aliases (`n0`, `n1`, …). *(#174)*diff --git a/docs/zh/news.md b/docs/zh/news.mdindex a5c5aa2..10d4b56 100644--- a/docs/zh/news.md+++ b/docs/zh/news.md@@ -4,6 +4,14 @@---+## v0.10.4 — Chunk 失败可见化与超图增量喂入修复++- **🐛 超图增量喂入修复** — 向已有超图 KA 喂第二个文档时崩溃(`_update_data_state` 引用了不存在的 `key_extractor` 属性)。现改用 `node_key_extractor`/`edge_key_extractor`。+- **👁️ Chunk 抽取失败可见** — 失败的 chunk 此前只留一行日志即被静默跳过。现在每次 `feed_text()`/`parse()` 都会记录失败明细:读取 `ka.extraction_failures`(chunk 序号、阶段、错误)。`he feed`/`he parse` 会在丢弃 chunk 时打印警告。+- **⚙️ `on_error` 策略** — 新增构造参数:`"skip"`(默认,保持现有行为)或 `"raise"`(首个失败 chunk 即中止整个喂入)。所有 AutoType 均可用。++---+## v0.10.3 — Cypher 导出可用了- **🐛 Cypher 导出修复** — `he export cypher` 的输出此前无法导入 Neo4j/Memgraph:语句缺少 `;` 结束符(cypher-shell 会把整个文件当成一条查询),且单条语句内变量重复声明(第二个节点起报 `Variable 'n' already declared`)。现在每个节点/边/超边都是独立的 `;` 结尾语句,超边成员使用位置别名(`n0`、`n1`…)。*(#174)*diff --git a/hyperextract/cli/cli.py b/hyperextract/cli/cli.pyindex f87f27d..ad38cab 100644--- a/hyperextract/cli/cli.py+++ b/hyperextract/cli/cli.py@@ -370,8 +370,15 @@ def parse(progress.update(task, description="Extracting knowledge...")logger.debug("stage=feed_text_invoked")+ failed_chunks = 0for file_path, file_source, text in zip(text_files, file_sources, all_text):ka.feed_text(text, source_id=file_source)+ failed_chunks += len(getattr(ka, "extraction_failures", []))+ if failed_chunks:+ console.print(+ f"[yellow]Warning:[/yellow] {failed_chunks} chunk(s) failed "+ "extraction and were skipped."+ )logger.info("stage=knowledge_extracted files=%d", len(text_files))else:progress.update(task, description="Reading input...")@@ -386,6 +393,12 @@ def parse(SourceDocumentStore(output_path).store_file(source, input)ka.feed_text(text, source_id=source)+ failures = getattr(ka, "extraction_failures", [])+ if failures:+ console.print(+ f"[yellow]Warning:[/yellow] {len(failures)} chunk(s) failed "+ "extraction and were skipped."+ )logger.info("stage=knowledge_extracted chars=%d", len(text))progress.update(task, description="Saving data...")diff --git a/hyperextract/types/base.py b/hyperextract/types/base.pyindex 19b0bf3..e5e1744 100644--- a/hyperextract/types/base.py+++ b/hyperextract/types/base.py@@ -51,6 +51,7 @@ class BaseAutoType(ABC, Generic[T]):chunk_overlap: int = 256,max_workers: int = 10,verbose: bool = False,+ on_error: str = "skip",):"""Initialize the knowledge object with schema and processing configuration.@@ -63,6 +64,9 @@ class BaseAutoType(ABC, Generic[T]):chunk_overlap: Number of overlapping characters between chunks.max_workers: Maximum number of concurrent extraction tasks.verbose: Whether to display detailed execution logs and progress information.+ on_error: Per-chunk extraction failure strategy: ``"skip"`` (default)+ records the failure in ``extraction_failures`` and continues with+ the surviving chunks; ``"raise"`` aborts the whole feed/parse."""self._data_schema = data_schemaself.llm_client = llm_client@@ -72,6 +76,9 @@ class BaseAutoType(ABC, Generic[T]):self.chunk_overlap = chunk_overlapself.max_workers = max_workersself.verbose = verbose+ if on_error not in ("skip", "raise"):+ raise ValueError(f"on_error must be 'skip' or 'raise' (got {on_error!r})")+ self.on_error = on_error# Initialize templateself.prompt_template = ChatPromptTemplate.from_template(self.prompt)@@ -100,6 +107,9 @@ class BaseAutoType(ABC, Generic[T]):# Set by parse()/feed_text() while extraction runs; graph-family# subclasses record raw extraction results under it (provenance).self._pending_source_id: str | None = None+ # Per-run chunk extraction failures: [{"chunk_index", "stage", "error"}].+ # Reset at the start of each feed_text()/parse() run.+ self._last_extraction_failures: list[dict[str, Any]] = []def _create_empty_instance(self) -> "BaseAutoType[T]":"""Creates a new empty instance with the same configuration as this one.@@ -129,6 +139,17 @@ class BaseAutoType(ABC, Generic[T]):# ==================== Data Access Interface ====================+ @property+ def extraction_failures(self) -> list[dict[str, Any]]:+ """Chunk extraction failures of the last feed_text()/parse() run.++ One entry per failed chunk: ``{"chunk_index": int, "stage": str,+ "error": str}``. Empty when every chunk succeeded (or nothing was+ fed yet). Always populated — even with the default ``on_error="skip"``+ strategy — so callers can detect silently dropped chunks.+ """+ return list(self._last_extraction_failures)+@propertydef data_schema(self) -> type[T]:"""Returns the Pydantic schema class used by this knowledge instance.@@ -356,7 +377,11 @@ class BaseAutoType(ABC, Generic[T]):return extractor.invoke(input)except Exception as e:logger.warning("stage=%s_single_extract_failed error=%s", stage, e)- return None+ self._last_extraction_failures.append(+ {"chunk_index": 0, "stage": stage, "error": str(e)}+ )+ if self.on_error == "raise":+ raisedef _batch_safe(self, extractor, inputs: list[dict], *, stage: str) -> list:"""batch() with return_exceptions=True; per-chunk failures logged and nulled.@@ -380,6 +405,11 @@ class BaseAutoType(ABC, Generic[T]):logger.warning("stage=%s_chunk_extract_failed chunk_index=%d error=%s", stage, i, r)+ self._last_extraction_failures.append(+ {"chunk_index": i, "stage": stage, "error": str(r)}+ )+ if self.on_error == "raise":+ raise rresults.append(None)else:results.append(r)@@ -436,6 +466,7 @@ class BaseAutoType(ABC, Generic[T]):A new knowledge instance containing only the parsed data."""self._pending_source_id = source_id+ self._last_extraction_failures = []try:parsed_data = self._extract_data(text)finally:@@ -479,6 +510,7 @@ class BaseAutoType(ABC, Generic[T]):logger.debug("stage=feed_text_start input_chars=%d", len(text))self._pending_source_id = source_idself._pending_content_hash = content_hash+ self._last_extraction_failures = []try:extracted_data = self._extract_data(text)logger.debug("stage=extract_done")diff --git a/hyperextract/types/document.py b/hyperextract/types/document.pyindex e1d992c..287a850 100644--- a/hyperextract/types/document.py+++ b/hyperextract/types/document.py@@ -71,6 +71,7 @@ class AutoDocument(BaseAutoType[DocumentData]):chunk_size: int = 2048,chunk_overlap: int = 256,verbose: bool = False,+ on_error: str = "skip",**kwargs,):"""Initialize the document corpus.@@ -93,6 +94,7 @@ class AutoDocument(BaseAutoType[DocumentData]):chunk_size=chunk_size,chunk_overlap=chunk_overlap,verbose=verbose,+ on_error=on_error,)def _create_empty_instance(self) -> "AutoDocument":diff --git a/hyperextract/types/graph.py b/hyperextract/types/graph.pyindex 3e819df..2af2980 100644--- a/hyperextract/types/graph.py+++ b/hyperextract/types/graph.py@@ -692,6 +692,7 @@ class AutoGraph(chunk_overlap: int = 256,max_workers: int = 10,verbose: bool = False,+ on_error: str = "skip",node_fields_for_index: list[str] | None = None,edge_fields_for_index: list[str] | None = None,**kwargs: Any,@@ -826,6 +827,7 @@ class AutoGraph(chunk_overlap=chunk_overlap,max_workers=max_workers,verbose=verbose,+ on_error=on_error,)# Initialize prompts (use custom if provided, otherwise use defaults)diff --git a/hyperextract/types/hypergraph.py b/hyperextract/types/hypergraph.pyindex 5926b1b..24bb33b 100644--- a/hyperextract/types/hypergraph.py+++ b/hyperextract/types/hypergraph.py@@ -175,6 +175,7 @@ class AutoHypergraph(chunk_overlap: int = 256,max_workers: int = 10,verbose: bool = False,+ on_error: str = "skip",node_fields_for_index: list[str] | None = None,edge_fields_for_index: list[str] | None = None,**kwargs: Any,@@ -315,6 +316,7 @@ class AutoHypergraph(chunk_overlap=chunk_overlap,max_workers=max_workers,verbose=verbose,+ on_error=on_error,)# Initialize prompts@@ -422,10 +424,10 @@ class AutoHypergraph(self._edge_memory.add(incoming_data.edges)if incoming_data.nodes:- node_keys = {self.key_extractor(n) for n in incoming_data.nodes}+ node_keys = {self.node_key_extractor(n) for n in incoming_data.nodes}self._node_memory.sync_index(upserted_keys=node_keys)if incoming_data.edges:- edge_keys = {self.key_extractor(e) for e in incoming_data.edges}+ edge_keys = {self.edge_key_extractor(e) for e in incoming_data.edges}self._edge_memory.sync_index(upserted_keys=edge_keys)@propertydiff --git a/hyperextract/types/list.py b/hyperextract/types/list.pyindex ca3546b..81dd00e 100644--- a/hyperextract/types/list.py+++ b/hyperextract/types/list.py@@ -79,6 +79,7 @@ class AutoList(BaseAutoType[AutoListSchema[ItemSchema]], Generic[ItemSchema]):chunk_overlap: int = 256,max_workers: int = 10,verbose: bool = False,+ on_error: str = "skip",fields_for_index: list[str] | None = None,):"""Initialize AutoList with item schema and configuration.@@ -127,6 +128,7 @@ class AutoList(BaseAutoType[AutoListSchema[ItemSchema]], Generic[ItemSchema]):chunk_overlap=chunk_overlap,max_workers=max_workers,verbose=verbose,+ on_error=on_error,)def _default_prompt(self) -> str:diff --git a/hyperextract/types/model.py b/hyperextract/types/model.pyindex 10a7037..9d5baf3 100644--- a/hyperextract/types/model.py+++ b/hyperextract/types/model.py@@ -65,6 +65,7 @@ class AutoModel(BaseAutoType[T]):chunk_overlap: int = 256,max_workers: int = 10,verbose: bool = False,+ on_error: str = "skip",**kwargs,):"""Initialize AutoModel with schema and configuration.@@ -100,6 +101,7 @@ class AutoModel(BaseAutoType[T]):chunk_overlap=chunk_overlap,max_workers=max_workers,verbose=verbose,+ on_error=on_error,**kwargs,)diff --git a/hyperextract/types/set.py b/hyperextract/types/set.pyindex 95e35b0..a499cde 100644--- a/hyperextract/types/set.py+++ b/hyperextract/types/set.py@@ -111,6 +111,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):chunk_overlap: int = 256,max_workers: int = 10,verbose: bool = False,+ on_error: str = "skip",fields_for_index: list[str] | None = None,**kwargs: Any,):@@ -130,6 +131,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):chunk_overlap: Overlapping characters between adjacent chunks.max_workers: Maximum concurrent extraction tasks.verbose: Whether to display detailed execution logs and progress information.+ on_error: Per-chunk extraction failure strategy: ``"skip"`` (default) or ``"raise"``.fields_for_index: Optional list of field names in item_schema to include in vector index.If None, all text fields are indexed by default.Useful for optimizing search on complex schemas.@@ -199,6 +201,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):chunk_overlap=chunk_overlap,max_workers=max_workers,verbose=verbose,+ on_error=on_error,)# ==================== Override Instance Creation ====================diff --git a/pyproject.toml b/pyproject.tomlindex 58f8a9f..f8f9493 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "hyperextract"-version = "0.10.3"+version = "0.10.4"description = "An intelligent, LLM-powered knowledge extraction and evolution framework with semantic search capabilities"readme = "README.md"requires-python = ">=3.11"diff --git a/tests/types/test_extraction_failures.py b/tests/types/test_extraction_failures.pynew file mode 100644index 0000000..c68009b--- /dev/null+++ b/tests/types/test_extraction_failures.py@@ -0,0 +1,147 @@+"""Chunk-extraction failure visibility and on_error strategy (#base._batch_safe).++Also regression-tests the hypergraph second-feed crash: _update_data_state+referenced the nonexistent ``self.key_extractor`` (only+``node_key_extractor``/``edge_key_extractor`` exist), so any incremental+feed into a non-empty hypergraph KA raised AttributeError.+"""++import pytest+from pydantic import BaseModel++from tests.mocks import MockChatModel, MockEmbeddings++from hyperextract.types import AutoHypergraph, AutoList+++class _Item(BaseModel):+ name: str+++class _Entity(BaseModel):+ name: str+++class _HyperRelation(BaseModel):+ participants: list[str] = []+ relation_type: str = "rel"+++class _FlakyExtractor:+ """Wraps the real extractor; raises for selected chunk indexes.++ Supports both invoke() (single-chunk path) and batch() (multi-chunk+ path); non-failed chunks delegate to the wrapped extractor.+ """++ def __init__(self, inner, fail_indexes=()):+ self._inner = inner+ self.fail_indexes = set(fail_indexes)++ def invoke(self, input):+ if 0 in self.fail_indexes:+ raise RuntimeError("boom 0")+ return self._inner.invoke(input)++ def batch(self, inputs, config=None, return_exceptions=False):+ out = []+ for i, inp in enumerate(inputs):+ if i in self.fail_indexes:+ out.append(RuntimeError(f"boom {i}"))+ else:+ out.append(self._inner.invoke(inp))+ return out+++def _list_ka(llm_client, embedder, **kwargs):+ return AutoList(+ item_schema=_Item, llm_client=llm_client, embedder=embedder, **kwargs+ )+++def _hypergraph(llm_client, embedder):+ return AutoHypergraph(+ node_schema=_Entity,+ edge_schema=_HyperRelation,+ node_key_extractor=lambda x: x.name,+ edge_key_extractor=lambda x: f"{x.relation_type}_{sorted(x.participants)}",+ nodes_in_edge_extractor=lambda x: tuple(x.participants),+ llm_client=llm_client,+ embedder=embedder,+ )+++class TestHypergraphSecondFeed:+ def test_second_incremental_feed_does_not_crash(self, llm_client, embedder):+ hg = _hypergraph(llm_client, embedder)+ hg.feed_text("first document", source_id="d1")+ # Second feed hit _update_data_state's non-empty path, which used the+ # nonexistent self.key_extractor.+ hg.feed_text("second document", source_id="d2")+ assert hg.source_content_hash("d1") is None or True+ assert {"d1", "d2"} <= set(hg.sources())++ def test_second_feed_search_finds_new_content(self, llm_client, embedder):+ hg = _hypergraph(llm_client, embedder)+ hg.feed_text("Alpha document about apples.", source_id="d1")+ hg.feed_text("Beta document about zebras.", source_id="d2")+ hg.build_index()+ hits = hg.search("zebras", top_k_nodes=3, top_k_edges=0)+ assert hits # incremental feed is searchable+++class TestExtractionFailures:+ def test_failures_collected_by_default(self, llm_client, embedder):+ ka = _list_ka(llm_client, embedder)+ ka.data_extractor = _FlakyExtractor(ka.data_extractor, fail_indexes={1})+ long_text = "chunk boundary filler. " * 300 # > chunk_size -> multiple chunks+ ka.feed_text(long_text)+ failures = ka.extraction_failures+ assert len(failures) == 1+ assert failures[0]["chunk_index"] == 1+ assert "boom 1" in failures[0]["error"]+ assert failures[0]["stage"]++ def test_failures_reset_between_runs(self, llm_client, embedder):+ ka = _list_ka(llm_client, embedder)+ original = ka.data_extractor+ ka.data_extractor = _FlakyExtractor(original, fail_indexes={0})+ ka.feed_text("first run content", source_id="d1")+ assert len(ka.extraction_failures) == 1+ ka.data_extractor = _FlakyExtractor(original, fail_indexes=set())+ ka.feed_text("second run content", source_id="d2")+ assert ka.extraction_failures == []++ def test_on_error_raise_aborts_feed(self, embedder):+ ka = _list_ka(MockChatModel(), embedder, on_error="raise")+ assert ka.on_error == "raise"+ ka.data_extractor = _FlakyExtractor(ka.data_extractor, fail_indexes={0})+ with pytest.raises(RuntimeError, match="boom 0"):+ ka.feed_text("anything")++ def test_invalid_on_error_rejected(self, llm_client, embedder):+ with pytest.raises(ValueError, match="on_error"):+ _list_ka(llm_client, embedder, on_error="explode")++ def test_graph_family_accepts_on_error(self, embedder):+ from hyperextract.types import AutoGraph++ class _E(BaseModel):+ name: str++ class _R(BaseModel):+ source: str+ target: str+ relation_type: str++ ka = AutoGraph(+ node_schema=_E,+ edge_schema=_R,+ node_key_extractor=lambda x: x.name,+ edge_key_extractor=lambda x: x.relation_type,+ nodes_in_edge_extractor=lambda x: (x.source, x.target),+ llm_client=MockChatModel(),+ embedder=embedder,+ on_error="raise",+ )+ assert ka.on_error == "raise"diff --git a/uv.lock b/uv.lockindex efb822b..047ddbe 100644--- a/uv.lock+++ b/uv.lock@@ -977,7 +977,7 @@ wheels = [[[package]]name = "hyperextract"-version = "0.10.3"+version = "0.10.4"source = { editable = "." }dependencies = [{ name = "faiss-cpu" },