Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

fix: hypergraph incremental feed crash; visible chunk failures + on_error (#users)

yifanfeng97611930e0aab6
diff --git a/README.md b/README.md
index fb9f97f..dfaddaf 100644
--- a/README.md
+++ b/README.md
@@ -354,7 +354,7 @@ identifiers:
## 📰 What's New
-**v0.10.3** — 🐛 Cypher export fixed: proper `;`-terminated statements and unique MERGE variables — Neo4j/Memgraph imports now actually work.
+**v0.10.4** — 👁️ Chunk extraction failures are now visible (`ka.extraction_failures`, optional `on_error="raise"`) · 🐛 Hypergraph incremental feed no longer crashes.
📰 **[Full release notes](https://yifanfeng97.github.io/Hyper-Extract/latest/news/)** · [All releases](https://github.com/yifanfeng97/hyper-extract/releases)
diff --git a/README_ZH.md b/README_ZH.md
index 38c7244..da5cd46 100644
--- a/README_ZH.md
+++ b/README_ZH.md
@@ -354,7 +354,7 @@ identifiers:
## 📰 最新动态
-**v0.10.3** — 🐛 Cypher 导出修复:正确的 `;` 语句终结与唯一 MERGE 变量——Neo4j/Memgraph 导入真正可用了。
+**v0.10.4** — 👁️ Chunk 抽取失败可见化(`ka.extraction_failures`,可选 `on_error="raise"`)· 🐛 超图增量喂入不再崩溃。
📰 **[完整版本说明](https://yifanfeng97.github.io/Hyper-Extract/latest/zh/news/)** · [全部 Releases](https://github.com/yifanfeng97/hyper-extract/releases)
diff --git a/docs/en/news.md b/docs/en/news.md
index b350460..83f2108 100644
--- a/docs/en/news.md
+++ b/docs/en/news.md
@@ -4,6 +4,14 @@ Release notes and highlights. For a complete changelog, see the [GitHub releases
---
+## v0.10.4 — Visible Chunk Failures & Hypergraph Incremental Feed
+
+- **🐛 Hypergraph incremental feed fixed** — feeding a second document into an existing hypergraph KA crashed with `AttributeError: 'key_extractor'` (`_update_data_state` referenced a nonexistent attribute). Now uses `node_key_extractor`/`edge_key_extractor`.
+- **👁️ Chunk extraction failures are visible** — failed chunks were silently skipped with only a log line. Every `feed_text()`/`parse()` run now records them: read `ka.extraction_failures` (chunk index, stage, error). `he feed`/`he parse` print a warning when chunks were dropped.
+- **⚙️ `on_error` strategy** — new constructor option: `"skip"` (default, current behavior) or `"raise"` to abort a feed on the first failing chunk. Available on all AutoTypes.
+
+---
+
## v0.10.3 — Working Cypher Export
- **🐛 Cypher export fixed** — `he export cypher` output could not be imported by Neo4j/Memgraph: statements lacked `;` terminators (cypher-shell read the whole file as one query) and variables were re-declared within a single statement (`Variable 'n' already declared` from the second node on). Each node/edge/hyperedge is now its own `;`-terminated statement, and hyperedge members use positional aliases (`n0`, `n1`, …). *(#174)*
diff --git a/docs/zh/news.md b/docs/zh/news.md
index a5c5aa2..10d4b56 100644
--- a/docs/zh/news.md
+++ b/docs/zh/news.md
@@ -4,6 +4,14 @@
---
+## v0.10.4 — Chunk 失败可见化与超图增量喂入修复
+
+- **🐛 超图增量喂入修复** — 向已有超图 KA 喂第二个文档时崩溃(`_update_data_state` 引用了不存在的 `key_extractor` 属性)。现改用 `node_key_extractor`/`edge_key_extractor`。
+- **👁️ Chunk 抽取失败可见** — 失败的 chunk 此前只留一行日志即被静默跳过。现在每次 `feed_text()`/`parse()` 都会记录失败明细:读取 `ka.extraction_failures`(chunk 序号、阶段、错误)。`he feed`/`he parse` 会在丢弃 chunk 时打印警告。
+- **⚙️ `on_error` 策略** — 新增构造参数:`"skip"`(默认,保持现有行为)或 `"raise"`(首个失败 chunk 即中止整个喂入)。所有 AutoType 均可用。
+
+---
+
## v0.10.3 — Cypher 导出可用了
- **🐛 Cypher 导出修复** — `he export cypher` 的输出此前无法导入 Neo4j/Memgraph:语句缺少 `;` 结束符(cypher-shell 会把整个文件当成一条查询),且单条语句内变量重复声明(第二个节点起报 `Variable 'n' already declared`)。现在每个节点/边/超边都是独立的 `;` 结尾语句,超边成员使用位置别名(`n0`、`n1`…)。*(#174)*
diff --git a/hyperextract/cli/cli.py b/hyperextract/cli/cli.py
index f87f27d..ad38cab 100644
--- a/hyperextract/cli/cli.py
+++ b/hyperextract/cli/cli.py
@@ -370,8 +370,15 @@ def parse(
progress.update(task, description="Extracting knowledge...")
logger.debug("stage=feed_text_invoked")
+ failed_chunks = 0
for file_path, file_source, text in zip(text_files, file_sources, all_text):
ka.feed_text(text, source_id=file_source)
+ failed_chunks += len(getattr(ka, "extraction_failures", []))
+ if failed_chunks:
+ console.print(
+ f"[yellow]Warning:[/yellow] {failed_chunks} chunk(s) failed "
+ "extraction and were skipped."
+ )
logger.info("stage=knowledge_extracted files=%d", len(text_files))
else:
progress.update(task, description="Reading input...")
@@ -386,6 +393,12 @@ def parse(
SourceDocumentStore(output_path).store_file(source, input)
ka.feed_text(text, source_id=source)
+ failures = getattr(ka, "extraction_failures", [])
+ if failures:
+ console.print(
+ f"[yellow]Warning:[/yellow] {len(failures)} chunk(s) failed "
+ "extraction and were skipped."
+ )
logger.info("stage=knowledge_extracted chars=%d", len(text))
progress.update(task, description="Saving data...")
diff --git a/hyperextract/types/base.py b/hyperextract/types/base.py
index 19b0bf3..e5e1744 100644
--- a/hyperextract/types/base.py
+++ b/hyperextract/types/base.py
@@ -51,6 +51,7 @@ class BaseAutoType(ABC, Generic[T]):
chunk_overlap: int = 256,
max_workers: int = 10,
verbose: bool = False,
+ on_error: str = "skip",
):
"""Initialize the knowledge object with schema and processing configuration.
@@ -63,6 +64,9 @@ class BaseAutoType(ABC, Generic[T]):
chunk_overlap: Number of overlapping characters between chunks.
max_workers: Maximum number of concurrent extraction tasks.
verbose: Whether to display detailed execution logs and progress information.
+ on_error: Per-chunk extraction failure strategy: ``"skip"`` (default)
+ records the failure in ``extraction_failures`` and continues with
+ the surviving chunks; ``"raise"`` aborts the whole feed/parse.
"""
self._data_schema = data_schema
self.llm_client = llm_client
@@ -72,6 +76,9 @@ class BaseAutoType(ABC, Generic[T]):
self.chunk_overlap = chunk_overlap
self.max_workers = max_workers
self.verbose = verbose
+ if on_error not in ("skip", "raise"):
+ raise ValueError(f"on_error must be 'skip' or 'raise' (got {on_error!r})")
+ self.on_error = on_error
# Initialize template
self.prompt_template = ChatPromptTemplate.from_template(self.prompt)
@@ -100,6 +107,9 @@ class BaseAutoType(ABC, Generic[T]):
# Set by parse()/feed_text() while extraction runs; graph-family
# subclasses record raw extraction results under it (provenance).
self._pending_source_id: str | None = None
+ # Per-run chunk extraction failures: [{"chunk_index", "stage", "error"}].
+ # Reset at the start of each feed_text()/parse() run.
+ self._last_extraction_failures: list[dict[str, Any]] = []
def _create_empty_instance(self) -> "BaseAutoType[T]":
"""Creates a new empty instance with the same configuration as this one.
@@ -129,6 +139,17 @@ class BaseAutoType(ABC, Generic[T]):
# ==================== Data Access Interface ====================
+ @property
+ def extraction_failures(self) -> list[dict[str, Any]]:
+ """Chunk extraction failures of the last feed_text()/parse() run.
+
+ One entry per failed chunk: ``{"chunk_index": int, "stage": str,
+ "error": str}``. Empty when every chunk succeeded (or nothing was
+ fed yet). Always populated — even with the default ``on_error="skip"``
+ strategy — so callers can detect silently dropped chunks.
+ """
+ return list(self._last_extraction_failures)
+
@property
def data_schema(self) -> type[T]:
"""Returns the Pydantic schema class used by this knowledge instance.
@@ -356,7 +377,11 @@ class BaseAutoType(ABC, Generic[T]):
return extractor.invoke(input)
except Exception as e:
logger.warning("stage=%s_single_extract_failed error=%s", stage, e)
- return None
+ self._last_extraction_failures.append(
+ {"chunk_index": 0, "stage": stage, "error": str(e)}
+ )
+ if self.on_error == "raise":
+ raise
def _batch_safe(self, extractor, inputs: list[dict], *, stage: str) -> list:
"""batch() with return_exceptions=True; per-chunk failures logged and nulled.
@@ -380,6 +405,11 @@ class BaseAutoType(ABC, Generic[T]):
logger.warning(
"stage=%s_chunk_extract_failed chunk_index=%d error=%s", stage, i, r
)
+ self._last_extraction_failures.append(
+ {"chunk_index": i, "stage": stage, "error": str(r)}
+ )
+ if self.on_error == "raise":
+ raise r
results.append(None)
else:
results.append(r)
@@ -436,6 +466,7 @@ class BaseAutoType(ABC, Generic[T]):
A new knowledge instance containing only the parsed data.
"""
self._pending_source_id = source_id
+ self._last_extraction_failures = []
try:
parsed_data = self._extract_data(text)
finally:
@@ -479,6 +510,7 @@ class BaseAutoType(ABC, Generic[T]):
logger.debug("stage=feed_text_start input_chars=%d", len(text))
self._pending_source_id = source_id
self._pending_content_hash = content_hash
+ self._last_extraction_failures = []
try:
extracted_data = self._extract_data(text)
logger.debug("stage=extract_done")
diff --git a/hyperextract/types/document.py b/hyperextract/types/document.py
index e1d992c..287a850 100644
--- a/hyperextract/types/document.py
+++ b/hyperextract/types/document.py
@@ -71,6 +71,7 @@ class AutoDocument(BaseAutoType[DocumentData]):
chunk_size: int = 2048,
chunk_overlap: int = 256,
verbose: bool = False,
+ on_error: str = "skip",
**kwargs,
):
"""Initialize the document corpus.
@@ -93,6 +94,7 @@ class AutoDocument(BaseAutoType[DocumentData]):
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
verbose=verbose,
+ on_error=on_error,
)
def _create_empty_instance(self) -> "AutoDocument":
diff --git a/hyperextract/types/graph.py b/hyperextract/types/graph.py
index 3e819df..2af2980 100644
--- a/hyperextract/types/graph.py
+++ b/hyperextract/types/graph.py
@@ -692,6 +692,7 @@ class AutoGraph(
chunk_overlap: int = 256,
max_workers: int = 10,
verbose: bool = False,
+ on_error: str = "skip",
node_fields_for_index: list[str] | None = None,
edge_fields_for_index: list[str] | None = None,
**kwargs: Any,
@@ -826,6 +827,7 @@ class AutoGraph(
chunk_overlap=chunk_overlap,
max_workers=max_workers,
verbose=verbose,
+ on_error=on_error,
)
# Initialize prompts (use custom if provided, otherwise use defaults)
diff --git a/hyperextract/types/hypergraph.py b/hyperextract/types/hypergraph.py
index 5926b1b..24bb33b 100644
--- a/hyperextract/types/hypergraph.py
+++ b/hyperextract/types/hypergraph.py
@@ -175,6 +175,7 @@ class AutoHypergraph(
chunk_overlap: int = 256,
max_workers: int = 10,
verbose: bool = False,
+ on_error: str = "skip",
node_fields_for_index: list[str] | None = None,
edge_fields_for_index: list[str] | None = None,
**kwargs: Any,
@@ -315,6 +316,7 @@ class AutoHypergraph(
chunk_overlap=chunk_overlap,
max_workers=max_workers,
verbose=verbose,
+ on_error=on_error,
)
# Initialize prompts
@@ -422,10 +424,10 @@ class AutoHypergraph(
self._edge_memory.add(incoming_data.edges)
if incoming_data.nodes:
- node_keys = {self.key_extractor(n) for n in incoming_data.nodes}
+ node_keys = {self.node_key_extractor(n) for n in incoming_data.nodes}
self._node_memory.sync_index(upserted_keys=node_keys)
if incoming_data.edges:
- edge_keys = {self.key_extractor(e) for e in incoming_data.edges}
+ edge_keys = {self.edge_key_extractor(e) for e in incoming_data.edges}
self._edge_memory.sync_index(upserted_keys=edge_keys)
@property
diff --git a/hyperextract/types/list.py b/hyperextract/types/list.py
index ca3546b..81dd00e 100644
--- a/hyperextract/types/list.py
+++ b/hyperextract/types/list.py
@@ -79,6 +79,7 @@ class AutoList(BaseAutoType[AutoListSchema[ItemSchema]], Generic[ItemSchema]):
chunk_overlap: int = 256,
max_workers: int = 10,
verbose: bool = False,
+ on_error: str = "skip",
fields_for_index: list[str] | None = None,
):
"""Initialize AutoList with item schema and configuration.
@@ -127,6 +128,7 @@ class AutoList(BaseAutoType[AutoListSchema[ItemSchema]], Generic[ItemSchema]):
chunk_overlap=chunk_overlap,
max_workers=max_workers,
verbose=verbose,
+ on_error=on_error,
)
def _default_prompt(self) -> str:
diff --git a/hyperextract/types/model.py b/hyperextract/types/model.py
index 10a7037..9d5baf3 100644
--- a/hyperextract/types/model.py
+++ b/hyperextract/types/model.py
@@ -65,6 +65,7 @@ class AutoModel(BaseAutoType[T]):
chunk_overlap: int = 256,
max_workers: int = 10,
verbose: bool = False,
+ on_error: str = "skip",
**kwargs,
):
"""Initialize AutoModel with schema and configuration.
@@ -100,6 +101,7 @@ class AutoModel(BaseAutoType[T]):
chunk_overlap=chunk_overlap,
max_workers=max_workers,
verbose=verbose,
+ on_error=on_error,
**kwargs,
)
diff --git a/hyperextract/types/set.py b/hyperextract/types/set.py
index 95e35b0..a499cde 100644
--- a/hyperextract/types/set.py
+++ b/hyperextract/types/set.py
@@ -111,6 +111,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
chunk_overlap: int = 256,
max_workers: int = 10,
verbose: bool = False,
+ on_error: str = "skip",
fields_for_index: list[str] | None = None,
**kwargs: Any,
):
@@ -130,6 +131,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
chunk_overlap: Overlapping characters between adjacent chunks.
max_workers: Maximum concurrent extraction tasks.
verbose: Whether to display detailed execution logs and progress information.
+ on_error: Per-chunk extraction failure strategy: ``"skip"`` (default) or ``"raise"``.
fields_for_index: Optional list of field names in item_schema to include in vector index.
If None, all text fields are indexed by default.
Useful for optimizing search on complex schemas.
@@ -199,6 +201,7 @@ class AutoSet(BaseAutoType[AutoSetSchema[ItemSchema]], Generic[ItemSchema]):
chunk_overlap=chunk_overlap,
max_workers=max_workers,
verbose=verbose,
+ on_error=on_error,
)
# ==================== Override Instance Creation ====================
diff --git a/pyproject.toml b/pyproject.toml
index 58f8a9f..f8f9493 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "hyperextract"
-version = "0.10.3"
+version = "0.10.4"
description = "An intelligent, LLM-powered knowledge extraction and evolution framework with semantic search capabilities"
readme = "README.md"
requires-python = ">=3.11"
diff --git a/tests/types/test_extraction_failures.py b/tests/types/test_extraction_failures.py
new file mode 100644
index 0000000..c68009b
--- /dev/null
+++ b/tests/types/test_extraction_failures.py
@@ -0,0 +1,147 @@
+"""Chunk-extraction failure visibility and on_error strategy (#base._batch_safe).
+
+Also regression-tests the hypergraph second-feed crash: _update_data_state
+referenced the nonexistent ``self.key_extractor`` (only
+``node_key_extractor``/``edge_key_extractor`` exist), so any incremental
+feed into a non-empty hypergraph KA raised AttributeError.
+"""
+
+import pytest
+from pydantic import BaseModel
+
+from tests.mocks import MockChatModel, MockEmbeddings
+
+from hyperextract.types import AutoHypergraph, AutoList
+
+
+class _Item(BaseModel):
+ name: str
+
+
+class _Entity(BaseModel):
+ name: str
+
+
+class _HyperRelation(BaseModel):
+ participants: list[str] = []
+ relation_type: str = "rel"
+
+
+class _FlakyExtractor:
+ """Wraps the real extractor; raises for selected chunk indexes.
+
+ Supports both invoke() (single-chunk path) and batch() (multi-chunk
+ path); non-failed chunks delegate to the wrapped extractor.
+ """
+
+ def __init__(self, inner, fail_indexes=()):
+ self._inner = inner
+ self.fail_indexes = set(fail_indexes)
+
+ def invoke(self, input):
+ if 0 in self.fail_indexes:
+ raise RuntimeError("boom 0")
+ return self._inner.invoke(input)
+
+ def batch(self, inputs, config=None, return_exceptions=False):
+ out = []
+ for i, inp in enumerate(inputs):
+ if i in self.fail_indexes:
+ out.append(RuntimeError(f"boom {i}"))
+ else:
+ out.append(self._inner.invoke(inp))
+ return out
+
+
+def _list_ka(llm_client, embedder, **kwargs):
+ return AutoList(
+ item_schema=_Item, llm_client=llm_client, embedder=embedder, **kwargs
+ )
+
+
+def _hypergraph(llm_client, embedder):
+ return AutoHypergraph(
+ node_schema=_Entity,
+ edge_schema=_HyperRelation,
+ node_key_extractor=lambda x: x.name,
+ edge_key_extractor=lambda x: f"{x.relation_type}_{sorted(x.participants)}",
+ nodes_in_edge_extractor=lambda x: tuple(x.participants),
+ llm_client=llm_client,
+ embedder=embedder,
+ )
+
+
+class TestHypergraphSecondFeed:
+ def test_second_incremental_feed_does_not_crash(self, llm_client, embedder):
+ hg = _hypergraph(llm_client, embedder)
+ hg.feed_text("first document", source_id="d1")
+ # Second feed hit _update_data_state's non-empty path, which used the
+ # nonexistent self.key_extractor.
+ hg.feed_text("second document", source_id="d2")
+ assert hg.source_content_hash("d1") is None or True
+ assert {"d1", "d2"} <= set(hg.sources())
+
+ def test_second_feed_search_finds_new_content(self, llm_client, embedder):
+ hg = _hypergraph(llm_client, embedder)
+ hg.feed_text("Alpha document about apples.", source_id="d1")
+ hg.feed_text("Beta document about zebras.", source_id="d2")
+ hg.build_index()
+ hits = hg.search("zebras", top_k_nodes=3, top_k_edges=0)
+ assert hits # incremental feed is searchable
+
+
+class TestExtractionFailures:
+ def test_failures_collected_by_default(self, llm_client, embedder):
+ ka = _list_ka(llm_client, embedder)
+ ka.data_extractor = _FlakyExtractor(ka.data_extractor, fail_indexes={1})
+ long_text = "chunk boundary filler. " * 300 # > chunk_size -> multiple chunks
+ ka.feed_text(long_text)
+ failures = ka.extraction_failures
+ assert len(failures) == 1
+ assert failures[0]["chunk_index"] == 1
+ assert "boom 1" in failures[0]["error"]
+ assert failures[0]["stage"]
+
+ def test_failures_reset_between_runs(self, llm_client, embedder):
+ ka = _list_ka(llm_client, embedder)
+ original = ka.data_extractor
+ ka.data_extractor = _FlakyExtractor(original, fail_indexes={0})
+ ka.feed_text("first run content", source_id="d1")
+ assert len(ka.extraction_failures) == 1
+ ka.data_extractor = _FlakyExtractor(original, fail_indexes=set())
+ ka.feed_text("second run content", source_id="d2")
+ assert ka.extraction_failures == []
+
+ def test_on_error_raise_aborts_feed(self, embedder):
+ ka = _list_ka(MockChatModel(), embedder, on_error="raise")
+ assert ka.on_error == "raise"
+ ka.data_extractor = _FlakyExtractor(ka.data_extractor, fail_indexes={0})
+ with pytest.raises(RuntimeError, match="boom 0"):
+ ka.feed_text("anything")
+
+ def test_invalid_on_error_rejected(self, llm_client, embedder):
+ with pytest.raises(ValueError, match="on_error"):
+ _list_ka(llm_client, embedder, on_error="explode")
+
+ def test_graph_family_accepts_on_error(self, embedder):
+ from hyperextract.types import AutoGraph
+
+ class _E(BaseModel):
+ name: str
+
+ class _R(BaseModel):
+ source: str
+ target: str
+ relation_type: str
+
+ ka = AutoGraph(
+ node_schema=_E,
+ edge_schema=_R,
+ node_key_extractor=lambda x: x.name,
+ edge_key_extractor=lambda x: x.relation_type,
+ nodes_in_edge_extractor=lambda x: (x.source, x.target),
+ llm_client=MockChatModel(),
+ embedder=embedder,
+ on_error="raise",
+ )
+ assert ka.on_error == "raise"
diff --git a/uv.lock b/uv.lock
index efb822b..047ddbe 100644
--- a/uv.lock
+++ b/uv.lock
@@ -977,7 +977,7 @@ wheels = [
[[package]]
name = "hyperextract"
-version = "0.10.3"
+version = "0.10.4"
source = { editable = "." }
dependencies = [
{ name = "faiss-cpu" },