Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

feat: 新增 iText2KG_Star 模块及演示,优化知识图谱提取与语义去重功能

yifanfeng97caa0691384b4
diff --git a/examples/itext2kg_star_demo.py b/examples/itext2kg_star_demo.py
new file mode 100644
index 0000000..4edb2a2
--- /dev/null
+++ b/examples/itext2kg_star_demo.py
@@ -0,0 +1,291 @@
+"""
+iText2KG_Star 演示:知识图谱提取与语义去重
+
+这个演示展示了 iText2KG_Star 的高级功能:
+1. 单阶段提取:直接从文本中提取关系(三元组),自动推导实体。
+2. 语义去重:使用 SemHash 算法合并相似实体(例如 "雷霆之怒" 和 "会长雷霆之怒")。
+"""
+
+import os
+import sys
+from collections import Counter
+
+# 添加项目根目录到路径
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dotenv import load_dotenv
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+
+from hyperextract.graphs.itext2kg_star import iText2KG_Star
+
+load_dotenv()
+
+# ==============================================================================
+# 示例文本 (同 iText2KG Demo)
+# ==============================================================================
+EXAMPLE_TEXT = """
+【818】星辰阁公会昨晚解散的真相!团长黑装备还骂人!这简直是最大的内讧事件!
+
+楼主:匿名玩家(ID已隐藏)
+发帖时间:今天 23:45
+
+我是星辰阁的一个普通成员,昨晚目睹了整个悲剧的发生。我觉得必须把真相告诉大家。
+
+首先介绍一下我们的团队阵容:
+- 会长兼主坦克:雷霆之怒(真实ID:张三,40级战士,装备评分9999)
+- 副坦克:坚如磐石(真实ID:李四,38级盾战士,评分8800,人很老实)
+- 主奶妈(治疗):小甜甜(真实ID:王五,38级圣骑士,评分8600)
+- 副奶妈:月光治愈者(真实ID:赵六,35级德鲁伊,评分7500)
+- DPS输出1(弓箭手):暗影刺客(真实ID:孙七,39级猎手,评分9200)
+- DPS输出2(法师):火球术一级(真实ID:周八,37级法师,评分8400)
+- DPS输出3(剑客):风云剑侠(真实ID:吴九,36级剑士,评分7800)
+- 远程法师:冰霜法咒(真实ID:郑十,35级冰法,评分7600)
+
+昨晚晚上8点,我们集合在副本入口。这是星辰阁第三次挑战"深渊魔龙"这个最终BOSS。
+前两次都失败了,主要是因为配合不够默契。但是昨晚会长雷霆之怒发誓说一定要拿下这个BOSS。
+
+副本开始,一切似乎很顺利。前5个小怪都被我们轻松秒杀。
+然后大家来到了BOSS房间——深渊魔龙的巢穴。这是一个超级庞大的紫龙,满血时候气势相当恐怖。
+
+战斗一开始,雷霆之怒(会长)冲上去拉住了仇恨值。坚如磐石(副坦克)站在侧翼进行补充。
+小甜甜不断给雷霆之怒刷HOT(持续恢复)。月光治愈者给坚如磐石加盾。
+前期看起来非常完美,大家的DPS输出也打得相当漂亮。
+
+但是,大约打了5分钟,问题出现了。
+
+坚如磐石(副坦克)突然掉线了!这太糟糕了。我们在公会频道里看到他的在线状态变成了灰色。
+公会会长雷霆之怒看到这一幕,血压瞬间上升(我可以想象他的脸色)。
+
+为什么副坦会掉线呢?大家后来才明白——坚如磐石的家里停电了。这真的是太不走运了。
+
+没有副坦克的补充,所有的伤害都集中在了雷霆之怒身上。
+但是这个时候,事情开始变得诡异。
+
+小甜甜(主奶妈)本应该全力治疗会长,但是我通过伤害统计面板发现,她的治疗量明显不足。
+更奇怪的是,她把大量的治疗都给了暗影刺客(弓箭手)。
+
+这很不正常。暗影刺客作为DPS,血量本不应该这么低。我仔细看了一下暗影刺客的走位——
+哎哟,这哥们为了最大化伤害输出,站位超级靠前,几乎要贴到BOSS身上了。
+结果BOSS的爪子扫过来,一下子就把暗影刺客的血条扫到了20%以下。
+
+关键时刻,小甜甜却把大治疗术给了暗影刺客。这让会长雷霆之怒的血量开始急速下降。
+
+我当时就在YY频道里喊:"小甜甜,快给会长加血!"
+
+月光治愈者也在拼命给会长加盾,但是她的输出能力有限,根本抵挡不住BOSS的伤害。
+
+而且我还发现,月光治愈者的治疗也有一部分莫名其妙地给了其他人。看起来公会里有人在划水。
+
+等等,我再看一下。哦天哪,火球术一级(法师)和冰霜法咒在BOSS的AoE范围内站了太长时间,
+导致他们频繁掉血。但是,更夸张的是,暗影刺客和小甜甜之间似乎有什么默契——
+每当暗影刺客掉血,小甜甜就立马给他加。
+
+这时候,我想到了一个可能性。我在游戏论坛里看过一些帖子,有人提到暗影刺客和小甜甜好像是现实中的情侣!
+
+妈呀,这下问题大了。在这么关键的时刻,小甜甜竟然偏心给她的男朋友加血,而忽视了作为团队大脑的会长!
+
+大约在15分钟的时候,BOSS放出了一个大招——"魔龙吐息"。
+这个技能造成的伤害足以秒杀任何一个DPS。但是坦克(雷霆之怒)应该能扛住。
+
+可是,小甜甜的治疗没有及时跟上。雷霆之怒的血条瞬间掉到了0。
+
+会长倒地了!
+
+随后,失去了坦克的护佑,暗影刺客也被BOSS的普通攻击打成了残血,然后被秒杀。
+紧接着是风云剑侠,然后是火球术一级。整个团队在短短30秒内全灭了。
+
+副本失败。大家都复活在了副本入口。
+
+这个时候,雷霆之怒(会长)的脾气彻底炸裂了。
+
+他在YY频道里破口大骂:"小甜甜,你是干什么呢?!你是团队奶妈,不是暗影刺客的私人医生!
+治疗职业就是要保坦克,你这样的治疗我见过,但从没见过这么烂的!"
+
+小甜甜试图辩解:"我……我只是……"
+
+雷霆之怒直接打断她:"别给我狡辩!我决定了,你从今天起被踢出公会!
+而且暗影刺客这个摸鱼的家伙,我也要把他赶出去。两个人都别在星辰阁待着!"
+
+但就在这个时候,事情发生了急转弯。
+
+暗影刺客一句话都没说,只见他的角色突然冲向了BOSS房间。
+而就在这时,副本的第二个阶段秘密掉落物品——传说级别的武器"魔龙之牙"出现了!
+
+这是一把超级稀有的武器,全服务器可能只有两三把。价值高达几百万金币。
+
+没等任何人反应过来,暗影刺客就冲上去,一把拿起了"魔龙之牙"!
+
+我们都懵了。"魔龙之牙"本来应该由队长(雷霆之怒)来分配,决定给谁。
+但是暗影刺客直接黑吃黑,把这把传说武器直接塞进了自己的背包里!
+
+会长还在语音里骂人,根本没注意到这一幕。
+
+紧接着,更狠的来了——暗影刺客在游戏里按下了"退出公会"的按钮。
+他直接,光速,毫不犹豫地退出了星辰阁!
+
+然后他的角色在世界频道发了一句话:"拜拜,各位。我去找个真正尊重我的公会。"
+
+说完,这哥们直接下线了。
+
+我们的公会频道顿时炸锅了。
+
+小甜甜也没做任何辩解,直接退出了公会。她甚至没有说一句话。
+
+会长雷霆之怒这时候才意识到发生了什么。他看到了掉落物品的分配记录——
+"魔龙之牙"被暗影刺客拿走了,而不是团队共有。
+
+他疯了。直接在世界频道开始刷屏:
+"各位玩家,我要举报星辰阁的暗影刺客和小甜甜这两条狗!他们合谋黑装备!
+不要信任他们!全服通缉这两个垃圾!"
+
+火球术一级(法师)现在也开始助威。他在世界频道里连发了十多条信息:
+"赞同!这两个人太过分了!狗男女!不要和他们做交易!"
+
+"冰霜法咒在副本里的表现也不太理想,看起来他好像和那对情侣有什么关系,
+因为我注意到在暗影刺客掉线的时候,冰霜法咒也莫名其妙地不进行输出了。"
+
+有人开始怀疑冰霜法咒是不是也是那对情侣的朋友。
+
+坚如磐石(副坦)这时候才上线。他看到了聊天记录,一脸懵逼。
+他在公会频道里问:"发生什么了?我家里停电了……"
+
+没有人回答他。因为大家都在看这场大戏。
+
+月光治愈者试图调和一下,她说:"大家冷静……这件事可能有误会……"
+
+但是没有人听她的。因为这个时候,世界频道上的玩家也开始评论这件事。
+
+有人说:"这就是为什么我不玩大公会。大公会水太深。"
+
+还有人说:"我早就听说暗影刺客这个人在游戏里名声不太好。多次被指控在副本里贪图装备。"
+
+有人挖出了暗影刺客的历史记录。原来他曾经在另一个公会里,也因为类似的事情被踢出来过。
+
+这哥们简直就是个"装备贪心鬼"。
+
+风云剑侠开始反思:"也许我们团队的问题不只是小甜甜。我们可能选错人了。"
+
+火球术一级继续在世界频道里刷屏:"各位,我建议大家抵制这两个玩家。不要和他们一个队。"
+
+这时候,一个名字叫做"游戏管理员001"的官方账号出现了。
+他说:"我们收到了关于玩家暗影刺客的举报。我们正在调查这件事。如果确认违规,将给予封号处理。"
+
+暗影刺客的名字从世界频道消失了——他已经下线了。而且下线之前把"魔龙之牙"从背包里转移到了仓库。
+没人知道他把这把武器藏在了哪里。
+
+最后,星辰阁的会长雷霆之怒在公会频道里宣布:
+"各位成员,我决定了。星辰阁正式解散。这个公会不适合我继续管理。
+我会转移到一个小公会,和志同道合的朋友重新开始。"
+
+很多成员试图挽留他。但是他的决心已定。
+
+现在,星辰阁已经成为了一个"死公会"。只有一些离不开这个ID的成员还在。
+
+这就是昨晚发生的一切。我作为一个旁观者,见证了这场游戏江湖的大戏。
+
+最后的最后,我想说:游戏就是游戏。团队合作最重要。装备是身外之物。
+但是,为了装备而背叛队友,这绝对是游戏玩家中最可耻的行为。
+
+希望游戏官方能够严肃处理暗影刺客的违规行为。
+也希望星辰阁的所有成员能够吸取教训,在各自的新公会里找到快乐。
+"""
+
+
+def print_stats(kg, title: str):
+ """打印图谱统计信息"""
+ print("\n" + "-" * 60)
+ print(f"📊 {title}")
+ print("-" * 60)
+ print(f" 🔹 节点数量: {len(kg.nodes)}")
+ print(f" 🔹 关系数量: {len(kg.edges)}")
+
+ # 列出 Top 5 节点 (按度数)
+ entity_degree = Counter()
+ for edge in kg.edges:
+ entity_degree[edge.startNode.name] += 1
+ entity_degree[edge.endNode.name] += 1
+
+ print("\n 🏆 活跃实体 (Top 5):")
+ for name, count in entity_degree.most_common(5):
+ print(f" - {name}: {count} 次引用")
+
+
+def main():
+ print("=" * 70)
+ print("🌟 iText2KG_Star 演示:提取与语义去重")
+ print("=" * 70)
+
+ # 1. 初始化
+ print("\n📦 初始化组件...")
+ llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+ embedder = OpenAIEmbeddings()
+
+ print("🚀 创建 iText2KG_Star 实例...")
+ kg = iText2KG_Star(llm_client=llm, embedder=embedder, verbose=True)
+
+ # 2. 提取 (Phase 1)
+ print(f"\n📄 [Phase 1] 正在处理文本 ({len(EXAMPLE_TEXT)} 字符)...")
+ kg.feed_text(EXAMPLE_TEXT)
+
+ print("\n✅ 提取完成 (原始图谱)")
+ print_stats(kg, "去重前统计")
+
+ # 打印前 10 个节点名称,看看是否有冗余
+ print("\n 🔍 节点快照 (前 10 个):")
+ for n in kg.nodes[:10]:
+ print(f" - {n.name} [{n.label}]")
+
+ # 3. 语义去重 (Phase 2)
+ threshold = 0.85
+ print(f"\n\n🔄 [Phase 2] 执行语义去重 (SemHash, Threshold={threshold})...")
+ print(" 这会将类似 '雷霆之怒' 和 '会长雷霆之怒' 的实体合并。")
+ print(" 正在生成 embeddings 并计算语义相似度...")
+
+ kg.match_nodes_and_update_edges(threshold=threshold)
+
+ print("\n✅ 去重完成")
+ print_stats(kg, "去重后统计")
+
+ # 打印更新后的节点
+ print("\n 🔍 节点快照 (去重后,前 10 个):")
+ for n in kg.nodes[:10]:
+ print(f" - {n.name} [{n.label}]")
+
+ # 4. 展示最终关系
+ print("\n" + "=" * 70)
+ print("🔗 最终关系列表 (前 20 条):")
+ print("=" * 70)
+ for i, edge in enumerate(kg.edges[:20], 1):
+ obs_date = (
+ edge.properties.observation_date
+ if edge.properties and edge.properties.observation_date
+ else "未设置"
+ )
+ print(
+ f" {i:2d}. {edge.startNode.name} --[{edge.name}]--> {edge.endNode.name}"
+ )
+ print(f" └─ 观测日期: {obs_date}")
+
+ # 5. 对比统计
+ print("\n" + "=" * 70)
+ print("📈 去重效果总结")
+ print("=" * 70)
+ print(" ✓ 语义去重成功地合并了相似实体")
+ print(" ✓ 每条边都自动添加了观测日期(提取时间)")
+ print(" ✓ 图谱结构得到了优化和规范化")
+
+ # 保存
+ try:
+ output_dir = "temp/itext2kg_star_demo"
+ kg.dump(output_dir)
+ print(f"\n💾 结果已保存至: {output_dir}/")
+ except Exception as e:
+ print(f"⚠ 保存失败: {e}")
+
+ print("\n" + "=" * 70)
+ print("✨ 演示完成!")
+ print("=" * 70)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/hyperextract/graphs/__init__.py b/hyperextract/graphs/__init__.py
index a505a3e..cd4b7fb 100644
--- a/hyperextract/graphs/__init__.py
+++ b/hyperextract/graphs/__init__.py
@@ -1,6 +1,7 @@
from .base import AutoGraph
from .kg_gen import KG_Gen
from .itext2kg import iText2KG
+from .itext2kg_star import iText2KG_Star
-__all__ = ["AutoGraph", "KG_Gen", "iText2KG"]
+__all__ = ["AutoGraph", "KG_Gen", "iText2KG", "iText2KG_Star"]
diff --git a/hyperextract/graphs/base.py b/hyperextract/graphs/base.py
index 8cd9f83..3dd493d 100644
--- a/hyperextract/graphs/base.py
+++ b/hyperextract/graphs/base.py
@@ -30,6 +30,41 @@ Node = TypeVar("Node", bound=BaseModel)
Edge = TypeVar("Edge", bound=BaseModel)
+# ==============================================================================
+# Default Prompts - Defined outside the class for clarity and reusability
+# ==============================================================================
+
+DEFAULT_GRAPH_PROMPT = (
+ "You are an expert knowledge graph extraction assistant. "
+ "Extract all entities (nodes) and their relationships (edges) from the following text. "
+ "Focus on being comprehensive and capturing the complete knowledge structure.\n\n"
+ "CRITICAL CONSTRAINT: Every edge must connect two nodes that are present in the extracted nodes list. "
+ "Do not create edges between entities that are not explicitly identified as nodes.\n\n"
+ "### Source Text:\n"
+)
+
+DEFAULT_NODE_PROMPT = (
+ "You are an expert information extraction assistant specialized in entity/node recognition. "
+ "Extract ALL relevant entities, concepts, or nodes from the following text with high precision.\n\n"
+ "Focus on:\n"
+ "- Being EXHAUSTIVE: capture all entity types mentioned\n"
+ "- Being PRECISE: extract exact entity names and descriptions\n"
+ "- Clarity: provide clear, concise descriptions for each entity\n\n"
+ "Do not attempt to extract relationships at this stage, only identify entities.\n\n"
+ "### Source Text:\n"
+)
+
+DEFAULT_EDGE_PROMPT = (
+ "You are an expert relationship extraction assistant. "
+ "Extract relationships (edges) between the provided entities.\n\n"
+ "CRITICAL RULES:\n"
+ "1. ONLY extract edges connecting entities from the known entity list below\n"
+ "2. DO NOT invent or hallucinate new entities that are not listed\n"
+ "3. If an entity in the text is not in the known list, DO NOT create edges involving it\n"
+ "4. Focus on explicit relationships mentioned in the text\n\n"
+)
+
+
class AutoGraphSchema(BaseModel, Generic[Node, Edge]):
"""Generic schema container for graph-based knowledge patterns."""
@@ -166,8 +201,8 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
self.edge_fields_for_index = edge_fields_for_index
# Initialize prompts (use custom if provided, otherwise use defaults)
- self.node_prompt = prompt_for_node_extraction or self._default_node_prompt()
- self.edge_prompt = prompt_for_edge_extraction or self._default_edge_prompt()
+ self.node_prompt = prompt_for_node_extraction or DEFAULT_NODE_PROMPT
+ self.edge_prompt = prompt_for_edge_extraction or DEFAULT_EDGE_PROMPT
# Create dynamic GraphSchema containers
graph_schema_name = f"{node_schema.__name__}{edge_schema.__name__}Graph"
@@ -252,7 +287,7 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
def _default_prompt(self) -> str:
"""Returns the default prompt for one-stage graph extraction."""
- return self._default_graph_prompt
+ return DEFAULT_GRAPH_PROMPT
def _create_empty_instance(self) -> "AutoGraph[Node, Edge]":
"""Creates a new empty AutoGraph instance with the same configuration as this one.
@@ -284,55 +319,6 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
edge_fields_for_index=self.edge_fields_for_index, # Persist edge index field configuration
)
- @property
- def _default_graph_prompt(self) -> str:
- """Default prompt for one-stage graph extraction (nodes + edges together).
-
- Emphasizes comprehensive extraction and constraint enforcement.
- """
- return (
- "You are an expert knowledge graph extraction assistant. "
- "Extract all entities (nodes) and their relationships (edges) from the following text. "
- "Focus on being comprehensive and capturing the complete knowledge structure.\n\n"
- "CRITICAL CONSTRAINT: Every edge must connect two nodes that are present in the extracted nodes list. "
- "Do not create edges between entities that are not explicitly identified as nodes.\n\n"
- "### Source Text:\n"
- )
-
- @property
- def _default_node_prompt(self) -> str:
- """Default prompt for two-stage node extraction (Step 1).
-
- Emphasizes exhaustiveness and precision in entity identification.
- """
- return (
- "You are an expert information extraction assistant specialized in entity/node recognition. "
- "Extract ALL relevant entities, concepts, or nodes from the following text with high precision.\n\n"
- "Focus on:\n"
- "- Being EXHAUSTIVE: capture all entity types mentioned\n"
- "- Being PRECISE: extract exact entity names and descriptions\n"
- "- Clarity: provide clear, concise descriptions for each entity\n\n"
- "Do not attempt to extract relationships at this stage, only identify entities.\n\n"
- "### Source Text:\n"
- )
-
- @property
- def _default_edge_prompt(self) -> str:
- """Default prompt for two-stage edge extraction (Step 2).
-
- Emphasizes strict validation: only extract edges where BOTH endpoints exist in provided entities.
- Includes warnings about hallucination prevention.
- """
- return (
- "You are an expert relationship extraction assistant. "
- "Extract relationships (edges) between the provided entities.\n\n"
- "CRITICAL RULES:\n"
- "1. ONLY extract edges connecting entities from the known entity list below\n"
- "2. DO NOT invent or hallucinate new entities that are not listed\n"
- "3. If an entity in the text is not in the known list, DO NOT create edges involving it\n"
- "4. Focus on explicit relationships mentioned in the text\n\n"
- )
-
@property
def data(self) -> AutoGraphSchema[Node, Edge]:
"""Returns the current graph state (nodes and edges).
@@ -544,7 +530,7 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
inputs.append({"chunk_text": chunk, "node_context": node_context})
prompt_template = ChatPromptTemplate.from_template(
- f"{self.edge_prompt}{{node_context}}\n\n### Text Chunk:\n{{chunk_text}}"
+ f"{self.edge_prompt}{{node_context}}\n\n### Source Text:\n{{chunk_text}}"
)
llm_chain = prompt_template | self.llm_client.with_structured_output(
self.edge_list_schema
@@ -597,7 +583,8 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
def merge_batch_data(
self,
- data_list_or_tuple: List[AutoGraphSchema] | Tuple[List[List[Node]], List[List[Edge]]],
+ data_list_or_tuple: List[AutoGraphSchema]
+ | Tuple[List[List[Node]], List[List[Edge]]],
) -> AutoGraphSchema:
"""Merge multiple graphs or node/edge tuples into one.
@@ -606,7 +593,7 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
- Tuple of (List[List[Node]], List[List[Edge]]) (optimization for batch processing)
Args:
- data_list_or_tuple: Either a list of AutoGraphSchema objects or a tuple of
+ data_list_or_tuple: Either a list of AutoGraphSchema objects or a tuple of
(nodes_lists, edges_lists) where each list contains items from multiple chunks.
Returns:
diff --git a/hyperextract/graphs/itext2kg_star.py b/hyperextract/graphs/itext2kg_star.py
index e69de29..abf5a1c 100644
--- a/hyperextract/graphs/itext2kg_star.py
+++ b/hyperextract/graphs/itext2kg_star.py
@@ -0,0 +1,341 @@
+"""iText2KG_Star implementation using AutoGraph as the core engine.
+
+This module provides a specialized implementation of the iText2KG_Star algorithm
+using the AutoGraph framework, designed for extracting high-quality,
+standardized triple-based knowledge graphs.
+
+Prompts and schemas are adapted from the original iText2KG_Star implementation.
+"""
+
+from typing import Optional, List
+from datetime import datetime
+from pydantic import BaseModel, Field
+from semhash import SemHash
+from langchain_core.language_models import BaseChatModel
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.embeddings import Embeddings
+from ontomem.merger import MergeStrategy
+
+from .base import AutoGraph, AutoGraphSchema, NodeListSchema, EdgeListSchema
+from ..utils.logging import logger
+
+# ==============================================================================
+# 1. Schema Definition - Consistent with original iText2KG_Star implementation
+# ==============================================================================
+
+
+class NodeSchema(BaseModel):
+ """Knowledge Graph Node: Standard node structure"""
+
+ label: str = Field(
+ description=(
+ "The semantic category of the entity (e.g., 'Person', 'Event', 'Location', 'Methodology', 'Position'). "
+ "Use 'Relationship' objects if the concept is inherently relational or verbal (e.g., 'plans'). "
+ "Prefer consistent, single-word categories where possible (e.g., 'Person', not 'Person_Entity'). "
+ )
+ )
+ name: str = Field(
+ description=(
+ "The unique name or title identifying this entity, representing exactly one concept. "
+ "For example, 'Yassir', 'CEO', or 'X'. Avoid combining multiple concepts (e.g., 'CEO of X'), "
+ "since linking them should be done via Relationship objects. "
+ "Verbs or multi-concept phrases (e.g., 'plans an escape') typically belong in Relationship objects. "
+ )
+ )
+
+
+class EdgeProperties(BaseModel):
+ """Properties for Knowledge Graph Edge"""
+
+ observation_date: Optional[str] = Field(
+ default=None,
+ description=(
+ "The date when the relationship was observed or recorded. "
+ "This field is populated manually by the user after extraction. "
+ "If multiple observation dates exist, use the latest date."
+ ),
+ )
+
+
+class EdgeSchema(BaseModel):
+ """Knowledge Graph Edge: Standard edge structure representing relationships between nodes"""
+
+ startNode: NodeSchema = Field(
+ description=(
+ "The 'subject' or source entity of this relationship, which must appear in the known entities."
+ )
+ )
+ endNode: NodeSchema = Field(
+ description=(
+ "The 'object' or target entity of this relationship, which must also appear in the known entities."
+ )
+ )
+ name: str = Field(
+ description=(
+ "A single, canonical predicate capturing how the startNode and endNode relate (e.g., 'is_CEO', "
+ "'holds_position', 'located_in'). Avoid compound verbs (e.g., 'plans_and_executes'). "
+ "AVOID relation names as prepositions 'of', 'in' or similar."
+ )
+ )
+ properties: EdgeProperties = Field(
+ description="Additional properties describing the relationship."
+ )
+
+
+# ==============================================================================
+# 2. Prompt Definition - Adapted from original iText2KG_Star implementation
+# ==============================================================================
+
+
+iText2KG_Star_EDGE_EXTRACTION_PROMPT = """
+# DIRECTIVES :
+- Extract all meaningful relationships directly from the provided context.
+- For each relationship, identify the start entity and end entity involved.
+- Each entity should have a clear name and appropriate label/type.
+- Avoid reflexive relations (entity relating to itself).
+- IMPORTANT: Set 'observation_date' to null in all cases. Do NOT extract or infer dates from the context.
+- The observation_date field should be populated by the user manually after extraction.
+"""
+
+
+# ==============================================================================
+# 3. iText2KG_Star Implementation - Inherits from AutoGraph
+# ==============================================================================
+
+
+class iText2KG_Star(AutoGraph[NodeSchema, EdgeSchema]):
+ """
+ iText2KG_Star: A specialized AutoGraph for extracting high-quality triple-based KGs.
+
+ Features:
+ - One-stage extraction: Extracts edges directly, then derives nodes automatically
+ - Customized prompts from original iText2KG_Star implementation
+ - Semantic Deduplication: Includes `match_nodes_and_update_edges` using SemHash with embeddings
+ - Automatic date tracking: Observation date set to extraction time
+ - Nested node schema: Maintains rich semantic information (name + label)
+
+ Example:
+ >>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+ >>> llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+ >>> embedder = OpenAIEmbeddings()
+ >>> kg = iText2KG_Star(llm_client=llm, embedder=embedder)
+
+ >>> # 1. Extract relationships and derive nodes
+ >>> kg.feed_text("Elon Musk is the CEO of SpaceX. Musk leads Tesla.")
+ >>> print(f"Before dedup - Nodes: {len(kg.nodes)}, Edges: {len(kg.edges)}")
+
+ >>> # 2. Deduplicate: Merges 'Elon Musk' and 'Musk'
+ >>> kg.match_nodes_and_update_edges(threshold=0.85)
+ >>> print(f"After dedup - Nodes: {len(kg.nodes)}, Edges: {len(kg.edges)}")
+ """
+
+ def __init__(
+ self,
+ llm_client: BaseChatModel,
+ embedder: Embeddings,
+ observation_date: str | None = None,
+ chunk_size: int = 2000,
+ chunk_overlap: int = 200,
+ max_workers: int = 10,
+ verbose: bool = True,
+ ):
+ """Initialize iText2KG_Star.
+
+ Args:
+ llm_client: Language model for extraction
+ embedder: Embedding model for vector indexing
+ observation_date: Date when the extraction was performed, like '1997-10-10' or '1997-10-10 23:59:59'.
+ If None, uses current date and time.
+ chunk_size: Characters per chunk
+ chunk_overlap: Overlapping characters between chunks
+ max_workers: Max concurrent extraction workers
+ verbose: Display detailed execution logs and progress information
+ """
+
+ self.observation_date = observation_date
+
+ # 1. Define Key Extractors (critical for deduplication)
+ # Node deduplication: exact match by name
+ node_key_fn = lambda x: x.name
+
+ # Edge deduplication: combination of subject-predicate-object triple
+ edge_key_fn = lambda x: f"{x.startNode.name}|{x.name}|{x.endNode.name}"
+
+ # 2. Edge consistency check: tell AutoGraph which nodes this edge connects
+ nodes_in_edge_fn = lambda x: (x.startNode.name, x.endNode.name)
+
+ # 3. Call parent class initialization
+ logger.info("🚀 Initializing iText2KG_Star")
+ super().__init__(
+ node_schema=NodeSchema,
+ edge_schema=EdgeSchema,
+ node_key_extractor=node_key_fn,
+ edge_key_extractor=edge_key_fn,
+ nodes_in_edge_extractor=nodes_in_edge_fn,
+ llm_client=llm_client,
+ embedder=embedder,
+ # Enforce one-stage extraction (first extract edges, then derive nodes)
+ extraction_mode="one_stage",
+ # Inject customized prompts
+ prompt_for_edge_extraction=iText2KG_Star_EDGE_EXTRACTION_PROMPT,
+ # Configure deduplication strategy
+ node_strategy_or_merger=MergeStrategy.KEEP_EXISTING,
+ edge_strategy_or_merger=MergeStrategy.KEEP_EXISTING,
+ # Optimize indexing: only index name field
+ node_fields_for_index=["name", "label"],
+ edge_fields_for_index=["startNode", "name", "endNode"],
+ # Other parameters
+ chunk_size=chunk_size,
+ chunk_overlap=chunk_overlap,
+ max_workers=max_workers,
+ verbose=verbose,
+ )
+
+ # ==================== Extraction Pipeline ====================
+
+ def _extract_data(self, text: str) -> AutoGraphSchema:
+ """Extract edges directly, then derive nodes (One-stage extraction).
+
+ Process:
+ 1. Split text into chunks.
+ 2. Batch extract edges directly from chunks (Edges contain node info).
+ 3. Post-process: Set observation date to current date and time.
+ 4. Derive nodes from edges.
+ 5. Merge all partial graphs into one global graph.
+
+ Args:
+ text: Input text.
+
+ Returns:
+ Extracted and validated graph.
+ """
+ # 1. Prepare chunks
+ if len(text) <= self.chunk_size:
+ chunks = [text]
+ else:
+ chunks = self.text_splitter.split_text(text)
+
+ # 2. Batch Extract Edges directly
+ prompt_template = ChatPromptTemplate.from_template(
+ f"{self.edge_prompt}\n\n### Source Text:\n{{chunk_text}}"
+ )
+ llm_chain = prompt_template | self.llm_client.with_structured_output(
+ self.edge_list_schema
+ )
+
+ inputs = [{"chunk_text": chunk} for chunk in chunks]
+ chunk_edge_lists = llm_chain.batch(
+ inputs, config={"max_concurrency": self.max_workers}
+ )
+
+ # 3. Post-process: Derive Nodes & Set Observation Date
+ all_nodes_lists, all_edges_lists = [], []
+
+ for edge_list in chunk_edge_lists:
+ derived_nodes, processed_edges = [], []
+
+ if edge_list and edge_list.items:
+ for edge in edge_list.items:
+ # Post-process: Set observation date to current extraction time
+ edge: EdgeSchema
+ observation_date = self.observation_date or datetime.now().strftime(
+ "%Y-%m-%d %H:%M:%S"
+ )
+ edge.properties.observation_date = observation_date
+
+ # Derive nodes from edges
+ derived_nodes.append(edge.startNode)
+ derived_nodes.append(edge.endNode)
+ processed_edges.append(edge)
+
+ all_nodes_lists.append(derived_nodes)
+ all_edges_lists.append(processed_edges)
+
+ # 4. Construct Partial Graphs (Tuple format for merge optimization)
+ partial_graphs = (all_nodes_lists, all_edges_lists)
+
+ # 5. Global Merge (passes tuples to merge_batch_data)
+ raw_graph = self.merge_batch_data(partial_graphs)
+
+ # Prune dangling edges to ensure graph consistency
+ return self._prune_dangling_edges(raw_graph)
+
+ # ==================== Node Deduplication ====================
+
+ def match_nodes_and_update_edges(self, threshold: float = 0.8) -> "iText2KG_Star":
+ """Match nodes in the graph and update edges accordingly using SemHash with embeddings.
+
+ This method identifies and merges similar nodes based on semantic similarity
+ using embeddings from the instance's embedder. It updates edges to reflect
+ any changes in node identities.
+
+ Args:
+ threshold: Similarity threshold for matching nodes (0.0 to 1.0). Defaults to 0.8.
+
+ Returns:
+ The updated iText2KG_Star instance with matched nodes and updated edges.
+ """
+ logger.info(
+ f"🔄 Starting node matching and edge update (threshold={threshold})..."
+ )
+
+ nodes, edges = self.nodes, self.edges
+
+ if not nodes:
+ logger.warning("⚠️ No nodes to match; skipping matching.")
+ return self
+
+ # 1. Generate Embeddings using self.embedder.embed_documents
+ node_names = [n.name for n in nodes]
+ logger.info(f"📊 Generating embeddings for {len(node_names)} nodes...")
+
+ try:
+ embeddings = self.embedder.embed_documents(node_names)
+ except Exception as e:
+ logger.error(f"❌ Failed to generate embeddings: {e}")
+ return self
+
+ # 2. Initialize SemHash from embeddings
+ semhash = SemHash.from_embeddings(
+ embeddings=embeddings,
+ records=node_names,
+ model=None, # Embeddings already provided
+ )
+
+ # 3. Run Deduplication
+ result = semhash.self_deduplicate(threshold=threshold)
+
+ # 4. Build Mapping from duplicates result
+ mapping = {} # old_name -> new_name
+ for record in result.filtered:
+ if record.duplicates and len(record.duplicates) > 0:
+ # Map the record to the first duplicate (representative)
+ mapping[record.record] = record.duplicates[0][0]
+
+ if not mapping:
+ logger.info("✅ No similar nodes found above threshold.")
+ return self
+
+ logger.info(f"🔗 Found {len(mapping)} nodes to merge.")
+
+ # 5. Apply mapping to graph data
+ # Update Node names
+ for node in nodes:
+ if node.name in mapping:
+ node.name = mapping[node.name]
+
+ # Update Edge references (startNode and endNode are nested NodeSchema objects)
+ for edge in edges:
+ if edge.startNode.name in mapping:
+ edge.startNode.name = mapping[edge.startNode.name]
+ if edge.endNode.name in mapping:
+ edge.endNode.name = mapping[edge.endNode.name]
+
+ # 6. Update internal state
+ new_data = AutoGraphSchema(nodes=nodes, edges=edges)
+ self._set_data_state(new_data)
+
+ logger.info(
+ f"✅ Node matching complete: Nodes {len(nodes)} -> {len(self.nodes)}, Edges: {len(edges)}"
+ )
+ return self
diff --git a/hyperextract/graphs/kg_gen.py b/hyperextract/graphs/kg_gen.py
index 296eff2..1e91899 100644
--- a/hyperextract/graphs/kg_gen.py
+++ b/hyperextract/graphs/kg_gen.py
@@ -27,7 +27,6 @@ class NodeSchema(BaseModel):
"""Knowledge Graph Node: Minimal entity with only a name field"""
name: str = Field(
- ...,
description="The unique name or identifier of the entity. Must be thorough and accurate to the source text.",
)
@@ -36,15 +35,12 @@ class EdgeSchema(BaseModel):
"""Knowledge Graph Edge: Standard triple structure (Subject-Predicate-Object)"""
subject: str = Field(
- ...,
description="Subject entity name. Must be an exact match from the extracted entities list.",
)
predicate: str = Field(
- ...,
description="The relationship/predicate between subject and object. Should be concise and descriptive.",
)
object: str = Field(
- ...,
description="Object entity name. Must be an exact match from the extracted entities list.",
)
@@ -168,12 +164,12 @@ class KG_Gen(AutoGraph[NodeSchema, EdgeSchema]):
model: str | SentenceTransformer = "sentence-transformers/all-MiniLM-L6-v2",
):
"""Internal helper to apply SemHash deduplication on a graph data object.
-
+
Args:
graph_data: The graph data object (nodes/edges) to process in-place.
threshold: SemHash similarity threshold (0.0 to 1.0).
model: Embedding model for SemHash.
-
+
Returns:
The modified graph_data object.
"""
diff --git a/hyperextract/hypergraphs/base.py b/hyperextract/hypergraphs/base.py
index d733b52..6c92bc4 100644
--- a/hyperextract/hypergraphs/base.py
+++ b/hyperextract/hypergraphs/base.py
@@ -460,7 +460,7 @@ class AutoHypergraph(
inputs.append({"chunk_text": chunk, "node_context": node_context})
prompt_template = ChatPromptTemplate.from_template(
- f"{self.edge_prompt}{{node_context}}\n\n### Text Chunk:\n{{chunk_text}}"
+ f"{self.edge_prompt}{{node_context}}\n\n### Source Text:\n{{chunk_text}}"
)
llm_chain = prompt_template | self.llm_client.with_structured_output(
self.edge_list_schema
diff --git a/hyperextract/hypergraphs/hyper_rag.py b/hyperextract/hypergraphs/hyper_rag.py
index 81a59c8..a37be29 100644
--- a/hyperextract/hypergraphs/hyper_rag.py
+++ b/hyperextract/hypergraphs/hyper_rag.py
@@ -18,13 +18,11 @@ from ontomem.merger import CustomRuleMerger
class NodeSchema(BaseModel):
"""Represents an entity extracted from the source text."""
- name: str = Field(..., description="Name of the entity, capitalized if in English")
+ name: str = Field(..., description="Name of the entity")
type: str = Field(
- ...,
description="Entity type (person, organization, geo, event, role, concept, etc.)",
)
description: str = Field(
- ...,
description="Comprehensive description of the entity's attributes and activities",
)
@@ -38,17 +36,15 @@ class EdgeSchema(BaseModel):
"""Represents a relationship or event connecting multiple entities (low-order or high-order)."""
participants: List[str] = Field(
- ..., description="Names of entities involved in this relationship"
+ description="Names of entities involved in this relationship"
)
description: str = Field(
- ..., description="Detailed explanation of the relationship or event"
+ description="Detailed explanation of the relationship or event"
)
keywords: List[str] = Field(
- ...,
description="List of keywords summarizing the relationship themes (e.g., ['conflict', 'trade', 'alliance'])",
)
strength: int = Field(
- ...,
ge=1,
le=10,
description="Numerical score indicating relationship strength (1-10)",
@@ -87,7 +83,7 @@ It treats all involved entities as **participants** in a shared context
# Merge Templates for LLM.CUSTOM_RULE Strategy
# ============================================================================
-NODE_MERGE_TEMPLATE = """You are an intelligent data merging assistant.
+NODE_MERGE_RULE = """You are an intelligent data merging assistant.
You will receive a list of objects representing the same Entity.
Your task is to merge them into a SINGLE object exactly matching the schema.
@@ -97,7 +93,7 @@ Merge strategy:
2. **description**: Synthesize a single, comprehensive description containing all unique details from the input descriptions. Write it in the third person. Resolve any contradictions coherently.
"""
-EDGE_MERGE_TEMPLATE = """You are an intelligent data merging assistant.
+EDGE_MERGE_RULE = """You are an intelligent data merging assistant.
You will receive a list of objects representing the same Hyperedge (Relationship/Event).
Your task is to merge them into a SINGLE object exactly matching the schema.
@@ -163,7 +159,7 @@ class Hyper_RAG(AutoHypergraph[NodeSchema, EdgeSchema]):
key_extractor=node_key_fn,
llm_client=llm_client,
item_schema=NodeSchema,
- rule=NODE_MERGE_TEMPLATE,
+ rule=NODE_MERGE_RULE,
)
# Edge merger: merges relationship descriptions using EDGE_MERGE_TEMPLATE
@@ -171,7 +167,7 @@ class Hyper_RAG(AutoHypergraph[NodeSchema, EdgeSchema]):
key_extractor=edge_key_fn,
llm_client=llm_client,
item_schema=EdgeSchema,
- rule=EDGE_MERGE_TEMPLATE,
+ rule=EDGE_MERGE_RULE,
)
# 4. Call parent class initialization