Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: 新增 iText2KG_Star 模块及演示,优化知识图谱提取与语义去重功能
yifanfeng97caa0691384b4
diff --git a/examples/itext2kg_star_demo.py b/examples/itext2kg_star_demo.pynew file mode 100644index 0000000..4edb2a2--- /dev/null+++ b/examples/itext2kg_star_demo.py@@ -0,0 +1,291 @@+"""+iText2KG_Star 演示:知识图谱提取与语义去重++这个演示展示了 iText2KG_Star 的高级功能:+1. 单阶段提取:直接从文本中提取关系(三元组),自动推导实体。+2. 语义去重:使用 SemHash 算法合并相似实体(例如 "雷霆之怒" 和 "会长雷霆之怒")。+"""++import os+import sys+from collections import Counter++# 添加项目根目录到路径+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from dotenv import load_dotenv+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++from hyperextract.graphs.itext2kg_star import iText2KG_Star++load_dotenv()++# ==============================================================================+# 示例文本 (同 iText2KG Demo)+# ==============================================================================+EXAMPLE_TEXT = """+【818】星辰阁公会昨晚解散的真相!团长黑装备还骂人!这简直是最大的内讧事件!++楼主:匿名玩家(ID已隐藏)+发帖时间:今天 23:45++我是星辰阁的一个普通成员,昨晚目睹了整个悲剧的发生。我觉得必须把真相告诉大家。++首先介绍一下我们的团队阵容:+- 会长兼主坦克:雷霆之怒(真实ID:张三,40级战士,装备评分9999)+- 副坦克:坚如磐石(真实ID:李四,38级盾战士,评分8800,人很老实)+- 主奶妈(治疗):小甜甜(真实ID:王五,38级圣骑士,评分8600)+- 副奶妈:月光治愈者(真实ID:赵六,35级德鲁伊,评分7500)+- DPS输出1(弓箭手):暗影刺客(真实ID:孙七,39级猎手,评分9200)+- DPS输出2(法师):火球术一级(真实ID:周八,37级法师,评分8400)+- DPS输出3(剑客):风云剑侠(真实ID:吴九,36级剑士,评分7800)+- 远程法师:冰霜法咒(真实ID:郑十,35级冰法,评分7600)++昨晚晚上8点,我们集合在副本入口。这是星辰阁第三次挑战"深渊魔龙"这个最终BOSS。+前两次都失败了,主要是因为配合不够默契。但是昨晚会长雷霆之怒发誓说一定要拿下这个BOSS。++副本开始,一切似乎很顺利。前5个小怪都被我们轻松秒杀。+然后大家来到了BOSS房间——深渊魔龙的巢穴。这是一个超级庞大的紫龙,满血时候气势相当恐怖。++战斗一开始,雷霆之怒(会长)冲上去拉住了仇恨值。坚如磐石(副坦克)站在侧翼进行补充。+小甜甜不断给雷霆之怒刷HOT(持续恢复)。月光治愈者给坚如磐石加盾。+前期看起来非常完美,大家的DPS输出也打得相当漂亮。++但是,大约打了5分钟,问题出现了。++坚如磐石(副坦克)突然掉线了!这太糟糕了。我们在公会频道里看到他的在线状态变成了灰色。+公会会长雷霆之怒看到这一幕,血压瞬间上升(我可以想象他的脸色)。++为什么副坦会掉线呢?大家后来才明白——坚如磐石的家里停电了。这真的是太不走运了。++没有副坦克的补充,所有的伤害都集中在了雷霆之怒身上。+但是这个时候,事情开始变得诡异。++小甜甜(主奶妈)本应该全力治疗会长,但是我通过伤害统计面板发现,她的治疗量明显不足。+更奇怪的是,她把大量的治疗都给了暗影刺客(弓箭手)。++这很不正常。暗影刺客作为DPS,血量本不应该这么低。我仔细看了一下暗影刺客的走位——+哎哟,这哥们为了最大化伤害输出,站位超级靠前,几乎要贴到BOSS身上了。+结果BOSS的爪子扫过来,一下子就把暗影刺客的血条扫到了20%以下。++关键时刻,小甜甜却把大治疗术给了暗影刺客。这让会长雷霆之怒的血量开始急速下降。++我当时就在YY频道里喊:"小甜甜,快给会长加血!"++月光治愈者也在拼命给会长加盾,但是她的输出能力有限,根本抵挡不住BOSS的伤害。++而且我还发现,月光治愈者的治疗也有一部分莫名其妙地给了其他人。看起来公会里有人在划水。++等等,我再看一下。哦天哪,火球术一级(法师)和冰霜法咒在BOSS的AoE范围内站了太长时间,+导致他们频繁掉血。但是,更夸张的是,暗影刺客和小甜甜之间似乎有什么默契——+每当暗影刺客掉血,小甜甜就立马给他加。++这时候,我想到了一个可能性。我在游戏论坛里看过一些帖子,有人提到暗影刺客和小甜甜好像是现实中的情侣!++妈呀,这下问题大了。在这么关键的时刻,小甜甜竟然偏心给她的男朋友加血,而忽视了作为团队大脑的会长!++大约在15分钟的时候,BOSS放出了一个大招——"魔龙吐息"。+这个技能造成的伤害足以秒杀任何一个DPS。但是坦克(雷霆之怒)应该能扛住。++可是,小甜甜的治疗没有及时跟上。雷霆之怒的血条瞬间掉到了0。++会长倒地了!++随后,失去了坦克的护佑,暗影刺客也被BOSS的普通攻击打成了残血,然后被秒杀。+紧接着是风云剑侠,然后是火球术一级。整个团队在短短30秒内全灭了。++副本失败。大家都复活在了副本入口。++这个时候,雷霆之怒(会长)的脾气彻底炸裂了。++他在YY频道里破口大骂:"小甜甜,你是干什么呢?!你是团队奶妈,不是暗影刺客的私人医生!+治疗职业就是要保坦克,你这样的治疗我见过,但从没见过这么烂的!"++小甜甜试图辩解:"我……我只是……"++雷霆之怒直接打断她:"别给我狡辩!我决定了,你从今天起被踢出公会!+而且暗影刺客这个摸鱼的家伙,我也要把他赶出去。两个人都别在星辰阁待着!"++但就在这个时候,事情发生了急转弯。++暗影刺客一句话都没说,只见他的角色突然冲向了BOSS房间。+而就在这时,副本的第二个阶段秘密掉落物品——传说级别的武器"魔龙之牙"出现了!++这是一把超级稀有的武器,全服务器可能只有两三把。价值高达几百万金币。++没等任何人反应过来,暗影刺客就冲上去,一把拿起了"魔龙之牙"!++我们都懵了。"魔龙之牙"本来应该由队长(雷霆之怒)来分配,决定给谁。+但是暗影刺客直接黑吃黑,把这把传说武器直接塞进了自己的背包里!++会长还在语音里骂人,根本没注意到这一幕。++紧接着,更狠的来了——暗影刺客在游戏里按下了"退出公会"的按钮。+他直接,光速,毫不犹豫地退出了星辰阁!++然后他的角色在世界频道发了一句话:"拜拜,各位。我去找个真正尊重我的公会。"++说完,这哥们直接下线了。++我们的公会频道顿时炸锅了。++小甜甜也没做任何辩解,直接退出了公会。她甚至没有说一句话。++会长雷霆之怒这时候才意识到发生了什么。他看到了掉落物品的分配记录——+"魔龙之牙"被暗影刺客拿走了,而不是团队共有。++他疯了。直接在世界频道开始刷屏:+"各位玩家,我要举报星辰阁的暗影刺客和小甜甜这两条狗!他们合谋黑装备!+不要信任他们!全服通缉这两个垃圾!"++火球术一级(法师)现在也开始助威。他在世界频道里连发了十多条信息:+"赞同!这两个人太过分了!狗男女!不要和他们做交易!"++"冰霜法咒在副本里的表现也不太理想,看起来他好像和那对情侣有什么关系,+因为我注意到在暗影刺客掉线的时候,冰霜法咒也莫名其妙地不进行输出了。"++有人开始怀疑冰霜法咒是不是也是那对情侣的朋友。++坚如磐石(副坦)这时候才上线。他看到了聊天记录,一脸懵逼。+他在公会频道里问:"发生什么了?我家里停电了……"++没有人回答他。因为大家都在看这场大戏。++月光治愈者试图调和一下,她说:"大家冷静……这件事可能有误会……"++但是没有人听她的。因为这个时候,世界频道上的玩家也开始评论这件事。++有人说:"这就是为什么我不玩大公会。大公会水太深。"++还有人说:"我早就听说暗影刺客这个人在游戏里名声不太好。多次被指控在副本里贪图装备。"++有人挖出了暗影刺客的历史记录。原来他曾经在另一个公会里,也因为类似的事情被踢出来过。++这哥们简直就是个"装备贪心鬼"。++风云剑侠开始反思:"也许我们团队的问题不只是小甜甜。我们可能选错人了。"++火球术一级继续在世界频道里刷屏:"各位,我建议大家抵制这两个玩家。不要和他们一个队。"++这时候,一个名字叫做"游戏管理员001"的官方账号出现了。+他说:"我们收到了关于玩家暗影刺客的举报。我们正在调查这件事。如果确认违规,将给予封号处理。"++暗影刺客的名字从世界频道消失了——他已经下线了。而且下线之前把"魔龙之牙"从背包里转移到了仓库。+没人知道他把这把武器藏在了哪里。++最后,星辰阁的会长雷霆之怒在公会频道里宣布:+"各位成员,我决定了。星辰阁正式解散。这个公会不适合我继续管理。+我会转移到一个小公会,和志同道合的朋友重新开始。"++很多成员试图挽留他。但是他的决心已定。++现在,星辰阁已经成为了一个"死公会"。只有一些离不开这个ID的成员还在。++这就是昨晚发生的一切。我作为一个旁观者,见证了这场游戏江湖的大戏。++最后的最后,我想说:游戏就是游戏。团队合作最重要。装备是身外之物。+但是,为了装备而背叛队友,这绝对是游戏玩家中最可耻的行为。++希望游戏官方能够严肃处理暗影刺客的违规行为。+也希望星辰阁的所有成员能够吸取教训,在各自的新公会里找到快乐。+"""+++def print_stats(kg, title: str):+ """打印图谱统计信息"""+ print("\n" + "-" * 60)+ print(f"📊 {title}")+ print("-" * 60)+ print(f" 🔹 节点数量: {len(kg.nodes)}")+ print(f" 🔹 关系数量: {len(kg.edges)}")++ # 列出 Top 5 节点 (按度数)+ entity_degree = Counter()+ for edge in kg.edges:+ entity_degree[edge.startNode.name] += 1+ entity_degree[edge.endNode.name] += 1++ print("\n 🏆 活跃实体 (Top 5):")+ for name, count in entity_degree.most_common(5):+ print(f" - {name}: {count} 次引用")+++def main():+ print("=" * 70)+ print("🌟 iText2KG_Star 演示:提取与语义去重")+ print("=" * 70)++ # 1. 初始化+ print("\n📦 初始化组件...")+ llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+ embedder = OpenAIEmbeddings()++ print("🚀 创建 iText2KG_Star 实例...")+ kg = iText2KG_Star(llm_client=llm, embedder=embedder, verbose=True)++ # 2. 提取 (Phase 1)+ print(f"\n📄 [Phase 1] 正在处理文本 ({len(EXAMPLE_TEXT)} 字符)...")+ kg.feed_text(EXAMPLE_TEXT)++ print("\n✅ 提取完成 (原始图谱)")+ print_stats(kg, "去重前统计")++ # 打印前 10 个节点名称,看看是否有冗余+ print("\n 🔍 节点快照 (前 10 个):")+ for n in kg.nodes[:10]:+ print(f" - {n.name} [{n.label}]")++ # 3. 语义去重 (Phase 2)+ threshold = 0.85+ print(f"\n\n🔄 [Phase 2] 执行语义去重 (SemHash, Threshold={threshold})...")+ print(" 这会将类似 '雷霆之怒' 和 '会长雷霆之怒' 的实体合并。")+ print(" 正在生成 embeddings 并计算语义相似度...")++ kg.match_nodes_and_update_edges(threshold=threshold)++ print("\n✅ 去重完成")+ print_stats(kg, "去重后统计")++ # 打印更新后的节点+ print("\n 🔍 节点快照 (去重后,前 10 个):")+ for n in kg.nodes[:10]:+ print(f" - {n.name} [{n.label}]")++ # 4. 展示最终关系+ print("\n" + "=" * 70)+ print("🔗 最终关系列表 (前 20 条):")+ print("=" * 70)+ for i, edge in enumerate(kg.edges[:20], 1):+ obs_date = (+ edge.properties.observation_date+ if edge.properties and edge.properties.observation_date+ else "未设置"+ )+ print(+ f" {i:2d}. {edge.startNode.name} --[{edge.name}]--> {edge.endNode.name}"+ )+ print(f" └─ 观测日期: {obs_date}")++ # 5. 对比统计+ print("\n" + "=" * 70)+ print("📈 去重效果总结")+ print("=" * 70)+ print(" ✓ 语义去重成功地合并了相似实体")+ print(" ✓ 每条边都自动添加了观测日期(提取时间)")+ print(" ✓ 图谱结构得到了优化和规范化")++ # 保存+ try:+ output_dir = "temp/itext2kg_star_demo"+ kg.dump(output_dir)+ print(f"\n💾 结果已保存至: {output_dir}/")+ except Exception as e:+ print(f"⚠ 保存失败: {e}")++ print("\n" + "=" * 70)+ print("✨ 演示完成!")+ print("=" * 70)+++if __name__ == "__main__":+ main()diff --git a/hyperextract/graphs/__init__.py b/hyperextract/graphs/__init__.pyindex a505a3e..cd4b7fb 100644--- a/hyperextract/graphs/__init__.py+++ b/hyperextract/graphs/__init__.py@@ -1,6 +1,7 @@from .base import AutoGraphfrom .kg_gen import KG_Genfrom .itext2kg import iText2KG+from .itext2kg_star import iText2KG_Star-__all__ = ["AutoGraph", "KG_Gen", "iText2KG"]+__all__ = ["AutoGraph", "KG_Gen", "iText2KG", "iText2KG_Star"]diff --git a/hyperextract/graphs/base.py b/hyperextract/graphs/base.pyindex 8cd9f83..3dd493d 100644--- a/hyperextract/graphs/base.py+++ b/hyperextract/graphs/base.py@@ -30,6 +30,41 @@ Node = TypeVar("Node", bound=BaseModel)Edge = TypeVar("Edge", bound=BaseModel)+# ==============================================================================+# Default Prompts - Defined outside the class for clarity and reusability+# ==============================================================================++DEFAULT_GRAPH_PROMPT = (+ "You are an expert knowledge graph extraction assistant. "+ "Extract all entities (nodes) and their relationships (edges) from the following text. "+ "Focus on being comprehensive and capturing the complete knowledge structure.\n\n"+ "CRITICAL CONSTRAINT: Every edge must connect two nodes that are present in the extracted nodes list. "+ "Do not create edges between entities that are not explicitly identified as nodes.\n\n"+ "### Source Text:\n"+)++DEFAULT_NODE_PROMPT = (+ "You are an expert information extraction assistant specialized in entity/node recognition. "+ "Extract ALL relevant entities, concepts, or nodes from the following text with high precision.\n\n"+ "Focus on:\n"+ "- Being EXHAUSTIVE: capture all entity types mentioned\n"+ "- Being PRECISE: extract exact entity names and descriptions\n"+ "- Clarity: provide clear, concise descriptions for each entity\n\n"+ "Do not attempt to extract relationships at this stage, only identify entities.\n\n"+ "### Source Text:\n"+)++DEFAULT_EDGE_PROMPT = (+ "You are an expert relationship extraction assistant. "+ "Extract relationships (edges) between the provided entities.\n\n"+ "CRITICAL RULES:\n"+ "1. ONLY extract edges connecting entities from the known entity list below\n"+ "2. DO NOT invent or hallucinate new entities that are not listed\n"+ "3. If an entity in the text is not in the known list, DO NOT create edges involving it\n"+ "4. Focus on explicit relationships mentioned in the text\n\n"+)++class AutoGraphSchema(BaseModel, Generic[Node, Edge]):"""Generic schema container for graph-based knowledge patterns."""@@ -166,8 +201,8 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):self.edge_fields_for_index = edge_fields_for_index# Initialize prompts (use custom if provided, otherwise use defaults)- self.node_prompt = prompt_for_node_extraction or self._default_node_prompt()- self.edge_prompt = prompt_for_edge_extraction or self._default_edge_prompt()+ self.node_prompt = prompt_for_node_extraction or DEFAULT_NODE_PROMPT+ self.edge_prompt = prompt_for_edge_extraction or DEFAULT_EDGE_PROMPT# Create dynamic GraphSchema containersgraph_schema_name = f"{node_schema.__name__}{edge_schema.__name__}Graph"@@ -252,7 +287,7 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):def _default_prompt(self) -> str:"""Returns the default prompt for one-stage graph extraction."""- return self._default_graph_prompt+ return DEFAULT_GRAPH_PROMPTdef _create_empty_instance(self) -> "AutoGraph[Node, Edge]":"""Creates a new empty AutoGraph instance with the same configuration as this one.@@ -284,55 +319,6 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):edge_fields_for_index=self.edge_fields_for_index, # Persist edge index field configuration)- @property- def _default_graph_prompt(self) -> str:- """Default prompt for one-stage graph extraction (nodes + edges together).-- Emphasizes comprehensive extraction and constraint enforcement.- """- return (- "You are an expert knowledge graph extraction assistant. "- "Extract all entities (nodes) and their relationships (edges) from the following text. "- "Focus on being comprehensive and capturing the complete knowledge structure.\n\n"- "CRITICAL CONSTRAINT: Every edge must connect two nodes that are present in the extracted nodes list. "- "Do not create edges between entities that are not explicitly identified as nodes.\n\n"- "### Source Text:\n"- )-- @property- def _default_node_prompt(self) -> str:- """Default prompt for two-stage node extraction (Step 1).-- Emphasizes exhaustiveness and precision in entity identification.- """- return (- "You are an expert information extraction assistant specialized in entity/node recognition. "- "Extract ALL relevant entities, concepts, or nodes from the following text with high precision.\n\n"- "Focus on:\n"- "- Being EXHAUSTIVE: capture all entity types mentioned\n"- "- Being PRECISE: extract exact entity names and descriptions\n"- "- Clarity: provide clear, concise descriptions for each entity\n\n"- "Do not attempt to extract relationships at this stage, only identify entities.\n\n"- "### Source Text:\n"- )-- @property- def _default_edge_prompt(self) -> str:- """Default prompt for two-stage edge extraction (Step 2).-- Emphasizes strict validation: only extract edges where BOTH endpoints exist in provided entities.- Includes warnings about hallucination prevention.- """- return (- "You are an expert relationship extraction assistant. "- "Extract relationships (edges) between the provided entities.\n\n"- "CRITICAL RULES:\n"- "1. ONLY extract edges connecting entities from the known entity list below\n"- "2. DO NOT invent or hallucinate new entities that are not listed\n"- "3. If an entity in the text is not in the known list, DO NOT create edges involving it\n"- "4. Focus on explicit relationships mentioned in the text\n\n"- )-@propertydef data(self) -> AutoGraphSchema[Node, Edge]:"""Returns the current graph state (nodes and edges).@@ -544,7 +530,7 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):inputs.append({"chunk_text": chunk, "node_context": node_context})prompt_template = ChatPromptTemplate.from_template(- f"{self.edge_prompt}{{node_context}}\n\n### Text Chunk:\n{{chunk_text}}"+ f"{self.edge_prompt}{{node_context}}\n\n### Source Text:\n{{chunk_text}}")llm_chain = prompt_template | self.llm_client.with_structured_output(self.edge_list_schema@@ -597,7 +583,8 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):def merge_batch_data(self,- data_list_or_tuple: List[AutoGraphSchema] | Tuple[List[List[Node]], List[List[Edge]]],+ data_list_or_tuple: List[AutoGraphSchema]+ | Tuple[List[List[Node]], List[List[Edge]]],) -> AutoGraphSchema:"""Merge multiple graphs or node/edge tuples into one.@@ -606,7 +593,7 @@ class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):- Tuple of (List[List[Node]], List[List[Edge]]) (optimization for batch processing)Args:- data_list_or_tuple: Either a list of AutoGraphSchema objects or a tuple of+ data_list_or_tuple: Either a list of AutoGraphSchema objects or a tuple of(nodes_lists, edges_lists) where each list contains items from multiple chunks.Returns:diff --git a/hyperextract/graphs/itext2kg_star.py b/hyperextract/graphs/itext2kg_star.pyindex e69de29..abf5a1c 100644--- a/hyperextract/graphs/itext2kg_star.py+++ b/hyperextract/graphs/itext2kg_star.py@@ -0,0 +1,341 @@+"""iText2KG_Star implementation using AutoGraph as the core engine.++This module provides a specialized implementation of the iText2KG_Star algorithm+using the AutoGraph framework, designed for extracting high-quality,+standardized triple-based knowledge graphs.++Prompts and schemas are adapted from the original iText2KG_Star implementation.+"""++from typing import Optional, List+from datetime import datetime+from pydantic import BaseModel, Field+from semhash import SemHash+from langchain_core.language_models import BaseChatModel+from langchain_core.prompts import ChatPromptTemplate+from langchain_core.embeddings import Embeddings+from ontomem.merger import MergeStrategy++from .base import AutoGraph, AutoGraphSchema, NodeListSchema, EdgeListSchema+from ..utils.logging import logger++# ==============================================================================+# 1. Schema Definition - Consistent with original iText2KG_Star implementation+# ==============================================================================+++class NodeSchema(BaseModel):+ """Knowledge Graph Node: Standard node structure"""++ label: str = Field(+ description=(+ "The semantic category of the entity (e.g., 'Person', 'Event', 'Location', 'Methodology', 'Position'). "+ "Use 'Relationship' objects if the concept is inherently relational or verbal (e.g., 'plans'). "+ "Prefer consistent, single-word categories where possible (e.g., 'Person', not 'Person_Entity'). "+ )+ )+ name: str = Field(+ description=(+ "The unique name or title identifying this entity, representing exactly one concept. "+ "For example, 'Yassir', 'CEO', or 'X'. Avoid combining multiple concepts (e.g., 'CEO of X'), "+ "since linking them should be done via Relationship objects. "+ "Verbs or multi-concept phrases (e.g., 'plans an escape') typically belong in Relationship objects. "+ )+ )+++class EdgeProperties(BaseModel):+ """Properties for Knowledge Graph Edge"""++ observation_date: Optional[str] = Field(+ default=None,+ description=(+ "The date when the relationship was observed or recorded. "+ "This field is populated manually by the user after extraction. "+ "If multiple observation dates exist, use the latest date."+ ),+ )+++class EdgeSchema(BaseModel):+ """Knowledge Graph Edge: Standard edge structure representing relationships between nodes"""++ startNode: NodeSchema = Field(+ description=(+ "The 'subject' or source entity of this relationship, which must appear in the known entities."+ )+ )+ endNode: NodeSchema = Field(+ description=(+ "The 'object' or target entity of this relationship, which must also appear in the known entities."+ )+ )+ name: str = Field(+ description=(+ "A single, canonical predicate capturing how the startNode and endNode relate (e.g., 'is_CEO', "+ "'holds_position', 'located_in'). Avoid compound verbs (e.g., 'plans_and_executes'). "+ "AVOID relation names as prepositions 'of', 'in' or similar."+ )+ )+ properties: EdgeProperties = Field(+ description="Additional properties describing the relationship."+ )+++# ==============================================================================+# 2. Prompt Definition - Adapted from original iText2KG_Star implementation+# ==============================================================================+++iText2KG_Star_EDGE_EXTRACTION_PROMPT = """+# DIRECTIVES :+- Extract all meaningful relationships directly from the provided context.+- For each relationship, identify the start entity and end entity involved.+- Each entity should have a clear name and appropriate label/type.+- Avoid reflexive relations (entity relating to itself).+- IMPORTANT: Set 'observation_date' to null in all cases. Do NOT extract or infer dates from the context.+- The observation_date field should be populated by the user manually after extraction.+"""+++# ==============================================================================+# 3. iText2KG_Star Implementation - Inherits from AutoGraph+# ==============================================================================+++class iText2KG_Star(AutoGraph[NodeSchema, EdgeSchema]):+ """+ iText2KG_Star: A specialized AutoGraph for extracting high-quality triple-based KGs.++ Features:+ - One-stage extraction: Extracts edges directly, then derives nodes automatically+ - Customized prompts from original iText2KG_Star implementation+ - Semantic Deduplication: Includes `match_nodes_and_update_edges` using SemHash with embeddings+ - Automatic date tracking: Observation date set to extraction time+ - Nested node schema: Maintains rich semantic information (name + label)++ Example:+ >>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings+ >>> llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+ >>> embedder = OpenAIEmbeddings()+ >>> kg = iText2KG_Star(llm_client=llm, embedder=embedder)++ >>> # 1. Extract relationships and derive nodes+ >>> kg.feed_text("Elon Musk is the CEO of SpaceX. Musk leads Tesla.")+ >>> print(f"Before dedup - Nodes: {len(kg.nodes)}, Edges: {len(kg.edges)}")++ >>> # 2. Deduplicate: Merges 'Elon Musk' and 'Musk'+ >>> kg.match_nodes_and_update_edges(threshold=0.85)+ >>> print(f"After dedup - Nodes: {len(kg.nodes)}, Edges: {len(kg.edges)}")+ """++ def __init__(+ self,+ llm_client: BaseChatModel,+ embedder: Embeddings,+ observation_date: str | None = None,+ chunk_size: int = 2000,+ chunk_overlap: int = 200,+ max_workers: int = 10,+ verbose: bool = True,+ ):+ """Initialize iText2KG_Star.++ Args:+ llm_client: Language model for extraction+ embedder: Embedding model for vector indexing+ observation_date: Date when the extraction was performed, like '1997-10-10' or '1997-10-10 23:59:59'.+ If None, uses current date and time.+ chunk_size: Characters per chunk+ chunk_overlap: Overlapping characters between chunks+ max_workers: Max concurrent extraction workers+ verbose: Display detailed execution logs and progress information+ """++ self.observation_date = observation_date++ # 1. Define Key Extractors (critical for deduplication)+ # Node deduplication: exact match by name+ node_key_fn = lambda x: x.name++ # Edge deduplication: combination of subject-predicate-object triple+ edge_key_fn = lambda x: f"{x.startNode.name}|{x.name}|{x.endNode.name}"++ # 2. Edge consistency check: tell AutoGraph which nodes this edge connects+ nodes_in_edge_fn = lambda x: (x.startNode.name, x.endNode.name)++ # 3. Call parent class initialization+ logger.info("🚀 Initializing iText2KG_Star")+ super().__init__(+ node_schema=NodeSchema,+ edge_schema=EdgeSchema,+ node_key_extractor=node_key_fn,+ edge_key_extractor=edge_key_fn,+ nodes_in_edge_extractor=nodes_in_edge_fn,+ llm_client=llm_client,+ embedder=embedder,+ # Enforce one-stage extraction (first extract edges, then derive nodes)+ extraction_mode="one_stage",+ # Inject customized prompts+ prompt_for_edge_extraction=iText2KG_Star_EDGE_EXTRACTION_PROMPT,+ # Configure deduplication strategy+ node_strategy_or_merger=MergeStrategy.KEEP_EXISTING,+ edge_strategy_or_merger=MergeStrategy.KEEP_EXISTING,+ # Optimize indexing: only index name field+ node_fields_for_index=["name", "label"],+ edge_fields_for_index=["startNode", "name", "endNode"],+ # Other parameters+ chunk_size=chunk_size,+ chunk_overlap=chunk_overlap,+ max_workers=max_workers,+ verbose=verbose,+ )++ # ==================== Extraction Pipeline ====================++ def _extract_data(self, text: str) -> AutoGraphSchema:+ """Extract edges directly, then derive nodes (One-stage extraction).++ Process:+ 1. Split text into chunks.+ 2. Batch extract edges directly from chunks (Edges contain node info).+ 3. Post-process: Set observation date to current date and time.+ 4. Derive nodes from edges.+ 5. Merge all partial graphs into one global graph.++ Args:+ text: Input text.++ Returns:+ Extracted and validated graph.+ """+ # 1. Prepare chunks+ if len(text) <= self.chunk_size:+ chunks = [text]+ else:+ chunks = self.text_splitter.split_text(text)++ # 2. Batch Extract Edges directly+ prompt_template = ChatPromptTemplate.from_template(+ f"{self.edge_prompt}\n\n### Source Text:\n{{chunk_text}}"+ )+ llm_chain = prompt_template | self.llm_client.with_structured_output(+ self.edge_list_schema+ )++ inputs = [{"chunk_text": chunk} for chunk in chunks]+ chunk_edge_lists = llm_chain.batch(+ inputs, config={"max_concurrency": self.max_workers}+ )++ # 3. Post-process: Derive Nodes & Set Observation Date+ all_nodes_lists, all_edges_lists = [], []++ for edge_list in chunk_edge_lists:+ derived_nodes, processed_edges = [], []++ if edge_list and edge_list.items:+ for edge in edge_list.items:+ # Post-process: Set observation date to current extraction time+ edge: EdgeSchema+ observation_date = self.observation_date or datetime.now().strftime(+ "%Y-%m-%d %H:%M:%S"+ )+ edge.properties.observation_date = observation_date++ # Derive nodes from edges+ derived_nodes.append(edge.startNode)+ derived_nodes.append(edge.endNode)+ processed_edges.append(edge)++ all_nodes_lists.append(derived_nodes)+ all_edges_lists.append(processed_edges)++ # 4. Construct Partial Graphs (Tuple format for merge optimization)+ partial_graphs = (all_nodes_lists, all_edges_lists)++ # 5. Global Merge (passes tuples to merge_batch_data)+ raw_graph = self.merge_batch_data(partial_graphs)++ # Prune dangling edges to ensure graph consistency+ return self._prune_dangling_edges(raw_graph)++ # ==================== Node Deduplication ====================++ def match_nodes_and_update_edges(self, threshold: float = 0.8) -> "iText2KG_Star":+ """Match nodes in the graph and update edges accordingly using SemHash with embeddings.++ This method identifies and merges similar nodes based on semantic similarity+ using embeddings from the instance's embedder. It updates edges to reflect+ any changes in node identities.++ Args:+ threshold: Similarity threshold for matching nodes (0.0 to 1.0). Defaults to 0.8.++ Returns:+ The updated iText2KG_Star instance with matched nodes and updated edges.+ """+ logger.info(+ f"🔄 Starting node matching and edge update (threshold={threshold})..."+ )++ nodes, edges = self.nodes, self.edges++ if not nodes:+ logger.warning("⚠️ No nodes to match; skipping matching.")+ return self++ # 1. Generate Embeddings using self.embedder.embed_documents+ node_names = [n.name for n in nodes]+ logger.info(f"📊 Generating embeddings for {len(node_names)} nodes...")++ try:+ embeddings = self.embedder.embed_documents(node_names)+ except Exception as e:+ logger.error(f"❌ Failed to generate embeddings: {e}")+ return self++ # 2. Initialize SemHash from embeddings+ semhash = SemHash.from_embeddings(+ embeddings=embeddings,+ records=node_names,+ model=None, # Embeddings already provided+ )++ # 3. Run Deduplication+ result = semhash.self_deduplicate(threshold=threshold)++ # 4. Build Mapping from duplicates result+ mapping = {} # old_name -> new_name+ for record in result.filtered:+ if record.duplicates and len(record.duplicates) > 0:+ # Map the record to the first duplicate (representative)+ mapping[record.record] = record.duplicates[0][0]++ if not mapping:+ logger.info("✅ No similar nodes found above threshold.")+ return self++ logger.info(f"🔗 Found {len(mapping)} nodes to merge.")++ # 5. Apply mapping to graph data+ # Update Node names+ for node in nodes:+ if node.name in mapping:+ node.name = mapping[node.name]++ # Update Edge references (startNode and endNode are nested NodeSchema objects)+ for edge in edges:+ if edge.startNode.name in mapping:+ edge.startNode.name = mapping[edge.startNode.name]+ if edge.endNode.name in mapping:+ edge.endNode.name = mapping[edge.endNode.name]++ # 6. Update internal state+ new_data = AutoGraphSchema(nodes=nodes, edges=edges)+ self._set_data_state(new_data)++ logger.info(+ f"✅ Node matching complete: Nodes {len(nodes)} -> {len(self.nodes)}, Edges: {len(edges)}"+ )+ return selfdiff --git a/hyperextract/graphs/kg_gen.py b/hyperextract/graphs/kg_gen.pyindex 296eff2..1e91899 100644--- a/hyperextract/graphs/kg_gen.py+++ b/hyperextract/graphs/kg_gen.py@@ -27,7 +27,6 @@ class NodeSchema(BaseModel):"""Knowledge Graph Node: Minimal entity with only a name field"""name: str = Field(- ...,description="The unique name or identifier of the entity. Must be thorough and accurate to the source text.",)@@ -36,15 +35,12 @@ class EdgeSchema(BaseModel):"""Knowledge Graph Edge: Standard triple structure (Subject-Predicate-Object)"""subject: str = Field(- ...,description="Subject entity name. Must be an exact match from the extracted entities list.",)predicate: str = Field(- ...,description="The relationship/predicate between subject and object. Should be concise and descriptive.",)object: str = Field(- ...,description="Object entity name. Must be an exact match from the extracted entities list.",)@@ -168,12 +164,12 @@ class KG_Gen(AutoGraph[NodeSchema, EdgeSchema]):model: str | SentenceTransformer = "sentence-transformers/all-MiniLM-L6-v2",):"""Internal helper to apply SemHash deduplication on a graph data object.-+Args:graph_data: The graph data object (nodes/edges) to process in-place.threshold: SemHash similarity threshold (0.0 to 1.0).model: Embedding model for SemHash.-+Returns:The modified graph_data object."""diff --git a/hyperextract/hypergraphs/base.py b/hyperextract/hypergraphs/base.pyindex d733b52..6c92bc4 100644--- a/hyperextract/hypergraphs/base.py+++ b/hyperextract/hypergraphs/base.py@@ -460,7 +460,7 @@ class AutoHypergraph(inputs.append({"chunk_text": chunk, "node_context": node_context})prompt_template = ChatPromptTemplate.from_template(- f"{self.edge_prompt}{{node_context}}\n\n### Text Chunk:\n{{chunk_text}}"+ f"{self.edge_prompt}{{node_context}}\n\n### Source Text:\n{{chunk_text}}")llm_chain = prompt_template | self.llm_client.with_structured_output(self.edge_list_schemadiff --git a/hyperextract/hypergraphs/hyper_rag.py b/hyperextract/hypergraphs/hyper_rag.pyindex 81a59c8..a37be29 100644--- a/hyperextract/hypergraphs/hyper_rag.py+++ b/hyperextract/hypergraphs/hyper_rag.py@@ -18,13 +18,11 @@ from ontomem.merger import CustomRuleMergerclass NodeSchema(BaseModel):"""Represents an entity extracted from the source text."""- name: str = Field(..., description="Name of the entity, capitalized if in English")+ name: str = Field(..., description="Name of the entity")type: str = Field(- ...,description="Entity type (person, organization, geo, event, role, concept, etc.)",)description: str = Field(- ...,description="Comprehensive description of the entity's attributes and activities",)@@ -38,17 +36,15 @@ class EdgeSchema(BaseModel):"""Represents a relationship or event connecting multiple entities (low-order or high-order)."""participants: List[str] = Field(- ..., description="Names of entities involved in this relationship"+ description="Names of entities involved in this relationship")description: str = Field(- ..., description="Detailed explanation of the relationship or event"+ description="Detailed explanation of the relationship or event")keywords: List[str] = Field(- ...,description="List of keywords summarizing the relationship themes (e.g., ['conflict', 'trade', 'alliance'])",)strength: int = Field(- ...,ge=1,le=10,description="Numerical score indicating relationship strength (1-10)",@@ -87,7 +83,7 @@ It treats all involved entities as **participants** in a shared context# Merge Templates for LLM.CUSTOM_RULE Strategy# ============================================================================-NODE_MERGE_TEMPLATE = """You are an intelligent data merging assistant.+NODE_MERGE_RULE = """You are an intelligent data merging assistant.You will receive a list of objects representing the same Entity.Your task is to merge them into a SINGLE object exactly matching the schema.@@ -97,7 +93,7 @@ Merge strategy:2. **description**: Synthesize a single, comprehensive description containing all unique details from the input descriptions. Write it in the third person. Resolve any contradictions coherently."""-EDGE_MERGE_TEMPLATE = """You are an intelligent data merging assistant.+EDGE_MERGE_RULE = """You are an intelligent data merging assistant.You will receive a list of objects representing the same Hyperedge (Relationship/Event).Your task is to merge them into a SINGLE object exactly matching the schema.@@ -163,7 +159,7 @@ class Hyper_RAG(AutoHypergraph[NodeSchema, EdgeSchema]):key_extractor=node_key_fn,llm_client=llm_client,item_schema=NodeSchema,- rule=NODE_MERGE_TEMPLATE,+ rule=NODE_MERGE_RULE,)# Edge merger: merges relationship descriptions using EDGE_MERGE_TEMPLATE@@ -171,7 +167,7 @@ class Hyper_RAG(AutoHypergraph[NodeSchema, EdgeSchema]):key_extractor=edge_key_fn,llm_client=llm_client,item_schema=EdgeSchema,- rule=EDGE_MERGE_TEMPLATE,+ rule=EDGE_MERGE_RULE,)# 4. Call parent class initialization