Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
新增 AutoGraph 示例,展示 MMORPG 公会内讧的知识图谱提取;定义玩家和互动的 Pydantic 模型,模拟论坛爆料贴数据源,执行两阶段提取并分析结果;实现语义搜索功能以查找特定角色和事件,支持知识图谱的保存。
yifanfeng97c0413e93cf5d
diff --git a/examples/auto_graph_game_demo.py b/examples/auto_graph_game_demo.pynew file mode 100644index 0000000..ff1cc4b--- /dev/null+++ b/examples/auto_graph_game_demo.py@@ -0,0 +1,403 @@+"""+AutoGraph Demo: MMORPG Guild Drama (游戏公会恩仇录) 🎮⚔️++场景背景:+在一个名为《永恒之境》的游戏中,顶尖公会"星辰阁"在攻略最终BOSS时发生了严重的内讧。+这段文本是一篇游戏论坛的"818"(八卦爆料)帖子。++难点:+1. 术语混杂:ID、职业代称(主T、奶妈)、外号混用。+2. 关系复杂:涉及治疗、仇恨(Aggro)、装备分配(Loot)、公会管理。+3. 讽刺与隐喻:LLM 需要理解"划水"、"黑装备"等游戏黑话。+"""++import sys+from pathlib import Path++# 添加项目根目录到路径+project_root = Path(__file__).resolve().parent.parent+sys.path.append(str(project_root))++from pydantic import BaseModel, Field+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++from hyperextract.graphs.base import AutoGraph+from ontomem import MergeStrategy++import dotenv++dotenv.load_dotenv()+# ==================== 1. 定义游戏知识结构 (Schema) ====================++class Player(BaseModel):+ """玩家/角色节点"""+ game_id: str = Field(description="游戏角色ID")+ role_class: str = Field(description="职业/职责,如:坦克(Tank)、治疗(Healer)、法师(DPS)", default="未知")+ guild: str = Field(description="所属公会", default="无公会")++ def __repr__(self):+ return f"🎮 ID:[{self.game_id}] 职业:<{self.role_class}> 公会:{self.guild}"++class GameInteraction(BaseModel):+ """游戏互动边"""+ source: str = Field(description="发起互动的玩家")+ target: str = Field(description="互动的对象(玩家或BOSS)")+ action_type: str = Field(description="互动类型:'治疗', '攻击', '抢装备', '辱骂', '踢出队伍', '拉入队伍'")+ details: str = Field(description="具体的互动描述或原因")++ def __repr__(self):+ return f"⚡ {self.source} --[{self.action_type}]--> {self.target} ({self.details})"++# ==================== 2. 模拟论坛爆料贴 (Data Source - 加长版) ====================++GAME_FORUM_POST = """+【818】星辰阁公会昨晚解散的真相!团长黑装备还骂人!这简直是最大的内讧事件!++楼主:匿名玩家(ID已隐藏)+发帖时间:今天 23:45++我是星辰阁的一个普通成员,昨晚目睹了整个悲剧的发生。我觉得必须把真相告诉大家。++首先介绍一下我们的团队阵容:+- 会长兼主坦克:雷霆之怒(真实ID:张三,40级战士,装备评分9999)+- 副坦克:坚如磐石(真实ID:李四,38级盾战士,评分8800,人很老实)+- 主奶妈(治疗):小甜甜(真实ID:王五,38级圣骑士,评分8600)+- 副奶妈:月光治愈者(真实ID:赵六,35级德鲁伊,评分7500)+- DPS输出1(弓箭手):暗影刺客(真实ID:孙七,39级猎手,评分9200)+- DPS输出2(法师):火球术一级(真实ID:周八,37级法师,评分8400)+- DPS输出3(剑客):风云剑侠(真实ID:吴九,36级剑士,评分7800)+- 远程法师:冰霜法咒(真实ID:郑十,35级冰法,评分7600)++昨晚晚上8点,我们集合在副本入口。这是星辰阁第三次挑战"深渊魔龙"这个最终BOSS。+前两次都失败了,主要是因为配合不够默契。但是昨晚会长雷霆之怒发誓说一定要拿下这个BOSS。++副本开始,一切似乎很顺利。前5个小怪都被我们轻松秒杀。+然后大家来到了BOSS房间——深渊魔龙的巢穴。这是一个超级庞大的紫龙,满血时候气势相当恐怖。++战斗一开始,雷霆之怒(会长)冲上去拉住了仇恨值。坚如磐石(副坦克)站在侧翼进行补充。+小甜甜不断给雷霆之怒刷HOT(持续恢复)。月光治愈者给坚如磐石加盾。+前期看起来非常完美,大家的DPS输出也打得相当漂亮。++但是,大约打了5分钟,问题出现了。++坚如磐石(副坦克)突然掉线了!这太糟糕了。我们在公会频道里看到他的在线状态变成了灰色。+公会会长雷霆之怒看到这一幕,血压瞬间上升(我可以想象他的脸色)。++为什么副坦会掉线呢?大家后来才明白——坚如磐石的家里停电了。这真的是太不走运了。++没有副坦克的补充,所有的伤害都集中在了雷霆之怒身上。+但是这个时候,事情开始变得诡异。++小甜甜(主奶妈)本应该全力治疗会长,但是我通过伤害统计面板发现,她的治疗量明显不足。+更奇怪的是,她把大量的治疗都给了暗影刺客(弓箭手)。++这很不正常。暗影刺客作为DPS,血量本不应该这么低。我仔细看了一下暗影刺客的走位——+哎哟,这哥们为了最大化伤害输出,站位超级靠前,几乎要贴到BOSS身上了。+结果BOSS的爪子扫过来,一下子就把暗影刺客的血条扫到了20%以下。++关键时刻,小甜甜却把大治疗术给了暗影刺客。这让会长雷霆之怒的血量开始急速下降。++我当时就在YY频道里喊:"小甜甜,快给会长加血!"++月光治愈者也在拼命给会长加盾,但是她的输出能力有限,根本抵挡不住BOSS的伤害。++而且我还发现,月光治愈者的治疗也有一部分莫名其妙地给了其他人。看起来公会里有人在划水。++等等,我再看一下。哦天哪,火球术一级(法师)和冰霜法咒在BOSS的AoE范围内站了太长时间,+导致他们频繁掉血。但是,更夸张的是,暗影刺客和小甜甜之间似乎有什么默契——+每当暗影刺客掉血,小甜甜就立马给他加。++这时候,我想到了一个可能性。我在游戏论坛里看过一些帖子,有人提到暗影刺客和小甜甜好像是现实中的情侣!++妈呀,这下问题大了。在这么关键的时刻,小甜甜竟然偏心给她的男朋友加血,而忽视了作为团队大脑的会长!++大约在15分钟的时候,BOSS放出了一个大招——"魔龙吐息"。+这个技能造成的伤害足以秒杀任何一个DPS。但是坦克(雷霆之怒)应该能扛住。++可是,小甜甜的治疗没有及时跟上。雷霆之怒的血条瞬间掉到了0。++会长倒地了!++随后,失去了坦克的护佑,暗影刺客也被BOSS的普通攻击打成了残血,然后被秒杀。+紧接着是风云剑侠,然后是火球术一级。整个团队在短短30秒内全灭了。++副本失败。大家都复活在了副本入口。++这个时候,雷霆之怒(会长)的脾气彻底炸裂了。++他在YY频道里破口大骂:"小甜甜,你是干什么呢?!你是团队奶妈,不是暗影刺客的私人医生!+治疗职业就是要保坦克,你这样的治疗我见过,但从没见过这么烂的!"++小甜甜试图辩解:"我……我只是……"++雷霆之怒直接打断她:"别给我狡辩!我决定了,你从今天起被踢出公会!+而且暗影刺客这个摸鱼的家伙,我也要把他赶出去。两个人都别在星辰阁待着!"++但就在这个时候,事情发生了急转弯。++暗影刺客一句话都没说,只见他的角色突然冲向了BOSS房间。+而就在这时,副本的第二个阶段秘密掉落物品——传说级别的武器"魔龙之牙"出现了!++这是一把超级稀有的武器,全服务器可能只有两三把。价值高达几百万金币。++没等任何人反应过来,暗影刺客就冲上去,一把拿起了"魔龙之牙"!++我们都懵了。"魔龙之牙"本来应该由队长(雷霆之怒)来分配,决定给谁。+但是暗影刺客直接黑吃黑,把这把传说武器直接塞进了自己的背包里!++会长还在语音里骂人,根本没注意到这一幕。++紧接着,更狠的来了——暗影刺客在游戏里按下了"退出公会"的按钮。+他直接,光速,毫不犹豫地退出了星辰阁!++然后他的角色在世界频道发了一句话:"拜拜,各位。我去找个真正尊重我的公会。"++说完,这哥们直接下线了。++我们的公会频道顿时炸锅了。++小甜甜也没做任何辩解,直接退出了公会。她甚至没有说一句话。++会长雷霆之怒这时候才意识到发生了什么。他看到了掉落物品的分配记录——+"魔龙之牙"被暗影刺客拿走了,而不是团队共有。++他疯了。直接在世界频道开始刷屏:+"各位玩家,我要举报星辰阁的暗影刺客和小甜甜这两条狗!他们合谋黑装备!+不要信任他们!全服通缉这两个垃圾!"++火球术一级(法师)现在也开始助威。他在世界频道里连发了十多条信息:+"赞同!这两个人太过分了!狗男女!不要和他们做交易!"++"冰霜法咒在副本里的表现也不太理想,看起来他好像和那对情侣有什么关系,+因为我注意到在暗影刺客掉线的时候,冰霜法咒也莫名其妙地不进行输出了。"++有人开始怀疑冰霜法咒是不是也是那对情侣的朋友。++坚如磐石(副坦)这时候才上线。他看到了聊天记录,一脸懵逼。+他在公会频道里问:"发生什么了?我家里停电了……"++没有人回答他。因为大家都在看这场大戏。++月光治愈者试图调和一下,她说:"大家冷静……这件事可能有误会……"++但是没有人听她的。因为这个时候,世界频道上的玩家也开始评论这件事。++有人说:"这就是为什么我不玩大公会。大公会水太深。"++还有人说:"我早就听说暗影刺客这个人在游戏里名声不太好。多次被指控在副本里贪图装备。"++有人挖出了暗影刺客的历史记录。原来他曾经在另一个公会里,也因为类似的事情被踢出来过。++这哥们简直就是个"装备贪心鬼"。++风云剑侠开始反思:"也许我们团队的问题不只是小甜甜。我们可能选错人了。"++火球术一级继续在世界频道里刷屏:"各位,我建议大家抵制这两个玩家。不要和他们一个队。"++这时候,一个名字叫做"游戏管理员001"的官方账号出现了。+他说:"我们收到了关于玩家暗影刺客的举报。我们正在调查这件事。如果确认违规,将给予封号处理。"++暗影刺客的名字从世界频道消失了——他已经下线了。而且下线之前把"魔龙之牙"从背包里转移到了仓库。+没人知道他把这把武器藏在了哪里。++最后,星辰阁的会长雷霆之怒在公会频道里宣布:+"各位成员,我决定了。星辰阁正式解散。这个公会不适合我继续管理。+我会转移到一个小公会,和志同道合的朋友重新开始。"++很多成员试图挽留他。但是他的决心已定。++现在,星辰阁已经成为了一个"死公会"。只有一些离不开这个ID的成员还在。++这就是昨晚发生的一切。我作为一个旁观者,见证了这场游戏江湖的大戏。++最后的最后,我想说:游戏就是游戏。团队合作最重要。装备是身外之物。+但是,为了装备而背叛队友,这绝对是游戏玩家中最可耻的行为。++希望游戏官方能够严肃处理暗影刺客的违规行为。+也希望星辰阁的所有成员能够吸取教训,在各自的新公会里找到快乐。++---++【回复楼层】++1楼:网络大侠+妈呀,这比小说还精彩。我已经截图保存,等着看后续。++2楼:游戏评论家+我注意到"冰霜法咒"在这件事里也有可疑之处。为什么他突然停止输出?+楼主,能不能深入调查一下他和暗影刺客的关系?++3楼:前星辰阁成员(风云剑侠)+我想出来说两句话。昨晚我在现场。+真相是:小甜甜确实有偏心。但是暗影刺客黑装备这件事更过分。+会长的反应虽然情绪化,但也能理解。+我已经离开了星辰阁,现在申请加入"日月神教"这个新兴公会。++4楼:月光治愈者(副奶妈)+我想为自己辩解一下。我在昨晚的治疗中没有任何划水行为。+是小甜甜的治疗分配有问题,不是我。+顺便说一下,我现在还在星辰阁,因为我对公会有感情。+但是我也不太确定会不会继续留下来。++5楼:装备交易商+这就是为什么装备交易需要系统中间作为担保的原因。+建议官方加强对装备分配的监管。++6楼:游戏社区版主+这个帖子已经标记为"关键事件记录"。+我们会持续关注这件事的后续发展。+请各位不要进行人身攻击和骚扰。相关玩家可以选择转服。+"""++# ==================== 3. 运行提取 ====================++def run_demo():+ print("=" * 60)+ print("🕹️ 正在加载游戏日志分析模块...")+ print("=" * 60)++ # 初始化 LLM+ print("\n🔧 初始化 LLM 和嵌入模型...")+ llm = ChatOpenAI(model="gpt-4o", temperature=0)+ embedder = OpenAIEmbeddings()++ # 创建 AutoGraph+ print("📊 构建 AutoGraph 实例...\n")+ graph = AutoGraph[Player, GameInteraction](+ node_schema=Player,+ edge_schema=GameInteraction,++ # 1. ID 提取规则+ node_key_extractor=lambda x: x.game_id,+ # 2. 互动唯一性规则+ edge_key_extractor=lambda x: f"{x.source}_{x.action_type}_{x.target}",+ # 3. 验证规则+ nodes_in_edge_extractor=lambda x: (x.source, x.target),++ llm_client=llm,+ embedder=embedder,++ # 两阶段提取,适合处理这种连贯的剧情+ extraction_mode="two_stage",++ # 智能合并:能识别 "主T" = "雷霆之怒" = "会长"+ node_strategy_or_merger=MergeStrategy.LLM.BALANCED,++ # 针对游戏的特化提示词+ prompt_for_node_extraction=(+ "提取文本中的游戏玩家和BOSS。注意识别ID、职业(如坦克/治疗/DPS)和公会归属。"+ "将别名(如'会长'、'主T'、'奶妈')统一合并到其主要游戏ID上。"+ "提取玩家的真实身份(括号内的信息)。"+ ),+ prompt_for_edge_extraction=(+ "提取玩家之间和玩家与BOSS之间的互动行为。包括战斗行为(治疗/攻击/防守)、"+ "社交行为(辱骂/踢人/拉人)以及物品分配(抢装备/黑装备)。"+ "确保只记录文中明确发生的事件。重点关注有争议的互动。"+ ),++ show_progress=True+ )++ print(f"\n📖 正在阅读游戏论坛爆料帖...")+ print("=" * 60)+ print(GAME_FORUM_POST[:200] + "...")+ print("=" * 60)++ # 执行提取+ print("\n⚙️ 执行两阶段提取...\n")+ graph.extract(GAME_FORUM_POST)++ # ==================== 4. 战报分析 ====================++ print("\n" + "=" * 60)+ print(f"✅ 分析完成!")+ print("=" * 60)+ print(f" 找到玩家/BOSS: {len(graph.nodes)} 个")+ print(f" 记录的互动事件: {len(graph.edges)} 个")++ print("\n" + "-" * 60)+ print("🛡️ 角色面板 (Nodes)")+ print("-" * 60)+ for i, p in enumerate(graph.nodes, 1):+ print(f"{i}. {p}")++ print("\n" + "-" * 60)+ print("⚔️ 战斗和互动记录 (Edges)")+ print("-" * 60)+ for i, e in enumerate(graph.edges, 1):+ print(f"{i}. {e}")++ # ==================== 5. 语义搜索 (查内鬼) ====================++ print("\n" + "=" * 60)+ print("🔍 游戏管理员(GM)调查查询模块")+ print("=" * 60)++ graph.build_index() # 建立索引++ # 场景1: 找谁拿了装备?+ q1 = "谁拿走了传说武器?"+ print(f"\n❓ Query: {q1}")+ try:+ interactions = graph.search_edges(q1, top_k=2)+ if interactions:+ for i in interactions:+ print(f" 🚨 {i}")+ else:+ print(" (没有找到相关记录)")+ except Exception as e:+ print(f" (搜索出错: {e})")++ # 场景2: 找特定的职业+ q2 = "谁是治疗职业的?"+ print(f"\n❓ Query: {q2}")+ try:+ healers = graph.search_nodes(q2, top_k=2)+ if healers:+ for h in healers:+ print(f" 🚑 {h}")+ else:+ print(" (没有找到治疗职业)")+ except Exception as e:+ print(f" (搜索出错: {e})")++ # 场景3: 找矛盾中心+ q3 = "谁和谁发生了冲突?"+ print(f"\n❓ Query: {q3}")+ try:+ conflicts = graph.search_edges(q3, top_k=3)+ if conflicts:+ for c in conflicts:+ print(f" 🔥 {c}")+ else:+ print(" (没有找到冲突记录)")+ except Exception as e:+ print(f" (搜索出错: {e})")++ # 场景4: 找坦克+ q4 = "谁是主坦克?"+ print(f"\n❓ Query: {q4}")+ try:+ tanks = graph.search_nodes(q4, top_k=1)+ if tanks:+ for t in tanks:+ print(f" 🛡️ {t}")+ else:+ print(" (没有找到坦克)")+ except Exception as e:+ print(f" (搜索出错: {e})")++ # ==================== 6. 保存知识图谱 ====================++ print("\n" + "=" * 60)+ print("💾 保存知识图谱到本地...")+ print("=" * 60)+ save_path = project_root / "temp" / "game_guild_graph"+ try:+ graph.dump(save_path)+ print(f"✅ 图谱已保存到: {save_path}")+ except Exception as e:+ print(f"⚠️ 保存失败: {e}")++if __name__ == "__main__":+ run_demo()diff --git a/hyperextract/graphs/base.py b/hyperextract/graphs/base.pyindex 41b6166..464c3b4 100644--- a/hyperextract/graphs/base.py+++ b/hyperextract/graphs/base.py@@ -4,7 +4,17 @@ Provides automatic deduplication for both nodes and edges using OMem.Supports single-stage and two-stage extraction strategies with consistency validation."""-from typing import Any, List, Type, Tuple, Callable, TypeVar, Generic, TYPE_CHECKING, Union+from typing import (+ Any,+ List,+ Type,+ Tuple,+ Callable,+ TypeVar,+ Generic,+ TYPE_CHECKING,+ Union,+)from pathlib import Pathfrom pydantic import BaseModel, Field, create_modelfrom langchain_core.embeddings import Embeddings@@ -24,9 +34,7 @@ Edge = TypeVar("Edge", bound=BaseModel)class AutoGraphSchema(BaseModel, Generic[Node, Edge]):"""Generic schema container for graph-based knowledge patterns."""- nodes: List[Node] = Field(- default_factory=list, description="Graph nodes/entities"- )+ nodes: List[Node] = Field(default_factory=list, description="Graph nodes/entities")edges: List[Edge] = Field(default_factory=list, description="Graph edges/relationships")@@ -34,23 +42,23 @@ class AutoGraphSchema(BaseModel, Generic[Node, Edge]):class NodeListSchema(BaseModel, Generic[Node]):"""Intermediate schema for batch node extraction."""+items: List[Node] = Field(default_factory=list,- description="List of identified entities or nodes found in the text."+ description="List of identified entities or nodes found in the text.",)class EdgeListSchema(BaseModel, Generic[Edge]):"""Intermediate schema for batch edge extraction."""+items: List[Edge] = Field(default_factory=list,- description="List of identified relationships or edges found in the text."+ description="List of identified relationships or edges found in the text.",)-class AutoGraph(- BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]-):+class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):"""AutoGraph - extracts knowledge graphs with nodes and edges from text.This pattern extracts structured knowledge graphs consisting of entities (nodes) and@@ -83,7 +91,7 @@ class AutoGraph(... edge_schema=Relation,... node_key_extractor=lambda x: x.name,... edge_key_extractor=lambda x: f"{x.source}-{x.relation_type}-{x.target}",- ... edge_to_nodes_extractor=lambda x: (x.source, x.target),+ ... nodes_in_edge_extractor=lambda x: (x.source, x.target),... llm_client=llm,... embedder=embedder,... extraction_mode="two_stage"@@ -99,7 +107,7 @@ class AutoGraph(edge_schema: Type[Edge],node_key_extractor: Callable[[Node], str],edge_key_extractor: Callable[[Edge], str],- edge_to_nodes_extractor: Callable[[Edge], Tuple[str, str]],+ nodes_in_edge_extractor: Callable[[Edge], Tuple[str, str]],llm_client: BaseChatModel,embedder: Embeddings,*,@@ -107,8 +115,9 @@ class AutoGraph(node_strategy_or_merger: MergeStrategy| BaseMerger = MergeStrategy.LLM.BALANCED,edge_strategy_or_merger: MergeStrategy | BaseMerger = MergeStrategy.MERGE_FIELD,- repair_strategy: str = "drop", # "drop" or "create_ghost"prompt: str = "",+ prompt_for_node_extraction: str = "",+ prompt_for_edge_extraction: str = "",chunk_size: int = 2000,chunk_overlap: int = 200,max_workers: int = 10,@@ -122,14 +131,15 @@ class AutoGraph(edge_schema: Pydantic BaseModel for edges/relationships.node_key_extractor: Function to extract unique key from node (e.g., lambda x: x.id).edge_key_extractor: Function to extract unique key from edge (e.g., lambda x: f"{x.src}-{x.rel}-{x.dst}").- edge_to_nodes_extractor: Function to extract (source_key, target_key) from edge for validation.+ nodes_in_edge_extractor: Function to extract (source_key, target_key) node keys from an edge for validation.llm_client: Language model client for extraction.embedder: Embedding model for vector indexing.extraction_mode: "one_stage" (extract nodes+edges together) or "two_stage" (nodes first, then edges).node_strategy_or_merger: Merge strategy for duplicate nodes (default: LLM.BALANCED).edge_strategy_or_merger: Merge strategy for duplicate edges (default: MERGE_FIELD).- repair_strategy: How to handle dangling edges - "drop" (remove) or "create_ghost" (create minimal nodes).- prompt: Custom extraction prompt.+ prompt: Custom extraction prompt for one-stage mode.+ prompt_for_node_extraction: Custom extraction prompt for two-stage node extraction.+ prompt_for_edge_extraction: Custom extraction prompt for two-stage edge extraction.chunk_size: Maximum characters per chunk.chunk_overlap: Overlapping characters between chunks.max_workers: Maximum concurrent extraction tasks.@@ -142,9 +152,12 @@ class AutoGraph(self.edge_schema = edge_schemaself.node_key_extractor = node_key_extractorself.edge_key_extractor = edge_key_extractor- self.edge_to_nodes_extractor = edge_to_nodes_extractor+ self.nodes_in_edge_extractor = nodes_in_edge_extractorself.extraction_mode = extraction_mode- self.repair_strategy = repair_strategy++ # Initialize prompts (use custom if provided, otherwise use defaults)+ self.node_prompt = prompt_for_node_extraction or self._default_node_prompt()+ self.edge_prompt = prompt_for_edge_extraction or self._default_edge_prompt()# Create dynamic GraphSchema containersgraph_schema_name = f"{node_schema.__name__}{edge_schema.__name__}Graph"@@ -156,10 +169,18 @@ class AutoGraph(# Create schema for list extraction (two-stage mode) with 'items' fieldself.node_list_schema = create_model(- "NodeList", items=(List[node_schema], Field(default_factory=list, description="Extracted nodes"))+ "NodeList",+ items=(+ List[node_schema],+ Field(default_factory=list, description="Extracted nodes"),+ ),)self.edge_list_schema = create_model(- "EdgeList", items=(List[edge_schema], Field(default_factory=list, description="Extracted edges"))+ "EdgeList",+ items=(+ List[edge_schema],+ Field(default_factory=list, description="Extracted edges"),+ ),)# Initialize Node Merger@@ -218,21 +239,57 @@ class AutoGraph()def _default_prompt(self) -> str:- """Returns the default extraction prompt based on extraction mode."""- if self.extraction_mode == "one_stage":- return (- "You are an expert knowledge graph extraction assistant. "- "Extract entities (nodes) and their relationships (edges) from the following text. "- "IMPORTANT CONSTRAINT: Every edge must connect two nodes that are present in the nodes list. "- "Do not create edges between entities that are not explicitly listed as nodes.\n\n"- "### Source Text:\n"- )- else: # two_stage- return (- "You are an expert knowledge extraction assistant. "- "Extract structured information from the following text according to the specified schema.\n\n"- "### Source Text:\n"- )+ """Returns the default prompt for one-stage graph extraction."""+ return self._default_graph_prompt++ @property+ def _default_graph_prompt(self) -> str:+ """Default prompt for one-stage graph extraction (nodes + edges together).++ Emphasizes comprehensive extraction and constraint enforcement.+ """+ return (+ "You are an expert knowledge graph extraction assistant. "+ "Extract all entities (nodes) and their relationships (edges) from the following text. "+ "Focus on being comprehensive and capturing the complete knowledge structure.\n\n"+ "CRITICAL CONSTRAINT: Every edge must connect two nodes that are present in the extracted nodes list. "+ "Do not create edges between entities that are not explicitly identified as nodes.\n\n"+ "### Source Text:\n"+ )++ @property+ def _default_node_prompt(self) -> str:+ """Default prompt for two-stage node extraction (Step 1).++ Emphasizes exhaustiveness and precision in entity identification.+ """+ return (+ "You are an expert information extraction assistant specialized in entity/node recognition. "+ "Extract ALL relevant entities, concepts, or nodes from the following text with high precision.\n\n"+ "Focus on:\n"+ "- Being EXHAUSTIVE: capture all entity types mentioned\n"+ "- Being PRECISE: extract exact entity names and descriptions\n"+ "- Clarity: provide clear, concise descriptions for each entity\n\n"+ "Do not attempt to extract relationships at this stage, only identify entities.\n\n"+ "### Source Text:\n"+ )++ @property+ def _default_edge_prompt(self) -> str:+ """Default prompt for two-stage edge extraction (Step 2).++ Emphasizes strict validation: only extract edges where BOTH endpoints exist in provided entities.+ Includes warnings about hallucination prevention.+ """+ return (+ "You are an expert relationship extraction assistant. "+ "Extract relationships (edges) between the provided entities.\n\n"+ "CRITICAL RULES:\n"+ "1. ONLY extract edges connecting entities from the known entity list below\n"+ "2. DO NOT invent or hallucinate new entities that are not listed\n"+ "3. If an entity in the text is not in the known list, DO NOT create edges involving it\n"+ "4. Focus on explicit relationships mentioned in the text\n\n"+ )@propertydef data(self) -> AutoGraphSchema[Node, Edge]:@@ -332,8 +389,8 @@ class AutoGraph(else:raise ValueError(f"Invalid extraction_mode: {self.extraction_mode}")- # Validate and repair graph consistency- return self._validate_and_repair(raw_graph)+ # Prune dangling edges to ensure graph consistency+ return self._prune_dangling_edges(raw_graph)def _extract_data_by_one_stage(self, text: str) -> AutoGraphSchema:"""Extract nodes and edges simultaneously using single LLM call.@@ -395,15 +452,13 @@ class AutoGraph(# 4. Construct Partial Graphs (Tuple format for merge optimization)partial_graphs = []- for node_list_obj, edge_list_obj in zip(chunk_node_lists, chunk_edge_lists):- nodes = node_list_obj.items if node_list_obj else []- edges = edge_list_obj.items if edge_list_obj else []- partial_graphs.append((nodes, edges))+ for node_list, edge_list in zip(chunk_node_lists, chunk_edge_lists):+ partial_graphs.append((node_list.items, edge_list.items))# 5. Global Merge (passes tuples to merge_batch_data)return self.merge_batch_data(partial_graphs)- def _extract_nodes_batch(self, chunks: List[str]) -> List[Any]:+ def _extract_nodes_batch(self, chunks: List[str]) -> List[NodeListSchema[Node]]:"""Batch extract nodes from multiple text chunks.Args:@@ -413,7 +468,7 @@ class AutoGraph(List of NodeListSchema objects with extracted nodes."""prompt_template = ChatPromptTemplate.from_template(- "Extract all relevant entities/nodes from the following text:\n\n{chunk_text}"+ f"{self.node_prompt}{{chunk_text}}")llm_chain = prompt_template | self.llm_client.with_structured_output(self.node_list_schema@@ -422,7 +477,9 @@ class AutoGraph(inputs = [{"chunk_text": chunk} for chunk in chunks]return llm_chain.batch(inputs, config={"max_concurrency": self.max_workers})- def _extract_edges_batch(self, chunks: List[str], node_lists: List[Any]) -> List[Any]:+ def _extract_edges_batch(+ self, chunks: List[str], node_lists: List[NodeListSchema[Node]]+ ) -> List[EdgeListSchema[Edge]]:"""Batch extract edges using corresponding node lists as context.Args:@@ -433,25 +490,18 @@ class AutoGraph(List of EdgeListSchema objects with extracted edges."""inputs = []- for chunk, node_list_obj in zip(chunks, node_lists):- nodes = node_list_obj.items if node_list_obj else []+ for chunk, node_list in zip(chunks, node_lists):+ nodes = node_list.items if node_list else []if not nodes:node_context = "No specific entities identified in this chunk."else:node_keys = [self.node_key_extractor(n) for n in nodes]node_context = "Known entities: " + ", ".join(node_keys)- inputs.append({- "chunk_text": chunk,- "node_context": node_context- })+ inputs.append({"chunk_text": chunk, "node_context": node_context})prompt_template = ChatPromptTemplate.from_template(- "Extract relationships/edges from the text based on the provided entities.\n"- "CONSTRAINT: Only extract edges connecting the provided entities. "- "Do not invent new entities.\n\n"- "{node_context}\n\n"- "Text:\n{chunk_text}"+ f"{self.edge_prompt}{{node_context}}\n\n### Text Chunk:\n{{chunk_text}}")llm_chain = prompt_template | self.llm_client.with_structured_output(self.edge_list_schema@@ -459,85 +509,52 @@ class AutoGraph(return llm_chain.batch(inputs, config={"max_concurrency": self.max_workers})- def _validate_and_repair(self, graph: AutoGraphSchema) -> AutoGraphSchema:- """Validate graph consistency and repair dangling edges.+ def _prune_dangling_edges(self, graph: AutoGraphSchema) -> AutoGraphSchema:+ """Prune edges that connect to non-existent nodes (Consistency Check).++ Ensures graph consistency by removing any edges where either endpoint+ (source or target) does not exist in the node list.Args:graph: Raw graph that may contain dangling edges.Returns:- Validated and repaired graph.+ Graph with only valid edges (endpoints must strictly exist in nodes)."""valid_nodes = graph.nodesvalid_node_keys = {self.node_key_extractor(n) for n in valid_nodes}refined_edges = []- ghost_nodes = []+ dropped_count = 0for edge in graph.edges:- src_key, dst_key = self.edge_to_nodes_extractor(edge)+ src_key, dst_key = self.nodes_in_edge_extractor(edge)# Check if both endpoints exist- src_exists = src_key in valid_node_keys- dst_exists = dst_key in valid_node_keys+ src_exists = src_key in valid_node_keys or src_key in self._node_memory+ dst_exists = dst_key in valid_node_keys or dst_key in self._node_memoryif src_exists and dst_exists:refined_edges.append(edge)- elif self.repair_strategy == "create_ghost":- # Create ghost nodes for missing endpoints- if not src_exists and src_key not in [- self.node_key_extractor(g) for g in ghost_nodes- ]:- ghost_node = self._create_ghost_node(src_key)- ghost_nodes.append(ghost_node)- valid_node_keys.add(src_key)-- if not dst_exists and dst_key not in [- self.node_key_extractor(g) for g in ghost_nodes- ]:- ghost_node = self._create_ghost_node(dst_key)- ghost_nodes.append(ghost_node)- valid_node_keys.add(dst_key)-- refined_edges.append(edge)- else: # repair_strategy == "drop"- logger.warning(- f"Dropping dangling edge: {src_key} -> {dst_key} "+ else:+ dropped_count += 1+ logger.debug(+ f"Pruning dangling edge: {src_key} -> {dst_key} "f"(src_exists={src_exists}, dst_exists={dst_exists})")- # Combine original nodes with ghost nodes- all_nodes = valid_nodes + ghost_nodes-- return self.graph_schema(nodes=all_nodes, edges=refined_edges)-- def _create_ghost_node(self, node_key: str) -> Node:- """Create a minimal ghost node with only the key field populated.+ if dropped_count > 0:+ logger.info(+ f"Pruned {dropped_count} dangling edges to ensure graph consistency."+ )- Args:- node_key: The key value for the ghost node.-- Returns:- Minimal node instance.- """- # Attempt to create minimal node with only required fields- node_dict = {}- for field_name, field_info in self.node_schema.model_fields.items():- if field_info.is_required():- # Try to populate with key if field name matches common patterns- if field_name in ["id", "name", "key", "entity", "node_id"]:- node_dict[field_name] = node_key- else:- # Use default or empty value- node_dict[field_name] = ""-- return self.node_schema(**node_dict)+ return self.graph_schema(nodes=valid_nodes, edges=refined_edges)# ==================== Merge Logic ====================def merge_batch_data(- self,- data_list_or_tuple: List[Union[AutoGraphSchema, Tuple[List[Node], List[Edge]]]]+ self,+ data_list_or_tuple: List[Union[AutoGraphSchema, Tuple[List[Node], List[Edge]]]],) -> AutoGraphSchema:"""Merge multiple graphs or node/edge tuples into one.@@ -559,15 +576,21 @@ class AutoGraph(all_edges.extend(graph.edges)else:- assert len(data_list_or_tuple) == 2, "Invalid input format for batch merging"- nodes_list, edges_list = data_list_or_tuple[0], data_list_or_tuple[1]- assert isinstance(nodes_list[0], NodeListSchema), "Expected NodeListSchema in first element"- assert isinstance(edges_list[0], EdgeListSchema), "Expected EdgeListSchema in second element"+ assert len(data_list_or_tuple) == 2, (+ "Invalid input format for batch merging"+ )+ nodes_lists, edges_lists = data_list_or_tuple[0], data_list_or_tuple[1]+ assert isinstance(nodes_lists[0], Node), (+ "Invalid node list format for batch merging"+ )+ assert isinstance(edges_lists[0], Edge), (+ "Invalid edge list format for batch merging"+ )all_nodes, all_edges = [], []- for nodes, edges in zip(nodes_list, edges_list):- all_nodes.extend(nodes.items)- all_edges.extend(edges.items)+ for node_list, edge_list in zip(nodes_lists, edges_lists):+ all_nodes.extend(node_list)+ all_edges.extend(edge_list)merged_nodes = self.node_merger.merge(all_nodes) if all_nodes else []merged_edges = self.edge_merger.merge(all_edges) if all_edges else []@@ -582,35 +605,88 @@ class AutoGraph(index_nodes: Whether to index nodes (default: True).index_edges: Whether to index edges (default: False)."""- if self.empty():- return-- # Typically we index nodes; edges are optionalif index_nodes:- self._node_memory.build_index()+ self.build_node_index()if index_edges:+ self.build_edge_index()++ def build_node_index(self) -> None:+ """Build vector index specifically for nodes."""+ if not self.empty():+ self._node_memory.build_index()++ def build_edge_index(self) -> None:+ """Build vector index specifically for edges."""+ if not self.empty():self._edge_memory.build_index()def search(- self, query: str, top_k: int = 3, search_type: str = "nodes"- ) -> List[Any]:- """Search the graph using semantic similarity.+ self,+ query: str,+ top_k: int = 3,+ search_nodes: bool = True,+ search_edges: bool = False,+ ) -> Union[List[Node], List[Edge], Tuple[List[Node], List[Edge]]]:+ """Unified graph search interface.++ Supports searching nodes only, edges only, or both simultaneously.+ When searching both, returns a tuple of (nodes_results, edges_results).+ When searching one type, returns a flat list.Args:- query: Search query.- top_k: Number of results.- search_type: "nodes" or "edges".+ query: Search query string.+ top_k: Number of results to return per type (default: 3).+ search_nodes: Whether to search nodes (default: True).+ search_edges: Whether to search edges (default: False).Returns:- List of relevant items.+ - List[Node] if only search_nodes is True.+ - List[Edge] if only search_edges is True.+ - Tuple[List[Node], List[Edge]] if both search_nodes and search_edges are True.++ Raises:+ ValueError: If neither search_nodes nor search_edges is True."""- if search_type == "nodes":- return self._node_memory.retrieve(query=query, top_k=top_k)- elif search_type == "edges":- return self._edge_memory.retrieve(query=query, top_k=top_k)- else:- raise ValueError(f"Invalid search_type: {search_type}")+ if not search_nodes and not search_edges:+ raise ValueError(+ "At least one of search_nodes or search_edges must be True."+ )++ if search_nodes and search_edges:+ nodes = self.search_nodes(query, top_k)+ edges = self.search_edges(query, top_k)+ return (nodes, edges)++ if search_nodes:+ return self.search_nodes(query, top_k)++ if search_edges:+ return self.search_edges(query, top_k)++ def search_nodes(self, query: str, top_k: int = 3) -> List[Node]:+ """Semantic search for nodes/entities only.++ Args:+ query: Search query string.+ top_k: Number of results to return (default: 3).++ Returns:+ List of matching nodes using semantic similarity.+ """+ return self._node_memory.search(query=query, top_k=top_k)++ def search_edges(self, query: str, top_k: int = 3) -> List[Edge]:+ """Semantic search for edges/relationships only.++ Args:+ query: Search query string.+ top_k: Number of results to return (default: 3).++ Returns:+ List of matching edges using semantic similarity.+ """+ return self._edge_memory.search(query=query, top_k=top_k)# ==================== Serialization ====================@@ -623,7 +699,7 @@ class AutoGraph(node_index_path = folder / "node_index"node_index_path.mkdir(exist_ok=True)try:- self._node_memory.save_to_disk(str(node_index_path))+ self._node_memory.dump_index(str(node_index_path))except Exception as e:logger.warning(f"Failed to save node index: {e}")@@ -631,7 +707,7 @@ class AutoGraph(edge_index_path = folder / "edge_index"edge_index_path.mkdir(exist_ok=True)try:- self._edge_memory.save_to_disk(str(edge_index_path))+ self._edge_memory.dump_index(str(edge_index_path))except Exception as e:logger.warning(f"Failed to save edge index: {e}")@@ -643,7 +719,7 @@ class AutoGraph(node_index_path = folder / "node_index"if node_index_path.exists():try:- self._node_memory.load_from_disk(str(node_index_path))+ self._node_memory.load_index(str(node_index_path))except Exception as e:logger.warning(f"Failed to load node index: {e}")@@ -651,6 +727,6 @@ class AutoGraph(edge_index_path = folder / "edge_index"if edge_index_path.exists():try:- self._edge_memory.load_from_disk(str(edge_index_path))+ self._edge_memory.load_index(str(edge_index_path))except Exception as e:logger.warning(f"Failed to load edge index: {e}")