Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

新增 AutoGraph 示例,展示 MMORPG 公会内讧的知识图谱提取;定义玩家和互动的 Pydantic 模型,模拟论坛爆料贴数据源,执行两阶段提取并分析结果;实现语义搜索功能以查找特定角色和事件,支持知识图谱的保存。

yifanfeng97c0413e93cf5d
diff --git a/examples/auto_graph_game_demo.py b/examples/auto_graph_game_demo.py
new file mode 100644
index 0000000..ff1cc4b
--- /dev/null
+++ b/examples/auto_graph_game_demo.py
@@ -0,0 +1,403 @@
+"""
+AutoGraph Demo: MMORPG Guild Drama (游戏公会恩仇录) 🎮⚔️
+
+场景背景:
+在一个名为《永恒之境》的游戏中,顶尖公会"星辰阁"在攻略最终BOSS时发生了严重的内讧。
+这段文本是一篇游戏论坛的"818"(八卦爆料)帖子。
+
+难点:
+1. 术语混杂:ID、职业代称(主T、奶妈)、外号混用。
+2. 关系复杂:涉及治疗、仇恨(Aggro)、装备分配(Loot)、公会管理。
+3. 讽刺与隐喻:LLM 需要理解"划水"、"黑装备"等游戏黑话。
+"""
+
+import sys
+from pathlib import Path
+
+# 添加项目根目录到路径
+project_root = Path(__file__).resolve().parent.parent
+sys.path.append(str(project_root))
+
+from pydantic import BaseModel, Field
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+
+from hyperextract.graphs.base import AutoGraph
+from ontomem import MergeStrategy
+
+import dotenv
+
+dotenv.load_dotenv()
+# ==================== 1. 定义游戏知识结构 (Schema) ====================
+
+class Player(BaseModel):
+ """玩家/角色节点"""
+ game_id: str = Field(description="游戏角色ID")
+ role_class: str = Field(description="职业/职责,如:坦克(Tank)、治疗(Healer)、法师(DPS)", default="未知")
+ guild: str = Field(description="所属公会", default="无公会")
+
+ def __repr__(self):
+ return f"🎮 ID:[{self.game_id}] 职业:<{self.role_class}> 公会:{self.guild}"
+
+class GameInteraction(BaseModel):
+ """游戏互动边"""
+ source: str = Field(description="发起互动的玩家")
+ target: str = Field(description="互动的对象(玩家或BOSS)")
+ action_type: str = Field(description="互动类型:'治疗', '攻击', '抢装备', '辱骂', '踢出队伍', '拉入队伍'")
+ details: str = Field(description="具体的互动描述或原因")
+
+ def __repr__(self):
+ return f"⚡ {self.source} --[{self.action_type}]--> {self.target} ({self.details})"
+
+# ==================== 2. 模拟论坛爆料贴 (Data Source - 加长版) ====================
+
+GAME_FORUM_POST = """
+【818】星辰阁公会昨晚解散的真相!团长黑装备还骂人!这简直是最大的内讧事件!
+
+楼主:匿名玩家(ID已隐藏)
+发帖时间:今天 23:45
+
+我是星辰阁的一个普通成员,昨晚目睹了整个悲剧的发生。我觉得必须把真相告诉大家。
+
+首先介绍一下我们的团队阵容:
+- 会长兼主坦克:雷霆之怒(真实ID:张三,40级战士,装备评分9999)
+- 副坦克:坚如磐石(真实ID:李四,38级盾战士,评分8800,人很老实)
+- 主奶妈(治疗):小甜甜(真实ID:王五,38级圣骑士,评分8600)
+- 副奶妈:月光治愈者(真实ID:赵六,35级德鲁伊,评分7500)
+- DPS输出1(弓箭手):暗影刺客(真实ID:孙七,39级猎手,评分9200)
+- DPS输出2(法师):火球术一级(真实ID:周八,37级法师,评分8400)
+- DPS输出3(剑客):风云剑侠(真实ID:吴九,36级剑士,评分7800)
+- 远程法师:冰霜法咒(真实ID:郑十,35级冰法,评分7600)
+
+昨晚晚上8点,我们集合在副本入口。这是星辰阁第三次挑战"深渊魔龙"这个最终BOSS。
+前两次都失败了,主要是因为配合不够默契。但是昨晚会长雷霆之怒发誓说一定要拿下这个BOSS。
+
+副本开始,一切似乎很顺利。前5个小怪都被我们轻松秒杀。
+然后大家来到了BOSS房间——深渊魔龙的巢穴。这是一个超级庞大的紫龙,满血时候气势相当恐怖。
+
+战斗一开始,雷霆之怒(会长)冲上去拉住了仇恨值。坚如磐石(副坦克)站在侧翼进行补充。
+小甜甜不断给雷霆之怒刷HOT(持续恢复)。月光治愈者给坚如磐石加盾。
+前期看起来非常完美,大家的DPS输出也打得相当漂亮。
+
+但是,大约打了5分钟,问题出现了。
+
+坚如磐石(副坦克)突然掉线了!这太糟糕了。我们在公会频道里看到他的在线状态变成了灰色。
+公会会长雷霆之怒看到这一幕,血压瞬间上升(我可以想象他的脸色)。
+
+为什么副坦会掉线呢?大家后来才明白——坚如磐石的家里停电了。这真的是太不走运了。
+
+没有副坦克的补充,所有的伤害都集中在了雷霆之怒身上。
+但是这个时候,事情开始变得诡异。
+
+小甜甜(主奶妈)本应该全力治疗会长,但是我通过伤害统计面板发现,她的治疗量明显不足。
+更奇怪的是,她把大量的治疗都给了暗影刺客(弓箭手)。
+
+这很不正常。暗影刺客作为DPS,血量本不应该这么低。我仔细看了一下暗影刺客的走位——
+哎哟,这哥们为了最大化伤害输出,站位超级靠前,几乎要贴到BOSS身上了。
+结果BOSS的爪子扫过来,一下子就把暗影刺客的血条扫到了20%以下。
+
+关键时刻,小甜甜却把大治疗术给了暗影刺客。这让会长雷霆之怒的血量开始急速下降。
+
+我当时就在YY频道里喊:"小甜甜,快给会长加血!"
+
+月光治愈者也在拼命给会长加盾,但是她的输出能力有限,根本抵挡不住BOSS的伤害。
+
+而且我还发现,月光治愈者的治疗也有一部分莫名其妙地给了其他人。看起来公会里有人在划水。
+
+等等,我再看一下。哦天哪,火球术一级(法师)和冰霜法咒在BOSS的AoE范围内站了太长时间,
+导致他们频繁掉血。但是,更夸张的是,暗影刺客和小甜甜之间似乎有什么默契——
+每当暗影刺客掉血,小甜甜就立马给他加。
+
+这时候,我想到了一个可能性。我在游戏论坛里看过一些帖子,有人提到暗影刺客和小甜甜好像是现实中的情侣!
+
+妈呀,这下问题大了。在这么关键的时刻,小甜甜竟然偏心给她的男朋友加血,而忽视了作为团队大脑的会长!
+
+大约在15分钟的时候,BOSS放出了一个大招——"魔龙吐息"。
+这个技能造成的伤害足以秒杀任何一个DPS。但是坦克(雷霆之怒)应该能扛住。
+
+可是,小甜甜的治疗没有及时跟上。雷霆之怒的血条瞬间掉到了0。
+
+会长倒地了!
+
+随后,失去了坦克的护佑,暗影刺客也被BOSS的普通攻击打成了残血,然后被秒杀。
+紧接着是风云剑侠,然后是火球术一级。整个团队在短短30秒内全灭了。
+
+副本失败。大家都复活在了副本入口。
+
+这个时候,雷霆之怒(会长)的脾气彻底炸裂了。
+
+他在YY频道里破口大骂:"小甜甜,你是干什么呢?!你是团队奶妈,不是暗影刺客的私人医生!
+治疗职业就是要保坦克,你这样的治疗我见过,但从没见过这么烂的!"
+
+小甜甜试图辩解:"我……我只是……"
+
+雷霆之怒直接打断她:"别给我狡辩!我决定了,你从今天起被踢出公会!
+而且暗影刺客这个摸鱼的家伙,我也要把他赶出去。两个人都别在星辰阁待着!"
+
+但就在这个时候,事情发生了急转弯。
+
+暗影刺客一句话都没说,只见他的角色突然冲向了BOSS房间。
+而就在这时,副本的第二个阶段秘密掉落物品——传说级别的武器"魔龙之牙"出现了!
+
+这是一把超级稀有的武器,全服务器可能只有两三把。价值高达几百万金币。
+
+没等任何人反应过来,暗影刺客就冲上去,一把拿起了"魔龙之牙"!
+
+我们都懵了。"魔龙之牙"本来应该由队长(雷霆之怒)来分配,决定给谁。
+但是暗影刺客直接黑吃黑,把这把传说武器直接塞进了自己的背包里!
+
+会长还在语音里骂人,根本没注意到这一幕。
+
+紧接着,更狠的来了——暗影刺客在游戏里按下了"退出公会"的按钮。
+他直接,光速,毫不犹豫地退出了星辰阁!
+
+然后他的角色在世界频道发了一句话:"拜拜,各位。我去找个真正尊重我的公会。"
+
+说完,这哥们直接下线了。
+
+我们的公会频道顿时炸锅了。
+
+小甜甜也没做任何辩解,直接退出了公会。她甚至没有说一句话。
+
+会长雷霆之怒这时候才意识到发生了什么。他看到了掉落物品的分配记录——
+"魔龙之牙"被暗影刺客拿走了,而不是团队共有。
+
+他疯了。直接在世界频道开始刷屏:
+"各位玩家,我要举报星辰阁的暗影刺客和小甜甜这两条狗!他们合谋黑装备!
+不要信任他们!全服通缉这两个垃圾!"
+
+火球术一级(法师)现在也开始助威。他在世界频道里连发了十多条信息:
+"赞同!这两个人太过分了!狗男女!不要和他们做交易!"
+
+"冰霜法咒在副本里的表现也不太理想,看起来他好像和那对情侣有什么关系,
+因为我注意到在暗影刺客掉线的时候,冰霜法咒也莫名其妙地不进行输出了。"
+
+有人开始怀疑冰霜法咒是不是也是那对情侣的朋友。
+
+坚如磐石(副坦)这时候才上线。他看到了聊天记录,一脸懵逼。
+他在公会频道里问:"发生什么了?我家里停电了……"
+
+没有人回答他。因为大家都在看这场大戏。
+
+月光治愈者试图调和一下,她说:"大家冷静……这件事可能有误会……"
+
+但是没有人听她的。因为这个时候,世界频道上的玩家也开始评论这件事。
+
+有人说:"这就是为什么我不玩大公会。大公会水太深。"
+
+还有人说:"我早就听说暗影刺客这个人在游戏里名声不太好。多次被指控在副本里贪图装备。"
+
+有人挖出了暗影刺客的历史记录。原来他曾经在另一个公会里,也因为类似的事情被踢出来过。
+
+这哥们简直就是个"装备贪心鬼"。
+
+风云剑侠开始反思:"也许我们团队的问题不只是小甜甜。我们可能选错人了。"
+
+火球术一级继续在世界频道里刷屏:"各位,我建议大家抵制这两个玩家。不要和他们一个队。"
+
+这时候,一个名字叫做"游戏管理员001"的官方账号出现了。
+他说:"我们收到了关于玩家暗影刺客的举报。我们正在调查这件事。如果确认违规,将给予封号处理。"
+
+暗影刺客的名字从世界频道消失了——他已经下线了。而且下线之前把"魔龙之牙"从背包里转移到了仓库。
+没人知道他把这把武器藏在了哪里。
+
+最后,星辰阁的会长雷霆之怒在公会频道里宣布:
+"各位成员,我决定了。星辰阁正式解散。这个公会不适合我继续管理。
+我会转移到一个小公会,和志同道合的朋友重新开始。"
+
+很多成员试图挽留他。但是他的决心已定。
+
+现在,星辰阁已经成为了一个"死公会"。只有一些离不开这个ID的成员还在。
+
+这就是昨晚发生的一切。我作为一个旁观者,见证了这场游戏江湖的大戏。
+
+最后的最后,我想说:游戏就是游戏。团队合作最重要。装备是身外之物。
+但是,为了装备而背叛队友,这绝对是游戏玩家中最可耻的行为。
+
+希望游戏官方能够严肃处理暗影刺客的违规行为。
+也希望星辰阁的所有成员能够吸取教训,在各自的新公会里找到快乐。
+
+---
+
+【回复楼层】
+
+1楼:网络大侠
+妈呀,这比小说还精彩。我已经截图保存,等着看后续。
+
+2楼:游戏评论家
+我注意到"冰霜法咒"在这件事里也有可疑之处。为什么他突然停止输出?
+楼主,能不能深入调查一下他和暗影刺客的关系?
+
+3楼:前星辰阁成员(风云剑侠)
+我想出来说两句话。昨晚我在现场。
+真相是:小甜甜确实有偏心。但是暗影刺客黑装备这件事更过分。
+会长的反应虽然情绪化,但也能理解。
+我已经离开了星辰阁,现在申请加入"日月神教"这个新兴公会。
+
+4楼:月光治愈者(副奶妈)
+我想为自己辩解一下。我在昨晚的治疗中没有任何划水行为。
+是小甜甜的治疗分配有问题,不是我。
+顺便说一下,我现在还在星辰阁,因为我对公会有感情。
+但是我也不太确定会不会继续留下来。
+
+5楼:装备交易商
+这就是为什么装备交易需要系统中间作为担保的原因。
+建议官方加强对装备分配的监管。
+
+6楼:游戏社区版主
+这个帖子已经标记为"关键事件记录"。
+我们会持续关注这件事的后续发展。
+请各位不要进行人身攻击和骚扰。相关玩家可以选择转服。
+"""
+
+# ==================== 3. 运行提取 ====================
+
+def run_demo():
+ print("=" * 60)
+ print("🕹️ 正在加载游戏日志分析模块...")
+ print("=" * 60)
+
+ # 初始化 LLM
+ print("\n🔧 初始化 LLM 和嵌入模型...")
+ llm = ChatOpenAI(model="gpt-4o", temperature=0)
+ embedder = OpenAIEmbeddings()
+
+ # 创建 AutoGraph
+ print("📊 构建 AutoGraph 实例...\n")
+ graph = AutoGraph[Player, GameInteraction](
+ node_schema=Player,
+ edge_schema=GameInteraction,
+
+ # 1. ID 提取规则
+ node_key_extractor=lambda x: x.game_id,
+ # 2. 互动唯一性规则
+ edge_key_extractor=lambda x: f"{x.source}_{x.action_type}_{x.target}",
+ # 3. 验证规则
+ nodes_in_edge_extractor=lambda x: (x.source, x.target),
+
+ llm_client=llm,
+ embedder=embedder,
+
+ # 两阶段提取,适合处理这种连贯的剧情
+ extraction_mode="two_stage",
+
+ # 智能合并:能识别 "主T" = "雷霆之怒" = "会长"
+ node_strategy_or_merger=MergeStrategy.LLM.BALANCED,
+
+ # 针对游戏的特化提示词
+ prompt_for_node_extraction=(
+ "提取文本中的游戏玩家和BOSS。注意识别ID、职业(如坦克/治疗/DPS)和公会归属。"
+ "将别名(如'会长'、'主T'、'奶妈')统一合并到其主要游戏ID上。"
+ "提取玩家的真实身份(括号内的信息)。"
+ ),
+ prompt_for_edge_extraction=(
+ "提取玩家之间和玩家与BOSS之间的互动行为。包括战斗行为(治疗/攻击/防守)、"
+ "社交行为(辱骂/踢人/拉人)以及物品分配(抢装备/黑装备)。"
+ "确保只记录文中明确发生的事件。重点关注有争议的互动。"
+ ),
+
+ show_progress=True
+ )
+
+ print(f"\n📖 正在阅读游戏论坛爆料帖...")
+ print("=" * 60)
+ print(GAME_FORUM_POST[:200] + "...")
+ print("=" * 60)
+
+ # 执行提取
+ print("\n⚙️ 执行两阶段提取...\n")
+ graph.extract(GAME_FORUM_POST)
+
+ # ==================== 4. 战报分析 ====================
+
+ print("\n" + "=" * 60)
+ print(f"✅ 分析完成!")
+ print("=" * 60)
+ print(f" 找到玩家/BOSS: {len(graph.nodes)} 个")
+ print(f" 记录的互动事件: {len(graph.edges)} 个")
+
+ print("\n" + "-" * 60)
+ print("🛡️ 角色面板 (Nodes)")
+ print("-" * 60)
+ for i, p in enumerate(graph.nodes, 1):
+ print(f"{i}. {p}")
+
+ print("\n" + "-" * 60)
+ print("⚔️ 战斗和互动记录 (Edges)")
+ print("-" * 60)
+ for i, e in enumerate(graph.edges, 1):
+ print(f"{i}. {e}")
+
+ # ==================== 5. 语义搜索 (查内鬼) ====================
+
+ print("\n" + "=" * 60)
+ print("🔍 游戏管理员(GM)调查查询模块")
+ print("=" * 60)
+
+ graph.build_index() # 建立索引
+
+ # 场景1: 找谁拿了装备?
+ q1 = "谁拿走了传说武器?"
+ print(f"\n❓ Query: {q1}")
+ try:
+ interactions = graph.search_edges(q1, top_k=2)
+ if interactions:
+ for i in interactions:
+ print(f" 🚨 {i}")
+ else:
+ print(" (没有找到相关记录)")
+ except Exception as e:
+ print(f" (搜索出错: {e})")
+
+ # 场景2: 找特定的职业
+ q2 = "谁是治疗职业的?"
+ print(f"\n❓ Query: {q2}")
+ try:
+ healers = graph.search_nodes(q2, top_k=2)
+ if healers:
+ for h in healers:
+ print(f" 🚑 {h}")
+ else:
+ print(" (没有找到治疗职业)")
+ except Exception as e:
+ print(f" (搜索出错: {e})")
+
+ # 场景3: 找矛盾中心
+ q3 = "谁和谁发生了冲突?"
+ print(f"\n❓ Query: {q3}")
+ try:
+ conflicts = graph.search_edges(q3, top_k=3)
+ if conflicts:
+ for c in conflicts:
+ print(f" 🔥 {c}")
+ else:
+ print(" (没有找到冲突记录)")
+ except Exception as e:
+ print(f" (搜索出错: {e})")
+
+ # 场景4: 找坦克
+ q4 = "谁是主坦克?"
+ print(f"\n❓ Query: {q4}")
+ try:
+ tanks = graph.search_nodes(q4, top_k=1)
+ if tanks:
+ for t in tanks:
+ print(f" 🛡️ {t}")
+ else:
+ print(" (没有找到坦克)")
+ except Exception as e:
+ print(f" (搜索出错: {e})")
+
+ # ==================== 6. 保存知识图谱 ====================
+
+ print("\n" + "=" * 60)
+ print("💾 保存知识图谱到本地...")
+ print("=" * 60)
+ save_path = project_root / "temp" / "game_guild_graph"
+ try:
+ graph.dump(save_path)
+ print(f"✅ 图谱已保存到: {save_path}")
+ except Exception as e:
+ print(f"⚠️ 保存失败: {e}")
+
+if __name__ == "__main__":
+ run_demo()
diff --git a/hyperextract/graphs/base.py b/hyperextract/graphs/base.py
index 41b6166..464c3b4 100644
--- a/hyperextract/graphs/base.py
+++ b/hyperextract/graphs/base.py
@@ -4,7 +4,17 @@ Provides automatic deduplication for both nodes and edges using OMem.
Supports single-stage and two-stage extraction strategies with consistency validation.
"""
-from typing import Any, List, Type, Tuple, Callable, TypeVar, Generic, TYPE_CHECKING, Union
+from typing import (
+ Any,
+ List,
+ Type,
+ Tuple,
+ Callable,
+ TypeVar,
+ Generic,
+ TYPE_CHECKING,
+ Union,
+)
from pathlib import Path
from pydantic import BaseModel, Field, create_model
from langchain_core.embeddings import Embeddings
@@ -24,9 +34,7 @@ Edge = TypeVar("Edge", bound=BaseModel)
class AutoGraphSchema(BaseModel, Generic[Node, Edge]):
"""Generic schema container for graph-based knowledge patterns."""
- nodes: List[Node] = Field(
- default_factory=list, description="Graph nodes/entities"
- )
+ nodes: List[Node] = Field(default_factory=list, description="Graph nodes/entities")
edges: List[Edge] = Field(
default_factory=list, description="Graph edges/relationships"
)
@@ -34,23 +42,23 @@ class AutoGraphSchema(BaseModel, Generic[Node, Edge]):
class NodeListSchema(BaseModel, Generic[Node]):
"""Intermediate schema for batch node extraction."""
+
items: List[Node] = Field(
default_factory=list,
- description="List of identified entities or nodes found in the text."
+ description="List of identified entities or nodes found in the text.",
)
class EdgeListSchema(BaseModel, Generic[Edge]):
"""Intermediate schema for batch edge extraction."""
+
items: List[Edge] = Field(
default_factory=list,
- description="List of identified relationships or edges found in the text."
+ description="List of identified relationships or edges found in the text.",
)
-class AutoGraph(
- BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]
-):
+class AutoGraph(BaseAutoType[AutoGraphSchema[Node, Edge]], Generic[Node, Edge]):
"""AutoGraph - extracts knowledge graphs with nodes and edges from text.
This pattern extracts structured knowledge graphs consisting of entities (nodes) and
@@ -83,7 +91,7 @@ class AutoGraph(
... edge_schema=Relation,
... node_key_extractor=lambda x: x.name,
... edge_key_extractor=lambda x: f"{x.source}-{x.relation_type}-{x.target}",
- ... edge_to_nodes_extractor=lambda x: (x.source, x.target),
+ ... nodes_in_edge_extractor=lambda x: (x.source, x.target),
... llm_client=llm,
... embedder=embedder,
... extraction_mode="two_stage"
@@ -99,7 +107,7 @@ class AutoGraph(
edge_schema: Type[Edge],
node_key_extractor: Callable[[Node], str],
edge_key_extractor: Callable[[Edge], str],
- edge_to_nodes_extractor: Callable[[Edge], Tuple[str, str]],
+ nodes_in_edge_extractor: Callable[[Edge], Tuple[str, str]],
llm_client: BaseChatModel,
embedder: Embeddings,
*,
@@ -107,8 +115,9 @@ class AutoGraph(
node_strategy_or_merger: MergeStrategy
| BaseMerger = MergeStrategy.LLM.BALANCED,
edge_strategy_or_merger: MergeStrategy | BaseMerger = MergeStrategy.MERGE_FIELD,
- repair_strategy: str = "drop", # "drop" or "create_ghost"
prompt: str = "",
+ prompt_for_node_extraction: str = "",
+ prompt_for_edge_extraction: str = "",
chunk_size: int = 2000,
chunk_overlap: int = 200,
max_workers: int = 10,
@@ -122,14 +131,15 @@ class AutoGraph(
edge_schema: Pydantic BaseModel for edges/relationships.
node_key_extractor: Function to extract unique key from node (e.g., lambda x: x.id).
edge_key_extractor: Function to extract unique key from edge (e.g., lambda x: f"{x.src}-{x.rel}-{x.dst}").
- edge_to_nodes_extractor: Function to extract (source_key, target_key) from edge for validation.
+ nodes_in_edge_extractor: Function to extract (source_key, target_key) node keys from an edge for validation.
llm_client: Language model client for extraction.
embedder: Embedding model for vector indexing.
extraction_mode: "one_stage" (extract nodes+edges together) or "two_stage" (nodes first, then edges).
node_strategy_or_merger: Merge strategy for duplicate nodes (default: LLM.BALANCED).
edge_strategy_or_merger: Merge strategy for duplicate edges (default: MERGE_FIELD).
- repair_strategy: How to handle dangling edges - "drop" (remove) or "create_ghost" (create minimal nodes).
- prompt: Custom extraction prompt.
+ prompt: Custom extraction prompt for one-stage mode.
+ prompt_for_node_extraction: Custom extraction prompt for two-stage node extraction.
+ prompt_for_edge_extraction: Custom extraction prompt for two-stage edge extraction.
chunk_size: Maximum characters per chunk.
chunk_overlap: Overlapping characters between chunks.
max_workers: Maximum concurrent extraction tasks.
@@ -142,9 +152,12 @@ class AutoGraph(
self.edge_schema = edge_schema
self.node_key_extractor = node_key_extractor
self.edge_key_extractor = edge_key_extractor
- self.edge_to_nodes_extractor = edge_to_nodes_extractor
+ self.nodes_in_edge_extractor = nodes_in_edge_extractor
self.extraction_mode = extraction_mode
- self.repair_strategy = repair_strategy
+
+ # Initialize prompts (use custom if provided, otherwise use defaults)
+ self.node_prompt = prompt_for_node_extraction or self._default_node_prompt()
+ self.edge_prompt = prompt_for_edge_extraction or self._default_edge_prompt()
# Create dynamic GraphSchema containers
graph_schema_name = f"{node_schema.__name__}{edge_schema.__name__}Graph"
@@ -156,10 +169,18 @@ class AutoGraph(
# Create schema for list extraction (two-stage mode) with 'items' field
self.node_list_schema = create_model(
- "NodeList", items=(List[node_schema], Field(default_factory=list, description="Extracted nodes"))
+ "NodeList",
+ items=(
+ List[node_schema],
+ Field(default_factory=list, description="Extracted nodes"),
+ ),
)
self.edge_list_schema = create_model(
- "EdgeList", items=(List[edge_schema], Field(default_factory=list, description="Extracted edges"))
+ "EdgeList",
+ items=(
+ List[edge_schema],
+ Field(default_factory=list, description="Extracted edges"),
+ ),
)
# Initialize Node Merger
@@ -218,21 +239,57 @@ class AutoGraph(
)
def _default_prompt(self) -> str:
- """Returns the default extraction prompt based on extraction mode."""
- if self.extraction_mode == "one_stage":
- return (
- "You are an expert knowledge graph extraction assistant. "
- "Extract entities (nodes) and their relationships (edges) from the following text. "
- "IMPORTANT CONSTRAINT: Every edge must connect two nodes that are present in the nodes list. "
- "Do not create edges between entities that are not explicitly listed as nodes.\n\n"
- "### Source Text:\n"
- )
- else: # two_stage
- return (
- "You are an expert knowledge extraction assistant. "
- "Extract structured information from the following text according to the specified schema.\n\n"
- "### Source Text:\n"
- )
+ """Returns the default prompt for one-stage graph extraction."""
+ return self._default_graph_prompt
+
+ @property
+ def _default_graph_prompt(self) -> str:
+ """Default prompt for one-stage graph extraction (nodes + edges together).
+
+ Emphasizes comprehensive extraction and constraint enforcement.
+ """
+ return (
+ "You are an expert knowledge graph extraction assistant. "
+ "Extract all entities (nodes) and their relationships (edges) from the following text. "
+ "Focus on being comprehensive and capturing the complete knowledge structure.\n\n"
+ "CRITICAL CONSTRAINT: Every edge must connect two nodes that are present in the extracted nodes list. "
+ "Do not create edges between entities that are not explicitly identified as nodes.\n\n"
+ "### Source Text:\n"
+ )
+
+ @property
+ def _default_node_prompt(self) -> str:
+ """Default prompt for two-stage node extraction (Step 1).
+
+ Emphasizes exhaustiveness and precision in entity identification.
+ """
+ return (
+ "You are an expert information extraction assistant specialized in entity/node recognition. "
+ "Extract ALL relevant entities, concepts, or nodes from the following text with high precision.\n\n"
+ "Focus on:\n"
+ "- Being EXHAUSTIVE: capture all entity types mentioned\n"
+ "- Being PRECISE: extract exact entity names and descriptions\n"
+ "- Clarity: provide clear, concise descriptions for each entity\n\n"
+ "Do not attempt to extract relationships at this stage, only identify entities.\n\n"
+ "### Source Text:\n"
+ )
+
+ @property
+ def _default_edge_prompt(self) -> str:
+ """Default prompt for two-stage edge extraction (Step 2).
+
+ Emphasizes strict validation: only extract edges where BOTH endpoints exist in provided entities.
+ Includes warnings about hallucination prevention.
+ """
+ return (
+ "You are an expert relationship extraction assistant. "
+ "Extract relationships (edges) between the provided entities.\n\n"
+ "CRITICAL RULES:\n"
+ "1. ONLY extract edges connecting entities from the known entity list below\n"
+ "2. DO NOT invent or hallucinate new entities that are not listed\n"
+ "3. If an entity in the text is not in the known list, DO NOT create edges involving it\n"
+ "4. Focus on explicit relationships mentioned in the text\n\n"
+ )
@property
def data(self) -> AutoGraphSchema[Node, Edge]:
@@ -332,8 +389,8 @@ class AutoGraph(
else:
raise ValueError(f"Invalid extraction_mode: {self.extraction_mode}")
- # Validate and repair graph consistency
- return self._validate_and_repair(raw_graph)
+ # Prune dangling edges to ensure graph consistency
+ return self._prune_dangling_edges(raw_graph)
def _extract_data_by_one_stage(self, text: str) -> AutoGraphSchema:
"""Extract nodes and edges simultaneously using single LLM call.
@@ -395,15 +452,13 @@ class AutoGraph(
# 4. Construct Partial Graphs (Tuple format for merge optimization)
partial_graphs = []
- for node_list_obj, edge_list_obj in zip(chunk_node_lists, chunk_edge_lists):
- nodes = node_list_obj.items if node_list_obj else []
- edges = edge_list_obj.items if edge_list_obj else []
- partial_graphs.append((nodes, edges))
+ for node_list, edge_list in zip(chunk_node_lists, chunk_edge_lists):
+ partial_graphs.append((node_list.items, edge_list.items))
# 5. Global Merge (passes tuples to merge_batch_data)
return self.merge_batch_data(partial_graphs)
- def _extract_nodes_batch(self, chunks: List[str]) -> List[Any]:
+ def _extract_nodes_batch(self, chunks: List[str]) -> List[NodeListSchema[Node]]:
"""Batch extract nodes from multiple text chunks.
Args:
@@ -413,7 +468,7 @@ class AutoGraph(
List of NodeListSchema objects with extracted nodes.
"""
prompt_template = ChatPromptTemplate.from_template(
- "Extract all relevant entities/nodes from the following text:\n\n{chunk_text}"
+ f"{self.node_prompt}{{chunk_text}}"
)
llm_chain = prompt_template | self.llm_client.with_structured_output(
self.node_list_schema
@@ -422,7 +477,9 @@ class AutoGraph(
inputs = [{"chunk_text": chunk} for chunk in chunks]
return llm_chain.batch(inputs, config={"max_concurrency": self.max_workers})
- def _extract_edges_batch(self, chunks: List[str], node_lists: List[Any]) -> List[Any]:
+ def _extract_edges_batch(
+ self, chunks: List[str], node_lists: List[NodeListSchema[Node]]
+ ) -> List[EdgeListSchema[Edge]]:
"""Batch extract edges using corresponding node lists as context.
Args:
@@ -433,25 +490,18 @@ class AutoGraph(
List of EdgeListSchema objects with extracted edges.
"""
inputs = []
- for chunk, node_list_obj in zip(chunks, node_lists):
- nodes = node_list_obj.items if node_list_obj else []
+ for chunk, node_list in zip(chunks, node_lists):
+ nodes = node_list.items if node_list else []
if not nodes:
node_context = "No specific entities identified in this chunk."
else:
node_keys = [self.node_key_extractor(n) for n in nodes]
node_context = "Known entities: " + ", ".join(node_keys)
- inputs.append({
- "chunk_text": chunk,
- "node_context": node_context
- })
+ inputs.append({"chunk_text": chunk, "node_context": node_context})
prompt_template = ChatPromptTemplate.from_template(
- "Extract relationships/edges from the text based on the provided entities.\n"
- "CONSTRAINT: Only extract edges connecting the provided entities. "
- "Do not invent new entities.\n\n"
- "{node_context}\n\n"
- "Text:\n{chunk_text}"
+ f"{self.edge_prompt}{{node_context}}\n\n### Text Chunk:\n{{chunk_text}}"
)
llm_chain = prompt_template | self.llm_client.with_structured_output(
self.edge_list_schema
@@ -459,85 +509,52 @@ class AutoGraph(
return llm_chain.batch(inputs, config={"max_concurrency": self.max_workers})
- def _validate_and_repair(self, graph: AutoGraphSchema) -> AutoGraphSchema:
- """Validate graph consistency and repair dangling edges.
+ def _prune_dangling_edges(self, graph: AutoGraphSchema) -> AutoGraphSchema:
+ """Prune edges that connect to non-existent nodes (Consistency Check).
+
+ Ensures graph consistency by removing any edges where either endpoint
+ (source or target) does not exist in the node list.
Args:
graph: Raw graph that may contain dangling edges.
Returns:
- Validated and repaired graph.
+ Graph with only valid edges (endpoints must strictly exist in nodes).
"""
valid_nodes = graph.nodes
valid_node_keys = {self.node_key_extractor(n) for n in valid_nodes}
refined_edges = []
- ghost_nodes = []
+ dropped_count = 0
for edge in graph.edges:
- src_key, dst_key = self.edge_to_nodes_extractor(edge)
+ src_key, dst_key = self.nodes_in_edge_extractor(edge)
# Check if both endpoints exist
- src_exists = src_key in valid_node_keys
- dst_exists = dst_key in valid_node_keys
+ src_exists = src_key in valid_node_keys or src_key in self._node_memory
+ dst_exists = dst_key in valid_node_keys or dst_key in self._node_memory
if src_exists and dst_exists:
refined_edges.append(edge)
- elif self.repair_strategy == "create_ghost":
- # Create ghost nodes for missing endpoints
- if not src_exists and src_key not in [
- self.node_key_extractor(g) for g in ghost_nodes
- ]:
- ghost_node = self._create_ghost_node(src_key)
- ghost_nodes.append(ghost_node)
- valid_node_keys.add(src_key)
-
- if not dst_exists and dst_key not in [
- self.node_key_extractor(g) for g in ghost_nodes
- ]:
- ghost_node = self._create_ghost_node(dst_key)
- ghost_nodes.append(ghost_node)
- valid_node_keys.add(dst_key)
-
- refined_edges.append(edge)
- else: # repair_strategy == "drop"
- logger.warning(
- f"Dropping dangling edge: {src_key} -> {dst_key} "
+ else:
+ dropped_count += 1
+ logger.debug(
+ f"Pruning dangling edge: {src_key} -> {dst_key} "
f"(src_exists={src_exists}, dst_exists={dst_exists})"
)
- # Combine original nodes with ghost nodes
- all_nodes = valid_nodes + ghost_nodes
-
- return self.graph_schema(nodes=all_nodes, edges=refined_edges)
-
- def _create_ghost_node(self, node_key: str) -> Node:
- """Create a minimal ghost node with only the key field populated.
+ if dropped_count > 0:
+ logger.info(
+ f"Pruned {dropped_count} dangling edges to ensure graph consistency."
+ )
- Args:
- node_key: The key value for the ghost node.
-
- Returns:
- Minimal node instance.
- """
- # Attempt to create minimal node with only required fields
- node_dict = {}
- for field_name, field_info in self.node_schema.model_fields.items():
- if field_info.is_required():
- # Try to populate with key if field name matches common patterns
- if field_name in ["id", "name", "key", "entity", "node_id"]:
- node_dict[field_name] = node_key
- else:
- # Use default or empty value
- node_dict[field_name] = ""
-
- return self.node_schema(**node_dict)
+ return self.graph_schema(nodes=valid_nodes, edges=refined_edges)
# ==================== Merge Logic ====================
def merge_batch_data(
- self,
- data_list_or_tuple: List[Union[AutoGraphSchema, Tuple[List[Node], List[Edge]]]]
+ self,
+ data_list_or_tuple: List[Union[AutoGraphSchema, Tuple[List[Node], List[Edge]]]],
) -> AutoGraphSchema:
"""Merge multiple graphs or node/edge tuples into one.
@@ -559,15 +576,21 @@ class AutoGraph(
all_edges.extend(graph.edges)
else:
- assert len(data_list_or_tuple) == 2, "Invalid input format for batch merging"
- nodes_list, edges_list = data_list_or_tuple[0], data_list_or_tuple[1]
- assert isinstance(nodes_list[0], NodeListSchema), "Expected NodeListSchema in first element"
- assert isinstance(edges_list[0], EdgeListSchema), "Expected EdgeListSchema in second element"
+ assert len(data_list_or_tuple) == 2, (
+ "Invalid input format for batch merging"
+ )
+ nodes_lists, edges_lists = data_list_or_tuple[0], data_list_or_tuple[1]
+ assert isinstance(nodes_lists[0], Node), (
+ "Invalid node list format for batch merging"
+ )
+ assert isinstance(edges_lists[0], Edge), (
+ "Invalid edge list format for batch merging"
+ )
all_nodes, all_edges = [], []
- for nodes, edges in zip(nodes_list, edges_list):
- all_nodes.extend(nodes.items)
- all_edges.extend(edges.items)
+ for node_list, edge_list in zip(nodes_lists, edges_lists):
+ all_nodes.extend(node_list)
+ all_edges.extend(edge_list)
merged_nodes = self.node_merger.merge(all_nodes) if all_nodes else []
merged_edges = self.edge_merger.merge(all_edges) if all_edges else []
@@ -582,35 +605,88 @@ class AutoGraph(
index_nodes: Whether to index nodes (default: True).
index_edges: Whether to index edges (default: False).
"""
- if self.empty():
- return
-
- # Typically we index nodes; edges are optional
if index_nodes:
- self._node_memory.build_index()
+ self.build_node_index()
if index_edges:
+ self.build_edge_index()
+
+ def build_node_index(self) -> None:
+ """Build vector index specifically for nodes."""
+ if not self.empty():
+ self._node_memory.build_index()
+
+ def build_edge_index(self) -> None:
+ """Build vector index specifically for edges."""
+ if not self.empty():
self._edge_memory.build_index()
def search(
- self, query: str, top_k: int = 3, search_type: str = "nodes"
- ) -> List[Any]:
- """Search the graph using semantic similarity.
+ self,
+ query: str,
+ top_k: int = 3,
+ search_nodes: bool = True,
+ search_edges: bool = False,
+ ) -> Union[List[Node], List[Edge], Tuple[List[Node], List[Edge]]]:
+ """Unified graph search interface.
+
+ Supports searching nodes only, edges only, or both simultaneously.
+ When searching both, returns a tuple of (nodes_results, edges_results).
+ When searching one type, returns a flat list.
Args:
- query: Search query.
- top_k: Number of results.
- search_type: "nodes" or "edges".
+ query: Search query string.
+ top_k: Number of results to return per type (default: 3).
+ search_nodes: Whether to search nodes (default: True).
+ search_edges: Whether to search edges (default: False).
Returns:
- List of relevant items.
+ - List[Node] if only search_nodes is True.
+ - List[Edge] if only search_edges is True.
+ - Tuple[List[Node], List[Edge]] if both search_nodes and search_edges are True.
+
+ Raises:
+ ValueError: If neither search_nodes nor search_edges is True.
"""
- if search_type == "nodes":
- return self._node_memory.retrieve(query=query, top_k=top_k)
- elif search_type == "edges":
- return self._edge_memory.retrieve(query=query, top_k=top_k)
- else:
- raise ValueError(f"Invalid search_type: {search_type}")
+ if not search_nodes and not search_edges:
+ raise ValueError(
+ "At least one of search_nodes or search_edges must be True."
+ )
+
+ if search_nodes and search_edges:
+ nodes = self.search_nodes(query, top_k)
+ edges = self.search_edges(query, top_k)
+ return (nodes, edges)
+
+ if search_nodes:
+ return self.search_nodes(query, top_k)
+
+ if search_edges:
+ return self.search_edges(query, top_k)
+
+ def search_nodes(self, query: str, top_k: int = 3) -> List[Node]:
+ """Semantic search for nodes/entities only.
+
+ Args:
+ query: Search query string.
+ top_k: Number of results to return (default: 3).
+
+ Returns:
+ List of matching nodes using semantic similarity.
+ """
+ return self._node_memory.search(query=query, top_k=top_k)
+
+ def search_edges(self, query: str, top_k: int = 3) -> List[Edge]:
+ """Semantic search for edges/relationships only.
+
+ Args:
+ query: Search query string.
+ top_k: Number of results to return (default: 3).
+
+ Returns:
+ List of matching edges using semantic similarity.
+ """
+ return self._edge_memory.search(query=query, top_k=top_k)
# ==================== Serialization ====================
@@ -623,7 +699,7 @@ class AutoGraph(
node_index_path = folder / "node_index"
node_index_path.mkdir(exist_ok=True)
try:
- self._node_memory.save_to_disk(str(node_index_path))
+ self._node_memory.dump_index(str(node_index_path))
except Exception as e:
logger.warning(f"Failed to save node index: {e}")
@@ -631,7 +707,7 @@ class AutoGraph(
edge_index_path = folder / "edge_index"
edge_index_path.mkdir(exist_ok=True)
try:
- self._edge_memory.save_to_disk(str(edge_index_path))
+ self._edge_memory.dump_index(str(edge_index_path))
except Exception as e:
logger.warning(f"Failed to save edge index: {e}")
@@ -643,7 +719,7 @@ class AutoGraph(
node_index_path = folder / "node_index"
if node_index_path.exists():
try:
- self._node_memory.load_from_disk(str(node_index_path))
+ self._node_memory.load_index(str(node_index_path))
except Exception as e:
logger.warning(f"Failed to load node index: {e}")
@@ -651,6 +727,6 @@ class AutoGraph(
edge_index_path = folder / "edge_index"
if edge_index_path.exists():
try:
- self._edge_memory.load_from_disk(str(edge_index_path))
+ self._edge_memory.load_index(str(edge_index_path))
except Exception as e:
logger.warning(f"Failed to load edge index: {e}")