Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: 新增 Atom 模块及演示,优化时序知识图谱提取功能,并更新相关合并策略
yifanfeng97fc006bcb54f3
diff --git a/examples/atom_demo.py b/examples/atom_demo.pynew file mode 100644index 0000000..32d0819--- /dev/null+++ b/examples/atom_demo.py@@ -0,0 +1,308 @@+"""+Atom 演示:产品历史中的时序知识图谱提取++这个演示展示了如何使用 Atom 从产品迭代历史中提取结构化知识。+重点展示:+1. 产品发布时间与版本信息的提取+2. 关键人物与产品的关联+3. 公司并购与产品策略变化+4. 时序关系的精确捕获(t_start, t_end)+5. 证据追踪与来源溯源+"""++import os+import sys+from collections import Counter, defaultdict++# 添加项目根目录到路径+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from dotenv import load_dotenv+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++from hyperextract.graphs import Atom++load_dotenv()++# ==============================================================================+# 示例文本:产品历史(技术产品迭代)+# ==============================================================================++# EXAMPLE_TEXT = """+# 2024年是李四职业生涯的转折点。6月15日,李四加入了Google,担任工程师一职。+# 他加入的是核心技术团队,主要从事后端服务开发。工作初期,李四表现出色,+# 与团队协作顺利,代码质量得到认可。++# 到了9月,由于李四在多个关键项目中的出色表现,Google决定将其提升为技术主管。+# 升职后,李四开始领导一个30人的技术团队,负责推荐系统的开发和优化。+# 他实施了多项技术改革,团队效率提升了40%。++# 然而,到了11月中旬,李四发现自己对当前的薪资待遇不满意。经过深思熟虑,+# 他决定寻求新的机会。最终,李四接受了字节跳动的邀约,并于11月25日正式加入。+# 在字节,李四担任高级工程师,继续从事推荐系统的研发工作。+# """++EXAMPLE_TEXT = """+"CyberPhone"是未来科技(FutureTech)公司旗下的旗舰智能手机系列,代表了移动设备革新的前沿。++第一代CyberPhone于2010年6月发布,凭其全息投影屏幕震惊业界。当时的首席设计师是乔纳森·李,他与工程主管陈思共同主导了这次创新。+这款产品上市后迅速成为全球最受欢迎的高端手机,首年销售额达到了2亿美元。++2012年9月,FutureTech推出了CyberPhone 2。这一代产品引入了脑机接口(BMI)的雏形功能,允许用户通过脑电波进行基础交互。+设计师乔纳森·李继续领导设计团队,而陈思则被晋升为首席工程师。+然而,这次创新也带来了风险:由于集成度过高,CyberPhone 2的电池安全问题频频出现。++到了2013年,由于电池缺陷引发的多起安全事故,FutureTech不得不宣布全球召回所有CyberPhone 2。+这场危机导致公司股价大幅下跌,管理层面临巨大压力。+时任CEO的王芳为了挽回信誉,做出了一个大胆的决定:整个2014年不发布新产品,所有资源投入到底层系统和安全机制的重构。++直到2015年底,经过两年打磨,CyberPhone 3终于问世。虽然发布时间大幅推迟,但这一代产品搭载了革命性的AI助手"Luna"。+Luna是由FutureTech在2015年年中收购的初创公司"Moon AI"开发的核心算法。+Moon AI的创始人兼CEO李想因此加入了FutureTech,担任AI创新部门的副总裁,任期从2015年7月至2019年6月。+CyberPhone 3一经发布就获得了业界的广泛赞誉,被誉为"手机史上的里程碑"。++自此之后,FutureTech建立了稳定的产品节奏,基本保持每两年推出一代新品的策略。+2017年3月发布的CyberPhone 4进一步完善了AI助手功能,并加入了增强现实(AR)显示技术。+2019年9月发布的CyberPhone 5则在续航能力和计算性能上取得了突破。+在这个阶段,乔纳森·李仍然是首席设计师,陈思继续担任首席工程师。++但转折点来了。2021年4月,乔纳森·李在为FutureTech服务了整整12年后,宣布离职并加入竞品公司"QuantumTech"担任设计总监。+乔纳森的离职引发了业界热议,许多人认为这标志着CyberPhone系列即将走向衰落。+与此同时,新晋设计师韩雪接替了乔纳森的职位,但她的设计风格与乔纳森截然不同。++2021年底发布的CyberPhone 6展现了全新的设计语言——极简风格取代了之前的科技感。+业界对这一变化评价不一,许多长期粉丝甚至戏称这是"最不像CyberPhone的一代"。+销售数据也反映了这一点:CyberPhone 6的市场反应相比前几代明显不足。++在这个困难时期,陈思于2022年年底晋升为首席产品官(CPO),获得了更大的权力重新调整产品方向。+同时,李想在2022年中期转任为AI伦理委员会主席,虽然仍在公司但已远离产品一线。++去年(2023年)全年,FutureTech几乎没有推出任何重大产品更新。这个沉默期引发了市场猜测。+直到今年初,公司宣布了一个雄心勃勃的计划:将在今年晚些时候发布CyberPhone X。++本周,FutureTech官方确认了CyberPhone X的发布时间定在今年Q3,并透露这款产品将是对CyberPhone系列的根本性重新定义。+根据公司声明,CyberPhone X旨在纪念该系列诞生15周年,同时引入突破性的新技术。+韩雪确认将继续担任首席设计师,而陈思作为CPO将主导这次产品重塑项目。+令人惊喜的是,曾经的设计大师乔纳森·李也获邀作为顾问加盟这个项目,尽管他目前仍就职于QuantumTech。++业界普遍认为,CyberPhone X将成为检验FutureTech是否能重获竞争力的关键产品。+"""++# ==============================================================================+# 主程序+# ==============================================================================++def main():+ print("=" * 80)+ print("🕒 Atom 演示:产品历史中的时序知识图谱提取")+ print("=" * 80)+ print()++ # 1. 初始化+ print("📦 初始化组件...")+ llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+ embedder = OpenAIEmbeddings(model="text-embedding-3-small")++ obs_date = "2024-12-15"+ print(f"🚀 创建 Atom 实例(观察日期:{obs_date})...")+ kg = Atom(+ llm_client=llm,+ embedder=embedder,+ observation_date=obs_date,+ facts_per_chunk=10,+ verbose=True,+ )++ # 2. 提取+ print(f"\n📄 正在处理产品历史文本(长度:{len(EXAMPLE_TEXT)} 字符)...")+ print("状态: 正在进行两阶段提取...")+ print(" [Phase 1] 原子事实提取")+ print(" [Phase 2] 关系提取(带时序和证据追踪)")+ print()+ kg.feed_text(EXAMPLE_TEXT)++ # 3. 显示初步结果+ print("\n" + "=" * 80)+ print("✅ Phase 1-2 提取完成!")+ print("=" * 80)++ # 统计信息+ print(f"\n📊 提取统计:")+ print(f" 节点数量: {len(kg.nodes)}")+ print(f" 关系数量: {len(kg.edges)}")++ # 统计标签分布+ labels = Counter(n.label for n in kg.nodes)+ print(f" 实体类型分布: {dict(labels)}")++ # 显示所有节点+ print("\n" + "-" * 80)+ print(f"🏷️ 提取的实体及其类型 ({len(kg.nodes)} 个):")+ print("-" * 80)+ for i, node in enumerate(kg.nodes, 1):+ print(f" {i:2d}. {node.name:<30} [类型: {node.label}]")++ # 显示所有关系(包括时间和证据信息)+ print("\n" + "-" * 80)+ print(f"🔗 提取的关系 ({len(kg.edges)} 个):")+ print("-" * 80)++ # 按时间排序关系+ sorted_edges = sorted(+ kg.edges,+ key=lambda e: e.t_start[0] if e.t_start else "9999-12-31"+ )++ for i, edge in enumerate(sorted_edges, 1):+ # 基本关系+ rel_str = f"{edge.startNode.name} --[{edge.name}]--> {edge.endNode.name}"++ # 添加时间信息+ time_info = ""+ if edge.t_start:+ time_info += f"(从 {edge.t_start[0]}"+ if edge.t_end:+ time_info += f" 到 {edge.t_end[0]})"+ else:+ time_info += ")"+ elif edge.t_end:+ time_info += f"(截止 {edge.t_end[0]})"++ print(f" {i:2d}. {rel_str:<65} {time_info}")++ # 显示证据(只显示第一条)+ if edge.atomic_facts:+ evidence = edge.atomic_facts[0].replace("\n", " ")+ if len(evidence) > 70:+ evidence = evidence[:70] + "..."+ print(f" 📌 证据: {evidence}")++ # 按谓词分组显示+ print("\n" + "-" * 80)+ print("📋 按关系类型分类:")+ print("-" * 80)++ relations_by_type = defaultdict(list)+ for edge in kg.edges:+ relations_by_type[edge.name].append((edge.startNode.name, edge.endNode.name))++ for pred in sorted(relations_by_type.keys()):+ pairs = relations_by_type[pred]+ print(f"\n ✓ {pred} ({len(pairs)} 条)")+ for s, o in pairs[:8]: # 显示前 8 条+ print(f" • {s} → {o}")+ if len(pairs) > 8:+ print(f" • ... 及 {len(pairs) - 8} 条其他关系")++ # 实体连接度分析+ print("\n" + "-" * 80)+ print("⭐ 高关联度实体 (Top 15):")+ print("-" * 80)++ entity_degree = Counter()+ for edge in kg.edges:+ entity_degree[edge.startNode.name] += 1+ entity_degree[edge.endNode.name] += 1++ for entity, count in entity_degree.most_common(15):+ print(f" • {entity:<30} 参与 {count:2d} 个关系")++ # ============================================================================+ # Phase 3: 语义去重+ # ============================================================================+ print("\n" + "=" * 80)+ print("🔄 Phase 3: 语义去重 (使用 SemHash)")+ print("=" * 80)++ print(f"\n去重前: 节点 {len(kg.nodes)}, 关系 {len(kg.edges)}")+ print(" 执行语义相似度匹配 (threshold=0.8)...")++ kg.match_nodes_and_update_edges(threshold=0.8)++ print(f"去重后: 节点 {len(kg.nodes)}, 关系 {len(kg.edges)}")++ # 对比统计+ print("\n📊 去重后统计:")++ # 更新标签分布+ labels_after = Counter(n.label for n in kg.nodes)+ print(f" 实体类型分布: {dict(labels_after)}")++ # 更新关系类型分布+ relations_after = defaultdict(int)+ for edge in kg.edges:+ relations_after[edge.name] += 1+ print(f" 关系类型: {len(relations_after)} 种")++ # 显示去重后的节点+ print("\n" + "-" * 80)+ print(f"🏷️ 去重后的实体 ({len(kg.nodes)} 个):")+ print("-" * 80)+ for i, node in enumerate(kg.nodes, 1):+ print(f" {i:2d}. {node.name:<30} [类型: {node.label}]")++ # 显示去重后的关系(简要版)+ print("\n" + "-" * 80)+ print(f"🔗 去重后的关系 ({len(kg.edges)} 个 - 简要版):")+ print("-" * 80)++ sorted_edges_after = sorted(+ kg.edges,+ key=lambda e: e.t_start[0] if e.t_start else "9999-12-31"+ )++ for i, edge in enumerate(sorted_edges_after, 1):+ rel_str = f"{edge.startNode.name} --[{edge.name}]--> {edge.endNode.name}"++ time_info = ""+ if edge.t_start:+ time_info += f"(从 {edge.t_start[0]}"+ if edge.t_end:+ time_info += f" 到 {edge.t_end[0]})"+ else:+ time_info += ")"+ elif edge.t_end:+ time_info += f"(截止 {edge.t_end[0]})"++ print(f" {i:2d}. {rel_str:<65} {time_info}")++ # ============================================================================+ # 保存结果+ # ============================================================================+ print("\n" + "=" * 80)+ print("💾 保存结果...")+ print("=" * 80)++ try:+ output_dir = "temp/atom_demo"+ kg.dump(output_dir)+ print(f"✓ 结果已保存到 {output_dir}/")++ # 列出保存的文件+ import os+ saved_files = []+ if os.path.exists(output_dir):+ for root, dirs, files in os.walk(output_dir):+ for file in files:+ rel_path = os.path.relpath(os.path.join(root, file), output_dir)+ saved_files.append(rel_path)++ if saved_files:+ print("\n 保存的文件:")+ for f in sorted(saved_files):+ print(f" - {f}")+ except Exception as e:+ print(f"⚠ 警告: 无法保存结果: {str(e)}")++ print("\n" + "=" * 80)+ print("✨ Atom 演示完成!")+ print("=" * 80)+ print("\n📌 演示要点:")+ print(" ✓ 两阶段提取:原子事实 → 关系提取")+ print(" ✓ 时序追踪:产品发布日期、人物任职期间")+ print(" ✓ 证据追踪:每条关系都关联到源事实")+ print(" ✓ 语义去重:自动合并同义实体")+++if __name__ == "__main__":+ main()diff --git a/hyperextract/graphs/__init__.py b/hyperextract/graphs/__init__.pyindex cd4b7fb..bf69b20 100644--- a/hyperextract/graphs/__init__.py+++ b/hyperextract/graphs/__init__.py@@ -2,6 +2,7 @@ from .base import AutoGraphfrom .kg_gen import KG_Genfrom .itext2kg import iText2KGfrom .itext2kg_star import iText2KG_Star+from .atom import Atom-__all__ = ["AutoGraph", "KG_Gen", "iText2KG", "iText2KG_Star"]+__all__ = ["AutoGraph", "KG_Gen", "iText2KG", "iText2KG_Star", "Atom"]diff --git a/hyperextract/graphs/atom.py b/hyperextract/graphs/atom.pyindex 5559d03..b806b9b 100644--- a/hyperextract/graphs/atom.py+++ b/hyperextract/graphs/atom.py@@ -7,14 +7,14 @@ standardized triple-based knowledge graphs.Prompts and schemas are adapted from the original Atom implementation."""-from typing import Optional+from typing import Optional, Listfrom datetime import datetimefrom pydantic import BaseModel, Fieldfrom semhash import SemHashfrom langchain_core.language_models import BaseChatModelfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_core.embeddings import Embeddings-from ontomem.merger import MergeStrategy+from ontomem.merger import MergeStrategy, CustomRuleMergerfrom .base import AutoGraph, AutoGraphSchemafrom ..utils.logging import logger@@ -128,7 +128,7 @@ class EdgeSchema(BaseModel):"A list of exact string copies of the atomic facts or sentences from the source text that provide evidence for this relationship. ""Select the specific facts from the input context that justify this edge. ""Do NOT paraphrase or modify the text. Copy it exactly as it appears in the source."- )+ ),)@@ -138,6 +138,8 @@ class EdgeSchema(BaseModel):Atom_FACTOID_EXTRACTION_PROMPT = """+Observation Date: {observation_date}+You are an expert factoid extraction engine. Your primary function is to read a news paragraph and its associated observation date, and then decompose the text into a comprehensive list of atomic, self-contained, and temporally-grounded facts.## Task@@ -196,7 +198,7 @@ Given an input paragraph and an `observation_date`, generate a list of all disti"""Atom_EDGE_EXTRACTION_PROMPT = """-Observation Date: {obs_timestamp}+Observation Date: {observation_date}You are a precise knowledge extraction engine designed to distill unstructured text into a structured Knowledge Graph.Your goal is to extract all meaningful relationships (edges) between entities, while rigorously capturing their temporal bounds and grounding evidence.@@ -220,8 +222,8 @@ Extract a list of relationships where each relationship consists of:#### 2. Temporal Extraction (Critical)- **Format**: All dates must be in `YYYY-MM-DD` or `YYYY` format.- **Lists**: `t_start` and `t_end` must always be lists of strings. If no date is found, use an empty list `[]`.-- **Relative Time Resolution**: Calculate absolute dates based on the **Observation Date** ({obs_timestamp}).- - "last year" -> {obs_timestamp} year - 1 (Jan 1st)+- **Relative Time Resolution**: Calculate absolute dates based on the **Observation Date** ({observation_date}).+ - "last year" -> {observation_date} year - 1 (Jan 1st)- "a few months ago" -> Estimate conservatively based on context.- "currently" -> implies the relationship is active (no `t_end`).- **End Actions**: If the text says someone "left" or "stopped", capture this in `t_end` while keeping the relation positive (e.g., Relation: "works_at", t_end: ["2023-01-01"]).@@ -261,6 +263,25 @@ Extract a list of relationships where each relationship consists of:"""+# ============================================================================+# Merge Templates for LLM.CUSTOM_RULE Strategy+# ============================================================================+++EDGE_MERGE_RULE = """You are an intelligent data merging assistant.+You will receive a list of edges representing the exact same relationship (Same Subject -> Same Predicate -> Same Object).++Your task is to merge them into a SINGLE edge object exactly matching the schema.++Merge strategy:+1. **startNode/endNode**: Keep the node definitions consistent.+2. **name**: Keep the canonical predicate name (since they are already grouped by this key).+3. **t_start**: Collect ALL distinct start dates/times from all inputs into a single list. Remove exact duplicates.+4. **t_end**: Collect ALL distinct end dates/times from all inputs into a single list. Remove exact duplicates.+5. **t_obs**: Collect ALL distinct observation dates from all inputs into a single list.+6. **atomic_facts**: Merge all evidence strings into a single list. Remove exact duplicate strings.+"""+# ==============================================================================# 3. Atom Implementation - Inherits from AutoGraph# ==============================================================================@@ -300,6 +321,7 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):observation_date: str | None = None,chunk_size: int = 2000,chunk_overlap: int = 200,+ facts_per_chunk: int = 10,max_workers: int = 10,verbose: bool = False,):@@ -312,10 +334,12 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):If None, uses current date and time.chunk_size: Characters per chunkchunk_overlap: Overlapping characters between chunks+ facts_per_chunk: Max number of atomic facts to group into a single extraction batch (default: 20)max_workers: Max concurrent extraction workersverbose: Display detailed execution logs and progress information"""+ self.facts_per_chunk = facts_per_chunkself.observation_date = observation_date# 1. Define Key Extractors (critical for deduplication)@@ -328,6 +352,14 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):# 2. Edge consistency check: tell AutoGraph which nodes this edge connectsnodes_in_edge_fn = lambda x: (x.startNode.name, x.endNode.name)+ # Edge merger: merges relationship descriptions using EDGE_MERGE_RULE+ edge_merger = CustomRuleMerger(+ key_extractor=edge_key_fn,+ llm_client=llm_client,+ item_schema=EdgeSchema,+ rule=EDGE_MERGE_RULE,+ )+# 3. Call parent class initializationlogger.info("🚀 Initializing Atom")super().__init__(@@ -344,7 +376,7 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):prompt_for_edge_extraction=Atom_EDGE_EXTRACTION_PROMPT,# Configure deduplication strategynode_strategy_or_merger=MergeStrategy.KEEP_EXISTING,- edge_strategy_or_merger=MergeStrategy.KEEP_EXISTING,+ edge_strategy_or_merger=edge_merger,# Optimize indexing: only index name fieldnode_fields_for_index=["name", "label"],edge_fields_for_index=["startNode", "name", "endNode"],@@ -376,10 +408,10 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):Extracted and validated graph."""obs_date_str = self.observation_date or datetime.now().strftime("%Y-%m-%d")-+# ==================== Step 1: Extract Atomic Facts ====================logger.info("🔍 [Phase 1] Extracting Atomic Facts...")-+# 1. Prepare raw chunksif len(text) <= self.chunk_size:raw_chunks = [text]@@ -399,7 +431,7 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):{"chunk_text": chunk, "observation_date": obs_date_str}for chunk in raw_chunks]- chunk_fact_lists = fact_chain.batch(+ chunk_fact_lists: List[AtomicFactSchema] = fact_chain.batch(fact_inputs, config={"max_concurrency": self.max_workers})@@ -413,23 +445,53 @@ class Atom(AutoGraph[NodeSchema, EdgeSchema]):if not all_facts:logger.warning("⚠️ No facts extracted. Returning empty graph.")- return self._create_empty_instance().data+ return AutoGraphSchema()# ==================== Step 2: Extract Edges from Facts ====================logger.info("🔍 [Phase 2] Extracting Edges from Atomic Facts...")- # 5. Create new context from facts (join with newlines)- fact_context_text = "\n".join(all_facts)-- # 6. Split Fact Context into chunks- if len(fact_context_text) <= self.chunk_size:- fact_chunks = [fact_context_text]- else:- fact_chunks = self.text_splitter.split_text(fact_context_text)-- # 7. Batch Extract Edges directly from Fact Chunks+ # 5. Group Facts into Chunks (Controlled by count AND size)+ # This ensures each fact remains intact and fact IDs reset per chunk+ fact_chunks = []+ current_batch = []+ current_char_count = 0++ # Estimate overhead: "[Fact 99]: " + newline is approx 12 chars+ PREFIX_OVERHEAD = 12++ for fact in all_facts:+ fact_len = len(fact) + PREFIX_OVERHEAD++ # Check constraints:+ # 1. Size limit (fuzzy check to keep under chunk_size)+ # 2. Count limit (facts_per_chunk)+ size_limit_reached = (current_char_count + fact_len) > self.chunk_size+ count_limit_reached = len(current_batch) >= self.facts_per_chunk++ if (size_limit_reached or count_limit_reached) and current_batch:+ # Flush current batch+ formatted_chunk = "\n".join(+ [f"[Fact {i+1}]: {f}" for i, f in enumerate(current_batch)]+ )+ fact_chunks.append(formatted_chunk)++ # Reset+ current_batch = []+ current_char_count = 0++ current_batch.append(fact)+ current_char_count += fact_len++ # Flush remaining items+ if current_batch:+ formatted_chunk = "\n".join(+ [f"[Fact {i+1}]: {f}" for i, f in enumerate(current_batch)]+ )+ fact_chunks.append(formatted_chunk)++ # 6. Batch Extract Edges directly from Fact Chunks# Format the edge prompt with observation date- edge_prompt_with_date = self.edge_prompt.format(obs_timestamp=obs_date_str)+ edge_prompt_with_date = self.edge_prompt.format(observation_date=obs_date_str)edge_prompt_template = ChatPromptTemplate.from_template(f"{edge_prompt_with_date}\n\n### Source Text:\n{{chunk_text}}")diff --git a/hyperextract/hypergraphs/hyper_rag.py b/hyperextract/hypergraphs/hyper_rag.pyindex a37be29..d47b339 100644--- a/hyperextract/hypergraphs/hyper_rag.py+++ b/hyperextract/hypergraphs/hyper_rag.py@@ -154,7 +154,7 @@ class Hyper_RAG(AutoHypergraph[NodeSchema, EdgeSchema]):nodes_in_edge_fn = lambda x: x.participants# 3. Create custom Mergers with LLM.CUSTOM_RULE strategy- # Node merger: merges entity descriptions using NODE_MERGE_TEMPLATE+ # Node merger: merges entity descriptions using NODE_MERGE_RULEnode_merger = CustomRuleMerger(key_extractor=node_key_fn,llm_client=llm_client,@@ -162,7 +162,7 @@ class Hyper_RAG(AutoHypergraph[NodeSchema, EdgeSchema]):rule=NODE_MERGE_RULE,)- # Edge merger: merges relationship descriptions using EDGE_MERGE_TEMPLATE+ # Edge merger: merges relationship descriptions using EDGE_MERGE_RULEedge_merger = CustomRuleMerger(key_extractor=edge_key_fn,llm_client=llm_client,