Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
更新 .gitignore 文件,添加 tmp/ 目录;重构知识图谱相关类,合并 SimpleGraph 实现至 base.py,删除冗余文件。
yifanfeng9741cf1fcc8ccd
diff --git a/.gitignore b/.gitignoreindex f1f9d7d..c8fd3ff 100644--- a/.gitignore+++ b/.gitignore@@ -1,3 +1,5 @@+tmp/+# Byte-compiled / optimized / DLL files__pycache__/*.py[cod]diff --git a/README.md b/README.mdindex 6e2b753..b8dcb26 100644--- a/README.md+++ b/README.md@@ -1 +1,111 @@-Hyper-Extract\ No newline at end of file++# Hyper-Extract 🧠++> **An Intelligent, LLM-Driven Knowledge Container Framework.**+>+> 一个基于 LLM 的智能知识提取与演化框架。不仅仅是存储数据,它能“阅读”、“思考”并“进化”。++## 📖 简介 (Introduction)++**Hyper-Extract** 是一个重新定义的非结构化数据处理框架。与传统的 ETL 工具不同,Hyper-Extract 的核心单元是 **Knowledge Container(知识容器)**。++每个容器(如 `SimpleGraph`, `HyperGraph`)都持有一个 **LLM 实例**。它们不仅仅是被动的数据结构,而是具有生命周期的智能体:++1. **Extract (摄入)**: 主动利用 LLM 从文本中提取信息,并执行去重、对齐、合并。+2. **Evolve (演化)**: 对已有知识进行内省(Introspection)。包括推理新关系、剪枝错误节点、图卷积传播等。+3. **Dump (产出)**: 将清洗后的高质量知识导出为标准格式(JSON/NetworkX/Neo4j)。++## ✨ 核心特性 (Features)++* **LLM Native**: 容器初始化即绑定 LangChain 模型实例,所有操作皆由大模型驱动。+* **统一生命周期**: 所有知识类型遵循 `Extract` -> `Evolve` -> `Dump` 的标准范式。+* **多模态结构支持**:+ * **Graph**: 标准实体关系图谱 (Nodes + Edges)。+ * **Hypergraph**: 支持超边(Hyperedge)连接多个节点的复杂语义结构 (HyperRAG 实现)。+* **类型安全**: 全面基于 Pydantic Schema,确保结构化输出的稳定性。++## 🛠️ 安装 (Installation)++本项目使用 `uv` 进行包管理(也支持标准 pip)。++```bash+# 克隆项目+git clone https://github.com/your-username/hyper-extract.git+cd hyper-extract++# 使用 uv 安装依赖 (推荐)+uv sync++# 或者使用 pip+pip install -r requirements.txt+```++## 🚀 快速开始 (Quick Start)++### 1. 定义与初始化++```python+import asyncio+from langchain_openai import ChatOpenAI+from src.knowledge.graph.simple_graph import SimpleGraph++async def main():+ # 1. 准备大模型实例+ llm = ChatOpenAI(model="gpt-4o", temperature=0)++ # 2. 初始化智能容器+ # 容器就像一个空的大脑,准备好接收信息+ kg = SimpleGraph(llm=llm)++ # 3. Extract: 摄入知识+ text = "Elon Musk co-founded OpenAI in 2015. He later left the board in 2018."+ print(f"正在提取: {text}...")+ await kg.extract(text)++ # 4. Evolve: 知识演化 (可选)+ # 例如:推断隐含关系或清理孤立节点+ await kg.evolve()++ # 5. Dump: 导出结果+ data = kg.dump(format="json")+ print("最终知识图谱:", data)++if __name__ == "__main__":+ asyncio.run(main())+```++## 📂 项目结构 (Project Structure)++```text+hyper-extract/+├── src/+│ ├── core/ # 核心抽象定义+│ ├── driver/ # LLM 驱动与适配器+│ ├── schema/ # Pydantic 数据结构定义 (GraphSchema, etc.)+│ └── knowledge/ # 知识容器实现+│ ├── base.py # [核心] BaseKnowledge 基类+│ ├── graph/ # 普通图谱实现+│ │ ├── simple_graph.py # 基础图提取+│ │ └── kg_gen_graph.py # 复杂生成式图谱+│ └── hypergraph/ # 超图实现+│ ├── simple_hypergraph.py+│ └── hyperrag_hypergraph.py+├── tests/ # 测试用例+├── main.py # 入口示例+├── pyproject.toml # 项目配置+└── uv.lock # 依赖锁定+```++## 🧩 扩展指南 (Development)++如果你想添加一种新的知识结构(例如“思维导图”):++1. 继承 `src.knowledge.base.BaseKnowledge`。+2. 定义你的数据 Schema (Pydantic)。+3. 实现 **`extract`**: 定义如何用 Prompt 将文本转化为你的 Schema。+4. 实现 **`evolve`** (可选): 定义如何优化你的数据结构。+5. 实现 **`dump`**: 定义导出格式。++## 📄 License++MIT License\ No newline at end of filediff --git a/src/knowledge/graph/base.py b/src/knowledge/graph/base.pyindex e69de29..5999216 100644--- a/src/knowledge/graph/base.py+++ b/src/knowledge/graph/base.py@@ -0,0 +1,80 @@+from ..base import BaseKnowledge+from typing import List, Optional, Dict, Any+from pydantic import BaseModel, Field+++class Entity(BaseModel):+ """Represents a node in the knowledge graph."""++ id: str = Field(+ ...,+ description="The unique name or identifier of the entity (e.g., 'Elon Musk').",+ )+ type: str = Field(+ "Unknown",+ description="The category of the entity (e.g., 'Person', 'Organization', 'Concept').",+ )+ description: Optional[str] = Field(+ None, description="A brief summary of the entity found in the text."+ )+ # Using a dict allows storing extra attributes without changing the schema structure+ properties: Dict[str, Any] = Field(+ default_factory=dict,+ description="Additional attributes (e.g., {'age': 50, 'status': 'active'}).",+ )+++class Relationship(BaseModel):+ """Represents an edge between two entities."""++ source: str = Field(..., description="The id of the source entity.")+ target: str = Field(..., description="The id of the target entity.")+ type: str = Field(+ ..., description="The type of the relationship (e.g., 'FOUNDED', 'LOCATED_IN')."+ )+ description: Optional[str] = Field(+ None, description="Contextual details about this relationship."+ )+ weight: float = Field(+ 1.0,+ description="The confidence score or strength of the connection (0.0 to 1.0).",+ )+++class GraphSchema(BaseModel):+ """The container for the entire graph structure."""++ nodes: List[Entity] = Field(default_factory=list)+ edges: List[Relationship] = Field(default_factory=list)++ def get_node_ids(self) -> set[str]:+ """Helper to get all current node IDs for quick lookup."""+ return {n.id for n in self.nodes}+++class SimpleGraph(BaseKnowledge[GraphSchema]):+ def _get_schema_class(self):+ return GraphSchema++ async def extract(self, text: str, **kwargs):+ # 1. 构造 prompt+ prompt = f"Extract nodes and edges from: {text}"++ # 2. 调用自带的 LLM+ response = await self.llm.ainvoke(prompt)++ # 3. 解析结果 (假设 parsed_data 是解析后的 GraphSchema 对象)+ parsed_data = self._parse(response)++ # 4. 【直接在这里做合并逻辑】+ # 简单的去重合并+ existing_node_ids = {n.id for n in self.data.nodes}+ for node in parsed_data.nodes:+ if node.id not in existing_node_ids:+ self.data.nodes.append(node)+ self.data.edges.extend(parsed_data.edges)++ def dump(self, format="json"):+ if format == "json":+ return self.data.model_dump_json()+ return self.data.model_dump()diff --git a/src/knowledge/graph/simple_graph.py b/src/knowledge/graph/simple_graph.pydeleted file mode 100644index 5999216..0000000--- a/src/knowledge/graph/simple_graph.py+++ /dev/null@@ -1,80 +0,0 @@-from ..base import BaseKnowledge-from typing import List, Optional, Dict, Any-from pydantic import BaseModel, Field---class Entity(BaseModel):- """Represents a node in the knowledge graph."""-- id: str = Field(- ...,- description="The unique name or identifier of the entity (e.g., 'Elon Musk').",- )- type: str = Field(- "Unknown",- description="The category of the entity (e.g., 'Person', 'Organization', 'Concept').",- )- description: Optional[str] = Field(- None, description="A brief summary of the entity found in the text."- )- # Using a dict allows storing extra attributes without changing the schema structure- properties: Dict[str, Any] = Field(- default_factory=dict,- description="Additional attributes (e.g., {'age': 50, 'status': 'active'}).",- )---class Relationship(BaseModel):- """Represents an edge between two entities."""-- source: str = Field(..., description="The id of the source entity.")- target: str = Field(..., description="The id of the target entity.")- type: str = Field(- ..., description="The type of the relationship (e.g., 'FOUNDED', 'LOCATED_IN')."- )- description: Optional[str] = Field(- None, description="Contextual details about this relationship."- )- weight: float = Field(- 1.0,- description="The confidence score or strength of the connection (0.0 to 1.0).",- )---class GraphSchema(BaseModel):- """The container for the entire graph structure."""-- nodes: List[Entity] = Field(default_factory=list)- edges: List[Relationship] = Field(default_factory=list)-- def get_node_ids(self) -> set[str]:- """Helper to get all current node IDs for quick lookup."""- return {n.id for n in self.nodes}---class SimpleGraph(BaseKnowledge[GraphSchema]):- def _get_schema_class(self):- return GraphSchema-- async def extract(self, text: str, **kwargs):- # 1. 构造 prompt- prompt = f"Extract nodes and edges from: {text}"-- # 2. 调用自带的 LLM- response = await self.llm.ainvoke(prompt)-- # 3. 解析结果 (假设 parsed_data 是解析后的 GraphSchema 对象)- parsed_data = self._parse(response)-- # 4. 【直接在这里做合并逻辑】- # 简单的去重合并- existing_node_ids = {n.id for n in self.data.nodes}- for node in parsed_data.nodes:- if node.id not in existing_node_ids:- self.data.nodes.append(node)- self.data.edges.extend(parsed_data.edges)-- def dump(self, format="json"):- if format == "json":- return self.data.model_dump_json()- return self.data.model_dump()diff --git a/src/knowledge/hypergraph/simple_hypergraph.py b/src/knowledge/hypergraph/simple_hypergraph.pydeleted file mode 100644index e69de29..0000000