Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

Refactor AutoList, AutoModel, and AutoSet classes for improved functionality and clarity

yifanfeng975b2ade887980
diff --git a/README.md b/README.md
index 835f34a..7d78bd7 100644
--- a/README.md
+++ b/README.md
@@ -1,19 +1,19 @@
# Hyper-Extract 🧠✨
-> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**
+> **An Intelligent, LLM-Powered Knowledge Extraction Framework**
>
-> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.
+> Transform unstructured text into structured Auto containers with the power of Large Language Models.
---
## 🎯 Vision
-**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.
+**Hyper-Extract** provides a powerful framework for intelligent knowledge extraction. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, and index knowledge from unstructured text.
Every Auto container in Hyper-Extract follows a unified lifecycle:
1. **Extract** - Intelligent extraction from text with automatic chunking and merging
-2. **Evolve** - Self-refinement through reasoning, pruning, and optimization
+2. **Merge** - Automatic aggregation with configurable merge strategies
3. **Index** - Semantic vector search capabilities
4. **Serialize** - Persist and reload knowledge with full fidelity
@@ -29,7 +29,7 @@ Every Auto container in Hyper-Extract follows a unified lifecycle:
### 🏗️ Unified Architecture
- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types
- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility
-- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline
+- **Lifecycle Management**: Standardized `extract` → `merge` → `search` → `dump/load` pipeline
### 🔍 Semantic Search
- **FAISS Integration**: Fast vector similarity search for knowledge retrieval
diff --git a/docs/index.md b/docs/index.md
index d42b0a8..519a4e6 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,19 +1,19 @@
# Hyper-Extract 🧠✨
-> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**
+> **An Intelligent, LLM-Powered Knowledge Extraction Framework**
>
-> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.
+> Transform unstructured text into structured Auto containers with the power of Large Language Models.
---
## 🎯 Vision
-**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.
+**Hyper-Extract** provides a powerful framework for intelligent knowledge extraction. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, and index knowledge from unstructured text.
Every Auto container in Hyper-Extract follows a unified lifecycle:
1. **Extract** - Intelligent extraction from text with automatic chunking and merging
-2. **Evolve** - Self-refinement through reasoning, pruning, and optimization
+2. **Merge** - Automatic aggregation with configurable merge strategies
3. **Index** - Semantic vector search capabilities
4. **Serialize** - Persist and reload knowledge with full fidelity
@@ -29,7 +29,7 @@ Every Auto container in Hyper-Extract follows a unified lifecycle:
### 🏗️ Unified Architecture
- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types
- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility
-- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline
+- **Lifecycle Management**: Standardized `extract` → `merge` → `search` → `dump/load` pipeline
### 🔍 Semantic Search
- **FAISS Integration**: Fast vector similarity search for knowledge retrieval
diff --git a/docs/zh/index.md b/docs/zh/index.md
index af158a4..fe6d99f 100644
--- a/docs/zh/index.md
+++ b/docs/zh/index.md
@@ -1,19 +1,19 @@
# Hyper-Extract 🧠✨
-> **一个智能的、基于 LLM 的知识提取与演化框架**
+> **一个智能的、基于 LLM 的知识提取框架**
>
-> 借助大语言模型的力量,将非结构化文本转化为结构化、可演化的知识容器。
+> 借助大语言模型的力量,将非结构化文本转化为结构化的知识容器。
---
## 🎯 愿景
-**Hyper-Extract** 将知识提取重新定义为一个智能的、自我演化的过程。我们不是将数据结构视为被动容器,而是赋予它们 **LLM 驱动的智能**,使其能够主动地从非结构化文本中提取、合并、演化和索引知识。
+**Hyper-Extract** 提供一个强大的智能知识提取框架。我们不是将数据结构视为被动容器,而是赋予它们 **LLM 驱动的智能**,使其能够主动地从非结构化文本中提取、合并和索引知识。
Hyper-Extract 中的每个知识容器都遵循统一的生命周期:
1. **Extract(提取)** - 使用 LLM 从文本智能提取,自动处理分块和合并
-2. **Evolve(演化)** - 通过推理、剪枝和优化实现自我优化
+2. **Merge(合并)** - 通过可配置的合并策略实现自动聚合
3. **Index(索引)** - 提供语义向量搜索能力
4. **Serialize(序列化)** - 完整保存和加载知识状态
@@ -29,7 +29,7 @@ Hyper-Extract 中的每个知识容器都遵循统一的生命周期:
### 🏗️ 统一架构
- **抽象基类**:`BaseAutoType[T]` 为所有知识类型提供一致的接口
- **泛型类型支持**:使用 `Generic[T]` 实现完整的 Python 类型提示
-- **生命周期管理**:标准化的 `extract` → `merge` → `evolve` → `search` → `dump/load` 流程
+- **生命周期管理**:标准化的 `extract` → `merge` → `search` → `dump/load` 流程
### 🔍 语义搜索
- **FAISS 集成**:快速的向量相似度搜索
diff --git a/examples/auto_list_demo.py b/examples/auto_list_demo.py
index 49e9384..6c0d07c 100644
--- a/examples/auto_list_demo.py
+++ b/examples/auto_list_demo.py
@@ -21,7 +21,6 @@ from hyperextract import AutoList
class NewsEvent(BaseModel):
"""新闻事件结构"""
- event_id: Optional[str] = Field(default=None, description="事件唯一标识")
title: str = Field(description="事件标题")
date: str = Field(description="事件发生日期")
location: str = Field(description="事件发生地点")
@@ -151,15 +150,15 @@ def main():
show_progress=True,
)
- # 4. 提取知识 (默认存储模式)
- print("\n[4] 提取新闻事件列表 (默认存储模式)...")
+ # 4. 提取知识 (使用 feed 模式)
+ print("\n[4] 提取新闻事件列表 (使用 feed 模式)...")
print("-" * 80)
- events = a_event_list.extract(news_text)
+ a_event_list.feed(news_text)
print("-" * 80)
# 5. 显示提取结果
- print(f"\n[5] 提取到 {len(events)} 个新闻事件:")
- for i, event in enumerate(events[:5], 1): # 只显示前5个
+ print(f"\n[5] 提取到 {len(a_event_list.items)} 个新闻事件:")
+ for i, event in enumerate(a_event_list.items[:5], 1): # 只显示前5个
print(f"\n 事件 {i}:")
print(f" 标题: {event.title}")
print(f" 日期: {event.date}")
@@ -169,8 +168,8 @@ def main():
if event.impact:
print(f" 影响: {event.impact[:60]}...")
- if len(events) > 5:
- print(f"\n ... 还有 {len(events) - 5} 个事件未显示")
+ if len(a_event_list) > 5:
+ print(f"\n ... 还有 {len(a_event_list) - 5} 个事件未显示")
# 6. 测试列表大小
print(f"\n[6] 知识列表大小: {len(a_event_list)} 个事件")
@@ -194,8 +193,8 @@ def main():
print(f" [{i}] {event.title} ({event.date})")
print(f" 地点: {event.location}")
- # 9. 测试增量提取(默认合并模式)
- print("\n[9] 测试增量提取(默认合并模式)...")
+ # 9. 测试增量提取(feed 模式)
+ print("\n[9] 测试增量提取(feed 模式)...")
additional_news = """
### 事件11: 华为发布6G测试网络
2024年6月15日,华为在深圳总部宣布建成全球首个6G技术测试网络。
@@ -205,7 +204,7 @@ def main():
print(" 添加补充新闻...")
before_count = len(a_event_list)
- a_event_list.extract(additional_news) # 默认 store=True,自动合并
+ a_event_list.feed(additional_news) # 使用 feed 自动合并
after_count = len(a_event_list)
print(f" 合并前: {before_count} 个事件")
print(f" 合并后: {after_count} 个事件")
diff --git a/examples/auto_model_demo.py b/examples/auto_model_demo.py
index f85cd02..32aa31f 100644
--- a/examples/auto_model_demo.py
+++ b/examples/auto_model_demo.py
@@ -161,10 +161,11 @@ def main():
show_progress=True,
)
- # 4. 提取知识 (默认存储模式)
- print("\n[4] 提取知识 (默认存储模式)...")
+ # 4. 提取知识 (使用 feed 模式)
+ print("\n[4] 提取知识 (使用 feed 模式)...")
print("-" * 80)
- extracted_data = a_document_model.extract(document)
+ a_document_model.feed(document)
+ extracted_data = a_document_model.data
print("-" * 80)
# 5. 显示提取结果
@@ -203,15 +204,15 @@ def main():
field_value = result[field_name]
print(f" [{i}] {field_name}: {str(field_value)[:80]}...")
- # 9. 测试增量提取(默认合并模式)
- print("\n[9] 测试增量提取(默认合并模式)...")
+ # 9. 测试增量提取(feed 模式)
+ print("\n[9] 测试增量提取(feed 模式)...")
additional_text = """
补充信息:
本研究获得了国家自然科学基金支持,项目编号:NSFC-2023-12345。
研究团队来自清华大学医学院和北京协和医院。
"""
print(" 添加补充文本...")
- a_document_model.extract(additional_text) # 默认 store=True,自动合并
+ a_document_model.feed(additional_text) # 使用 feed 自动合并
print(f" 更新后的作者信息: {a_document_model.data.author}")
# 10. 保存知识
diff --git a/examples/auto_set_demo.py b/examples/auto_set_demo.py
index 05d3252..afa7b70 100644
--- a/examples/auto_set_demo.py
+++ b/examples/auto_set_demo.py
@@ -1,7 +1,7 @@
-"""Set Knowledge Demo - Extracting unique skills from multiple job descriptions
+"""集合知识示例 - 从冒险者日志中提取和去重怪物信息
-This example demonstrates how AutoSet automatically deduplicates
-extracted items based on a unique key field.
+这个示例演示了AutoSet如何基于唯一键字段自动去重提取的项目。
+使用RPG怪物图鉴场景展示LLM_MERGE策略如何智能合并来自多个来源的冲突信息。
"""
import sys
@@ -12,6 +12,8 @@ project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))
import os
+import io
+# sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
@@ -19,199 +21,246 @@ from hyperextract import AutoSet
from hyperextract.utils.merger import MergeStrategy
-# Define the schema for skills
-class SkillSchema(BaseModel):
- """A skill mentioned in job descriptions"""
+# 定义怪物信息的数据模型
+class MonsterEntry(BaseModel):
+ """RPG游戏中的怪物信息"""
name: str = Field(
- description="Name of the skill (e.g., 'Python', 'Machine Learning')"
+ description="怪物的名称(例如:'地精小卒'、'洞穴巨魔')"
)
- category: str | None = Field(
+ species: str | None = Field(
None,
- description="Category (e.g., 'Programming Language', 'Framework', 'Soft Skill')",
+ description="怪物的种族分类(例如:'类人型'、'巨人'、'野兽')",
)
- proficiency_level: str | None = Field(
+ danger_level: int | None = Field(
None,
- description="Required proficiency level if mentioned (e.g., 'Expert', 'Intermediate')",
+ description="危险等级(1-10),用于评估战斗难度",
)
- description: str | None = Field(
- None, description="Brief description or context of how the skill is used"
+ habitats: str | None = Field(
+ None,
+ description="怪物常见的栖息地或活动场所"
+ )
+ characteristics: str | None = Field(
+ None,
+ description="怪物的外观特征和行为习性描述"
+ )
+ weaknesses: str | None = Field(
+ None,
+ description="怪物对特定伤害类型或攻击方式的弱点"
+ )
+ loot_drops: str | None = Field(
+ None,
+ description="击败怪物可能掉落的物品和资源"
)
def main():
- # Initialize LLM and embeddings
+ # 初始化LLM和嵌入模型
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(base_url=os.getenv("OPENAI_BASE_URL"))
- # Create AutoSet with FIELD_MERGE strategy
- # This means when the same skill appears multiple times,
- # the new extraction will fill in any missing fields from the existing one
- a_skill_set = AutoSet(
- item_schema=SkillSchema,
+ # 创建AutoSet,使用LLM_MERGE策略
+ # 当同一个怪物在多个日志中出现时,LLM会智能合并冲突的信息
+ monster_codex = AutoSet(
+ item_schema=MonsterEntry,
llm_client=llm,
embedder=embedder,
- key_extractor=lambda x: x.name, # Deduplicate based on skill name
- merge_item_strategy=MergeStrategy.FIELD_MERGE,
- prompt="""Extract all technical skills, frameworks, tools, and soft skills mentioned in the text.
-
-For each skill, provide:
-- name: The exact name of the skill
-- category: What type of skill it is
-- proficiency_level: If mentioned, the required level
-- description: Brief context about how it's used
+ key_extractor=lambda x: x.name, # 基于怪物名称去重
+ merge_item_strategy=MergeStrategy.LLM_MERGE,
+ prompt="""从冒险者的日志记录中提取所有怪物的相关信息。
+
+对于每只怪物,请提供:
+- name: 怪物的确切名称
+- species: 怪物的种族分类(类人型、野兽、不死生物等)
+- danger_level: 战斗危险等级(1-10)
+- habitats: 常见的栖息地或地理位置
+- characteristics: 详细的外观特征和行为描述
+- weaknesses: 已知的弱点和克制方法
+- loot_drops: 击败后可能掉落的物品
-Be specific and extract all mentioned skills.""",
+要尽可能准确和详细,包括所有提到的怪物和它们的属性。如果信息不完整,请基于已有信息做出合理推断。""",
)
- # Sample job descriptions (with overlapping skills)
- job_descriptions = [
+ # 来自冒险者的日志 - 包含重复的怪物但信息略有不同
+ adventurer_logs = [
"""
- Senior Python Developer
+ 【第一个冒险日记 - 林地远征】
+ 日期:秋月15日
+
+ 今天在幽暗森林中遭遇了几种怪物。首先遇到了一群地精小卒,他们大约有3-4英尺高,绿皮肤,
+ 穿着破烂的皮盔甲。这些家伙相当狡猾,喜欢群体狩猎。危险等级大概是2-3。它们掉落了
+ 一些铜币和破旧的匕首。
- We are seeking an expert Python developer with 5+ years of experience.
+ 在森林深处,我还遇到了一只洞穴巨魔。这个家伙大约8英尺高,皮肤呈灰褐色,力量惊人。
+ 对火焰特别敏感——我用火焰咒语成功地逼退了它。危险等级至少是7。它的巢穴里有一些金币。
- Requirements:
- - Expert level Python programming
- - Strong experience with FastAPI framework for building REST APIs
- - Proficiency in PostgreSQL database design and optimization
- - Experience with Docker for containerization
- - Strong problem-solving and communication skills
- - Experience with Git version control
+ 还有森林狼群,非常聪明的野兽,它们似乎有高度的群体智能。危险等级4-5。
""",
+
"""
- Full Stack Engineer
+ 【第二个冒险日记 - 深矿探险】
+ 日期:秋月18日
+
+ 在深矿洞中进行了危险的探险。地精小卒在这里也有出现!看来他们在地下还有殖民地。
+ 这次的地精穿着更好的装备,显然这是地精的精英部队。他们配备了铁制短剑,训练更充分。
+ 危险等级可能达到了3-4。掉落了更多的铜币和一些魔晶。
- Join our dynamic team as a Full Stack Engineer!
+ 还有一种我之前没遇过的怪物——地下爬行虫。这些生物有分节的身体,长约2-3米,
+ 用触须来感知周围环境。它们不是很聪明,但速度很快。危险等级3。掉落虫壳碎片。
- Required Skills:
- - Python for backend development (3+ years)
- - React for frontend development
- - Experience with FastAPI or Django
- - PostgreSQL or MySQL database experience
- - Docker and Kubernetes for deployment
- - Excellent teamwork and communication abilities
- - Git workflow knowledge
+ 巨魔在深处也出现了,但这次的巨魔看起来比森林里的那只更大更凶悍。
+ 它对寒冰魔法有抵抗力,但仍然惧怕火焰。危险等级估计8级。
""",
+
"""
- Machine Learning Engineer
+ 【第三个冒险日记 - 废墟遗迹】
+ 日期:秋月22日
- Looking for an ML Engineer to join our AI team.
+ 在古老遗迹的探索中发现了新的敌人类型。最常见的是地精小卒的变种——遗迹守卫。
+ 他们看起来是被之前的文明改造过的,具有更强的战斗能力。有人说他们被魔法增强过。
+ 危险等级跳到了4-5。这些怪物掉落了古代金币和遗迹碎片。
- Must Have:
- - Advanced Python skills for ML development
- - PyTorch or TensorFlow experience
- - Strong understanding of machine learning algorithms
- - Experience with Docker for model deployment
- - PostgreSQL for data storage
- - Git for code versioning
- - Strong analytical and problem-solving skills
- """,
+ 还有一个更大的怪物——遗迹看守者,是一只3米高的怪物。它有三只眼睛,可能具有某种魔法能力。
+ 非常危险!危险等级至少9级。击败它时获得了一个古老的魔晶。
+
+ 巨魔在这里也有据点,防守非常严密。这些可能是智商更高的个体。
+ 它们使用简单的工具和陷阱来狩猎冒险者。危险等级8-9。
+ """
]
print("=" * 80)
- print("SET KNOWLEDGE DEMO: Extracting Unique Skills from Job Descriptions")
+ print("集合知识演示:从冒险者日志中构建怪物图鉴")
print("=" * 80)
- # Extract skills from all job descriptions
- print("\n📄 Processing job descriptions...")
- for i, job_desc in enumerate(job_descriptions, 1):
- print(f"\n Processing job description {i}...")
- a_skill_set.extract(job_desc)
- print(f" ✓ Total unique skills so far: {len(a_skill_set)}")
+ # 从所有日志中提取怪物信息
+ print("\n[*] 正在处理冒险者日志...")
+ for i, log in enumerate(adventurer_logs, 1):
+ print(f"\n 处理日志 {i}...")
+ monster_codex.feed(log) # 喂养数据
+ print(f" √ 目前已识别的怪物数量:{len(monster_codex)}")
- # Display all unique skills
+ # 显示所有提取的怪物
print("\n" + "=" * 80)
- print(f"📊 EXTRACTED UNIQUE SKILLS: {len(a_skill_set)} total")
+ print(f"[*] 提取的怪物统计:共 {len(monster_codex)} 种不同的怪物")
print("=" * 80)
- # Group by category
- skills_by_category = {}
- for skill in a_skill_set.items:
- category = skill.category or "Uncategorized"
- if category not in skills_by_category:
- skills_by_category[category] = []
- skills_by_category[category].append(skill)
-
- # Display organized results
- for category, category_skills in sorted(skills_by_category.items()):
- print(f"\n{category}:")
- for skill in sorted(category_skills, key=lambda s: s.name):
- print(f" • {skill.name}")
- if skill.proficiency_level:
- print(f" Level: {skill.proficiency_level}")
- if skill.description:
- print(f" Context: {skill.description}")
-
- # Demonstrate set operations
+ # 按危险等级分组显示
+ monsters_by_danger = {}
+ for monster in monster_codex.items:
+ level = monster.danger_level or 0
+ if level not in monsters_by_danger:
+ monsters_by_danger[level] = []
+ monsters_by_danger[level].append(monster)
+
+ # 从低到高显示怪物
+ print("\n[怪物按危险等级分类]\n")
+ for level in sorted(monsters_by_danger.keys()):
+ level_monsters = monsters_by_danger[level]
+ print(f"[!] 危险等级 {level}:")
+ for monster in sorted(level_monsters, key=lambda m: m.name):
+ print(f"\n ● 怪物名称:{monster.name}")
+ if monster.species:
+ print(f" 种族:{monster.species}")
+ if monster.habitats:
+ print(f" 栖息地:{monster.habitats}")
+ if monster.characteristics:
+ print(f" 特征:{monster.characteristics}")
+ if monster.weaknesses:
+ print(f" 弱点:{monster.weaknesses}")
+ if monster.loot_drops:
+ print(f" 掉落物:{monster.loot_drops}")
+
+ # 演示集合操作
print("\n" + "=" * 80)
- print("🔍 DEMONSTRATING SET OPERATIONS")
+ print("[*] 演示集合操作和语义搜索")
print("=" * 80)
- # Build index for semantic search
- print("\nBuilding vector index for semantic search...")
- a_skill_set.build_index()
- print("✓ Index built successfully")
-
- # Search for related skills
- print("\n1. Semantic Search - Finding skills related to 'backend development':")
- backend_skills = a_skill_set.search("backend development", top_k=5)
- for skill in backend_skills:
- print(f" • {skill.name}")
- if skill.category:
- print(f" Category: {skill.category}")
-
- # Check if specific skills exist
- print("\n2. Membership Testing:")
- test_skills = ["Python", "Java", "Docker", "Rust"]
- for skill_name in test_skills:
- exists = a_skill_set.contains(skill_name)
- status = "✓" if exists else "✗"
- print(f" {status} {skill_name}: {'Found' if exists else 'Not found'}")
-
- # Get specific skill details
- print("\n3. Get Specific Skill Details:")
- python_skill = a_skill_set.get("Python")
- if python_skill:
- print(f" Skill: {python_skill.name}")
- print(f" Category: {python_skill.category}")
- print(f" Level: {python_skill.proficiency_level or 'Not specified'}")
- print(f" Description: {python_skill.description or 'No description'}")
-
- # Save the extracted knowledge
+ # 构建向量索引以支持语义搜索
+ print("\n正在构建向量索引...")
+ monster_codex.build_index()
+ print("√ 索引构建成功")
+
+ # 语义搜索示例
+ print("\n[1] 语义搜索 - 寻找掉落金币的怪物:")
+ treasure_monsters = monster_codex.search("掉落金币财宝", top_k=5)
+ for monster in treasure_monsters:
+ print(f" • {monster.name}")
+ if monster.loot_drops:
+ print(f" 掉落物:{monster.loot_drops}")
+
+ print("\n[2] 语义搜索 - 寻找怕火焰的怪物:")
+ fire_weak_monsters = monster_codex.search("害怕火焰弱点火", top_k=5)
+ for monster in fire_weak_monsters:
+ print(f" • {monster.name}")
+ if monster.weaknesses:
+ print(f" 弱点:{monster.weaknesses}")
+
+ print("\n[3] 语义搜索 - 寻找在地下出现的怪物:")
+ underground_monsters = monster_codex.search("地下洞穴矿井", top_k=5)
+ for monster in underground_monsters:
+ print(f" • {monster.name}")
+ if monster.habitats:
+ print(f" 栖息地:{monster.habitats}")
+
+ # 检查特定怪物是否存在
+ print("\n[4] 成员检验 - 检查特定怪物是否在图鉴中:")
+ check_monsters = ["地精小卒", "洞穴巨魔", "龙", "骷髅战士"]
+ for monster_name in check_monsters:
+ exists = monster_codex.contains(monster_name)
+ status = "√" if exists else "×"
+ print(f" {status} {monster_name}:{'已记录' if exists else '未记录'}")
+
+ # 获取特定怪物的详细信息
+ print("\n[5] 查询特定怪物详情:")
+ goblin = monster_codex.get("地精小卒")
+ if goblin:
+ print(f" 怪物名称:{goblin.name}")
+ print(f" 种族:{goblin.species or '未知'}")
+ print(f" 危险等级:{goblin.danger_level or '未知'}/10")
+ print(f" 栖息地:{goblin.habitats or '未知'}")
+ print(f" 特征:{goblin.characteristics or '无'}")
+ print(f" 弱点:{goblin.weaknesses or '无已知弱点'}")
+ print(f" 掉落物:{goblin.loot_drops or '无'}")
+
+ # 保存提取的知识
print("\n" + "=" * 80)
- print("💾 SAVING EXTRACTED KNOWLEDGE")
+ print("[*] 保存怪物图鉴")
print("=" * 80)
- save_path = "tmp/auto_set_demo"
- a_skill_set.dump(save_path)
- print(f"✓ Saved to: {save_path}")
- print(" - state.json (structured data)")
- print(" - faiss_index/ (vector index for semantic search)")
+ save_path = "tmp/monster_codex"
+ monster_codex.dump(save_path)
+ print(f"√ 已保存到:{save_path}")
+ print(" - state.json(结构化数据)")
+ print(" - faiss_index/(向量索引)")
- # Demonstrate loading
- print("\n📂 Loading saved knowledge...")
- a_loaded_skill_set = AutoSet(
- item_schema=SkillSchema,
+ # 演示加载功能
+ print("\n[*] 加载已保存的怪物图鉴...")
+ loaded_codex = AutoSet(
+ item_schema=MonsterEntry,
llm_client=llm,
embedder=embedder,
key_extractor=lambda x: x.name,
)
- a_loaded_skill_set.load(save_path)
- print(f"✓ Loaded {len(a_loaded_skill_set)} unique skills")
+ loaded_codex.load(save_path)
+ print(f"√ 加载成功!图鉴中有 {len(loaded_codex)} 只怪物")
+
+ # 显示已加载的怪物列表
+ print("\n[已加载怪物列表]")
+ for monster in loaded_codex.items:
+ print(f" • {monster.name} (危险等级:{monster.danger_level}/10)")
- # Summary
+ # 总结
print("\n" + "=" * 80)
- print("📈 SUMMARY")
+ print("[*] 演示总结")
print("=" * 80)
- print(f" Total unique skills extracted: {len(a_skill_set)}")
- print(f" Job descriptions processed: {len(job_descriptions)}")
- print(" Deduplication strategy: FIELD_MERGE")
- print(" Key extractor: lambda x: x.name")
- print("\n ✓ Automatically deduplicated based on skill names")
- print(" ✓ Merged information from multiple occurrences")
- print(" ✓ Supports semantic search across all skills")
- print(" ✓ Persistent storage with dump/load")
+ print(f" 提取的不同怪物数量:{len(monster_codex)}")
+ print(f" 处理的冒险者日志数:{len(adventurer_logs)}")
+ print(" 去重策略:LLM_MERGE(智能合并)")
+ print(" 去重键:lambda x: x.name")
+ print("\n √ 基于怪物名称自动去重")
+ print(" √ 智能合并多个来源的冲突信息")
+ print(" √ 支持所有怪物的语义搜索")
+ print(" √ 持久化存储和加载")
print("=" * 80)
diff --git a/hyperextract/core/base.py b/hyperextract/core/base.py
index ba1fdc4..6672854 100644
--- a/hyperextract/core/base.py
+++ b/hyperextract/core/base.py
@@ -2,6 +2,7 @@ from abc import ABC, abstractmethod
from pathlib import Path
from typing import TypeVar, Generic, Any, Dict, Type, List
from datetime import datetime
+import json
from pydantic import BaseModel
from langchain_core.language_models.chat_models import BaseChatModel
from langchain_core.embeddings import Embeddings
@@ -16,7 +17,7 @@ T = TypeVar("T", bound=BaseModel)
class BaseAutoType(ABC, Generic[T]):
- """Unified knowledge base class integrating extraction, storage, aggregation, and evolution.
+ """Unified knowledge base class integrating extraction, storage, and aggregation.
This abstract base class provides a complete framework for managing structured knowledge
extracted from text. It handles the full lifecycle from extraction to serialization.
@@ -26,7 +27,6 @@ class BaseAutoType(ABC, Generic[T]):
- Automatically handle long text chunking and parallel processing
- Store and aggregate extracted knowledge with configurable merge strategies
- Build and maintain vector indices for semantic search
- - Support knowledge evolution and optimization
- Provide serialization and deserialization capabilities
"""
@@ -46,7 +46,7 @@ class BaseAutoType(ABC, Generic[T]):
Args:
data_schema: Pydantic BaseModel subclass defining the knowledge structure.
- llm_client: Language model client for extraction and evolution.
+ llm_client: Language model client for extraction.
embedder: Embedding model for semantic search and similarity computation.
prompt: Custom prompt template for extraction (defaults to generic prompt).
chunk_size: Maximum chunk size for splitting long texts.
@@ -87,6 +87,9 @@ class BaseAutoType(ABC, Generic[T]):
"updated_at": datetime.now(),
}
+ # Initialize internal state (calls hook for subclass setup)
+ self._init_internal_state()
+
def _create_new_instance(self) -> "BaseAutoType[T]":
"""Creates a new instance with the same configuration as this one.
@@ -137,11 +140,49 @@ class BaseAutoType(ABC, Generic[T]):
"""
return self._data
+ # ==================== State Management Lifecycle Hooks ====================
+
+ def _init_internal_state(self) -> None:
+ """
+ Protected hook to initialize internal state.
+ Called at the END of __init__ to ensure all basic attributes are set first.
+
+ Subclasses can override to initialize their own structures (e.g., _items_dict for AutoSet).
+ """
+ self._data = self._data_schema()
+ self._index = None
+
+ def _set_internal_state(self, data: T) -> None:
+ """
+ Protected hook to update internal state.
+ Called whenever data is modified (extract, feed, load).
+
+ Responsibilities:
+ 1. Update self._data
+ 2. Invalidate vector index (data changed)
+ 3. Subclasses override to sync auxiliary structures
+
+ Args:
+ data: The new data object to set.
+ """
+ self._data = data
+ self.clear_index()
+
+ def _clear_internal_state(self) -> None:
+ """
+ Protected hook to fully clear internal state.
+ Called in clear() method.
+
+ Default: Reset to empty schema instance via _set_internal_state hook.
+ """
+ self._set_internal_state(self._data_schema())
+
+ # ==================== Data Management ====================
+
def clear(self):
"""Clears all knowledge including data and vector index."""
+ self._clear_internal_state()
self.metadata["updated_at"] = datetime.now()
- self._data = self._data_schema()
- self.clear_index()
def clear_index(self):
"""Clears the vector index without affecting the stored data."""
@@ -149,31 +190,72 @@ class BaseAutoType(ABC, Generic[T]):
# ==================== Extraction & Merge ====================
- @abstractmethod
- def extract(self, text: str, *, store: bool = True) -> T:
- """Extracts structured knowledge from text with automatic chunking and merging.
+ def _extract_data(self, text: str) -> T:
+ """
+ Internal: Unified extraction logic (Chunking -> LLM -> Merge).
+ Implemented in BaseAutoType for code reuse by List and Set.
- This is the main entry point for knowledge extraction. It handles the complete pipeline
- from text preprocessing to knowledge merging.
+ Args:
+ text: Input text.
+
+ Returns:
+ The extracted knowledge data.
+ """
+ if len(text) <= self.chunk_size:
+ extracted_data = self.llm_chain_extract.invoke({"chunk_text": text})
+ extracted_data_list = [extracted_data]
+ else:
+ chunks = self.text_splitter.split_text(text)
+ inputs = [{"chunk_text": chunk} for chunk in chunks]
+ extracted_data_list = self.llm_chain_extract.batch(
+ inputs, config={"max_concurrency": self.max_workers}
+ )
+
+ merged_data = self.merge(extracted_data_list)
+ return merged_data
+
+ def extract(self, text: str) -> "BaseAutoType[T]":
+ """
+ Extracts knowledge into a NEW instance without modifying the current one.
- Extraction pipeline:
- 1. Determine if text chunking is needed based on length
- 2. Extract knowledge from each chunk using LLM
- 3. Merge all extracted data using the merge() method
- 4. If store=True: merge with existing data and update internal state
- 5. If store=False: return extracted data without modifying internal state
- 6. Return the merged knowledge
+ Use this for previewing data or branching knowledge bases.
Args:
- text: Input text to extract knowledge from (can be short or long).
- store: Controls whether to store extracted knowledge internally.
- - True (default): Store mode - merge with existing knowledge and update internal state
- - False: Temporary mode - return extracted data without modifying internal state
+ text: Input text.
Returns:
- The extracted and merged knowledge data.
+ A new knowledge instance containing only the extracted data.
"""
- pass
+ extracted_data = self._extract_data(text)
+
+ new_instance = self._create_new_instance()
+ new_instance._set_internal_state(extracted_data)
+
+ new_instance.metadata["created_at"] = datetime.now()
+ new_instance.metadata["updated_at"] = datetime.now()
+
+ return new_instance
+
+ def feed(self, text: str) -> "BaseAutoType[T]":
+ """
+ Ingests text into the CURRENT knowledge base instance.
+
+ This modifies the internal state by merging new data with existing data.
+ Supports method chaining (e.g., kb.feed(text1).feed(text2)).
+
+ Args:
+ text: Input text.
+
+ Returns:
+ Self (the current instance).
+ """
+ extracted_data = self._extract_data(text)
+ merged_data = self.merge([self._data, extracted_data])
+
+ self._set_internal_state(merged_data)
+ self.metadata["updated_at"] = datetime.now()
+
+ return self
@abstractmethod
def merge(self, data_list: List[T]) -> T:
@@ -195,24 +277,6 @@ class BaseAutoType(ABC, Generic[T]):
"""
pass
- # ==================== Evolution ====================
-
- def evolve(self, **kwargs) -> T:
- """Evolves and optimizes the internal knowledge through inference and refinement.
-
- Subclasses can implement this method to perform knowledge-specific evolution strategies.
-
- Evolution capabilities:
- - Infer implicit relationships between entities
- - Prune low-confidence nodes and connections
- - Apply clustering for better organization
- - Complete missing knowledge through reasoning
-
- Returns:
- The evolved knowledge data.
- """
- pass
-
# ==================== Indexing & Query ====================
@abstractmethod
@@ -246,34 +310,123 @@ class BaseAutoType(ABC, Generic[T]):
# ==================== Serialization ====================
- @abstractmethod
- def dump(self, folder_path: str | Path):
+ def dump(self, folder_path: str | Path) -> None:
"""Exports knowledge to a specified folder.
- Subclasses must implement this method to persist their knowledge data.
-
- Files to save:
- 1. Structured data (self._data) - saved as JSON file
+ Saves to the folder:
+ 1. Structured data (self._data) - saved as state.json
2. Vector index (self._index) - saved as FAISS index files
Args:
- folder_path: Target folder path for saving knowledge.
+ folder_path: Target folder path for saving.
"""
- pass
+ from langchain_community.vectorstores import FAISS
- @abstractmethod
- def load(self, folder_path: str | Path):
- """Loads knowledge from a specified folder.
+ folder = Path(folder_path)
+ if folder.exists() and folder.is_file():
+ raise Exception("Folder path is a file, please provide a folder path.")
+
+ if not folder.exists():
+ folder.mkdir(parents=True, exist_ok=True)
+
+ try:
+ # 1. Save structured data
+ data = {
+ "schema_name": self._data_schema.__name__,
+ "data_schema": self._data_schema.model_json_schema(),
+ "data": self._data.model_dump(),
+ "metadata": self._prepare_metadata_for_dump(),
+ }
+
+ # Allow subclasses to add extra fields
+ data.update(self._get_extra_dump_data())
- Subclasses must implement this method to restore previously saved knowledge.
+ json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)
+ data_file = folder / "state.json"
+ with open(data_file, "w", encoding="utf-8") as f:
+ f.write(json_str)
- Files to load:
- 1. Structured data (self._data) - loaded from JSON file
+ # 2. Save vector index
+ if self._index is not None:
+ index_path = str(folder / "faiss_index")
+ self._index.save_local(index_path)
+
+ except Exception as e:
+ raise e
+
+ def load(self, folder_path: str | Path) -> None:
+ """Loads knowledge from a specified folder.
+
+ Loads from the folder:
+ 1. Structured data (self._data) - loaded from state.json
2. Vector index (self._index) - loaded from FAISS index files
Args:
folder_path: Source folder path containing saved knowledge.
"""
+ from langchain_community.vectorstores import FAISS
+
+ folder = Path(folder_path)
+ if not folder.is_dir():
+ raise ValueError(f"Folder does not exist: {folder_path}")
+
+ # 1. Load structured data
+ data_file = folder / "state.json"
+ if not data_file.is_file():
+ raise ValueError(f"Data file not found: {data_file}")
+
+ with open(data_file, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ # Load data using hook for proper state sync
+ loaded_data = self._data_schema.model_validate(data.get("data", {}))
+ self._set_internal_state(loaded_data)
+
+ # Update metadata with loaded values
+ if "metadata" in data:
+ self.metadata.update(data["metadata"])
+ self.metadata["updated_at"] = datetime.now()
+
+ # Allow subclasses to load extra data
+ self._load_extra_data(data)
+
+ # 2. Load vector index
+ index_path = str(folder / "faiss_index")
+ if Path(index_path).exists():
+ try:
+ self._index = FAISS.load_local(
+ index_path, self.embedder, allow_dangerous_deserialization=True
+ )
+ except Exception as e:
+ self._index = None
+
+ # ==================== Serialization Helpers ====================
+
+ def _prepare_metadata_for_dump(self) -> Dict[str, Any]:
+ """Helper to serialize metadata values (e.g., datetimes)."""
+ return {
+ k: str(v) if isinstance(v, datetime) else v
+ for k, v in self.metadata.items()
+ }
+
+ def _get_extra_dump_data(self) -> Dict[str, Any]:
+ """Hook for subclasses to add extra fields to state.json.
+
+ Override this method to save subclass-specific data (e.g., merge strategy).
+
+ Returns:
+ Dictionary of extra data to include in state.json.
+ """
+ return {}
+
+ def _load_extra_data(self, json_data: Dict[str, Any]) -> None:
+ """Hook for subclasses to extract extra fields from state.json.
+
+ Override this method to restore subclass-specific data.
+
+ Args:
+ json_data: The loaded JSON data dictionary.
+ """
pass
# ==================== Operator Overloads ====================
@@ -285,11 +438,11 @@ class BaseAutoType(ABC, Generic[T]):
The new instance inherits configuration from the left operand (self).
Usage:
- >>> kb1 = UnitKnowledge(PersonSchema, ...)
- >>> kb2 = UnitKnowledge(PersonSchema, ...)
+ >>> kb1 = AutoList(PersonSchema, ...)
+ >>> kb2 = AutoList(PersonSchema, ...)
>>> kb3 = kb1 + kb2 # ✅ Same schema, creates merged instance
>>>
- >>> kb4 = UnitKnowledge(CompanySchema, ...)
+ >>> kb4 = AutoList(CompanySchema, ...)
>>> kb5 = kb1 + kb4 # ❌ TypeError: Different schemas
Args:
@@ -323,12 +476,11 @@ class BaseAutoType(ABC, Generic[T]):
# Create a new instance with the same configuration
new_instance = self._create_new_instance()
- # Set the merged data and update metadata
- new_instance._data = merged_data
+ # Set the merged data using hook and update metadata
+ new_instance._set_internal_state(merged_data)
new_instance.metadata["created_at"] = min(
self.metadata["created_at"], other.metadata["created_at"]
)
new_instance.metadata["updated_at"] = datetime.now()
- new_instance.clear_index() # Index needs to be rebuilt
return new_instance
diff --git a/hyperextract/core/list.py b/hyperextract/core/list.py
index 0c43039..3bd3a37 100644
--- a/hyperextract/core/list.py
+++ b/hyperextract/core/list.py
@@ -1,9 +1,18 @@
"""List Knowledge Pattern - extracts a collection of objects from text."""
-import json
-from typing import List, Any, Type, TypeVar, Generic, TYPE_CHECKING
+from typing import (
+ List,
+ Any,
+ Type,
+ TypeVar,
+ Generic,
+ TYPE_CHECKING,
+ Iterator,
+ Callable,
+ Iterable,
+ Union,
+)
from datetime import datetime
-from pathlib import Path
from pydantic import BaseModel, Field, create_model
from langchain_core.language_models.chat_models import BaseChatModel
from langchain_core.embeddings import Embeddings
@@ -132,66 +141,6 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
show_progress=self.show_progress,
)
- def extract(self, text: str, *, store: bool = True) -> AutoListSchema:
- """Extracts a list of items using LangChain native batch processing.
-
- Args:
- text: Input text to extract items from.
- store: Controls whether to store extracted knowledge internally.
- - True (default): Store mode - merge with existing knowledge and update internal state
- - False: Temporary mode - return extracted data without modifying internal state
-
- Returns:
- The list of extracted items.
- """
- start_time = datetime.now()
-
- # Determine if chunking is needed
- if len(text) <= self.chunk_size:
- # Short text: extract in one pass
- if self.show_progress:
- logger.info(f"Processing single text (length: {len(text)})...")
-
- extracted_data = self.llm_chain_extract.invoke({"chunk_text": text})
- extracted_data_list = [extracted_data]
-
- else:
- # Long text: extract by chunking
- chunks = self.text_splitter.split_text(text)
- logger.info(f"Split text into {len(chunks)} chunks")
-
- if self.show_progress:
- logger.info(f"Processing {len(chunks)} chunk(s)...")
-
- inputs = [{"chunk_text": chunk} for chunk in chunks]
- extracted_data_list = self.llm_chain_extract.batch(
- inputs, config={"max_concurrency": self.max_workers}
- )
-
- if self.show_progress:
- logger.info(f"Extracted {len(extracted_data_list)} chunks")
-
- logger.info("Merging extracted knowledge...")
- merged_data = self.merge(extracted_data_list)
-
- # If store=True, merge with existing data and update internal state
- if store:
- if self._data and len(self.items) > 0:
- self._data = self.merge([self._data, merged_data])
- else:
- self._data = merged_data
-
- self.clear_index()
- self.metadata["updated_at"] = datetime.now()
-
- logger.info("Knowledge extraction completed")
- logger.info(
- f"Duration: {(datetime.now() - start_time).total_seconds():.2f} seconds"
- )
-
- # Return items list (either from stored data or temporary merged data)
- return self.items if store else merged_data.items
-
def merge(self, data_list: List[AutoListSchema]) -> AutoListSchema:
"""Pure data merge method implementing list append strategy.
@@ -215,7 +164,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
# Return a new container object
return self.item_list_schema(items=all_items)
- def build_index(self):
+ def build_index(self) -> None:
"""Builds independent vector index for each item in the list."""
items = self.items
if not items:
@@ -244,7 +193,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
logger.error("FAISS not available. Install with: pip install faiss-cpu")
raise
- def search(self, query: str, top_k: int = 3) -> List[Any]:
+ def search(self, query: str, top_k: int = 3) -> List[Item]:
"""Searches items in the list using semantic similarity.
Args:
@@ -262,7 +211,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
raise Exception("Vector store not initialized")
docs = self._index.similarity_search(query, k=top_k)
- results = []
+ results: List[Item] = []
for doc in docs:
# Attempt to restore as object
try:
@@ -276,89 +225,13 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
logger.info(f"Found {len(results)} results for query: {query[:50]}...")
return results
- def dump(self, folder_path: str) -> Any:
- """Exports knowledge to a specified folder.
-
- Uses the same serialization logic as UnitKnowledge since the container
- is also a BaseModel.
- """
- folder = Path(folder_path)
- folder.mkdir(parents=True, exist_ok=True)
-
- # 1. Save structured data
- data = {
- "schema_name": self.item_list_schema.__name__,
- "item_schema_name": self.item_schema.__name__,
- "item_schema": self.item_schema.model_json_schema(),
- "data": self.data.model_dump(),
- "metadata": {
- k: str(v) if isinstance(v, datetime) else v
- for k, v in self.metadata.items()
- },
- }
-
- json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)
- data_file = folder / "state.json"
- with open(data_file, "w", encoding="utf-8") as f:
- f.write(json_str)
- logger.info(f"Saved data to {data_file}")
-
- # 2. Save vector index
- if self._index is not None:
- index_path = str(folder / "faiss_index")
- self._index.save_local(index_path)
- logger.info(f"Saved FAISS index to {index_path}")
- else:
- logger.warning("No index to save")
-
- return json_str
-
- def load(self, folder_path: str):
- """Loads knowledge from a specified folder."""
- folder = Path(folder_path)
- if not folder.is_dir():
- raise ValueError(f"Folder does not exist: {folder_path}")
-
- # 1. Load structured data
- data_file = folder / "state.json"
- if not data_file.is_file():
- raise ValueError(f"Data file not found: {data_file}")
-
- with open(data_file, "r", encoding="utf-8") as f:
- data = json.load(f)
- logger.info(f"Loaded data from {data_file}")
-
- self._data = self.item_list_schema.model_validate(data.get("data", {}))
-
- # Update metadata with loaded values
- if "metadata" in data:
- self.metadata.update(data["metadata"])
- self.metadata["updated_at"] = datetime.now()
-
- # 2. Load vector index
- index_path = str(folder / "faiss_index")
- if Path(index_path).exists():
- try:
- self._index = FAISS.load_local(
- index_path, self.embedder, allow_dangerous_deserialization=True
- )
- logger.info(f"Loaded FAISS index from {index_path}")
- except Exception as e:
- logger.warning(f"Failed to load FAISS index: {e}")
- self._index = None
- else:
- logger.warning("No index file found, will rebuild on next search")
- self._index = None
-
- logger.info(f"Loaded knowledge successfully with {len(self)} items")
-
# ==================== Pythonic Sequence Operations ====================
def __len__(self) -> int:
"""Returns the number of elements in the list."""
return len(self.items)
- def __getitem__(self, key):
+ def __getitem__(self, key: Union[int, slice]) -> Union[Item, "AutoList[Item]"]:
"""Support index access and slicing.
Args:
@@ -392,7 +265,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
f"List indices must be integers or slices, not {type(key).__name__}"
)
- def __setitem__(self, index: int, item):
+ def __setitem__(self, index: int, item: Item) -> None:
"""Support index assignment.
Args:
@@ -416,17 +289,17 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
self.clear_index()
self.metadata["updated_at"] = datetime.now()
- def __add__(self, other):
+ def __add__(self, other: "BaseAutoType[Item]") -> "AutoList[Item]":
"""Operator overload for '+' to combine knowledge instances.
Supports multiple combination patterns:
- AutoList + AutoList → AutoList (merge lists)
- - AutoList + UnitKnowledge → AutoList (append unit to list)
+ - AutoList + AutoModel → AutoList (append model to list)
- This enables chain operations like: unit1 + unit2 + unit3
+ This enables chain operations like: model1 + model2 + model3
Args:
- other: Another AutoList or UnitKnowledge with compatible schema.
+ other: Another AutoList or AutoModel with compatible schema.
Returns:
New AutoList with combined items.
@@ -436,7 +309,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
"""
from .model import AutoModel
- # Case 1: AutoList + AutoList (call parent implementation)
+ # Case 1: AutoList + AutoList
if isinstance(other, AutoList):
# Check schema compatibility
if self.item_schema != other.item_schema:
@@ -444,7 +317,6 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
f"Cannot add AutoList instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
- # Manually merge without calling base class (to avoid _data_schema check)
# Create new instance with merged items
new_instance = self._create_new_instance()
new_instance._data.items = self.items + other.items
@@ -452,7 +324,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
new_instance.metadata["updated_at"] = self.metadata.get("updated_at")
return new_instance
- # Case 2: AutoList + AutoModel → AutoList (append unit)
+ # Case 2: AutoList + AutoModel → AutoList (append model)
elif isinstance(other, AutoModel):
# Check schema compatibility
if self.item_schema != other._data_schema:
@@ -461,7 +333,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
f"List: {self.item_schema.__name__}, Unit: {other._data_schema.__name__}"
)
- # Create new AutoList with appended unit
+ # Create new AutoList with appended model
new_list = self._create_new_instance()
new_list._data.items = self.items + [other._data]
@@ -478,7 +350,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
f"Unsupported operand type for +: 'AutoList' and '{type(other).__name__}'"
)
- def __delitem__(self, index: int):
+ def __delitem__(self, index: int) -> None:
"""Support del operation for removing items by index.
Args:
@@ -499,7 +371,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
self.clear_index()
self.metadata["updated_at"] = datetime.now()
- def __iter__(self):
+ def __iter__(self) -> Iterator[Item]:
"""Support iteration over items.
Returns:
@@ -514,7 +386,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
"""
return iter(self.items)
- def __contains__(self, item) -> bool:
+ def __contains__(self, item: Item) -> bool:
"""Support 'in' operator for membership testing.
Args:
@@ -564,7 +436,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
# ==================== List Modification Methods ====================
- def append(self, item) -> None:
+ def append(self, item: Item) -> None:
"""Append a single item to the end of the list.
Args:
@@ -585,7 +457,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
self.clear_index()
self.metadata["updated_at"] = datetime.now()
- def extend(self, items) -> None:
+ def extend(self, items: Union[Iterable[Item], "AutoList[Item]"]) -> None:
"""Extend the list by appending multiple items.
Args:
@@ -626,7 +498,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
self.clear_index()
self.metadata["updated_at"] = datetime.now()
- def insert(self, index: int, item) -> None:
+ def insert(self, index: int, item: Item) -> None:
"""Insert an item at a specific position.
Args:
@@ -649,7 +521,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
self.clear_index()
self.metadata["updated_at"] = datetime.now()
- def remove(self, item) -> None:
+ def remove(self, item: Item) -> None:
"""Remove the first occurrence of an item from the list.
Args:
@@ -678,7 +550,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
# Item not found
raise ValueError(f"{item} is not in list")
- def pop(self, index: int = -1):
+ def pop(self, index: int = -1) -> Item:
"""Remove and return an item at the given position.
Args:
@@ -708,7 +580,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
# ==================== Query and Utility Methods ====================
- def index(self, item, start: int = 0, stop: int = None) -> int:
+ def index(self, item: Item, start: int = 0, stop: int | None = None) -> int:
"""Return the index of the first occurrence of item.
Args:
@@ -736,7 +608,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
raise ValueError(f"{item} is not in list")
- def count(self, item) -> int:
+ def count(self, item: Item) -> int:
"""Return the number of times item appears in the list.
Args:
@@ -800,7 +672,9 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
self._index = None
self.build_index()
- def sort(self, key=None, reverse: bool = False) -> None:
+ def sort(
+ self, key: Union[Callable[[Item], Any], None] = None, reverse: bool = False
+ ) -> None:
"""Sort the items in place.
Args:
@@ -839,7 +713,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
# ==================== Helper Methods ====================
- def _validate_item_schema(self, item) -> None:
+ def _validate_item_schema(self, item: Any) -> None:
"""Validate that item's schema matches item_schema.
Args:
@@ -884,7 +758,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
raise TypeError(" ".join(error_parts))
- def _items_equal(self, item1, item2) -> bool:
+ def _items_equal(self, item1: BaseModel, item2: BaseModel) -> bool:
"""Check if two items are equal.
Args:
diff --git a/hyperextract/core/model.py b/hyperextract/core/model.py
index da7b40e..b7654d7 100644
--- a/hyperextract/core/model.py
+++ b/hyperextract/core/model.py
@@ -1,9 +1,7 @@
"""Unit Knowledge Pattern - extracts a single structured object from text."""
-import json
from typing import List, Any, Type
from datetime import datetime
-from pathlib import Path
from langchain_core.language_models.chat_models import BaseChatModel
from langchain_core.embeddings import Embeddings
from langchain_core.documents import Document
@@ -225,134 +223,38 @@ class AutoModel(BaseAutoType[T]):
logger.info(f"Found {len(results)} results for query: {query[:50]}...")
return results
- # ==================== Serialization ====================
-
- def dump(self, folder_path: str | Path):
- """Exports knowledge to a specified folder.
-
- Saves to the folder:
- 1. Structured data (self._data) - saved as state.json
- 2. Vector index (self._index) - saved as FAISS index files
-
- Args:
- folder_path: Target folder path for saving.
- """
-
- folder = Path(folder_path)
- if folder.exists() and folder.is_file():
- raise Exception("Folder path is a file, please provide a folder path.")
-
- if not folder.exists():
- logger.info(f"Creating folder: {folder}")
- folder.mkdir(parents=True, exist_ok=True)
-
- try:
- # 1. Save structured data
- data = {
- "schema_name": self.data_schema.__name__,
- "data_schema": self.data_schema.model_json_schema(),
- "data": self.data.model_dump(),
- "metadata": {
- k: str(v) if isinstance(v, datetime) else v
- for k, v in self.metadata.items()
- },
- }
-
- json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)
- data_file = folder / "state.json"
- with open(data_file, "w", encoding="utf-8") as f:
- f.write(json_str)
- logger.info(f"Saved data to {data_file}")
-
- # 2. Save vector index
- if self._index is not None:
- index_path = str(folder / "faiss_index")
- self._index.save_local(index_path)
- logger.info(f"Saved FAISS index to {index_path}")
- else:
- logger.warning("No index to save")
-
- except Exception as e:
- logger.error(f"Failed to dump knowledge: {e}")
-
- def load(self, folder_path: str | Path):
- """Loads knowledge from a specified folder.
-
- Loads from the folder:
- 1. Structured data (self._data) - loaded from state.json
- 2. Vector index (self._index) - loaded from FAISS index files
-
- Args:
- folder_path: Source folder path containing saved knowledge.
- """
- folder = Path(folder_path)
- if not folder.is_dir():
- raise ValueError(f"Folder does not exist: {folder_path}")
-
- # 1. Load structured data
- data_file = folder / "state.json"
- if not data_file.is_file():
- raise ValueError(f"Data file not found: {data_file}")
-
- with open(data_file, "r", encoding="utf-8") as f:
- data = json.load(f)
- logger.info(f"Loaded data from {data_file}")
-
- self._data = self.data_schema.model_validate(data.get("data", {}))
-
- # Update metadata with loaded values
- if "metadata" in data:
- self.metadata.update(data["metadata"])
- self.metadata["updated_at"] = datetime.now()
-
- # 2. Load vector index
- index_path = str(folder / "faiss_index")
- if Path(index_path).exists():
- try:
- from langchain_community.vectorstores import FAISS
-
- self._index = FAISS.load_local(
- index_path, self.embedder, allow_dangerous_deserialization=True
- )
- logger.info(f"Loaded FAISS index from {index_path}")
- except Exception as e:
- logger.warning(f"Failed to load FAISS index: {e}")
- self._index = None
- else:
- logger.warning("No index file found, will rebuild on next search")
- self._index = None
-
- logger.info(f"Loaded knowledge successfully with {len(self)} unit(s)")
-
# ==================== Operators ====================
def __add__(self, other):
- """Operator overload for '+' to combine AutoModel instances into ListKnowledge.
+ """Operator overload for '+' to combine AutoModel instances into AutoList.
+
+ Supports multiple combination patterns:
+ - AutoModel + AutoModel → AutoList (create list from both items)
+ - AutoModel + AutoList → AutoList (prepend model to list)
- When two AutoModel instances are added, they are combined into a ListKnowledge
- containing both objects as separate items. This enables intuitive collection building.
+ This enables intuitive chain operations like: unit1 + unit2 + unit3
Usage:
>>> unit1 = AutoModel(PersonSchema, ...)
>>> unit2 = AutoModel(PersonSchema, ...)
- >>> person_list = unit1 + unit2 # → ListKnowledge[PersonSchema]
+ >>> person_list = unit1 + unit2 # → AutoList[PersonSchema]
>>>
>>> # Chain operations
>>> unit3 = AutoModel(PersonSchema, ...)
- >>> person_list = unit1 + unit2 + unit3 # → ListKnowledge with 3 items
+ >>> person_list = unit1 + unit2 + unit3 # → AutoList with 3 items
Args:
- other: Another AutoModel with the same data schema, or ListKnowledge.
+ other: Another AutoModel with the same data schema, or AutoList.
Returns:
- ListKnowledge containing both objects as items.
+ AutoList containing both objects as items.
Raises:
TypeError: If schemas don't match or invalid operand type.
"""
- from .list import ListKnowledge
+ from .list import AutoList
- # Case 1: AutoModel + AutoModel → ListKnowledge
+ # Case 1: AutoModel + AutoModel → AutoList
if isinstance(other, AutoModel):
# Check schema compatibility
if self._data_schema != other._data_schema:
@@ -361,8 +263,8 @@ class AutoModel(BaseAutoType[T]):
f"Left: {self._data_schema.__name__}, Right: {other._data_schema.__name__}"
)
- # Create new ListKnowledge
- list_kb = ListKnowledge(
+ # Create new AutoList
+ list_kb = AutoList(
item_schema=self._data_schema,
llm_client=self.llm_client,
embedder=self.embedder,
@@ -384,17 +286,17 @@ class AutoModel(BaseAutoType[T]):
return list_kb
- # Case 2: AutoModel + ListKnowledge → ListKnowledge (for reverse order)
- elif isinstance(other, ListKnowledge):
+ # Case 2: AutoModel + AutoList → AutoList (prepend model)
+ elif isinstance(other, AutoList):
# Check schema compatibility
if self._data_schema != other.item_schema:
raise TypeError(
- f"Cannot add AutoModel to ListKnowledge with different schemas. "
+ f"Cannot add AutoModel to AutoList with different schemas. "
f"Unit: {self._data_schema.__name__}, List: {other.item_schema.__name__}"
)
- # Create new ListKnowledge with unit prepended
- new_list = ListKnowledge(
+ # Create new AutoList with unit prepended
+ new_list = AutoList(
item_schema=self._data_schema,
llm_client=self.llm_client,
embedder=self.embedder,
diff --git a/hyperextract/core/set.py b/hyperextract/core/set.py
index db9198f..70153c7 100644
--- a/hyperextract/core/set.py
+++ b/hyperextract/core/set.py
@@ -4,10 +4,8 @@ Provides automatic deduplication based on a user-specified unique key field.
Supports multiple merge strategies including LLM-powered intelligent merging.
"""
-import json
from typing import List, Any, Type, TypeVar, Generic, Dict, Optional, Callable
from datetime import datetime
-from pathlib import Path
from pydantic import BaseModel, Field, create_model
from langchain_core.language_models.chat_models import BaseChatModel
from langchain_core.embeddings import Embeddings
@@ -136,14 +134,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
**kwargs,
)
- # AutoSet-specific attributes
+ # AutoSet-specific attributes (before _init_internal_state is called by parent)
self.key_extractor = key_extractor
self.merge_item_strategy = merge_item_strategy
self.llm_batch_size = llm_batch_size
- # Internal storage: Dict for O(1) deduplication
- self._items_dict: Dict[Any, Item] = {}
-
# Create Merger instance based on strategy
self._merger: BaseMerger[Item] = self._create_merger(
item_schema=item_schema,
@@ -289,70 +284,42 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
"""
return iter(self.items)
- # ==================== Core Override Methods ====================
-
- def clear(self):
- """Clears all items from the set."""
- self._data = self.item_set_schema()
- self._items_dict.clear()
- self.clear_index()
- self.metadata["updated_at"] = datetime.now()
-
- def extract(self, text: str, *, store: bool = True):
- """Extracts a set of unique items using LLM with automatic deduplication.
-
- Args:
- text: Input text to extract items from.
- store: Whether to store extracted knowledge internally.
+ # ==================== State Management Lifecycle Overrides ====================
- Returns:
- The set of extracted unique items.
+ def _init_internal_state(self) -> None:
"""
- start_time = datetime.now()
-
- # Determine if chunking is needed
- if len(text) <= self.chunk_size:
- # Short text: extract in one pass
- if self.show_progress:
- logger.info(f"Processing single text (length: {len(text)})...")
-
- extracted_data = self.llm_chain_extract.invoke({"chunk_text": text})
- extracted_data_list = [extracted_data]
-
- else:
- # Long text: extract by chunking
- chunks = self.text_splitter.split_text(text)
- logger.info(f"Split text into {len(chunks)} chunks")
-
- if self.show_progress:
- logger.info(f"Processing {len(chunks)} chunk(s)...")
-
- inputs = [{"chunk_text": chunk} for chunk in chunks]
- extracted_data_list = self.llm_chain_extract.batch(
- inputs, config={"max_concurrency": self.max_workers}
- )
+ Overrides base hook to initialize AutoSet-specific structures.
+ Called at the end of __init__.
+ """
+ # 1. Initialize the dict (AutoSet-specific)
+ self._items_dict: Dict[Any, Item] = {}
- if self.show_progress:
- logger.info(f"Extracted {len(extracted_data_list)} chunks")
+ # 2. Call parent to initialize standard structures (_data, _index)
+ super()._init_internal_state()
- logger.info("Merging extracted knowledge...")
- merged_data = self.merge(extracted_data_list)
+ def _set_internal_state(self, data: AutoSetSchema[Item]) -> None:
+ """
+ Overrides base hook to ensure _items_dict stays synchronized with _data.
+ This is the synchronization point for AutoSet's dual-structure design.
+ """
+ # 1. Sync dictionary (AutoSet-specific: the Real Source of Truth for lookups)
+ self._items_dict.clear()
+ for item in data.items:
+ key = self._get_key_value(item)
+ if key is not None:
+ self._items_dict[key] = item
- # If store=True, merge with existing data and update internal state
- if store:
- if self._data and len(self.items) > 0:
- final_data = self.merge([self._data, merged_data])
- else:
- final_data = merged_data
- self._update_internal_state(final_data)
+ # 2. Call parent to set _data and clear index
+ super()._set_internal_state(data)
- logger.info("Knowledge extraction completed")
- logger.info(
- f"Duration: {(datetime.now() - start_time).total_seconds():.2f} seconds"
- )
+ def _clear_internal_state(self) -> None:
+ """
+ Overrides base hook to explicitly clear the dictionary before clearing _data.
+ """
+ self._items_dict.clear()
+ super()._clear_internal_state()
- # Return items list
- return self.items if store else merged_data.items
+ # ==================== Core Override Methods ====================
def merge(self, data_list: List[AutoSetSchema[Item]]) -> AutoSetSchema[Item]:
"""Merges multiple data containers with automatic deduplication.
@@ -391,25 +358,6 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
# Step 3: Return new container (does not modify self)
return self.item_set_schema(items=merged_items)
- def _update_internal_state(self, data: AutoSetSchema[Item]) -> None:
- """Updates internal state from a data container.
-
- Private method that synchronizes _items_dict and _data with new data.
- Also clears vector index and updates metadata timestamp.
-
- Args:
- data: The data container to update from.
- """
- self._items_dict.clear()
- for item in data.items:
- key = self._get_key_value(item)
- if key is not None:
- self._items_dict[key] = item
-
- self._data = data
- self.clear_index()
- self.metadata["updated_at"] = datetime.now()
-
def _get_key_value(self, item: Item) -> Optional[Any]:
"""Extracts the unique key value from an item using key_extractor.
@@ -503,92 +451,19 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
logger.info(f"Found {len(results)} results for query: {query[:50]}...")
return results
- # ==================== Serialization ====================
-
- def dump(self, folder_path: str | Path):
- """Exports knowledge to a specified folder.
+ # ==================== Serialization Helpers ====================
- Saves both the items and the AutoSet-specific metadata.
+ def _get_extra_dump_data(self) -> Dict[str, Any]:
+ """Add AutoSet specific data to the dump.
- Args:
- folder_path: Target folder path for saving.
+ Returns:
+ Dictionary with extra AutoSet-specific fields.
"""
- folder = Path(folder_path)
- folder.mkdir(parents=True, exist_ok=True)
-
- # Save structured data
- data = {
- "schema_name": self.item_set_schema.__name__,
+ return {
"item_schema_name": self.item_schema.__name__,
- "item_schema": self.item_schema.model_json_schema(),
"merge_item_strategy": self.merge_item_strategy.value,
- "data": self.data.model_dump(),
- "metadata": {
- k: str(v) if isinstance(v, datetime) else v
- for k, v in self.metadata.items()
- },
}
- json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)
- data_file = folder / "state.json"
- with open(data_file, "w", encoding="utf-8") as f:
- f.write(json_str)
- logger.info(f"Saved data to {data_file}")
-
- # Save vector index
- if self._index is not None:
- index_path = str(folder / "faiss_index")
- self._index.save_local(index_path)
- logger.info(f"Saved FAISS index to {index_path}")
- else:
- logger.warning("No index to save")
-
- def load(self, folder_path: str | Path):
- """Loads knowledge from a specified folder.
-
- Args:
- folder_path: Source folder path containing saved knowledge.
- """
- folder = Path(folder_path)
- if not folder.is_dir():
- raise ValueError(f"Folder does not exist: {folder_path}")
-
- # Load structured data
- data_file = folder / "state.json"
- if not data_file.is_file():
- raise ValueError(f"Data file not found: {data_file}")
-
- with open(data_file, "r", encoding="utf-8") as f:
- data = json.load(f)
- logger.info(f"Loaded data from {data_file}")
-
- loaded_data = self.item_set_schema.model_validate(data.get("data", {}))
- self._update_internal_state(loaded_data)
-
- # Update metadata from saved data
- if "metadata" in data:
- self.metadata.update(data["metadata"])
- self.metadata["updated_at"] = datetime.now()
-
- # Load vector index
- index_path = str(folder / "faiss_index")
- if Path(index_path).exists():
- try:
- from langchain_community.vectorstores import FAISS
-
- self._index = FAISS.load_local(
- index_path, self.embedder, allow_dangerous_deserialization=True
- )
- logger.info(f"Loaded FAISS index from {index_path}")
- except Exception as e:
- logger.warning(f"Failed to load FAISS index: {e}")
- self._index = None
- else:
- logger.warning("No index file found, will rebuild on next search")
- self._index = None
-
- logger.info(f"Loaded knowledge successfully with {len(self)} unique items")
-
# ==================== Set-Specific Methods ====================
def add(self, item: Item) -> None:
@@ -610,9 +485,9 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
# Add new
self._items_dict[key_value] = item.model_copy(deep=True)
- # Update internal state
- self._data.items = list(self._items_dict.values())
- self.clear_index()
+ # Update state via hook
+ new_data = self.item_set_schema(items=list(self._items_dict.values()))
+ self._set_internal_state(new_data)
self.metadata["updated_at"] = datetime.now()
def remove(self, key: Any) -> Optional[Item]:
@@ -626,8 +501,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
"""
removed = self._items_dict.pop(key, None)
if removed:
- self._data.items = list(self._items_dict.values())
- self.clear_index()
+ new_data = self.item_set_schema(items=list(self._items_dict.values()))
+ self._set_internal_state(new_data)
self.metadata["updated_at"] = datetime.now()
logger.debug(f"Removed item with key '{key}'")
return removed
@@ -681,8 +556,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
"""
if key in self._items_dict:
self._items_dict.pop(key)
- self._data.items = list(self._items_dict.values())
- self.clear_index()
+ new_data = self.item_set_schema(items=list(self._items_dict.values()))
+ self._set_internal_state(new_data)
self.metadata["updated_at"] = datetime.now()
def pop(self) -> Item:
@@ -710,8 +585,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
key = next(iter(self._items_dict))
item = self._items_dict.pop(key)
- self._data.items = list(self._items_dict.values())
- self.clear_index()
+ new_data = self.item_set_schema(items=list(self._items_dict.values()))
+ self._set_internal_state(new_data)
self.metadata["updated_at"] = datetime.now()
return item
@@ -728,10 +603,13 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
>>> # Original skills unchanged
"""
new_set = self._create_new_instance()
+ # Populate the dictionary directly, then sync via hook
new_set._items_dict = {
key: item.model_copy(deep=True) for key, item in self._items_dict.items()
}
- new_set._data.items = list(new_set._items_dict.values())
+ # Use hook to sync _data
+ new_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))
+ new_set._set_internal_state(new_data)
new_set.metadata = self.metadata.copy()
new_set.metadata["created_at"] = datetime.now()
@@ -774,7 +652,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
else:
new_set._items_dict[key] = item.model_copy(deep=True)
- new_set._data.items = list(new_set._items_dict.values())
+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))
+ new_set._set_internal_state(merged_data)
new_set.metadata["updated_at"] = datetime.now()
return new_set
@@ -814,7 +693,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
if merged:
new_set._items_dict[key] = merged
- new_set._data.items = list(new_set._items_dict.values())
+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))
+ new_set._set_internal_state(merged_data)
new_set.metadata["updated_at"] = datetime.now()
return new_set
@@ -849,7 +729,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
if key not in other._items_dict:
new_set._items_dict[key] = item.model_copy(deep=True)
- new_set._data.items = list(new_set._items_dict.values())
+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))
+ new_set._set_internal_state(merged_data)
new_set.metadata["updated_at"] = datetime.now()
return new_set
@@ -889,7 +770,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
if key not in self._items_dict:
new_set._items_dict[key] = item.model_copy(deep=True)
- new_set._data.items = list(new_set._items_dict.values())
+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))
+ new_set._set_internal_state(merged_data)
new_set.metadata["updated_at"] = datetime.now()
return new_set
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tests/test_knowledge.py b/tests/test_knowledge.py
deleted file mode 100644
index 612069a..0000000
--- a/tests/test_knowledge.py
+++ /dev/null
@@ -1,343 +0,0 @@
-"""
-测试 BaseAutoType 类的功能。
-"""
-import pytest
-from typing import List, Optional
-from pydantic import BaseModel, Field
-from unittest.mock import MagicMock, AsyncMock
-import json
-
-from hyperextract.core import BaseAutoType
-
-
-# 定义测试用的 Schema
-class Article(BaseModel):
- """文章 Schema"""
- title: str = Field(..., description="文章标题")
- authors: List[str] = Field(default_factory=list, description="作者列表")
- keywords: List[str] = Field(default_factory=list, description="关键词")
- summary: str = Field(default="", description="摘要")
-
-
-class Person(BaseModel):
- """人物 Schema"""
- name: str = Field(..., description="人物名字")
- age: int = Field(..., description="年龄")
- occupation: str = Field(default="", description="职业")
-
-
-def mock_llm():
- """创建 mock LLM 客户端"""
- llm = AsyncMock()
- return llm
-
-
-def mock_embedder():
- """创建 mock Embeddings 客户端"""
- embedder = MagicMock()
- return embedder
-
-
-def a_article(mock_llm, mock_embedder):
- """创建一个 Article 类型的 BaseAutoType 实例"""
- return BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- storage="memory",
- )
-
-
-def a_person(mock_llm, mock_embedder):
- """创建一个 Person 类型的 BaseAutoType 实例"""
- return BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Person,
- storage="memory",
- )
-
-
-class TestBaseAutoTypeInitialization:
- """测试 BaseAutoType 类的初始化"""
-
- def test_auto_type_init_with_article_schema(self, a_article):
- """测试使用 Article schema 初始化"""
- assert a_article.schema_class == Article
- assert a_article.storage == "memory"
- assert a_article.data is not None
-
- def test_auto_type_init_with_initial_data(self, mock_llm, mock_embedder):
- """测试带初始数据的初始化"""
- initial_article = Article(
- title="Test Article",
- authors=["Author 1"],
- keywords=["test"],
- summary="A test article"
- )
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- initial_data=initial_article,
- )
- assert knowledge.data == initial_article
- assert knowledge.data.title == "Test Article"
-
- def test_auto_type_with_different_schemas(self, mock_llm, mock_embedder):
- """测试使用不同 Schema 的 BaseAutoType"""
- article_k = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- )
- person_k = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Person,
- )
- assert article_k.schema_class == Article
- assert person_k.schema_class == Person
-
-
-class TestBaseAutoTypeExtract:
- """测试 BaseAutoType 的 extract 方法"""
-
- @pytest.mark.asyncio
- async def test_extract_article(self, a_article, mock_llm):
- """测试提取文章信息"""
- # 模拟 LLM 返回的响应(with_structured_output 直接返回对象)
- expected_article = Article(
- title="AI in 2024",
- authors=["John Doe", "Jane Smith"],
- keywords=["AI", "Machine Learning"],
- summary="An overview of AI trends in 2024"
- )
-
- # 模拟 with_structured_output 返回的 structured_llm
- mock_structured_llm = AsyncMock()
- mock_structured_llm.ainvoke.return_value = expected_article
- mock_llm.with_structured_output.return_value = mock_structured_llm
-
- text = "Artificial Intelligence continues to advance in 2024..."
- result = await a_article.extract(text)
-
- assert result.title == "AI in 2024"
- assert len(result.authors) == 2
- assert "AI" in result.keywords
- assert a_article.data == result
-
- @pytest.mark.asyncio
- async def test_extract_with_markdown_response(self, a_article, mock_llm):
- """测试 with_structured_output 自动处理响应格式"""
- # 使用 with_structured_output 无需处理 markdown 格式
- expected_article = Article(
- title="Test Article",
- authors=["Author"],
- keywords=["test"],
- summary="Summary"
- )
-
- mock_structured_llm = AsyncMock()
- mock_structured_llm.ainvoke.return_value = expected_article
- mock_llm.with_structured_output.return_value = mock_structured_llm
-
- result = await a_article.extract("test text")
- assert result.title == "Test Article"
- assert result.authors[0] == "Author"
-
- @pytest.mark.asyncio
- async def test_aextract_calls_extract(self, a_article, mock_llm):
- """测试 aextract 方法调用 extract"""
- expected_article = Article(
- title="Test",
- authors=[],
- keywords=[],
- summary=""
- )
-
- mock_structured_llm = AsyncMock()
- mock_structured_llm.ainvoke.return_value = expected_article
- mock_llm.with_structured_output.return_value = mock_structured_llm
-
- result = await a_article.aextract("test")
- assert result.title == "Test"
-
-
-class TestBaseAutoTypeDump:
- """测试 BaseAutoType 的 dump 方法"""
-
- def test_dump_to_json(self, mock_llm, mock_embedder):
- """测试导出为 JSON 格式"""
- article = Article(
- title="Test",
- authors=["Alice"],
- keywords=["test"],
- summary="A test"
- )
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- initial_data=article,
- )
-
- json_str = knowledge.dump(format="json")
- assert isinstance(json_str, str)
- parsed = json.loads(json_str)
- assert parsed["title"] == "Test"
- assert parsed["authors"] == ["Alice"]
-
- def test_dump_to_dict(self, mock_llm, mock_embedder):
- """测试导出为字典格式"""
- article = Article(
- title="Test",
- authors=["Alice"],
- keywords=["test"],
- summary="A test"
- )
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- initial_data=article,
- )
-
- data_dict = knowledge.dump(format="dict")
- assert isinstance(data_dict, dict)
- assert data_dict["title"] == "Test"
-
-
-class TestBaseAutoTypeLoad:
- """测试 BaseAutoType 的 load 方法"""
-
- def test_load_from_dict(self, mock_llm, mock_embedder):
- """测试从字典加载"""
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- )
-
- data = {
- "title": "Loaded Article",
- "authors": ["Author1"],
- "keywords": ["loaded"],
- "summary": "Loaded from dict"
- }
- knowledge.load(data)
-
- assert knowledge.data.title == "Loaded Article"
- assert knowledge.data.authors == ["Author1"]
-
- def test_load_from_json_string(self, mock_llm, mock_embedder):
- """测试从 JSON 字符串加载"""
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- )
-
- json_str = json.dumps({
- "title": "JSON Article",
- "authors": ["JSON Author"],
- "keywords": [],
- "summary": ""
- })
- knowledge.load(json_str)
-
- assert knowledge.data.title == "JSON Article"
-
- def test_load_invalid_data_raises_error(self, mock_llm, mock_embedder):
- """测试加载无效数据时抛出错误"""
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- )
-
- # 缺少必需字段
- invalid_data = {"authors": ["Author"]}
- with pytest.raises(Exception):
- knowledge.load(invalid_data)
-
-
-class TestBaseAutoTypeSearch:
- """测试 BaseAutoType 的 search 方法"""
-
- @pytest.mark.asyncio
- async def test_search_returns_dict(self, mock_llm, mock_embedder):
- """测试 search 返回字典格式的数据"""
- article = Article(
- title="Search Test",
- authors=["Searcher"],
- keywords=["search"],
- summary="Testing search"
- )
- knowledge = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- initial_data=article,
- )
-
- result = await knowledge.search("search")
- assert isinstance(result, dict)
- assert result["title"] == "Search Test"
-
-
-class TestBaseAutoTypeEvolve:
- """测试 BaseAutoType 的 evolve 和 aevolve 方法"""
-
- @pytest.mark.asyncio
- async def test_aevolve_default_implementation(self, a_article):
- """测试 aevolve 的默认实现(不做任何操作)"""
- await a_article.aevolve()
- # 如果没有抛出异常,则测试通过
-
-
-class TestBaseAutoTypeMultipleSchemas:
- """测试使用多个不同 Schema 的场景"""
-
- def test_different_schemas_independent(self, mock_llm, mock_embedder):
- """测试不同 Schema 的 BaseAutoType 相互独立"""
- article_k = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Article,
- )
- person_k = BaseAutoType(
- llm_client=mock_llm,
- embedder=mock_embedder,
- schema_class=Person,
- )
-
- # 分别为两个 knowledge 加载数据
- article_data = {
- "title": "Article",
- "authors": [],
- "keywords": [],
- "summary": ""
- }
- person_data = {
- "name": "John",
- "age": 30,
- "occupation": "Engineer"
- }
-
- article_k.load(article_data)
- person_k.load(person_data)
-
- # 验证数据相互独立
- assert article_k.data.title == "Article"
- assert person_k.data.name == "John"
- assert isinstance(article_k.data, Article)
- assert isinstance(person_k.data, Person)
-
-
-if __name__ == "__main__":
- pytest.main([__file__, "-v"])