Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
Refactor AutoList, AutoModel, and AutoSet classes for improved functionality and clarity
yifanfeng975b2ade887980
diff --git a/README.md b/README.mdindex 835f34a..7d78bd7 100644--- a/README.md+++ b/README.md@@ -1,19 +1,19 @@# Hyper-Extract 🧠✨-> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**+> **An Intelligent, LLM-Powered Knowledge Extraction Framework**>-> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.+> Transform unstructured text into structured Auto containers with the power of Large Language Models.---## 🎯 Vision-**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.+**Hyper-Extract** provides a powerful framework for intelligent knowledge extraction. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, and index knowledge from unstructured text.Every Auto container in Hyper-Extract follows a unified lifecycle:1. **Extract** - Intelligent extraction from text with automatic chunking and merging-2. **Evolve** - Self-refinement through reasoning, pruning, and optimization+2. **Merge** - Automatic aggregation with configurable merge strategies3. **Index** - Semantic vector search capabilities4. **Serialize** - Persist and reload knowledge with full fidelity@@ -29,7 +29,7 @@ Every Auto container in Hyper-Extract follows a unified lifecycle:### 🏗️ Unified Architecture- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility-- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline+- **Lifecycle Management**: Standardized `extract` → `merge` → `search` → `dump/load` pipeline### 🔍 Semantic Search- **FAISS Integration**: Fast vector similarity search for knowledge retrievaldiff --git a/docs/index.md b/docs/index.mdindex d42b0a8..519a4e6 100644--- a/docs/index.md+++ b/docs/index.md@@ -1,19 +1,19 @@# Hyper-Extract 🧠✨-> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**+> **An Intelligent, LLM-Powered Knowledge Extraction Framework**>-> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.+> Transform unstructured text into structured Auto containers with the power of Large Language Models.---## 🎯 Vision-**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.+**Hyper-Extract** provides a powerful framework for intelligent knowledge extraction. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, and index knowledge from unstructured text.Every Auto container in Hyper-Extract follows a unified lifecycle:1. **Extract** - Intelligent extraction from text with automatic chunking and merging-2. **Evolve** - Self-refinement through reasoning, pruning, and optimization+2. **Merge** - Automatic aggregation with configurable merge strategies3. **Index** - Semantic vector search capabilities4. **Serialize** - Persist and reload knowledge with full fidelity@@ -29,7 +29,7 @@ Every Auto container in Hyper-Extract follows a unified lifecycle:### 🏗️ Unified Architecture- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility-- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline+- **Lifecycle Management**: Standardized `extract` → `merge` → `search` → `dump/load` pipeline### 🔍 Semantic Search- **FAISS Integration**: Fast vector similarity search for knowledge retrievaldiff --git a/docs/zh/index.md b/docs/zh/index.mdindex af158a4..fe6d99f 100644--- a/docs/zh/index.md+++ b/docs/zh/index.md@@ -1,19 +1,19 @@# Hyper-Extract 🧠✨-> **一个智能的、基于 LLM 的知识提取与演化框架**+> **一个智能的、基于 LLM 的知识提取框架**>-> 借助大语言模型的力量,将非结构化文本转化为结构化、可演化的知识容器。+> 借助大语言模型的力量,将非结构化文本转化为结构化的知识容器。---## 🎯 愿景-**Hyper-Extract** 将知识提取重新定义为一个智能的、自我演化的过程。我们不是将数据结构视为被动容器,而是赋予它们 **LLM 驱动的智能**,使其能够主动地从非结构化文本中提取、合并、演化和索引知识。+**Hyper-Extract** 提供一个强大的智能知识提取框架。我们不是将数据结构视为被动容器,而是赋予它们 **LLM 驱动的智能**,使其能够主动地从非结构化文本中提取、合并和索引知识。Hyper-Extract 中的每个知识容器都遵循统一的生命周期:1. **Extract(提取)** - 使用 LLM 从文本智能提取,自动处理分块和合并-2. **Evolve(演化)** - 通过推理、剪枝和优化实现自我优化+2. **Merge(合并)** - 通过可配置的合并策略实现自动聚合3. **Index(索引)** - 提供语义向量搜索能力4. **Serialize(序列化)** - 完整保存和加载知识状态@@ -29,7 +29,7 @@ Hyper-Extract 中的每个知识容器都遵循统一的生命周期:### 🏗️ 统一架构- **抽象基类**:`BaseAutoType[T]` 为所有知识类型提供一致的接口- **泛型类型支持**:使用 `Generic[T]` 实现完整的 Python 类型提示-- **生命周期管理**:标准化的 `extract` → `merge` → `evolve` → `search` → `dump/load` 流程+- **生命周期管理**:标准化的 `extract` → `merge` → `search` → `dump/load` 流程### 🔍 语义搜索- **FAISS 集成**:快速的向量相似度搜索diff --git a/examples/auto_list_demo.py b/examples/auto_list_demo.pyindex 49e9384..6c0d07c 100644--- a/examples/auto_list_demo.py+++ b/examples/auto_list_demo.py@@ -21,7 +21,6 @@ from hyperextract import AutoListclass NewsEvent(BaseModel):"""新闻事件结构"""- event_id: Optional[str] = Field(default=None, description="事件唯一标识")title: str = Field(description="事件标题")date: str = Field(description="事件发生日期")location: str = Field(description="事件发生地点")@@ -151,15 +150,15 @@ def main():show_progress=True,)- # 4. 提取知识 (默认存储模式)- print("\n[4] 提取新闻事件列表 (默认存储模式)...")+ # 4. 提取知识 (使用 feed 模式)+ print("\n[4] 提取新闻事件列表 (使用 feed 模式)...")print("-" * 80)- events = a_event_list.extract(news_text)+ a_event_list.feed(news_text)print("-" * 80)# 5. 显示提取结果- print(f"\n[5] 提取到 {len(events)} 个新闻事件:")- for i, event in enumerate(events[:5], 1): # 只显示前5个+ print(f"\n[5] 提取到 {len(a_event_list.items)} 个新闻事件:")+ for i, event in enumerate(a_event_list.items[:5], 1): # 只显示前5个print(f"\n 事件 {i}:")print(f" 标题: {event.title}")print(f" 日期: {event.date}")@@ -169,8 +168,8 @@ def main():if event.impact:print(f" 影响: {event.impact[:60]}...")- if len(events) > 5:- print(f"\n ... 还有 {len(events) - 5} 个事件未显示")+ if len(a_event_list) > 5:+ print(f"\n ... 还有 {len(a_event_list) - 5} 个事件未显示")# 6. 测试列表大小print(f"\n[6] 知识列表大小: {len(a_event_list)} 个事件")@@ -194,8 +193,8 @@ def main():print(f" [{i}] {event.title} ({event.date})")print(f" 地点: {event.location}")- # 9. 测试增量提取(默认合并模式)- print("\n[9] 测试增量提取(默认合并模式)...")+ # 9. 测试增量提取(feed 模式)+ print("\n[9] 测试增量提取(feed 模式)...")additional_news = """### 事件11: 华为发布6G测试网络2024年6月15日,华为在深圳总部宣布建成全球首个6G技术测试网络。@@ -205,7 +204,7 @@ def main():print(" 添加补充新闻...")before_count = len(a_event_list)- a_event_list.extract(additional_news) # 默认 store=True,自动合并+ a_event_list.feed(additional_news) # 使用 feed 自动合并after_count = len(a_event_list)print(f" 合并前: {before_count} 个事件")print(f" 合并后: {after_count} 个事件")diff --git a/examples/auto_model_demo.py b/examples/auto_model_demo.pyindex f85cd02..32aa31f 100644--- a/examples/auto_model_demo.py+++ b/examples/auto_model_demo.py@@ -161,10 +161,11 @@ def main():show_progress=True,)- # 4. 提取知识 (默认存储模式)- print("\n[4] 提取知识 (默认存储模式)...")+ # 4. 提取知识 (使用 feed 模式)+ print("\n[4] 提取知识 (使用 feed 模式)...")print("-" * 80)- extracted_data = a_document_model.extract(document)+ a_document_model.feed(document)+ extracted_data = a_document_model.dataprint("-" * 80)# 5. 显示提取结果@@ -203,15 +204,15 @@ def main():field_value = result[field_name]print(f" [{i}] {field_name}: {str(field_value)[:80]}...")- # 9. 测试增量提取(默认合并模式)- print("\n[9] 测试增量提取(默认合并模式)...")+ # 9. 测试增量提取(feed 模式)+ print("\n[9] 测试增量提取(feed 模式)...")additional_text = """补充信息:本研究获得了国家自然科学基金支持,项目编号:NSFC-2023-12345。研究团队来自清华大学医学院和北京协和医院。"""print(" 添加补充文本...")- a_document_model.extract(additional_text) # 默认 store=True,自动合并+ a_document_model.feed(additional_text) # 使用 feed 自动合并print(f" 更新后的作者信息: {a_document_model.data.author}")# 10. 保存知识diff --git a/examples/auto_set_demo.py b/examples/auto_set_demo.pyindex 05d3252..afa7b70 100644--- a/examples/auto_set_demo.py+++ b/examples/auto_set_demo.py@@ -1,7 +1,7 @@-"""Set Knowledge Demo - Extracting unique skills from multiple job descriptions+"""集合知识示例 - 从冒险者日志中提取和去重怪物信息-This example demonstrates how AutoSet automatically deduplicates-extracted items based on a unique key field.+这个示例演示了AutoSet如何基于唯一键字段自动去重提取的项目。+使用RPG怪物图鉴场景展示LLM_MERGE策略如何智能合并来自多个来源的冲突信息。"""import sys@@ -12,6 +12,8 @@ project_root = Path(__file__).parent.parentsys.path.insert(0, str(project_root))import os+import io+# sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')from pydantic import BaseModel, Fieldfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings@@ -19,199 +21,246 @@ from hyperextract import AutoSetfrom hyperextract.utils.merger import MergeStrategy-# Define the schema for skills-class SkillSchema(BaseModel):- """A skill mentioned in job descriptions"""+# 定义怪物信息的数据模型+class MonsterEntry(BaseModel):+ """RPG游戏中的怪物信息"""name: str = Field(- description="Name of the skill (e.g., 'Python', 'Machine Learning')"+ description="怪物的名称(例如:'地精小卒'、'洞穴巨魔')")- category: str | None = Field(+ species: str | None = Field(None,- description="Category (e.g., 'Programming Language', 'Framework', 'Soft Skill')",+ description="怪物的种族分类(例如:'类人型'、'巨人'、'野兽')",)- proficiency_level: str | None = Field(+ danger_level: int | None = Field(None,- description="Required proficiency level if mentioned (e.g., 'Expert', 'Intermediate')",+ description="危险等级(1-10),用于评估战斗难度",)- description: str | None = Field(- None, description="Brief description or context of how the skill is used"+ habitats: str | None = Field(+ None,+ description="怪物常见的栖息地或活动场所"+ )+ characteristics: str | None = Field(+ None,+ description="怪物的外观特征和行为习性描述"+ )+ weaknesses: str | None = Field(+ None,+ description="怪物对特定伤害类型或攻击方式的弱点"+ )+ loot_drops: str | None = Field(+ None,+ description="击败怪物可能掉落的物品和资源")def main():- # Initialize LLM and embeddings+ # 初始化LLM和嵌入模型llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(base_url=os.getenv("OPENAI_BASE_URL"))- # Create AutoSet with FIELD_MERGE strategy- # This means when the same skill appears multiple times,- # the new extraction will fill in any missing fields from the existing one- a_skill_set = AutoSet(- item_schema=SkillSchema,+ # 创建AutoSet,使用LLM_MERGE策略+ # 当同一个怪物在多个日志中出现时,LLM会智能合并冲突的信息+ monster_codex = AutoSet(+ item_schema=MonsterEntry,llm_client=llm,embedder=embedder,- key_extractor=lambda x: x.name, # Deduplicate based on skill name- merge_item_strategy=MergeStrategy.FIELD_MERGE,- prompt="""Extract all technical skills, frameworks, tools, and soft skills mentioned in the text.--For each skill, provide:-- name: The exact name of the skill-- category: What type of skill it is-- proficiency_level: If mentioned, the required level-- description: Brief context about how it's used+ key_extractor=lambda x: x.name, # 基于怪物名称去重+ merge_item_strategy=MergeStrategy.LLM_MERGE,+ prompt="""从冒险者的日志记录中提取所有怪物的相关信息。++对于每只怪物,请提供:+- name: 怪物的确切名称+- species: 怪物的种族分类(类人型、野兽、不死生物等)+- danger_level: 战斗危险等级(1-10)+- habitats: 常见的栖息地或地理位置+- characteristics: 详细的外观特征和行为描述+- weaknesses: 已知的弱点和克制方法+- loot_drops: 击败后可能掉落的物品-Be specific and extract all mentioned skills.""",+要尽可能准确和详细,包括所有提到的怪物和它们的属性。如果信息不完整,请基于已有信息做出合理推断。""",)- # Sample job descriptions (with overlapping skills)- job_descriptions = [+ # 来自冒险者的日志 - 包含重复的怪物但信息略有不同+ adventurer_logs = ["""- Senior Python Developer+ 【第一个冒险日记 - 林地远征】+ 日期:秋月15日++ 今天在幽暗森林中遭遇了几种怪物。首先遇到了一群地精小卒,他们大约有3-4英尺高,绿皮肤,+ 穿着破烂的皮盔甲。这些家伙相当狡猾,喜欢群体狩猎。危险等级大概是2-3。它们掉落了+ 一些铜币和破旧的匕首。- We are seeking an expert Python developer with 5+ years of experience.+ 在森林深处,我还遇到了一只洞穴巨魔。这个家伙大约8英尺高,皮肤呈灰褐色,力量惊人。+ 对火焰特别敏感——我用火焰咒语成功地逼退了它。危险等级至少是7。它的巢穴里有一些金币。- Requirements:- - Expert level Python programming- - Strong experience with FastAPI framework for building REST APIs- - Proficiency in PostgreSQL database design and optimization- - Experience with Docker for containerization- - Strong problem-solving and communication skills- - Experience with Git version control+ 还有森林狼群,非常聪明的野兽,它们似乎有高度的群体智能。危险等级4-5。""",+"""- Full Stack Engineer+ 【第二个冒险日记 - 深矿探险】+ 日期:秋月18日++ 在深矿洞中进行了危险的探险。地精小卒在这里也有出现!看来他们在地下还有殖民地。+ 这次的地精穿着更好的装备,显然这是地精的精英部队。他们配备了铁制短剑,训练更充分。+ 危险等级可能达到了3-4。掉落了更多的铜币和一些魔晶。- Join our dynamic team as a Full Stack Engineer!+ 还有一种我之前没遇过的怪物——地下爬行虫。这些生物有分节的身体,长约2-3米,+ 用触须来感知周围环境。它们不是很聪明,但速度很快。危险等级3。掉落虫壳碎片。- Required Skills:- - Python for backend development (3+ years)- - React for frontend development- - Experience with FastAPI or Django- - PostgreSQL or MySQL database experience- - Docker and Kubernetes for deployment- - Excellent teamwork and communication abilities- - Git workflow knowledge+ 巨魔在深处也出现了,但这次的巨魔看起来比森林里的那只更大更凶悍。+ 它对寒冰魔法有抵抗力,但仍然惧怕火焰。危险等级估计8级。""",+"""- Machine Learning Engineer+ 【第三个冒险日记 - 废墟遗迹】+ 日期:秋月22日- Looking for an ML Engineer to join our AI team.+ 在古老遗迹的探索中发现了新的敌人类型。最常见的是地精小卒的变种——遗迹守卫。+ 他们看起来是被之前的文明改造过的,具有更强的战斗能力。有人说他们被魔法增强过。+ 危险等级跳到了4-5。这些怪物掉落了古代金币和遗迹碎片。- Must Have:- - Advanced Python skills for ML development- - PyTorch or TensorFlow experience- - Strong understanding of machine learning algorithms- - Experience with Docker for model deployment- - PostgreSQL for data storage- - Git for code versioning- - Strong analytical and problem-solving skills- """,+ 还有一个更大的怪物——遗迹看守者,是一只3米高的怪物。它有三只眼睛,可能具有某种魔法能力。+ 非常危险!危险等级至少9级。击败它时获得了一个古老的魔晶。++ 巨魔在这里也有据点,防守非常严密。这些可能是智商更高的个体。+ 它们使用简单的工具和陷阱来狩猎冒险者。危险等级8-9。+ """]print("=" * 80)- print("SET KNOWLEDGE DEMO: Extracting Unique Skills from Job Descriptions")+ print("集合知识演示:从冒险者日志中构建怪物图鉴")print("=" * 80)- # Extract skills from all job descriptions- print("\n📄 Processing job descriptions...")- for i, job_desc in enumerate(job_descriptions, 1):- print(f"\n Processing job description {i}...")- a_skill_set.extract(job_desc)- print(f" ✓ Total unique skills so far: {len(a_skill_set)}")+ # 从所有日志中提取怪物信息+ print("\n[*] 正在处理冒险者日志...")+ for i, log in enumerate(adventurer_logs, 1):+ print(f"\n 处理日志 {i}...")+ monster_codex.feed(log) # 喂养数据+ print(f" √ 目前已识别的怪物数量:{len(monster_codex)}")- # Display all unique skills+ # 显示所有提取的怪物print("\n" + "=" * 80)- print(f"📊 EXTRACTED UNIQUE SKILLS: {len(a_skill_set)} total")+ print(f"[*] 提取的怪物统计:共 {len(monster_codex)} 种不同的怪物")print("=" * 80)- # Group by category- skills_by_category = {}- for skill in a_skill_set.items:- category = skill.category or "Uncategorized"- if category not in skills_by_category:- skills_by_category[category] = []- skills_by_category[category].append(skill)-- # Display organized results- for category, category_skills in sorted(skills_by_category.items()):- print(f"\n{category}:")- for skill in sorted(category_skills, key=lambda s: s.name):- print(f" • {skill.name}")- if skill.proficiency_level:- print(f" Level: {skill.proficiency_level}")- if skill.description:- print(f" Context: {skill.description}")-- # Demonstrate set operations+ # 按危险等级分组显示+ monsters_by_danger = {}+ for monster in monster_codex.items:+ level = monster.danger_level or 0+ if level not in monsters_by_danger:+ monsters_by_danger[level] = []+ monsters_by_danger[level].append(monster)++ # 从低到高显示怪物+ print("\n[怪物按危险等级分类]\n")+ for level in sorted(monsters_by_danger.keys()):+ level_monsters = monsters_by_danger[level]+ print(f"[!] 危险等级 {level}:")+ for monster in sorted(level_monsters, key=lambda m: m.name):+ print(f"\n ● 怪物名称:{monster.name}")+ if monster.species:+ print(f" 种族:{monster.species}")+ if monster.habitats:+ print(f" 栖息地:{monster.habitats}")+ if monster.characteristics:+ print(f" 特征:{monster.characteristics}")+ if monster.weaknesses:+ print(f" 弱点:{monster.weaknesses}")+ if monster.loot_drops:+ print(f" 掉落物:{monster.loot_drops}")++ # 演示集合操作print("\n" + "=" * 80)- print("🔍 DEMONSTRATING SET OPERATIONS")+ print("[*] 演示集合操作和语义搜索")print("=" * 80)- # Build index for semantic search- print("\nBuilding vector index for semantic search...")- a_skill_set.build_index()- print("✓ Index built successfully")-- # Search for related skills- print("\n1. Semantic Search - Finding skills related to 'backend development':")- backend_skills = a_skill_set.search("backend development", top_k=5)- for skill in backend_skills:- print(f" • {skill.name}")- if skill.category:- print(f" Category: {skill.category}")-- # Check if specific skills exist- print("\n2. Membership Testing:")- test_skills = ["Python", "Java", "Docker", "Rust"]- for skill_name in test_skills:- exists = a_skill_set.contains(skill_name)- status = "✓" if exists else "✗"- print(f" {status} {skill_name}: {'Found' if exists else 'Not found'}")-- # Get specific skill details- print("\n3. Get Specific Skill Details:")- python_skill = a_skill_set.get("Python")- if python_skill:- print(f" Skill: {python_skill.name}")- print(f" Category: {python_skill.category}")- print(f" Level: {python_skill.proficiency_level or 'Not specified'}")- print(f" Description: {python_skill.description or 'No description'}")-- # Save the extracted knowledge+ # 构建向量索引以支持语义搜索+ print("\n正在构建向量索引...")+ monster_codex.build_index()+ print("√ 索引构建成功")++ # 语义搜索示例+ print("\n[1] 语义搜索 - 寻找掉落金币的怪物:")+ treasure_monsters = monster_codex.search("掉落金币财宝", top_k=5)+ for monster in treasure_monsters:+ print(f" • {monster.name}")+ if monster.loot_drops:+ print(f" 掉落物:{monster.loot_drops}")++ print("\n[2] 语义搜索 - 寻找怕火焰的怪物:")+ fire_weak_monsters = monster_codex.search("害怕火焰弱点火", top_k=5)+ for monster in fire_weak_monsters:+ print(f" • {monster.name}")+ if monster.weaknesses:+ print(f" 弱点:{monster.weaknesses}")++ print("\n[3] 语义搜索 - 寻找在地下出现的怪物:")+ underground_monsters = monster_codex.search("地下洞穴矿井", top_k=5)+ for monster in underground_monsters:+ print(f" • {monster.name}")+ if monster.habitats:+ print(f" 栖息地:{monster.habitats}")++ # 检查特定怪物是否存在+ print("\n[4] 成员检验 - 检查特定怪物是否在图鉴中:")+ check_monsters = ["地精小卒", "洞穴巨魔", "龙", "骷髅战士"]+ for monster_name in check_monsters:+ exists = monster_codex.contains(monster_name)+ status = "√" if exists else "×"+ print(f" {status} {monster_name}:{'已记录' if exists else '未记录'}")++ # 获取特定怪物的详细信息+ print("\n[5] 查询特定怪物详情:")+ goblin = monster_codex.get("地精小卒")+ if goblin:+ print(f" 怪物名称:{goblin.name}")+ print(f" 种族:{goblin.species or '未知'}")+ print(f" 危险等级:{goblin.danger_level or '未知'}/10")+ print(f" 栖息地:{goblin.habitats or '未知'}")+ print(f" 特征:{goblin.characteristics or '无'}")+ print(f" 弱点:{goblin.weaknesses or '无已知弱点'}")+ print(f" 掉落物:{goblin.loot_drops or '无'}")++ # 保存提取的知识print("\n" + "=" * 80)- print("💾 SAVING EXTRACTED KNOWLEDGE")+ print("[*] 保存怪物图鉴")print("=" * 80)- save_path = "tmp/auto_set_demo"- a_skill_set.dump(save_path)- print(f"✓ Saved to: {save_path}")- print(" - state.json (structured data)")- print(" - faiss_index/ (vector index for semantic search)")+ save_path = "tmp/monster_codex"+ monster_codex.dump(save_path)+ print(f"√ 已保存到:{save_path}")+ print(" - state.json(结构化数据)")+ print(" - faiss_index/(向量索引)")- # Demonstrate loading- print("\n📂 Loading saved knowledge...")- a_loaded_skill_set = AutoSet(- item_schema=SkillSchema,+ # 演示加载功能+ print("\n[*] 加载已保存的怪物图鉴...")+ loaded_codex = AutoSet(+ item_schema=MonsterEntry,llm_client=llm,embedder=embedder,key_extractor=lambda x: x.name,)- a_loaded_skill_set.load(save_path)- print(f"✓ Loaded {len(a_loaded_skill_set)} unique skills")+ loaded_codex.load(save_path)+ print(f"√ 加载成功!图鉴中有 {len(loaded_codex)} 只怪物")++ # 显示已加载的怪物列表+ print("\n[已加载怪物列表]")+ for monster in loaded_codex.items:+ print(f" • {monster.name} (危险等级:{monster.danger_level}/10)")- # Summary+ # 总结print("\n" + "=" * 80)- print("📈 SUMMARY")+ print("[*] 演示总结")print("=" * 80)- print(f" Total unique skills extracted: {len(a_skill_set)}")- print(f" Job descriptions processed: {len(job_descriptions)}")- print(" Deduplication strategy: FIELD_MERGE")- print(" Key extractor: lambda x: x.name")- print("\n ✓ Automatically deduplicated based on skill names")- print(" ✓ Merged information from multiple occurrences")- print(" ✓ Supports semantic search across all skills")- print(" ✓ Persistent storage with dump/load")+ print(f" 提取的不同怪物数量:{len(monster_codex)}")+ print(f" 处理的冒险者日志数:{len(adventurer_logs)}")+ print(" 去重策略:LLM_MERGE(智能合并)")+ print(" 去重键:lambda x: x.name")+ print("\n √ 基于怪物名称自动去重")+ print(" √ 智能合并多个来源的冲突信息")+ print(" √ 支持所有怪物的语义搜索")+ print(" √ 持久化存储和加载")print("=" * 80)diff --git a/hyperextract/core/base.py b/hyperextract/core/base.pyindex ba1fdc4..6672854 100644--- a/hyperextract/core/base.py+++ b/hyperextract/core/base.py@@ -2,6 +2,7 @@ from abc import ABC, abstractmethodfrom pathlib import Pathfrom typing import TypeVar, Generic, Any, Dict, Type, Listfrom datetime import datetime+import jsonfrom pydantic import BaseModelfrom langchain_core.language_models.chat_models import BaseChatModelfrom langchain_core.embeddings import Embeddings@@ -16,7 +17,7 @@ T = TypeVar("T", bound=BaseModel)class BaseAutoType(ABC, Generic[T]):- """Unified knowledge base class integrating extraction, storage, aggregation, and evolution.+ """Unified knowledge base class integrating extraction, storage, and aggregation.This abstract base class provides a complete framework for managing structured knowledgeextracted from text. It handles the full lifecycle from extraction to serialization.@@ -26,7 +27,6 @@ class BaseAutoType(ABC, Generic[T]):- Automatically handle long text chunking and parallel processing- Store and aggregate extracted knowledge with configurable merge strategies- Build and maintain vector indices for semantic search- - Support knowledge evolution and optimization- Provide serialization and deserialization capabilities"""@@ -46,7 +46,7 @@ class BaseAutoType(ABC, Generic[T]):Args:data_schema: Pydantic BaseModel subclass defining the knowledge structure.- llm_client: Language model client for extraction and evolution.+ llm_client: Language model client for extraction.embedder: Embedding model for semantic search and similarity computation.prompt: Custom prompt template for extraction (defaults to generic prompt).chunk_size: Maximum chunk size for splitting long texts.@@ -87,6 +87,9 @@ class BaseAutoType(ABC, Generic[T]):"updated_at": datetime.now(),}+ # Initialize internal state (calls hook for subclass setup)+ self._init_internal_state()+def _create_new_instance(self) -> "BaseAutoType[T]":"""Creates a new instance with the same configuration as this one.@@ -137,11 +140,49 @@ class BaseAutoType(ABC, Generic[T]):"""return self._data+ # ==================== State Management Lifecycle Hooks ====================++ def _init_internal_state(self) -> None:+ """+ Protected hook to initialize internal state.+ Called at the END of __init__ to ensure all basic attributes are set first.++ Subclasses can override to initialize their own structures (e.g., _items_dict for AutoSet).+ """+ self._data = self._data_schema()+ self._index = None++ def _set_internal_state(self, data: T) -> None:+ """+ Protected hook to update internal state.+ Called whenever data is modified (extract, feed, load).++ Responsibilities:+ 1. Update self._data+ 2. Invalidate vector index (data changed)+ 3. Subclasses override to sync auxiliary structures++ Args:+ data: The new data object to set.+ """+ self._data = data+ self.clear_index()++ def _clear_internal_state(self) -> None:+ """+ Protected hook to fully clear internal state.+ Called in clear() method.++ Default: Reset to empty schema instance via _set_internal_state hook.+ """+ self._set_internal_state(self._data_schema())++ # ==================== Data Management ====================+def clear(self):"""Clears all knowledge including data and vector index."""+ self._clear_internal_state()self.metadata["updated_at"] = datetime.now()- self._data = self._data_schema()- self.clear_index()def clear_index(self):"""Clears the vector index without affecting the stored data."""@@ -149,31 +190,72 @@ class BaseAutoType(ABC, Generic[T]):# ==================== Extraction & Merge ====================- @abstractmethod- def extract(self, text: str, *, store: bool = True) -> T:- """Extracts structured knowledge from text with automatic chunking and merging.+ def _extract_data(self, text: str) -> T:+ """+ Internal: Unified extraction logic (Chunking -> LLM -> Merge).+ Implemented in BaseAutoType for code reuse by List and Set.- This is the main entry point for knowledge extraction. It handles the complete pipeline- from text preprocessing to knowledge merging.+ Args:+ text: Input text.++ Returns:+ The extracted knowledge data.+ """+ if len(text) <= self.chunk_size:+ extracted_data = self.llm_chain_extract.invoke({"chunk_text": text})+ extracted_data_list = [extracted_data]+ else:+ chunks = self.text_splitter.split_text(text)+ inputs = [{"chunk_text": chunk} for chunk in chunks]+ extracted_data_list = self.llm_chain_extract.batch(+ inputs, config={"max_concurrency": self.max_workers}+ )++ merged_data = self.merge(extracted_data_list)+ return merged_data++ def extract(self, text: str) -> "BaseAutoType[T]":+ """+ Extracts knowledge into a NEW instance without modifying the current one.- Extraction pipeline:- 1. Determine if text chunking is needed based on length- 2. Extract knowledge from each chunk using LLM- 3. Merge all extracted data using the merge() method- 4. If store=True: merge with existing data and update internal state- 5. If store=False: return extracted data without modifying internal state- 6. Return the merged knowledge+ Use this for previewing data or branching knowledge bases.Args:- text: Input text to extract knowledge from (can be short or long).- store: Controls whether to store extracted knowledge internally.- - True (default): Store mode - merge with existing knowledge and update internal state- - False: Temporary mode - return extracted data without modifying internal state+ text: Input text.Returns:- The extracted and merged knowledge data.+ A new knowledge instance containing only the extracted data."""- pass+ extracted_data = self._extract_data(text)++ new_instance = self._create_new_instance()+ new_instance._set_internal_state(extracted_data)++ new_instance.metadata["created_at"] = datetime.now()+ new_instance.metadata["updated_at"] = datetime.now()++ return new_instance++ def feed(self, text: str) -> "BaseAutoType[T]":+ """+ Ingests text into the CURRENT knowledge base instance.++ This modifies the internal state by merging new data with existing data.+ Supports method chaining (e.g., kb.feed(text1).feed(text2)).++ Args:+ text: Input text.++ Returns:+ Self (the current instance).+ """+ extracted_data = self._extract_data(text)+ merged_data = self.merge([self._data, extracted_data])++ self._set_internal_state(merged_data)+ self.metadata["updated_at"] = datetime.now()++ return self@abstractmethoddef merge(self, data_list: List[T]) -> T:@@ -195,24 +277,6 @@ class BaseAutoType(ABC, Generic[T]):"""pass- # ==================== Evolution ====================-- def evolve(self, **kwargs) -> T:- """Evolves and optimizes the internal knowledge through inference and refinement.-- Subclasses can implement this method to perform knowledge-specific evolution strategies.-- Evolution capabilities:- - Infer implicit relationships between entities- - Prune low-confidence nodes and connections- - Apply clustering for better organization- - Complete missing knowledge through reasoning-- Returns:- The evolved knowledge data.- """- pass-# ==================== Indexing & Query ====================@abstractmethod@@ -246,34 +310,123 @@ class BaseAutoType(ABC, Generic[T]):# ==================== Serialization ====================- @abstractmethod- def dump(self, folder_path: str | Path):+ def dump(self, folder_path: str | Path) -> None:"""Exports knowledge to a specified folder.- Subclasses must implement this method to persist their knowledge data.-- Files to save:- 1. Structured data (self._data) - saved as JSON file+ Saves to the folder:+ 1. Structured data (self._data) - saved as state.json2. Vector index (self._index) - saved as FAISS index filesArgs:- folder_path: Target folder path for saving knowledge.+ folder_path: Target folder path for saving."""- pass+ from langchain_community.vectorstores import FAISS- @abstractmethod- def load(self, folder_path: str | Path):- """Loads knowledge from a specified folder.+ folder = Path(folder_path)+ if folder.exists() and folder.is_file():+ raise Exception("Folder path is a file, please provide a folder path.")++ if not folder.exists():+ folder.mkdir(parents=True, exist_ok=True)++ try:+ # 1. Save structured data+ data = {+ "schema_name": self._data_schema.__name__,+ "data_schema": self._data_schema.model_json_schema(),+ "data": self._data.model_dump(),+ "metadata": self._prepare_metadata_for_dump(),+ }++ # Allow subclasses to add extra fields+ data.update(self._get_extra_dump_data())- Subclasses must implement this method to restore previously saved knowledge.+ json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)+ data_file = folder / "state.json"+ with open(data_file, "w", encoding="utf-8") as f:+ f.write(json_str)- Files to load:- 1. Structured data (self._data) - loaded from JSON file+ # 2. Save vector index+ if self._index is not None:+ index_path = str(folder / "faiss_index")+ self._index.save_local(index_path)++ except Exception as e:+ raise e++ def load(self, folder_path: str | Path) -> None:+ """Loads knowledge from a specified folder.++ Loads from the folder:+ 1. Structured data (self._data) - loaded from state.json2. Vector index (self._index) - loaded from FAISS index filesArgs:folder_path: Source folder path containing saved knowledge."""+ from langchain_community.vectorstores import FAISS++ folder = Path(folder_path)+ if not folder.is_dir():+ raise ValueError(f"Folder does not exist: {folder_path}")++ # 1. Load structured data+ data_file = folder / "state.json"+ if not data_file.is_file():+ raise ValueError(f"Data file not found: {data_file}")++ with open(data_file, "r", encoding="utf-8") as f:+ data = json.load(f)++ # Load data using hook for proper state sync+ loaded_data = self._data_schema.model_validate(data.get("data", {}))+ self._set_internal_state(loaded_data)++ # Update metadata with loaded values+ if "metadata" in data:+ self.metadata.update(data["metadata"])+ self.metadata["updated_at"] = datetime.now()++ # Allow subclasses to load extra data+ self._load_extra_data(data)++ # 2. Load vector index+ index_path = str(folder / "faiss_index")+ if Path(index_path).exists():+ try:+ self._index = FAISS.load_local(+ index_path, self.embedder, allow_dangerous_deserialization=True+ )+ except Exception as e:+ self._index = None++ # ==================== Serialization Helpers ====================++ def _prepare_metadata_for_dump(self) -> Dict[str, Any]:+ """Helper to serialize metadata values (e.g., datetimes)."""+ return {+ k: str(v) if isinstance(v, datetime) else v+ for k, v in self.metadata.items()+ }++ def _get_extra_dump_data(self) -> Dict[str, Any]:+ """Hook for subclasses to add extra fields to state.json.++ Override this method to save subclass-specific data (e.g., merge strategy).++ Returns:+ Dictionary of extra data to include in state.json.+ """+ return {}++ def _load_extra_data(self, json_data: Dict[str, Any]) -> None:+ """Hook for subclasses to extract extra fields from state.json.++ Override this method to restore subclass-specific data.++ Args:+ json_data: The loaded JSON data dictionary.+ """pass# ==================== Operator Overloads ====================@@ -285,11 +438,11 @@ class BaseAutoType(ABC, Generic[T]):The new instance inherits configuration from the left operand (self).Usage:- >>> kb1 = UnitKnowledge(PersonSchema, ...)- >>> kb2 = UnitKnowledge(PersonSchema, ...)+ >>> kb1 = AutoList(PersonSchema, ...)+ >>> kb2 = AutoList(PersonSchema, ...)>>> kb3 = kb1 + kb2 # ✅ Same schema, creates merged instance>>>- >>> kb4 = UnitKnowledge(CompanySchema, ...)+ >>> kb4 = AutoList(CompanySchema, ...)>>> kb5 = kb1 + kb4 # ❌ TypeError: Different schemasArgs:@@ -323,12 +476,11 @@ class BaseAutoType(ABC, Generic[T]):# Create a new instance with the same configurationnew_instance = self._create_new_instance()- # Set the merged data and update metadata- new_instance._data = merged_data+ # Set the merged data using hook and update metadata+ new_instance._set_internal_state(merged_data)new_instance.metadata["created_at"] = min(self.metadata["created_at"], other.metadata["created_at"])new_instance.metadata["updated_at"] = datetime.now()- new_instance.clear_index() # Index needs to be rebuiltreturn new_instancediff --git a/hyperextract/core/list.py b/hyperextract/core/list.pyindex 0c43039..3bd3a37 100644--- a/hyperextract/core/list.py+++ b/hyperextract/core/list.py@@ -1,9 +1,18 @@"""List Knowledge Pattern - extracts a collection of objects from text."""-import json-from typing import List, Any, Type, TypeVar, Generic, TYPE_CHECKING+from typing import (+ List,+ Any,+ Type,+ TypeVar,+ Generic,+ TYPE_CHECKING,+ Iterator,+ Callable,+ Iterable,+ Union,+)from datetime import datetime-from pathlib import Pathfrom pydantic import BaseModel, Field, create_modelfrom langchain_core.language_models.chat_models import BaseChatModelfrom langchain_core.embeddings import Embeddings@@ -132,66 +141,6 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):show_progress=self.show_progress,)- def extract(self, text: str, *, store: bool = True) -> AutoListSchema:- """Extracts a list of items using LangChain native batch processing.-- Args:- text: Input text to extract items from.- store: Controls whether to store extracted knowledge internally.- - True (default): Store mode - merge with existing knowledge and update internal state- - False: Temporary mode - return extracted data without modifying internal state-- Returns:- The list of extracted items.- """- start_time = datetime.now()-- # Determine if chunking is needed- if len(text) <= self.chunk_size:- # Short text: extract in one pass- if self.show_progress:- logger.info(f"Processing single text (length: {len(text)})...")-- extracted_data = self.llm_chain_extract.invoke({"chunk_text": text})- extracted_data_list = [extracted_data]-- else:- # Long text: extract by chunking- chunks = self.text_splitter.split_text(text)- logger.info(f"Split text into {len(chunks)} chunks")-- if self.show_progress:- logger.info(f"Processing {len(chunks)} chunk(s)...")-- inputs = [{"chunk_text": chunk} for chunk in chunks]- extracted_data_list = self.llm_chain_extract.batch(- inputs, config={"max_concurrency": self.max_workers}- )-- if self.show_progress:- logger.info(f"Extracted {len(extracted_data_list)} chunks")-- logger.info("Merging extracted knowledge...")- merged_data = self.merge(extracted_data_list)-- # If store=True, merge with existing data and update internal state- if store:- if self._data and len(self.items) > 0:- self._data = self.merge([self._data, merged_data])- else:- self._data = merged_data-- self.clear_index()- self.metadata["updated_at"] = datetime.now()-- logger.info("Knowledge extraction completed")- logger.info(- f"Duration: {(datetime.now() - start_time).total_seconds():.2f} seconds"- )-- # Return items list (either from stored data or temporary merged data)- return self.items if store else merged_data.items-def merge(self, data_list: List[AutoListSchema]) -> AutoListSchema:"""Pure data merge method implementing list append strategy.@@ -215,7 +164,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):# Return a new container objectreturn self.item_list_schema(items=all_items)- def build_index(self):+ def build_index(self) -> None:"""Builds independent vector index for each item in the list."""items = self.itemsif not items:@@ -244,7 +193,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):logger.error("FAISS not available. Install with: pip install faiss-cpu")raise- def search(self, query: str, top_k: int = 3) -> List[Any]:+ def search(self, query: str, top_k: int = 3) -> List[Item]:"""Searches items in the list using semantic similarity.Args:@@ -262,7 +211,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):raise Exception("Vector store not initialized")docs = self._index.similarity_search(query, k=top_k)- results = []+ results: List[Item] = []for doc in docs:# Attempt to restore as objecttry:@@ -276,89 +225,13 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):logger.info(f"Found {len(results)} results for query: {query[:50]}...")return results- def dump(self, folder_path: str) -> Any:- """Exports knowledge to a specified folder.-- Uses the same serialization logic as UnitKnowledge since the container- is also a BaseModel.- """- folder = Path(folder_path)- folder.mkdir(parents=True, exist_ok=True)-- # 1. Save structured data- data = {- "schema_name": self.item_list_schema.__name__,- "item_schema_name": self.item_schema.__name__,- "item_schema": self.item_schema.model_json_schema(),- "data": self.data.model_dump(),- "metadata": {- k: str(v) if isinstance(v, datetime) else v- for k, v in self.metadata.items()- },- }-- json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)- data_file = folder / "state.json"- with open(data_file, "w", encoding="utf-8") as f:- f.write(json_str)- logger.info(f"Saved data to {data_file}")-- # 2. Save vector index- if self._index is not None:- index_path = str(folder / "faiss_index")- self._index.save_local(index_path)- logger.info(f"Saved FAISS index to {index_path}")- else:- logger.warning("No index to save")-- return json_str-- def load(self, folder_path: str):- """Loads knowledge from a specified folder."""- folder = Path(folder_path)- if not folder.is_dir():- raise ValueError(f"Folder does not exist: {folder_path}")-- # 1. Load structured data- data_file = folder / "state.json"- if not data_file.is_file():- raise ValueError(f"Data file not found: {data_file}")-- with open(data_file, "r", encoding="utf-8") as f:- data = json.load(f)- logger.info(f"Loaded data from {data_file}")-- self._data = self.item_list_schema.model_validate(data.get("data", {}))-- # Update metadata with loaded values- if "metadata" in data:- self.metadata.update(data["metadata"])- self.metadata["updated_at"] = datetime.now()-- # 2. Load vector index- index_path = str(folder / "faiss_index")- if Path(index_path).exists():- try:- self._index = FAISS.load_local(- index_path, self.embedder, allow_dangerous_deserialization=True- )- logger.info(f"Loaded FAISS index from {index_path}")- except Exception as e:- logger.warning(f"Failed to load FAISS index: {e}")- self._index = None- else:- logger.warning("No index file found, will rebuild on next search")- self._index = None-- logger.info(f"Loaded knowledge successfully with {len(self)} items")-# ==================== Pythonic Sequence Operations ====================def __len__(self) -> int:"""Returns the number of elements in the list."""return len(self.items)- def __getitem__(self, key):+ def __getitem__(self, key: Union[int, slice]) -> Union[Item, "AutoList[Item]"]:"""Support index access and slicing.Args:@@ -392,7 +265,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):f"List indices must be integers or slices, not {type(key).__name__}")- def __setitem__(self, index: int, item):+ def __setitem__(self, index: int, item: Item) -> None:"""Support index assignment.Args:@@ -416,17 +289,17 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):self.clear_index()self.metadata["updated_at"] = datetime.now()- def __add__(self, other):+ def __add__(self, other: "BaseAutoType[Item]") -> "AutoList[Item]":"""Operator overload for '+' to combine knowledge instances.Supports multiple combination patterns:- AutoList + AutoList → AutoList (merge lists)- - AutoList + UnitKnowledge → AutoList (append unit to list)+ - AutoList + AutoModel → AutoList (append model to list)- This enables chain operations like: unit1 + unit2 + unit3+ This enables chain operations like: model1 + model2 + model3Args:- other: Another AutoList or UnitKnowledge with compatible schema.+ other: Another AutoList or AutoModel with compatible schema.Returns:New AutoList with combined items.@@ -436,7 +309,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):"""from .model import AutoModel- # Case 1: AutoList + AutoList (call parent implementation)+ # Case 1: AutoList + AutoListif isinstance(other, AutoList):# Check schema compatibilityif self.item_schema != other.item_schema:@@ -444,7 +317,6 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):f"Cannot add AutoList instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")- # Manually merge without calling base class (to avoid _data_schema check)# Create new instance with merged itemsnew_instance = self._create_new_instance()new_instance._data.items = self.items + other.items@@ -452,7 +324,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):new_instance.metadata["updated_at"] = self.metadata.get("updated_at")return new_instance- # Case 2: AutoList + AutoModel → AutoList (append unit)+ # Case 2: AutoList + AutoModel → AutoList (append model)elif isinstance(other, AutoModel):# Check schema compatibilityif self.item_schema != other._data_schema:@@ -461,7 +333,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):f"List: {self.item_schema.__name__}, Unit: {other._data_schema.__name__}")- # Create new AutoList with appended unit+ # Create new AutoList with appended modelnew_list = self._create_new_instance()new_list._data.items = self.items + [other._data]@@ -478,7 +350,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):f"Unsupported operand type for +: 'AutoList' and '{type(other).__name__}'")- def __delitem__(self, index: int):+ def __delitem__(self, index: int) -> None:"""Support del operation for removing items by index.Args:@@ -499,7 +371,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):self.clear_index()self.metadata["updated_at"] = datetime.now()- def __iter__(self):+ def __iter__(self) -> Iterator[Item]:"""Support iteration over items.Returns:@@ -514,7 +386,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):"""return iter(self.items)- def __contains__(self, item) -> bool:+ def __contains__(self, item: Item) -> bool:"""Support 'in' operator for membership testing.Args:@@ -564,7 +436,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):# ==================== List Modification Methods ====================- def append(self, item) -> None:+ def append(self, item: Item) -> None:"""Append a single item to the end of the list.Args:@@ -585,7 +457,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):self.clear_index()self.metadata["updated_at"] = datetime.now()- def extend(self, items) -> None:+ def extend(self, items: Union[Iterable[Item], "AutoList[Item]"]) -> None:"""Extend the list by appending multiple items.Args:@@ -626,7 +498,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):self.clear_index()self.metadata["updated_at"] = datetime.now()- def insert(self, index: int, item) -> None:+ def insert(self, index: int, item: Item) -> None:"""Insert an item at a specific position.Args:@@ -649,7 +521,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):self.clear_index()self.metadata["updated_at"] = datetime.now()- def remove(self, item) -> None:+ def remove(self, item: Item) -> None:"""Remove the first occurrence of an item from the list.Args:@@ -678,7 +550,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):# Item not foundraise ValueError(f"{item} is not in list")- def pop(self, index: int = -1):+ def pop(self, index: int = -1) -> Item:"""Remove and return an item at the given position.Args:@@ -708,7 +580,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):# ==================== Query and Utility Methods ====================- def index(self, item, start: int = 0, stop: int = None) -> int:+ def index(self, item: Item, start: int = 0, stop: int | None = None) -> int:"""Return the index of the first occurrence of item.Args:@@ -736,7 +608,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):raise ValueError(f"{item} is not in list")- def count(self, item) -> int:+ def count(self, item: Item) -> int:"""Return the number of times item appears in the list.Args:@@ -800,7 +672,9 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):self._index = Noneself.build_index()- def sort(self, key=None, reverse: bool = False) -> None:+ def sort(+ self, key: Union[Callable[[Item], Any], None] = None, reverse: bool = False+ ) -> None:"""Sort the items in place.Args:@@ -839,7 +713,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):# ==================== Helper Methods ====================- def _validate_item_schema(self, item) -> None:+ def _validate_item_schema(self, item: Any) -> None:"""Validate that item's schema matches item_schema.Args:@@ -884,7 +758,7 @@ class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):raise TypeError(" ".join(error_parts))- def _items_equal(self, item1, item2) -> bool:+ def _items_equal(self, item1: BaseModel, item2: BaseModel) -> bool:"""Check if two items are equal.Args:diff --git a/hyperextract/core/model.py b/hyperextract/core/model.pyindex da7b40e..b7654d7 100644--- a/hyperextract/core/model.py+++ b/hyperextract/core/model.py@@ -1,9 +1,7 @@"""Unit Knowledge Pattern - extracts a single structured object from text."""-import jsonfrom typing import List, Any, Typefrom datetime import datetime-from pathlib import Pathfrom langchain_core.language_models.chat_models import BaseChatModelfrom langchain_core.embeddings import Embeddingsfrom langchain_core.documents import Document@@ -225,134 +223,38 @@ class AutoModel(BaseAutoType[T]):logger.info(f"Found {len(results)} results for query: {query[:50]}...")return results- # ==================== Serialization ====================-- def dump(self, folder_path: str | Path):- """Exports knowledge to a specified folder.-- Saves to the folder:- 1. Structured data (self._data) - saved as state.json- 2. Vector index (self._index) - saved as FAISS index files-- Args:- folder_path: Target folder path for saving.- """-- folder = Path(folder_path)- if folder.exists() and folder.is_file():- raise Exception("Folder path is a file, please provide a folder path.")-- if not folder.exists():- logger.info(f"Creating folder: {folder}")- folder.mkdir(parents=True, exist_ok=True)-- try:- # 1. Save structured data- data = {- "schema_name": self.data_schema.__name__,- "data_schema": self.data_schema.model_json_schema(),- "data": self.data.model_dump(),- "metadata": {- k: str(v) if isinstance(v, datetime) else v- for k, v in self.metadata.items()- },- }-- json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)- data_file = folder / "state.json"- with open(data_file, "w", encoding="utf-8") as f:- f.write(json_str)- logger.info(f"Saved data to {data_file}")-- # 2. Save vector index- if self._index is not None:- index_path = str(folder / "faiss_index")- self._index.save_local(index_path)- logger.info(f"Saved FAISS index to {index_path}")- else:- logger.warning("No index to save")-- except Exception as e:- logger.error(f"Failed to dump knowledge: {e}")-- def load(self, folder_path: str | Path):- """Loads knowledge from a specified folder.-- Loads from the folder:- 1. Structured data (self._data) - loaded from state.json- 2. Vector index (self._index) - loaded from FAISS index files-- Args:- folder_path: Source folder path containing saved knowledge.- """- folder = Path(folder_path)- if not folder.is_dir():- raise ValueError(f"Folder does not exist: {folder_path}")-- # 1. Load structured data- data_file = folder / "state.json"- if not data_file.is_file():- raise ValueError(f"Data file not found: {data_file}")-- with open(data_file, "r", encoding="utf-8") as f:- data = json.load(f)- logger.info(f"Loaded data from {data_file}")-- self._data = self.data_schema.model_validate(data.get("data", {}))-- # Update metadata with loaded values- if "metadata" in data:- self.metadata.update(data["metadata"])- self.metadata["updated_at"] = datetime.now()-- # 2. Load vector index- index_path = str(folder / "faiss_index")- if Path(index_path).exists():- try:- from langchain_community.vectorstores import FAISS-- self._index = FAISS.load_local(- index_path, self.embedder, allow_dangerous_deserialization=True- )- logger.info(f"Loaded FAISS index from {index_path}")- except Exception as e:- logger.warning(f"Failed to load FAISS index: {e}")- self._index = None- else:- logger.warning("No index file found, will rebuild on next search")- self._index = None-- logger.info(f"Loaded knowledge successfully with {len(self)} unit(s)")-# ==================== Operators ====================def __add__(self, other):- """Operator overload for '+' to combine AutoModel instances into ListKnowledge.+ """Operator overload for '+' to combine AutoModel instances into AutoList.++ Supports multiple combination patterns:+ - AutoModel + AutoModel → AutoList (create list from both items)+ - AutoModel + AutoList → AutoList (prepend model to list)- When two AutoModel instances are added, they are combined into a ListKnowledge- containing both objects as separate items. This enables intuitive collection building.+ This enables intuitive chain operations like: unit1 + unit2 + unit3Usage:>>> unit1 = AutoModel(PersonSchema, ...)>>> unit2 = AutoModel(PersonSchema, ...)- >>> person_list = unit1 + unit2 # → ListKnowledge[PersonSchema]+ >>> person_list = unit1 + unit2 # → AutoList[PersonSchema]>>>>>> # Chain operations>>> unit3 = AutoModel(PersonSchema, ...)- >>> person_list = unit1 + unit2 + unit3 # → ListKnowledge with 3 items+ >>> person_list = unit1 + unit2 + unit3 # → AutoList with 3 itemsArgs:- other: Another AutoModel with the same data schema, or ListKnowledge.+ other: Another AutoModel with the same data schema, or AutoList.Returns:- ListKnowledge containing both objects as items.+ AutoList containing both objects as items.Raises:TypeError: If schemas don't match or invalid operand type."""- from .list import ListKnowledge+ from .list import AutoList- # Case 1: AutoModel + AutoModel → ListKnowledge+ # Case 1: AutoModel + AutoModel → AutoListif isinstance(other, AutoModel):# Check schema compatibilityif self._data_schema != other._data_schema:@@ -361,8 +263,8 @@ class AutoModel(BaseAutoType[T]):f"Left: {self._data_schema.__name__}, Right: {other._data_schema.__name__}")- # Create new ListKnowledge- list_kb = ListKnowledge(+ # Create new AutoList+ list_kb = AutoList(item_schema=self._data_schema,llm_client=self.llm_client,embedder=self.embedder,@@ -384,17 +286,17 @@ class AutoModel(BaseAutoType[T]):return list_kb- # Case 2: AutoModel + ListKnowledge → ListKnowledge (for reverse order)- elif isinstance(other, ListKnowledge):+ # Case 2: AutoModel + AutoList → AutoList (prepend model)+ elif isinstance(other, AutoList):# Check schema compatibilityif self._data_schema != other.item_schema:raise TypeError(- f"Cannot add AutoModel to ListKnowledge with different schemas. "+ f"Cannot add AutoModel to AutoList with different schemas. "f"Unit: {self._data_schema.__name__}, List: {other.item_schema.__name__}")- # Create new ListKnowledge with unit prepended- new_list = ListKnowledge(+ # Create new AutoList with unit prepended+ new_list = AutoList(item_schema=self._data_schema,llm_client=self.llm_client,embedder=self.embedder,diff --git a/hyperextract/core/set.py b/hyperextract/core/set.pyindex db9198f..70153c7 100644--- a/hyperextract/core/set.py+++ b/hyperextract/core/set.py@@ -4,10 +4,8 @@ Provides automatic deduplication based on a user-specified unique key field.Supports multiple merge strategies including LLM-powered intelligent merging."""-import jsonfrom typing import List, Any, Type, TypeVar, Generic, Dict, Optional, Callablefrom datetime import datetime-from pathlib import Pathfrom pydantic import BaseModel, Field, create_modelfrom langchain_core.language_models.chat_models import BaseChatModelfrom langchain_core.embeddings import Embeddings@@ -136,14 +134,11 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):**kwargs,)- # AutoSet-specific attributes+ # AutoSet-specific attributes (before _init_internal_state is called by parent)self.key_extractor = key_extractorself.merge_item_strategy = merge_item_strategyself.llm_batch_size = llm_batch_size- # Internal storage: Dict for O(1) deduplication- self._items_dict: Dict[Any, Item] = {}-# Create Merger instance based on strategyself._merger: BaseMerger[Item] = self._create_merger(item_schema=item_schema,@@ -289,70 +284,42 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):"""return iter(self.items)- # ==================== Core Override Methods ====================-- def clear(self):- """Clears all items from the set."""- self._data = self.item_set_schema()- self._items_dict.clear()- self.clear_index()- self.metadata["updated_at"] = datetime.now()-- def extract(self, text: str, *, store: bool = True):- """Extracts a set of unique items using LLM with automatic deduplication.-- Args:- text: Input text to extract items from.- store: Whether to store extracted knowledge internally.+ # ==================== State Management Lifecycle Overrides ====================- Returns:- The set of extracted unique items.+ def _init_internal_state(self) -> None:"""- start_time = datetime.now()-- # Determine if chunking is needed- if len(text) <= self.chunk_size:- # Short text: extract in one pass- if self.show_progress:- logger.info(f"Processing single text (length: {len(text)})...")-- extracted_data = self.llm_chain_extract.invoke({"chunk_text": text})- extracted_data_list = [extracted_data]-- else:- # Long text: extract by chunking- chunks = self.text_splitter.split_text(text)- logger.info(f"Split text into {len(chunks)} chunks")-- if self.show_progress:- logger.info(f"Processing {len(chunks)} chunk(s)...")-- inputs = [{"chunk_text": chunk} for chunk in chunks]- extracted_data_list = self.llm_chain_extract.batch(- inputs, config={"max_concurrency": self.max_workers}- )+ Overrides base hook to initialize AutoSet-specific structures.+ Called at the end of __init__.+ """+ # 1. Initialize the dict (AutoSet-specific)+ self._items_dict: Dict[Any, Item] = {}- if self.show_progress:- logger.info(f"Extracted {len(extracted_data_list)} chunks")+ # 2. Call parent to initialize standard structures (_data, _index)+ super()._init_internal_state()- logger.info("Merging extracted knowledge...")- merged_data = self.merge(extracted_data_list)+ def _set_internal_state(self, data: AutoSetSchema[Item]) -> None:+ """+ Overrides base hook to ensure _items_dict stays synchronized with _data.+ This is the synchronization point for AutoSet's dual-structure design.+ """+ # 1. Sync dictionary (AutoSet-specific: the Real Source of Truth for lookups)+ self._items_dict.clear()+ for item in data.items:+ key = self._get_key_value(item)+ if key is not None:+ self._items_dict[key] = item- # If store=True, merge with existing data and update internal state- if store:- if self._data and len(self.items) > 0:- final_data = self.merge([self._data, merged_data])- else:- final_data = merged_data- self._update_internal_state(final_data)+ # 2. Call parent to set _data and clear index+ super()._set_internal_state(data)- logger.info("Knowledge extraction completed")- logger.info(- f"Duration: {(datetime.now() - start_time).total_seconds():.2f} seconds"- )+ def _clear_internal_state(self) -> None:+ """+ Overrides base hook to explicitly clear the dictionary before clearing _data.+ """+ self._items_dict.clear()+ super()._clear_internal_state()- # Return items list- return self.items if store else merged_data.items+ # ==================== Core Override Methods ====================def merge(self, data_list: List[AutoSetSchema[Item]]) -> AutoSetSchema[Item]:"""Merges multiple data containers with automatic deduplication.@@ -391,25 +358,6 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):# Step 3: Return new container (does not modify self)return self.item_set_schema(items=merged_items)- def _update_internal_state(self, data: AutoSetSchema[Item]) -> None:- """Updates internal state from a data container.-- Private method that synchronizes _items_dict and _data with new data.- Also clears vector index and updates metadata timestamp.-- Args:- data: The data container to update from.- """- self._items_dict.clear()- for item in data.items:- key = self._get_key_value(item)- if key is not None:- self._items_dict[key] = item-- self._data = data- self.clear_index()- self.metadata["updated_at"] = datetime.now()-def _get_key_value(self, item: Item) -> Optional[Any]:"""Extracts the unique key value from an item using key_extractor.@@ -503,92 +451,19 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):logger.info(f"Found {len(results)} results for query: {query[:50]}...")return results- # ==================== Serialization ====================-- def dump(self, folder_path: str | Path):- """Exports knowledge to a specified folder.+ # ==================== Serialization Helpers ====================- Saves both the items and the AutoSet-specific metadata.+ def _get_extra_dump_data(self) -> Dict[str, Any]:+ """Add AutoSet specific data to the dump.- Args:- folder_path: Target folder path for saving.+ Returns:+ Dictionary with extra AutoSet-specific fields."""- folder = Path(folder_path)- folder.mkdir(parents=True, exist_ok=True)-- # Save structured data- data = {- "schema_name": self.item_set_schema.__name__,+ return {"item_schema_name": self.item_schema.__name__,- "item_schema": self.item_schema.model_json_schema(),"merge_item_strategy": self.merge_item_strategy.value,- "data": self.data.model_dump(),- "metadata": {- k: str(v) if isinstance(v, datetime) else v- for k, v in self.metadata.items()- },}- json_str = json.dumps(data, indent=2, ensure_ascii=False, default=str)- data_file = folder / "state.json"- with open(data_file, "w", encoding="utf-8") as f:- f.write(json_str)- logger.info(f"Saved data to {data_file}")-- # Save vector index- if self._index is not None:- index_path = str(folder / "faiss_index")- self._index.save_local(index_path)- logger.info(f"Saved FAISS index to {index_path}")- else:- logger.warning("No index to save")-- def load(self, folder_path: str | Path):- """Loads knowledge from a specified folder.-- Args:- folder_path: Source folder path containing saved knowledge.- """- folder = Path(folder_path)- if not folder.is_dir():- raise ValueError(f"Folder does not exist: {folder_path}")-- # Load structured data- data_file = folder / "state.json"- if not data_file.is_file():- raise ValueError(f"Data file not found: {data_file}")-- with open(data_file, "r", encoding="utf-8") as f:- data = json.load(f)- logger.info(f"Loaded data from {data_file}")-- loaded_data = self.item_set_schema.model_validate(data.get("data", {}))- self._update_internal_state(loaded_data)-- # Update metadata from saved data- if "metadata" in data:- self.metadata.update(data["metadata"])- self.metadata["updated_at"] = datetime.now()-- # Load vector index- index_path = str(folder / "faiss_index")- if Path(index_path).exists():- try:- from langchain_community.vectorstores import FAISS-- self._index = FAISS.load_local(- index_path, self.embedder, allow_dangerous_deserialization=True- )- logger.info(f"Loaded FAISS index from {index_path}")- except Exception as e:- logger.warning(f"Failed to load FAISS index: {e}")- self._index = None- else:- logger.warning("No index file found, will rebuild on next search")- self._index = None-- logger.info(f"Loaded knowledge successfully with {len(self)} unique items")-# ==================== Set-Specific Methods ====================def add(self, item: Item) -> None:@@ -610,9 +485,9 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):# Add newself._items_dict[key_value] = item.model_copy(deep=True)- # Update internal state- self._data.items = list(self._items_dict.values())- self.clear_index()+ # Update state via hook+ new_data = self.item_set_schema(items=list(self._items_dict.values()))+ self._set_internal_state(new_data)self.metadata["updated_at"] = datetime.now()def remove(self, key: Any) -> Optional[Item]:@@ -626,8 +501,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):"""removed = self._items_dict.pop(key, None)if removed:- self._data.items = list(self._items_dict.values())- self.clear_index()+ new_data = self.item_set_schema(items=list(self._items_dict.values()))+ self._set_internal_state(new_data)self.metadata["updated_at"] = datetime.now()logger.debug(f"Removed item with key '{key}'")return removed@@ -681,8 +556,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):"""if key in self._items_dict:self._items_dict.pop(key)- self._data.items = list(self._items_dict.values())- self.clear_index()+ new_data = self.item_set_schema(items=list(self._items_dict.values()))+ self._set_internal_state(new_data)self.metadata["updated_at"] = datetime.now()def pop(self) -> Item:@@ -710,8 +585,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):key = next(iter(self._items_dict))item = self._items_dict.pop(key)- self._data.items = list(self._items_dict.values())- self.clear_index()+ new_data = self.item_set_schema(items=list(self._items_dict.values()))+ self._set_internal_state(new_data)self.metadata["updated_at"] = datetime.now()return item@@ -728,10 +603,13 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):>>> # Original skills unchanged"""new_set = self._create_new_instance()+ # Populate the dictionary directly, then sync via hooknew_set._items_dict = {key: item.model_copy(deep=True) for key, item in self._items_dict.items()}- new_set._data.items = list(new_set._items_dict.values())+ # Use hook to sync _data+ new_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))+ new_set._set_internal_state(new_data)new_set.metadata = self.metadata.copy()new_set.metadata["created_at"] = datetime.now()@@ -774,7 +652,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):else:new_set._items_dict[key] = item.model_copy(deep=True)- new_set._data.items = list(new_set._items_dict.values())+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))+ new_set._set_internal_state(merged_data)new_set.metadata["updated_at"] = datetime.now()return new_set@@ -814,7 +693,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):if merged:new_set._items_dict[key] = merged- new_set._data.items = list(new_set._items_dict.values())+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))+ new_set._set_internal_state(merged_data)new_set.metadata["updated_at"] = datetime.now()return new_set@@ -849,7 +729,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):if key not in other._items_dict:new_set._items_dict[key] = item.model_copy(deep=True)- new_set._data.items = list(new_set._items_dict.values())+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))+ new_set._set_internal_state(merged_data)new_set.metadata["updated_at"] = datetime.now()return new_set@@ -889,7 +770,8 @@ class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):if key not in self._items_dict:new_set._items_dict[key] = item.model_copy(deep=True)- new_set._data.items = list(new_set._items_dict.values())+ merged_data = new_set.item_set_schema(items=list(new_set._items_dict.values()))+ new_set._set_internal_state(merged_data)new_set.metadata["updated_at"] = datetime.now()return new_setdiff --git a/tests/__init__.py b/tests/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/tests/test_knowledge.py b/tests/test_knowledge.pydeleted file mode 100644index 612069a..0000000--- a/tests/test_knowledge.py+++ /dev/null@@ -1,343 +0,0 @@-"""-测试 BaseAutoType 类的功能。-"""-import pytest-from typing import List, Optional-from pydantic import BaseModel, Field-from unittest.mock import MagicMock, AsyncMock-import json--from hyperextract.core import BaseAutoType---# 定义测试用的 Schema-class Article(BaseModel):- """文章 Schema"""- title: str = Field(..., description="文章标题")- authors: List[str] = Field(default_factory=list, description="作者列表")- keywords: List[str] = Field(default_factory=list, description="关键词")- summary: str = Field(default="", description="摘要")---class Person(BaseModel):- """人物 Schema"""- name: str = Field(..., description="人物名字")- age: int = Field(..., description="年龄")- occupation: str = Field(default="", description="职业")---def mock_llm():- """创建 mock LLM 客户端"""- llm = AsyncMock()- return llm---def mock_embedder():- """创建 mock Embeddings 客户端"""- embedder = MagicMock()- return embedder---def a_article(mock_llm, mock_embedder):- """创建一个 Article 类型的 BaseAutoType 实例"""- return BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- storage="memory",- )---def a_person(mock_llm, mock_embedder):- """创建一个 Person 类型的 BaseAutoType 实例"""- return BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Person,- storage="memory",- )---class TestBaseAutoTypeInitialization:- """测试 BaseAutoType 类的初始化"""-- def test_auto_type_init_with_article_schema(self, a_article):- """测试使用 Article schema 初始化"""- assert a_article.schema_class == Article- assert a_article.storage == "memory"- assert a_article.data is not None-- def test_auto_type_init_with_initial_data(self, mock_llm, mock_embedder):- """测试带初始数据的初始化"""- initial_article = Article(- title="Test Article",- authors=["Author 1"],- keywords=["test"],- summary="A test article"- )- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- initial_data=initial_article,- )- assert knowledge.data == initial_article- assert knowledge.data.title == "Test Article"-- def test_auto_type_with_different_schemas(self, mock_llm, mock_embedder):- """测试使用不同 Schema 的 BaseAutoType"""- article_k = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- )- person_k = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Person,- )- assert article_k.schema_class == Article- assert person_k.schema_class == Person---class TestBaseAutoTypeExtract:- """测试 BaseAutoType 的 extract 方法"""-- @pytest.mark.asyncio- async def test_extract_article(self, a_article, mock_llm):- """测试提取文章信息"""- # 模拟 LLM 返回的响应(with_structured_output 直接返回对象)- expected_article = Article(- title="AI in 2024",- authors=["John Doe", "Jane Smith"],- keywords=["AI", "Machine Learning"],- summary="An overview of AI trends in 2024"- )-- # 模拟 with_structured_output 返回的 structured_llm- mock_structured_llm = AsyncMock()- mock_structured_llm.ainvoke.return_value = expected_article- mock_llm.with_structured_output.return_value = mock_structured_llm-- text = "Artificial Intelligence continues to advance in 2024..."- result = await a_article.extract(text)-- assert result.title == "AI in 2024"- assert len(result.authors) == 2- assert "AI" in result.keywords- assert a_article.data == result-- @pytest.mark.asyncio- async def test_extract_with_markdown_response(self, a_article, mock_llm):- """测试 with_structured_output 自动处理响应格式"""- # 使用 with_structured_output 无需处理 markdown 格式- expected_article = Article(- title="Test Article",- authors=["Author"],- keywords=["test"],- summary="Summary"- )-- mock_structured_llm = AsyncMock()- mock_structured_llm.ainvoke.return_value = expected_article- mock_llm.with_structured_output.return_value = mock_structured_llm-- result = await a_article.extract("test text")- assert result.title == "Test Article"- assert result.authors[0] == "Author"-- @pytest.mark.asyncio- async def test_aextract_calls_extract(self, a_article, mock_llm):- """测试 aextract 方法调用 extract"""- expected_article = Article(- title="Test",- authors=[],- keywords=[],- summary=""- )-- mock_structured_llm = AsyncMock()- mock_structured_llm.ainvoke.return_value = expected_article- mock_llm.with_structured_output.return_value = mock_structured_llm-- result = await a_article.aextract("test")- assert result.title == "Test"---class TestBaseAutoTypeDump:- """测试 BaseAutoType 的 dump 方法"""-- def test_dump_to_json(self, mock_llm, mock_embedder):- """测试导出为 JSON 格式"""- article = Article(- title="Test",- authors=["Alice"],- keywords=["test"],- summary="A test"- )- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- initial_data=article,- )-- json_str = knowledge.dump(format="json")- assert isinstance(json_str, str)- parsed = json.loads(json_str)- assert parsed["title"] == "Test"- assert parsed["authors"] == ["Alice"]-- def test_dump_to_dict(self, mock_llm, mock_embedder):- """测试导出为字典格式"""- article = Article(- title="Test",- authors=["Alice"],- keywords=["test"],- summary="A test"- )- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- initial_data=article,- )-- data_dict = knowledge.dump(format="dict")- assert isinstance(data_dict, dict)- assert data_dict["title"] == "Test"---class TestBaseAutoTypeLoad:- """测试 BaseAutoType 的 load 方法"""-- def test_load_from_dict(self, mock_llm, mock_embedder):- """测试从字典加载"""- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- )-- data = {- "title": "Loaded Article",- "authors": ["Author1"],- "keywords": ["loaded"],- "summary": "Loaded from dict"- }- knowledge.load(data)-- assert knowledge.data.title == "Loaded Article"- assert knowledge.data.authors == ["Author1"]-- def test_load_from_json_string(self, mock_llm, mock_embedder):- """测试从 JSON 字符串加载"""- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- )-- json_str = json.dumps({- "title": "JSON Article",- "authors": ["JSON Author"],- "keywords": [],- "summary": ""- })- knowledge.load(json_str)-- assert knowledge.data.title == "JSON Article"-- def test_load_invalid_data_raises_error(self, mock_llm, mock_embedder):- """测试加载无效数据时抛出错误"""- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- )-- # 缺少必需字段- invalid_data = {"authors": ["Author"]}- with pytest.raises(Exception):- knowledge.load(invalid_data)---class TestBaseAutoTypeSearch:- """测试 BaseAutoType 的 search 方法"""-- @pytest.mark.asyncio- async def test_search_returns_dict(self, mock_llm, mock_embedder):- """测试 search 返回字典格式的数据"""- article = Article(- title="Search Test",- authors=["Searcher"],- keywords=["search"],- summary="Testing search"- )- knowledge = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- initial_data=article,- )-- result = await knowledge.search("search")- assert isinstance(result, dict)- assert result["title"] == "Search Test"---class TestBaseAutoTypeEvolve:- """测试 BaseAutoType 的 evolve 和 aevolve 方法"""-- @pytest.mark.asyncio- async def test_aevolve_default_implementation(self, a_article):- """测试 aevolve 的默认实现(不做任何操作)"""- await a_article.aevolve()- # 如果没有抛出异常,则测试通过---class TestBaseAutoTypeMultipleSchemas:- """测试使用多个不同 Schema 的场景"""-- def test_different_schemas_independent(self, mock_llm, mock_embedder):- """测试不同 Schema 的 BaseAutoType 相互独立"""- article_k = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Article,- )- person_k = BaseAutoType(- llm_client=mock_llm,- embedder=mock_embedder,- schema_class=Person,- )-- # 分别为两个 knowledge 加载数据- article_data = {- "title": "Article",- "authors": [],- "keywords": [],- "summary": ""- }- person_data = {- "name": "John",- "age": 30,- "occupation": "Engineer"- }-- article_k.load(article_data)- person_k.load(person_data)-- # 验证数据相互独立- assert article_k.data.title == "Article"- assert person_k.data.name == "John"- assert isinstance(article_k.data, Article)- assert isinstance(person_k.data, Person)---if __name__ == "__main__":- pytest.main([__file__, "-v"])