Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: 添加 AutoSet 机制以支持基于键值的信息累加,增强学术名词和菜肴条目的合并能力
yifanfeng970d1225eb6a36
diff --git a/hyperextract/templates/INSTRUCTION.md b/hyperextract/templates/INSTRUCTION.mdindex 615a6e9..db6bb88 100644--- a/hyperextract/templates/INSTRUCTION.md+++ b/hyperextract/templates/INSTRUCTION.md@@ -39,6 +39,50 @@ Before designing a Schema, ask yourself:---+## 1.1 Special Guidance: AutoSet (Key-Based Accumulation)++The `AutoSet` base class is designed for building **deduplicated, enriched knowledge collections** through **exact key matching**.+It is fundamentally different from `AutoList` (which has no deduplication) and from semantic clustering/embedding-based merging.++### How AutoSet Works++**Core Mechanism**:+- Defines a `key_extractor` function that maps each item to a unique identifier (e.g., `key_extractor=lambda x: x.name.strip().lower()`).+- When `feed_text()` is called, the framework extracts items and groups them by their keys.+- **All items with the same key are automatically merged into one comprehensive entry**.+- The **`embedder` parameter is NOT used for merging logic**; it is only used for semantic search and knowledge graph visualization.++**Example Scenario**:+```python+from hyperextract.templates.food.culinary_dish import CulinaryDishSet++dishes = CulinaryDishSet(llm_client=..., embedder=...)++# Text 1: Extract basic dish info+dishes.feed_text("Kung Pao Chicken is a Sichuan dish with peanuts and chilies.")+# Result: {name: "kung pao chicken", cuisine: "Sichuan", primary_ingredients: "peanuts, chilies"}++# Text 2: Add complementary details about the same dish+dishes.feed_text("Kung Pao Chicken is stir-fried rapidly, creating a sweet and spicy flavor.")+# Auto-merged: {name: "kung pao chicken", cuisine: "Sichuan", primary_ingredients: "peanuts, chilies", flavor_profile: "sweet and spicy"}++dishes.show() # Shows the merged, enriched entry+```++**Key Design Principles**:+1. **Exact Key Matching**: The `key_extractor` defines what makes items "the same". Only items with identical keys are merged.+2. **Information Accumulation**: Merging combines and enriches all attributes from multiple mentions of the same item.+3. **No Semantic Similarity**: Unlike clustering, AutoSet does NOT group similar-but-different items (e.g., "Kung Pao Chicken" and "Gong Bao Chicken" with different keys remain separate).+4. **LLM Responsibility**: The LLM's extraction prompt should normalize/standardize names to ensure correct grouping by key.++**Embedder Role**:+- **NOT used for deduplication**. The `embedder` is used for:+ - Semantic retrieval during chat/search queries+ - Visualizing relationships in the knowledge graph+ - Future question-answering capabilities++---+## 2. Implementation Rules### A. Prompt Definitiondiff --git a/hyperextract/templates/INSTRUCTION_ZH.md b/hyperextract/templates/INSTRUCTION_ZH.mdindex c5b7dd4..7248538 100644--- a/hyperextract/templates/INSTRUCTION_ZH.md+++ b/hyperextract/templates/INSTRUCTION_ZH.md@@ -39,6 +39,50 @@---+## 1.1 特别说明:AutoSet(基于键值的信息累加)++`AutoSet` 基类专为构建**去重、富集的知识库**而设计,采用**精确键值匹配**机制。+它**完全不同于** `AutoList`(无去重)和基于语义聚类/嵌入相似度的合并方式。++### AutoSet 工作原理++**核心机制**:+- 定义一个 `key_extractor` 函数将每个条目映射到唯一标识符(如 `key_extractor=lambda x: x.name.strip().lower()`)。+- 调用 `feed_text()` 时,框架提取条目**并按键值分组**。+- **所有键值相同的条目自动合并为一条综合条目**。+- `embedder` 参数**不用于合并逻辑**,仅用于语义检索和知识图谱可视化。++**场景示例**:+```python+from hyperextract.templates.food.culinary_dish import CulinaryDishSet++dishes = CulinaryDishSet(llm_client=..., embedder=...)++# 文本 1:提取菜肴基础信息+dishes.feed_text("宫保鸡丁是四川菜,用花生和干辣椒制作。")+# 结果:{name: "宫保鸡丁", cuisine: "四川", primary_ingredients: "花生,干辣椒"}++# 文本 2:添加关于同一菜肴的补充信息+dishes.feed_text("宫保鸡丁是大火快速炒制,整体风味是甜辣鲜香。")+# 自动合并:{name: "宫保鸡丁", cuisine: "四川", primary_ingredients: "花生,干辣椒", flavor_profile: "甜辣鲜香"}++dishes.show() # 显示合并后的富集条目+```++**关键设计原则**:+1. **精确键值匹配**:`key_extractor` 定义什么是"同一个条目"。**只有**键值完全相同的条目才会合并。+2. **信息累加**:合并时**融合并丰富**来自多个信息源的所有属性。+3. **非语义相似性**:不同于聚类,AutoSet **不会**将语义相近但键值不同的条目分组(如"宫保鸡丁"和"宫爆鸡丁"如果被标准化为不同名称,会保持分离)。+4. **LLM 责任**:提取 Prompt 应确保名称标准化,以便 LLM 正确归组到同一键值下。++**Embedder 的角色**:+- **不用于去重逻辑**,`embedder` 用于:+ - 聊天/搜索查询时的语义检索+ - 知识图谱中关系的可视化展示+ - 未来的问答能力++---+## 2. 实现规范 (Implementation Rules)### A. Prompt 定义diff --git a/hyperextract/templates/en/education/academic_lexicon.py b/hyperextract/templates/en/education/academic_lexicon.pyindex 684fed4..0b2e05d 100644--- a/hyperextract/templates/en/education/academic_lexicon.py+++ b/hyperextract/templates/en/education/academic_lexicon.py@@ -51,16 +51,17 @@ _PROMPT = (class AcademicLexiconSet(AutoSet[AcademicTerm]):"""- Educational template for building a deduplicated, synthesized dictionary of academic terms.- Leverages AutoSet to merge conflicting or complementary definitions from multiple text sources.+ Educational template for building a synthesized dictionary of academic terms through key-based information accumulation.+ Automatically merges information from multiple text sources about the same term (identified by term name)+ into a single comprehensive entry, enriching definitions with context from all sources.Example Usage:>>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings>>> from hyperextract.templates.en.education.academic_lexicon import AcademicLexiconSet>>> lexicon = AcademicLexiconSet(llm_client, embedder)- >>> lexicon.feed_text("Source 1: Photosynthesis is a process used by plants...")- >>> lexicon.feed_text("Source 2: Photosynthesis converts light energy into chemical energy...")- >>> lexicon.extract() # Will merge definitions of "Photosynthesis"+ >>> lexicon.feed_text("Source 1: Photosynthesis is a process used by plants to convert light energy...") # Extract: term='photosynthesis', definition=...+ >>> lexicon.feed_text("Source 2: Photosynthesis occurs in chloroplasts and involves electron transport...") # Auto-merge into existing 'photosynthesis' entry+ >>> lexicon.show() # Display merged academic terms"""def __init__(@@ -80,7 +81,7 @@ class AcademicLexiconSet(AutoSet[AcademicTerm]):Args:llm_client (BaseChatModel): The LLM used for extraction.- embedder (Embeddings): The embedding model for entity deduplication and merging.+ embedder (Embeddings): The embedding model for semantic retrieval and knowledge graph visualization. Note: Key-based accumulation uses the term name, not embeddings for merging.extraction_mode (str): Mode of extraction.chunk_size (int): Size of text chunks for processing large documents.chunk_overlap (int): Overlap between chunks.diff --git a/hyperextract/templates/en/environment/endangered_species.py b/hyperextract/templates/en/environment/endangered_species.pyindex aefbbeb..aec13fb 100644--- a/hyperextract/templates/en/environment/endangered_species.py+++ b/hyperextract/templates/en/environment/endangered_species.py@@ -81,12 +81,12 @@ class EndangeredSpeciesList(AutoSet[SpeciesStatus]):Args:llm_client (BaseChatModel): Language model client for extracting species and threat information.- embedder (Embeddings): Embedding model for deduplicating species entries and indexing threat descriptions.+ embedder (Embeddings): Embedding model for semantic search and knowledge graph visualization. Note: Species accumulation is based on exact species name matching, not semantic similarity.chunk_size (int, optional): Maximum characters per text chunk. Defaults to 2048.chunk_overlap (int, optional): Overlapping characters between chunks for context preservation. Defaults to 256.max_workers (int, optional): Maximum concurrent extraction workers. Defaults to 10.- verbose (bool, optional): If True, prints detailed extraction and deduplication progress logs. Defaults to False.- **kwargs: Additional arguments passed to the AutoSet base class for deduplication control.+ verbose (bool, optional): If True, prints detailed extraction and accumulation progress logs. Defaults to False.+ **kwargs: Additional arguments passed to the AutoSet base class."""super().__init__(item_schema=SpeciesStatus,diff --git a/hyperextract/templates/en/food/culinary_dish.py b/hyperextract/templates/en/food/culinary_dish.pyindex c7b4cea..7509e10 100644--- a/hyperextract/templates/en/food/culinary_dish.py+++ b/hyperextract/templates/en/food/culinary_dish.py@@ -74,19 +74,20 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):Applicable to: Restaurant Menus, Cookbooks, Food Encyclopedias, Recipe Websites,Food Blogs, Culinary Magazine Articles, Restaurant Review Databases, Dining Guides.- Template for building a deduplicated collection of canonical culinary dishes.- Standardizes dish names across cultural and linguistic variations, capturing their- defining characteristics. Useful for culinary reference systems, recipe discovery,- and food journalism.+ Template for building a standardized culinary foundation through **key-based information accumulation**.+ Automatically merges information about the same dish (identified by standardized dish name) from+ multiple sources into a single enriched entry, capturing ingredients, preparation, origin, and characteristics.Example Usage:>>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings>>> llm = ChatOpenAI(model="gpt-4o")>>> embedder = OpenAIEmbeddings()>>> dishes = CulinaryDishSet(llm_client=llm, embedder=embedder)- >>> menu_text = "Our menu features Gong Bao Chicken, Kung Pao Chicken, and authentic Sichuan peanut chicken..."- >>> dishes.feed_text(menu_text)- >>> dishes.show() # Automatically accumulates information for identical dish keys (e.g. 'Kung Pao Chicken')+ >>> # Source 1: Extract basic info (name, cuisine, region)+ >>> dishes.feed_text("Kung Pao Chicken is a famous dish from Sichuan province with peanuts and dried chilies.")+ >>> # Source 2: Extract complementary details (ingredients, flavor, technique)+ >>> dishes.feed_text("Kung Pao Chicken features rapid stir-frying of chicken with roasted peanuts, creating a sweet and spicy taste.")+ >>> dishes.show() # Shows merged 'Kung Pao Chicken' entry with accumulated details from both sources"""def __init__(@@ -105,7 +106,7 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):Args:llm_client (BaseChatModel): The LLM for dish extraction and standardization.- embedder (Embeddings): Embedding model for retrieval and visualization.+ embedder (Embeddings): Embedding model for semantic retrieval and knowledge graph visualization.chunk_size (int): Size of text chunks for processing.chunk_overlap (int): Overlap between chunks.max_workers (int): Parallel processing workers.diff --git a/hyperextract/templates/zh/education/academic_lexicon.py b/hyperextract/templates/zh/education/academic_lexicon.pyindex 45f563f..eb630d6 100644--- a/hyperextract/templates/zh/education/academic_lexicon.py+++ b/hyperextract/templates/zh/education/academic_lexicon.py@@ -47,14 +47,14 @@ _PROMPT = (class AcademicLexiconSet(AutoSet[AcademicTerm]):"""- 用于构建去重且合成的学术名词百科集的教育模版。- 利用 AutoSet 的合并策略,将来自不同教材或文献的定义融合成唯一的、信息最全的条目。+ 用于构建学术名词百科集的教育模板,通过**精确的术语键值进行信息累加**。+ 自动将来自多个文本源的同一术语信息合并为一条完整条目,从各源中丰富定义内容。用法示例:>>> lexicon = AcademicLexiconSet(llm_client, embedder)- >>> lexicon.feed_text("来源A:光合作用是植物利用光能...")- >>> lexicon.feed_text("来源B:光合作用涉及叶绿素吸收光子...")- >>> lexicon.extract() # 自动通过“项合并”功能将关于“光合作用”的不同侧重定义合并+ >>> lexicon.feed_text("来源A:光合作用是植物利用太阳能进行的生化反应...") # 提取:term='光合作用', definition=...+ >>> lexicon.feed_text("来源B:光合作用在叶绿体中进行,包含光反应和暗反应...") # 自动合并到已有的'光合作用'条目中+ >>> lexicon.show() # 显示合并后的学术术语集"""def __init__(@@ -74,13 +74,13 @@ class AcademicLexiconSet(AutoSet[AcademicTerm]):参数描述:llm_client (BaseChatModel): 用于提取的语言模型客户端。- embedder (Embeddings): 用于实体合并和去重的嵌入模型(AutoSet 的核心)。+ embedder (Embeddings): 用于语义检索和知识图谱可视化的嵌入模型。注意:信息累加基于术语名称(key),而非嵌入向量。extraction_mode (str): 提取模式策略。chunk_size (int): 文本分片大小。chunk_overlap (int): 分片重叠大小。max_workers (int): 并行处理进程数。verbose (bool): 是否开启详细日志。- **kwargs: 透传给 AutoSet 基类的其他参数(如自定义 merge_strategy)。+ **kwargs: 透传给 AutoSet 基类的其他参数。"""super().__init__(item_schema=AcademicTerm,diff --git a/hyperextract/templates/zh/environment/endangered_species.py b/hyperextract/templates/zh/environment/endangered_species.pyindex fd6c032..04e5d7d 100644--- a/hyperextract/templates/zh/environment/endangered_species.py+++ b/hyperextract/templates/zh/environment/endangered_species.py@@ -80,12 +80,12 @@ class EndangeredSpeciesList(AutoSet[SpeciesStatus]):Args:llm_client (BaseChatModel):用于提取物种和威胁信息的语言模型客户端。- embedder (Embeddings):用于物种去重和威胁描述索引的嵌入模型。+ embedder (Embeddings):用于语义检索和知识图谱可视化的嵌入模型。注意:物种信息累加基于精确的物种名称匹配,而非语义相似度。chunk_size (int, optional):每个文本块的最大字符数。默认为 2048。chunk_overlap (int, optional):块之间的重叠字符数,用于保持上下文连贯性。默认为 256。max_workers (int, optional):并发提取的最大工作进程数。默认为 10。- verbose (bool, optional):如果为 True,则打印详细的提取和去重进度日志。默认为 False。- **kwargs:传递给 AutoSet 基类的参数,用于控制去重行为。+ verbose (bool, optional):如果为 True,则打印详细的提取和累加进度日志。默认为 False。+ **kwargs:传递给 AutoSet 基类的参数。"""super().__init__(item_schema=SpeciesStatus,diff --git a/hyperextract/templates/zh/food/culinary_dish.py b/hyperextract/templates/zh/food/culinary_dish.pyindex 9f31448..8169ede 100644--- a/hyperextract/templates/zh/food/culinary_dish.py+++ b/hyperextract/templates/zh/food/culinary_dish.py@@ -70,16 +70,19 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):"""适用文档: 餐厅菜单、美食博客、烹饪书籍、食材供应商列表、菜肴数据库、美食指南、社交媒体。- 模板用于从多个来源提取和去重菜肴条目。在菜肴名称的变体(例如"宫保鸡丁"与"宫爆鸡丁")中自动融合相同的菜肴。支持菜单标准化、美食知识积累和跨文化菜肴对应。+ 模板用于构建标准化的菜肴知识库,通过**精确的菜肴名称为键进行信息累加**。+ 自动将来自多个文本源的同一菜肴信息合并为一条完整条目,从各源中积累食材、烹饪方法、地域特征和风味描述。使用示例:>>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings>>> llm = ChatOpenAI(model="gpt-4o")>>> embedder = OpenAIEmbeddings()>>> dishes = CulinaryDishSet(llm_client=llm, embedder=embedder)- >>> menu = "1. Kung Pao Chicken - 宫保鸡丁。2. 宫爆鸡丁(四川焦糖鸡丁)"- >>> dishes.feed_text(menu)- >>> dishes.show()+ >>> # 来源 1:提取基本信息(名称、菜系、地域)+ >>> dishes.feed_text("宫保鸡丁是来源于四川的著名菜肴,主要食材有鸡肉、花生、干辣椒。")+ >>> # 来源 2:提取补充信息(风味、烹饪手法)+ >>> dishes.feed_text("宫保鸡丁采用大火快速炒制,花生要烘烤,整体风味是甜辣鲜香。")+ >>> dishes.show() # 显示合并后包含两个来源信息的"宫保鸡丁"条目"""def __init__(@@ -98,7 +101,7 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):Args:llm_client (BaseChatModel): 用于提取的 LLM。- embedder (Embeddings): 用于检索和可视化的嵌入模型。+ embedder (Embeddings): 用于语义检索和知识图谱可视化的嵌入模型。chunk_size (int): 每个分块的最大字符数。chunk_overlap (int): 分块之间的重叠。max_workers (int): 并行处理工作线程数。