Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

feat: 添加 AutoSet 机制以支持基于键值的信息累加,增强学术名词和菜肴条目的合并能力

yifanfeng970d1225eb6a36
diff --git a/hyperextract/templates/INSTRUCTION.md b/hyperextract/templates/INSTRUCTION.md
index 615a6e9..db6bb88 100644
--- a/hyperextract/templates/INSTRUCTION.md
+++ b/hyperextract/templates/INSTRUCTION.md
@@ -39,6 +39,50 @@ Before designing a Schema, ask yourself:
---
+## 1.1 Special Guidance: AutoSet (Key-Based Accumulation)
+
+The `AutoSet` base class is designed for building **deduplicated, enriched knowledge collections** through **exact key matching**.
+It is fundamentally different from `AutoList` (which has no deduplication) and from semantic clustering/embedding-based merging.
+
+### How AutoSet Works
+
+**Core Mechanism**:
+- Defines a `key_extractor` function that maps each item to a unique identifier (e.g., `key_extractor=lambda x: x.name.strip().lower()`).
+- When `feed_text()` is called, the framework extracts items and groups them by their keys.
+- **All items with the same key are automatically merged into one comprehensive entry**.
+- The **`embedder` parameter is NOT used for merging logic**; it is only used for semantic search and knowledge graph visualization.
+
+**Example Scenario**:
+```python
+from hyperextract.templates.food.culinary_dish import CulinaryDishSet
+
+dishes = CulinaryDishSet(llm_client=..., embedder=...)
+
+# Text 1: Extract basic dish info
+dishes.feed_text("Kung Pao Chicken is a Sichuan dish with peanuts and chilies.")
+# Result: {name: "kung pao chicken", cuisine: "Sichuan", primary_ingredients: "peanuts, chilies"}
+
+# Text 2: Add complementary details about the same dish
+dishes.feed_text("Kung Pao Chicken is stir-fried rapidly, creating a sweet and spicy flavor.")
+# Auto-merged: {name: "kung pao chicken", cuisine: "Sichuan", primary_ingredients: "peanuts, chilies", flavor_profile: "sweet and spicy"}
+
+dishes.show() # Shows the merged, enriched entry
+```
+
+**Key Design Principles**:
+1. **Exact Key Matching**: The `key_extractor` defines what makes items "the same". Only items with identical keys are merged.
+2. **Information Accumulation**: Merging combines and enriches all attributes from multiple mentions of the same item.
+3. **No Semantic Similarity**: Unlike clustering, AutoSet does NOT group similar-but-different items (e.g., "Kung Pao Chicken" and "Gong Bao Chicken" with different keys remain separate).
+4. **LLM Responsibility**: The LLM's extraction prompt should normalize/standardize names to ensure correct grouping by key.
+
+**Embedder Role**:
+- **NOT used for deduplication**. The `embedder` is used for:
+ - Semantic retrieval during chat/search queries
+ - Visualizing relationships in the knowledge graph
+ - Future question-answering capabilities
+
+---
+
## 2. Implementation Rules
### A. Prompt Definition
diff --git a/hyperextract/templates/INSTRUCTION_ZH.md b/hyperextract/templates/INSTRUCTION_ZH.md
index c5b7dd4..7248538 100644
--- a/hyperextract/templates/INSTRUCTION_ZH.md
+++ b/hyperextract/templates/INSTRUCTION_ZH.md
@@ -39,6 +39,50 @@
---
+## 1.1 特别说明:AutoSet(基于键值的信息累加)
+
+`AutoSet` 基类专为构建**去重、富集的知识库**而设计,采用**精确键值匹配**机制。
+它**完全不同于** `AutoList`(无去重)和基于语义聚类/嵌入相似度的合并方式。
+
+### AutoSet 工作原理
+
+**核心机制**:
+- 定义一个 `key_extractor` 函数将每个条目映射到唯一标识符(如 `key_extractor=lambda x: x.name.strip().lower()`)。
+- 调用 `feed_text()` 时,框架提取条目**并按键值分组**。
+- **所有键值相同的条目自动合并为一条综合条目**。
+- `embedder` 参数**不用于合并逻辑**,仅用于语义检索和知识图谱可视化。
+
+**场景示例**:
+```python
+from hyperextract.templates.food.culinary_dish import CulinaryDishSet
+
+dishes = CulinaryDishSet(llm_client=..., embedder=...)
+
+# 文本 1:提取菜肴基础信息
+dishes.feed_text("宫保鸡丁是四川菜,用花生和干辣椒制作。")
+# 结果:{name: "宫保鸡丁", cuisine: "四川", primary_ingredients: "花生,干辣椒"}
+
+# 文本 2:添加关于同一菜肴的补充信息
+dishes.feed_text("宫保鸡丁是大火快速炒制,整体风味是甜辣鲜香。")
+# 自动合并:{name: "宫保鸡丁", cuisine: "四川", primary_ingredients: "花生,干辣椒", flavor_profile: "甜辣鲜香"}
+
+dishes.show() # 显示合并后的富集条目
+```
+
+**关键设计原则**:
+1. **精确键值匹配**:`key_extractor` 定义什么是"同一个条目"。**只有**键值完全相同的条目才会合并。
+2. **信息累加**:合并时**融合并丰富**来自多个信息源的所有属性。
+3. **非语义相似性**:不同于聚类,AutoSet **不会**将语义相近但键值不同的条目分组(如"宫保鸡丁"和"宫爆鸡丁"如果被标准化为不同名称,会保持分离)。
+4. **LLM 责任**:提取 Prompt 应确保名称标准化,以便 LLM 正确归组到同一键值下。
+
+**Embedder 的角色**:
+- **不用于去重逻辑**,`embedder` 用于:
+ - 聊天/搜索查询时的语义检索
+ - 知识图谱中关系的可视化展示
+ - 未来的问答能力
+
+---
+
## 2. 实现规范 (Implementation Rules)
### A. Prompt 定义
diff --git a/hyperextract/templates/en/education/academic_lexicon.py b/hyperextract/templates/en/education/academic_lexicon.py
index 684fed4..0b2e05d 100644
--- a/hyperextract/templates/en/education/academic_lexicon.py
+++ b/hyperextract/templates/en/education/academic_lexicon.py
@@ -51,16 +51,17 @@ _PROMPT = (
class AcademicLexiconSet(AutoSet[AcademicTerm]):
"""
- Educational template for building a deduplicated, synthesized dictionary of academic terms.
- Leverages AutoSet to merge conflicting or complementary definitions from multiple text sources.
+ Educational template for building a synthesized dictionary of academic terms through key-based information accumulation.
+ Automatically merges information from multiple text sources about the same term (identified by term name)
+ into a single comprehensive entry, enriching definitions with context from all sources.
Example Usage:
>>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings
>>> from hyperextract.templates.en.education.academic_lexicon import AcademicLexiconSet
>>> lexicon = AcademicLexiconSet(llm_client, embedder)
- >>> lexicon.feed_text("Source 1: Photosynthesis is a process used by plants...")
- >>> lexicon.feed_text("Source 2: Photosynthesis converts light energy into chemical energy...")
- >>> lexicon.extract() # Will merge definitions of "Photosynthesis"
+ >>> lexicon.feed_text("Source 1: Photosynthesis is a process used by plants to convert light energy...") # Extract: term='photosynthesis', definition=...
+ >>> lexicon.feed_text("Source 2: Photosynthesis occurs in chloroplasts and involves electron transport...") # Auto-merge into existing 'photosynthesis' entry
+ >>> lexicon.show() # Display merged academic terms
"""
def __init__(
@@ -80,7 +81,7 @@ class AcademicLexiconSet(AutoSet[AcademicTerm]):
Args:
llm_client (BaseChatModel): The LLM used for extraction.
- embedder (Embeddings): The embedding model for entity deduplication and merging.
+ embedder (Embeddings): The embedding model for semantic retrieval and knowledge graph visualization. Note: Key-based accumulation uses the term name, not embeddings for merging.
extraction_mode (str): Mode of extraction.
chunk_size (int): Size of text chunks for processing large documents.
chunk_overlap (int): Overlap between chunks.
diff --git a/hyperextract/templates/en/environment/endangered_species.py b/hyperextract/templates/en/environment/endangered_species.py
index aefbbeb..aec13fb 100644
--- a/hyperextract/templates/en/environment/endangered_species.py
+++ b/hyperextract/templates/en/environment/endangered_species.py
@@ -81,12 +81,12 @@ class EndangeredSpeciesList(AutoSet[SpeciesStatus]):
Args:
llm_client (BaseChatModel): Language model client for extracting species and threat information.
- embedder (Embeddings): Embedding model for deduplicating species entries and indexing threat descriptions.
+ embedder (Embeddings): Embedding model for semantic search and knowledge graph visualization. Note: Species accumulation is based on exact species name matching, not semantic similarity.
chunk_size (int, optional): Maximum characters per text chunk. Defaults to 2048.
chunk_overlap (int, optional): Overlapping characters between chunks for context preservation. Defaults to 256.
max_workers (int, optional): Maximum concurrent extraction workers. Defaults to 10.
- verbose (bool, optional): If True, prints detailed extraction and deduplication progress logs. Defaults to False.
- **kwargs: Additional arguments passed to the AutoSet base class for deduplication control.
+ verbose (bool, optional): If True, prints detailed extraction and accumulation progress logs. Defaults to False.
+ **kwargs: Additional arguments passed to the AutoSet base class.
"""
super().__init__(
item_schema=SpeciesStatus,
diff --git a/hyperextract/templates/en/food/culinary_dish.py b/hyperextract/templates/en/food/culinary_dish.py
index c7b4cea..7509e10 100644
--- a/hyperextract/templates/en/food/culinary_dish.py
+++ b/hyperextract/templates/en/food/culinary_dish.py
@@ -74,19 +74,20 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):
Applicable to: Restaurant Menus, Cookbooks, Food Encyclopedias, Recipe Websites,
Food Blogs, Culinary Magazine Articles, Restaurant Review Databases, Dining Guides.
- Template for building a deduplicated collection of canonical culinary dishes.
- Standardizes dish names across cultural and linguistic variations, capturing their
- defining characteristics. Useful for culinary reference systems, recipe discovery,
- and food journalism.
+ Template for building a standardized culinary foundation through **key-based information accumulation**.
+ Automatically merges information about the same dish (identified by standardized dish name) from
+ multiple sources into a single enriched entry, capturing ingredients, preparation, origin, and characteristics.
Example Usage:
>>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings
>>> llm = ChatOpenAI(model="gpt-4o")
>>> embedder = OpenAIEmbeddings()
>>> dishes = CulinaryDishSet(llm_client=llm, embedder=embedder)
- >>> menu_text = "Our menu features Gong Bao Chicken, Kung Pao Chicken, and authentic Sichuan peanut chicken..."
- >>> dishes.feed_text(menu_text)
- >>> dishes.show() # Automatically accumulates information for identical dish keys (e.g. 'Kung Pao Chicken')
+ >>> # Source 1: Extract basic info (name, cuisine, region)
+ >>> dishes.feed_text("Kung Pao Chicken is a famous dish from Sichuan province with peanuts and dried chilies.")
+ >>> # Source 2: Extract complementary details (ingredients, flavor, technique)
+ >>> dishes.feed_text("Kung Pao Chicken features rapid stir-frying of chicken with roasted peanuts, creating a sweet and spicy taste.")
+ >>> dishes.show() # Shows merged 'Kung Pao Chicken' entry with accumulated details from both sources
"""
def __init__(
@@ -105,7 +106,7 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):
Args:
llm_client (BaseChatModel): The LLM for dish extraction and standardization.
- embedder (Embeddings): Embedding model for retrieval and visualization.
+ embedder (Embeddings): Embedding model for semantic retrieval and knowledge graph visualization.
chunk_size (int): Size of text chunks for processing.
chunk_overlap (int): Overlap between chunks.
max_workers (int): Parallel processing workers.
diff --git a/hyperextract/templates/zh/education/academic_lexicon.py b/hyperextract/templates/zh/education/academic_lexicon.py
index 45f563f..eb630d6 100644
--- a/hyperextract/templates/zh/education/academic_lexicon.py
+++ b/hyperextract/templates/zh/education/academic_lexicon.py
@@ -47,14 +47,14 @@ _PROMPT = (
class AcademicLexiconSet(AutoSet[AcademicTerm]):
"""
- 用于构建去重且合成的学术名词百科集的教育模版。
- 利用 AutoSet 的合并策略,将来自不同教材或文献的定义融合成唯一的、信息最全的条目。
+ 用于构建学术名词百科集的教育模板,通过**精确的术语键值进行信息累加**。
+ 自动将来自多个文本源的同一术语信息合并为一条完整条目,从各源中丰富定义内容。
用法示例:
>>> lexicon = AcademicLexiconSet(llm_client, embedder)
- >>> lexicon.feed_text("来源A:光合作用是植物利用光能...")
- >>> lexicon.feed_text("来源B:光合作用涉及叶绿素吸收光子...")
- >>> lexicon.extract() # 自动通过“项合并”功能将关于“光合作用”的不同侧重定义合并
+ >>> lexicon.feed_text("来源A:光合作用是植物利用太阳能进行的生化反应...") # 提取:term='光合作用', definition=...
+ >>> lexicon.feed_text("来源B:光合作用在叶绿体中进行,包含光反应和暗反应...") # 自动合并到已有的'光合作用'条目中
+ >>> lexicon.show() # 显示合并后的学术术语集
"""
def __init__(
@@ -74,13 +74,13 @@ class AcademicLexiconSet(AutoSet[AcademicTerm]):
参数描述:
llm_client (BaseChatModel): 用于提取的语言模型客户端。
- embedder (Embeddings): 用于实体合并和去重的嵌入模型(AutoSet 的核心)。
+ embedder (Embeddings): 用于语义检索和知识图谱可视化的嵌入模型。注意:信息累加基于术语名称(key),而非嵌入向量。
extraction_mode (str): 提取模式策略。
chunk_size (int): 文本分片大小。
chunk_overlap (int): 分片重叠大小。
max_workers (int): 并行处理进程数。
verbose (bool): 是否开启详细日志。
- **kwargs: 透传给 AutoSet 基类的其他参数(如自定义 merge_strategy)。
+ **kwargs: 透传给 AutoSet 基类的其他参数。
"""
super().__init__(
item_schema=AcademicTerm,
diff --git a/hyperextract/templates/zh/environment/endangered_species.py b/hyperextract/templates/zh/environment/endangered_species.py
index fd6c032..04e5d7d 100644
--- a/hyperextract/templates/zh/environment/endangered_species.py
+++ b/hyperextract/templates/zh/environment/endangered_species.py
@@ -80,12 +80,12 @@ class EndangeredSpeciesList(AutoSet[SpeciesStatus]):
Args:
llm_client (BaseChatModel):用于提取物种和威胁信息的语言模型客户端。
- embedder (Embeddings):用于物种去重和威胁描述索引的嵌入模型。
+ embedder (Embeddings):用于语义检索和知识图谱可视化的嵌入模型。注意:物种信息累加基于精确的物种名称匹配,而非语义相似度。
chunk_size (int, optional):每个文本块的最大字符数。默认为 2048。
chunk_overlap (int, optional):块之间的重叠字符数,用于保持上下文连贯性。默认为 256。
max_workers (int, optional):并发提取的最大工作进程数。默认为 10。
- verbose (bool, optional):如果为 True,则打印详细的提取和去重进度日志。默认为 False。
- **kwargs:传递给 AutoSet 基类的参数,用于控制去重行为。
+ verbose (bool, optional):如果为 True,则打印详细的提取和累加进度日志。默认为 False。
+ **kwargs:传递给 AutoSet 基类的参数。
"""
super().__init__(
item_schema=SpeciesStatus,
diff --git a/hyperextract/templates/zh/food/culinary_dish.py b/hyperextract/templates/zh/food/culinary_dish.py
index 9f31448..8169ede 100644
--- a/hyperextract/templates/zh/food/culinary_dish.py
+++ b/hyperextract/templates/zh/food/culinary_dish.py
@@ -70,16 +70,19 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):
"""
适用文档: 餐厅菜单、美食博客、烹饪书籍、食材供应商列表、菜肴数据库、美食指南、社交媒体。
- 模板用于从多个来源提取和去重菜肴条目。在菜肴名称的变体(例如"宫保鸡丁"与"宫爆鸡丁")中自动融合相同的菜肴。支持菜单标准化、美食知识积累和跨文化菜肴对应。
+ 模板用于构建标准化的菜肴知识库,通过**精确的菜肴名称为键进行信息累加**。
+ 自动将来自多个文本源的同一菜肴信息合并为一条完整条目,从各源中积累食材、烹饪方法、地域特征和风味描述。
使用示例:
>>> from langchain_openai import ChatOpenAI, OpenAIEmbeddings
>>> llm = ChatOpenAI(model="gpt-4o")
>>> embedder = OpenAIEmbeddings()
>>> dishes = CulinaryDishSet(llm_client=llm, embedder=embedder)
- >>> menu = "1. Kung Pao Chicken - 宫保鸡丁。2. 宫爆鸡丁(四川焦糖鸡丁)"
- >>> dishes.feed_text(menu)
- >>> dishes.show()
+ >>> # 来源 1:提取基本信息(名称、菜系、地域)
+ >>> dishes.feed_text("宫保鸡丁是来源于四川的著名菜肴,主要食材有鸡肉、花生、干辣椒。")
+ >>> # 来源 2:提取补充信息(风味、烹饪手法)
+ >>> dishes.feed_text("宫保鸡丁采用大火快速炒制,花生要烘烤,整体风味是甜辣鲜香。")
+ >>> dishes.show() # 显示合并后包含两个来源信息的"宫保鸡丁"条目
"""
def __init__(
@@ -98,7 +101,7 @@ class CulinaryDishSet(AutoSet[CulinaryDish]):
Args:
llm_client (BaseChatModel): 用于提取的 LLM。
- embedder (Embeddings): 用于检索和可视化的嵌入模型。
+ embedder (Embeddings): 用于语义检索和知识图谱可视化的嵌入模型。
chunk_size (int): 每个分块的最大字符数。
chunk_overlap (int): 分块之间的重叠。
max_workers (int): 并行处理工作线程数。