Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: Introduce graph and hypergraph structures for knowledge extraction
yifanfeng979e39a78b7faa
diff --git a/README.md b/README.mdindex c73e5d2..835f34a 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**>-> Transform unstructured text into structured, evolving knowledge containers with the power of Large Language Models.+> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.---@@ -10,7 +10,7 @@**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.-Every knowledge container in Hyper-Extract follows a unified lifecycle:+Every Auto container in Hyper-Extract follows a unified lifecycle:1. **Extract** - Intelligent extraction from text with automatic chunking and merging2. **Evolve** - Self-refinement through reasoning, pruning, and optimization@@ -27,7 +27,7 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:- **Intelligent Merging**: First-priority merge strategy for chunk-based extraction### 🏗️ Unified Architecture-- **Abstract Base Class**: `BaseKnowledge[T]` provides consistent interface across all knowledge types+- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline@@ -37,11 +37,11 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:- **Field-Level Search**: Search across all schema fields with metadata preservation### 📦 Multiple Knowledge Types (Roadmap)-- ✅ **UnitKnowledge**: Universal schema-agnostic knowledge extraction-- 🚧 **EntityKnowledge**: Entity-centric extraction with attribute management-- 🚧 **RelationKnowledge**: Relationship extraction with entity linking-- 🚧 **GraphKnowledge**: Knowledge graph construction (nodes + edges)-- 🚧 **HypergraphKnowledge**: Hypergraph representation for complex multi-entity relations+- ✅ **AutoModel**: Universal schema-agnostic knowledge extraction+- ✅ **AutoList**: Collection-based extraction for lists of items+- ✅ **AutoSet**: Unique collection with automatic deduplication+- 🚧 **AutoGraph**: Knowledge graph construction (nodes + edges)+- 🚧 **AutoHypergraph**: Hypergraph representation for complex multi-entity relations- 🚧 **ExtractionExperience**: Meta-learning from extraction patterns---@@ -77,7 +77,7 @@ pip install -r requirements.txtfrom pydantic import BaseModel, Fieldfrom typing import List-class ArticleKnowledge(BaseModel):+class ArticleSchema(BaseModel):"""Schema for article knowledge extraction"""title: str = Field(default="", description="Article title")author: str = Field(default="", description="Author name")@@ -86,19 +86,19 @@ class ArticleKnowledge(BaseModel):topics: List[str] = Field(default_factory=list, description="Topic tags")```-### 2. Initialize Knowledge Container+### 2. Initialize Auto Container```pythonfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract import AutoModel# Initialize LLM and embedderllm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")-# Create knowledge container-knowledge = UnitKnowledge(- data_schema=ArticleKnowledge,+# Create Auto container+a_article = AutoModel(+ data_schema=ArticleSchema,llm_client=llm,embedder=embedder,chunk_size=2000, # Auto-split long texts@@ -119,23 +119,23 @@ Artificial intelligence is transforming industries..."""# Extract with automatic chunking and merging-result = knowledge.extract(article_text)+result = a_article.extract(article_text)print(f"Extraction completed: {result}")# Access extracted data-print(f"Title: {knowledge.data.title}")-print(f"Author: {knowledge.data.author}")-print(f"Key points: {knowledge.data.key_points}")+print(f"Title: {a_article.data.title}")+print(f"Author: {a_article.data.author}")+print(f"Key points: {a_article.data.key_points}")```### 4. Semantic Search```python# Build vector index-knowledge.build_index()+a_article.build_index()# Search with natural language-results = knowledge.search("What are the main topics?", top_k=3)+results = a_article.search("What are the main topics?", top_k=3)for result in results:print(result)```@@ -144,10 +144,10 @@ for result in results:```python# Save to folder (includes data + FAISS index)-knowledge.dump("./output/my_knowledge")+a_article.dump("./output/my_article")# Load from folder-knowledge.load("./output/my_knowledge")+a_article.load("./output/my_article")```---@@ -156,7 +156,7 @@ knowledge.load("./output/my_knowledge")### Design Principles-1. **Template Method Pattern**: `BaseKnowledge` defines the extraction skeleton, subclasses customize behavior+1. **Template Method Pattern**: `BaseAutoType` defines the extraction skeleton, subclasses customize behavior2. **First-Priority Merge**: `model_copy(update=model_dump(exclude_none=True))` for conflict resolution3. **Lazy Indexing**: Build FAISS indices only when needed, track with `_index_dirty` flag4. **Type Safety**: Leverage Pydantic for runtime validation and IDE support@@ -167,7 +167,7 @@ knowledge.load("./output/my_knowledge")Check out the `examples/` directory for complete working examples:-- [`generic_knowledge_demo.py`](examples/generic_knowledge_demo.py) - Full UnitKnowledge walkthrough+- [`auto_model_demo.py`](examples/auto_model_demo.py) - Full AutoModel walkthrough- More examples coming soon!---@@ -175,17 +175,16 @@ Check out the `examples/` directory for complete working examples:## 🗺️ Roadmap### Phase 1: Foundation (Current) ✅-- [x] Abstract `BaseKnowledge` class-- [x] `UnitKnowledge` implementation+- [x] Abstract `BaseAutoType` class+- [x] `AutoModel` implementation- [x] LangChain integration with structured output- [x] FAISS vector search- [x] Serialization (dump/load)### Phase 2: Specialized Knowledge Types 🚧-- [ ] `EntityKnowledge` - Entity extraction with coreference resolution-- [ ] `RelationKnowledge` - Relationship extraction and linking-- [ ] `GraphKnowledge` - Full knowledge graph construction-- [ ] `HypergraphKnowledge` - Hypergraph for complex relations+- [ ] `AutoGraph` - Full knowledge graph construction+- [ ] `AutoHypergraph` - Hypergraph for complex relations+- [ ] `AutoTree` - Hierarchical tree structures### Phase 3: Evolution & Learning 🔮- [ ] `evolve()` implementation with reasoningdiff --git a/docs/BUILDING.md b/docs/BUILDING.mdindex 0cd6674..c6bb2b1 100644--- a/docs/BUILDING.md+++ b/docs/BUILDING.md@@ -104,10 +104,10 @@ docs/│ └── knowledge-patterns.md # Pattern selection guide├── api/│ ├── index.md # API overview-│ ├── base.md # BaseKnowledge API-│ ├── unit.md # UnitKnowledge API-│ ├── list.md # ListKnowledge API-│ └── set.md # SetKnowledge API+│ ├── base.md # BaseAutoType API+│ ├── unit.md # AutoModel API+│ ├── list.md # AutoList API+│ └── set.md # AutoSet API└── zh/└── index.md # Chinese homepage (中文首页)```@@ -205,9 +205,9 @@ plugins:Usage in markdown:```markdown-# BaseKnowledge API+# BaseAutoType API-::: hyperextract.knowledge.base.BaseKnowledge+::: hyperextract.core.base.BaseAutoTypeoptions:show_source: trueheading_level: 2@@ -290,7 +290,7 @@ jobs:1. **Use code blocks with language hints**:````markdown```python- from hyperextract import UnitKnowledge+ from hyperextract import AutoModel```````diff --git a/docs/api/list.md b/docs/api/auto_list.mdsimilarity index 88%rename from docs/api/list.mdrename to docs/api/auto_list.mdindex 2289e37..02e6f68 100644--- a/docs/api/list.md+++ b/docs/api/auto_list.md@@ -1,6 +1,6 @@-# ListKnowledge API+# AutoList API-::: hyperextract.knowledge.generic.list.ListKnowledge+::: hyperextract.core.list.AutoListoptions:show_root_heading: trueshow_source: true@@ -8,12 +8,12 @@## Overview-`ListKnowledge` implements the multi-item list extraction pattern. It extracts multiple independent objects of the same type from a document, suitable for entity lists, events, citations, and more.+`AutoList` implements the multi-item list extraction pattern. It extracts multiple independent objects of the same type from a document, suitable for entity lists, events, citations, and more.## Class Signature```python-class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):+class AutoList(BaseAutoType[ItemListSchema[Item]], Generic[Item]):"""List Knowledge Pattern - extracts a collection of objects from text."""```@@ -27,7 +27,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):## Initialization```python-from hyperextract.knowledge.generic import ListKnowledge+from hyperextract.core import AutoListfrom pydantic import BaseModel, Fieldfrom typing import Optional@@ -36,7 +36,7 @@ class Person(BaseModel):role: Optional[str] = Field(None, description="Role or title")organization: Optional[str] = Field(None, description="Organization")-people = ListKnowledge(+people = AutoList(item_schema=Person, # Note: item_schema, not data_schema!llm_client=llm,embedder=embedder@@ -74,7 +74,7 @@ print(len(container.items))## List Operations-`ListKnowledge` supports Pythonic list operations:+`AutoList` supports Pythonic list operations:### Indexing```python@@ -215,7 +215,7 @@ class Entity(BaseModel):type: str = Field(..., description="Entity type (person/org/location)")context: str = Field(default="", description="Surrounding context")-entities = ListKnowledge(+entities = AutoList(item_schema=Entity,llm_client=llm,embedder=embedder@@ -230,7 +230,7 @@ class Event(BaseModel):participants: List[str] = Field(default_factory=list)location: Optional[str] = None-events = ListKnowledge(+events = AutoList(item_schema=Event,llm_client=llm,embedder=embedder@@ -245,7 +245,7 @@ class Citation(BaseModel):year: Optional[int] = Nonevenue: Optional[str] = None-citations = ListKnowledge(+citations = AutoList(item_schema=Citation,llm_client=llm,embedder=embedder@@ -259,7 +259,7 @@ citations = ListKnowledge(Subclass to implement custom deduplication:```python-class SmartListKnowledge(ListKnowledge[Item]):+class SmartAutoList(AutoList[Item]):def _merge_strategy(self, old_data, new_data):# Custom deduplication logicexisting_names = {item.name for item in old_data.items}@@ -312,18 +312,18 @@ sorted_people = sorted(people.items, key=attrgetter('organization', 'name'))4. **Chunk Size**: Smaller chunks for dense information```python# For entity-rich text- ListKnowledge(..., chunk_size=1500, chunk_overlap=150)+ AutoList(..., chunk_size=1500, chunk_overlap=150)```## Serialization```python# Save-people.dump("./people_knowledge")+people.dump("./a_people")# Load-loaded = ListKnowledge.load(- "./people_knowledge",+loaded = AutoList.load(+ "./a_people",item_schema=Person,llm_client=llm,embedder=embedder@@ -332,9 +332,9 @@ loaded = ListKnowledge.load(assert len(loaded) == len(people)```-## Comparison with SetKnowledge+## Comparison with AutoSet-| Feature | ListKnowledge | SetKnowledge |+| Feature | AutoList | AutoSet ||---------|---------------|--------------|| Duplicates | Allowed (basic dedup) | Prevented (by unique_key) || Merge | Simple append | Configurable strategies |@@ -346,6 +346,6 @@ assert len(loaded) == len(people)## See Also-- [SetKnowledge API](set.md) - Unique collection with deduplication-- [BaseKnowledge API](base.md) - Base class documentation+- [AutoSet API](set.md) - Unique collection with deduplication+- [BaseAutoType API](base.md) - Base class documentation- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selectiondiff --git a/docs/api/set.md b/docs/api/auto_set.mdsimilarity index 89%rename from docs/api/set.mdrename to docs/api/auto_set.mdindex dfca60e..4464427 100644--- a/docs/api/set.md+++ b/docs/api/auto_set.md@@ -1,6 +1,6 @@-# SetKnowledge API+# AutoSet API-::: hyperextract.knowledge.generic.set.SetKnowledge+::: hyperextract.core.set.AutoSetoptions:show_root_heading: trueshow_source: true@@ -8,12 +8,12 @@## Overview-`SetKnowledge` implements the unique collection pattern with automatic deduplication. It extracts items and ensures uniqueness based on a user-specified key field, with configurable merge strategies for handling duplicates.+`AutoSet` implements the unique collection pattern with automatic deduplication. It extracts items and ensures uniqueness based on a user-specified key field, with configurable merge strategies for handling duplicates.## Class Signature```python-class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):+class AutoSet(BaseAutoType[ItemSetSchema[Item]], Generic[Item]):"""Set Knowledge Pattern - extracts a unique collection of objects."""```@@ -28,7 +28,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):## Initialization```python-from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy+from hyperextract.core import AutoSet, MergeItemStrategyfrom pydantic import BaseModel, Fieldclass Keyword(BaseModel):@@ -36,7 +36,7 @@ class Keyword(BaseModel):category: Optional[str] = Field(None, description="Category")importance: Optional[str] = Field(None, description="Importance level")-keywords = SetKnowledge(+keywords = AutoSet(item_schema=Keyword,llm_client=llm,embedder=embedder,@@ -57,7 +57,7 @@ keywords = SetKnowledge(### KEEP_OLDKeep existing item, discard new duplicate:```python-SetKnowledge(+AutoSet(...,merge_strategy=MergeItemStrategy.KEEP_OLD)@@ -73,7 +73,7 @@ SetKnowledge(### KEEP_NEW (Default)Replace existing with new:```python-SetKnowledge(+AutoSet(...,merge_strategy=MergeItemStrategy.KEEP_NEW)@@ -89,7 +89,7 @@ SetKnowledge(### FIELD_MERGEMerge fields - new fills old's None/empty fields:```python-SetKnowledge(+AutoSet(...,merge_strategy=MergeItemStrategy.FIELD_MERGE)@@ -105,7 +105,7 @@ SetKnowledge(### LLM_MERGEUse LLM to intelligently merge:```python-SetKnowledge(+AutoSet(...,merge_strategy=MergeItemStrategy.LLM_MERGE)@@ -216,7 +216,7 @@ if keywords.contains("Python"):## Set Operations-`SetKnowledge` supports mathematical set operations:+`AutoSet` supports mathematical set operations:### Union (`|`)Combine two sets:@@ -248,8 +248,8 @@ exclusive = keywords1 ^ keywords2### Example:```python-tech_keywords = SetKnowledge(item_schema=Keyword, ...)-science_keywords = SetKnowledge(item_schema=Keyword, ...)+tech_keywords = AutoSet(item_schema=Keyword, ...)+science_keywords = AutoSet(item_schema=Keyword, ...)# Extract from different sourcestech_keywords.extract(tech_article)@@ -283,7 +283,7 @@ class Entity(BaseModel):type: str = Field(..., description="Entity type")description: Optional[str] = None-entities = SetKnowledge(+entities = AutoSet(item_schema=Entity,llm_client=llm,embedder=embedder,@@ -299,7 +299,7 @@ class Topic(BaseModel):subtopics: List[str] = Field(default_factory=list)relevance: Optional[str] = None-topics = SetKnowledge(+topics = AutoSet(item_schema=Topic,llm_client=llm,embedder=embedder,@@ -315,7 +315,7 @@ class Skill(BaseModel):proficiency: Optional[str] = Noneyears_experience: Optional[int] = None-skills = SetKnowledge(+skills = AutoSet(item_schema=Skill,llm_client=llm,embedder=embedder,@@ -331,7 +331,7 @@ skills = SetKnowledge(For complex merging, use `LLM_MERGE` or subclass:```python-class SmartSetKnowledge(SetKnowledge[Item]):+class SmartAutoSet(AutoSet[Item]):def _merge_items_llm(self, old_item: Item, new_item: Item) -> Item:# Custom LLM prompt for mergingcustom_prompt = f'''@@ -385,7 +385,7 @@ terms_only = [k.term for k in keywords.items]3. **Performance Considerations**:```python# LLM_MERGE is slower (makes LLM calls)- SetKnowledge(..., merge_strategy=MergeItemStrategy.LLM_MERGE)+ AutoSet(..., merge_strategy=MergeItemStrategy.LLM_MERGE)# For large datasets, consider FIELD_MERGE or KEEP_NEW```@@ -397,7 +397,7 @@ terms_only = [k.term for k in keywords.items]id: str = Field(..., description="Unique ID") # Required!...- knowledge = SetKnowledge(..., unique_key="id")+ knowledge = AutoSet(..., unique_key="id")```## Serialization@@ -407,7 +407,7 @@ terms_only = [k.term for k in keywords.items]keywords.dump("./keywords_set")# Load (preserves uniqueness)-loaded = SetKnowledge.load(+loaded = AutoSet.load("./keywords_set",item_schema=Keyword,llm_client=llm,@@ -419,7 +419,7 @@ loaded = SetKnowledge.load(## Comparison-| Feature | ListKnowledge | SetKnowledge |+| Feature | AutoList | AutoSet ||---------|---------------|--------------|| Duplicates | Allowed | Prevented by unique_key || Merge Strategies | Simple append | 4 configurable strategies |@@ -429,6 +429,6 @@ loaded = SetKnowledge.load(## See Also-- [ListKnowledge API](list.md) - Multi-item list extraction-- [BaseKnowledge API](base.md) - Base class documentation+- [AutoList API](list.md) - Multi-item list extraction+- [BaseAutoType API](base.md) - Base class documentation- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guidediff --git a/docs/api/base.md b/docs/api/base.mdindex a29b8d8..f1cad3c 100644--- a/docs/api/base.md+++ b/docs/api/base.md@@ -1,6 +1,6 @@-# BaseKnowledge API+# BaseAutoType API-::: hyperextract.knowledge.base.BaseKnowledge+::: hyperextract.core.base.BaseAutoTypeoptions:show_root_heading: trueshow_source: true@@ -8,12 +8,12 @@## Overview-`BaseKnowledge` is the abstract base class that provides the foundation for all knowledge extraction patterns in Hyper-Extract. It implements the core lifecycle operations: extraction, merging, indexing, evolution, and serialization.+`BaseAutoType` is the abstract base class that provides the foundation for all knowledge extraction patterns in Hyper-Extract. It implements the core lifecycle operations: extraction, merging, indexing, evolution, and serialization.## Class Signature```python-class BaseKnowledge(ABC, Generic[T]):+class BaseAutoType(ABC, Generic[T]):"""Unified knowledge base class integrating extraction, storage,aggregation, and evolution."""```@@ -38,7 +38,7 @@ Where `T` is a Pydantic `BaseModel` subclass defining your knowledge schema.### Example```python-from hyperextract.knowledge.base import BaseKnowledge+from hyperextract.core.base import BaseAutoTypefrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom pydantic import BaseModel, Field@@ -49,8 +49,8 @@ class MySchema(BaseModel):llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")-# Note: BaseKnowledge is abstract, use concrete implementations-# knowledge = UnitKnowledge(...)+# Note: BaseAutoType is abstract, use concrete implementations+# knowledge = AutoModel(...)```## Properties@@ -189,7 +189,7 @@ def load(llm_client: BaseChatModel,embedder: Embeddings,**kwargs-) -> "BaseKnowledge[T]"+) -> "BaseAutoType[T]"```Load knowledge from disk.@@ -205,7 +205,7 @@ Load knowledge from disk.**Example**:```python-loaded = UnitKnowledge.load(+loaded = AutoModel.load("./my_knowledge",data_schema=MySchema,llm_client=llm,@@ -268,14 +268,14 @@ Handles multi-chunk extraction with parallel processing and merging.## Usage in Subclasses-When creating custom knowledge patterns, inherit from `BaseKnowledge`:+When creating custom knowledge patterns, inherit from `BaseAutoType`:```python-from hyperextract.knowledge.base import BaseKnowledge, T+from hyperextract.core.base import BaseAutoType, Tfrom typing import Listfrom langchain_core.documents import Document-class CustomKnowledge(BaseKnowledge[T]):+class CustomKnowledge(BaseAutoType[T]):@staticmethoddef _default_prompt() -> str:return "Extract information according to schema: "@@ -291,6 +291,6 @@ class CustomKnowledge(BaseKnowledge[T]):## See Also-- [UnitKnowledge API](unit.md) - Single-object extraction-- [ListKnowledge API](list.md) - Multi-item extraction-- [SetKnowledge API](set.md) - Unique collection extraction+- [AutoModel API](unit.md) - Single-object extraction+- [AutoList API](list.md) - Multi-item extraction+- [AutoSet API](set.md) - Unique collection extractiondiff --git a/docs/api/index.md b/docs/api/index.mdindex 5c145e6..2d039c1 100644--- a/docs/api/index.md+++ b/docs/api/index.md@@ -7,19 +7,19 @@ This section provides detailed API documentation for all Hyper-Extract classes a```hyperextract/├── knowledge/-│ ├── base.py # BaseKnowledge abstract class+│ ├── base.py # BaseAutoType abstract class│ └── generic/-│ ├── unit.py # UnitKnowledge-│ ├── list.py # ListKnowledge-│ └── set.py # SetKnowledge+│ ├── unit.py # AutoModel+│ ├── list.py # AutoList+│ └── set.py # AutoSet```## Quick Links-- [BaseKnowledge](base.md) - Abstract base class for all knowledge patterns-- [UnitKnowledge](unit.md) - Single-object extraction pattern-- [ListKnowledge](list.md) - Multi-item list extraction pattern-- [SetKnowledge](set.md) - Unique collection with deduplication+- [BaseAutoType](base.md) - Abstract base class for all knowledge patterns+- [AutoModel](unit.md) - Single-object extraction pattern+- [AutoList](list.md) - Multi-item list extraction pattern+- [AutoSet](set.md) - Unique collection with deduplication## Common Patterns@@ -59,14 +59,14 @@ Every knowledge class implements these methods:All classes are fully typed with Python generics:```python-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract.core import AutoModelfrom pydantic import BaseModelclass MySchema(BaseModel):field: str# Type-safe: knowledge.data is of type MySchema-knowledge: UnitKnowledge[MySchema] = UnitKnowledge(+knowledge: AutoModel[MySchema] = AutoModel(data_schema=MySchema,llm_client=llm,embedder=embedder@@ -75,7 +75,7 @@ knowledge: UnitKnowledge[MySchema] = UnitKnowledge(## Next: Detailed API Docs-- [BaseKnowledge API](base.md)-- [UnitKnowledge API](unit.md)-- [ListKnowledge API](list.md)-- [SetKnowledge API](set.md)+- [BaseAutoType API](base.md)+- [AutoModel API](unit.md)+- [AutoList API](list.md)+- [AutoSet API](set.md)diff --git a/docs/api/unit.md b/docs/api/model.mdsimilarity index 86%rename from docs/api/unit.mdrename to docs/api/model.mdindex 164b79e..a77fba2 100644--- a/docs/api/unit.md+++ b/docs/api/model.md@@ -1,6 +1,6 @@-# UnitKnowledge API+# AutoModel API-::: hyperextract.knowledge.generic.unit.UnitKnowledge+::: hyperextract.core.model.AutoModeloptions:show_root_heading: trueshow_source: true@@ -8,12 +8,12 @@## Overview-`UnitKnowledge` implements the single-object extraction pattern. It's designed to extract exactly one structured object per document, making it ideal for document-level information like summaries, metadata, or aggregate statistics.+`AutoModel` implements the single-object extraction pattern. It's designed to extract exactly one structured object per document, making it ideal for document-level information like summaries, metadata, or aggregate statistics.## Class Signature```python-class UnitKnowledge(BaseKnowledge[T]):+class AutoModel(BaseAutoType[T]):"""Unit Knowledge Pattern - extracts a single structured object from text."""```@@ -26,10 +26,10 @@ class UnitKnowledge(BaseKnowledge[T]):## Initialization-Inherits all parameters from `BaseKnowledge` with same defaults.+Inherits all parameters from `BaseAutoType` with same defaults.```python-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract.core import AutoModelfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom pydantic import BaseModel, Field@@ -41,7 +41,7 @@ class DocumentMetadata(BaseModel):llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")-knowledge = UnitKnowledge(+knowledge = AutoModel(data_schema=DocumentMetadata,llm_client=llm,embedder=embedder,@@ -178,7 +178,7 @@ class Summary(BaseModel):key_points: List[str] = Field(default_factory=list, description="Key points")conclusion: str = Field(default="", description="Conclusion")-summary_knowledge = UnitKnowledge(+summary_knowledge = AutoModel(data_schema=Summary,llm_client=llm,embedder=embedder@@ -193,7 +193,7 @@ class Article(BaseModel):author: str = Field(default="", description="Author")category: str = Field(default="", description="Category")-article_knowledge = UnitKnowledge(+article_knowledge = AutoModel(data_schema=Article,llm_client=llm,embedder=embedder@@ -208,7 +208,7 @@ class Sentiment(BaseModel):negative_aspects: List[str] = Field(default_factory=list)confidence: float = Field(default=0.0, description="Confidence score")-sentiment_knowledge = UnitKnowledge(+sentiment_knowledge = AutoModel(data_schema=Sentiment,llm_client=llm,embedder=embedder@@ -231,10 +231,10 @@ sentiment_knowledge = UnitKnowledge(3. **Chunk Sizing**: Adjust based on where information appears```python# For metadata (usually at start)- UnitKnowledge(..., chunk_size=1000, chunk_overlap=100)+ AutoModel(..., chunk_size=1000, chunk_overlap=100)# For summary (may span document)- UnitKnowledge(..., chunk_size=3000, chunk_overlap=300)+ AutoModel(..., chunk_size=3000, chunk_overlap=300)```4. **Custom Prompts**: Provide domain-specific guidance@@ -246,7 +246,7 @@ sentiment_knowledge = UnitKnowledge(- Citation count if mentioned'''- knowledge = UnitKnowledge(+ knowledge = AutoModel(data_schema=PaperMetadata,llm_client=llm,embedder=embedder,@@ -261,7 +261,7 @@ sentiment_knowledge = UnitKnowledge(knowledge.dump("./document_metadata")# Load-loaded = UnitKnowledge.load(+loaded = AutoModel.load("./document_metadata",data_schema=DocumentMetadata,llm_client=llm,@@ -273,7 +273,7 @@ assert loaded.data.title == knowledge.data.title## See Also-- [BaseKnowledge API](base.md) - Base class documentation-- [ListKnowledge API](list.md) - For extracting multiple items-- [SetKnowledge API](set.md) - For unique collections+- [BaseAutoType API](base.md) - Base class documentation+- [AutoList API](list.md) - For extracting multiple items+- [AutoSet API](set.md) - For unique collections- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guidediff --git a/docs/index.md b/docs/index.mdindex db96777..d42b0a8 100644--- a/docs/index.md+++ b/docs/index.md@@ -2,7 +2,7 @@> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**>-> Transform unstructured text into structured, evolving knowledge containers with the power of Large Language Models.+> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.---@@ -10,7 +10,7 @@**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.-Every knowledge container in Hyper-Extract follows a unified lifecycle:+Every Auto container in Hyper-Extract follows a unified lifecycle:1. **Extract** - Intelligent extraction from text with automatic chunking and merging2. **Evolve** - Self-refinement through reasoning, pruning, and optimization@@ -27,7 +27,7 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:- **Intelligent Merging**: First-priority merge strategy for chunk-based extraction### 🏗️ Unified Architecture-- **Abstract Base Class**: `BaseKnowledge[T]` provides consistent interface across all knowledge types+- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline@@ -39,15 +39,14 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:### 📦 Knowledge Patterns#### ✅ Implemented-- **UnitKnowledge**: Single-object extraction for document-level information (summaries, metadata)-- **ListKnowledge**: Collection-based extraction for entities, events, and structured lists-- **SetKnowledge**: Unique collection with automatic deduplication and intelligent merge strategies+- **AutoModel**: Single-object extraction for document-level information (summaries, metadata)+- **AutoList**: Collection-based extraction for entities, events, and structured lists+- **AutoSet**: Unique collection with automatic deduplication and intelligent merge strategies#### 🚧 In Development-- **EntityKnowledge**: Entity-centric extraction with attribute management-- **RelationKnowledge**: Relationship extraction with entity linking-- **GraphKnowledge**: Knowledge graph construction (nodes + edges)-- **HypergraphKnowledge**: Hypergraph representation for complex multi-entity relations+- **AutoGraph**: Knowledge graph construction (nodes + edges)+- **AutoHypergraph**: Hypergraph representation for complex multi-entity relations+- **AutoTree**: Hierarchical tree structures for nested data---@@ -82,7 +81,7 @@ pip install -e .from pydantic import BaseModel, Fieldfrom typing import List-class ArticleKnowledge(BaseModel):+class ArticleSchema(BaseModel):"""Schema for article knowledge extraction"""title: str = Field(default="", description="Article title")author: str = Field(default="", description="Author name")@@ -95,15 +94,15 @@ class ArticleKnowledge(BaseModel):```pythonfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract.core import AutoModel# Initialize LLM and embedderllm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")-# Create knowledge container-knowledge = UnitKnowledge(- data_schema=ArticleKnowledge,+# Create Auto container+a_article = AutoModel(+ data_schema=ArticleSchema,llm_client=llm,embedder=embedder)@@ -118,18 +117,18 @@ text = """written by Professor Li Ming..."""-knowledge.extract(text)-print(knowledge.data) # Access extracted structured knowledge+a_article.extract(text)+print(a_article.data) # Access extracted structured knowledge```### 4. Semantic Search```python# Build index-knowledge.build_index()+a_article.build_index()# Search for relevant content-results = knowledge.search("applications of deep learning", k=3)+results = a_article.search("applications of deep learning", k=3)for doc, score in results:print(f"Score: {score}, Content: {doc.page_content}")```@@ -138,12 +137,12 @@ for doc, score in results:```python# Save to disk-knowledge.dump("./my_knowledge")+a_article.dump("./my_knowledge")# Load-loaded_knowledge = UnitKnowledge.load(+a_loaded = AutoModel.load("./my_knowledge",- data_schema=ArticleKnowledge,+ data_schema=ArticleSchema,llm_client=llm,embedder=embedder)diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.mdindex cef91c0..953a86d 100644--- a/docs/user-guide/getting-started.md+++ b/docs/user-guide/getting-started.md@@ -63,7 +63,7 @@ class BookReview(BaseModel):### Knowledge Container-A **knowledge container** is an instance of a knowledge pattern class (e.g., `UnitKnowledge`, `ListKnowledge`, `SetKnowledge`) that manages extraction, storage, and retrieval of structured knowledge.+A **Auto container** is an instance of a knowledge pattern class (e.g., `AutoModel`, `AutoList`, `AutoSet`) that manages extraction, storage, and retrieval of structured knowledge.### Extraction Pipeline@@ -109,10 +109,10 @@ embedder = OpenAIEmbeddings(model="text-embedding-3-small")### Step 3: Create Knowledge Container```python-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract.core import AutoModel# Create container-paper_knowledge = UnitKnowledge(+paper_knowledge = AutoModel(data_schema=ResearchPaper,llm_client=llm,embedder=embedder,@@ -172,7 +172,7 @@ for doc, score in results:paper_knowledge.dump("./paper_knowledge")# Later, load it back-loaded_knowledge = UnitKnowledge.load(+loaded_knowledge = AutoModel.load("./paper_knowledge",data_schema=ResearchPaper,llm_client=llm,@@ -194,7 +194,7 @@ Hyper-Extract automatically handles long documents by:You can control chunking behavior:```python-knowledge = UnitKnowledge(+knowledge = AutoModel(data_schema=YourSchema,llm_client=llm,embedder=embedder,diff --git a/docs/user-guide/knowledge-patterns.md b/docs/user-guide/knowledge-patterns.mdindex d34a28a..26aa4ed 100644--- a/docs/user-guide/knowledge-patterns.md+++ b/docs/user-guide/knowledge-patterns.md@@ -8,17 +8,17 @@ Hyper-Extract provides three fundamental knowledge patterns, each optimized for| Pattern | Use Case | Deduplication | Merge Strategy ||---------|----------|---------------|----------------|-| **UnitKnowledge** | Single object per document | N/A | Field-level merge |-| **ListKnowledge** | Multiple independent items | Basic | Append |-| **SetKnowledge** | Unique collection | Automatic | Configurable |+| **AutoModel** | Single object per document | N/A | Field-level merge |+| **AutoList** | Multiple independent items | Basic | Append |+| **AutoSet** | Unique collection | Automatic | Configurable |----## UnitKnowledge Pattern+## AutoModel Pattern### When to Use-Use `UnitKnowledge` when you want to extract **exactly one structured object** from a document, regardless of its length.+Use `AutoModel` when you want to extract **exactly one structured object** from a document, regardless of its length.**Perfect for**:- Document summaries@@ -39,7 +39,7 @@ Use `UnitKnowledge` when you want to extract **exactly one structured object** ffrom pydantic import BaseModel, Fieldfrom typing import Listfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract.core import AutoModel# Define schemaclass DocumentMetadata(BaseModel):@@ -54,7 +54,7 @@ class DocumentMetadata(BaseModel):llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")-metadata = UnitKnowledge(+metadata = AutoModel(data_schema=DocumentMetadata,llm_client=llm,embedder=embedder@@ -84,11 +84,11 @@ results = metadata.search("machine learning applications", k=5)----## ListKnowledge Pattern+## AutoList Pattern### When to Use-Use `ListKnowledge` when you need to extract **multiple independent items** of the same type from a document.+Use `AutoList` when you need to extract **multiple independent items** of the same type from a document.**Perfect for**:- Entity lists (people, organizations, locations)@@ -108,7 +108,7 @@ Use `ListKnowledge` when you need to extract **multiple independent items** of t```pythonfrom pydantic import BaseModel, Fieldfrom typing import List, Optional-from hyperextract.knowledge.generic import ListKnowledge+from hyperextract.core import AutoList# Define item schemaclass Person(BaseModel):@@ -118,7 +118,7 @@ class Person(BaseModel):context: Optional[str] = Field(None, description="Context of mention")# Initialize-people = ListKnowledge(+people = AutoList(item_schema=Person, # Note: item_schema, not data_schemallm_client=llm,embedder=embedder@@ -142,7 +142,7 @@ print(f"Total people extracted: {len(people.items)}")### List Operations-`ListKnowledge` supports Pythonic list operations:+`AutoList` supports Pythonic list operations:```python# Append new items@@ -185,11 +185,11 @@ for doc, score in results:----## SetKnowledge Pattern+## AutoSet Pattern### When to Use-Use `SetKnowledge` when you need a **unique collection** with automatic deduplication based on a key field.+Use `AutoSet` when you need a **unique collection** with automatic deduplication based on a key field.**Perfect for**:- Keywords/tags (deduplicated by term)@@ -210,7 +210,7 @@ Use `SetKnowledge` when you need a **unique collection** with automatic deduplic```pythonfrom pydantic import BaseModel, Fieldfrom typing import Optional-from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy+from hyperextract.core import AutoSet, MergeItemStrategy# Define item schema with unique keyclass Keyword(BaseModel):@@ -220,7 +220,7 @@ class Keyword(BaseModel):context: Optional[str] = Field(None, description="Usage context")# Initialize with unique_key-keywords = SetKnowledge(+keywords = AutoSet(item_schema=Keyword,llm_client=llm,embedder=embedder,@@ -246,7 +246,7 @@ for kw in keywords.items:#### 1. KEEP_OLDKeeps existing item, discards new duplicate:```python-keywords = SetKnowledge(+keywords = AutoSet(item_schema=Keyword,llm_client=llm,embedder=embedder,@@ -275,7 +275,7 @@ merge_strategy=MergeItemStrategy.LLM_MERGE### Set Operations-`SetKnowledge` supports set operations:+`AutoSet` supports set operations:```python# Union: combine two sets@@ -293,7 +293,7 @@ exclusive = keywords1 ^ keywords2### Indexing Strategy-Like ListKnowledge, each unique item is indexed:+Like AutoList, each unique item is indexed:```pythonkeywords.build_index()@@ -308,39 +308,39 @@ results = keywords.search("AI technologies", k=5)```Is it a single object per document?-├─ Yes → Use UnitKnowledge+├─ Yes → Use AutoModel└─ No → Are duplicates acceptable?- ├─ Yes → Use ListKnowledge- └─ No → Use SetKnowledge (specify unique_key)+ ├─ Yes → Use AutoList+ └─ No → Use AutoSet (specify unique_key)```### Examples by Domain**News Article Processing**:-- `UnitKnowledge`: Article metadata, summary-- `ListKnowledge`: All mentioned events-- `SetKnowledge`: Unique entities, locations+- `AutoModel`: Article metadata, summary+- `AutoList`: All mentioned events+- `AutoSet`: Unique entities, locations**Research Paper Analysis**:-- `UnitKnowledge`: Paper metadata, abstract-- `ListKnowledge`: All citations-- `SetKnowledge`: Unique keywords, methodologies+- `AutoModel`: Paper metadata, abstract+- `AutoList`: All citations+- `AutoSet`: Unique keywords, methodologies**Customer Reviews**:-- `UnitKnowledge`: Overall sentiment, rating-- `ListKnowledge`: All complaints/praises-- `SetKnowledge`: Unique features mentioned+- `AutoModel`: Overall sentiment, rating+- `AutoList`: All complaints/praises+- `AutoSet`: Unique features mentioned---## Advanced: Custom Patterns-You can create custom patterns by subclassing `BaseKnowledge`:+You can create custom patterns by subclassing `BaseAutoType`:```python-from hyperextract.knowledge.base import BaseKnowledge+from hyperextract.core.base import BaseAutoType-class MyCustomPattern(BaseKnowledge[MySchema]):+class MyCustomPattern(BaseAutoType[MySchema]):def _merge_strategy(self, old_data, new_data):# Implement custom merge logicpassdiff --git a/docs/zh/index.md b/docs/zh/index.mdindex cc21c48..af158a4 100644--- a/docs/zh/index.md+++ b/docs/zh/index.md@@ -27,7 +27,7 @@ Hyper-Extract 中的每个知识容器都遵循统一的生命周期:- **智能合并**:针对分块提取的优先级合并策略### 🏗️ 统一架构-- **抽象基类**:`BaseKnowledge[T]` 为所有知识类型提供一致的接口+- **抽象基类**:`BaseAutoType[T]` 为所有知识类型提供一致的接口- **泛型类型支持**:使用 `Generic[T]` 实现完整的 Python 类型提示- **生命周期管理**:标准化的 `extract` → `merge` → `evolve` → `search` → `dump/load` 流程@@ -39,15 +39,14 @@ Hyper-Extract 中的每个知识容器都遵循统一的生命周期:### 📦 知识模式#### ✅ 已实现-- **UnitKnowledge**:单对象提取,适用于文档级信息(摘要、元数据)-- **ListKnowledge**:列表提取,适用于实体、事件和结构化列表-- **SetKnowledge**:唯一集合,支持自动去重和智能合并策略+- **AutoModel**:单对象提取,适用于文档级信息(摘要、元数据)+- **AutoList**:列表提取,适用于实体、事件和结构化列表+- **AutoSet**:唯一集合,支持自动去重和智能合并策略#### 🚧 开发中-- **EntityKnowledge**:实体中心的提取,支持属性管理-- **RelationKnowledge**:关系提取,支持实体链接- **GraphKnowledge**:知识图谱构建(节点 + 边)- **HypergraphKnowledge**:超图表示,用于复杂的多实体关系+- **AutoTree**:层级树结构,用于嵌套数据---@@ -95,14 +94,14 @@ class ArticleKnowledge(BaseModel):```pythonfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract.core import AutoModel# 初始化 LLM 和嵌入模型llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")# 创建知识容器-knowledge = UnitKnowledge(+knowledge = AutoModel(data_schema=ArticleKnowledge,llm_client=llm,embedder=embedder@@ -140,7 +139,7 @@ for doc, score in results:knowledge.dump("./my_knowledge")# 加载-loaded_knowledge = UnitKnowledge.load(+loaded_knowledge = AutoModel.load("./my_knowledge",data_schema=ArticleKnowledge,llm_client=llm,diff --git a/examples/list_knowledge_demo.py b/examples/auto_list_demo.pysimilarity index 88%rename from examples/list_knowledge_demo.pyrename to examples/auto_list_demo.pyindex 31b5de8..49e9384 100644--- a/examples/list_knowledge_demo.py+++ b/examples/auto_list_demo.py@@ -1,7 +1,7 @@"""-ListKnowledge 多 Chunk 提取示例+AutoList 多 Chunk 提取示例-演示如何使用 ListKnowledge 处理长文本,测试多 chunk 并发提取和列表合并功能。+演示如何使用 AutoList 处理长文本,测试多 chunk 并发提取和列表合并功能。"""import sys@@ -14,7 +14,7 @@ sys.path.insert(0, str(project_root))from pydantic import BaseModel, Fieldfrom typing import Optionalfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import ListKnowledge+from hyperextract import AutoList# 定义单个新闻事件的 Schema@@ -125,7 +125,7 @@ Jeff Bezos在发布会上表示,新一代无人机可承载5公斤货物,飞def main():print("=" * 80)- print("ListKnowledge 多 Chunk 提取示例")+ print("AutoList 多 Chunk 提取示例")print("=" * 80)# 1. 生成长文本@@ -139,9 +139,9 @@ def main():llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")- # 3. 创建 ListKnowledge 实例 (使用小的 chunk_size 测试多 chunk)- print("\n[3] 创建 ListKnowledge 实例 (chunk_size=300)...")- knowledge = ListKnowledge(+ # 3. 创建 AutoList 实例 (使用小的 chunk_size 测试多 chunk)+ print("\n[3] 创建 AutoList 实例 (chunk_size=300)...")+ a_event_list = AutoList(item_schema=NewsEvent,llm_client=llm,embedder=embedder,@@ -154,7 +154,7 @@ def main():# 4. 提取知识 (默认存储模式)print("\n[4] 提取新闻事件列表 (默认存储模式)...")print("-" * 80)- events = knowledge.extract(news_text)+ events = a_event_list.extract(news_text)print("-" * 80)# 5. 显示提取结果@@ -173,11 +173,11 @@ def main():print(f"\n ... 还有 {len(events) - 5} 个事件未显示")# 6. 测试列表大小- print(f"\n[6] 知识列表大小: {len(knowledge)} 个事件")+ print(f"\n[6] 知识列表大小: {len(a_event_list)} 个事件")# 7. 构建索引print("\n[7] 构建向量索引...")- knowledge.build_index()+ a_event_list.build_index()# 8. 测试搜索print("\n[8] 测试事件搜索...")@@ -189,7 +189,7 @@ def main():for query in queries:print(f"\n 查询: '{query}'")- results = knowledge.search(query, top_k=3)+ results = a_event_list.search(query, top_k=3)for i, event in enumerate(results, 1):print(f" [{i}] {event.title} ({event.date})")print(f" 地点: {event.location}")@@ -204,9 +204,9 @@ def main():"""print(" 添加补充新闻...")- before_count = len(knowledge)- knowledge.extract(additional_news) # 默认 store=True,自动合并- after_count = len(knowledge)+ before_count = len(a_event_list)+ a_event_list.extract(additional_news) # 默认 store=True,自动合并+ after_count = len(a_event_list)print(f" 合并前: {before_count} 个事件")print(f" 合并后: {after_count} 个事件")print(f" 新增: {after_count - before_count} 个事件")@@ -214,32 +214,32 @@ def main():# 10. 显示最新添加的事件if after_count > before_count:print("\n 最新添加的事件:")- for event in knowledge.items[-3:]:+ for event in a_event_list.items[-3:]:print(f" - {event.title} ({event.date})")- knowledge.build_index()+ a_event_list.build_index()# 11. 保存知识print("\n[10] 保存知识到文件...")- save_path = Path(__file__).parent.parent / "tmp" / "list_knowledge_multi_chunk"- knowledge.dump(str(save_path))+ save_path = Path(__file__).parent.parent / "tmp" / "auto_list_demo"+ a_event_list.dump(str(save_path))print(f" 已保存到: {save_path}")# 12. 重新加载测试print("\n[11] 测试知识加载...")- new_knowledge = ListKnowledge(+ a_loaded_event_list = AutoList(item_schema=NewsEvent,llm_client=llm,embedder=embedder,)- new_knowledge.load(str(save_path))- print(f" 加载成功! 事件数: {len(new_knowledge)}")- print(f" 第一个事件: {new_knowledge.items[0].title}")+ a_loaded_event_list.load(str(save_path))+ print(f" 加载成功! 事件数: {len(a_loaded_event_list)}")+ print(f" 第一个事件: {a_loaded_event_list.items[0].title}")# 13. 测试清空功能print("\n[12] 测试清空功能...")- print(f" 清空前: {len(new_knowledge)} 个事件")- new_knowledge.clear()- print(f" 清空后: {len(new_knowledge)} 个事件")+ print(f" 清空前: {len(a_loaded_event_list)} 个事件")+ a_loaded_event_list.clear()+ print(f" 清空后: {len(a_loaded_event_list)} 个事件")print("\n" + "=" * 80)print("示例运行完成!")diff --git a/examples/list_operations_demo.py b/examples/auto_list_operations_demo.pysimilarity index 66%rename from examples/list_operations_demo.pyrename to examples/auto_list_operations_demo.pyindex 132679d..e302f20 100644--- a/examples/list_operations_demo.py+++ b/examples/auto_list_operations_demo.py@@ -1,7 +1,7 @@"""-演示 ListKnowledge 的 Pythonic 列表操作+演示 AutoList 的 Pythonic 列表操作-这个示例展示了如何像使用 Python 原生 list 一样使用 ListKnowledge:+这个示例展示了如何像使用 Python 原生 list 一样使用 AutoList:- 索引访问和切片- 迭代和成员测试- 追加、插入和扩展@@ -22,7 +22,7 @@ from dotenv import load_dotenvfrom pydantic import BaseModel, Fieldfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import ListKnowledge+from hyperextract import AutoList# 加载环境变量load_dotenv()@@ -49,55 +49,55 @@ def main():api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL"))- # 创建 ListKnowledge 实例- tasks = ListKnowledge(+ # 创建 AutoList 实例+ a_task_list = AutoList(item_schema=Task,llm_client=llm,embedder=embedder,)print("=" * 70)- print("ListKnowledge Pythonic 操作演示")+ print("AutoList Pythonic 操作演示")print("=" * 70)# 1. 使用 append 添加任务print("\n1. 使用 append() 添加任务:")- tasks.append(Task(name="实现用户登录", priority=5, status="done"))- tasks.append(Task(name="设计数据库", priority=4, status="done"))- tasks.append(Task(name="编写API文档", priority=3, status="in_progress"))- print(f" 已添加 {len(tasks)} 个任务")+ a_task_list.append(Task(name="实现用户登录", priority=5, status="done"))+ a_task_list.append(Task(name="设计数据库", priority=4, status="done"))+ a_task_list.append(Task(name="编写API文档", priority=3, status="in_progress"))+ print(f" 已添加 {len(a_task_list)} 个任务")# 2. 索引访问print("\n2. 索引访问:")- print(f" 第一个任务: {tasks[0].name} (优先级 {tasks[0].priority})")- print(f" 最后一个任务: {tasks[-1].name} (状态: {tasks[-1].status})")+ print(f" 第一个任务: {a_task_list[0].name} (优先级 {a_task_list[0].priority})")+ print(f" 最后一个任务: {a_task_list[-1].name} (状态: {a_task_list[-1].status})")# 3. 切片print("\n3. 切片操作:")- high_priority = tasks[:2]+ high_priority = a_task_list[:2]print(f" 前两个高优先级任务:")for task in high_priority:print(f" - {task.name} (P{task.priority})")# 4. 迭代print("\n4. 遍历所有任务:")- for i, task in enumerate(tasks, 1):+ for i, task in enumerate(a_task_list, 1):print(f" {i}. {task.name} - {task.status}")# 5. 使用列表推导式print("\n5. 列表推导式 - 获取所有任务名称:")- task_names = [task.name for task in tasks]+ task_names = [task.name for task in a_task_list]print(f" {task_names}")# 6. 成员测试print("\n6. 成员测试 (in 操作符):")test_task = Task(name="编写API文档", priority=3, status="in_progress")- print(f" 测试任务在列表中: {test_task in tasks}")+ print(f" 测试任务在列表中: {test_task in a_task_list}")# 7. 插入任务print("\n7. 使用 insert() 在开头插入紧急任务:")- tasks.insert(0, Task(name="修复生产BUG", priority=5, status="in_progress"))- print(f" 新的第一个任务: {tasks[0].name}")+ a_task_list.insert(0, Task(name="修复生产BUG", priority=5, status="in_progress"))+ print(f" 新的第一个任务: {a_task_list[0].name}")# 8. 扩展多个任务print("\n8. 使用 extend() 批量添加任务:")@@ -105,79 +105,79 @@ def main():Task(name="代码审查", priority=3, status="pending"),Task(name="性能优化", priority=2, status="pending"),]- tasks.extend(new_tasks)- print(f" 现在共有 {len(tasks)} 个任务")+ a_task_list.extend(new_tasks)+ print(f" 现在共有 {len(a_task_list)} 个任务")# 9. 查找和计数print("\n9. 查找和计数:")- in_progress_count = sum(1 for t in tasks if t.status == "in_progress")+ in_progress_count = sum(1 for t in a_task_list if t.status == "in_progress")print(f" 进行中的任务数量: {in_progress_count}")# 查找第一个 pending 任务的位置- for i, task in enumerate(tasks):+ for i, task in enumerate(a_task_list):if task.status == "pending":print(f" 第一个待处理任务位置: 索引 {i} - {task.name}")break# 10. 修改任务print("\n10. 修改任务状态 (通过索引赋值):")- print(f" 修改前: {tasks[-1].name} - {tasks[-1].status}")- updated_task = tasks[-1]+ print(f" 修改前: {a_task_list[-1].name} - {a_task_list[-1].status}")+ updated_task = a_task_list[-1]updated_task.status = "done"- tasks[-1] = updated_task- print(f" 修改后: {tasks[-1].name} - {tasks[-1].status}")+ a_task_list[-1] = updated_task+ print(f" 修改后: {a_task_list[-1].name} - {a_task_list[-1].status}")# 11. 删除任务print("\n11. 删除已完成的任务:")- print(f" 删除前共有 {len(tasks)} 个任务")+ print(f" 删除前共有 {len(a_task_list)} 个任务")# 删除第一个任务- del tasks[0]- print(f" 删除后共有 {len(tasks)} 个任务")+ del a_task_list[0]+ print(f" 删除后共有 {len(a_task_list)} 个任务")# 12. Pop 操作print("\n12. 使用 pop() 移除并返回任务:")- last_task = tasks.pop()+ last_task = a_task_list.pop()print(f" 弹出的任务: {last_task.name}")- print(f" 剩余 {len(tasks)} 个任务")+ print(f" 剩余 {len(a_task_list)} 个任务")# 13. 复制print("\n13. 创建任务列表的副本:")- tasks_backup = tasks.copy()+ tasks_backup = a_task_list.copy()tasks_backup.append(Task(name="新任务", priority=1, status="pending"))- print(f" 原始列表: {len(tasks)} 个任务")+ print(f" 原始列表: {len(a_task_list)} 个任务")print(f" 副本列表: {len(tasks_backup)} 个任务")print(" ✓ 副本是独立的")# 14. 排序print("\n14. 按优先级排序任务 (降序):")- tasks.sort(key=lambda t: t.priority, reverse=True)+ a_task_list.sort(key=lambda t: t.priority, reverse=True)print(" 排序后的任务列表:")- for task in tasks:+ for task in a_task_list:print(f" - P{task.priority}: {task.name}")# 15. 反转print("\n15. 反转任务列表:")- tasks.reverse()+ a_task_list.reverse()print(" 反转后:")- for task in tasks:+ for task in a_task_list:print(f" - {task.name}")# 16. 字符串表示print("\n16. 字符串表示:")- print(f" repr: {repr(tasks)}")- print(f" str: {str(tasks)}")+ print(f" repr: {repr(a_task_list)}")+ print(f" str: {str(a_task_list)}")# 17. 组合操作- print("\n17. 组合两个 ListKnowledge (+ 操作符):")- more_tasks = ListKnowledge(+ print("\n17. 组合两个 AutoList (+ 操作符):")+ more_tasks = AutoList(item_schema=Task,llm_client=llm,embedder=embedder,)more_tasks.append(Task(name="部署到生产", priority=5, status="pending"))- combined = tasks + more_tasks- print(f" 原始任务: {len(tasks)} 个")+ combined = a_task_list + more_tasks+ print(f" 原始任务: {len(a_task_list)} 个")print(f" 新增任务: {len(more_tasks)} 个")print(f" 组合后: {len(combined)} 个")diff --git a/examples/unit_knowledge_demo.py b/examples/auto_model_demo.pysimilarity index 87%rename from examples/unit_knowledge_demo.pyrename to examples/auto_model_demo.pyindex 0435f74..f85cd02 100644--- a/examples/unit_knowledge_demo.py+++ b/examples/auto_model_demo.py@@ -1,7 +1,7 @@"""-UnitKnowledge 多 Chunk 提取示例+AutoModel 多 Chunk 提取示例-演示如何使用 UnitKnowledge 处理长文本,测试多 chunk 并发提取和合并功能。+演示如何使用 AutoModel 处理长文本,测试多 chunk 并发提取和合并功能。"""import sys@@ -14,11 +14,11 @@ sys.path.insert(0, str(project_root))from pydantic import BaseModel, Fieldfrom typing import List, Optionalfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import UnitKnowledge+from hyperextract import AutoModel# 定义文档知识 Schema-class DocumentKnowledge(BaseModel):+class DocumentSchema(BaseModel):"""文档综合知识结构"""title: str = Field(default="", description="文档标题")@@ -135,7 +135,7 @@ AI医疗诊断系统的可信度和安全性将进一步提升。def main():print("=" * 80)- print("UnitKnowledge 多 Chunk 提取示例")+ print("AutoModel 多 Chunk 提取示例")print("=" * 80)# 1. 生成长文档@@ -149,10 +149,10 @@ def main():llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(model="text-embedding-3-small")- # 3. 创建 UnitKnowledge 实例 (使用小的 chunk_size 测试多 chunk)- print("\n[3] 创建 UnitKnowledge 实例 (chunk_size=200)...")- knowledge = UnitKnowledge(- data_schema=DocumentKnowledge,+ # 3. 创建 AutoModel 实例 (使用小的 chunk_size 测试多 chunk)+ print("\n[3] 创建 AutoModel 实例 (chunk_size=200)...")+ a_document_model = AutoModel(+ data_schema=DocumentSchema,llm_client=llm,embedder=embedder,chunk_size=200, # 设置小的 chunk_size 来测试多 chunk 提取@@ -164,7 +164,7 @@ def main():# 4. 提取知识 (默认存储模式)print("\n[4] 提取知识 (默认存储模式)...")print("-" * 80)- extracted_data = knowledge.extract(document)+ extracted_data = a_document_model.extract(document)print("-" * 80)# 5. 显示提取结果@@ -181,11 +181,11 @@ def main():print(f" 关键词数量: {len(extracted_data.keywords)}")# 6. 测试字段数量- print(f"\n[6] 知识字段数: {len(knowledge)}")+ print(f"\n[6] 知识字段数: {len(a_document_model)}")# 7. 构建索引print("\n[7] 构建向量索引...")- knowledge.build_index()+ a_document_model.build_index()# 8. 测试搜索print("\n[8] 测试知识搜索...")@@ -197,7 +197,7 @@ def main():for query in queries:print(f"\n 查询: '{query}'")- results = knowledge.search(query, top_k=2)+ results = a_document_model.search(query, top_k=2)for i, result in enumerate(results, 1):field_name = list(result.keys())[0]field_value = result[field_name]@@ -211,25 +211,25 @@ def main():研究团队来自清华大学医学院和北京协和医院。"""print(" 添加补充文本...")- knowledge.extract(additional_text) # 默认 store=True,自动合并- print(f" 更新后的作者信息: {knowledge.data.author}")+ a_document_model.extract(additional_text) # 默认 store=True,自动合并+ print(f" 更新后的作者信息: {a_document_model.data.author}")# 10. 保存知识print("\n[10] 保存知识到文件...")- save_path = Path(__file__).parent.parent / "tmp" / "unit_knowledge_multi_chunk"- knowledge.dump(str(save_path))+ save_path = Path(__file__).parent.parent / "tmp" / "auto_model_demo"+ a_document_model.dump(str(save_path))print(f" 已保存到: {save_path}")# 11. 重新加载测试print("\n[11] 测试知识加载...")- new_knowledge = UnitKnowledge(- data_schema=DocumentKnowledge,+ a_loaded_document_model = AutoModel(+ data_schema=DocumentSchema,llm_client=llm,embedder=embedder,)- new_knowledge.load(str(save_path))- print(f" 加载成功! 字段数: {len(new_knowledge)}")- print(f" 标题: {new_knowledge.data.title}")+ a_loaded_document_model.load(str(save_path))+ print(f" 加载成功! 字段数: {len(a_loaded_document_model)}")+ print(f" 标题: {a_loaded_document_model.data.title}")print("\n" + "=" * 80)print("示例运行完成!")diff --git a/examples/set_knowledge_demo.py b/examples/auto_set_demo.pysimilarity index 89%rename from examples/set_knowledge_demo.pyrename to examples/auto_set_demo.pyindex b372453..05d3252 100644--- a/examples/set_knowledge_demo.py+++ b/examples/auto_set_demo.py@@ -1,6 +1,6 @@"""Set Knowledge Demo - Extracting unique skills from multiple job descriptions-This example demonstrates how SetKnowledge automatically deduplicates+This example demonstrates how AutoSet automatically deduplicatesextracted items based on a unique key field."""@@ -15,7 +15,7 @@ import osfrom pydantic import BaseModel, Fieldfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic.set import SetKnowledge+from hyperextract import AutoSetfrom hyperextract.utils.merger import MergeStrategy@@ -44,10 +44,10 @@ def main():llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)embedder = OpenAIEmbeddings(base_url=os.getenv("OPENAI_BASE_URL"))- # Create SetKnowledge with FIELD_MERGE strategy+ # Create AutoSet with FIELD_MERGE strategy# This means when the same skill appears multiple times,# the new extraction will fill in any missing fields from the existing one- skills = SetKnowledge(+ a_skill_set = AutoSet(item_schema=SkillSchema,llm_client=llm,embedder=embedder,@@ -117,17 +117,17 @@ Be specific and extract all mentioned skills.""",print("\n📄 Processing job descriptions...")for i, job_desc in enumerate(job_descriptions, 1):print(f"\n Processing job description {i}...")- skills.extract(job_desc)- print(f" ✓ Total unique skills so far: {len(skills)}")+ a_skill_set.extract(job_desc)+ print(f" ✓ Total unique skills so far: {len(a_skill_set)}")# Display all unique skillsprint("\n" + "=" * 80)- print(f"📊 EXTRACTED UNIQUE SKILLS: {len(skills)} total")+ print(f"📊 EXTRACTED UNIQUE SKILLS: {len(a_skill_set)} total")print("=" * 80)# Group by categoryskills_by_category = {}- for skill in skills.items:+ for skill in a_skill_set.items:category = skill.category or "Uncategorized"if category not in skills_by_category:skills_by_category[category] = []@@ -150,12 +150,12 @@ Be specific and extract all mentioned skills.""",# Build index for semantic searchprint("\nBuilding vector index for semantic search...")- skills.build_index()+ a_skill_set.build_index()print("✓ Index built successfully")# Search for related skillsprint("\n1. Semantic Search - Finding skills related to 'backend development':")- backend_skills = skills.search("backend development", top_k=5)+ backend_skills = a_skill_set.search("backend development", top_k=5)for skill in backend_skills:print(f" • {skill.name}")if skill.category:@@ -165,13 +165,13 @@ Be specific and extract all mentioned skills.""",print("\n2. Membership Testing:")test_skills = ["Python", "Java", "Docker", "Rust"]for skill_name in test_skills:- exists = skills.contains(skill_name)+ exists = a_skill_set.contains(skill_name)status = "✓" if exists else "✗"print(f" {status} {skill_name}: {'Found' if exists else 'Not found'}")# Get specific skill detailsprint("\n3. Get Specific Skill Details:")- python_skill = skills.get("Python")+ python_skill = a_skill_set.get("Python")if python_skill:print(f" Skill: {python_skill.name}")print(f" Category: {python_skill.category}")@@ -183,28 +183,28 @@ Be specific and extract all mentioned skills.""",print("💾 SAVING EXTRACTED KNOWLEDGE")print("=" * 80)- save_path = "tmp/set_knowledge_demo"- skills.dump(save_path)+ save_path = "tmp/auto_set_demo"+ a_skill_set.dump(save_path)print(f"✓ Saved to: {save_path}")print(" - state.json (structured data)")print(" - faiss_index/ (vector index for semantic search)")# Demonstrate loadingprint("\n📂 Loading saved knowledge...")- loaded_skills = SetKnowledge(+ a_loaded_skill_set = AutoSet(item_schema=SkillSchema,llm_client=llm,embedder=embedder,key_extractor=lambda x: x.name,)- loaded_skills.load(save_path)- print(f"✓ Loaded {len(loaded_skills)} unique skills")+ a_loaded_skill_set.load(save_path)+ print(f"✓ Loaded {len(a_loaded_skill_set)} unique skills")# Summaryprint("\n" + "=" * 80)print("📈 SUMMARY")print("=" * 80)- print(f" Total unique skills extracted: {len(skills)}")+ print(f" Total unique skills extracted: {len(a_skill_set)}")print(f" Job descriptions processed: {len(job_descriptions)}")print(" Deduplication strategy: FIELD_MERGE")print(" Key extractor: lambda x: x.name")diff --git a/examples/generic_knowledge_demo.py b/examples/generic_knowledge_demo.pydeleted file mode 100644index 1a6bb81..0000000--- a/examples/generic_knowledge_demo.py+++ /dev/null@@ -1,130 +0,0 @@-"""-UnitKnowledge 使用示例--演示如何使用 UnitKnowledge 类进行知识提取、搜索、保存和加载。-"""--import sys-from pathlib import Path--# 添加项目根目录到 Python 路径-project_root = Path(__file__).parent.parent-sys.path.insert(0, str(project_root))--from pydantic import BaseModel, Field-from typing import List-from langchain_openai import ChatOpenAI, OpenAIEmbeddings-from hyperextract.knowledge.generic import UnitKnowledge---# 1. 定义知识 Schema-class ArticleKnowledge(BaseModel):- """文章知识结构"""-- title: str = Field(default="", description="文章标题")- author: str = Field(default="", description="作者")- summary: str = Field(default="", description="摘要")- key_points: List[str] = Field(default_factory=list, description="关键要点")- topics: List[str] = Field(default_factory=list, description="主题标签")---def main():- print("=" * 60)- print("UnitKnowledge 使用示例")- print("=" * 60)-- # 2. 初始化 LLM 和 Embedder- print("\n[1] 初始化 LLM 和 Embedder...")- llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)- embedder = OpenAIEmbeddings(model="text-embedding-3-small")-- # 3. 创建 UnitKnowledge 实例- print("[2] 创建 UnitKnowledge 实例...")- knowledge = UnitKnowledge(- data_schema=ArticleKnowledge,- llm_client=llm,- embedder=embedder,- prompt="", # 使用默认 prompt- chunk_size=2000,- max_workers=5,- show_progress=True,- )-- # 4. 提取知识- print("\n[3] 从文本中提取知识...")- sample_text = """- Title: The Future of Artificial Intelligence- Author: Dr. Jane Smith-- Artificial intelligence (AI) is rapidly transforming our world. This groundbreaking- technology has the potential to revolutionize industries ranging from healthcare to- transportation. In this article, we explore the key developments and future directions- of AI research.-- Key Points:- 1. Machine learning algorithms are becoming increasingly sophisticated- 2. AI is being applied to solve complex real-world problems- 3. Ethical considerations are crucial in AI development- 4. Collaboration between humans and AI will define the future-- Topics: artificial intelligence, machine learning, ethics, future technology- """-- result = knowledge.extract(sample_text)- print(f"✓ 提取完成: {result}")-- # 5. 查看提取的数据- print("\n[4] 查看提取的知识数据:")- extracted_data = knowledge.data- print(f" 标题: {extracted_data.title}")- print(f" 作者: {extracted_data.author}")- print(f" 摘要: {extracted_data.summary[:100]}...")- print(f" 关键要点数量: {len(extracted_data.key_points)}")- print(f" 主题标签: {', '.join(extracted_data.topics)}")-- # 6. 构建索引- print("\n[5] 构建向量索引...")- knowledge.build_index()- print(f"✓ 索引构建完成,共 {len(knowledge)} 个字段")-- # 7. 语义搜索- print("\n[6] 执行语义搜索...")- search_query = "What are the main topics of this article?"- results = knowledge.search(search_query, top_k=3)- print(f" 查询: {search_query}")- print(f" 搜索结果数量: {len(results)}")- for i, result in enumerate(results, 1):- print(f" 结果 {i}: {result}")-- # 8. 保存知识- print("\n[7] 保存知识到文件夹...")- save_path = Path(__file__).parent.parent / "tmp" / "article_knowledge"- knowledge.dump(save_path)- print(f"✓ 知识已保存到: {save_path}")-- # 9. 加载知识- print("\n[8] 从文件夹加载知识...")- new_knowledge = UnitKnowledge(- data_schema=ArticleKnowledge,- llm_client=llm,- embedder=embedder,- )- new_knowledge.load(save_path)- print("✓ 知识加载完成")- print(f" 加载的标题: {new_knowledge.data.title}")- print(f" 加载的字段数: {len(new_knowledge)}")-- # 10. 测试长文本提取(分块)- print("\n[9] 测试长文本提取(自动分块)...")- long_text = sample_text * 5 # 重复文本模拟长文本- knowledge.clear()- result = knowledge.extract(long_text)- print(f"✓ 长文本提取完成: {result}")-- print("\n" + "=" * 60)- print("示例执行完成!")- print("=" * 60)---if __name__ == "__main__":- main()diff --git a/hyperextract/__init__.py b/hyperextract/__init__.pyindex e69de29..032ad1f 100644--- a/hyperextract/__init__.py+++ b/hyperextract/__init__.py@@ -0,0 +1,15 @@+"""HyperExtract - AI-powered data structure extraction from unstructured text.++This library provides Auto-prefixed intelligent data structures that automatically+extract structured information from text using Large Language Models (LLMs).+"""++from .core import AutoModel, AutoList, AutoSet++__version__ = "0.2.0"++__all__ = [+ "AutoModel",+ "AutoList",+ "AutoSet",+]diff --git a/hyperextract/core/__init__.py b/hyperextract/core/__init__.pynew file mode 100644index 0000000..3c763a7--- /dev/null+++ b/hyperextract/core/__init__.py@@ -0,0 +1,13 @@+"""Core knowledge patterns - fundamental data structure patterns."""++from .base import BaseAutoType+from .model import AutoModel+from .list import AutoList+from .set import AutoSet++__all__ = [+ "BaseAutoType",+ "AutoModel",+ "AutoList",+ "AutoSet",+]diff --git a/hyperextract/knowledge/base.py b/hyperextract/core/base.pysimilarity index 98%rename from hyperextract/knowledge/base.pyrename to hyperextract/core/base.pyindex a0fbf20..ba1fdc4 100644--- a/hyperextract/knowledge/base.py+++ b/hyperextract/core/base.py@@ -15,7 +15,7 @@ T = TypeVar("T", bound=BaseModel)# ===================== Knowledge Base Class =====================-class BaseKnowledge(ABC, Generic[T]):+class BaseAutoType(ABC, Generic[T]):"""Unified knowledge base class integrating extraction, storage, aggregation, and evolution.This abstract base class provides a complete framework for managing structured knowledge@@ -87,7 +87,7 @@ class BaseKnowledge(ABC, Generic[T]):"updated_at": datetime.now(),}- def _create_new_instance(self) -> "BaseKnowledge[T]":+ def _create_new_instance(self) -> "BaseAutoType[T]":"""Creates a new instance with the same configuration as this one.Subclasses can override this method if they have special initialization requirements.@@ -278,7 +278,7 @@ class BaseKnowledge(ABC, Generic[T]):# ==================== Operator Overloads ====================- def __add__(self, other: "BaseKnowledge[T]") -> "BaseKnowledge[T]":+ def __add__(self, other: "BaseAutoType[T]") -> "BaseAutoType[T]":"""Operator overload for '+' to merge two knowledge instances.Creates a new knowledge instance by merging the data from both instances.diff --git a/hyperextract/knowledge/generic/list.py b/hyperextract/core/list.pysimilarity index 91%rename from hyperextract/knowledge/generic/list.pyrename to hyperextract/core/list.pyindex 852345d..0c43039 100644--- a/hyperextract/knowledge/generic/list.py+++ b/hyperextract/core/list.py@@ -10,7 +10,7 @@ from langchain_core.embeddings import Embeddingsfrom langchain_core.documents import Documentfrom langchain_community.vectorstores import FAISS-from ..base import BaseKnowledge+from .base import BaseAutoTypetry:from hyperextract.config import logger@@ -23,14 +23,14 @@ except ImportError:Item = TypeVar("Item", bound=BaseModel)-class ItemListSchema(BaseModel, Generic[Item]):+class AutoListSchema(BaseModel, Generic[Item]):"""Generic schema container for list-based knowledge patterns."""items: List[Item] = Field(default_factory=list, description="Item list")-class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):- """List Knowledge Pattern - extracts a collection of objects from text.+class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):+ """AutoList - extracts a collection of objects from text.This pattern extracts multiple independent objects from a document, suitable forextracting entities, events, references, or any collection of structured items.@@ -40,14 +40,14 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):- Merge strategy: Append with basic deduplication (extensible by subclasses)- Indexing strategy: Each item in the list is indexed independently- Comparison with UnitKnowledge:- - UnitKnowledge: Extracts a single structured object (e.g., summary, metadata)- - ListKnowledge: Extracts multiple independent objects (e.g., entity list, event list)+ Comparison with AutoModel:+ - AutoModel: Extracts a single structured object (e.g., summary, metadata)+ - AutoList: Extracts multiple independent objects (e.g., entity list, event list)"""if TYPE_CHECKING:# Use generic version during type checking to maintain complete type hints- item_list_schema: Type[ItemListSchema[Item]]+ item_list_schema: Type[AutoListSchema[Item]]def __init__(self,@@ -62,7 +62,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):show_progress: bool = True,**kwargs,):- """Initialize ListKnowledge with item schema and configuration.+ """Initialize AutoList with item schema and configuration.Args:item_schema: Pydantic BaseModel subclass for individual list items.@@ -113,13 +113,13 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):"""Returns the internal list of extracted items."""return getattr(self._data, "items", [])- def _create_new_instance(self) -> "ListKnowledge[Item]":+ def _create_new_instance(self) -> "AutoList[Item]":"""Creates a new instance with the same configuration as this one.- Overrides base class method to handle ListKnowledge's item_schema parameter.+ Overrides base class method to handle AutoList's item_schema parameter.Returns:- A new ListKnowledge instance with the same configuration.+ A new AutoList instance with the same configuration."""return self.__class__(item_schema=self.item_schema,@@ -132,7 +132,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):show_progress=self.show_progress,)- def extract(self, text: str, *, store: bool = True) -> ItemListSchema:+ def extract(self, text: str, *, store: bool = True) -> AutoListSchema:"""Extracts a list of items using LangChain native batch processing.Args:@@ -192,18 +192,18 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):# Return items list (either from stored data or temporary merged data)return self.items if store else merged_data.items- def merge(self, data_list: List[ItemListSchema]) -> ItemListSchema:+ def merge(self, data_list: List[AutoListSchema]) -> AutoListSchema:"""Pure data merge method implementing list append strategy.Merge strategy: Collects all items from all container objects and merges theminto a single list. Subclasses can override this method to implement more- sophisticated deduplication logic (e.g., EntityKnowledge).+ sophisticated deduplication logic (e.g., AutoSet with custom key_extractor).Args:data_list: List of container objects to merge.Returns:- A new merged ItemListSchema object.+ A new merged AutoListSchema object."""all_items = []@@ -366,7 +366,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):Returns:- For integer index: Returns the Item at that position- - For slice: Returns a new ListKnowledge instance with sliced items+ - For slice: Returns a new AutoList instance with sliced itemsRaises:IndexError: If index is out of range.@@ -375,14 +375,14 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):Examples:>>> knowledge[0] # First item>>> knowledge[-1] # Last item- >>> knowledge[1:3] # New ListKnowledge with items [1:3]+ >>> knowledge[1:3] # New AutoList with items [1:3]>>> knowledge[:5] # First 5 items as new instance"""if isinstance(key, int):# Integer index: return Item directlyreturn self.items[key]elif isinstance(key, slice):- # Slice: return new ListKnowledge instance+ # Slice: return new AutoList instancenew_instance = self._create_new_instance()new_instance._data.items = self.items[key]new_instance.metadata["updated_at"] = datetime.now()@@ -420,28 +420,28 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):"""Operator overload for '+' to combine knowledge instances.Supports multiple combination patterns:- - ListKnowledge + ListKnowledge → ListKnowledge (merge lists)- - ListKnowledge + UnitKnowledge → ListKnowledge (append unit to list)+ - AutoList + AutoList → AutoList (merge lists)+ - AutoList + UnitKnowledge → AutoList (append unit to list)This enables chain operations like: unit1 + unit2 + unit3Args:- other: Another ListKnowledge or UnitKnowledge with compatible schema.+ other: Another AutoList or UnitKnowledge with compatible schema.Returns:- New ListKnowledge with combined items.+ New AutoList with combined items.Raises:TypeError: If schemas don't match or invalid operand type."""- from .unit import UnitKnowledge+ from .model import AutoModel- # Case 1: ListKnowledge + ListKnowledge (call parent implementation)- if isinstance(other, ListKnowledge):+ # Case 1: AutoList + AutoList (call parent implementation)+ if isinstance(other, AutoList):# Check schema compatibilityif self.item_schema != other.item_schema:raise TypeError(- f"Cannot add ListKnowledge instances with different schemas. "+ f"Cannot add AutoList instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")# Manually merge without calling base class (to avoid _data_schema check)@@ -452,16 +452,16 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):new_instance.metadata["updated_at"] = self.metadata.get("updated_at")return new_instance- # Case 2: ListKnowledge + UnitKnowledge → ListKnowledge (append unit)- elif isinstance(other, UnitKnowledge):+ # Case 2: AutoList + AutoModel → AutoList (append unit)+ elif isinstance(other, AutoModel):# Check schema compatibilityif self.item_schema != other._data_schema:raise TypeError(- f"Cannot add UnitKnowledge to ListKnowledge with different schemas. "+ f"Cannot add AutoModel to AutoList with different schemas. "f"List: {self.item_schema.__name__}, Unit: {other._data_schema.__name__}")- # Create new ListKnowledge with appended unit+ # Create new AutoList with appended unitnew_list = self._create_new_instance()new_list._data.items = self.items + [other._data]@@ -475,7 +475,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):else:raise TypeError(- f"Unsupported operand type for +: 'ListKnowledge' and '{type(other).__name__}'"+ f"Unsupported operand type for +: 'AutoList' and '{type(other).__name__}'")def __delitem__(self, index: int):@@ -544,7 +544,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):Examples:>>> repr(knowledge)- 'ListKnowledge[PersonSchema](5 items)'+ 'AutoList[PersonSchema](5 items)'"""return (f"{self.__class__.__name__}[{self.item_schema.__name__}]({len(self)} items)"@@ -558,7 +558,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):Examples:>>> str(knowledge)- 'ListKnowledge with 5 PersonSchema items'+ 'AutoList with 5 PersonSchema items'"""return f"{self.__class__.__name__} with {len(self)} {self.item_schema.__name__} items"@@ -591,7 +591,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):Args:items: Iterable of items to append. Can be:- List of items- - Another ListKnowledge instance+ - Another AutoList instance- Any iterable yielding itemsRaises:@@ -605,11 +605,11 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):- Clears the vector index (needs rebuild)- Updates metadata timestamp"""- # Handle ListKnowledge instance- if isinstance(items, ListKnowledge):+ # Handle AutoList instance+ if isinstance(items, AutoList):if self.item_schema != items.item_schema:raise TypeError(- f"Cannot extend with ListKnowledge of different schema. "+ f"Cannot extend with AutoList of different schema. "f"Expected: {self.item_schema.__name__}, "f"Got: {items.item_schema.__name__}")@@ -757,11 +757,11 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):count += 1return count- def copy(self) -> "ListKnowledge[Item]":- """Create a deep copy of this ListKnowledge instance.+ def copy(self) -> "AutoList[Item]":+ """Create a deep copy of this AutoList instance.Returns:- A new ListKnowledge instance with copied items and metadata.+ A new AutoList instance with copied items and metadata.Note:The vector index is not copied; it needs to be rebuilt if needed.diff --git a/hyperextract/knowledge/generic/unit.py b/hyperextract/core/model.pysimilarity index 92%rename from hyperextract/knowledge/generic/unit.pyrename to hyperextract/core/model.pyindex cf613c7..da7b40e 100644--- a/hyperextract/knowledge/generic/unit.py+++ b/hyperextract/core/model.py@@ -9,7 +9,7 @@ from langchain_core.embeddings import Embeddingsfrom langchain_core.documents import Documentfrom langchain_community.vectorstores import FAISS-from ..base import BaseKnowledge, T+from .base import BaseAutoType, Ttry:from hyperextract.config import logger@@ -19,8 +19,8 @@ except ImportError:logger = logging.getLogger(__name__)-class UnitKnowledge(BaseKnowledge[T]):- """Unit Knowledge Pattern - extracts a single structured object from text.+class AutoModel(BaseAutoType[T]):+ """AutoModel - extracts a single structured object from text.This pattern is designed for extracting exactly one structured object per document,regardless of document length. Suitable for document-level information like summaries,@@ -47,7 +47,7 @@ class UnitKnowledge(BaseKnowledge[T]):show_progress: bool = True,**kwargs,):- """Initialize UnitKnowledge with schema and configuration.+ """Initialize AutoModel with schema and configuration.Args:data_schema: Pydantic BaseModel subclass defining the object structure.@@ -327,22 +327,22 @@ class UnitKnowledge(BaseKnowledge[T]):# ==================== Operators ====================def __add__(self, other):- """Operator overload for '+' to combine UnitKnowledge instances into ListKnowledge.+ """Operator overload for '+' to combine AutoModel instances into ListKnowledge.- When two UnitKnowledge instances are added, they are combined into a ListKnowledge+ When two AutoModel instances are added, they are combined into a ListKnowledgecontaining both objects as separate items. This enables intuitive collection building.Usage:- >>> unit1 = UnitKnowledge(PersonSchema, ...)- >>> unit2 = UnitKnowledge(PersonSchema, ...)+ >>> unit1 = AutoModel(PersonSchema, ...)+ >>> unit2 = AutoModel(PersonSchema, ...)>>> person_list = unit1 + unit2 # → ListKnowledge[PersonSchema]>>>>>> # Chain operations- >>> unit3 = UnitKnowledge(PersonSchema, ...)+ >>> unit3 = AutoModel(PersonSchema, ...)>>> person_list = unit1 + unit2 + unit3 # → ListKnowledge with 3 itemsArgs:- other: Another UnitKnowledge with the same data schema, or ListKnowledge.+ other: Another AutoModel with the same data schema, or ListKnowledge.Returns:ListKnowledge containing both objects as items.@@ -352,12 +352,12 @@ class UnitKnowledge(BaseKnowledge[T]):"""from .list import ListKnowledge- # Case 1: UnitKnowledge + UnitKnowledge → ListKnowledge- if isinstance(other, UnitKnowledge):+ # Case 1: AutoModel + AutoModel → ListKnowledge+ if isinstance(other, AutoModel):# Check schema compatibilityif self._data_schema != other._data_schema:raise TypeError(- f"Cannot add UnitKnowledge instances with different schemas. "+ f"Cannot add AutoModel instances with different schemas. "f"Left: {self._data_schema.__name__}, Right: {other._data_schema.__name__}")@@ -384,12 +384,12 @@ class UnitKnowledge(BaseKnowledge[T]):return list_kb- # Case 2: UnitKnowledge + ListKnowledge → ListKnowledge (for reverse order)+ # Case 2: AutoModel + ListKnowledge → ListKnowledge (for reverse order)elif isinstance(other, ListKnowledge):# Check schema compatibilityif self._data_schema != other.item_schema:raise TypeError(- f"Cannot add UnitKnowledge to ListKnowledge with different schemas. "+ f"Cannot add AutoModel to ListKnowledge with different schemas. "f"Unit: {self._data_schema.__name__}, List: {other.item_schema.__name__}")@@ -418,5 +418,5 @@ class UnitKnowledge(BaseKnowledge[T]):else:raise TypeError(- f"Unsupported operand type for +: 'UnitKnowledge' and '{type(other).__name__}'"+ f"Unsupported operand type for +: 'AutoModel' and '{type(other).__name__}'")diff --git a/hyperextract/knowledge/generic/set.py b/hyperextract/core/set.pysimilarity index 86%rename from hyperextract/knowledge/generic/set.pyrename to hyperextract/core/set.pyindex e551691..db9198f 100644--- a/hyperextract/knowledge/generic/set.py+++ b/hyperextract/core/set.py@@ -13,7 +13,7 @@ from langchain_core.language_models.chat_models import BaseChatModelfrom langchain_core.embeddings import Embeddingsfrom langchain_core.documents import Document-from ..base import BaseKnowledge+from .base import BaseAutoTypefrom hyperextract.utils.merger import (BaseMerger,MergeStrategy,@@ -34,14 +34,14 @@ except ImportError:Item = TypeVar("Item", bound=BaseModel)-class ItemSetSchema(BaseModel, Generic[Item]):+class AutoSetSchema(BaseModel, Generic[Item]):"""Generic schema container for set-based knowledge patterns."""items: List[Item] = Field(default_factory=list, description="Set of unique items")-class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):- """Set Knowledge Pattern - extracts a unique collection of objects.+class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):+ """AutoSet - extracts a unique collection of objects.This pattern automatically deduplicates items based on a user-specifiedkey extractor function. Provides flexible merge strategies including LLM-powered@@ -55,9 +55,9 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):- External interface: List (via items property)- Set operations: union (|), intersection (&), difference (-)- Comparison with ListKnowledge:- - ListKnowledge: Allows duplicates, simple append merge- - SetKnowledge: Automatic deduplication, intelligent merge strategies+ Comparison with AutoList:+ - AutoList: Allows duplicates, simple append merge+ - AutoSet: Automatic deduplication, intelligent merge strategiesExample:>>> class KeywordSchema(BaseModel):@@ -65,7 +65,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):... category: str | None = None... frequency: int | None = None>>>- >>> keywords = SetKnowledge(+ >>> keywords = AutoSet(... item_schema=KeywordSchema,... llm_client=llm,... embedder=embedder,@@ -93,7 +93,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):llm_batch_size: int = 10,**kwargs,):- """Initialize SetKnowledge with key extractor and merge strategy.+ """Initialize AutoSet with key extractor and merge strategy.Args:item_schema: Pydantic BaseModel subclass for individual items.@@ -113,7 +113,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):# Store item_schemaself.item_schema = item_schema- # Create ItemSetSchema container dynamically (similar to ListKnowledge's ItemListSchema)+ # Create AutoSetSchema container dynamically (similar to AutoList's AutoListSchema)container_name = f"{item_schema.__name__}Set"self.item_set_schema = create_model(container_name,@@ -123,7 +123,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):),)- # Call BaseKnowledge.__init__ (not ListKnowledge)+ # Call BaseAutoType.__init__ (not ListKnowledge)super().__init__(data_schema=self.item_set_schema,llm_client=llm_client,@@ -136,7 +136,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):**kwargs,)- # SetKnowledge-specific attributes+ # AutoSet-specific attributesself.key_extractor = key_extractorself.merge_item_strategy = merge_item_strategyself.llm_batch_size = llm_batch_size@@ -153,13 +153,13 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):# ==================== Override Instance Creation ====================- def _create_new_instance(self) -> "SetKnowledge[Item]":+ def _create_new_instance(self) -> "AutoSet[Item]":"""Creates a new instance with the same configuration.- Overrides parent method to include SetKnowledge-specific parameters.+ Overrides parent method to include AutoSet-specific parameters.Returns:- New SetKnowledge instance.+ New AutoSet instance."""return self.__class__(item_schema=self.item_schema,@@ -270,11 +270,11 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):def __repr__(self) -> str:"""Returns a developer-friendly representation."""- return f"SetKnowledge[{self.item_schema.__name__}]({len(self)} unique items)"+ return f"AutoSet[{self.item_schema.__name__}]({len(self)} unique items)"def __str__(self) -> str:"""Returns a user-friendly string representation."""- return f"SetKnowledge with {len(self)} unique {self.item_schema.__name__} items"+ return f"AutoSet with {len(self)} unique {self.item_schema.__name__} items"def __iter__(self):"""Enables iteration over all items in the set.@@ -354,7 +354,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):# Return items listreturn self.items if store else merged_data.items- def merge(self, data_list: List[ItemSetSchema[Item]]) -> ItemSetSchema[Item]:+ def merge(self, data_list: List[AutoSetSchema[Item]]) -> AutoSetSchema[Item]:"""Merges multiple data containers with automatic deduplication.Pure function: Does not modify internal state.@@ -366,7 +366,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):data_list: List of container objects to merge.Returns:- New merged ItemSetSchema with deduplicated items.+ New merged AutoSetSchema with deduplicated items."""# Step 1: Collect all items from all containersall_items = []@@ -391,7 +391,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):# Step 3: Return new container (does not modify self)return self.item_set_schema(items=merged_items)- def _update_internal_state(self, data: ItemSetSchema[Item]) -> None:+ def _update_internal_state(self, data: AutoSetSchema[Item]) -> None:"""Updates internal state from a data container.Private method that synchronizes _items_dict and _data with new data.@@ -508,7 +508,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):def dump(self, folder_path: str | Path):"""Exports knowledge to a specified folder.- Saves both the items and the SetKnowledge-specific metadata.+ Saves both the items and the AutoSet-specific metadata.Args:folder_path: Target folder path for saving.@@ -706,7 +706,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):- Updates metadata timestamp"""if not self._items_dict:- raise KeyError("pop from an empty SetKnowledge")+ raise KeyError("pop from an empty AutoSet")key = next(iter(self._items_dict))item = self._items_dict.pop(key)@@ -716,11 +716,11 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):return item- def copy(self) -> "SetKnowledge[Item]":+ def copy(self) -> "AutoSet[Item]":"""Creates a deep copy of the set.Returns:- A new SetKnowledge instance with copies of all items.+ A new AutoSet instance with copies of all items.Examples:>>> backup = skills.copy()@@ -739,25 +739,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):# ==================== Set Operations ====================- def __or__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def __or__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Union operation: set1 | set2.Returns a new set containing all items from both sets.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:- New SetKnowledge with union of items.+ New AutoSet with union of items."""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):raise TypeError(- f"Unsupported operand type for |: 'SetKnowledge' and '{type(other).__name__}'"+ f"Unsupported operand type for |: 'AutoSet' and '{type(other).__name__}'")if self.item_schema != other.item_schema:raise TypeError(- f"Cannot union SetKnowledge instances with different schemas. "+ f"Cannot union AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")@@ -779,25 +779,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):return new_set- def __and__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def __and__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Intersection operation: set1 & set2.Returns a new set containing only items present in both sets.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:- New SetKnowledge with intersection of items.+ New AutoSet with intersection of items."""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):raise TypeError(- f"Unsupported operand type for &: 'SetKnowledge' and '{type(other).__name__}'"+ f"Unsupported operand type for &: 'AutoSet' and '{type(other).__name__}'")if self.item_schema != other.item_schema:raise TypeError(- f"Cannot intersect SetKnowledge instances with different schemas. "+ f"Cannot intersect AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")@@ -819,25 +819,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):return new_set- def __sub__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def __sub__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Difference operation: set1 - set2.Returns a new set containing items in self but not in other.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:- New SetKnowledge with difference of items.+ New AutoSet with difference of items."""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):raise TypeError(- f"Unsupported operand type for -: 'SetKnowledge' and '{type(other).__name__}'"+ f"Unsupported operand type for -: 'AutoSet' and '{type(other).__name__}'")if self.item_schema != other.item_schema:raise TypeError(- f"Cannot subtract SetKnowledge instances with different schemas. "+ f"Cannot subtract AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")@@ -854,25 +854,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):return new_set- def __xor__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def __xor__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Symmetric difference operation: set1 ^ set2.Returns a new set containing items in either set but not in both.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:- New SetKnowledge with symmetric difference of items.+ New AutoSet with symmetric difference of items."""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):raise TypeError(- f"Unsupported operand type for ^: 'SetKnowledge' and '{type(other).__name__}'"+ f"Unsupported operand type for ^: 'AutoSet' and '{type(other).__name__}'")if self.item_schema != other.item_schema:raise TypeError(- f"Cannot compute symmetric difference of SetKnowledge instances with different schemas. "+ f"Cannot compute symmetric difference of AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")@@ -896,19 +896,19 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):# ==================== Named Set Operations ====================- def union(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def union(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Union operation (named method)."""return self | other- def intersection(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def intersection(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Intersection operation (named method)."""return self & other- def difference(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def difference(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Difference operation (named method)."""return self - other- def symmetric_difference(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":+ def symmetric_difference(self, other: "AutoSet[Item]") -> "AutoSet[Item]":"""Symmetric difference operation (named method)."""return self ^ other@@ -929,7 +929,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):Examples:>>> skills1 == skills2 # True if same keys"""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):return Falseif self.item_schema != other.item_schema:@@ -945,37 +945,37 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):"""return not self.__eq__(other)- def __le__(self, other: "SetKnowledge[Item]") -> bool:+ def __le__(self, other: "AutoSet[Item]") -> bool:"""Subset comparison: set1 <= set2.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if self is a subset of other (all keys in self are in other).Raises:- TypeError: If other is not a SetKnowledge or has different schema.+ TypeError: If other is not a AutoSet or has different schema.Examples:>>> skills1 <= skills2 # True if skills1 is subset of skills2"""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):return NotImplementedif self.item_schema != other.item_schema:raise TypeError(- f"Cannot compare SetKnowledge instances with different schemas. "+ f"Cannot compare AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")return set(self._items_dict.keys()).issubset(set(other._items_dict.keys()))- def __lt__(self, other: "SetKnowledge[Item]") -> bool:+ def __lt__(self, other: "AutoSet[Item]") -> bool:"""Proper subset comparison: set1 < set2.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if self is a proper subset of other (subset and not equal).@@ -983,16 +983,16 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):Examples:>>> skills1 < skills2 # True if skills1 is proper subset"""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):return NotImplementedreturn self <= other and self != other- def __ge__(self, other: "SetKnowledge[Item]") -> bool:+ def __ge__(self, other: "AutoSet[Item]") -> bool:"""Superset comparison: set1 >= set2.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if self is a superset of other (all keys in other are in self).@@ -1000,22 +1000,22 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):Examples:>>> skills1 >= skills2 # True if skills1 is superset of skills2"""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):return NotImplementedif self.item_schema != other.item_schema:raise TypeError(- f"Cannot compare SetKnowledge instances with different schemas. "+ f"Cannot compare AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")return set(self._items_dict.keys()).issuperset(set(other._items_dict.keys()))- def __gt__(self, other: "SetKnowledge[Item]") -> bool:+ def __gt__(self, other: "AutoSet[Item]") -> bool:"""Proper superset comparison: set1 > set2.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if self is a proper superset of other (superset and not equal).@@ -1023,16 +1023,16 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):Examples:>>> skills1 > skills2 # True if skills1 is proper superset"""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):return NotImplementedreturn self >= other and self != other- def issubset(self, other: "SetKnowledge[Item]") -> bool:+ def issubset(self, other: "AutoSet[Item]") -> bool:"""Test whether every key in the set is in other.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if self is a subset of other.@@ -1042,11 +1042,11 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):"""return self <= other- def issuperset(self, other: "SetKnowledge[Item]") -> bool:+ def issuperset(self, other: "AutoSet[Item]") -> bool:"""Test whether every key in other is in the set.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if self is a superset of other.@@ -1056,29 +1056,29 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):"""return self >= other- def isdisjoint(self, other: "SetKnowledge[Item]") -> bool:+ def isdisjoint(self, other: "AutoSet[Item]") -> bool:"""Test whether the set has no keys in common with other.Args:- other: Another SetKnowledge instance.+ other: Another AutoSet instance.Returns:True if the two sets have no keys in common.Raises:- TypeError: If other is not a SetKnowledge or has different schema.+ TypeError: If other is not a AutoSet or has different schema.Examples:>>> skills1.isdisjoint(skills2) # True if no common skills"""- if not isinstance(other, SetKnowledge):+ if not isinstance(other, AutoSet):raise TypeError(- f"isdisjoint() argument must be SetKnowledge, not {type(other).__name__}"+ f"isdisjoint() argument must be AutoSet, not {type(other).__name__}")if self.item_schema != other.item_schema:raise TypeError(- f"Cannot compare SetKnowledge instances with different schemas. "+ f"Cannot compare AutoSet instances with different schemas. "f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}")diff --git a/hyperextract/knowledge/graph/__init__.py b/hyperextract/graphs/__init__.pysimilarity index 100%rename from hyperextract/knowledge/graph/__init__.pyrename to hyperextract/graphs/__init__.pydiff --git a/hyperextract/knowledge/graph/base.py b/hyperextract/graphs/base.pysimilarity index 96%rename from hyperextract/knowledge/graph/base.pyrename to hyperextract/graphs/base.pyindex 1a27d1e..74bbe2f 100644--- a/hyperextract/knowledge/graph/base.py+++ b/hyperextract/graphs/base.py@@ -1,4 +1,4 @@-from ..base import BaseKnowledge+from ..core.base import BaseAutoTypefrom typing import List, Optional, Dict, Anyfrom pydantic import BaseModel, Field@@ -43,7 +43,7 @@ class GraphSchema(BaseModel):return {n.id for n in self.nodes}-class SimpleGraph(BaseKnowledge[GraphSchema]):+class SimpleGraph(BaseAutoType[GraphSchema]):def _get_schema_class(self):return GraphSchemadiff --git a/hyperextract/knowledge/graph/kg_gen.py b/hyperextract/graphs/kg_gen.pysimilarity index 100%rename from hyperextract/knowledge/graph/kg_gen.pyrename to hyperextract/graphs/kg_gen.pydiff --git a/hyperextract/knowledge/hypergraph/__init__.py b/hyperextract/hypergraphs/__init__.pysimilarity index 100%rename from hyperextract/knowledge/hypergraph/__init__.pyrename to hyperextract/hypergraphs/__init__.pydiff --git a/hyperextract/knowledge/hypergraph/base.py b/hyperextract/hypergraphs/base.pysimilarity index 100%rename from hyperextract/knowledge/hypergraph/base.pyrename to hyperextract/hypergraphs/base.pydiff --git a/hyperextract/knowledge/hypergraph/hyperrag.py b/hyperextract/hypergraphs/hyperrag.pysimilarity index 100%rename from hyperextract/knowledge/hypergraph/hyperrag.pyrename to hyperextract/hypergraphs/hyperrag.pydiff --git a/hyperextract/knowledge/__init__.py b/hyperextract/knowledge/__init__.pydeleted file mode 100644index 518926d..0000000--- a/hyperextract/knowledge/__init__.py+++ /dev/null@@ -1,45 +0,0 @@-"""Knowledge extraction framework.--This package provides a hierarchical knowledge extraction framework:- - generic: Base knowledge patterns (UnitKnowledge, ListKnowledge, SetKnowledge)- - typical: Common knowledge types (EntityKnowledge, RelationKnowledge)- - domain: Domain-specific knowledge patterns (reserved for future use)- - graph: Graph-based knowledge structures- - hypergraph: Hypergraph-based knowledge structures--Usage:- All main classes can be imported directly from hyperextract.knowledge:-- >>> from hyperextract.knowledge import UnitKnowledge, ListKnowledge, EntityKnowledge- >>> from hyperextract.knowledge import SimpleGraph-"""--from .base import BaseKnowledge--# Generic patterns-from .generic import UnitKnowledge, ListKnowledge, ItemListSchema--# Typical patterns-from .typical import EntityKnowledge--# Graph patterns-from .graph.base import SimpleGraph, Entity, Relationship, GraphSchema--__all__ = [- # Base- "BaseKnowledge",-- # Generic patterns- "UnitKnowledge",- "ListKnowledge",- "ItemListSchema",-- # Typical patterns- "EntityKnowledge",-- # Graph patterns- "SimpleGraph",- "Entity",- "Relationship",- "GraphSchema",-]diff --git a/hyperextract/knowledge/domain/__init__.py b/hyperextract/knowledge/domain/__init__.pydeleted file mode 100644index b3e6ce6..0000000--- a/hyperextract/knowledge/domain/__init__.py+++ /dev/null@@ -1,11 +0,0 @@-"""Domain-specific knowledge patterns.--This module is reserved for domain-specific knowledge implementations such as:- - DocumentKnowledge: Document-level knowledge (title, author, sections, etc.)- - ConversationKnowledge: Multi-turn conversation knowledge- - AcademicPaperKnowledge: Academic paper structure and metadata- - CodeKnowledge: Source code analysis and documentation- - etc.-"""--__all__ = []diff --git a/hyperextract/knowledge/generic/__init__.py b/hyperextract/knowledge/generic/__init__.pydeleted file mode 100644index 79fca3e..0000000--- a/hyperextract/knowledge/generic/__init__.py+++ /dev/null@@ -1,12 +0,0 @@-"""Generic knowledge patterns - fundamental data structure patterns."""--from .unit import UnitKnowledge-from .list import ListKnowledge, ItemListSchema-from .set import SetKnowledge--__all__ = [- "UnitKnowledge",- "ListKnowledge",- "ItemListSchema",- "SetKnowledge",-]diff --git a/hyperextract/knowledge/typical/__init__.py b/hyperextract/knowledge/typical/__init__.pydeleted file mode 100644index 2daeea9..0000000--- a/hyperextract/knowledge/typical/__init__.py+++ /dev/null@@ -1,7 +0,0 @@-"""Typical knowledge patterns - common knowledge representation types."""--from .entity import EntityKnowledge--__all__ = [- "EntityKnowledge",-]diff --git a/hyperextract/knowledge/typical/entity.py b/hyperextract/knowledge/typical/entity.pydeleted file mode 100644index 6189e06..0000000--- a/hyperextract/knowledge/typical/entity.py+++ /dev/null@@ -1,390 +0,0 @@-"""Entity knowledge extraction module.--Provides entity extraction implementation based on ListKnowledge.-"""--from typing import List, Dict, Type, TypeVar, Generic-from pydantic import BaseModel, Field as PydanticField-from langchain_core.language_models.chat_models import BaseChatModel-from langchain_core.embeddings import Embeddings-from langchain_core.documents import Document-from langchain_core.prompts import ChatPromptTemplate-from langchain_community.vectorstores import FAISS--from ..generic import ListKnowledge--try:- from hyperextract.config import logger-except ImportError:- import logging-- logger = logging.getLogger(__name__)---# Entity type variable-E = TypeVar("E", bound="BaseEntitySchema")---class BaseEntitySchema(BaseModel):- """Base entity class defining required core fields for all entities.-- Users can inherit from this class to add custom fields.- """-- name: str = PydanticField(description="Unique entity name", min_length=1)- description: str = PydanticField(description="Entity description")-- class Config:- """Configuration allowing users to extend with additional fields."""-- extra = "allow" # Allow additional fields- validate_assignment = True # Validate on assignment---class EntityKnowledge(ListKnowledge[E], Generic[E]):- """Specialized knowledge class for entity extraction and management.-- Key features:- 1. Automatic deduplication based on entity.name- 2. Intelligent LLM-powered merging for entities with the same name- 3. Each entity indexed as an independent document- 4. Support for entity-level semantic search- """-- def __init__(- self,- entity_schema: Type[E],- llm_client: BaseChatModel,- embedder: Embeddings,- *,- prompt: str = "",- **kwargs,- ):- """Initialize EntityKnowledge with entity schema and configuration.-- Args:- entity_schema: User-defined entity class (must inherit from BaseEntity).- llm_client: Language model client for extraction and merging.- embedder: Embedding model for vector indexing.- prompt: Custom extraction prompt.- """- self._entity_schema = entity_schema-- super().__init__(- item_schema=entity_schema,- llm_client=llm_client,- embedder=embedder,- prompt=prompt or self._default_prompt(),- **kwargs,- )-- self._merge_chain = self._create_merge_chain()-- # Global entity mapping (primary storage, type-safe)- self._entity_map: Dict[str, E] = {}-- @staticmethod- def _default_prompt() -> str:- """Returns the default entity extraction prompt."""- return (- "You are an expert entity extraction assistant. "- "Carefully read the following text and extract all entities "- "according to the specified schema. Be precise and comprehensive."- "Extract all entities without adding information not present in the text.\n\n"- "### Source Text:\n"- )-- def _create_merge_chain(self):- """Creates an LLM chain for intelligently merging entities with the same name."""- merge_template = """You are an expert at merging entity information.-Given two instances of the same entity, intelligently merge their fields into one.--Rules:-1. Keep the 'name' field unchanged-2. Merge 'description' by combining both descriptions (remove duplicates)-3. For other fields: prefer non-null values, if both exist choose the more complete/accurate one--Entity A:-{entity_a}--Entity B:-{entity_b}--Please merge these two entities intelligently and return the result."""-- merge_prompt = ChatPromptTemplate.from_template(merge_template)-- return merge_prompt | self.llm_client.with_structured_output(- self._entity_schema- )-- # ==================== Data Management ====================-- def clear(self):- """Clears all entities from storage."""- super().clear()- self._entity_map.clear()-- # ==================== Extraction & Aggregation ====================-- def extract(self, text: str, *, store: bool = True) -> BaseModel:- """Extracts entity knowledge with support for store/temporary modes.-- Mode explanation:- - store=True (default): Store mode- Merges newly extracted entities with existing knowledge and updates internal state-- - store=False: Temporary mode- Returns extracted entities without modifying internal state-- Args:- text: Input text to extract entities from.- store: Whether to store extracted knowledge (default True).-- Returns:- Container with extracted entities.- """- # Call parent extraction method with store parameter- result = super().extract(text, store=store)-- # Only synchronize to entity_map if storing internally- if store:- self._sync_items_to_map()- logger.info(- "Entity extraction completed (Store mode)"- )- logger.info(f"Total entities: {len(self._entity_map)}")- else:- logger.info(- "Entity extraction completed (Temporary mode)"- )-- return result-- def merge(self, data_list: List[BaseModel]) -> BaseModel:- """Entity-level intelligent merging using LLM.-- Strategy:- 1. Call parent merge to get all extracted items- 2. Assign result to self._data- 3. Synchronize to entity_map for LLM-powered deduplication- 4. Return deduplicated data-- Args:- data_list: Multiple container objects.-- Returns:- Merged entity data.- """- # Call parent merge to get merged result- merged_result = super().merge(data_list)-- # Temporarily assign to self._data so _sync_items_to_map can access self.items- self._data = merged_result-- # Synchronize to entity_map for deduplication- self._sync_items_to_map()-- # Return final self._data (updated by _sync_items_to_map)- return self._data-- def _sync_items_to_map(self):- """Synchronizes self.items to entity_map, performing deduplication and LLM merging."""- total_entities = 0- duplicate_count = 0-- # Traverse parent's items and merge with existing entity_map- for entity in self.items:- total_entities += 1- entity_name = entity.name.strip()-- if not entity_name: # Skip empty names- logger.warning(f"Skipping entity with empty name: {entity}")- continue-- if entity_name in self._entity_map:- # Entity exists - merge using LLM- duplicate_count += 1- existing = self._entity_map[entity_name]- merged = self._merge_entity_with_llm(existing, entity)- if merged:- self._entity_map[entity_name] = merged- else:- logger.warning(- f"LLM merge failed for entity '{entity_name}', keeping existing"- )- else:- # New entity - deep copy and add- self._entity_map[entity_name] = entity.model_copy(deep=True)-- # Write deduplicated results back to self.items- self._data.items = list(self._entity_map.values())- self.clear_index()-- logger.info(- f"Merged {len(self._entity_map)} unique entities "- f"from {total_entities} total (removed {duplicate_count} duplicates)"- )-- def _merge_entity_with_llm(self, entity_a: E, entity_b: E) -> E | None:- """Uses LLM to intelligently merge two entities with the same name.-- Args:- entity_a: First entity.- entity_b: Second entity.-- Returns:- Merged entity (returns None on failure).- """- try:- # Prepare inputs- entity_a_json = entity_a.model_dump_json(indent=2)- entity_b_json = entity_b.model_dump_json(indent=2)-- # Call LLM for merging- merged_entity = self._merge_chain.invoke(- {- "entity_a": entity_a_json,- "entity_b": entity_b_json,- }- )-- logger.debug(f"Successfully merged entity: {merged_entity.name}")- return merged_entity-- except Exception as e:- logger.error(f"Error merging entity '{entity_a.name}' with LLM: {e}")- return None-- # ==================== Indexing & Query ====================-- def build_index(self):- """Builds independent vector index document for each entity.-- Each entity → one Document:- - page_content: "{name}: {description} + other fields"- - metadata: {"entity": entity.model_dump()}- """- if len(self._entity_map) == 0:- logger.warning("No entities to index")- return-- if self._index is not None:- return # Index already exists-- documents = []- for entity in self._entity_map.values():- # Build document content for vectorization- content_parts = [- f"name: {entity.name}",- f"description: {entity.description}",- ]-- # Add other fields if user extended the schema- for field_name in self._entity_schema.model_fields:- if field_name not in ("name", "description"):- value = getattr(entity, field_name, None)- if value is not None:- content_parts.append(f"{field_name}: {value}")-- content = "\n".join(content_parts)-- # Save complete entity data to metadata- documents.append(- Document(- page_content=content,- metadata={"raw": entity.model_dump()},- )- )-- self._index = FAISS.from_documents(documents, self.embedder)- logger.info(f"Built FAISS index with {len(documents)} entities")--- def search(self, query: str, top_k: int = 3, return_raw: bool = False) -> List[E]:- """Performs semantic search over entities.-- Args:- query: Search query string.- top_k: Number of results to return.- return_raw: Whether to return raw entity objects (default returns dict).-- Returns:- List of entities.- """- if len(self) == 0:- logger.warning("No items to search")- return []-- if self._index is None:- raise Exception("Index is not built, please build the index first.")-- docs = self._index.similarity_search(query, k=top_k)-- # Restore entity objects- results = []- for doc in docs:- try:- entity_data = doc.metadata["raw"]- if return_raw:- # Return Pydantic object- entity = self._entity_schema.model_validate(entity_data)- results.append(entity)- else:- # Return dictionary- results.append(entity_data)- except Exception as e:- logger.warning(f"Failed to restore entity: {e}")-- logger.info(f"Found {len(results)} entities for query: {query[:50]}...")- return results-- def __len__(self) -> int:- """Returns the number of entities (reads directly from entity_map)."""- return len(self._entity_map)-- # ==================== Convenience Properties ====================-- @property- def entities(self) -> List[E]:- """Quick access to entity list."""- return self.items-- @property- def entity_schema(self) -> Type[E]:- """Returns the user-defined entity schema."""- return self._entity_schema-- @property- def entity_names(self) -> List[str]:- """Returns all entity names (reads directly from entity_map)."""- return list(self._entity_map.keys())-- # ==================== Optional Enhancement Methods ====================-- def get_entity_by_name(self, name: str) -> E | None:- """Retrieves entity by name (O(1) dictionary lookup)."""- return self._entity_map.get(name)-- def remove_entity(self, name: str) -> bool:- """Removes specified entity (from both entity_map and items)."""- if name in self._entity_map:- del self._entity_map[name]- self._data.items = list(self._entity_map.values())- self.clear_index()- logger.info(f"Removed entity: {name}")- return True- return False-- # ==================== Serialization ====================-- def load(self, folder_path: str, **kwargs):- """Loads knowledge and rebuilds entity_map."""- super().load(folder_path, **kwargs)-- # Rebuild entity_map from items- self._entity_map = {- entity.name: entity.model_copy(deep=True) for entity in self.items- }-- logger.info(f"Loaded {len(self._entity_map)} entities")diff --git a/hyperextract/knowledge/typical/tree.py b/hyperextract/knowledge/typical/tree.pydeleted file mode 100644index e69de29..0000000diff --git a/hyperextract/skill/__init__.py b/hyperextract/skill/__init__.pydeleted file mode 100644index e69de29..0000000diff --git a/hyperextract/trees/__init__.py b/hyperextract/trees/__init__.pynew file mode 100644index 0000000..b0ce327--- /dev/null+++ b/hyperextract/trees/__init__.py@@ -0,0 +1,10 @@+"""Tree structures for hierarchical data extraction.++This module provides tree-based Auto types for extracting hierarchical+and nested data structures from unstructured text.+"""++# Tree implementations will be added here as they are developed+# from .base import AutoTree++__all__ = []diff --git a/hyperextract/knowledge/typical/relation.py b/hyperextract/trees/base.pysimilarity index 100%rename from hyperextract/knowledge/typical/relation.pyrename to hyperextract/trees/base.pydiff --git a/mkdocs.yml b/mkdocs.ymlindex 5f74227..3af7b1a 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -95,9 +95,9 @@ nav:- Knowledge Patterns: user-guide/knowledge-patterns.md- API Reference:- Overview: api/index.md- - BaseKnowledge: api/base.md- - UnitKnowledge: api/unit.md- - ListKnowledge: api/list.md- - SetKnowledge: api/set.md+ - BaseAutoType: api/base.md+ - AutoModel: api/model.md+ - AutoList: api/auto_list.md+ - AutoSet: api/auto_set.md- 中文文档:- 首页: zh/index.mddiff --git a/tests/test_knowledge.py b/tests/test_knowledge.pyindex ad82fda..612069a 100644--- a/tests/test_knowledge.py+++ b/tests/test_knowledge.py@@ -1,5 +1,5 @@"""-测试 Knowledge 类的功能。+测试 BaseAutoType 类的功能。"""import pytestfrom typing import List, Optional@@ -7,7 +7,7 @@ from pydantic import BaseModel, Fieldfrom unittest.mock import MagicMock, AsyncMockimport json-from hyperextract.knowledge.base import Knowledge+from hyperextract.core import BaseAutoType# 定义测试用的 Schema@@ -41,9 +41,9 @@ def mock_embedder():@pytest.fixture-def article_knowledge(mock_llm, mock_embedder):- """创建一个 Article 类型的 Knowledge 实例"""- return Knowledge(+def a_article(mock_llm, mock_embedder):+ """创建一个 Article 类型的 BaseAutoType 实例"""+ return BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -52,9 +52,9 @@ def article_knowledge(mock_llm, mock_embedder):@pytest.fixture-def person_knowledge(mock_llm, mock_embedder):- """创建一个 Person 类型的 Knowledge 实例"""- return Knowledge(+def a_person(mock_llm, mock_embedder):+ """创建一个 Person 类型的 BaseAutoType 实例"""+ return BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Person,@@ -62,16 +62,16 @@ def person_knowledge(mock_llm, mock_embedder):)-class TestKnowledgeInitialization:- """测试 Knowledge 类的初始化"""+class TestBaseAutoTypeInitialization:+ """测试 BaseAutoType 类的初始化"""- def test_knowledge_init_with_article_schema(self, article_knowledge):+ def test_auto_type_init_with_article_schema(self, a_article):"""测试使用 Article schema 初始化"""- assert article_knowledge.schema_class == Article- assert article_knowledge.storage == "memory"- assert article_knowledge.data is not None+ assert a_article.schema_class == Article+ assert a_article.storage == "memory"+ assert a_article.data is not None- def test_knowledge_init_with_initial_data(self, mock_llm, mock_embedder):+ def test_auto_type_init_with_initial_data(self, mock_llm, mock_embedder):"""测试带初始数据的初始化"""initial_article = Article(title="Test Article",@@ -79,7 +79,7 @@ class TestKnowledgeInitialization:keywords=["test"],summary="A test article")- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -88,14 +88,14 @@ class TestKnowledgeInitialization:assert knowledge.data == initial_articleassert knowledge.data.title == "Test Article"- def test_knowledge_with_different_schemas(self, mock_llm, mock_embedder):- """测试使用不同 Schema 的 Knowledge"""- article_k = Knowledge(+ def test_auto_type_with_different_schemas(self, mock_llm, mock_embedder):+ """测试使用不同 Schema 的 BaseAutoType"""+ article_k = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,)- person_k = Knowledge(+ person_k = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Person,@@ -104,11 +104,11 @@ class TestKnowledgeInitialization:assert person_k.schema_class == Person-class TestKnowledgeExtract:- """测试 Knowledge 的 extract 方法"""+class TestBaseAutoTypeExtract:+ """测试 BaseAutoType 的 extract 方法"""@pytest.mark.asyncio- async def test_extract_article(self, article_knowledge, mock_llm):+ async def test_extract_article(self, a_article, mock_llm):"""测试提取文章信息"""# 模拟 LLM 返回的响应(with_structured_output 直接返回对象)expected_article = Article(@@ -124,15 +124,15 @@ class TestKnowledgeExtract:mock_llm.with_structured_output.return_value = mock_structured_llmtext = "Artificial Intelligence continues to advance in 2024..."- result = await article_knowledge.extract(text)+ result = await a_article.extract(text)assert result.title == "AI in 2024"assert len(result.authors) == 2assert "AI" in result.keywords- assert article_knowledge.data == result+ assert a_article.data == result@pytest.mark.asyncio- async def test_extract_with_markdown_response(self, article_knowledge, mock_llm):+ async def test_extract_with_markdown_response(self, a_article, mock_llm):"""测试 with_structured_output 自动处理响应格式"""# 使用 with_structured_output 无需处理 markdown 格式expected_article = Article(@@ -146,12 +146,12 @@ class TestKnowledgeExtract:mock_structured_llm.ainvoke.return_value = expected_articlemock_llm.with_structured_output.return_value = mock_structured_llm- result = await article_knowledge.extract("test text")+ result = await a_article.extract("test text")assert result.title == "Test Article"assert result.authors[0] == "Author"@pytest.mark.asyncio- async def test_aextract_calls_extract(self, article_knowledge, mock_llm):+ async def test_aextract_calls_extract(self, a_article, mock_llm):"""测试 aextract 方法调用 extract"""expected_article = Article(title="Test",@@ -164,12 +164,12 @@ class TestKnowledgeExtract:mock_structured_llm.ainvoke.return_value = expected_articlemock_llm.with_structured_output.return_value = mock_structured_llm- result = await article_knowledge.aextract("test")+ result = await a_article.aextract("test")assert result.title == "Test"-class TestKnowledgeDump:- """测试 Knowledge 的 dump 方法"""+class TestBaseAutoTypeDump:+ """测试 BaseAutoType 的 dump 方法"""def test_dump_to_json(self, mock_llm, mock_embedder):"""测试导出为 JSON 格式"""@@ -179,7 +179,7 @@ class TestKnowledgeDump:keywords=["test"],summary="A test")- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -200,7 +200,7 @@ class TestKnowledgeDump:keywords=["test"],summary="A test")- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -212,12 +212,12 @@ class TestKnowledgeDump:assert data_dict["title"] == "Test"-class TestKnowledgeLoad:- """测试 Knowledge 的 load 方法"""+class TestBaseAutoTypeLoad:+ """测试 BaseAutoType 的 load 方法"""def test_load_from_dict(self, mock_llm, mock_embedder):"""测试从字典加载"""- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -236,7 +236,7 @@ class TestKnowledgeLoad:def test_load_from_json_string(self, mock_llm, mock_embedder):"""测试从 JSON 字符串加载"""- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -254,7 +254,7 @@ class TestKnowledgeLoad:def test_load_invalid_data_raises_error(self, mock_llm, mock_embedder):"""测试加载无效数据时抛出错误"""- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -266,8 +266,8 @@ class TestKnowledgeLoad:knowledge.load(invalid_data)-class TestKnowledgeSearch:- """测试 Knowledge 的 search 方法"""+class TestBaseAutoTypeSearch:+ """测试 BaseAutoType 的 search 方法"""@pytest.mark.asyncioasync def test_search_returns_dict(self, mock_llm, mock_embedder):@@ -278,7 +278,7 @@ class TestKnowledgeSearch:keywords=["search"],summary="Testing search")- knowledge = Knowledge(+ knowledge = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,@@ -290,27 +290,27 @@ class TestKnowledgeSearch:assert result["title"] == "Search Test"-class TestKnowledgeEvolve:- """测试 Knowledge 的 evolve 和 aevolve 方法"""+class TestBaseAutoTypeEvolve:+ """测试 BaseAutoType 的 evolve 和 aevolve 方法"""@pytest.mark.asyncio- async def test_aevolve_default_implementation(self, article_knowledge):+ async def test_aevolve_default_implementation(self, a_article):"""测试 aevolve 的默认实现(不做任何操作)"""- await article_knowledge.aevolve()+ await a_article.aevolve()# 如果没有抛出异常,则测试通过-class TestKnowledgeMultipleSchemas:+class TestBaseAutoTypeMultipleSchemas:"""测试使用多个不同 Schema 的场景"""def test_different_schemas_independent(self, mock_llm, mock_embedder):- """测试不同 Schema 的 Knowledge 相互独立"""- article_k = Knowledge(+ """测试不同 Schema 的 BaseAutoType 相互独立"""+ article_k = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Article,)- person_k = Knowledge(+ person_k = BaseAutoType(llm_client=mock_llm,embedder=mock_embedder,schema_class=Person,