Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

feat: Introduce graph and hypergraph structures for knowledge extraction

yifanfeng979e39a78b7faa
diff --git a/README.md b/README.md
index c73e5d2..835f34a 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**
>
-> Transform unstructured text into structured, evolving knowledge containers with the power of Large Language Models.
+> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.
---
@@ -10,7 +10,7 @@
**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.
-Every knowledge container in Hyper-Extract follows a unified lifecycle:
+Every Auto container in Hyper-Extract follows a unified lifecycle:
1. **Extract** - Intelligent extraction from text with automatic chunking and merging
2. **Evolve** - Self-refinement through reasoning, pruning, and optimization
@@ -27,7 +27,7 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:
- **Intelligent Merging**: First-priority merge strategy for chunk-based extraction
### 🏗️ Unified Architecture
-- **Abstract Base Class**: `BaseKnowledge[T]` provides consistent interface across all knowledge types
+- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types
- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility
- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline
@@ -37,11 +37,11 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:
- **Field-Level Search**: Search across all schema fields with metadata preservation
### 📦 Multiple Knowledge Types (Roadmap)
-- ✅ **UnitKnowledge**: Universal schema-agnostic knowledge extraction
-- 🚧 **EntityKnowledge**: Entity-centric extraction with attribute management
-- 🚧 **RelationKnowledge**: Relationship extraction with entity linking
-- 🚧 **GraphKnowledge**: Knowledge graph construction (nodes + edges)
-- 🚧 **HypergraphKnowledge**: Hypergraph representation for complex multi-entity relations
+- ✅ **AutoModel**: Universal schema-agnostic knowledge extraction
+- ✅ **AutoList**: Collection-based extraction for lists of items
+- ✅ **AutoSet**: Unique collection with automatic deduplication
+- 🚧 **AutoGraph**: Knowledge graph construction (nodes + edges)
+- 🚧 **AutoHypergraph**: Hypergraph representation for complex multi-entity relations
- 🚧 **ExtractionExperience**: Meta-learning from extraction patterns
---
@@ -77,7 +77,7 @@ pip install -r requirements.txt
from pydantic import BaseModel, Field
from typing import List
-class ArticleKnowledge(BaseModel):
+class ArticleSchema(BaseModel):
"""Schema for article knowledge extraction"""
title: str = Field(default="", description="Article title")
author: str = Field(default="", description="Author name")
@@ -86,19 +86,19 @@ class ArticleKnowledge(BaseModel):
topics: List[str] = Field(default_factory=list, description="Topic tags")
```
-### 2. Initialize Knowledge Container
+### 2. Initialize Auto Container
```python
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract import AutoModel
# Initialize LLM and embedder
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
-# Create knowledge container
-knowledge = UnitKnowledge(
- data_schema=ArticleKnowledge,
+# Create Auto container
+a_article = AutoModel(
+ data_schema=ArticleSchema,
llm_client=llm,
embedder=embedder,
chunk_size=2000, # Auto-split long texts
@@ -119,23 +119,23 @@ Artificial intelligence is transforming industries...
"""
# Extract with automatic chunking and merging
-result = knowledge.extract(article_text)
+result = a_article.extract(article_text)
print(f"Extraction completed: {result}")
# Access extracted data
-print(f"Title: {knowledge.data.title}")
-print(f"Author: {knowledge.data.author}")
-print(f"Key points: {knowledge.data.key_points}")
+print(f"Title: {a_article.data.title}")
+print(f"Author: {a_article.data.author}")
+print(f"Key points: {a_article.data.key_points}")
```
### 4. Semantic Search
```python
# Build vector index
-knowledge.build_index()
+a_article.build_index()
# Search with natural language
-results = knowledge.search("What are the main topics?", top_k=3)
+results = a_article.search("What are the main topics?", top_k=3)
for result in results:
print(result)
```
@@ -144,10 +144,10 @@ for result in results:
```python
# Save to folder (includes data + FAISS index)
-knowledge.dump("./output/my_knowledge")
+a_article.dump("./output/my_article")
# Load from folder
-knowledge.load("./output/my_knowledge")
+a_article.load("./output/my_article")
```
---
@@ -156,7 +156,7 @@ knowledge.load("./output/my_knowledge")
### Design Principles
-1. **Template Method Pattern**: `BaseKnowledge` defines the extraction skeleton, subclasses customize behavior
+1. **Template Method Pattern**: `BaseAutoType` defines the extraction skeleton, subclasses customize behavior
2. **First-Priority Merge**: `model_copy(update=model_dump(exclude_none=True))` for conflict resolution
3. **Lazy Indexing**: Build FAISS indices only when needed, track with `_index_dirty` flag
4. **Type Safety**: Leverage Pydantic for runtime validation and IDE support
@@ -167,7 +167,7 @@ knowledge.load("./output/my_knowledge")
Check out the `examples/` directory for complete working examples:
-- [`generic_knowledge_demo.py`](examples/generic_knowledge_demo.py) - Full UnitKnowledge walkthrough
+- [`auto_model_demo.py`](examples/auto_model_demo.py) - Full AutoModel walkthrough
- More examples coming soon!
---
@@ -175,17 +175,16 @@ Check out the `examples/` directory for complete working examples:
## 🗺️ Roadmap
### Phase 1: Foundation (Current) ✅
-- [x] Abstract `BaseKnowledge` class
-- [x] `UnitKnowledge` implementation
+- [x] Abstract `BaseAutoType` class
+- [x] `AutoModel` implementation
- [x] LangChain integration with structured output
- [x] FAISS vector search
- [x] Serialization (dump/load)
### Phase 2: Specialized Knowledge Types 🚧
-- [ ] `EntityKnowledge` - Entity extraction with coreference resolution
-- [ ] `RelationKnowledge` - Relationship extraction and linking
-- [ ] `GraphKnowledge` - Full knowledge graph construction
-- [ ] `HypergraphKnowledge` - Hypergraph for complex relations
+- [ ] `AutoGraph` - Full knowledge graph construction
+- [ ] `AutoHypergraph` - Hypergraph for complex relations
+- [ ] `AutoTree` - Hierarchical tree structures
### Phase 3: Evolution & Learning 🔮
- [ ] `evolve()` implementation with reasoning
diff --git a/docs/BUILDING.md b/docs/BUILDING.md
index 0cd6674..c6bb2b1 100644
--- a/docs/BUILDING.md
+++ b/docs/BUILDING.md
@@ -104,10 +104,10 @@ docs/
│ └── knowledge-patterns.md # Pattern selection guide
├── api/
│ ├── index.md # API overview
-│ ├── base.md # BaseKnowledge API
-│ ├── unit.md # UnitKnowledge API
-│ ├── list.md # ListKnowledge API
-│ └── set.md # SetKnowledge API
+│ ├── base.md # BaseAutoType API
+│ ├── unit.md # AutoModel API
+│ ├── list.md # AutoList API
+│ └── set.md # AutoSet API
└── zh/
└── index.md # Chinese homepage (中文首页)
```
@@ -205,9 +205,9 @@ plugins:
Usage in markdown:
```markdown
-# BaseKnowledge API
+# BaseAutoType API
-::: hyperextract.knowledge.base.BaseKnowledge
+::: hyperextract.core.base.BaseAutoType
options:
show_source: true
heading_level: 2
@@ -290,7 +290,7 @@ jobs:
1. **Use code blocks with language hints**:
````markdown
```python
- from hyperextract import UnitKnowledge
+ from hyperextract import AutoModel
```
````
diff --git a/docs/api/list.md b/docs/api/auto_list.md
similarity index 88%
rename from docs/api/list.md
rename to docs/api/auto_list.md
index 2289e37..02e6f68 100644
--- a/docs/api/list.md
+++ b/docs/api/auto_list.md
@@ -1,6 +1,6 @@
-# ListKnowledge API
+# AutoList API
-::: hyperextract.knowledge.generic.list.ListKnowledge
+::: hyperextract.core.list.AutoList
options:
show_root_heading: true
show_source: true
@@ -8,12 +8,12 @@
## Overview
-`ListKnowledge` implements the multi-item list extraction pattern. It extracts multiple independent objects of the same type from a document, suitable for entity lists, events, citations, and more.
+`AutoList` implements the multi-item list extraction pattern. It extracts multiple independent objects of the same type from a document, suitable for entity lists, events, citations, and more.
## Class Signature
```python
-class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
+class AutoList(BaseAutoType[ItemListSchema[Item]], Generic[Item]):
"""List Knowledge Pattern - extracts a collection of objects from text."""
```
@@ -27,7 +27,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
## Initialization
```python
-from hyperextract.knowledge.generic import ListKnowledge
+from hyperextract.core import AutoList
from pydantic import BaseModel, Field
from typing import Optional
@@ -36,7 +36,7 @@ class Person(BaseModel):
role: Optional[str] = Field(None, description="Role or title")
organization: Optional[str] = Field(None, description="Organization")
-people = ListKnowledge(
+people = AutoList(
item_schema=Person, # Note: item_schema, not data_schema!
llm_client=llm,
embedder=embedder
@@ -74,7 +74,7 @@ print(len(container.items))
## List Operations
-`ListKnowledge` supports Pythonic list operations:
+`AutoList` supports Pythonic list operations:
### Indexing
```python
@@ -215,7 +215,7 @@ class Entity(BaseModel):
type: str = Field(..., description="Entity type (person/org/location)")
context: str = Field(default="", description="Surrounding context")
-entities = ListKnowledge(
+entities = AutoList(
item_schema=Entity,
llm_client=llm,
embedder=embedder
@@ -230,7 +230,7 @@ class Event(BaseModel):
participants: List[str] = Field(default_factory=list)
location: Optional[str] = None
-events = ListKnowledge(
+events = AutoList(
item_schema=Event,
llm_client=llm,
embedder=embedder
@@ -245,7 +245,7 @@ class Citation(BaseModel):
year: Optional[int] = None
venue: Optional[str] = None
-citations = ListKnowledge(
+citations = AutoList(
item_schema=Citation,
llm_client=llm,
embedder=embedder
@@ -259,7 +259,7 @@ citations = ListKnowledge(
Subclass to implement custom deduplication:
```python
-class SmartListKnowledge(ListKnowledge[Item]):
+class SmartAutoList(AutoList[Item]):
def _merge_strategy(self, old_data, new_data):
# Custom deduplication logic
existing_names = {item.name for item in old_data.items}
@@ -312,18 +312,18 @@ sorted_people = sorted(people.items, key=attrgetter('organization', 'name'))
4. **Chunk Size**: Smaller chunks for dense information
```python
# For entity-rich text
- ListKnowledge(..., chunk_size=1500, chunk_overlap=150)
+ AutoList(..., chunk_size=1500, chunk_overlap=150)
```
## Serialization
```python
# Save
-people.dump("./people_knowledge")
+people.dump("./a_people")
# Load
-loaded = ListKnowledge.load(
- "./people_knowledge",
+loaded = AutoList.load(
+ "./a_people",
item_schema=Person,
llm_client=llm,
embedder=embedder
@@ -332,9 +332,9 @@ loaded = ListKnowledge.load(
assert len(loaded) == len(people)
```
-## Comparison with SetKnowledge
+## Comparison with AutoSet
-| Feature | ListKnowledge | SetKnowledge |
+| Feature | AutoList | AutoSet |
|---------|---------------|--------------|
| Duplicates | Allowed (basic dedup) | Prevented (by unique_key) |
| Merge | Simple append | Configurable strategies |
@@ -346,6 +346,6 @@ assert len(loaded) == len(people)
## See Also
-- [SetKnowledge API](set.md) - Unique collection with deduplication
-- [BaseKnowledge API](base.md) - Base class documentation
+- [AutoSet API](set.md) - Unique collection with deduplication
+- [BaseAutoType API](base.md) - Base class documentation
- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection
diff --git a/docs/api/set.md b/docs/api/auto_set.md
similarity index 89%
rename from docs/api/set.md
rename to docs/api/auto_set.md
index dfca60e..4464427 100644
--- a/docs/api/set.md
+++ b/docs/api/auto_set.md
@@ -1,6 +1,6 @@
-# SetKnowledge API
+# AutoSet API
-::: hyperextract.knowledge.generic.set.SetKnowledge
+::: hyperextract.core.set.AutoSet
options:
show_root_heading: true
show_source: true
@@ -8,12 +8,12 @@
## Overview
-`SetKnowledge` implements the unique collection pattern with automatic deduplication. It extracts items and ensures uniqueness based on a user-specified key field, with configurable merge strategies for handling duplicates.
+`AutoSet` implements the unique collection pattern with automatic deduplication. It extracts items and ensures uniqueness based on a user-specified key field, with configurable merge strategies for handling duplicates.
## Class Signature
```python
-class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
+class AutoSet(BaseAutoType[ItemSetSchema[Item]], Generic[Item]):
"""Set Knowledge Pattern - extracts a unique collection of objects."""
```
@@ -28,7 +28,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
## Initialization
```python
-from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy
+from hyperextract.core import AutoSet, MergeItemStrategy
from pydantic import BaseModel, Field
class Keyword(BaseModel):
@@ -36,7 +36,7 @@ class Keyword(BaseModel):
category: Optional[str] = Field(None, description="Category")
importance: Optional[str] = Field(None, description="Importance level")
-keywords = SetKnowledge(
+keywords = AutoSet(
item_schema=Keyword,
llm_client=llm,
embedder=embedder,
@@ -57,7 +57,7 @@ keywords = SetKnowledge(
### KEEP_OLD
Keep existing item, discard new duplicate:
```python
-SetKnowledge(
+AutoSet(
...,
merge_strategy=MergeItemStrategy.KEEP_OLD
)
@@ -73,7 +73,7 @@ SetKnowledge(
### KEEP_NEW (Default)
Replace existing with new:
```python
-SetKnowledge(
+AutoSet(
...,
merge_strategy=MergeItemStrategy.KEEP_NEW
)
@@ -89,7 +89,7 @@ SetKnowledge(
### FIELD_MERGE
Merge fields - new fills old's None/empty fields:
```python
-SetKnowledge(
+AutoSet(
...,
merge_strategy=MergeItemStrategy.FIELD_MERGE
)
@@ -105,7 +105,7 @@ SetKnowledge(
### LLM_MERGE
Use LLM to intelligently merge:
```python
-SetKnowledge(
+AutoSet(
...,
merge_strategy=MergeItemStrategy.LLM_MERGE
)
@@ -216,7 +216,7 @@ if keywords.contains("Python"):
## Set Operations
-`SetKnowledge` supports mathematical set operations:
+`AutoSet` supports mathematical set operations:
### Union (`|`)
Combine two sets:
@@ -248,8 +248,8 @@ exclusive = keywords1 ^ keywords2
### Example:
```python
-tech_keywords = SetKnowledge(item_schema=Keyword, ...)
-science_keywords = SetKnowledge(item_schema=Keyword, ...)
+tech_keywords = AutoSet(item_schema=Keyword, ...)
+science_keywords = AutoSet(item_schema=Keyword, ...)
# Extract from different sources
tech_keywords.extract(tech_article)
@@ -283,7 +283,7 @@ class Entity(BaseModel):
type: str = Field(..., description="Entity type")
description: Optional[str] = None
-entities = SetKnowledge(
+entities = AutoSet(
item_schema=Entity,
llm_client=llm,
embedder=embedder,
@@ -299,7 +299,7 @@ class Topic(BaseModel):
subtopics: List[str] = Field(default_factory=list)
relevance: Optional[str] = None
-topics = SetKnowledge(
+topics = AutoSet(
item_schema=Topic,
llm_client=llm,
embedder=embedder,
@@ -315,7 +315,7 @@ class Skill(BaseModel):
proficiency: Optional[str] = None
years_experience: Optional[int] = None
-skills = SetKnowledge(
+skills = AutoSet(
item_schema=Skill,
llm_client=llm,
embedder=embedder,
@@ -331,7 +331,7 @@ skills = SetKnowledge(
For complex merging, use `LLM_MERGE` or subclass:
```python
-class SmartSetKnowledge(SetKnowledge[Item]):
+class SmartAutoSet(AutoSet[Item]):
def _merge_items_llm(self, old_item: Item, new_item: Item) -> Item:
# Custom LLM prompt for merging
custom_prompt = f'''
@@ -385,7 +385,7 @@ terms_only = [k.term for k in keywords.items]
3. **Performance Considerations**:
```python
# LLM_MERGE is slower (makes LLM calls)
- SetKnowledge(..., merge_strategy=MergeItemStrategy.LLM_MERGE)
+ AutoSet(..., merge_strategy=MergeItemStrategy.LLM_MERGE)
# For large datasets, consider FIELD_MERGE or KEEP_NEW
```
@@ -397,7 +397,7 @@ terms_only = [k.term for k in keywords.items]
id: str = Field(..., description="Unique ID") # Required!
...
- knowledge = SetKnowledge(..., unique_key="id")
+ knowledge = AutoSet(..., unique_key="id")
```
## Serialization
@@ -407,7 +407,7 @@ terms_only = [k.term for k in keywords.items]
keywords.dump("./keywords_set")
# Load (preserves uniqueness)
-loaded = SetKnowledge.load(
+loaded = AutoSet.load(
"./keywords_set",
item_schema=Keyword,
llm_client=llm,
@@ -419,7 +419,7 @@ loaded = SetKnowledge.load(
## Comparison
-| Feature | ListKnowledge | SetKnowledge |
+| Feature | AutoList | AutoSet |
|---------|---------------|--------------|
| Duplicates | Allowed | Prevented by unique_key |
| Merge Strategies | Simple append | 4 configurable strategies |
@@ -429,6 +429,6 @@ loaded = SetKnowledge.load(
## See Also
-- [ListKnowledge API](list.md) - Multi-item list extraction
-- [BaseKnowledge API](base.md) - Base class documentation
+- [AutoList API](list.md) - Multi-item list extraction
+- [BaseAutoType API](base.md) - Base class documentation
- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guide
diff --git a/docs/api/base.md b/docs/api/base.md
index a29b8d8..f1cad3c 100644
--- a/docs/api/base.md
+++ b/docs/api/base.md
@@ -1,6 +1,6 @@
-# BaseKnowledge API
+# BaseAutoType API
-::: hyperextract.knowledge.base.BaseKnowledge
+::: hyperextract.core.base.BaseAutoType
options:
show_root_heading: true
show_source: true
@@ -8,12 +8,12 @@
## Overview
-`BaseKnowledge` is the abstract base class that provides the foundation for all knowledge extraction patterns in Hyper-Extract. It implements the core lifecycle operations: extraction, merging, indexing, evolution, and serialization.
+`BaseAutoType` is the abstract base class that provides the foundation for all knowledge extraction patterns in Hyper-Extract. It implements the core lifecycle operations: extraction, merging, indexing, evolution, and serialization.
## Class Signature
```python
-class BaseKnowledge(ABC, Generic[T]):
+class BaseAutoType(ABC, Generic[T]):
"""Unified knowledge base class integrating extraction, storage,
aggregation, and evolution."""
```
@@ -38,7 +38,7 @@ Where `T` is a Pydantic `BaseModel` subclass defining your knowledge schema.
### Example
```python
-from hyperextract.knowledge.base import BaseKnowledge
+from hyperextract.core.base import BaseAutoType
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from pydantic import BaseModel, Field
@@ -49,8 +49,8 @@ class MySchema(BaseModel):
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
-# Note: BaseKnowledge is abstract, use concrete implementations
-# knowledge = UnitKnowledge(...)
+# Note: BaseAutoType is abstract, use concrete implementations
+# knowledge = AutoModel(...)
```
## Properties
@@ -189,7 +189,7 @@ def load(
llm_client: BaseChatModel,
embedder: Embeddings,
**kwargs
-) -> "BaseKnowledge[T]"
+) -> "BaseAutoType[T]"
```
Load knowledge from disk.
@@ -205,7 +205,7 @@ Load knowledge from disk.
**Example**:
```python
-loaded = UnitKnowledge.load(
+loaded = AutoModel.load(
"./my_knowledge",
data_schema=MySchema,
llm_client=llm,
@@ -268,14 +268,14 @@ Handles multi-chunk extraction with parallel processing and merging.
## Usage in Subclasses
-When creating custom knowledge patterns, inherit from `BaseKnowledge`:
+When creating custom knowledge patterns, inherit from `BaseAutoType`:
```python
-from hyperextract.knowledge.base import BaseKnowledge, T
+from hyperextract.core.base import BaseAutoType, T
from typing import List
from langchain_core.documents import Document
-class CustomKnowledge(BaseKnowledge[T]):
+class CustomKnowledge(BaseAutoType[T]):
@staticmethod
def _default_prompt() -> str:
return "Extract information according to schema: "
@@ -291,6 +291,6 @@ class CustomKnowledge(BaseKnowledge[T]):
## See Also
-- [UnitKnowledge API](unit.md) - Single-object extraction
-- [ListKnowledge API](list.md) - Multi-item extraction
-- [SetKnowledge API](set.md) - Unique collection extraction
+- [AutoModel API](unit.md) - Single-object extraction
+- [AutoList API](list.md) - Multi-item extraction
+- [AutoSet API](set.md) - Unique collection extraction
diff --git a/docs/api/index.md b/docs/api/index.md
index 5c145e6..2d039c1 100644
--- a/docs/api/index.md
+++ b/docs/api/index.md
@@ -7,19 +7,19 @@ This section provides detailed API documentation for all Hyper-Extract classes a
```
hyperextract/
├── knowledge/
-│ ├── base.py # BaseKnowledge abstract class
+│ ├── base.py # BaseAutoType abstract class
│ └── generic/
-│ ├── unit.py # UnitKnowledge
-│ ├── list.py # ListKnowledge
-│ └── set.py # SetKnowledge
+│ ├── unit.py # AutoModel
+│ ├── list.py # AutoList
+│ └── set.py # AutoSet
```
## Quick Links
-- [BaseKnowledge](base.md) - Abstract base class for all knowledge patterns
-- [UnitKnowledge](unit.md) - Single-object extraction pattern
-- [ListKnowledge](list.md) - Multi-item list extraction pattern
-- [SetKnowledge](set.md) - Unique collection with deduplication
+- [BaseAutoType](base.md) - Abstract base class for all knowledge patterns
+- [AutoModel](unit.md) - Single-object extraction pattern
+- [AutoList](list.md) - Multi-item list extraction pattern
+- [AutoSet](set.md) - Unique collection with deduplication
## Common Patterns
@@ -59,14 +59,14 @@ Every knowledge class implements these methods:
All classes are fully typed with Python generics:
```python
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract.core import AutoModel
from pydantic import BaseModel
class MySchema(BaseModel):
field: str
# Type-safe: knowledge.data is of type MySchema
-knowledge: UnitKnowledge[MySchema] = UnitKnowledge(
+knowledge: AutoModel[MySchema] = AutoModel(
data_schema=MySchema,
llm_client=llm,
embedder=embedder
@@ -75,7 +75,7 @@ knowledge: UnitKnowledge[MySchema] = UnitKnowledge(
## Next: Detailed API Docs
-- [BaseKnowledge API](base.md)
-- [UnitKnowledge API](unit.md)
-- [ListKnowledge API](list.md)
-- [SetKnowledge API](set.md)
+- [BaseAutoType API](base.md)
+- [AutoModel API](unit.md)
+- [AutoList API](list.md)
+- [AutoSet API](set.md)
diff --git a/docs/api/unit.md b/docs/api/model.md
similarity index 86%
rename from docs/api/unit.md
rename to docs/api/model.md
index 164b79e..a77fba2 100644
--- a/docs/api/unit.md
+++ b/docs/api/model.md
@@ -1,6 +1,6 @@
-# UnitKnowledge API
+# AutoModel API
-::: hyperextract.knowledge.generic.unit.UnitKnowledge
+::: hyperextract.core.model.AutoModel
options:
show_root_heading: true
show_source: true
@@ -8,12 +8,12 @@
## Overview
-`UnitKnowledge` implements the single-object extraction pattern. It's designed to extract exactly one structured object per document, making it ideal for document-level information like summaries, metadata, or aggregate statistics.
+`AutoModel` implements the single-object extraction pattern. It's designed to extract exactly one structured object per document, making it ideal for document-level information like summaries, metadata, or aggregate statistics.
## Class Signature
```python
-class UnitKnowledge(BaseKnowledge[T]):
+class AutoModel(BaseAutoType[T]):
"""Unit Knowledge Pattern - extracts a single structured object from text."""
```
@@ -26,10 +26,10 @@ class UnitKnowledge(BaseKnowledge[T]):
## Initialization
-Inherits all parameters from `BaseKnowledge` with same defaults.
+Inherits all parameters from `BaseAutoType` with same defaults.
```python
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract.core import AutoModel
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from pydantic import BaseModel, Field
@@ -41,7 +41,7 @@ class DocumentMetadata(BaseModel):
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
-knowledge = UnitKnowledge(
+knowledge = AutoModel(
data_schema=DocumentMetadata,
llm_client=llm,
embedder=embedder,
@@ -178,7 +178,7 @@ class Summary(BaseModel):
key_points: List[str] = Field(default_factory=list, description="Key points")
conclusion: str = Field(default="", description="Conclusion")
-summary_knowledge = UnitKnowledge(
+summary_knowledge = AutoModel(
data_schema=Summary,
llm_client=llm,
embedder=embedder
@@ -193,7 +193,7 @@ class Article(BaseModel):
author: str = Field(default="", description="Author")
category: str = Field(default="", description="Category")
-article_knowledge = UnitKnowledge(
+article_knowledge = AutoModel(
data_schema=Article,
llm_client=llm,
embedder=embedder
@@ -208,7 +208,7 @@ class Sentiment(BaseModel):
negative_aspects: List[str] = Field(default_factory=list)
confidence: float = Field(default=0.0, description="Confidence score")
-sentiment_knowledge = UnitKnowledge(
+sentiment_knowledge = AutoModel(
data_schema=Sentiment,
llm_client=llm,
embedder=embedder
@@ -231,10 +231,10 @@ sentiment_knowledge = UnitKnowledge(
3. **Chunk Sizing**: Adjust based on where information appears
```python
# For metadata (usually at start)
- UnitKnowledge(..., chunk_size=1000, chunk_overlap=100)
+ AutoModel(..., chunk_size=1000, chunk_overlap=100)
# For summary (may span document)
- UnitKnowledge(..., chunk_size=3000, chunk_overlap=300)
+ AutoModel(..., chunk_size=3000, chunk_overlap=300)
```
4. **Custom Prompts**: Provide domain-specific guidance
@@ -246,7 +246,7 @@ sentiment_knowledge = UnitKnowledge(
- Citation count if mentioned
'''
- knowledge = UnitKnowledge(
+ knowledge = AutoModel(
data_schema=PaperMetadata,
llm_client=llm,
embedder=embedder,
@@ -261,7 +261,7 @@ sentiment_knowledge = UnitKnowledge(
knowledge.dump("./document_metadata")
# Load
-loaded = UnitKnowledge.load(
+loaded = AutoModel.load(
"./document_metadata",
data_schema=DocumentMetadata,
llm_client=llm,
@@ -273,7 +273,7 @@ assert loaded.data.title == knowledge.data.title
## See Also
-- [BaseKnowledge API](base.md) - Base class documentation
-- [ListKnowledge API](list.md) - For extracting multiple items
-- [SetKnowledge API](set.md) - For unique collections
+- [BaseAutoType API](base.md) - Base class documentation
+- [AutoList API](list.md) - For extracting multiple items
+- [AutoSet API](set.md) - For unique collections
- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guide
diff --git a/docs/index.md b/docs/index.md
index db96777..d42b0a8 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -2,7 +2,7 @@
> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**
>
-> Transform unstructured text into structured, evolving knowledge containers with the power of Large Language Models.
+> Transform unstructured text into structured, evolving Auto containers with the power of Large Language Models.
---
@@ -10,7 +10,7 @@
**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.
-Every knowledge container in Hyper-Extract follows a unified lifecycle:
+Every Auto container in Hyper-Extract follows a unified lifecycle:
1. **Extract** - Intelligent extraction from text with automatic chunking and merging
2. **Evolve** - Self-refinement through reasoning, pruning, and optimization
@@ -27,7 +27,7 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:
- **Intelligent Merging**: First-priority merge strategy for chunk-based extraction
### 🏗️ Unified Architecture
-- **Abstract Base Class**: `BaseKnowledge[T]` provides consistent interface across all knowledge types
+- **Abstract Base Class**: `BaseAutoType[T]` provides consistent interface across all knowledge types
- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility
- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline
@@ -39,15 +39,14 @@ Every knowledge container in Hyper-Extract follows a unified lifecycle:
### 📦 Knowledge Patterns
#### ✅ Implemented
-- **UnitKnowledge**: Single-object extraction for document-level information (summaries, metadata)
-- **ListKnowledge**: Collection-based extraction for entities, events, and structured lists
-- **SetKnowledge**: Unique collection with automatic deduplication and intelligent merge strategies
+- **AutoModel**: Single-object extraction for document-level information (summaries, metadata)
+- **AutoList**: Collection-based extraction for entities, events, and structured lists
+- **AutoSet**: Unique collection with automatic deduplication and intelligent merge strategies
#### 🚧 In Development
-- **EntityKnowledge**: Entity-centric extraction with attribute management
-- **RelationKnowledge**: Relationship extraction with entity linking
-- **GraphKnowledge**: Knowledge graph construction (nodes + edges)
-- **HypergraphKnowledge**: Hypergraph representation for complex multi-entity relations
+- **AutoGraph**: Knowledge graph construction (nodes + edges)
+- **AutoHypergraph**: Hypergraph representation for complex multi-entity relations
+- **AutoTree**: Hierarchical tree structures for nested data
---
@@ -82,7 +81,7 @@ pip install -e .
from pydantic import BaseModel, Field
from typing import List
-class ArticleKnowledge(BaseModel):
+class ArticleSchema(BaseModel):
"""Schema for article knowledge extraction"""
title: str = Field(default="", description="Article title")
author: str = Field(default="", description="Author name")
@@ -95,15 +94,15 @@ class ArticleKnowledge(BaseModel):
```python
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract.core import AutoModel
# Initialize LLM and embedder
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
-# Create knowledge container
-knowledge = UnitKnowledge(
- data_schema=ArticleKnowledge,
+# Create Auto container
+a_article = AutoModel(
+ data_schema=ArticleSchema,
llm_client=llm,
embedder=embedder
)
@@ -118,18 +117,18 @@ text = """
written by Professor Li Ming...
"""
-knowledge.extract(text)
-print(knowledge.data) # Access extracted structured knowledge
+a_article.extract(text)
+print(a_article.data) # Access extracted structured knowledge
```
### 4. Semantic Search
```python
# Build index
-knowledge.build_index()
+a_article.build_index()
# Search for relevant content
-results = knowledge.search("applications of deep learning", k=3)
+results = a_article.search("applications of deep learning", k=3)
for doc, score in results:
print(f"Score: {score}, Content: {doc.page_content}")
```
@@ -138,12 +137,12 @@ for doc, score in results:
```python
# Save to disk
-knowledge.dump("./my_knowledge")
+a_article.dump("./my_knowledge")
# Load
-loaded_knowledge = UnitKnowledge.load(
+a_loaded = AutoModel.load(
"./my_knowledge",
- data_schema=ArticleKnowledge,
+ data_schema=ArticleSchema,
llm_client=llm,
embedder=embedder
)
diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.md
index cef91c0..953a86d 100644
--- a/docs/user-guide/getting-started.md
+++ b/docs/user-guide/getting-started.md
@@ -63,7 +63,7 @@ class BookReview(BaseModel):
### Knowledge Container
-A **knowledge container** is an instance of a knowledge pattern class (e.g., `UnitKnowledge`, `ListKnowledge`, `SetKnowledge`) that manages extraction, storage, and retrieval of structured knowledge.
+A **Auto container** is an instance of a knowledge pattern class (e.g., `AutoModel`, `AutoList`, `AutoSet`) that manages extraction, storage, and retrieval of structured knowledge.
### Extraction Pipeline
@@ -109,10 +109,10 @@ embedder = OpenAIEmbeddings(model="text-embedding-3-small")
### Step 3: Create Knowledge Container
```python
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract.core import AutoModel
# Create container
-paper_knowledge = UnitKnowledge(
+paper_knowledge = AutoModel(
data_schema=ResearchPaper,
llm_client=llm,
embedder=embedder,
@@ -172,7 +172,7 @@ for doc, score in results:
paper_knowledge.dump("./paper_knowledge")
# Later, load it back
-loaded_knowledge = UnitKnowledge.load(
+loaded_knowledge = AutoModel.load(
"./paper_knowledge",
data_schema=ResearchPaper,
llm_client=llm,
@@ -194,7 +194,7 @@ Hyper-Extract automatically handles long documents by:
You can control chunking behavior:
```python
-knowledge = UnitKnowledge(
+knowledge = AutoModel(
data_schema=YourSchema,
llm_client=llm,
embedder=embedder,
diff --git a/docs/user-guide/knowledge-patterns.md b/docs/user-guide/knowledge-patterns.md
index d34a28a..26aa4ed 100644
--- a/docs/user-guide/knowledge-patterns.md
+++ b/docs/user-guide/knowledge-patterns.md
@@ -8,17 +8,17 @@ Hyper-Extract provides three fundamental knowledge patterns, each optimized for
| Pattern | Use Case | Deduplication | Merge Strategy |
|---------|----------|---------------|----------------|
-| **UnitKnowledge** | Single object per document | N/A | Field-level merge |
-| **ListKnowledge** | Multiple independent items | Basic | Append |
-| **SetKnowledge** | Unique collection | Automatic | Configurable |
+| **AutoModel** | Single object per document | N/A | Field-level merge |
+| **AutoList** | Multiple independent items | Basic | Append |
+| **AutoSet** | Unique collection | Automatic | Configurable |
---
-## UnitKnowledge Pattern
+## AutoModel Pattern
### When to Use
-Use `UnitKnowledge` when you want to extract **exactly one structured object** from a document, regardless of its length.
+Use `AutoModel` when you want to extract **exactly one structured object** from a document, regardless of its length.
**Perfect for**:
- Document summaries
@@ -39,7 +39,7 @@ Use `UnitKnowledge` when you want to extract **exactly one structured object** f
from pydantic import BaseModel, Field
from typing import List
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract.core import AutoModel
# Define schema
class DocumentMetadata(BaseModel):
@@ -54,7 +54,7 @@ class DocumentMetadata(BaseModel):
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
-metadata = UnitKnowledge(
+metadata = AutoModel(
data_schema=DocumentMetadata,
llm_client=llm,
embedder=embedder
@@ -84,11 +84,11 @@ results = metadata.search("machine learning applications", k=5)
---
-## ListKnowledge Pattern
+## AutoList Pattern
### When to Use
-Use `ListKnowledge` when you need to extract **multiple independent items** of the same type from a document.
+Use `AutoList` when you need to extract **multiple independent items** of the same type from a document.
**Perfect for**:
- Entity lists (people, organizations, locations)
@@ -108,7 +108,7 @@ Use `ListKnowledge` when you need to extract **multiple independent items** of t
```python
from pydantic import BaseModel, Field
from typing import List, Optional
-from hyperextract.knowledge.generic import ListKnowledge
+from hyperextract.core import AutoList
# Define item schema
class Person(BaseModel):
@@ -118,7 +118,7 @@ class Person(BaseModel):
context: Optional[str] = Field(None, description="Context of mention")
# Initialize
-people = ListKnowledge(
+people = AutoList(
item_schema=Person, # Note: item_schema, not data_schema
llm_client=llm,
embedder=embedder
@@ -142,7 +142,7 @@ print(f"Total people extracted: {len(people.items)}")
### List Operations
-`ListKnowledge` supports Pythonic list operations:
+`AutoList` supports Pythonic list operations:
```python
# Append new items
@@ -185,11 +185,11 @@ for doc, score in results:
---
-## SetKnowledge Pattern
+## AutoSet Pattern
### When to Use
-Use `SetKnowledge` when you need a **unique collection** with automatic deduplication based on a key field.
+Use `AutoSet` when you need a **unique collection** with automatic deduplication based on a key field.
**Perfect for**:
- Keywords/tags (deduplicated by term)
@@ -210,7 +210,7 @@ Use `SetKnowledge` when you need a **unique collection** with automatic deduplic
```python
from pydantic import BaseModel, Field
from typing import Optional
-from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy
+from hyperextract.core import AutoSet, MergeItemStrategy
# Define item schema with unique key
class Keyword(BaseModel):
@@ -220,7 +220,7 @@ class Keyword(BaseModel):
context: Optional[str] = Field(None, description="Usage context")
# Initialize with unique_key
-keywords = SetKnowledge(
+keywords = AutoSet(
item_schema=Keyword,
llm_client=llm,
embedder=embedder,
@@ -246,7 +246,7 @@ for kw in keywords.items:
#### 1. KEEP_OLD
Keeps existing item, discards new duplicate:
```python
-keywords = SetKnowledge(
+keywords = AutoSet(
item_schema=Keyword,
llm_client=llm,
embedder=embedder,
@@ -275,7 +275,7 @@ merge_strategy=MergeItemStrategy.LLM_MERGE
### Set Operations
-`SetKnowledge` supports set operations:
+`AutoSet` supports set operations:
```python
# Union: combine two sets
@@ -293,7 +293,7 @@ exclusive = keywords1 ^ keywords2
### Indexing Strategy
-Like ListKnowledge, each unique item is indexed:
+Like AutoList, each unique item is indexed:
```python
keywords.build_index()
@@ -308,39 +308,39 @@ results = keywords.search("AI technologies", k=5)
```
Is it a single object per document?
-├─ Yes → Use UnitKnowledge
+├─ Yes → Use AutoModel
└─ No → Are duplicates acceptable?
- ├─ Yes → Use ListKnowledge
- └─ No → Use SetKnowledge (specify unique_key)
+ ├─ Yes → Use AutoList
+ └─ No → Use AutoSet (specify unique_key)
```
### Examples by Domain
**News Article Processing**:
-- `UnitKnowledge`: Article metadata, summary
-- `ListKnowledge`: All mentioned events
-- `SetKnowledge`: Unique entities, locations
+- `AutoModel`: Article metadata, summary
+- `AutoList`: All mentioned events
+- `AutoSet`: Unique entities, locations
**Research Paper Analysis**:
-- `UnitKnowledge`: Paper metadata, abstract
-- `ListKnowledge`: All citations
-- `SetKnowledge`: Unique keywords, methodologies
+- `AutoModel`: Paper metadata, abstract
+- `AutoList`: All citations
+- `AutoSet`: Unique keywords, methodologies
**Customer Reviews**:
-- `UnitKnowledge`: Overall sentiment, rating
-- `ListKnowledge`: All complaints/praises
-- `SetKnowledge`: Unique features mentioned
+- `AutoModel`: Overall sentiment, rating
+- `AutoList`: All complaints/praises
+- `AutoSet`: Unique features mentioned
---
## Advanced: Custom Patterns
-You can create custom patterns by subclassing `BaseKnowledge`:
+You can create custom patterns by subclassing `BaseAutoType`:
```python
-from hyperextract.knowledge.base import BaseKnowledge
+from hyperextract.core.base import BaseAutoType
-class MyCustomPattern(BaseKnowledge[MySchema]):
+class MyCustomPattern(BaseAutoType[MySchema]):
def _merge_strategy(self, old_data, new_data):
# Implement custom merge logic
pass
diff --git a/docs/zh/index.md b/docs/zh/index.md
index cc21c48..af158a4 100644
--- a/docs/zh/index.md
+++ b/docs/zh/index.md
@@ -27,7 +27,7 @@ Hyper-Extract 中的每个知识容器都遵循统一的生命周期:
- **智能合并**:针对分块提取的优先级合并策略
### 🏗️ 统一架构
-- **抽象基类**:`BaseKnowledge[T]` 为所有知识类型提供一致的接口
+- **抽象基类**:`BaseAutoType[T]` 为所有知识类型提供一致的接口
- **泛型类型支持**:使用 `Generic[T]` 实现完整的 Python 类型提示
- **生命周期管理**:标准化的 `extract` → `merge` → `evolve` → `search` → `dump/load` 流程
@@ -39,15 +39,14 @@ Hyper-Extract 中的每个知识容器都遵循统一的生命周期:
### 📦 知识模式
#### ✅ 已实现
-- **UnitKnowledge**:单对象提取,适用于文档级信息(摘要、元数据)
-- **ListKnowledge**:列表提取,适用于实体、事件和结构化列表
-- **SetKnowledge**:唯一集合,支持自动去重和智能合并策略
+- **AutoModel**:单对象提取,适用于文档级信息(摘要、元数据)
+- **AutoList**:列表提取,适用于实体、事件和结构化列表
+- **AutoSet**:唯一集合,支持自动去重和智能合并策略
#### 🚧 开发中
-- **EntityKnowledge**:实体中心的提取,支持属性管理
-- **RelationKnowledge**:关系提取,支持实体链接
- **GraphKnowledge**:知识图谱构建(节点 + 边)
- **HypergraphKnowledge**:超图表示,用于复杂的多实体关系
+- **AutoTree**:层级树结构,用于嵌套数据
---
@@ -95,14 +94,14 @@ class ArticleKnowledge(BaseModel):
```python
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract.core import AutoModel
# 初始化 LLM 和嵌入模型
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
# 创建知识容器
-knowledge = UnitKnowledge(
+knowledge = AutoModel(
data_schema=ArticleKnowledge,
llm_client=llm,
embedder=embedder
@@ -140,7 +139,7 @@ for doc, score in results:
knowledge.dump("./my_knowledge")
# 加载
-loaded_knowledge = UnitKnowledge.load(
+loaded_knowledge = AutoModel.load(
"./my_knowledge",
data_schema=ArticleKnowledge,
llm_client=llm,
diff --git a/examples/list_knowledge_demo.py b/examples/auto_list_demo.py
similarity index 88%
rename from examples/list_knowledge_demo.py
rename to examples/auto_list_demo.py
index 31b5de8..49e9384 100644
--- a/examples/list_knowledge_demo.py
+++ b/examples/auto_list_demo.py
@@ -1,7 +1,7 @@
"""
-ListKnowledge 多 Chunk 提取示例
+AutoList 多 Chunk 提取示例
-演示如何使用 ListKnowledge 处理长文本,测试多 chunk 并发提取和列表合并功能。
+演示如何使用 AutoList 处理长文本,测试多 chunk 并发提取和列表合并功能。
"""
import sys
@@ -14,7 +14,7 @@ sys.path.insert(0, str(project_root))
from pydantic import BaseModel, Field
from typing import Optional
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import ListKnowledge
+from hyperextract import AutoList
# 定义单个新闻事件的 Schema
@@ -125,7 +125,7 @@ Jeff Bezos在发布会上表示,新一代无人机可承载5公斤货物,飞
def main():
print("=" * 80)
- print("ListKnowledge 多 Chunk 提取示例")
+ print("AutoList 多 Chunk 提取示例")
print("=" * 80)
# 1. 生成长文本
@@ -139,9 +139,9 @@ def main():
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
- # 3. 创建 ListKnowledge 实例 (使用小的 chunk_size 测试多 chunk)
- print("\n[3] 创建 ListKnowledge 实例 (chunk_size=300)...")
- knowledge = ListKnowledge(
+ # 3. 创建 AutoList 实例 (使用小的 chunk_size 测试多 chunk)
+ print("\n[3] 创建 AutoList 实例 (chunk_size=300)...")
+ a_event_list = AutoList(
item_schema=NewsEvent,
llm_client=llm,
embedder=embedder,
@@ -154,7 +154,7 @@ def main():
# 4. 提取知识 (默认存储模式)
print("\n[4] 提取新闻事件列表 (默认存储模式)...")
print("-" * 80)
- events = knowledge.extract(news_text)
+ events = a_event_list.extract(news_text)
print("-" * 80)
# 5. 显示提取结果
@@ -173,11 +173,11 @@ def main():
print(f"\n ... 还有 {len(events) - 5} 个事件未显示")
# 6. 测试列表大小
- print(f"\n[6] 知识列表大小: {len(knowledge)} 个事件")
+ print(f"\n[6] 知识列表大小: {len(a_event_list)} 个事件")
# 7. 构建索引
print("\n[7] 构建向量索引...")
- knowledge.build_index()
+ a_event_list.build_index()
# 8. 测试搜索
print("\n[8] 测试事件搜索...")
@@ -189,7 +189,7 @@ def main():
for query in queries:
print(f"\n 查询: '{query}'")
- results = knowledge.search(query, top_k=3)
+ results = a_event_list.search(query, top_k=3)
for i, event in enumerate(results, 1):
print(f" [{i}] {event.title} ({event.date})")
print(f" 地点: {event.location}")
@@ -204,9 +204,9 @@ def main():
"""
print(" 添加补充新闻...")
- before_count = len(knowledge)
- knowledge.extract(additional_news) # 默认 store=True,自动合并
- after_count = len(knowledge)
+ before_count = len(a_event_list)
+ a_event_list.extract(additional_news) # 默认 store=True,自动合并
+ after_count = len(a_event_list)
print(f" 合并前: {before_count} 个事件")
print(f" 合并后: {after_count} 个事件")
print(f" 新增: {after_count - before_count} 个事件")
@@ -214,32 +214,32 @@ def main():
# 10. 显示最新添加的事件
if after_count > before_count:
print("\n 最新添加的事件:")
- for event in knowledge.items[-3:]:
+ for event in a_event_list.items[-3:]:
print(f" - {event.title} ({event.date})")
- knowledge.build_index()
+ a_event_list.build_index()
# 11. 保存知识
print("\n[10] 保存知识到文件...")
- save_path = Path(__file__).parent.parent / "tmp" / "list_knowledge_multi_chunk"
- knowledge.dump(str(save_path))
+ save_path = Path(__file__).parent.parent / "tmp" / "auto_list_demo"
+ a_event_list.dump(str(save_path))
print(f" 已保存到: {save_path}")
# 12. 重新加载测试
print("\n[11] 测试知识加载...")
- new_knowledge = ListKnowledge(
+ a_loaded_event_list = AutoList(
item_schema=NewsEvent,
llm_client=llm,
embedder=embedder,
)
- new_knowledge.load(str(save_path))
- print(f" 加载成功! 事件数: {len(new_knowledge)}")
- print(f" 第一个事件: {new_knowledge.items[0].title}")
+ a_loaded_event_list.load(str(save_path))
+ print(f" 加载成功! 事件数: {len(a_loaded_event_list)}")
+ print(f" 第一个事件: {a_loaded_event_list.items[0].title}")
# 13. 测试清空功能
print("\n[12] 测试清空功能...")
- print(f" 清空前: {len(new_knowledge)} 个事件")
- new_knowledge.clear()
- print(f" 清空后: {len(new_knowledge)} 个事件")
+ print(f" 清空前: {len(a_loaded_event_list)} 个事件")
+ a_loaded_event_list.clear()
+ print(f" 清空后: {len(a_loaded_event_list)} 个事件")
print("\n" + "=" * 80)
print("示例运行完成!")
diff --git a/examples/list_operations_demo.py b/examples/auto_list_operations_demo.py
similarity index 66%
rename from examples/list_operations_demo.py
rename to examples/auto_list_operations_demo.py
index 132679d..e302f20 100644
--- a/examples/list_operations_demo.py
+++ b/examples/auto_list_operations_demo.py
@@ -1,7 +1,7 @@
"""
-演示 ListKnowledge 的 Pythonic 列表操作
+演示 AutoList 的 Pythonic 列表操作
-这个示例展示了如何像使用 Python 原生 list 一样使用 ListKnowledge:
+这个示例展示了如何像使用 Python 原生 list 一样使用 AutoList:
- 索引访问和切片
- 迭代和成员测试
- 追加、插入和扩展
@@ -22,7 +22,7 @@ from dotenv import load_dotenv
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import ListKnowledge
+from hyperextract import AutoList
# 加载环境变量
load_dotenv()
@@ -49,55 +49,55 @@ def main():
api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL")
)
- # 创建 ListKnowledge 实例
- tasks = ListKnowledge(
+ # 创建 AutoList 实例
+ a_task_list = AutoList(
item_schema=Task,
llm_client=llm,
embedder=embedder,
)
print("=" * 70)
- print("ListKnowledge Pythonic 操作演示")
+ print("AutoList Pythonic 操作演示")
print("=" * 70)
# 1. 使用 append 添加任务
print("\n1. 使用 append() 添加任务:")
- tasks.append(Task(name="实现用户登录", priority=5, status="done"))
- tasks.append(Task(name="设计数据库", priority=4, status="done"))
- tasks.append(Task(name="编写API文档", priority=3, status="in_progress"))
- print(f" 已添加 {len(tasks)} 个任务")
+ a_task_list.append(Task(name="实现用户登录", priority=5, status="done"))
+ a_task_list.append(Task(name="设计数据库", priority=4, status="done"))
+ a_task_list.append(Task(name="编写API文档", priority=3, status="in_progress"))
+ print(f" 已添加 {len(a_task_list)} 个任务")
# 2. 索引访问
print("\n2. 索引访问:")
- print(f" 第一个任务: {tasks[0].name} (优先级 {tasks[0].priority})")
- print(f" 最后一个任务: {tasks[-1].name} (状态: {tasks[-1].status})")
+ print(f" 第一个任务: {a_task_list[0].name} (优先级 {a_task_list[0].priority})")
+ print(f" 最后一个任务: {a_task_list[-1].name} (状态: {a_task_list[-1].status})")
# 3. 切片
print("\n3. 切片操作:")
- high_priority = tasks[:2]
+ high_priority = a_task_list[:2]
print(f" 前两个高优先级任务:")
for task in high_priority:
print(f" - {task.name} (P{task.priority})")
# 4. 迭代
print("\n4. 遍历所有任务:")
- for i, task in enumerate(tasks, 1):
+ for i, task in enumerate(a_task_list, 1):
print(f" {i}. {task.name} - {task.status}")
# 5. 使用列表推导式
print("\n5. 列表推导式 - 获取所有任务名称:")
- task_names = [task.name for task in tasks]
+ task_names = [task.name for task in a_task_list]
print(f" {task_names}")
# 6. 成员测试
print("\n6. 成员测试 (in 操作符):")
test_task = Task(name="编写API文档", priority=3, status="in_progress")
- print(f" 测试任务在列表中: {test_task in tasks}")
+ print(f" 测试任务在列表中: {test_task in a_task_list}")
# 7. 插入任务
print("\n7. 使用 insert() 在开头插入紧急任务:")
- tasks.insert(0, Task(name="修复生产BUG", priority=5, status="in_progress"))
- print(f" 新的第一个任务: {tasks[0].name}")
+ a_task_list.insert(0, Task(name="修复生产BUG", priority=5, status="in_progress"))
+ print(f" 新的第一个任务: {a_task_list[0].name}")
# 8. 扩展多个任务
print("\n8. 使用 extend() 批量添加任务:")
@@ -105,79 +105,79 @@ def main():
Task(name="代码审查", priority=3, status="pending"),
Task(name="性能优化", priority=2, status="pending"),
]
- tasks.extend(new_tasks)
- print(f" 现在共有 {len(tasks)} 个任务")
+ a_task_list.extend(new_tasks)
+ print(f" 现在共有 {len(a_task_list)} 个任务")
# 9. 查找和计数
print("\n9. 查找和计数:")
- in_progress_count = sum(1 for t in tasks if t.status == "in_progress")
+ in_progress_count = sum(1 for t in a_task_list if t.status == "in_progress")
print(f" 进行中的任务数量: {in_progress_count}")
# 查找第一个 pending 任务的位置
- for i, task in enumerate(tasks):
+ for i, task in enumerate(a_task_list):
if task.status == "pending":
print(f" 第一个待处理任务位置: 索引 {i} - {task.name}")
break
# 10. 修改任务
print("\n10. 修改任务状态 (通过索引赋值):")
- print(f" 修改前: {tasks[-1].name} - {tasks[-1].status}")
- updated_task = tasks[-1]
+ print(f" 修改前: {a_task_list[-1].name} - {a_task_list[-1].status}")
+ updated_task = a_task_list[-1]
updated_task.status = "done"
- tasks[-1] = updated_task
- print(f" 修改后: {tasks[-1].name} - {tasks[-1].status}")
+ a_task_list[-1] = updated_task
+ print(f" 修改后: {a_task_list[-1].name} - {a_task_list[-1].status}")
# 11. 删除任务
print("\n11. 删除已完成的任务:")
- print(f" 删除前共有 {len(tasks)} 个任务")
+ print(f" 删除前共有 {len(a_task_list)} 个任务")
# 删除第一个任务
- del tasks[0]
- print(f" 删除后共有 {len(tasks)} 个任务")
+ del a_task_list[0]
+ print(f" 删除后共有 {len(a_task_list)} 个任务")
# 12. Pop 操作
print("\n12. 使用 pop() 移除并返回任务:")
- last_task = tasks.pop()
+ last_task = a_task_list.pop()
print(f" 弹出的任务: {last_task.name}")
- print(f" 剩余 {len(tasks)} 个任务")
+ print(f" 剩余 {len(a_task_list)} 个任务")
# 13. 复制
print("\n13. 创建任务列表的副本:")
- tasks_backup = tasks.copy()
+ tasks_backup = a_task_list.copy()
tasks_backup.append(Task(name="新任务", priority=1, status="pending"))
- print(f" 原始列表: {len(tasks)} 个任务")
+ print(f" 原始列表: {len(a_task_list)} 个任务")
print(f" 副本列表: {len(tasks_backup)} 个任务")
print(" ✓ 副本是独立的")
# 14. 排序
print("\n14. 按优先级排序任务 (降序):")
- tasks.sort(key=lambda t: t.priority, reverse=True)
+ a_task_list.sort(key=lambda t: t.priority, reverse=True)
print(" 排序后的任务列表:")
- for task in tasks:
+ for task in a_task_list:
print(f" - P{task.priority}: {task.name}")
# 15. 反转
print("\n15. 反转任务列表:")
- tasks.reverse()
+ a_task_list.reverse()
print(" 反转后:")
- for task in tasks:
+ for task in a_task_list:
print(f" - {task.name}")
# 16. 字符串表示
print("\n16. 字符串表示:")
- print(f" repr: {repr(tasks)}")
- print(f" str: {str(tasks)}")
+ print(f" repr: {repr(a_task_list)}")
+ print(f" str: {str(a_task_list)}")
# 17. 组合操作
- print("\n17. 组合两个 ListKnowledge (+ 操作符):")
- more_tasks = ListKnowledge(
+ print("\n17. 组合两个 AutoList (+ 操作符):")
+ more_tasks = AutoList(
item_schema=Task,
llm_client=llm,
embedder=embedder,
)
more_tasks.append(Task(name="部署到生产", priority=5, status="pending"))
- combined = tasks + more_tasks
- print(f" 原始任务: {len(tasks)} 个")
+ combined = a_task_list + more_tasks
+ print(f" 原始任务: {len(a_task_list)} 个")
print(f" 新增任务: {len(more_tasks)} 个")
print(f" 组合后: {len(combined)} 个")
diff --git a/examples/unit_knowledge_demo.py b/examples/auto_model_demo.py
similarity index 87%
rename from examples/unit_knowledge_demo.py
rename to examples/auto_model_demo.py
index 0435f74..f85cd02 100644
--- a/examples/unit_knowledge_demo.py
+++ b/examples/auto_model_demo.py
@@ -1,7 +1,7 @@
"""
-UnitKnowledge 多 Chunk 提取示例
+AutoModel 多 Chunk 提取示例
-演示如何使用 UnitKnowledge 处理长文本,测试多 chunk 并发提取和合并功能。
+演示如何使用 AutoModel 处理长文本,测试多 chunk 并发提取和合并功能。
"""
import sys
@@ -14,11 +14,11 @@ sys.path.insert(0, str(project_root))
from pydantic import BaseModel, Field
from typing import List, Optional
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import UnitKnowledge
+from hyperextract import AutoModel
# 定义文档知识 Schema
-class DocumentKnowledge(BaseModel):
+class DocumentSchema(BaseModel):
"""文档综合知识结构"""
title: str = Field(default="", description="文档标题")
@@ -135,7 +135,7 @@ AI医疗诊断系统的可信度和安全性将进一步提升。
def main():
print("=" * 80)
- print("UnitKnowledge 多 Chunk 提取示例")
+ print("AutoModel 多 Chunk 提取示例")
print("=" * 80)
# 1. 生成长文档
@@ -149,10 +149,10 @@ def main():
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(model="text-embedding-3-small")
- # 3. 创建 UnitKnowledge 实例 (使用小的 chunk_size 测试多 chunk)
- print("\n[3] 创建 UnitKnowledge 实例 (chunk_size=200)...")
- knowledge = UnitKnowledge(
- data_schema=DocumentKnowledge,
+ # 3. 创建 AutoModel 实例 (使用小的 chunk_size 测试多 chunk)
+ print("\n[3] 创建 AutoModel 实例 (chunk_size=200)...")
+ a_document_model = AutoModel(
+ data_schema=DocumentSchema,
llm_client=llm,
embedder=embedder,
chunk_size=200, # 设置小的 chunk_size 来测试多 chunk 提取
@@ -164,7 +164,7 @@ def main():
# 4. 提取知识 (默认存储模式)
print("\n[4] 提取知识 (默认存储模式)...")
print("-" * 80)
- extracted_data = knowledge.extract(document)
+ extracted_data = a_document_model.extract(document)
print("-" * 80)
# 5. 显示提取结果
@@ -181,11 +181,11 @@ def main():
print(f" 关键词数量: {len(extracted_data.keywords)}")
# 6. 测试字段数量
- print(f"\n[6] 知识字段数: {len(knowledge)}")
+ print(f"\n[6] 知识字段数: {len(a_document_model)}")
# 7. 构建索引
print("\n[7] 构建向量索引...")
- knowledge.build_index()
+ a_document_model.build_index()
# 8. 测试搜索
print("\n[8] 测试知识搜索...")
@@ -197,7 +197,7 @@ def main():
for query in queries:
print(f"\n 查询: '{query}'")
- results = knowledge.search(query, top_k=2)
+ results = a_document_model.search(query, top_k=2)
for i, result in enumerate(results, 1):
field_name = list(result.keys())[0]
field_value = result[field_name]
@@ -211,25 +211,25 @@ def main():
研究团队来自清华大学医学院和北京协和医院。
"""
print(" 添加补充文本...")
- knowledge.extract(additional_text) # 默认 store=True,自动合并
- print(f" 更新后的作者信息: {knowledge.data.author}")
+ a_document_model.extract(additional_text) # 默认 store=True,自动合并
+ print(f" 更新后的作者信息: {a_document_model.data.author}")
# 10. 保存知识
print("\n[10] 保存知识到文件...")
- save_path = Path(__file__).parent.parent / "tmp" / "unit_knowledge_multi_chunk"
- knowledge.dump(str(save_path))
+ save_path = Path(__file__).parent.parent / "tmp" / "auto_model_demo"
+ a_document_model.dump(str(save_path))
print(f" 已保存到: {save_path}")
# 11. 重新加载测试
print("\n[11] 测试知识加载...")
- new_knowledge = UnitKnowledge(
- data_schema=DocumentKnowledge,
+ a_loaded_document_model = AutoModel(
+ data_schema=DocumentSchema,
llm_client=llm,
embedder=embedder,
)
- new_knowledge.load(str(save_path))
- print(f" 加载成功! 字段数: {len(new_knowledge)}")
- print(f" 标题: {new_knowledge.data.title}")
+ a_loaded_document_model.load(str(save_path))
+ print(f" 加载成功! 字段数: {len(a_loaded_document_model)}")
+ print(f" 标题: {a_loaded_document_model.data.title}")
print("\n" + "=" * 80)
print("示例运行完成!")
diff --git a/examples/set_knowledge_demo.py b/examples/auto_set_demo.py
similarity index 89%
rename from examples/set_knowledge_demo.py
rename to examples/auto_set_demo.py
index b372453..05d3252 100644
--- a/examples/set_knowledge_demo.py
+++ b/examples/auto_set_demo.py
@@ -1,6 +1,6 @@
"""Set Knowledge Demo - Extracting unique skills from multiple job descriptions
-This example demonstrates how SetKnowledge automatically deduplicates
+This example demonstrates how AutoSet automatically deduplicates
extracted items based on a unique key field.
"""
@@ -15,7 +15,7 @@ import os
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic.set import SetKnowledge
+from hyperextract import AutoSet
from hyperextract.utils.merger import MergeStrategy
@@ -44,10 +44,10 @@ def main():
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embedder = OpenAIEmbeddings(base_url=os.getenv("OPENAI_BASE_URL"))
- # Create SetKnowledge with FIELD_MERGE strategy
+ # Create AutoSet with FIELD_MERGE strategy
# This means when the same skill appears multiple times,
# the new extraction will fill in any missing fields from the existing one
- skills = SetKnowledge(
+ a_skill_set = AutoSet(
item_schema=SkillSchema,
llm_client=llm,
embedder=embedder,
@@ -117,17 +117,17 @@ Be specific and extract all mentioned skills.""",
print("\n📄 Processing job descriptions...")
for i, job_desc in enumerate(job_descriptions, 1):
print(f"\n Processing job description {i}...")
- skills.extract(job_desc)
- print(f" ✓ Total unique skills so far: {len(skills)}")
+ a_skill_set.extract(job_desc)
+ print(f" ✓ Total unique skills so far: {len(a_skill_set)}")
# Display all unique skills
print("\n" + "=" * 80)
- print(f"📊 EXTRACTED UNIQUE SKILLS: {len(skills)} total")
+ print(f"📊 EXTRACTED UNIQUE SKILLS: {len(a_skill_set)} total")
print("=" * 80)
# Group by category
skills_by_category = {}
- for skill in skills.items:
+ for skill in a_skill_set.items:
category = skill.category or "Uncategorized"
if category not in skills_by_category:
skills_by_category[category] = []
@@ -150,12 +150,12 @@ Be specific and extract all mentioned skills.""",
# Build index for semantic search
print("\nBuilding vector index for semantic search...")
- skills.build_index()
+ a_skill_set.build_index()
print("✓ Index built successfully")
# Search for related skills
print("\n1. Semantic Search - Finding skills related to 'backend development':")
- backend_skills = skills.search("backend development", top_k=5)
+ backend_skills = a_skill_set.search("backend development", top_k=5)
for skill in backend_skills:
print(f" • {skill.name}")
if skill.category:
@@ -165,13 +165,13 @@ Be specific and extract all mentioned skills.""",
print("\n2. Membership Testing:")
test_skills = ["Python", "Java", "Docker", "Rust"]
for skill_name in test_skills:
- exists = skills.contains(skill_name)
+ exists = a_skill_set.contains(skill_name)
status = "✓" if exists else "✗"
print(f" {status} {skill_name}: {'Found' if exists else 'Not found'}")
# Get specific skill details
print("\n3. Get Specific Skill Details:")
- python_skill = skills.get("Python")
+ python_skill = a_skill_set.get("Python")
if python_skill:
print(f" Skill: {python_skill.name}")
print(f" Category: {python_skill.category}")
@@ -183,28 +183,28 @@ Be specific and extract all mentioned skills.""",
print("💾 SAVING EXTRACTED KNOWLEDGE")
print("=" * 80)
- save_path = "tmp/set_knowledge_demo"
- skills.dump(save_path)
+ save_path = "tmp/auto_set_demo"
+ a_skill_set.dump(save_path)
print(f"✓ Saved to: {save_path}")
print(" - state.json (structured data)")
print(" - faiss_index/ (vector index for semantic search)")
# Demonstrate loading
print("\n📂 Loading saved knowledge...")
- loaded_skills = SetKnowledge(
+ a_loaded_skill_set = AutoSet(
item_schema=SkillSchema,
llm_client=llm,
embedder=embedder,
key_extractor=lambda x: x.name,
)
- loaded_skills.load(save_path)
- print(f"✓ Loaded {len(loaded_skills)} unique skills")
+ a_loaded_skill_set.load(save_path)
+ print(f"✓ Loaded {len(a_loaded_skill_set)} unique skills")
# Summary
print("\n" + "=" * 80)
print("📈 SUMMARY")
print("=" * 80)
- print(f" Total unique skills extracted: {len(skills)}")
+ print(f" Total unique skills extracted: {len(a_skill_set)}")
print(f" Job descriptions processed: {len(job_descriptions)}")
print(" Deduplication strategy: FIELD_MERGE")
print(" Key extractor: lambda x: x.name")
diff --git a/examples/generic_knowledge_demo.py b/examples/generic_knowledge_demo.py
deleted file mode 100644
index 1a6bb81..0000000
--- a/examples/generic_knowledge_demo.py
+++ /dev/null
@@ -1,130 +0,0 @@
-"""
-UnitKnowledge 使用示例
-
-演示如何使用 UnitKnowledge 类进行知识提取、搜索、保存和加载。
-"""
-
-import sys
-from pathlib import Path
-
-# 添加项目根目录到 Python 路径
-project_root = Path(__file__).parent.parent
-sys.path.insert(0, str(project_root))
-
-from pydantic import BaseModel, Field
-from typing import List
-from langchain_openai import ChatOpenAI, OpenAIEmbeddings
-from hyperextract.knowledge.generic import UnitKnowledge
-
-
-# 1. 定义知识 Schema
-class ArticleKnowledge(BaseModel):
- """文章知识结构"""
-
- title: str = Field(default="", description="文章标题")
- author: str = Field(default="", description="作者")
- summary: str = Field(default="", description="摘要")
- key_points: List[str] = Field(default_factory=list, description="关键要点")
- topics: List[str] = Field(default_factory=list, description="主题标签")
-
-
-def main():
- print("=" * 60)
- print("UnitKnowledge 使用示例")
- print("=" * 60)
-
- # 2. 初始化 LLM 和 Embedder
- print("\n[1] 初始化 LLM 和 Embedder...")
- llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
- embedder = OpenAIEmbeddings(model="text-embedding-3-small")
-
- # 3. 创建 UnitKnowledge 实例
- print("[2] 创建 UnitKnowledge 实例...")
- knowledge = UnitKnowledge(
- data_schema=ArticleKnowledge,
- llm_client=llm,
- embedder=embedder,
- prompt="", # 使用默认 prompt
- chunk_size=2000,
- max_workers=5,
- show_progress=True,
- )
-
- # 4. 提取知识
- print("\n[3] 从文本中提取知识...")
- sample_text = """
- Title: The Future of Artificial Intelligence
- Author: Dr. Jane Smith
-
- Artificial intelligence (AI) is rapidly transforming our world. This groundbreaking
- technology has the potential to revolutionize industries ranging from healthcare to
- transportation. In this article, we explore the key developments and future directions
- of AI research.
-
- Key Points:
- 1. Machine learning algorithms are becoming increasingly sophisticated
- 2. AI is being applied to solve complex real-world problems
- 3. Ethical considerations are crucial in AI development
- 4. Collaboration between humans and AI will define the future
-
- Topics: artificial intelligence, machine learning, ethics, future technology
- """
-
- result = knowledge.extract(sample_text)
- print(f"✓ 提取完成: {result}")
-
- # 5. 查看提取的数据
- print("\n[4] 查看提取的知识数据:")
- extracted_data = knowledge.data
- print(f" 标题: {extracted_data.title}")
- print(f" 作者: {extracted_data.author}")
- print(f" 摘要: {extracted_data.summary[:100]}...")
- print(f" 关键要点数量: {len(extracted_data.key_points)}")
- print(f" 主题标签: {', '.join(extracted_data.topics)}")
-
- # 6. 构建索引
- print("\n[5] 构建向量索引...")
- knowledge.build_index()
- print(f"✓ 索引构建完成,共 {len(knowledge)} 个字段")
-
- # 7. 语义搜索
- print("\n[6] 执行语义搜索...")
- search_query = "What are the main topics of this article?"
- results = knowledge.search(search_query, top_k=3)
- print(f" 查询: {search_query}")
- print(f" 搜索结果数量: {len(results)}")
- for i, result in enumerate(results, 1):
- print(f" 结果 {i}: {result}")
-
- # 8. 保存知识
- print("\n[7] 保存知识到文件夹...")
- save_path = Path(__file__).parent.parent / "tmp" / "article_knowledge"
- knowledge.dump(save_path)
- print(f"✓ 知识已保存到: {save_path}")
-
- # 9. 加载知识
- print("\n[8] 从文件夹加载知识...")
- new_knowledge = UnitKnowledge(
- data_schema=ArticleKnowledge,
- llm_client=llm,
- embedder=embedder,
- )
- new_knowledge.load(save_path)
- print("✓ 知识加载完成")
- print(f" 加载的标题: {new_knowledge.data.title}")
- print(f" 加载的字段数: {len(new_knowledge)}")
-
- # 10. 测试长文本提取(分块)
- print("\n[9] 测试长文本提取(自动分块)...")
- long_text = sample_text * 5 # 重复文本模拟长文本
- knowledge.clear()
- result = knowledge.extract(long_text)
- print(f"✓ 长文本提取完成: {result}")
-
- print("\n" + "=" * 60)
- print("示例执行完成!")
- print("=" * 60)
-
-
-if __name__ == "__main__":
- main()
diff --git a/hyperextract/__init__.py b/hyperextract/__init__.py
index e69de29..032ad1f 100644
--- a/hyperextract/__init__.py
+++ b/hyperextract/__init__.py
@@ -0,0 +1,15 @@
+"""HyperExtract - AI-powered data structure extraction from unstructured text.
+
+This library provides Auto-prefixed intelligent data structures that automatically
+extract structured information from text using Large Language Models (LLMs).
+"""
+
+from .core import AutoModel, AutoList, AutoSet
+
+__version__ = "0.2.0"
+
+__all__ = [
+ "AutoModel",
+ "AutoList",
+ "AutoSet",
+]
diff --git a/hyperextract/core/__init__.py b/hyperextract/core/__init__.py
new file mode 100644
index 0000000..3c763a7
--- /dev/null
+++ b/hyperextract/core/__init__.py
@@ -0,0 +1,13 @@
+"""Core knowledge patterns - fundamental data structure patterns."""
+
+from .base import BaseAutoType
+from .model import AutoModel
+from .list import AutoList
+from .set import AutoSet
+
+__all__ = [
+ "BaseAutoType",
+ "AutoModel",
+ "AutoList",
+ "AutoSet",
+]
diff --git a/hyperextract/knowledge/base.py b/hyperextract/core/base.py
similarity index 98%
rename from hyperextract/knowledge/base.py
rename to hyperextract/core/base.py
index a0fbf20..ba1fdc4 100644
--- a/hyperextract/knowledge/base.py
+++ b/hyperextract/core/base.py
@@ -15,7 +15,7 @@ T = TypeVar("T", bound=BaseModel)
# ===================== Knowledge Base Class =====================
-class BaseKnowledge(ABC, Generic[T]):
+class BaseAutoType(ABC, Generic[T]):
"""Unified knowledge base class integrating extraction, storage, aggregation, and evolution.
This abstract base class provides a complete framework for managing structured knowledge
@@ -87,7 +87,7 @@ class BaseKnowledge(ABC, Generic[T]):
"updated_at": datetime.now(),
}
- def _create_new_instance(self) -> "BaseKnowledge[T]":
+ def _create_new_instance(self) -> "BaseAutoType[T]":
"""Creates a new instance with the same configuration as this one.
Subclasses can override this method if they have special initialization requirements.
@@ -278,7 +278,7 @@ class BaseKnowledge(ABC, Generic[T]):
# ==================== Operator Overloads ====================
- def __add__(self, other: "BaseKnowledge[T]") -> "BaseKnowledge[T]":
+ def __add__(self, other: "BaseAutoType[T]") -> "BaseAutoType[T]":
"""Operator overload for '+' to merge two knowledge instances.
Creates a new knowledge instance by merging the data from both instances.
diff --git a/hyperextract/knowledge/generic/list.py b/hyperextract/core/list.py
similarity index 91%
rename from hyperextract/knowledge/generic/list.py
rename to hyperextract/core/list.py
index 852345d..0c43039 100644
--- a/hyperextract/knowledge/generic/list.py
+++ b/hyperextract/core/list.py
@@ -10,7 +10,7 @@ from langchain_core.embeddings import Embeddings
from langchain_core.documents import Document
from langchain_community.vectorstores import FAISS
-from ..base import BaseKnowledge
+from .base import BaseAutoType
try:
from hyperextract.config import logger
@@ -23,14 +23,14 @@ except ImportError:
Item = TypeVar("Item", bound=BaseModel)
-class ItemListSchema(BaseModel, Generic[Item]):
+class AutoListSchema(BaseModel, Generic[Item]):
"""Generic schema container for list-based knowledge patterns."""
items: List[Item] = Field(default_factory=list, description="Item list")
-class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
- """List Knowledge Pattern - extracts a collection of objects from text.
+class AutoList(BaseAutoType[AutoListSchema[Item]], Generic[Item]):
+ """AutoList - extracts a collection of objects from text.
This pattern extracts multiple independent objects from a document, suitable for
extracting entities, events, references, or any collection of structured items.
@@ -40,14 +40,14 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
- Merge strategy: Append with basic deduplication (extensible by subclasses)
- Indexing strategy: Each item in the list is indexed independently
- Comparison with UnitKnowledge:
- - UnitKnowledge: Extracts a single structured object (e.g., summary, metadata)
- - ListKnowledge: Extracts multiple independent objects (e.g., entity list, event list)
+ Comparison with AutoModel:
+ - AutoModel: Extracts a single structured object (e.g., summary, metadata)
+ - AutoList: Extracts multiple independent objects (e.g., entity list, event list)
"""
if TYPE_CHECKING:
# Use generic version during type checking to maintain complete type hints
- item_list_schema: Type[ItemListSchema[Item]]
+ item_list_schema: Type[AutoListSchema[Item]]
def __init__(
self,
@@ -62,7 +62,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
show_progress: bool = True,
**kwargs,
):
- """Initialize ListKnowledge with item schema and configuration.
+ """Initialize AutoList with item schema and configuration.
Args:
item_schema: Pydantic BaseModel subclass for individual list items.
@@ -113,13 +113,13 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
"""Returns the internal list of extracted items."""
return getattr(self._data, "items", [])
- def _create_new_instance(self) -> "ListKnowledge[Item]":
+ def _create_new_instance(self) -> "AutoList[Item]":
"""Creates a new instance with the same configuration as this one.
- Overrides base class method to handle ListKnowledge's item_schema parameter.
+ Overrides base class method to handle AutoList's item_schema parameter.
Returns:
- A new ListKnowledge instance with the same configuration.
+ A new AutoList instance with the same configuration.
"""
return self.__class__(
item_schema=self.item_schema,
@@ -132,7 +132,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
show_progress=self.show_progress,
)
- def extract(self, text: str, *, store: bool = True) -> ItemListSchema:
+ def extract(self, text: str, *, store: bool = True) -> AutoListSchema:
"""Extracts a list of items using LangChain native batch processing.
Args:
@@ -192,18 +192,18 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
# Return items list (either from stored data or temporary merged data)
return self.items if store else merged_data.items
- def merge(self, data_list: List[ItemListSchema]) -> ItemListSchema:
+ def merge(self, data_list: List[AutoListSchema]) -> AutoListSchema:
"""Pure data merge method implementing list append strategy.
Merge strategy: Collects all items from all container objects and merges them
into a single list. Subclasses can override this method to implement more
- sophisticated deduplication logic (e.g., EntityKnowledge).
+ sophisticated deduplication logic (e.g., AutoSet with custom key_extractor).
Args:
data_list: List of container objects to merge.
Returns:
- A new merged ItemListSchema object.
+ A new merged AutoListSchema object.
"""
all_items = []
@@ -366,7 +366,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
Returns:
- For integer index: Returns the Item at that position
- - For slice: Returns a new ListKnowledge instance with sliced items
+ - For slice: Returns a new AutoList instance with sliced items
Raises:
IndexError: If index is out of range.
@@ -375,14 +375,14 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
Examples:
>>> knowledge[0] # First item
>>> knowledge[-1] # Last item
- >>> knowledge[1:3] # New ListKnowledge with items [1:3]
+ >>> knowledge[1:3] # New AutoList with items [1:3]
>>> knowledge[:5] # First 5 items as new instance
"""
if isinstance(key, int):
# Integer index: return Item directly
return self.items[key]
elif isinstance(key, slice):
- # Slice: return new ListKnowledge instance
+ # Slice: return new AutoList instance
new_instance = self._create_new_instance()
new_instance._data.items = self.items[key]
new_instance.metadata["updated_at"] = datetime.now()
@@ -420,28 +420,28 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
"""Operator overload for '+' to combine knowledge instances.
Supports multiple combination patterns:
- - ListKnowledge + ListKnowledge → ListKnowledge (merge lists)
- - ListKnowledge + UnitKnowledge → ListKnowledge (append unit to list)
+ - AutoList + AutoList → AutoList (merge lists)
+ - AutoList + UnitKnowledge → AutoList (append unit to list)
This enables chain operations like: unit1 + unit2 + unit3
Args:
- other: Another ListKnowledge or UnitKnowledge with compatible schema.
+ other: Another AutoList or UnitKnowledge with compatible schema.
Returns:
- New ListKnowledge with combined items.
+ New AutoList with combined items.
Raises:
TypeError: If schemas don't match or invalid operand type.
"""
- from .unit import UnitKnowledge
+ from .model import AutoModel
- # Case 1: ListKnowledge + ListKnowledge (call parent implementation)
- if isinstance(other, ListKnowledge):
+ # Case 1: AutoList + AutoList (call parent implementation)
+ if isinstance(other, AutoList):
# Check schema compatibility
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot add ListKnowledge instances with different schemas. "
+ f"Cannot add AutoList instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
# Manually merge without calling base class (to avoid _data_schema check)
@@ -452,16 +452,16 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
new_instance.metadata["updated_at"] = self.metadata.get("updated_at")
return new_instance
- # Case 2: ListKnowledge + UnitKnowledge → ListKnowledge (append unit)
- elif isinstance(other, UnitKnowledge):
+ # Case 2: AutoList + AutoModel → AutoList (append unit)
+ elif isinstance(other, AutoModel):
# Check schema compatibility
if self.item_schema != other._data_schema:
raise TypeError(
- f"Cannot add UnitKnowledge to ListKnowledge with different schemas. "
+ f"Cannot add AutoModel to AutoList with different schemas. "
f"List: {self.item_schema.__name__}, Unit: {other._data_schema.__name__}"
)
- # Create new ListKnowledge with appended unit
+ # Create new AutoList with appended unit
new_list = self._create_new_instance()
new_list._data.items = self.items + [other._data]
@@ -475,7 +475,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
else:
raise TypeError(
- f"Unsupported operand type for +: 'ListKnowledge' and '{type(other).__name__}'"
+ f"Unsupported operand type for +: 'AutoList' and '{type(other).__name__}'"
)
def __delitem__(self, index: int):
@@ -544,7 +544,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
Examples:
>>> repr(knowledge)
- 'ListKnowledge[PersonSchema](5 items)'
+ 'AutoList[PersonSchema](5 items)'
"""
return (
f"{self.__class__.__name__}[{self.item_schema.__name__}]({len(self)} items)"
@@ -558,7 +558,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
Examples:
>>> str(knowledge)
- 'ListKnowledge with 5 PersonSchema items'
+ 'AutoList with 5 PersonSchema items'
"""
return f"{self.__class__.__name__} with {len(self)} {self.item_schema.__name__} items"
@@ -591,7 +591,7 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
Args:
items: Iterable of items to append. Can be:
- List of items
- - Another ListKnowledge instance
+ - Another AutoList instance
- Any iterable yielding items
Raises:
@@ -605,11 +605,11 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
- Clears the vector index (needs rebuild)
- Updates metadata timestamp
"""
- # Handle ListKnowledge instance
- if isinstance(items, ListKnowledge):
+ # Handle AutoList instance
+ if isinstance(items, AutoList):
if self.item_schema != items.item_schema:
raise TypeError(
- f"Cannot extend with ListKnowledge of different schema. "
+ f"Cannot extend with AutoList of different schema. "
f"Expected: {self.item_schema.__name__}, "
f"Got: {items.item_schema.__name__}"
)
@@ -757,11 +757,11 @@ class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
count += 1
return count
- def copy(self) -> "ListKnowledge[Item]":
- """Create a deep copy of this ListKnowledge instance.
+ def copy(self) -> "AutoList[Item]":
+ """Create a deep copy of this AutoList instance.
Returns:
- A new ListKnowledge instance with copied items and metadata.
+ A new AutoList instance with copied items and metadata.
Note:
The vector index is not copied; it needs to be rebuilt if needed.
diff --git a/hyperextract/knowledge/generic/unit.py b/hyperextract/core/model.py
similarity index 92%
rename from hyperextract/knowledge/generic/unit.py
rename to hyperextract/core/model.py
index cf613c7..da7b40e 100644
--- a/hyperextract/knowledge/generic/unit.py
+++ b/hyperextract/core/model.py
@@ -9,7 +9,7 @@ from langchain_core.embeddings import Embeddings
from langchain_core.documents import Document
from langchain_community.vectorstores import FAISS
-from ..base import BaseKnowledge, T
+from .base import BaseAutoType, T
try:
from hyperextract.config import logger
@@ -19,8 +19,8 @@ except ImportError:
logger = logging.getLogger(__name__)
-class UnitKnowledge(BaseKnowledge[T]):
- """Unit Knowledge Pattern - extracts a single structured object from text.
+class AutoModel(BaseAutoType[T]):
+ """AutoModel - extracts a single structured object from text.
This pattern is designed for extracting exactly one structured object per document,
regardless of document length. Suitable for document-level information like summaries,
@@ -47,7 +47,7 @@ class UnitKnowledge(BaseKnowledge[T]):
show_progress: bool = True,
**kwargs,
):
- """Initialize UnitKnowledge with schema and configuration.
+ """Initialize AutoModel with schema and configuration.
Args:
data_schema: Pydantic BaseModel subclass defining the object structure.
@@ -327,22 +327,22 @@ class UnitKnowledge(BaseKnowledge[T]):
# ==================== Operators ====================
def __add__(self, other):
- """Operator overload for '+' to combine UnitKnowledge instances into ListKnowledge.
+ """Operator overload for '+' to combine AutoModel instances into ListKnowledge.
- When two UnitKnowledge instances are added, they are combined into a ListKnowledge
+ When two AutoModel instances are added, they are combined into a ListKnowledge
containing both objects as separate items. This enables intuitive collection building.
Usage:
- >>> unit1 = UnitKnowledge(PersonSchema, ...)
- >>> unit2 = UnitKnowledge(PersonSchema, ...)
+ >>> unit1 = AutoModel(PersonSchema, ...)
+ >>> unit2 = AutoModel(PersonSchema, ...)
>>> person_list = unit1 + unit2 # → ListKnowledge[PersonSchema]
>>>
>>> # Chain operations
- >>> unit3 = UnitKnowledge(PersonSchema, ...)
+ >>> unit3 = AutoModel(PersonSchema, ...)
>>> person_list = unit1 + unit2 + unit3 # → ListKnowledge with 3 items
Args:
- other: Another UnitKnowledge with the same data schema, or ListKnowledge.
+ other: Another AutoModel with the same data schema, or ListKnowledge.
Returns:
ListKnowledge containing both objects as items.
@@ -352,12 +352,12 @@ class UnitKnowledge(BaseKnowledge[T]):
"""
from .list import ListKnowledge
- # Case 1: UnitKnowledge + UnitKnowledge → ListKnowledge
- if isinstance(other, UnitKnowledge):
+ # Case 1: AutoModel + AutoModel → ListKnowledge
+ if isinstance(other, AutoModel):
# Check schema compatibility
if self._data_schema != other._data_schema:
raise TypeError(
- f"Cannot add UnitKnowledge instances with different schemas. "
+ f"Cannot add AutoModel instances with different schemas. "
f"Left: {self._data_schema.__name__}, Right: {other._data_schema.__name__}"
)
@@ -384,12 +384,12 @@ class UnitKnowledge(BaseKnowledge[T]):
return list_kb
- # Case 2: UnitKnowledge + ListKnowledge → ListKnowledge (for reverse order)
+ # Case 2: AutoModel + ListKnowledge → ListKnowledge (for reverse order)
elif isinstance(other, ListKnowledge):
# Check schema compatibility
if self._data_schema != other.item_schema:
raise TypeError(
- f"Cannot add UnitKnowledge to ListKnowledge with different schemas. "
+ f"Cannot add AutoModel to ListKnowledge with different schemas. "
f"Unit: {self._data_schema.__name__}, List: {other.item_schema.__name__}"
)
@@ -418,5 +418,5 @@ class UnitKnowledge(BaseKnowledge[T]):
else:
raise TypeError(
- f"Unsupported operand type for +: 'UnitKnowledge' and '{type(other).__name__}'"
+ f"Unsupported operand type for +: 'AutoModel' and '{type(other).__name__}'"
)
diff --git a/hyperextract/knowledge/generic/set.py b/hyperextract/core/set.py
similarity index 86%
rename from hyperextract/knowledge/generic/set.py
rename to hyperextract/core/set.py
index e551691..db9198f 100644
--- a/hyperextract/knowledge/generic/set.py
+++ b/hyperextract/core/set.py
@@ -13,7 +13,7 @@ from langchain_core.language_models.chat_models import BaseChatModel
from langchain_core.embeddings import Embeddings
from langchain_core.documents import Document
-from ..base import BaseKnowledge
+from .base import BaseAutoType
from hyperextract.utils.merger import (
BaseMerger,
MergeStrategy,
@@ -34,14 +34,14 @@ except ImportError:
Item = TypeVar("Item", bound=BaseModel)
-class ItemSetSchema(BaseModel, Generic[Item]):
+class AutoSetSchema(BaseModel, Generic[Item]):
"""Generic schema container for set-based knowledge patterns."""
items: List[Item] = Field(default_factory=list, description="Set of unique items")
-class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
- """Set Knowledge Pattern - extracts a unique collection of objects.
+class AutoSet(BaseAutoType[AutoSetSchema[Item]], Generic[Item]):
+ """AutoSet - extracts a unique collection of objects.
This pattern automatically deduplicates items based on a user-specified
key extractor function. Provides flexible merge strategies including LLM-powered
@@ -55,9 +55,9 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
- External interface: List (via items property)
- Set operations: union (|), intersection (&), difference (-)
- Comparison with ListKnowledge:
- - ListKnowledge: Allows duplicates, simple append merge
- - SetKnowledge: Automatic deduplication, intelligent merge strategies
+ Comparison with AutoList:
+ - AutoList: Allows duplicates, simple append merge
+ - AutoSet: Automatic deduplication, intelligent merge strategies
Example:
>>> class KeywordSchema(BaseModel):
@@ -65,7 +65,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
... category: str | None = None
... frequency: int | None = None
>>>
- >>> keywords = SetKnowledge(
+ >>> keywords = AutoSet(
... item_schema=KeywordSchema,
... llm_client=llm,
... embedder=embedder,
@@ -93,7 +93,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
llm_batch_size: int = 10,
**kwargs,
):
- """Initialize SetKnowledge with key extractor and merge strategy.
+ """Initialize AutoSet with key extractor and merge strategy.
Args:
item_schema: Pydantic BaseModel subclass for individual items.
@@ -113,7 +113,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
# Store item_schema
self.item_schema = item_schema
- # Create ItemSetSchema container dynamically (similar to ListKnowledge's ItemListSchema)
+ # Create AutoSetSchema container dynamically (similar to AutoList's AutoListSchema)
container_name = f"{item_schema.__name__}Set"
self.item_set_schema = create_model(
container_name,
@@ -123,7 +123,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
),
)
- # Call BaseKnowledge.__init__ (not ListKnowledge)
+ # Call BaseAutoType.__init__ (not ListKnowledge)
super().__init__(
data_schema=self.item_set_schema,
llm_client=llm_client,
@@ -136,7 +136,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
**kwargs,
)
- # SetKnowledge-specific attributes
+ # AutoSet-specific attributes
self.key_extractor = key_extractor
self.merge_item_strategy = merge_item_strategy
self.llm_batch_size = llm_batch_size
@@ -153,13 +153,13 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
# ==================== Override Instance Creation ====================
- def _create_new_instance(self) -> "SetKnowledge[Item]":
+ def _create_new_instance(self) -> "AutoSet[Item]":
"""Creates a new instance with the same configuration.
- Overrides parent method to include SetKnowledge-specific parameters.
+ Overrides parent method to include AutoSet-specific parameters.
Returns:
- New SetKnowledge instance.
+ New AutoSet instance.
"""
return self.__class__(
item_schema=self.item_schema,
@@ -270,11 +270,11 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
def __repr__(self) -> str:
"""Returns a developer-friendly representation."""
- return f"SetKnowledge[{self.item_schema.__name__}]({len(self)} unique items)"
+ return f"AutoSet[{self.item_schema.__name__}]({len(self)} unique items)"
def __str__(self) -> str:
"""Returns a user-friendly string representation."""
- return f"SetKnowledge with {len(self)} unique {self.item_schema.__name__} items"
+ return f"AutoSet with {len(self)} unique {self.item_schema.__name__} items"
def __iter__(self):
"""Enables iteration over all items in the set.
@@ -354,7 +354,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
# Return items list
return self.items if store else merged_data.items
- def merge(self, data_list: List[ItemSetSchema[Item]]) -> ItemSetSchema[Item]:
+ def merge(self, data_list: List[AutoSetSchema[Item]]) -> AutoSetSchema[Item]:
"""Merges multiple data containers with automatic deduplication.
Pure function: Does not modify internal state.
@@ -366,7 +366,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
data_list: List of container objects to merge.
Returns:
- New merged ItemSetSchema with deduplicated items.
+ New merged AutoSetSchema with deduplicated items.
"""
# Step 1: Collect all items from all containers
all_items = []
@@ -391,7 +391,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
# Step 3: Return new container (does not modify self)
return self.item_set_schema(items=merged_items)
- def _update_internal_state(self, data: ItemSetSchema[Item]) -> None:
+ def _update_internal_state(self, data: AutoSetSchema[Item]) -> None:
"""Updates internal state from a data container.
Private method that synchronizes _items_dict and _data with new data.
@@ -508,7 +508,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
def dump(self, folder_path: str | Path):
"""Exports knowledge to a specified folder.
- Saves both the items and the SetKnowledge-specific metadata.
+ Saves both the items and the AutoSet-specific metadata.
Args:
folder_path: Target folder path for saving.
@@ -706,7 +706,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
- Updates metadata timestamp
"""
if not self._items_dict:
- raise KeyError("pop from an empty SetKnowledge")
+ raise KeyError("pop from an empty AutoSet")
key = next(iter(self._items_dict))
item = self._items_dict.pop(key)
@@ -716,11 +716,11 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
return item
- def copy(self) -> "SetKnowledge[Item]":
+ def copy(self) -> "AutoSet[Item]":
"""Creates a deep copy of the set.
Returns:
- A new SetKnowledge instance with copies of all items.
+ A new AutoSet instance with copies of all items.
Examples:
>>> backup = skills.copy()
@@ -739,25 +739,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
# ==================== Set Operations ====================
- def __or__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def __or__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Union operation: set1 | set2.
Returns a new set containing all items from both sets.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
- New SetKnowledge with union of items.
+ New AutoSet with union of items.
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
raise TypeError(
- f"Unsupported operand type for |: 'SetKnowledge' and '{type(other).__name__}'"
+ f"Unsupported operand type for |: 'AutoSet' and '{type(other).__name__}'"
)
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot union SetKnowledge instances with different schemas. "
+ f"Cannot union AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
@@ -779,25 +779,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
return new_set
- def __and__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def __and__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Intersection operation: set1 & set2.
Returns a new set containing only items present in both sets.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
- New SetKnowledge with intersection of items.
+ New AutoSet with intersection of items.
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
raise TypeError(
- f"Unsupported operand type for &: 'SetKnowledge' and '{type(other).__name__}'"
+ f"Unsupported operand type for &: 'AutoSet' and '{type(other).__name__}'"
)
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot intersect SetKnowledge instances with different schemas. "
+ f"Cannot intersect AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
@@ -819,25 +819,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
return new_set
- def __sub__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def __sub__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Difference operation: set1 - set2.
Returns a new set containing items in self but not in other.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
- New SetKnowledge with difference of items.
+ New AutoSet with difference of items.
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
raise TypeError(
- f"Unsupported operand type for -: 'SetKnowledge' and '{type(other).__name__}'"
+ f"Unsupported operand type for -: 'AutoSet' and '{type(other).__name__}'"
)
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot subtract SetKnowledge instances with different schemas. "
+ f"Cannot subtract AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
@@ -854,25 +854,25 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
return new_set
- def __xor__(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def __xor__(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Symmetric difference operation: set1 ^ set2.
Returns a new set containing items in either set but not in both.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
- New SetKnowledge with symmetric difference of items.
+ New AutoSet with symmetric difference of items.
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
raise TypeError(
- f"Unsupported operand type for ^: 'SetKnowledge' and '{type(other).__name__}'"
+ f"Unsupported operand type for ^: 'AutoSet' and '{type(other).__name__}'"
)
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot compute symmetric difference of SetKnowledge instances with different schemas. "
+ f"Cannot compute symmetric difference of AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
@@ -896,19 +896,19 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
# ==================== Named Set Operations ====================
- def union(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def union(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Union operation (named method)."""
return self | other
- def intersection(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def intersection(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Intersection operation (named method)."""
return self & other
- def difference(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def difference(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Difference operation (named method)."""
return self - other
- def symmetric_difference(self, other: "SetKnowledge[Item]") -> "SetKnowledge[Item]":
+ def symmetric_difference(self, other: "AutoSet[Item]") -> "AutoSet[Item]":
"""Symmetric difference operation (named method)."""
return self ^ other
@@ -929,7 +929,7 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
Examples:
>>> skills1 == skills2 # True if same keys
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
return False
if self.item_schema != other.item_schema:
@@ -945,37 +945,37 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
"""
return not self.__eq__(other)
- def __le__(self, other: "SetKnowledge[Item]") -> bool:
+ def __le__(self, other: "AutoSet[Item]") -> bool:
"""Subset comparison: set1 <= set2.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if self is a subset of other (all keys in self are in other).
Raises:
- TypeError: If other is not a SetKnowledge or has different schema.
+ TypeError: If other is not a AutoSet or has different schema.
Examples:
>>> skills1 <= skills2 # True if skills1 is subset of skills2
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
return NotImplemented
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot compare SetKnowledge instances with different schemas. "
+ f"Cannot compare AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
return set(self._items_dict.keys()).issubset(set(other._items_dict.keys()))
- def __lt__(self, other: "SetKnowledge[Item]") -> bool:
+ def __lt__(self, other: "AutoSet[Item]") -> bool:
"""Proper subset comparison: set1 < set2.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if self is a proper subset of other (subset and not equal).
@@ -983,16 +983,16 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
Examples:
>>> skills1 < skills2 # True if skills1 is proper subset
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
return NotImplemented
return self <= other and self != other
- def __ge__(self, other: "SetKnowledge[Item]") -> bool:
+ def __ge__(self, other: "AutoSet[Item]") -> bool:
"""Superset comparison: set1 >= set2.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if self is a superset of other (all keys in other are in self).
@@ -1000,22 +1000,22 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
Examples:
>>> skills1 >= skills2 # True if skills1 is superset of skills2
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
return NotImplemented
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot compare SetKnowledge instances with different schemas. "
+ f"Cannot compare AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
return set(self._items_dict.keys()).issuperset(set(other._items_dict.keys()))
- def __gt__(self, other: "SetKnowledge[Item]") -> bool:
+ def __gt__(self, other: "AutoSet[Item]") -> bool:
"""Proper superset comparison: set1 > set2.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if self is a proper superset of other (superset and not equal).
@@ -1023,16 +1023,16 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
Examples:
>>> skills1 > skills2 # True if skills1 is proper superset
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
return NotImplemented
return self >= other and self != other
- def issubset(self, other: "SetKnowledge[Item]") -> bool:
+ def issubset(self, other: "AutoSet[Item]") -> bool:
"""Test whether every key in the set is in other.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if self is a subset of other.
@@ -1042,11 +1042,11 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
"""
return self <= other
- def issuperset(self, other: "SetKnowledge[Item]") -> bool:
+ def issuperset(self, other: "AutoSet[Item]") -> bool:
"""Test whether every key in other is in the set.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if self is a superset of other.
@@ -1056,29 +1056,29 @@ class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
"""
return self >= other
- def isdisjoint(self, other: "SetKnowledge[Item]") -> bool:
+ def isdisjoint(self, other: "AutoSet[Item]") -> bool:
"""Test whether the set has no keys in common with other.
Args:
- other: Another SetKnowledge instance.
+ other: Another AutoSet instance.
Returns:
True if the two sets have no keys in common.
Raises:
- TypeError: If other is not a SetKnowledge or has different schema.
+ TypeError: If other is not a AutoSet or has different schema.
Examples:
>>> skills1.isdisjoint(skills2) # True if no common skills
"""
- if not isinstance(other, SetKnowledge):
+ if not isinstance(other, AutoSet):
raise TypeError(
- f"isdisjoint() argument must be SetKnowledge, not {type(other).__name__}"
+ f"isdisjoint() argument must be AutoSet, not {type(other).__name__}"
)
if self.item_schema != other.item_schema:
raise TypeError(
- f"Cannot compare SetKnowledge instances with different schemas. "
+ f"Cannot compare AutoSet instances with different schemas. "
f"Left: {self.item_schema.__name__}, Right: {other.item_schema.__name__}"
)
diff --git a/hyperextract/knowledge/graph/__init__.py b/hyperextract/graphs/__init__.py
similarity index 100%
rename from hyperextract/knowledge/graph/__init__.py
rename to hyperextract/graphs/__init__.py
diff --git a/hyperextract/knowledge/graph/base.py b/hyperextract/graphs/base.py
similarity index 96%
rename from hyperextract/knowledge/graph/base.py
rename to hyperextract/graphs/base.py
index 1a27d1e..74bbe2f 100644
--- a/hyperextract/knowledge/graph/base.py
+++ b/hyperextract/graphs/base.py
@@ -1,4 +1,4 @@
-from ..base import BaseKnowledge
+from ..core.base import BaseAutoType
from typing import List, Optional, Dict, Any
from pydantic import BaseModel, Field
@@ -43,7 +43,7 @@ class GraphSchema(BaseModel):
return {n.id for n in self.nodes}
-class SimpleGraph(BaseKnowledge[GraphSchema]):
+class SimpleGraph(BaseAutoType[GraphSchema]):
def _get_schema_class(self):
return GraphSchema
diff --git a/hyperextract/knowledge/graph/kg_gen.py b/hyperextract/graphs/kg_gen.py
similarity index 100%
rename from hyperextract/knowledge/graph/kg_gen.py
rename to hyperextract/graphs/kg_gen.py
diff --git a/hyperextract/knowledge/hypergraph/__init__.py b/hyperextract/hypergraphs/__init__.py
similarity index 100%
rename from hyperextract/knowledge/hypergraph/__init__.py
rename to hyperextract/hypergraphs/__init__.py
diff --git a/hyperextract/knowledge/hypergraph/base.py b/hyperextract/hypergraphs/base.py
similarity index 100%
rename from hyperextract/knowledge/hypergraph/base.py
rename to hyperextract/hypergraphs/base.py
diff --git a/hyperextract/knowledge/hypergraph/hyperrag.py b/hyperextract/hypergraphs/hyperrag.py
similarity index 100%
rename from hyperextract/knowledge/hypergraph/hyperrag.py
rename to hyperextract/hypergraphs/hyperrag.py
diff --git a/hyperextract/knowledge/__init__.py b/hyperextract/knowledge/__init__.py
deleted file mode 100644
index 518926d..0000000
--- a/hyperextract/knowledge/__init__.py
+++ /dev/null
@@ -1,45 +0,0 @@
-"""Knowledge extraction framework.
-
-This package provides a hierarchical knowledge extraction framework:
- - generic: Base knowledge patterns (UnitKnowledge, ListKnowledge, SetKnowledge)
- - typical: Common knowledge types (EntityKnowledge, RelationKnowledge)
- - domain: Domain-specific knowledge patterns (reserved for future use)
- - graph: Graph-based knowledge structures
- - hypergraph: Hypergraph-based knowledge structures
-
-Usage:
- All main classes can be imported directly from hyperextract.knowledge:
-
- >>> from hyperextract.knowledge import UnitKnowledge, ListKnowledge, EntityKnowledge
- >>> from hyperextract.knowledge import SimpleGraph
-"""
-
-from .base import BaseKnowledge
-
-# Generic patterns
-from .generic import UnitKnowledge, ListKnowledge, ItemListSchema
-
-# Typical patterns
-from .typical import EntityKnowledge
-
-# Graph patterns
-from .graph.base import SimpleGraph, Entity, Relationship, GraphSchema
-
-__all__ = [
- # Base
- "BaseKnowledge",
-
- # Generic patterns
- "UnitKnowledge",
- "ListKnowledge",
- "ItemListSchema",
-
- # Typical patterns
- "EntityKnowledge",
-
- # Graph patterns
- "SimpleGraph",
- "Entity",
- "Relationship",
- "GraphSchema",
-]
diff --git a/hyperextract/knowledge/domain/__init__.py b/hyperextract/knowledge/domain/__init__.py
deleted file mode 100644
index b3e6ce6..0000000
--- a/hyperextract/knowledge/domain/__init__.py
+++ /dev/null
@@ -1,11 +0,0 @@
-"""Domain-specific knowledge patterns.
-
-This module is reserved for domain-specific knowledge implementations such as:
- - DocumentKnowledge: Document-level knowledge (title, author, sections, etc.)
- - ConversationKnowledge: Multi-turn conversation knowledge
- - AcademicPaperKnowledge: Academic paper structure and metadata
- - CodeKnowledge: Source code analysis and documentation
- - etc.
-"""
-
-__all__ = []
diff --git a/hyperextract/knowledge/generic/__init__.py b/hyperextract/knowledge/generic/__init__.py
deleted file mode 100644
index 79fca3e..0000000
--- a/hyperextract/knowledge/generic/__init__.py
+++ /dev/null
@@ -1,12 +0,0 @@
-"""Generic knowledge patterns - fundamental data structure patterns."""
-
-from .unit import UnitKnowledge
-from .list import ListKnowledge, ItemListSchema
-from .set import SetKnowledge
-
-__all__ = [
- "UnitKnowledge",
- "ListKnowledge",
- "ItemListSchema",
- "SetKnowledge",
-]
diff --git a/hyperextract/knowledge/typical/__init__.py b/hyperextract/knowledge/typical/__init__.py
deleted file mode 100644
index 2daeea9..0000000
--- a/hyperextract/knowledge/typical/__init__.py
+++ /dev/null
@@ -1,7 +0,0 @@
-"""Typical knowledge patterns - common knowledge representation types."""
-
-from .entity import EntityKnowledge
-
-__all__ = [
- "EntityKnowledge",
-]
diff --git a/hyperextract/knowledge/typical/entity.py b/hyperextract/knowledge/typical/entity.py
deleted file mode 100644
index 6189e06..0000000
--- a/hyperextract/knowledge/typical/entity.py
+++ /dev/null
@@ -1,390 +0,0 @@
-"""Entity knowledge extraction module.
-
-Provides entity extraction implementation based on ListKnowledge.
-"""
-
-from typing import List, Dict, Type, TypeVar, Generic
-from pydantic import BaseModel, Field as PydanticField
-from langchain_core.language_models.chat_models import BaseChatModel
-from langchain_core.embeddings import Embeddings
-from langchain_core.documents import Document
-from langchain_core.prompts import ChatPromptTemplate
-from langchain_community.vectorstores import FAISS
-
-from ..generic import ListKnowledge
-
-try:
- from hyperextract.config import logger
-except ImportError:
- import logging
-
- logger = logging.getLogger(__name__)
-
-
-# Entity type variable
-E = TypeVar("E", bound="BaseEntitySchema")
-
-
-class BaseEntitySchema(BaseModel):
- """Base entity class defining required core fields for all entities.
-
- Users can inherit from this class to add custom fields.
- """
-
- name: str = PydanticField(description="Unique entity name", min_length=1)
- description: str = PydanticField(description="Entity description")
-
- class Config:
- """Configuration allowing users to extend with additional fields."""
-
- extra = "allow" # Allow additional fields
- validate_assignment = True # Validate on assignment
-
-
-class EntityKnowledge(ListKnowledge[E], Generic[E]):
- """Specialized knowledge class for entity extraction and management.
-
- Key features:
- 1. Automatic deduplication based on entity.name
- 2. Intelligent LLM-powered merging for entities with the same name
- 3. Each entity indexed as an independent document
- 4. Support for entity-level semantic search
- """
-
- def __init__(
- self,
- entity_schema: Type[E],
- llm_client: BaseChatModel,
- embedder: Embeddings,
- *,
- prompt: str = "",
- **kwargs,
- ):
- """Initialize EntityKnowledge with entity schema and configuration.
-
- Args:
- entity_schema: User-defined entity class (must inherit from BaseEntity).
- llm_client: Language model client for extraction and merging.
- embedder: Embedding model for vector indexing.
- prompt: Custom extraction prompt.
- """
- self._entity_schema = entity_schema
-
- super().__init__(
- item_schema=entity_schema,
- llm_client=llm_client,
- embedder=embedder,
- prompt=prompt or self._default_prompt(),
- **kwargs,
- )
-
- self._merge_chain = self._create_merge_chain()
-
- # Global entity mapping (primary storage, type-safe)
- self._entity_map: Dict[str, E] = {}
-
- @staticmethod
- def _default_prompt() -> str:
- """Returns the default entity extraction prompt."""
- return (
- "You are an expert entity extraction assistant. "
- "Carefully read the following text and extract all entities "
- "according to the specified schema. Be precise and comprehensive."
- "Extract all entities without adding information not present in the text.\n\n"
- "### Source Text:\n"
- )
-
- def _create_merge_chain(self):
- """Creates an LLM chain for intelligently merging entities with the same name."""
- merge_template = """You are an expert at merging entity information.
-Given two instances of the same entity, intelligently merge their fields into one.
-
-Rules:
-1. Keep the 'name' field unchanged
-2. Merge 'description' by combining both descriptions (remove duplicates)
-3. For other fields: prefer non-null values, if both exist choose the more complete/accurate one
-
-Entity A:
-{entity_a}
-
-Entity B:
-{entity_b}
-
-Please merge these two entities intelligently and return the result."""
-
- merge_prompt = ChatPromptTemplate.from_template(merge_template)
-
- return merge_prompt | self.llm_client.with_structured_output(
- self._entity_schema
- )
-
- # ==================== Data Management ====================
-
- def clear(self):
- """Clears all entities from storage."""
- super().clear()
- self._entity_map.clear()
-
- # ==================== Extraction & Aggregation ====================
-
- def extract(self, text: str, *, store: bool = True) -> BaseModel:
- """Extracts entity knowledge with support for store/temporary modes.
-
- Mode explanation:
- - store=True (default): Store mode
- Merges newly extracted entities with existing knowledge and updates internal state
-
- - store=False: Temporary mode
- Returns extracted entities without modifying internal state
-
- Args:
- text: Input text to extract entities from.
- store: Whether to store extracted knowledge (default True).
-
- Returns:
- Container with extracted entities.
- """
- # Call parent extraction method with store parameter
- result = super().extract(text, store=store)
-
- # Only synchronize to entity_map if storing internally
- if store:
- self._sync_items_to_map()
- logger.info(
- "Entity extraction completed (Store mode)"
- )
- logger.info(f"Total entities: {len(self._entity_map)}")
- else:
- logger.info(
- "Entity extraction completed (Temporary mode)"
- )
-
- return result
-
- def merge(self, data_list: List[BaseModel]) -> BaseModel:
- """Entity-level intelligent merging using LLM.
-
- Strategy:
- 1. Call parent merge to get all extracted items
- 2. Assign result to self._data
- 3. Synchronize to entity_map for LLM-powered deduplication
- 4. Return deduplicated data
-
- Args:
- data_list: Multiple container objects.
-
- Returns:
- Merged entity data.
- """
- # Call parent merge to get merged result
- merged_result = super().merge(data_list)
-
- # Temporarily assign to self._data so _sync_items_to_map can access self.items
- self._data = merged_result
-
- # Synchronize to entity_map for deduplication
- self._sync_items_to_map()
-
- # Return final self._data (updated by _sync_items_to_map)
- return self._data
-
- def _sync_items_to_map(self):
- """Synchronizes self.items to entity_map, performing deduplication and LLM merging."""
- total_entities = 0
- duplicate_count = 0
-
- # Traverse parent's items and merge with existing entity_map
- for entity in self.items:
- total_entities += 1
- entity_name = entity.name.strip()
-
- if not entity_name: # Skip empty names
- logger.warning(f"Skipping entity with empty name: {entity}")
- continue
-
- if entity_name in self._entity_map:
- # Entity exists - merge using LLM
- duplicate_count += 1
- existing = self._entity_map[entity_name]
- merged = self._merge_entity_with_llm(existing, entity)
- if merged:
- self._entity_map[entity_name] = merged
- else:
- logger.warning(
- f"LLM merge failed for entity '{entity_name}', keeping existing"
- )
- else:
- # New entity - deep copy and add
- self._entity_map[entity_name] = entity.model_copy(deep=True)
-
- # Write deduplicated results back to self.items
- self._data.items = list(self._entity_map.values())
- self.clear_index()
-
- logger.info(
- f"Merged {len(self._entity_map)} unique entities "
- f"from {total_entities} total (removed {duplicate_count} duplicates)"
- )
-
- def _merge_entity_with_llm(self, entity_a: E, entity_b: E) -> E | None:
- """Uses LLM to intelligently merge two entities with the same name.
-
- Args:
- entity_a: First entity.
- entity_b: Second entity.
-
- Returns:
- Merged entity (returns None on failure).
- """
- try:
- # Prepare inputs
- entity_a_json = entity_a.model_dump_json(indent=2)
- entity_b_json = entity_b.model_dump_json(indent=2)
-
- # Call LLM for merging
- merged_entity = self._merge_chain.invoke(
- {
- "entity_a": entity_a_json,
- "entity_b": entity_b_json,
- }
- )
-
- logger.debug(f"Successfully merged entity: {merged_entity.name}")
- return merged_entity
-
- except Exception as e:
- logger.error(f"Error merging entity '{entity_a.name}' with LLM: {e}")
- return None
-
- # ==================== Indexing & Query ====================
-
- def build_index(self):
- """Builds independent vector index document for each entity.
-
- Each entity → one Document:
- - page_content: "{name}: {description} + other fields"
- - metadata: {"entity": entity.model_dump()}
- """
- if len(self._entity_map) == 0:
- logger.warning("No entities to index")
- return
-
- if self._index is not None:
- return # Index already exists
-
- documents = []
- for entity in self._entity_map.values():
- # Build document content for vectorization
- content_parts = [
- f"name: {entity.name}",
- f"description: {entity.description}",
- ]
-
- # Add other fields if user extended the schema
- for field_name in self._entity_schema.model_fields:
- if field_name not in ("name", "description"):
- value = getattr(entity, field_name, None)
- if value is not None:
- content_parts.append(f"{field_name}: {value}")
-
- content = "\n".join(content_parts)
-
- # Save complete entity data to metadata
- documents.append(
- Document(
- page_content=content,
- metadata={"raw": entity.model_dump()},
- )
- )
-
- self._index = FAISS.from_documents(documents, self.embedder)
- logger.info(f"Built FAISS index with {len(documents)} entities")
-
-
- def search(self, query: str, top_k: int = 3, return_raw: bool = False) -> List[E]:
- """Performs semantic search over entities.
-
- Args:
- query: Search query string.
- top_k: Number of results to return.
- return_raw: Whether to return raw entity objects (default returns dict).
-
- Returns:
- List of entities.
- """
- if len(self) == 0:
- logger.warning("No items to search")
- return []
-
- if self._index is None:
- raise Exception("Index is not built, please build the index first.")
-
- docs = self._index.similarity_search(query, k=top_k)
-
- # Restore entity objects
- results = []
- for doc in docs:
- try:
- entity_data = doc.metadata["raw"]
- if return_raw:
- # Return Pydantic object
- entity = self._entity_schema.model_validate(entity_data)
- results.append(entity)
- else:
- # Return dictionary
- results.append(entity_data)
- except Exception as e:
- logger.warning(f"Failed to restore entity: {e}")
-
- logger.info(f"Found {len(results)} entities for query: {query[:50]}...")
- return results
-
- def __len__(self) -> int:
- """Returns the number of entities (reads directly from entity_map)."""
- return len(self._entity_map)
-
- # ==================== Convenience Properties ====================
-
- @property
- def entities(self) -> List[E]:
- """Quick access to entity list."""
- return self.items
-
- @property
- def entity_schema(self) -> Type[E]:
- """Returns the user-defined entity schema."""
- return self._entity_schema
-
- @property
- def entity_names(self) -> List[str]:
- """Returns all entity names (reads directly from entity_map)."""
- return list(self._entity_map.keys())
-
- # ==================== Optional Enhancement Methods ====================
-
- def get_entity_by_name(self, name: str) -> E | None:
- """Retrieves entity by name (O(1) dictionary lookup)."""
- return self._entity_map.get(name)
-
- def remove_entity(self, name: str) -> bool:
- """Removes specified entity (from both entity_map and items)."""
- if name in self._entity_map:
- del self._entity_map[name]
- self._data.items = list(self._entity_map.values())
- self.clear_index()
- logger.info(f"Removed entity: {name}")
- return True
- return False
-
- # ==================== Serialization ====================
-
- def load(self, folder_path: str, **kwargs):
- """Loads knowledge and rebuilds entity_map."""
- super().load(folder_path, **kwargs)
-
- # Rebuild entity_map from items
- self._entity_map = {
- entity.name: entity.model_copy(deep=True) for entity in self.items
- }
-
- logger.info(f"Loaded {len(self._entity_map)} entities")
diff --git a/hyperextract/knowledge/typical/tree.py b/hyperextract/knowledge/typical/tree.py
deleted file mode 100644
index e69de29..0000000
diff --git a/hyperextract/skill/__init__.py b/hyperextract/skill/__init__.py
deleted file mode 100644
index e69de29..0000000
diff --git a/hyperextract/trees/__init__.py b/hyperextract/trees/__init__.py
new file mode 100644
index 0000000..b0ce327
--- /dev/null
+++ b/hyperextract/trees/__init__.py
@@ -0,0 +1,10 @@
+"""Tree structures for hierarchical data extraction.
+
+This module provides tree-based Auto types for extracting hierarchical
+and nested data structures from unstructured text.
+"""
+
+# Tree implementations will be added here as they are developed
+# from .base import AutoTree
+
+__all__ = []
diff --git a/hyperextract/knowledge/typical/relation.py b/hyperextract/trees/base.py
similarity index 100%
rename from hyperextract/knowledge/typical/relation.py
rename to hyperextract/trees/base.py
diff --git a/mkdocs.yml b/mkdocs.yml
index 5f74227..3af7b1a 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -95,9 +95,9 @@ nav:
- Knowledge Patterns: user-guide/knowledge-patterns.md
- API Reference:
- Overview: api/index.md
- - BaseKnowledge: api/base.md
- - UnitKnowledge: api/unit.md
- - ListKnowledge: api/list.md
- - SetKnowledge: api/set.md
+ - BaseAutoType: api/base.md
+ - AutoModel: api/model.md
+ - AutoList: api/auto_list.md
+ - AutoSet: api/auto_set.md
- 中文文档:
- 首页: zh/index.md
diff --git a/tests/test_knowledge.py b/tests/test_knowledge.py
index ad82fda..612069a 100644
--- a/tests/test_knowledge.py
+++ b/tests/test_knowledge.py
@@ -1,5 +1,5 @@
"""
-测试 Knowledge 类的功能。
+测试 BaseAutoType 类的功能。
"""
import pytest
from typing import List, Optional
@@ -7,7 +7,7 @@ from pydantic import BaseModel, Field
from unittest.mock import MagicMock, AsyncMock
import json
-from hyperextract.knowledge.base import Knowledge
+from hyperextract.core import BaseAutoType
# 定义测试用的 Schema
@@ -41,9 +41,9 @@ def mock_embedder():
@pytest.fixture
-def article_knowledge(mock_llm, mock_embedder):
- """创建一个 Article 类型的 Knowledge 实例"""
- return Knowledge(
+def a_article(mock_llm, mock_embedder):
+ """创建一个 Article 类型的 BaseAutoType 实例"""
+ return BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -52,9 +52,9 @@ def article_knowledge(mock_llm, mock_embedder):
@pytest.fixture
-def person_knowledge(mock_llm, mock_embedder):
- """创建一个 Person 类型的 Knowledge 实例"""
- return Knowledge(
+def a_person(mock_llm, mock_embedder):
+ """创建一个 Person 类型的 BaseAutoType 实例"""
+ return BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Person,
@@ -62,16 +62,16 @@ def person_knowledge(mock_llm, mock_embedder):
)
-class TestKnowledgeInitialization:
- """测试 Knowledge 类的初始化"""
+class TestBaseAutoTypeInitialization:
+ """测试 BaseAutoType 类的初始化"""
- def test_knowledge_init_with_article_schema(self, article_knowledge):
+ def test_auto_type_init_with_article_schema(self, a_article):
"""测试使用 Article schema 初始化"""
- assert article_knowledge.schema_class == Article
- assert article_knowledge.storage == "memory"
- assert article_knowledge.data is not None
+ assert a_article.schema_class == Article
+ assert a_article.storage == "memory"
+ assert a_article.data is not None
- def test_knowledge_init_with_initial_data(self, mock_llm, mock_embedder):
+ def test_auto_type_init_with_initial_data(self, mock_llm, mock_embedder):
"""测试带初始数据的初始化"""
initial_article = Article(
title="Test Article",
@@ -79,7 +79,7 @@ class TestKnowledgeInitialization:
keywords=["test"],
summary="A test article"
)
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -88,14 +88,14 @@ class TestKnowledgeInitialization:
assert knowledge.data == initial_article
assert knowledge.data.title == "Test Article"
- def test_knowledge_with_different_schemas(self, mock_llm, mock_embedder):
- """测试使用不同 Schema 的 Knowledge"""
- article_k = Knowledge(
+ def test_auto_type_with_different_schemas(self, mock_llm, mock_embedder):
+ """测试使用不同 Schema 的 BaseAutoType"""
+ article_k = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
)
- person_k = Knowledge(
+ person_k = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Person,
@@ -104,11 +104,11 @@ class TestKnowledgeInitialization:
assert person_k.schema_class == Person
-class TestKnowledgeExtract:
- """测试 Knowledge 的 extract 方法"""
+class TestBaseAutoTypeExtract:
+ """测试 BaseAutoType 的 extract 方法"""
@pytest.mark.asyncio
- async def test_extract_article(self, article_knowledge, mock_llm):
+ async def test_extract_article(self, a_article, mock_llm):
"""测试提取文章信息"""
# 模拟 LLM 返回的响应(with_structured_output 直接返回对象)
expected_article = Article(
@@ -124,15 +124,15 @@ class TestKnowledgeExtract:
mock_llm.with_structured_output.return_value = mock_structured_llm
text = "Artificial Intelligence continues to advance in 2024..."
- result = await article_knowledge.extract(text)
+ result = await a_article.extract(text)
assert result.title == "AI in 2024"
assert len(result.authors) == 2
assert "AI" in result.keywords
- assert article_knowledge.data == result
+ assert a_article.data == result
@pytest.mark.asyncio
- async def test_extract_with_markdown_response(self, article_knowledge, mock_llm):
+ async def test_extract_with_markdown_response(self, a_article, mock_llm):
"""测试 with_structured_output 自动处理响应格式"""
# 使用 with_structured_output 无需处理 markdown 格式
expected_article = Article(
@@ -146,12 +146,12 @@ class TestKnowledgeExtract:
mock_structured_llm.ainvoke.return_value = expected_article
mock_llm.with_structured_output.return_value = mock_structured_llm
- result = await article_knowledge.extract("test text")
+ result = await a_article.extract("test text")
assert result.title == "Test Article"
assert result.authors[0] == "Author"
@pytest.mark.asyncio
- async def test_aextract_calls_extract(self, article_knowledge, mock_llm):
+ async def test_aextract_calls_extract(self, a_article, mock_llm):
"""测试 aextract 方法调用 extract"""
expected_article = Article(
title="Test",
@@ -164,12 +164,12 @@ class TestKnowledgeExtract:
mock_structured_llm.ainvoke.return_value = expected_article
mock_llm.with_structured_output.return_value = mock_structured_llm
- result = await article_knowledge.aextract("test")
+ result = await a_article.aextract("test")
assert result.title == "Test"
-class TestKnowledgeDump:
- """测试 Knowledge 的 dump 方法"""
+class TestBaseAutoTypeDump:
+ """测试 BaseAutoType 的 dump 方法"""
def test_dump_to_json(self, mock_llm, mock_embedder):
"""测试导出为 JSON 格式"""
@@ -179,7 +179,7 @@ class TestKnowledgeDump:
keywords=["test"],
summary="A test"
)
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -200,7 +200,7 @@ class TestKnowledgeDump:
keywords=["test"],
summary="A test"
)
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -212,12 +212,12 @@ class TestKnowledgeDump:
assert data_dict["title"] == "Test"
-class TestKnowledgeLoad:
- """测试 Knowledge 的 load 方法"""
+class TestBaseAutoTypeLoad:
+ """测试 BaseAutoType 的 load 方法"""
def test_load_from_dict(self, mock_llm, mock_embedder):
"""测试从字典加载"""
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -236,7 +236,7 @@ class TestKnowledgeLoad:
def test_load_from_json_string(self, mock_llm, mock_embedder):
"""测试从 JSON 字符串加载"""
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -254,7 +254,7 @@ class TestKnowledgeLoad:
def test_load_invalid_data_raises_error(self, mock_llm, mock_embedder):
"""测试加载无效数据时抛出错误"""
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -266,8 +266,8 @@ class TestKnowledgeLoad:
knowledge.load(invalid_data)
-class TestKnowledgeSearch:
- """测试 Knowledge 的 search 方法"""
+class TestBaseAutoTypeSearch:
+ """测试 BaseAutoType 的 search 方法"""
@pytest.mark.asyncio
async def test_search_returns_dict(self, mock_llm, mock_embedder):
@@ -278,7 +278,7 @@ class TestKnowledgeSearch:
keywords=["search"],
summary="Testing search"
)
- knowledge = Knowledge(
+ knowledge = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
@@ -290,27 +290,27 @@ class TestKnowledgeSearch:
assert result["title"] == "Search Test"
-class TestKnowledgeEvolve:
- """测试 Knowledge 的 evolve 和 aevolve 方法"""
+class TestBaseAutoTypeEvolve:
+ """测试 BaseAutoType 的 evolve 和 aevolve 方法"""
@pytest.mark.asyncio
- async def test_aevolve_default_implementation(self, article_knowledge):
+ async def test_aevolve_default_implementation(self, a_article):
"""测试 aevolve 的默认实现(不做任何操作)"""
- await article_knowledge.aevolve()
+ await a_article.aevolve()
# 如果没有抛出异常,则测试通过
-class TestKnowledgeMultipleSchemas:
+class TestBaseAutoTypeMultipleSchemas:
"""测试使用多个不同 Schema 的场景"""
def test_different_schemas_independent(self, mock_llm, mock_embedder):
- """测试不同 Schema 的 Knowledge 相互独立"""
- article_k = Knowledge(
+ """测试不同 Schema 的 BaseAutoType 相互独立"""
+ article_k = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Article,
)
- person_k = Knowledge(
+ person_k = BaseAutoType(
llm_client=mock_llm,
embedder=mock_embedder,
schema_class=Person,