Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

Add UnitKnowledge API documentation and enhance user guide

yifanfeng97ae86ec15256c
diff --git a/docs/BUILDING.md b/docs/BUILDING.md
new file mode 100644
index 0000000..0cd6674
--- /dev/null
+++ b/docs/BUILDING.md
@@ -0,0 +1,398 @@
+# Building Documentation
+
+This guide explains how to build and preview the Hyper-Extract documentation.
+
+## Prerequisites
+
+The documentation uses MkDocs with Material theme. Required dependencies are in the `dev` dependency group:
+
+```toml
+[dependency-groups]
+dev = [
+ "mkdocs>=1.6.1",
+ "mkdocs-material>=9.7.1",
+ "mkdocstrings[python]>=1.0.0",
+]
+```
+
+## Installation
+
+### Using uv (Recommended)
+
+```bash
+# Install all dependencies including dev dependencies
+uv sync
+
+# Activate virtual environment
+.venv\Scripts\activate # Windows
+source .venv/bin/activate # Linux/macOS
+```
+
+### Using pip
+
+```bash
+# Install project with dev dependencies
+pip install -e ".[dev]"
+```
+
+## Building Documentation
+
+### Development Server (Live Preview)
+
+Start a local server with live reloading:
+
+```bash
+mkdocs serve
+```
+
+Then open your browser to `http://127.0.0.1:8000/`
+
+- Changes to `.md` files auto-reload
+- Hot reload for configuration changes
+- Useful for writing and testing documentation
+
+### Build Static Site
+
+Generate the static HTML site:
+
+```bash
+mkdocs build
+```
+
+Output is in `site/` directory:
+```
+site/
+├── index.html
+├── user-guide/
+│ ├── getting-started/
+│ └── knowledge-patterns/
+├── api/
+│ ├── base/
+│ ├── unit/
+│ ├── list/
+│ └── set/
+├── zh/
+│ └── index.html
+└── assets/
+```
+
+### Build with Clean
+
+Remove old build artifacts first:
+
+```bash
+mkdocs build --clean
+```
+
+### Strict Mode
+
+Build with warnings as errors:
+
+```bash
+mkdocs build --strict
+```
+
+Useful for CI/CD to catch documentation issues.
+
+## Documentation Structure
+
+```
+docs/
+├── index.md # English homepage
+├── user-guide/
+│ ├── getting-started.md # Installation and basic usage
+│ └── knowledge-patterns.md # Pattern selection guide
+├── api/
+│ ├── index.md # API overview
+│ ├── base.md # BaseKnowledge API
+│ ├── unit.md # UnitKnowledge API
+│ ├── list.md # ListKnowledge API
+│ └── set.md # SetKnowledge API
+└── zh/
+ └── index.md # Chinese homepage (中文首页)
+```
+
+## Multi-Language Support
+
+### Current Languages
+
+- **English** (default): `/`
+- **中文** (Chinese): `/zh/`
+
+### Adding New Languages
+
+1. Create language directory:
+ ```bash
+ mkdir docs/ja # Japanese example
+ ```
+
+2. Add translation files:
+ ```bash
+ cp docs/index.md docs/ja/index.md
+ # Translate content to Japanese
+ ```
+
+3. Update `mkdocs.yml`:
+ ```yaml
+ extra:
+ alternate:
+ - name: English
+ link: /
+ lang: en
+ - name: 中文
+ link: /zh/
+ lang: zh
+ - name: 日本語
+ link: /ja/
+ lang: ja
+ ```
+
+4. Add to navigation:
+ ```yaml
+ nav:
+ # ... existing nav ...
+ - 日本語:
+ - ホーム: ja/index.md
+ ```
+
+## Configuration
+
+### MkDocs Configuration (`mkdocs.yml`)
+
+Key sections:
+
+```yaml
+# Site metadata
+site_name: Hyper-Extract
+site_description: An intelligent, LLM-powered knowledge extraction framework
+
+# Theme
+theme:
+ name: material
+ language: en
+ features:
+ - navigation.tabs
+ - search.suggest
+ - content.code.copy
+
+# Plugins
+plugins:
+ - search
+ - mkdocstrings # Auto-generate API docs from docstrings
+
+# Markdown extensions
+markdown_extensions:
+ - pymdownx.highlight
+ - pymdownx.superfences
+ - admonition
+```
+
+### MkDocstrings Configuration
+
+Auto-generates API documentation from Python docstrings:
+
+```yaml
+plugins:
+ - mkdocstrings:
+ handlers:
+ python:
+ paths: [.] # Search path for modules
+ options:
+ show_source: true
+ show_root_heading: true
+ docstring_style: google
+```
+
+Usage in markdown:
+```markdown
+# BaseKnowledge API
+
+::: hyperextract.knowledge.base.BaseKnowledge
+ options:
+ show_source: true
+ heading_level: 2
+```
+
+## Deployment
+
+### GitHub Pages
+
+1. **Build documentation**:
+ ```bash
+ mkdocs gh-deploy
+ ```
+
+ This command:
+ - Builds the documentation
+ - Commits to `gh-pages` branch
+ - Pushes to GitHub
+
+2. **Configure GitHub Pages**:
+ - Go to repository Settings → Pages
+ - Source: Deploy from branch
+ - Branch: `gh-pages` / `root`
+
+### Manual Deployment
+
+1. Build static site:
+ ```bash
+ mkdocs build
+ ```
+
+2. Deploy `site/` directory to your hosting:
+ ```bash
+ # Example: Deploy to Netlify
+ netlify deploy --dir=site --prod
+
+ # Example: Deploy to Vercel
+ vercel --prod site/
+ ```
+
+## Continuous Integration
+
+### GitHub Actions
+
+Create `.github/workflows/docs.yml`:
+
+```yaml
+name: Deploy Documentation
+
+on:
+ push:
+ branches:
+ - main
+ paths:
+ - 'docs/**'
+ - 'mkdocs.yml'
+
+jobs:
+ deploy:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+
+ - uses: actions/setup-python@v5
+ with:
+ python-version: '3.10'
+
+ - name: Install dependencies
+ run: |
+ pip install mkdocs mkdocs-material mkdocstrings[python]
+
+ - name: Deploy documentation
+ run: mkdocs gh-deploy --force
+```
+
+## Tips
+
+### Writing Documentation
+
+1. **Use code blocks with language hints**:
+ ````markdown
+ ```python
+ from hyperextract import UnitKnowledge
+ ```
+ ````
+
+2. **Add admonitions for important notes**:
+ ```markdown
+ !!! note
+ This is an important note.
+
+ !!! warning
+ Be careful with this feature.
+ ```
+
+3. **Link to other pages**:
+ ```markdown
+ See [Getting Started](user-guide/getting-started.md)
+ See [API Reference](api/base.md#extract)
+ ```
+
+4. **Use tabs for alternatives**:
+ ````markdown
+ === "Python"
+ ```python
+ knowledge.extract(text)
+ ```
+
+ === "Result"
+ ```json
+ {"title": "Example"}
+ ```
+ ````
+
+### Auto-generating API Docs
+
+For classes with good docstrings:
+
+```markdown
+# MyClass API
+
+::: mymodule.MyClass
+ options:
+ show_source: true
+ members:
+ - __init__
+ - method1
+ - method2
+```
+
+### Search Optimization
+
+MkDocs indexes all content automatically. To improve searchability:
+
+- Use clear headings
+- Include keywords in descriptions
+- Add synonyms in metadata
+
+## Troubleshooting
+
+### MkDocs not found
+
+```bash
+# Install mkdocs
+pip install mkdocs mkdocs-material mkdocstrings[python]
+```
+
+### Module not found (mkdocstrings)
+
+Ensure the project root is in Python path:
+
+```yaml
+# mkdocs.yml
+plugins:
+ - mkdocstrings:
+ handlers:
+ python:
+ paths: [.] # Current directory
+```
+
+### Port already in use
+
+```bash
+# Use different port
+mkdocs serve --dev-addr=127.0.0.1:8001
+```
+
+### Build errors
+
+```bash
+# Verbose output
+mkdocs build --verbose
+
+# Clean and rebuild
+mkdocs build --clean
+```
+
+## Resources
+
+- [MkDocs Documentation](https://www.mkdocs.org/)
+- [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/)
+- [MkDocstrings](https://mkdocstrings.github.io/)
+
+## Need Help?
+
+- Check [MkDocs User Guide](https://www.mkdocs.org/user-guide/)
+- See [Material Theme Reference](https://squidfunk.github.io/mkdocs-material/reference/)
+- Open an issue on GitHub
diff --git a/docs/api/base.md b/docs/api/base.md
new file mode 100644
index 0000000..a29b8d8
--- /dev/null
+++ b/docs/api/base.md
@@ -0,0 +1,296 @@
+# BaseKnowledge API
+
+::: hyperextract.knowledge.base.BaseKnowledge
+ options:
+ show_root_heading: true
+ show_source: true
+ heading_level: 2
+
+## Overview
+
+`BaseKnowledge` is the abstract base class that provides the foundation for all knowledge extraction patterns in Hyper-Extract. It implements the core lifecycle operations: extraction, merging, indexing, evolution, and serialization.
+
+## Class Signature
+
+```python
+class BaseKnowledge(ABC, Generic[T]):
+ """Unified knowledge base class integrating extraction, storage,
+ aggregation, and evolution."""
+```
+
+Where `T` is a Pydantic `BaseModel` subclass defining your knowledge schema.
+
+## Initialization
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `data_schema` | `Type[T]` | Required | Pydantic BaseModel subclass defining knowledge structure |
+| `llm_client` | `BaseChatModel` | Required | LangChain LLM client for extraction |
+| `embedder` | `Embeddings` | Required | Embedding model for semantic search |
+| `prompt` | `str` | `""` | Custom extraction prompt (uses default if empty) |
+| `chunk_size` | `int` | `2000` | Maximum characters per text chunk |
+| `chunk_overlap` | `int` | `200` | Overlapping characters between chunks |
+| `max_workers` | `int` | `10` | Maximum concurrent extraction tasks |
+| `show_progress` | `bool` | `True` | Whether to display progress logs |
+
+### Example
+
+```python
+from hyperextract.knowledge.base import BaseKnowledge
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+from pydantic import BaseModel, Field
+
+class MySchema(BaseModel):
+ title: str = Field(default="", description="Document title")
+ summary: str = Field(default="", description="Brief summary")
+
+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+
+# Note: BaseKnowledge is abstract, use concrete implementations
+# knowledge = UnitKnowledge(...)
+```
+
+## Properties
+
+### `data`
+```python
+@property
+def data(self) -> T
+```
+Returns the extracted knowledge data. Type is your specified schema `T`.
+
+**Example**:
+```python
+knowledge.extract(text)
+print(knowledge.data.title)
+```
+
+### `metadata`
+```python
+@property
+def metadata(self) -> Dict[str, Any]
+```
+Returns metadata about the knowledge object, including creation and update timestamps.
+
+**Example**:
+```python
+print(knowledge.metadata["created_at"])
+print(knowledge.metadata["updated_at"])
+```
+
+## Core Methods
+
+### `extract()`
+```python
+def extract(
+ self,
+ text: str,
+ *,
+ additional_info: str = "",
+ merge: bool = True
+) -> T
+```
+
+Extract structured knowledge from text.
+
+**Parameters**:
+- `text` (str): Source text to extract from
+- `additional_info` (str): Optional context or instructions
+- `merge` (bool): Whether to merge with existing data (default: True)
+
+**Returns**: Extracted data of type `T`
+
+**Example**:
+```python
+result = knowledge.extract(
+ text="Long article text...",
+ additional_info="Focus on technical details",
+ merge=True
+)
+```
+
+### `build_index()`
+```python
+def build_index(self, force_rebuild: bool = False) -> None
+```
+
+Build or rebuild the FAISS vector index for semantic search.
+
+**Parameters**:
+- `force_rebuild` (bool): Force rebuild even if index exists
+
+**Example**:
+```python
+knowledge.build_index() # Build if not exists
+knowledge.build_index(force_rebuild=True) # Force rebuild
+```
+
+### `search()`
+```python
+def search(
+ self,
+ query: str,
+ k: int = 5,
+ **kwargs
+) -> List[Tuple[Document, float]]
+```
+
+Perform semantic search over indexed knowledge.
+
+**Parameters**:
+- `query` (str): Search query
+- `k` (int): Number of results to return
+- `**kwargs`: Additional arguments passed to FAISS
+
+**Returns**: List of (Document, similarity_score) tuples
+
+**Example**:
+```python
+results = knowledge.search("machine learning applications", k=3)
+for doc, score in results:
+ print(f"Score: {score:.3f}")
+ print(f"Content: {doc.page_content}")
+ print(f"Metadata: {doc.metadata}")
+```
+
+### `dump()`
+```python
+def dump(self, save_path: str | Path) -> None
+```
+
+Save knowledge to disk, including data, metadata, and index.
+
+**Parameters**:
+- `save_path` (str | Path): Directory path to save to
+
+**Structure**:
+```
+save_path/
+├── state.json # Data and metadata
+└── faiss_index/ # Vector index
+ └── index.faiss
+```
+
+**Example**:
+```python
+knowledge.dump("./my_knowledge")
+```
+
+### `load()` (Class Method)
+```python
+@classmethod
+def load(
+ cls,
+ load_path: str | Path,
+ data_schema: Type[T],
+ llm_client: BaseChatModel,
+ embedder: Embeddings,
+ **kwargs
+) -> "BaseKnowledge[T]"
+```
+
+Load knowledge from disk.
+
+**Parameters**:
+- `load_path` (str | Path): Directory to load from
+- `data_schema` (Type[T]): Schema class (must match saved data)
+- `llm_client` (BaseChatModel): LLM client for future operations
+- `embedder` (Embeddings): Embedder for search
+- `**kwargs`: Additional initialization parameters
+
+**Returns**: Loaded knowledge instance
+
+**Example**:
+```python
+loaded = UnitKnowledge.load(
+ "./my_knowledge",
+ data_schema=MySchema,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+## Abstract Methods
+
+Subclasses must implement these methods:
+
+### `_default_prompt()`
+```python
+@staticmethod
+@abstractmethod
+def _default_prompt() -> str
+```
+
+Returns the default extraction prompt for the pattern.
+
+### `_merge_strategy()`
+```python
+@abstractmethod
+def _merge_strategy(self, old_data: T, new_data: T) -> T
+```
+
+Defines how to merge old and new extracted data.
+
+### `_build_documents_for_indexing()`
+```python
+@abstractmethod
+def _build_documents_for_indexing(self) -> List[Document]
+```
+
+Converts knowledge data into LangChain Documents for indexing.
+
+## Internal Methods
+
+### `_extract_single_chunk()`
+```python
+def _extract_single_chunk(
+ self,
+ chunk_text: str,
+ additional_info: str = ""
+) -> T
+```
+
+Extracts knowledge from a single text chunk. Used internally by `extract()`.
+
+### `_extract_multi_chunk()`
+```python
+def _extract_multi_chunk(
+ self,
+ text: str,
+ additional_info: str = ""
+) -> T
+```
+
+Handles multi-chunk extraction with parallel processing and merging.
+
+## Usage in Subclasses
+
+When creating custom knowledge patterns, inherit from `BaseKnowledge`:
+
+```python
+from hyperextract.knowledge.base import BaseKnowledge, T
+from typing import List
+from langchain_core.documents import Document
+
+class CustomKnowledge(BaseKnowledge[T]):
+ @staticmethod
+ def _default_prompt() -> str:
+ return "Extract information according to schema: "
+
+ def _merge_strategy(self, old_data: T, new_data: T) -> T:
+ # Implement merge logic
+ return new_data # Simple replace
+
+ def _build_documents_for_indexing(self) -> List[Document]:
+ # Convert self._data to documents
+ return [Document(page_content=str(self._data))]
+```
+
+## See Also
+
+- [UnitKnowledge API](unit.md) - Single-object extraction
+- [ListKnowledge API](list.md) - Multi-item extraction
+- [SetKnowledge API](set.md) - Unique collection extraction
diff --git a/docs/api/index.md b/docs/api/index.md
new file mode 100644
index 0000000..5c145e6
--- /dev/null
+++ b/docs/api/index.md
@@ -0,0 +1,81 @@
+# API Reference
+
+This section provides detailed API documentation for all Hyper-Extract classes and methods.
+
+## Module Structure
+
+```
+hyperextract/
+├── knowledge/
+│ ├── base.py # BaseKnowledge abstract class
+│ └── generic/
+│ ├── unit.py # UnitKnowledge
+│ ├── list.py # ListKnowledge
+│ └── set.py # SetKnowledge
+```
+
+## Quick Links
+
+- [BaseKnowledge](base.md) - Abstract base class for all knowledge patterns
+- [UnitKnowledge](unit.md) - Single-object extraction pattern
+- [ListKnowledge](list.md) - Multi-item list extraction pattern
+- [SetKnowledge](set.md) - Unique collection with deduplication
+
+## Common Patterns
+
+### Initialization
+
+All knowledge classes share a common initialization pattern:
+
+```python
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+
+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+
+knowledge = PatternClass(
+ data_schema=YourSchema, # or item_schema for List/Set
+ llm_client=llm,
+ embedder=embedder,
+ chunk_size=2000,
+ chunk_overlap=200
+)
+```
+
+### Core Methods
+
+Every knowledge class implements these methods:
+
+| Method | Description | Returns |
+|--------|-------------|---------|
+| `extract(text)` | Extract knowledge from text | None (updates internal state) |
+| `build_index()` | Build FAISS vector index | None |
+| `search(query, k)` | Semantic search | List of (Document, score) tuples |
+| `dump(path)` | Save to disk | None |
+| `load(path, ...)` | Load from disk | Knowledge instance |
+
+### Type Hints
+
+All classes are fully typed with Python generics:
+
+```python
+from hyperextract.knowledge.generic import UnitKnowledge
+from pydantic import BaseModel
+
+class MySchema(BaseModel):
+ field: str
+
+# Type-safe: knowledge.data is of type MySchema
+knowledge: UnitKnowledge[MySchema] = UnitKnowledge(
+ data_schema=MySchema,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+## Next: Detailed API Docs
+
+- [BaseKnowledge API](base.md)
+- [UnitKnowledge API](unit.md)
+- [ListKnowledge API](list.md)
+- [SetKnowledge API](set.md)
diff --git a/docs/api/list.md b/docs/api/list.md
new file mode 100644
index 0000000..2289e37
--- /dev/null
+++ b/docs/api/list.md
@@ -0,0 +1,351 @@
+# ListKnowledge API
+
+::: hyperextract.knowledge.generic.list.ListKnowledge
+ options:
+ show_root_heading: true
+ show_source: true
+ heading_level: 2
+
+## Overview
+
+`ListKnowledge` implements the multi-item list extraction pattern. It extracts multiple independent objects of the same type from a document, suitable for entity lists, events, citations, and more.
+
+## Class Signature
+
+```python
+class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):
+ """List Knowledge Pattern - extracts a collection of objects from text."""
+```
+
+## Key Characteristics
+
+- **Extraction Target**: A collection of structured objects
+- **Merge Strategy**: Append with basic deduplication (exact matches)
+- **Indexing Strategy**: Each item indexed independently
+- **Container**: Uses `ItemListSchema[Item]` wrapper
+
+## Initialization
+
+```python
+from hyperextract.knowledge.generic import ListKnowledge
+from pydantic import BaseModel, Field
+from typing import Optional
+
+class Person(BaseModel):
+ name: str = Field(..., description="Person's full name")
+ role: Optional[str] = Field(None, description="Role or title")
+ organization: Optional[str] = Field(None, description="Organization")
+
+people = ListKnowledge(
+ item_schema=Person, # Note: item_schema, not data_schema!
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+## Properties
+
+### `items`
+```python
+@property
+def items(self) -> List[Item]
+```
+
+Returns the list of extracted items.
+
+```python
+people.extract(text)
+for person in people.items:
+ print(f"{person.name} - {person.role}")
+```
+
+### `data`
+```python
+@property
+def data(self) -> ItemListSchema[Item]
+```
+
+Returns the container wrapper with `.items` attribute.
+
+```python
+container = people.data
+print(len(container.items))
+```
+
+## List Operations
+
+`ListKnowledge` supports Pythonic list operations:
+
+### Indexing
+```python
+first = people[0]
+last = people[-1]
+slice_items = people[1:3]
+```
+
+### Iteration
+```python
+for person in people:
+ print(person.name)
+```
+
+### Length
+```python
+count = len(people)
+```
+
+### Membership
+```python
+if some_person in people:
+ print("Found!")
+```
+
+### Append
+```python
+people.append(Person(name="John Doe", role="Engineer"))
+```
+
+### Extend
+```python
+new_people = [
+ Person(name="Alice", role="Designer"),
+ Person(name="Bob", role="Manager")
+]
+people.extend(new_people)
+```
+
+### Clear
+```python
+people.clear() # Remove all items
+```
+
+## Methods
+
+### `extract()`
+
+Extracts multiple items from text:
+
+```python
+text = """
+Dr. Sarah Johnson leads the research team at MIT.
+Professor Michael Chen from Stanford collaborated on the project.
+Emily Rodriguez from Google coordinated the efforts.
+"""
+
+people.extract(text)
+print(f"Extracted {len(people)} people")
+
+for person in people:
+ print(f"- {person.name}: {person.role} at {person.organization}")
+```
+
+**Multi-chunk behavior**:
+- Each chunk extracts multiple items
+- All items are appended together
+- Basic deduplication removes exact duplicates
+
+### `build_index()`
+
+Indexes each item separately:
+
+```python
+people.build_index()
+# Creates one document per item with all fields serialized
+```
+
+### `search()`
+
+Searches across all items:
+
+```python
+results = people.search("researchers from MIT", k=3)
+for doc, score in results:
+ # doc.page_content contains serialized item
+ # doc.metadata contains item index
+ print(f"Match (score {score:.3f}): {doc.page_content}")
+```
+
+## Merge Strategy
+
+Appends new items with basic deduplication:
+
+```python
+def _merge_strategy(
+ self,
+ old_data: ItemListSchema[Item],
+ new_data: ItemListSchema[Item]
+) -> ItemListSchema[Item]:
+ """Append new items, removing exact duplicates."""
+```
+
+**Example**:
+```python
+# Old items
+[Person(name="Alice", role="Engineer")]
+
+# New items (from next chunk)
+[Person(name="Bob", role="Designer"),
+ Person(name="Alice", role="Engineer")] # Duplicate
+
+# Merged result
+[Person(name="Alice", role="Engineer"),
+ Person(name="Bob", role="Designer")]
+```
+
+## Indexing Strategy
+
+```python
+def _build_documents_for_indexing(self) -> List[Document]:
+ """Build one document per item."""
+ return [
+ Document(
+ page_content=item.model_dump_json(),
+ metadata={"item_index": i}
+ )
+ for i, item in enumerate(self.items)
+ ]
+```
+
+## Use Cases
+
+### Entity Extraction
+```python
+class Entity(BaseModel):
+ name: str = Field(..., description="Entity name")
+ type: str = Field(..., description="Entity type (person/org/location)")
+ context: str = Field(default="", description="Surrounding context")
+
+entities = ListKnowledge(
+ item_schema=Entity,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+### Event Extraction
+```python
+class Event(BaseModel):
+ date: str = Field(..., description="Event date")
+ description: str = Field(..., description="What happened")
+ participants: List[str] = Field(default_factory=list)
+ location: Optional[str] = None
+
+events = ListKnowledge(
+ item_schema=Event,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+### Citation Extraction
+```python
+class Citation(BaseModel):
+ authors: List[str] = Field(..., description="Author names")
+ title: str = Field(..., description="Paper title")
+ year: Optional[int] = None
+ venue: Optional[str] = None
+
+citations = ListKnowledge(
+ item_schema=Citation,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+## Advanced Usage
+
+### Custom Deduplication
+
+Subclass to implement custom deduplication:
+
+```python
+class SmartListKnowledge(ListKnowledge[Item]):
+ def _merge_strategy(self, old_data, new_data):
+ # Custom deduplication logic
+ existing_names = {item.name for item in old_data.items}
+ unique_new = [
+ item for item in new_data.items
+ if item.name not in existing_names
+ ]
+ old_data.items.extend(unique_new)
+ return old_data
+```
+
+### Filtering
+
+```python
+# Filter items after extraction
+engineers = [p for p in people if p.role == "Engineer"]
+mit_people = [p for p in people if "MIT" in (p.organization or "")]
+```
+
+### Sorting
+
+```python
+# Sort by field
+sorted_people = sorted(people.items, key=lambda p: p.name)
+
+# Sort by multiple fields
+from operator import attrgetter
+sorted_people = sorted(people.items, key=attrgetter('organization', 'name'))
+```
+
+## Best Practices
+
+1. **Required vs Optional Fields**:
+ ```python
+ class Item(BaseModel):
+ name: str = Field(..., description="Required field") # Must have value
+ details: Optional[str] = Field(None, description="Optional field")
+ ```
+
+2. **Field Descriptions**: Help LLM understand what to extract
+ ```python
+ date: str = Field(..., description="Date in YYYY-MM-DD format")
+ ```
+
+3. **List Fields**: Use `default_factory=list`
+ ```python
+ tags: List[str] = Field(default_factory=list, description="Tags")
+ ```
+
+4. **Chunk Size**: Smaller chunks for dense information
+ ```python
+ # For entity-rich text
+ ListKnowledge(..., chunk_size=1500, chunk_overlap=150)
+ ```
+
+## Serialization
+
+```python
+# Save
+people.dump("./people_knowledge")
+
+# Load
+loaded = ListKnowledge.load(
+ "./people_knowledge",
+ item_schema=Person,
+ llm_client=llm,
+ embedder=embedder
+)
+
+assert len(loaded) == len(people)
+```
+
+## Comparison with SetKnowledge
+
+| Feature | ListKnowledge | SetKnowledge |
+|---------|---------------|--------------|
+| Duplicates | Allowed (basic dedup) | Prevented (by unique_key) |
+| Merge | Simple append | Configurable strategies |
+| Use Case | Order matters, simple lists | Uniqueness required |
+
+**When to use List vs Set**:
+- Use **List** when: Order is important, items might legitimately appear multiple times
+- Use **Set** when: Uniqueness is critical, need smart deduplication
+
+## See Also
+
+- [SetKnowledge API](set.md) - Unique collection with deduplication
+- [BaseKnowledge API](base.md) - Base class documentation
+- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection
diff --git a/docs/api/set.md b/docs/api/set.md
new file mode 100644
index 0000000..dfca60e
--- /dev/null
+++ b/docs/api/set.md
@@ -0,0 +1,434 @@
+# SetKnowledge API
+
+::: hyperextract.knowledge.generic.set.SetKnowledge
+ options:
+ show_root_heading: true
+ show_source: true
+ heading_level: 2
+
+## Overview
+
+`SetKnowledge` implements the unique collection pattern with automatic deduplication. It extracts items and ensures uniqueness based on a user-specified key field, with configurable merge strategies for handling duplicates.
+
+## Class Signature
+
+```python
+class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):
+ """Set Knowledge Pattern - extracts a unique collection of objects."""
+```
+
+## Key Characteristics
+
+- **Extraction Target**: Unique collection of structured objects
+- **Deduplication**: Based on `unique_key` field
+- **Merge Strategy**: Configurable (keep_old/keep_new/field_merge/llm_merge)
+- **Internal Storage**: Dict for O(1) lookup
+- **External Interface**: List-like access via `items` property
+
+## Initialization
+
+```python
+from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy
+from pydantic import BaseModel, Field
+
+class Keyword(BaseModel):
+ term: str = Field(..., description="Keyword or phrase")
+ category: Optional[str] = Field(None, description="Category")
+ importance: Optional[str] = Field(None, description="Importance level")
+
+keywords = SetKnowledge(
+ item_schema=Keyword,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="term", # Deduplicate by this field
+ merge_strategy=MergeItemStrategy.FIELD_MERGE
+)
+```
+
+### Additional Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `unique_key` | `str` | Required | Field name for deduplication |
+| `merge_strategy` | `MergeItemStrategy` | `KEEP_NEW` | How to handle duplicates |
+
+## Merge Strategies
+
+### KEEP_OLD
+Keep existing item, discard new duplicate:
+```python
+SetKnowledge(
+ ...,
+ merge_strategy=MergeItemStrategy.KEEP_OLD
+)
+```
+
+**Example**:
+```python
+# Existing: Keyword(term="AI", category="Tech", importance="High")
+# New: Keyword(term="AI", category="Science", importance="Medium")
+# Result: Keyword(term="AI", category="Tech", importance="High") # Old kept
+```
+
+### KEEP_NEW (Default)
+Replace existing with new:
+```python
+SetKnowledge(
+ ...,
+ merge_strategy=MergeItemStrategy.KEEP_NEW
+)
+```
+
+**Example**:
+```python
+# Existing: Keyword(term="AI", category="Tech", importance="High")
+# New: Keyword(term="AI", category="Science", importance="Medium")
+# Result: Keyword(term="AI", category="Science", importance="Medium") # New kept
+```
+
+### FIELD_MERGE
+Merge fields - new fills old's None/empty fields:
+```python
+SetKnowledge(
+ ...,
+ merge_strategy=MergeItemStrategy.FIELD_MERGE
+)
+```
+
+**Example**:
+```python
+# Existing: Keyword(term="AI", category="Tech", importance=None)
+# New: Keyword(term="AI", category=None, importance="High")
+# Result: Keyword(term="AI", category="Tech", importance="High") # Combined
+```
+
+### LLM_MERGE
+Use LLM to intelligently merge:
+```python
+SetKnowledge(
+ ...,
+ merge_strategy=MergeItemStrategy.LLM_MERGE
+)
+```
+
+**Example**:
+```python
+# Existing: Keyword(term="ML", category="AI", importance="High")
+# New: Keyword(term="ML", category="Data Science", importance="Critical")
+# LLM decides: Keyword(term="ML", category="AI/Data Science", importance="Critical")
+```
+
+## Properties
+
+### `items`
+```python
+@property
+def items(self) -> List[Item]
+```
+
+Returns list of unique items:
+```python
+for keyword in keywords.items:
+ print(f"{keyword.term}: {keyword.category}")
+```
+
+### `unique_keys`
+```python
+@property
+def unique_keys(self) -> List[str]
+```
+
+Returns list of unique key values:
+```python
+all_terms = keywords.unique_keys
+print(f"Unique keywords: {', '.join(all_terms)}")
+```
+
+## Methods
+
+### `extract()`
+
+Extracts items with automatic deduplication:
+
+```python
+text = """
+Machine learning is a key AI technology. Deep learning, a subset of
+machine learning, has revolutionized AI. Neural networks are fundamental
+to deep learning. Machine learning continues to evolve.
+"""
+
+keywords.extract(text)
+# "machine learning" appears twice but stored once
+print(f"Extracted {len(keywords)} unique keywords")
+```
+
+### `add_item()`
+```python
+def add_item(self, item: Item) -> bool
+```
+
+Manually add an item, applying merge strategy if duplicate:
+
+```python
+new_keyword = Keyword(term="Python", category="Language")
+added = keywords.add_item(new_keyword)
+print(f"Item added: {added}")
+```
+
+**Returns**: `True` if new item added, `False` if merged with existing
+
+### `get_item()`
+```python
+def get_item(self, key_value: str) -> Optional[Item]
+```
+
+Retrieve item by unique key value:
+
+```python
+ai_keyword = keywords.get_item("AI")
+if ai_keyword:
+ print(f"Found: {ai_keyword.category}")
+```
+
+### `remove_item()`
+```python
+def remove_item(self, key_value: str) -> bool
+```
+
+Remove item by unique key:
+
+```python
+removed = keywords.remove_item("obsolete_term")
+print(f"Removed: {removed}")
+```
+
+### `contains()`
+```python
+def contains(self, key_value: str) -> bool
+```
+
+Check if key exists:
+
+```python
+if keywords.contains("Python"):
+ print("Python is in the set")
+```
+
+## Set Operations
+
+`SetKnowledge` supports mathematical set operations:
+
+### Union (`|`)
+Combine two sets:
+```python
+combined = keywords1 | keywords2
+# All unique items from both sets
+```
+
+### Intersection (`&`)
+Common items:
+```python
+common = keywords1 & keywords2
+# Only items present in both sets
+```
+
+### Difference (`-`)
+Items in first but not second:
+```python
+unique_to_first = keywords1 - keywords2
+# Items only in keywords1
+```
+
+### Symmetric Difference (`^`)
+Items in either but not both:
+```python
+exclusive = keywords1 ^ keywords2
+# Items in one set but not the other
+```
+
+### Example:
+```python
+tech_keywords = SetKnowledge(item_schema=Keyword, ...)
+science_keywords = SetKnowledge(item_schema=Keyword, ...)
+
+# Extract from different sources
+tech_keywords.extract(tech_article)
+science_keywords.extract(science_article)
+
+# Find common keywords
+common = tech_keywords & science_keywords
+print(f"Keywords in both: {[k.term for k in common.items]}")
+
+# Find tech-specific keywords
+tech_only = tech_keywords - science_keywords
+print(f"Tech-specific: {[k.term for k in tech_only.items]}")
+```
+
+## Indexing and Search
+
+```python
+keywords.build_index()
+
+results = keywords.search("artificial intelligence topics", k=5)
+for doc, score in results:
+ print(f"Match (score {score:.3f}): {doc.page_content}")
+```
+
+## Use Cases
+
+### Unique Entities
+```python
+class Entity(BaseModel):
+ name: str = Field(..., description="Entity name")
+ type: str = Field(..., description="Entity type")
+ description: Optional[str] = None
+
+entities = SetKnowledge(
+ item_schema=Entity,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="name",
+ merge_strategy=MergeItemStrategy.FIELD_MERGE
+)
+```
+
+### Topic Modeling
+```python
+class Topic(BaseModel):
+ topic: str = Field(..., description="Topic name")
+ subtopics: List[str] = Field(default_factory=list)
+ relevance: Optional[str] = None
+
+topics = SetKnowledge(
+ item_schema=Topic,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="topic",
+ merge_strategy=MergeItemStrategy.LLM_MERGE # Smart merging
+)
+```
+
+### Skills Extraction
+```python
+class Skill(BaseModel):
+ skill_name: str = Field(..., description="Skill name")
+ proficiency: Optional[str] = None
+ years_experience: Optional[int] = None
+
+skills = SetKnowledge(
+ item_schema=Skill,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="skill_name",
+ merge_strategy=MergeItemStrategy.KEEP_NEW # Latest info
+)
+```
+
+## Advanced Usage
+
+### Custom Merge Logic
+
+For complex merging, use `LLM_MERGE` or subclass:
+
+```python
+class SmartSetKnowledge(SetKnowledge[Item]):
+ def _merge_items_llm(self, old_item: Item, new_item: Item) -> Item:
+ # Custom LLM prompt for merging
+ custom_prompt = f'''
+ Intelligently merge these two items:
+ Old: {old_item.model_dump_json()}
+ New: {new_item.model_dump_json()}
+
+ Rules:
+ - Prefer more detailed information
+ - Combine complementary fields
+ - Resolve conflicts logically
+ '''
+ # Use your LLM to merge
+ ...
+```
+
+### Filtering and Transformation
+
+```python
+# Filter by field
+important = [k for k in keywords if k.importance == "High"]
+
+# Group by category
+from collections import defaultdict
+by_category = defaultdict(list)
+for k in keywords.items:
+ by_category[k.category].append(k)
+
+# Transform
+terms_only = [k.term for k in keywords.items]
+```
+
+## Best Practices
+
+1. **Choose Unique Key Carefully**:
+ ```python
+ # Good: Natural unique identifier
+ unique_key="email" # For people
+ unique_key="isbn" # For books
+
+ # Careful: May have false duplicates
+ unique_key="name" # "John Smith" appears twice
+ ```
+
+2. **Merge Strategy Selection**:
+ - `KEEP_OLD`: When first mention is most authoritative
+ - `KEEP_NEW`: When latest information is better
+ - `FIELD_MERGE`: When information is complementary
+ - `LLM_MERGE`: When conflicts need intelligent resolution
+
+3. **Performance Considerations**:
+ ```python
+ # LLM_MERGE is slower (makes LLM calls)
+ SetKnowledge(..., merge_strategy=MergeItemStrategy.LLM_MERGE)
+
+ # For large datasets, consider FIELD_MERGE or KEEP_NEW
+ ```
+
+4. **Unique Key Validation**:
+ ```python
+ # Ensure unique_key field exists and is required
+ class Item(BaseModel):
+ id: str = Field(..., description="Unique ID") # Required!
+ ...
+
+ knowledge = SetKnowledge(..., unique_key="id")
+ ```
+
+## Serialization
+
+```python
+# Save
+keywords.dump("./keywords_set")
+
+# Load (preserves uniqueness)
+loaded = SetKnowledge.load(
+ "./keywords_set",
+ item_schema=Keyword,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="term", # Must specify again
+ merge_strategy=MergeItemStrategy.FIELD_MERGE
+)
+```
+
+## Comparison
+
+| Feature | ListKnowledge | SetKnowledge |
+|---------|---------------|--------------|
+| Duplicates | Allowed | Prevented by unique_key |
+| Merge Strategies | Simple append | 4 configurable strategies |
+| Storage | List | Dict (O(1) lookup) |
+| Set Operations | No | Yes (∪, ∩, -, ⊕) |
+| Use Case | Simple collections | Unique items, deduplication |
+
+## See Also
+
+- [ListKnowledge API](list.md) - Multi-item list extraction
+- [BaseKnowledge API](base.md) - Base class documentation
+- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guide
diff --git a/docs/api/unit.md b/docs/api/unit.md
new file mode 100644
index 0000000..164b79e
--- /dev/null
+++ b/docs/api/unit.md
@@ -0,0 +1,279 @@
+# UnitKnowledge API
+
+::: hyperextract.knowledge.generic.unit.UnitKnowledge
+ options:
+ show_root_heading: true
+ show_source: true
+ heading_level: 2
+
+## Overview
+
+`UnitKnowledge` implements the single-object extraction pattern. It's designed to extract exactly one structured object per document, making it ideal for document-level information like summaries, metadata, or aggregate statistics.
+
+## Class Signature
+
+```python
+class UnitKnowledge(BaseKnowledge[T]):
+ """Unit Knowledge Pattern - extracts a single structured object from text."""
+```
+
+## Key Characteristics
+
+- **Extraction Target**: One unique structured object per document
+- **Merge Strategy**: Field-level merge - first extraction wins, subsequent fills missing fields
+- **Indexing Strategy**: Each non-null field indexed independently
+- **Processing**: Uses LangChain batch processing for efficient multi-chunk handling
+
+## Initialization
+
+Inherits all parameters from `BaseKnowledge` with same defaults.
+
+```python
+from hyperextract.knowledge.generic import UnitKnowledge
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+from pydantic import BaseModel, Field
+
+class DocumentMetadata(BaseModel):
+ title: str = Field(default="", description="Document title")
+ author: str = Field(default="", description="Author name")
+ summary: str = Field(default="", description="Brief summary")
+
+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+
+knowledge = UnitKnowledge(
+ data_schema=DocumentMetadata,
+ llm_client=llm,
+ embedder=embedder,
+ chunk_size=2000,
+ chunk_overlap=200
+)
+```
+
+## Properties
+
+### `data`
+```python
+@property
+def data(self) -> T
+```
+
+Returns the single extracted object.
+
+**Example**:
+```python
+knowledge.extract(text)
+metadata = knowledge.data
+print(f"Title: {metadata.title}")
+print(f"Author: {metadata.author}")
+```
+
+## Methods
+
+### `extract()`
+
+Extracts a single object from text. For long texts:
+1. Splits into overlapping chunks
+2. Extracts object from each chunk in parallel
+3. Merges using first-priority strategy
+
+**Merge Behavior**:
+- First extraction: Populates all fields
+- Subsequent extractions: Only fill `None` or empty fields
+
+```python
+text = """
+Long document spanning multiple chunks...
+Title mentioned in chunk 1...
+Author mentioned in chunk 2...
+Summary in chunk 3...
+"""
+
+knowledge.extract(text)
+# Result: Single object with title, author, and summary
+```
+
+### `build_index()`
+
+Builds index from all non-null fields:
+
+```python
+knowledge.build_index()
+# Creates one document per field:
+# - Document(page_content=title, metadata={"field": "title"})
+# - Document(page_content=author, metadata={"field": "author"})
+# - Document(page_content=summary, metadata={"field": "summary"})
+```
+
+### `search()`
+
+Searches across all indexed fields:
+
+```python
+results = knowledge.search("research methodology", k=5)
+for doc, score in results:
+ field_name = doc.metadata["field"]
+ content = doc.page_content
+ print(f"Field '{field_name}' matched with score {score:.3f}")
+ print(f"Content: {content}")
+```
+
+## Merge Strategy Details
+
+The `_merge_strategy` method implements field-level merging:
+
+```python
+def _merge_strategy(self, old_data: T, new_data: T) -> T:
+ """Merge strategy: Keep old values, fill only None/empty fields from new."""
+```
+
+**Example**:
+```python
+# Chunk 1 extracts
+old_data = DocumentMetadata(
+ title="AI Research",
+ author="",
+ summary=""
+)
+
+# Chunk 2 extracts
+new_data = DocumentMetadata(
+ title="Different Title", # Ignored - old has value
+ author="John Smith", # Kept - old was empty
+ summary="A study..." # Kept - old was empty
+)
+
+# Merged result
+merged = DocumentMetadata(
+ title="AI Research", # From old
+ author="John Smith", # From new
+ summary="A study..." # From new
+)
+```
+
+## Indexing Strategy
+
+Each field becomes a separate document:
+
+```python
+def _build_documents_for_indexing(self) -> List[Document]:
+ """Build documents from non-null fields."""
+ documents = []
+ for field_name, value in self._data.model_dump().items():
+ if value and value != [] and value != "":
+ doc = Document(
+ page_content=str(value),
+ metadata={"field": field_name}
+ )
+ documents.append(doc)
+ return documents
+```
+
+## Use Cases
+
+### Document Summary
+```python
+class Summary(BaseModel):
+ main_topic: str = Field(default="", description="Main topic")
+ key_points: List[str] = Field(default_factory=list, description="Key points")
+ conclusion: str = Field(default="", description="Conclusion")
+
+summary_knowledge = UnitKnowledge(
+ data_schema=Summary,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+### Article Metadata
+```python
+class Article(BaseModel):
+ title: str = Field(default="", description="Article title")
+ publication_date: str = Field(default="", description="Publication date")
+ author: str = Field(default="", description="Author")
+ category: str = Field(default="", description="Category")
+
+article_knowledge = UnitKnowledge(
+ data_schema=Article,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+### Sentiment Analysis
+```python
+class Sentiment(BaseModel):
+ overall_sentiment: str = Field(default="", description="Overall sentiment")
+ positive_aspects: List[str] = Field(default_factory=list)
+ negative_aspects: List[str] = Field(default_factory=list)
+ confidence: float = Field(default=0.0, description="Confidence score")
+
+sentiment_knowledge = UnitKnowledge(
+ data_schema=Sentiment,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+## Best Practices
+
+1. **Clear Field Descriptions**: LLM uses descriptions during extraction
+ ```python
+ title: str = Field(default="", description="Full article title including subtitle")
+ ```
+
+2. **Appropriate Defaults**: Use empty strings for text, empty lists for sequences
+ ```python
+ name: str = Field(default="") # Not None
+ tags: List[str] = Field(default_factory=list) # Not None
+ ```
+
+3. **Chunk Sizing**: Adjust based on where information appears
+ ```python
+ # For metadata (usually at start)
+ UnitKnowledge(..., chunk_size=1000, chunk_overlap=100)
+
+ # For summary (may span document)
+ UnitKnowledge(..., chunk_size=3000, chunk_overlap=300)
+ ```
+
+4. **Custom Prompts**: Provide domain-specific guidance
+ ```python
+ custom_prompt = '''
+ Extract academic paper metadata. Pay special attention to:
+ - Author affiliations
+ - Publication venue
+ - Citation count if mentioned
+ '''
+
+ knowledge = UnitKnowledge(
+ data_schema=PaperMetadata,
+ llm_client=llm,
+ embedder=embedder,
+ prompt=custom_prompt
+ )
+ ```
+
+## Serialization
+
+```python
+# Save
+knowledge.dump("./document_metadata")
+
+# Load
+loaded = UnitKnowledge.load(
+ "./document_metadata",
+ data_schema=DocumentMetadata,
+ llm_client=llm,
+ embedder=embedder
+)
+
+assert loaded.data.title == knowledge.data.title
+```
+
+## See Also
+
+- [BaseKnowledge API](base.md) - Base class documentation
+- [ListKnowledge API](list.md) - For extracting multiple items
+- [SetKnowledge API](set.md) - For unique collections
+- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guide
diff --git a/docs/index.md b/docs/index.md
index 000ea34..db96777 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,17 +1,170 @@
-# Welcome to MkDocs
+# Hyper-Extract 🧠✨
-For full documentation visit [mkdocs.org](https://www.mkdocs.org).
+> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**
+>
+> Transform unstructured text into structured, evolving knowledge containers with the power of Large Language Models.
-## Commands
+---
-* `mkdocs new [dir-name]` - Create a new project.
-* `mkdocs serve` - Start the live-reloading docs server.
-* `mkdocs build` - Build the documentation site.
-* `mkdocs -h` - Print help message and exit.
+## 🎯 Vision
-## Project layout
+**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.
- mkdocs.yml # The configuration file.
- docs/
- index.md # The documentation homepage.
- ... # Other markdown pages, images and other files.
+Every knowledge container in Hyper-Extract follows a unified lifecycle:
+
+1. **Extract** - Intelligent extraction from text with automatic chunking and merging
+2. **Evolve** - Self-refinement through reasoning, pruning, and optimization
+3. **Index** - Semantic vector search capabilities
+4. **Serialize** - Persist and reload knowledge with full fidelity
+
+---
+
+## ✨ Key Features
+
+### 🤖 LLM-Native Design
+- **Deep LangChain Integration**: Every container is initialized with LLM and embedder instances
+- **Structured Output**: Leverage `with_structured_output` for type-safe Pydantic schema extraction
+- **Intelligent Merging**: First-priority merge strategy for chunk-based extraction
+
+### 🏗️ Unified Architecture
+- **Abstract Base Class**: `BaseKnowledge[T]` provides consistent interface across all knowledge types
+- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility
+- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline
+
+### 🔍 Semantic Search
+- **FAISS Integration**: Fast vector similarity search for knowledge retrieval
+- **Automatic Indexing**: Build and maintain indices with lazy loading and dirty tracking
+- **Field-Level Search**: Search across all schema fields with metadata preservation
+
+### 📦 Knowledge Patterns
+
+#### ✅ Implemented
+- **UnitKnowledge**: Single-object extraction for document-level information (summaries, metadata)
+- **ListKnowledge**: Collection-based extraction for entities, events, and structured lists
+- **SetKnowledge**: Unique collection with automatic deduplication and intelligent merge strategies
+
+#### 🚧 In Development
+- **EntityKnowledge**: Entity-centric extraction with attribute management
+- **RelationKnowledge**: Relationship extraction with entity linking
+- **GraphKnowledge**: Knowledge graph construction (nodes + edges)
+- **HypergraphKnowledge**: Hypergraph representation for complex multi-entity relations
+
+---
+
+## 🛠️ Installation
+
+### Prerequisites
+- Python 3.10+
+- OpenAI API key (or compatible LLM provider)
+
+### Install with uv (Recommended)
+```bash
+# Clone the repository
+git clone https://github.com/your-username/hyper-extract.git
+cd hyper-extract
+
+# Install dependencies with uv
+uv sync
+```
+
+### Install with pip
+```bash
+pip install -e .
+```
+
+---
+
+## 🚀 Quick Start
+
+### 1. Define Your Knowledge Schema
+
+```python
+from pydantic import BaseModel, Field
+from typing import List
+
+class ArticleKnowledge(BaseModel):
+ """Schema for article knowledge extraction"""
+ title: str = Field(default="", description="Article title")
+ author: str = Field(default="", description="Author name")
+ summary: str = Field(default="", description="Article summary")
+ key_points: List[str] = Field(default_factory=list, description="Key points")
+ topics: List[str] = Field(default_factory=list, description="Topic tags")
+```
+
+### 2. Initialize Knowledge Container
+
+```python
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+from hyperextract.knowledge.generic import UnitKnowledge
+
+# Initialize LLM and embedder
+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+
+# Create knowledge container
+knowledge = UnitKnowledge(
+ data_schema=ArticleKnowledge,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+### 3. Extract Knowledge
+
+```python
+# Extract from text
+text = """
+'Introduction to Deep Learning' is a comprehensive textbook
+written by Professor Li Ming...
+"""
+
+knowledge.extract(text)
+print(knowledge.data) # Access extracted structured knowledge
+```
+
+### 4. Semantic Search
+
+```python
+# Build index
+knowledge.build_index()
+
+# Search for relevant content
+results = knowledge.search("applications of deep learning", k=3)
+for doc, score in results:
+ print(f"Score: {score}, Content: {doc.page_content}")
+```
+
+### 5. Persistence
+
+```python
+# Save to disk
+knowledge.dump("./my_knowledge")
+
+# Load
+loaded_knowledge = UnitKnowledge.load(
+ "./my_knowledge",
+ data_schema=ArticleKnowledge,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+---
+
+## 📚 Documentation
+
+- **[Getting Started](user-guide/getting-started.md)** - Detailed tutorial and setup guide
+- **[Knowledge Patterns](user-guide/knowledge-patterns.md)** - Understanding different extraction patterns
+- **[API Reference](api/base.md)** - Complete API documentation
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome! Please see our contributing guidelines for details.
+
+---
+
+## 📄 License
+
+This project is licensed under the MIT License.
diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.md
new file mode 100644
index 0000000..cef91c0
--- /dev/null
+++ b/docs/user-guide/getting-started.md
@@ -0,0 +1,231 @@
+# Getting Started
+
+This guide will walk you through the basics of using Hyper-Extract to extract structured knowledge from unstructured text.
+
+## Installation
+
+### Prerequisites
+
+Before you begin, ensure you have:
+
+- **Python 3.10 or higher** installed
+- An **OpenAI API key** (or access to a compatible LLM provider)
+- Basic understanding of Python and Pydantic
+
+### Setting Up Your Environment
+
+1. **Clone the repository**:
+ ```bash
+ git clone https://github.com/your-username/hyper-extract.git
+ cd hyper-extract
+ ```
+
+2. **Install dependencies** (recommended: use `uv`):
+ ```bash
+ uv sync
+ ```
+
+ Or with pip:
+ ```bash
+ pip install -e .
+ ```
+
+3. **Configure your API key**:
+ Create a `.env` file in the project root:
+ ```bash
+ OPENAI_API_KEY=your-api-key-here
+ ```
+
+## Core Concepts
+
+### Knowledge Schema
+
+A **knowledge schema** is a Pydantic model that defines the structure of information you want to extract. It should include:
+
+- Field names and types
+- Default values
+- Descriptive annotations (used by the LLM during extraction)
+
+Example:
+```python
+from pydantic import BaseModel, Field
+from typing import List
+
+class BookReview(BaseModel):
+ """Schema for book review extraction"""
+ title: str = Field(default="", description="Book title")
+ author: str = Field(default="", description="Author name")
+ rating: float = Field(default=0.0, description="Rating out of 5")
+ pros: List[str] = Field(default_factory=list, description="Positive aspects")
+ cons: List[str] = Field(default_factory=list, description="Negative aspects")
+ recommendation: str = Field(default="", description="Final recommendation")
+```
+
+### Knowledge Container
+
+A **knowledge container** is an instance of a knowledge pattern class (e.g., `UnitKnowledge`, `ListKnowledge`, `SetKnowledge`) that manages extraction, storage, and retrieval of structured knowledge.
+
+### Extraction Pipeline
+
+The extraction pipeline automatically:
+
+1. **Chunks** long texts into manageable pieces
+2. **Extracts** structured data from each chunk in parallel
+3. **Merges** results according to the pattern's merge strategy
+4. **Indexes** the data for semantic search
+
+## Basic Usage Example
+
+### Step 1: Define Your Schema
+
+```python
+from pydantic import BaseModel, Field
+from typing import List
+
+class ResearchPaper(BaseModel):
+ """Research paper metadata extraction"""
+ title: str = Field(default="", description="Paper title")
+ authors: List[str] = Field(default_factory=list, description="Author names")
+ abstract: str = Field(default="", description="Paper abstract")
+ keywords: List[str] = Field(default_factory=list, description="Keywords")
+ methodology: str = Field(default="", description="Research methodology")
+ findings: List[str] = Field(default_factory=list, description="Key findings")
+ conclusion: str = Field(default="", description="Main conclusion")
+```
+
+### Step 2: Initialize LLM and Embedder
+
+```python
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+
+# Initialize models
+llm = ChatOpenAI(
+ model="gpt-4o-mini",
+ temperature=0 # Use 0 for consistent extraction
+)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+```
+
+### Step 3: Create Knowledge Container
+
+```python
+from hyperextract.knowledge.generic import UnitKnowledge
+
+# Create container
+paper_knowledge = UnitKnowledge(
+ data_schema=ResearchPaper,
+ llm_client=llm,
+ embedder=embedder,
+ chunk_size=2000, # Characters per chunk
+ chunk_overlap=200 # Overlap between chunks
+)
+```
+
+### Step 4: Extract Knowledge
+
+```python
+# Your source text
+paper_text = """
+Title: Deep Learning for Natural Language Processing
+
+Authors: John Smith, Jane Doe, Bob Johnson
+
+Abstract: This paper presents a comprehensive study on applying
+deep learning techniques to natural language processing tasks...
+
+[... rest of the paper ...]
+"""
+
+# Extract knowledge
+paper_knowledge.extract(paper_text)
+
+# Access extracted data
+print(paper_knowledge.data.title)
+print(paper_knowledge.data.authors)
+print(paper_knowledge.data.abstract)
+```
+
+### Step 5: Semantic Search
+
+```python
+# Build search index
+paper_knowledge.build_index()
+
+# Search for specific information
+results = paper_knowledge.search(
+ "What methodology was used?",
+ k=3 # Top 3 results
+)
+
+# Display results
+for doc, score in results:
+ print(f"Relevance: {score:.3f}")
+ print(f"Content: {doc.page_content}")
+ print(f"Metadata: {doc.metadata}")
+ print("---")
+```
+
+### Step 6: Save and Load
+
+```python
+# Save knowledge to disk
+paper_knowledge.dump("./paper_knowledge")
+
+# Later, load it back
+loaded_knowledge = UnitKnowledge.load(
+ "./paper_knowledge",
+ data_schema=ResearchPaper,
+ llm_client=llm,
+ embedder=embedder
+)
+
+# Knowledge is fully restored
+print(loaded_knowledge.data)
+```
+
+## Working with Long Documents
+
+Hyper-Extract automatically handles long documents by:
+
+1. **Splitting** text into overlapping chunks
+2. **Processing** chunks in parallel for efficiency
+3. **Merging** extracted information intelligently
+
+You can control chunking behavior:
+
+```python
+knowledge = UnitKnowledge(
+ data_schema=YourSchema,
+ llm_client=llm,
+ embedder=embedder,
+ chunk_size=3000, # Larger chunks for more context
+ chunk_overlap=300, # More overlap to avoid missing information
+ max_workers=5 # Control parallelism
+)
+```
+
+## Error Handling
+
+Always wrap extraction in try-except blocks:
+
+```python
+try:
+ knowledge.extract(text)
+except Exception as e:
+ print(f"Extraction failed: {e}")
+ # Handle error appropriately
+```
+
+## Next Steps
+
+- Learn about different [Knowledge Patterns](knowledge-patterns.md)
+- Explore the [API Reference](../api/base.md)
+- Check out the [examples](https://github.com/your-username/hyper-extract/tree/main/examples) folder
+
+## Tips for Better Extraction
+
+1. **Clear Descriptions**: Write detailed field descriptions in your schema - the LLM uses these
+2. **Appropriate Defaults**: Set sensible default values for optional fields
+3. **Type Hints**: Use proper Python type hints for better extraction accuracy
+4. **Chunk Size**: Adjust based on your document structure (2000-4000 chars works well)
+5. **Temperature**: Use 0 for consistent extraction, higher for creative tasks
diff --git a/docs/user-guide/knowledge-patterns.md b/docs/user-guide/knowledge-patterns.md
new file mode 100644
index 0000000..d34a28a
--- /dev/null
+++ b/docs/user-guide/knowledge-patterns.md
@@ -0,0 +1,359 @@
+# Knowledge Patterns
+
+Hyper-Extract provides three fundamental knowledge patterns, each optimized for different extraction scenarios. Understanding when to use each pattern is key to effective knowledge extraction.
+
+---
+
+## Overview
+
+| Pattern | Use Case | Deduplication | Merge Strategy |
+|---------|----------|---------------|----------------|
+| **UnitKnowledge** | Single object per document | N/A | Field-level merge |
+| **ListKnowledge** | Multiple independent items | Basic | Append |
+| **SetKnowledge** | Unique collection | Automatic | Configurable |
+
+---
+
+## UnitKnowledge Pattern
+
+### When to Use
+
+Use `UnitKnowledge` when you want to extract **exactly one structured object** from a document, regardless of its length.
+
+**Perfect for**:
+- Document summaries
+- Article metadata (title, author, date)
+- Aggregate statistics
+- Overall sentiment analysis
+- Document classification
+
+### How It Works
+
+1. **Extraction**: Processes text in chunks, extracting a schema instance from each
+2. **Merging**: Uses "first-wins" strategy - first extraction populates fields, subsequent extractions only fill missing values
+3. **Result**: Single consolidated object with information from entire document
+
+### Example: Document Metadata
+
+```python
+from pydantic import BaseModel, Field
+from typing import List
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+from hyperextract.knowledge.generic import UnitKnowledge
+
+# Define schema
+class DocumentMetadata(BaseModel):
+ title: str = Field(default="", description="Document title")
+ author: str = Field(default="", description="Primary author")
+ publication_date: str = Field(default="", description="Publication date")
+ category: str = Field(default="", description="Document category")
+ summary: str = Field(default="", description="Brief summary")
+ word_count_estimate: int = Field(default=0, description="Approximate word count")
+
+# Initialize
+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+
+metadata = UnitKnowledge(
+ data_schema=DocumentMetadata,
+ llm_client=llm,
+ embedder=embedder
+)
+
+# Extract
+long_article = """
+[Your long article text here...]
+"""
+metadata.extract(long_article)
+
+# Access single consolidated object
+print(metadata.data.title)
+print(metadata.data.summary)
+```
+
+### Indexing Strategy
+
+Each non-null field is indexed separately, allowing field-specific semantic search:
+
+```python
+metadata.build_index()
+
+# Search across all fields
+results = metadata.search("machine learning applications", k=5)
+```
+
+---
+
+## ListKnowledge Pattern
+
+### When to Use
+
+Use `ListKnowledge` when you need to extract **multiple independent items** of the same type from a document.
+
+**Perfect for**:
+- Entity lists (people, organizations, locations)
+- Event sequences
+- Product mentions
+- Citation extraction
+- Task lists
+
+### How It Works
+
+1. **Extraction**: Processes text in chunks, extracting multiple items from each
+2. **Merging**: Appends all items with basic deduplication (exact matches only)
+3. **Result**: List of all extracted items
+
+### Example: Entity Extraction
+
+```python
+from pydantic import BaseModel, Field
+from typing import List, Optional
+from hyperextract.knowledge.generic import ListKnowledge
+
+# Define item schema
+class Person(BaseModel):
+ name: str = Field(..., description="Person's full name")
+ role: Optional[str] = Field(None, description="Their role or title")
+ organization: Optional[str] = Field(None, description="Associated organization")
+ context: Optional[str] = Field(None, description="Context of mention")
+
+# Initialize
+people = ListKnowledge(
+ item_schema=Person, # Note: item_schema, not data_schema
+ llm_client=llm,
+ embedder=embedder
+)
+
+# Extract
+text = """
+Dr. Sarah Johnson, the lead researcher at MIT, collaborated with
+Professor Michael Chen from Stanford University. The project manager,
+Emily Rodriguez from Google, coordinated the efforts...
+"""
+people.extract(text)
+
+# Access list of items
+for person in people.items:
+ print(f"{person.name} - {person.role} at {person.organization}")
+
+# Or use the property
+print(f"Total people extracted: {len(people.items)}")
+```
+
+### List Operations
+
+`ListKnowledge` supports Pythonic list operations:
+
+```python
+# Append new items
+people.append(Person(name="John Doe", role="Engineer"))
+
+# Extend with multiple items
+new_people = [
+ Person(name="Alice Smith", role="Designer"),
+ Person(name="Bob Jones", role="Manager")
+]
+people.extend(new_people)
+
+# Access by index
+first_person = people[0]
+
+# Iterate
+for person in people:
+ print(person.name)
+
+# Length
+count = len(people)
+
+# Check membership
+if some_person in people:
+ print("Found!")
+```
+
+### Indexing Strategy
+
+Each item in the list is indexed independently:
+
+```python
+people.build_index()
+
+# Search returns relevant items with scores
+results = people.search("researchers from MIT", k=3)
+for doc, score in results:
+ print(f"Match: {doc.page_content} (score: {score})")
+```
+
+---
+
+## SetKnowledge Pattern
+
+### When to Use
+
+Use `SetKnowledge` when you need a **unique collection** with automatic deduplication based on a key field.
+
+**Perfect for**:
+- Keywords/tags (deduplicated by term)
+- Unique entities (deduplicated by ID or name)
+- Topic modeling (unique topics)
+- Skill extraction (unique skills)
+- Location mentions (unique places)
+
+### How It Works
+
+1. **Extraction**: Processes text in chunks, extracting multiple items
+2. **Deduplication**: Uses specified `unique_key` field to identify duplicates
+3. **Merging**: Configurable strategy (keep_old, keep_new, field_merge, llm_merge)
+4. **Result**: Unique collection with smart duplicate handling
+
+### Example: Keyword Extraction with Deduplication
+
+```python
+from pydantic import BaseModel, Field
+from typing import Optional
+from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy
+
+# Define item schema with unique key
+class Keyword(BaseModel):
+ term: str = Field(..., description="The keyword or phrase")
+ category: Optional[str] = Field(None, description="Keyword category")
+ importance: Optional[str] = Field(None, description="Importance level")
+ context: Optional[str] = Field(None, description="Usage context")
+
+# Initialize with unique_key
+keywords = SetKnowledge(
+ item_schema=Keyword,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="term", # Deduplicate by term
+ merge_strategy=MergeItemStrategy.FIELD_MERGE # Fill missing fields
+)
+
+# Extract
+text = """
+Machine learning is a key technology. Deep learning, a subset of
+machine learning, has revolutionized AI. Neural networks are fundamental
+to deep learning...
+"""
+keywords.extract(text)
+
+# Access unique items
+for kw in keywords.items:
+ print(f"{kw.term}: {kw.category} ({kw.importance})")
+```
+
+### Merge Strategies
+
+#### 1. KEEP_OLD
+Keeps existing item, discards new duplicate:
+```python
+keywords = SetKnowledge(
+ item_schema=Keyword,
+ llm_client=llm,
+ embedder=embedder,
+ unique_key="term",
+ merge_strategy=MergeItemStrategy.KEEP_OLD
+)
+```
+
+#### 2. KEEP_NEW (Default)
+Replaces existing item with new one:
+```python
+merge_strategy=MergeItemStrategy.KEEP_NEW
+```
+
+#### 3. FIELD_MERGE
+Merges fields - new values fill old's None/empty fields:
+```python
+merge_strategy=MergeItemStrategy.FIELD_MERGE
+```
+
+#### 4. LLM_MERGE
+Uses LLM to intelligently merge both items:
+```python
+merge_strategy=MergeItemStrategy.LLM_MERGE
+```
+
+### Set Operations
+
+`SetKnowledge` supports set operations:
+
+```python
+# Union: combine two sets
+combined = keywords1 | keywords2
+
+# Intersection: common items
+common = keywords1 & keywords2
+
+# Difference: items in first but not second
+unique = keywords1 - keywords2
+
+# Symmetric difference: items in either but not both
+exclusive = keywords1 ^ keywords2
+```
+
+### Indexing Strategy
+
+Like ListKnowledge, each unique item is indexed:
+
+```python
+keywords.build_index()
+results = keywords.search("AI technologies", k=5)
+```
+
+---
+
+## Choosing the Right Pattern
+
+### Decision Tree
+
+```
+Is it a single object per document?
+├─ Yes → Use UnitKnowledge
+└─ No → Are duplicates acceptable?
+ ├─ Yes → Use ListKnowledge
+ └─ No → Use SetKnowledge (specify unique_key)
+```
+
+### Examples by Domain
+
+**News Article Processing**:
+- `UnitKnowledge`: Article metadata, summary
+- `ListKnowledge`: All mentioned events
+- `SetKnowledge`: Unique entities, locations
+
+**Research Paper Analysis**:
+- `UnitKnowledge`: Paper metadata, abstract
+- `ListKnowledge`: All citations
+- `SetKnowledge`: Unique keywords, methodologies
+
+**Customer Reviews**:
+- `UnitKnowledge`: Overall sentiment, rating
+- `ListKnowledge`: All complaints/praises
+- `SetKnowledge`: Unique features mentioned
+
+---
+
+## Advanced: Custom Patterns
+
+You can create custom patterns by subclassing `BaseKnowledge`:
+
+```python
+from hyperextract.knowledge.base import BaseKnowledge
+
+class MyCustomPattern(BaseKnowledge[MySchema]):
+ def _merge_strategy(self, old_data, new_data):
+ # Implement custom merge logic
+ pass
+
+ def _build_documents_for_indexing(self):
+ # Implement custom indexing logic
+ pass
+```
+
+---
+
+## Next Steps
+
+- Explore [API Reference](../api/base.md) for detailed method documentation
+- Check [Getting Started](getting-started.md) for setup instructions
+- Review [examples](https://github.com/your-username/hyper-extract/tree/main/examples) for complete code samples
diff --git a/docs/zh/index.md b/docs/zh/index.md
new file mode 100644
index 0000000..cc21c48
--- /dev/null
+++ b/docs/zh/index.md
@@ -0,0 +1,169 @@
+# Hyper-Extract 🧠✨
+
+> **一个智能的、基于 LLM 的知识提取与演化框架**
+>
+> 借助大语言模型的力量,将非结构化文本转化为结构化、可演化的知识容器。
+
+---
+
+## 🎯 愿景
+
+**Hyper-Extract** 将知识提取重新定义为一个智能的、自我演化的过程。我们不是将数据结构视为被动容器,而是赋予它们 **LLM 驱动的智能**,使其能够主动地从非结构化文本中提取、合并、演化和索引知识。
+
+Hyper-Extract 中的每个知识容器都遵循统一的生命周期:
+
+1. **Extract(提取)** - 使用 LLM 从文本智能提取,自动处理分块和合并
+2. **Evolve(演化)** - 通过推理、剪枝和优化实现自我优化
+3. **Index(索引)** - 提供语义向量搜索能力
+4. **Serialize(序列化)** - 完整保存和加载知识状态
+
+---
+
+## ✨ 核心特性
+
+### 🤖 LLM 原生设计
+- **深度 LangChain 集成**:每个容器都使用 LLM 和嵌入模型初始化
+- **结构化输出**:利用 `with_structured_output` 实现类型安全的 Pydantic 模式提取
+- **智能合并**:针对分块提取的优先级合并策略
+
+### 🏗️ 统一架构
+- **抽象基类**:`BaseKnowledge[T]` 为所有知识类型提供一致的接口
+- **泛型类型支持**:使用 `Generic[T]` 实现完整的 Python 类型提示
+- **生命周期管理**:标准化的 `extract` → `merge` → `evolve` → `search` → `dump/load` 流程
+
+### 🔍 语义搜索
+- **FAISS 集成**:快速的向量相似度搜索
+- **自动索引**:支持懒加载和脏数据跟踪
+- **字段级搜索**:可以在所有 schema 字段中搜索,并保留元数据
+
+### 📦 知识模式
+
+#### ✅ 已实现
+- **UnitKnowledge**:单对象提取,适用于文档级信息(摘要、元数据)
+- **ListKnowledge**:列表提取,适用于实体、事件和结构化列表
+- **SetKnowledge**:唯一集合,支持自动去重和智能合并策略
+
+#### 🚧 开发中
+- **EntityKnowledge**:实体中心的提取,支持属性管理
+- **RelationKnowledge**:关系提取,支持实体链接
+- **GraphKnowledge**:知识图谱构建(节点 + 边)
+- **HypergraphKnowledge**:超图表示,用于复杂的多实体关系
+
+---
+
+## 🛠️ 安装
+
+### 前置要求
+- Python 3.10+
+- OpenAI API 密钥(或兼容的 LLM 提供商)
+
+### 使用 uv 安装(推荐)
+```bash
+# 克隆仓库
+git clone https://github.com/your-username/hyper-extract.git
+cd hyper-extract
+
+# 使用 uv 安装依赖
+uv sync
+```
+
+### 使用 pip 安装
+```bash
+pip install -e .
+```
+
+---
+
+## 🚀 快速开始
+
+### 1. 定义知识 Schema
+
+```python
+from pydantic import BaseModel, Field
+from typing import List
+
+class ArticleKnowledge(BaseModel):
+ """文章知识提取的 Schema"""
+ title: str = Field(default="", description="文章标题")
+ author: str = Field(default="", description="作者名称")
+ summary: str = Field(default="", description="文章摘要")
+ key_points: List[str] = Field(default_factory=list, description="关键要点")
+ topics: List[str] = Field(default_factory=list, description="主题标签")
+```
+
+### 2. 初始化知识容器
+
+```python
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+from hyperextract.knowledge.generic import UnitKnowledge
+
+# 初始化 LLM 和嵌入模型
+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
+embedder = OpenAIEmbeddings(model="text-embedding-3-small")
+
+# 创建知识容器
+knowledge = UnitKnowledge(
+ data_schema=ArticleKnowledge,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+### 3. 提取知识
+
+```python
+# 从文本提取
+text = """
+《深度学习导论》是由李明教授撰写的一本综合性教材...
+"""
+
+knowledge.extract(text)
+print(knowledge.data) # 访问提取的结构化知识
+```
+
+### 4. 语义搜索
+
+```python
+# 构建索引
+knowledge.build_index()
+
+# 搜索相关内容
+results = knowledge.search("深度学习的主要应用", k=3)
+for doc, score in results:
+ print(f"相关度: {score}, 内容: {doc.page_content}")
+```
+
+### 5. 持久化
+
+```python
+# 保存到磁盘
+knowledge.dump("./my_knowledge")
+
+# 加载
+loaded_knowledge = UnitKnowledge.load(
+ "./my_knowledge",
+ data_schema=ArticleKnowledge,
+ llm_client=llm,
+ embedder=embedder
+)
+```
+
+---
+
+## 📚 文档导航
+
+- **[快速入门指南](user-guide/getting-started.md)** - 详细的入门教程
+- **[知识模式指南](user-guide/knowledge-patterns.md)** - 了解不同的知识提取模式
+- **[API 参考](api/base.md)** - 完整的 API 文档
+
+---
+
+## 🤝 贡献
+
+欢迎贡献!请查看我们的贡献指南了解详情。
+
+---
+
+## 📄 许可证
+
+本项目基于 MIT 许可证开源。
diff --git a/hyperextract/config.py b/hyperextract/config.py
index 5ff8553..e5d6112 100644
--- a/hyperextract/config.py
+++ b/hyperextract/config.py
@@ -1,20 +1,20 @@
-# src/hyperextract/config.py
+# hyperextract/config.py
import sys
import os
from pathlib import Path
from dotenv import load_dotenv
from loguru import logger
-# 1. 自动加载 .env
-# 寻找项目根目录的 .env 文件
+# 1. Auto load .env
+# Find the .env file in the project root directory
env_path = Path(__file__).parent.parent / ".env"
load_dotenv(dotenv_path=env_path)
-# 2. 配置 Loguru
-# 移除默认的 handler (避免重复或格式不统一)
+# 2. Configure Loguru
+# Remove default handler (to avoid duplication or inconsistent formatting)
logger.remove()
-# 添加控制台输出 (开发模式下级别可以是 DEBUG)
+# Add console output (development mode level can be DEBUG)
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
logger.add(
sys.stderr,
@@ -22,15 +22,15 @@ logger.add(
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>",
)
-# 添加文件输出 (自动轮转,保存日志文件)
+# Add file output (automatic rotation, save log files)
logger.add(
"logs/hyperextract.log",
- rotation="10 MB", # 文件超过 10MB 就分割
- retention="10 days", # 只保留最近 10 天
- level="DEBUG", # 文件里记录更详细的信息
- compression="zip", # 历史日志压缩
+ rotation="10 MB", # Split file when it exceeds 10MB
+ retention="10 days", # Only keep recent 10 days
+ level="DEBUG", # Record more detailed information in file
+ compression="zip", # Compress historical logs
)
-# 导出 logger 供其他模块使用
-__all__ = ["logger"]
+# Export logger for use in other modules
+__all__ = ["logger"]
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index cc5e49d..5f74227 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -1,18 +1,22 @@
-site_name: HyperExtract
-site_description: Context-aware knowledge extraction library.
-site_url: https://your-github-username.github.io/hyperextract/
+site_name: Hyper-Extract
+site_description: An intelligent, LLM-powered knowledge extraction and evolution framework
+site_url: https://your-github-username.github.io/hyper-extract/
+repo_url: https://github.com/your-username/hyper-extract
+repo_name: hyper-extract
-# 核心主题配置
+# Theme configuration
theme:
name: material
- palette:
- # 切换日间/夜间模式的按钮
+ language: en
+ palette:
+ # Light mode
- scheme: default
primary: teal
accent: purple
toggle:
icon: material/brightness-7
name: Switch to dark mode
+ # Dark mode
- scheme: slate
primary: teal
accent: lime
@@ -20,24 +24,80 @@ theme:
icon: material/brightness-4
name: Switch to light mode
features:
- - content.code.copy # 代码块复制按钮
- - navigation.tabs # 顶部导航栏
+ - content.code.copy # Code copy button
+ - content.code.annotate # Code annotations
+ - navigation.tabs # Top navigation tabs
+ - navigation.tabs.sticky # Sticky tabs
+ - navigation.sections # Navigation sections
+ - navigation.expand # Expand navigation by default
+ - navigation.top # Back to top button
+ - search.suggest # Search suggestions
+ - search.highlight # Highlight search results
+ - toc.follow # TOC follows scroll
+ icon:
+ repo: fontawesome/brands/github
-# 插件配置
+# Plugins configuration
plugins:
- - search
+ - search:
+ lang:
+ - en
+ - zh
- mkdocstrings:
default_handler: python
handlers:
python:
- paths: [src] # 告诉插件源码在哪里
+ paths: [.]
+ options:
+ show_source: true
+ show_root_heading: true
+ show_root_full_path: false
+ heading_level: 2
+ docstring_style: google
-# 导航栏结构
+# Markdown extensions
+markdown_extensions:
+ - pymdownx.highlight:
+ anchor_linenums: true
+ line_spans: __span
+ pygments_lang_class: true
+ - pymdownx.inlinehilite
+ - pymdownx.snippets
+ - pymdownx.superfences
+ - pymdownx.tabbed:
+ alternate_style: true
+ - pymdownx.details
+ - admonition
+ - tables
+ - attr_list
+ - md_in_html
+ - toc:
+ permalink: true
+
+# Extra configuration
+extra:
+ social:
+ - icon: fontawesome/brands/github
+ link: https://github.com/your-username/hyper-extract
+ alternate:
+ - name: English
+ link: /
+ lang: en
+ - name: 中文
+ link: /zh/
+ lang: zh
+
+# Navigation structure
nav:
- Home: index.md
- User Guide:
- Getting Started: user-guide/getting-started.md
- - Configuration: user-guide/configuration.md
- - API Reference: # 这里用来自动生成 API 文档
- - Extractor: api/extractor.md
- - Strategies: api/strategies.md
+ - Knowledge Patterns: user-guide/knowledge-patterns.md
+ - API Reference:
+ - Overview: api/index.md
+ - BaseKnowledge: api/base.md
+ - UnitKnowledge: api/unit.md
+ - ListKnowledge: api/list.md
+ - SetKnowledge: api/set.md
+ - 中文文档:
+ - 首页: zh/index.md