Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
Add UnitKnowledge API documentation and enhance user guide
yifanfeng97ae86ec15256c
diff --git a/docs/BUILDING.md b/docs/BUILDING.mdnew file mode 100644index 0000000..0cd6674--- /dev/null+++ b/docs/BUILDING.md@@ -0,0 +1,398 @@+# Building Documentation++This guide explains how to build and preview the Hyper-Extract documentation.++## Prerequisites++The documentation uses MkDocs with Material theme. Required dependencies are in the `dev` dependency group:++```toml+[dependency-groups]+dev = [+ "mkdocs>=1.6.1",+ "mkdocs-material>=9.7.1",+ "mkdocstrings[python]>=1.0.0",+]+```++## Installation++### Using uv (Recommended)++```bash+# Install all dependencies including dev dependencies+uv sync++# Activate virtual environment+.venv\Scripts\activate # Windows+source .venv/bin/activate # Linux/macOS+```++### Using pip++```bash+# Install project with dev dependencies+pip install -e ".[dev]"+```++## Building Documentation++### Development Server (Live Preview)++Start a local server with live reloading:++```bash+mkdocs serve+```++Then open your browser to `http://127.0.0.1:8000/`++- Changes to `.md` files auto-reload+- Hot reload for configuration changes+- Useful for writing and testing documentation++### Build Static Site++Generate the static HTML site:++```bash+mkdocs build+```++Output is in `site/` directory:+```+site/+├── index.html+├── user-guide/+│ ├── getting-started/+│ └── knowledge-patterns/+├── api/+│ ├── base/+│ ├── unit/+│ ├── list/+│ └── set/+├── zh/+│ └── index.html+└── assets/+```++### Build with Clean++Remove old build artifacts first:++```bash+mkdocs build --clean+```++### Strict Mode++Build with warnings as errors:++```bash+mkdocs build --strict+```++Useful for CI/CD to catch documentation issues.++## Documentation Structure++```+docs/+├── index.md # English homepage+├── user-guide/+│ ├── getting-started.md # Installation and basic usage+│ └── knowledge-patterns.md # Pattern selection guide+├── api/+│ ├── index.md # API overview+│ ├── base.md # BaseKnowledge API+│ ├── unit.md # UnitKnowledge API+│ ├── list.md # ListKnowledge API+│ └── set.md # SetKnowledge API+└── zh/+ └── index.md # Chinese homepage (中文首页)+```++## Multi-Language Support++### Current Languages++- **English** (default): `/`+- **中文** (Chinese): `/zh/`++### Adding New Languages++1. Create language directory:+ ```bash+ mkdir docs/ja # Japanese example+ ```++2. Add translation files:+ ```bash+ cp docs/index.md docs/ja/index.md+ # Translate content to Japanese+ ```++3. Update `mkdocs.yml`:+ ```yaml+ extra:+ alternate:+ - name: English+ link: /+ lang: en+ - name: 中文+ link: /zh/+ lang: zh+ - name: 日本語+ link: /ja/+ lang: ja+ ```++4. Add to navigation:+ ```yaml+ nav:+ # ... existing nav ...+ - 日本語:+ - ホーム: ja/index.md+ ```++## Configuration++### MkDocs Configuration (`mkdocs.yml`)++Key sections:++```yaml+# Site metadata+site_name: Hyper-Extract+site_description: An intelligent, LLM-powered knowledge extraction framework++# Theme+theme:+ name: material+ language: en+ features:+ - navigation.tabs+ - search.suggest+ - content.code.copy++# Plugins+plugins:+ - search+ - mkdocstrings # Auto-generate API docs from docstrings++# Markdown extensions+markdown_extensions:+ - pymdownx.highlight+ - pymdownx.superfences+ - admonition+```++### MkDocstrings Configuration++Auto-generates API documentation from Python docstrings:++```yaml+plugins:+ - mkdocstrings:+ handlers:+ python:+ paths: [.] # Search path for modules+ options:+ show_source: true+ show_root_heading: true+ docstring_style: google+```++Usage in markdown:+```markdown+# BaseKnowledge API++::: hyperextract.knowledge.base.BaseKnowledge+ options:+ show_source: true+ heading_level: 2+```++## Deployment++### GitHub Pages++1. **Build documentation**:+ ```bash+ mkdocs gh-deploy+ ```++ This command:+ - Builds the documentation+ - Commits to `gh-pages` branch+ - Pushes to GitHub++2. **Configure GitHub Pages**:+ - Go to repository Settings → Pages+ - Source: Deploy from branch+ - Branch: `gh-pages` / `root`++### Manual Deployment++1. Build static site:+ ```bash+ mkdocs build+ ```++2. Deploy `site/` directory to your hosting:+ ```bash+ # Example: Deploy to Netlify+ netlify deploy --dir=site --prod++ # Example: Deploy to Vercel+ vercel --prod site/+ ```++## Continuous Integration++### GitHub Actions++Create `.github/workflows/docs.yml`:++```yaml+name: Deploy Documentation++on:+ push:+ branches:+ - main+ paths:+ - 'docs/**'+ - 'mkdocs.yml'++jobs:+ deploy:+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4++ - uses: actions/setup-python@v5+ with:+ python-version: '3.10'++ - name: Install dependencies+ run: |+ pip install mkdocs mkdocs-material mkdocstrings[python]++ - name: Deploy documentation+ run: mkdocs gh-deploy --force+```++## Tips++### Writing Documentation++1. **Use code blocks with language hints**:+ ````markdown+ ```python+ from hyperextract import UnitKnowledge+ ```+ ````++2. **Add admonitions for important notes**:+ ```markdown+ !!! note+ This is an important note.++ !!! warning+ Be careful with this feature.+ ```++3. **Link to other pages**:+ ```markdown+ See [Getting Started](user-guide/getting-started.md)+ See [API Reference](api/base.md#extract)+ ```++4. **Use tabs for alternatives**:+ ````markdown+ === "Python"+ ```python+ knowledge.extract(text)+ ```++ === "Result"+ ```json+ {"title": "Example"}+ ```+ ````++### Auto-generating API Docs++For classes with good docstrings:++```markdown+# MyClass API++::: mymodule.MyClass+ options:+ show_source: true+ members:+ - __init__+ - method1+ - method2+```++### Search Optimization++MkDocs indexes all content automatically. To improve searchability:++- Use clear headings+- Include keywords in descriptions+- Add synonyms in metadata++## Troubleshooting++### MkDocs not found++```bash+# Install mkdocs+pip install mkdocs mkdocs-material mkdocstrings[python]+```++### Module not found (mkdocstrings)++Ensure the project root is in Python path:++```yaml+# mkdocs.yml+plugins:+ - mkdocstrings:+ handlers:+ python:+ paths: [.] # Current directory+```++### Port already in use++```bash+# Use different port+mkdocs serve --dev-addr=127.0.0.1:8001+```++### Build errors++```bash+# Verbose output+mkdocs build --verbose++# Clean and rebuild+mkdocs build --clean+```++## Resources++- [MkDocs Documentation](https://www.mkdocs.org/)+- [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/)+- [MkDocstrings](https://mkdocstrings.github.io/)++## Need Help?++- Check [MkDocs User Guide](https://www.mkdocs.org/user-guide/)+- See [Material Theme Reference](https://squidfunk.github.io/mkdocs-material/reference/)+- Open an issue on GitHubdiff --git a/docs/api/base.md b/docs/api/base.mdnew file mode 100644index 0000000..a29b8d8--- /dev/null+++ b/docs/api/base.md@@ -0,0 +1,296 @@+# BaseKnowledge API++::: hyperextract.knowledge.base.BaseKnowledge+ options:+ show_root_heading: true+ show_source: true+ heading_level: 2++## Overview++`BaseKnowledge` is the abstract base class that provides the foundation for all knowledge extraction patterns in Hyper-Extract. It implements the core lifecycle operations: extraction, merging, indexing, evolution, and serialization.++## Class Signature++```python+class BaseKnowledge(ABC, Generic[T]):+ """Unified knowledge base class integrating extraction, storage,+ aggregation, and evolution."""+```++Where `T` is a Pydantic `BaseModel` subclass defining your knowledge schema.++## Initialization++### Parameters++| Parameter | Type | Default | Description |+|-----------|------|---------|-------------|+| `data_schema` | `Type[T]` | Required | Pydantic BaseModel subclass defining knowledge structure |+| `llm_client` | `BaseChatModel` | Required | LangChain LLM client for extraction |+| `embedder` | `Embeddings` | Required | Embedding model for semantic search |+| `prompt` | `str` | `""` | Custom extraction prompt (uses default if empty) |+| `chunk_size` | `int` | `2000` | Maximum characters per text chunk |+| `chunk_overlap` | `int` | `200` | Overlapping characters between chunks |+| `max_workers` | `int` | `10` | Maximum concurrent extraction tasks |+| `show_progress` | `bool` | `True` | Whether to display progress logs |++### Example++```python+from hyperextract.knowledge.base import BaseKnowledge+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from pydantic import BaseModel, Field++class MySchema(BaseModel):+ title: str = Field(default="", description="Document title")+ summary: str = Field(default="", description="Brief summary")++llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")++# Note: BaseKnowledge is abstract, use concrete implementations+# knowledge = UnitKnowledge(...)+```++## Properties++### `data`+```python+@property+def data(self) -> T+```+Returns the extracted knowledge data. Type is your specified schema `T`.++**Example**:+```python+knowledge.extract(text)+print(knowledge.data.title)+```++### `metadata`+```python+@property+def metadata(self) -> Dict[str, Any]+```+Returns metadata about the knowledge object, including creation and update timestamps.++**Example**:+```python+print(knowledge.metadata["created_at"])+print(knowledge.metadata["updated_at"])+```++## Core Methods++### `extract()`+```python+def extract(+ self,+ text: str,+ *,+ additional_info: str = "",+ merge: bool = True+) -> T+```++Extract structured knowledge from text.++**Parameters**:+- `text` (str): Source text to extract from+- `additional_info` (str): Optional context or instructions+- `merge` (bool): Whether to merge with existing data (default: True)++**Returns**: Extracted data of type `T`++**Example**:+```python+result = knowledge.extract(+ text="Long article text...",+ additional_info="Focus on technical details",+ merge=True+)+```++### `build_index()`+```python+def build_index(self, force_rebuild: bool = False) -> None+```++Build or rebuild the FAISS vector index for semantic search.++**Parameters**:+- `force_rebuild` (bool): Force rebuild even if index exists++**Example**:+```python+knowledge.build_index() # Build if not exists+knowledge.build_index(force_rebuild=True) # Force rebuild+```++### `search()`+```python+def search(+ self,+ query: str,+ k: int = 5,+ **kwargs+) -> List[Tuple[Document, float]]+```++Perform semantic search over indexed knowledge.++**Parameters**:+- `query` (str): Search query+- `k` (int): Number of results to return+- `**kwargs`: Additional arguments passed to FAISS++**Returns**: List of (Document, similarity_score) tuples++**Example**:+```python+results = knowledge.search("machine learning applications", k=3)+for doc, score in results:+ print(f"Score: {score:.3f}")+ print(f"Content: {doc.page_content}")+ print(f"Metadata: {doc.metadata}")+```++### `dump()`+```python+def dump(self, save_path: str | Path) -> None+```++Save knowledge to disk, including data, metadata, and index.++**Parameters**:+- `save_path` (str | Path): Directory path to save to++**Structure**:+```+save_path/+├── state.json # Data and metadata+└── faiss_index/ # Vector index+ └── index.faiss+```++**Example**:+```python+knowledge.dump("./my_knowledge")+```++### `load()` (Class Method)+```python+@classmethod+def load(+ cls,+ load_path: str | Path,+ data_schema: Type[T],+ llm_client: BaseChatModel,+ embedder: Embeddings,+ **kwargs+) -> "BaseKnowledge[T]"+```++Load knowledge from disk.++**Parameters**:+- `load_path` (str | Path): Directory to load from+- `data_schema` (Type[T]): Schema class (must match saved data)+- `llm_client` (BaseChatModel): LLM client for future operations+- `embedder` (Embeddings): Embedder for search+- `**kwargs`: Additional initialization parameters++**Returns**: Loaded knowledge instance++**Example**:+```python+loaded = UnitKnowledge.load(+ "./my_knowledge",+ data_schema=MySchema,+ llm_client=llm,+ embedder=embedder+)+```++## Abstract Methods++Subclasses must implement these methods:++### `_default_prompt()`+```python+@staticmethod+@abstractmethod+def _default_prompt() -> str+```++Returns the default extraction prompt for the pattern.++### `_merge_strategy()`+```python+@abstractmethod+def _merge_strategy(self, old_data: T, new_data: T) -> T+```++Defines how to merge old and new extracted data.++### `_build_documents_for_indexing()`+```python+@abstractmethod+def _build_documents_for_indexing(self) -> List[Document]+```++Converts knowledge data into LangChain Documents for indexing.++## Internal Methods++### `_extract_single_chunk()`+```python+def _extract_single_chunk(+ self,+ chunk_text: str,+ additional_info: str = ""+) -> T+```++Extracts knowledge from a single text chunk. Used internally by `extract()`.++### `_extract_multi_chunk()`+```python+def _extract_multi_chunk(+ self,+ text: str,+ additional_info: str = ""+) -> T+```++Handles multi-chunk extraction with parallel processing and merging.++## Usage in Subclasses++When creating custom knowledge patterns, inherit from `BaseKnowledge`:++```python+from hyperextract.knowledge.base import BaseKnowledge, T+from typing import List+from langchain_core.documents import Document++class CustomKnowledge(BaseKnowledge[T]):+ @staticmethod+ def _default_prompt() -> str:+ return "Extract information according to schema: "++ def _merge_strategy(self, old_data: T, new_data: T) -> T:+ # Implement merge logic+ return new_data # Simple replace++ def _build_documents_for_indexing(self) -> List[Document]:+ # Convert self._data to documents+ return [Document(page_content=str(self._data))]+```++## See Also++- [UnitKnowledge API](unit.md) - Single-object extraction+- [ListKnowledge API](list.md) - Multi-item extraction+- [SetKnowledge API](set.md) - Unique collection extractiondiff --git a/docs/api/index.md b/docs/api/index.mdnew file mode 100644index 0000000..5c145e6--- /dev/null+++ b/docs/api/index.md@@ -0,0 +1,81 @@+# API Reference++This section provides detailed API documentation for all Hyper-Extract classes and methods.++## Module Structure++```+hyperextract/+├── knowledge/+│ ├── base.py # BaseKnowledge abstract class+│ └── generic/+│ ├── unit.py # UnitKnowledge+│ ├── list.py # ListKnowledge+│ └── set.py # SetKnowledge+```++## Quick Links++- [BaseKnowledge](base.md) - Abstract base class for all knowledge patterns+- [UnitKnowledge](unit.md) - Single-object extraction pattern+- [ListKnowledge](list.md) - Multi-item list extraction pattern+- [SetKnowledge](set.md) - Unique collection with deduplication++## Common Patterns++### Initialization++All knowledge classes share a common initialization pattern:++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")++knowledge = PatternClass(+ data_schema=YourSchema, # or item_schema for List/Set+ llm_client=llm,+ embedder=embedder,+ chunk_size=2000,+ chunk_overlap=200+)+```++### Core Methods++Every knowledge class implements these methods:++| Method | Description | Returns |+|--------|-------------|---------|+| `extract(text)` | Extract knowledge from text | None (updates internal state) |+| `build_index()` | Build FAISS vector index | None |+| `search(query, k)` | Semantic search | List of (Document, score) tuples |+| `dump(path)` | Save to disk | None |+| `load(path, ...)` | Load from disk | Knowledge instance |++### Type Hints++All classes are fully typed with Python generics:++```python+from hyperextract.knowledge.generic import UnitKnowledge+from pydantic import BaseModel++class MySchema(BaseModel):+ field: str++# Type-safe: knowledge.data is of type MySchema+knowledge: UnitKnowledge[MySchema] = UnitKnowledge(+ data_schema=MySchema,+ llm_client=llm,+ embedder=embedder+)+```++## Next: Detailed API Docs++- [BaseKnowledge API](base.md)+- [UnitKnowledge API](unit.md)+- [ListKnowledge API](list.md)+- [SetKnowledge API](set.md)diff --git a/docs/api/list.md b/docs/api/list.mdnew file mode 100644index 0000000..2289e37--- /dev/null+++ b/docs/api/list.md@@ -0,0 +1,351 @@+# ListKnowledge API++::: hyperextract.knowledge.generic.list.ListKnowledge+ options:+ show_root_heading: true+ show_source: true+ heading_level: 2++## Overview++`ListKnowledge` implements the multi-item list extraction pattern. It extracts multiple independent objects of the same type from a document, suitable for entity lists, events, citations, and more.++## Class Signature++```python+class ListKnowledge(BaseKnowledge[ItemListSchema[Item]], Generic[Item]):+ """List Knowledge Pattern - extracts a collection of objects from text."""+```++## Key Characteristics++- **Extraction Target**: A collection of structured objects+- **Merge Strategy**: Append with basic deduplication (exact matches)+- **Indexing Strategy**: Each item indexed independently+- **Container**: Uses `ItemListSchema[Item]` wrapper++## Initialization++```python+from hyperextract.knowledge.generic import ListKnowledge+from pydantic import BaseModel, Field+from typing import Optional++class Person(BaseModel):+ name: str = Field(..., description="Person's full name")+ role: Optional[str] = Field(None, description="Role or title")+ organization: Optional[str] = Field(None, description="Organization")++people = ListKnowledge(+ item_schema=Person, # Note: item_schema, not data_schema!+ llm_client=llm,+ embedder=embedder+)+```++## Properties++### `items`+```python+@property+def items(self) -> List[Item]+```++Returns the list of extracted items.++```python+people.extract(text)+for person in people.items:+ print(f"{person.name} - {person.role}")+```++### `data`+```python+@property+def data(self) -> ItemListSchema[Item]+```++Returns the container wrapper with `.items` attribute.++```python+container = people.data+print(len(container.items))+```++## List Operations++`ListKnowledge` supports Pythonic list operations:++### Indexing+```python+first = people[0]+last = people[-1]+slice_items = people[1:3]+```++### Iteration+```python+for person in people:+ print(person.name)+```++### Length+```python+count = len(people)+```++### Membership+```python+if some_person in people:+ print("Found!")+```++### Append+```python+people.append(Person(name="John Doe", role="Engineer"))+```++### Extend+```python+new_people = [+ Person(name="Alice", role="Designer"),+ Person(name="Bob", role="Manager")+]+people.extend(new_people)+```++### Clear+```python+people.clear() # Remove all items+```++## Methods++### `extract()`++Extracts multiple items from text:++```python+text = """+Dr. Sarah Johnson leads the research team at MIT.+Professor Michael Chen from Stanford collaborated on the project.+Emily Rodriguez from Google coordinated the efforts.+"""++people.extract(text)+print(f"Extracted {len(people)} people")++for person in people:+ print(f"- {person.name}: {person.role} at {person.organization}")+```++**Multi-chunk behavior**:+- Each chunk extracts multiple items+- All items are appended together+- Basic deduplication removes exact duplicates++### `build_index()`++Indexes each item separately:++```python+people.build_index()+# Creates one document per item with all fields serialized+```++### `search()`++Searches across all items:++```python+results = people.search("researchers from MIT", k=3)+for doc, score in results:+ # doc.page_content contains serialized item+ # doc.metadata contains item index+ print(f"Match (score {score:.3f}): {doc.page_content}")+```++## Merge Strategy++Appends new items with basic deduplication:++```python+def _merge_strategy(+ self,+ old_data: ItemListSchema[Item],+ new_data: ItemListSchema[Item]+) -> ItemListSchema[Item]:+ """Append new items, removing exact duplicates."""+```++**Example**:+```python+# Old items+[Person(name="Alice", role="Engineer")]++# New items (from next chunk)+[Person(name="Bob", role="Designer"),+ Person(name="Alice", role="Engineer")] # Duplicate++# Merged result+[Person(name="Alice", role="Engineer"),+ Person(name="Bob", role="Designer")]+```++## Indexing Strategy++```python+def _build_documents_for_indexing(self) -> List[Document]:+ """Build one document per item."""+ return [+ Document(+ page_content=item.model_dump_json(),+ metadata={"item_index": i}+ )+ for i, item in enumerate(self.items)+ ]+```++## Use Cases++### Entity Extraction+```python+class Entity(BaseModel):+ name: str = Field(..., description="Entity name")+ type: str = Field(..., description="Entity type (person/org/location)")+ context: str = Field(default="", description="Surrounding context")++entities = ListKnowledge(+ item_schema=Entity,+ llm_client=llm,+ embedder=embedder+)+```++### Event Extraction+```python+class Event(BaseModel):+ date: str = Field(..., description="Event date")+ description: str = Field(..., description="What happened")+ participants: List[str] = Field(default_factory=list)+ location: Optional[str] = None++events = ListKnowledge(+ item_schema=Event,+ llm_client=llm,+ embedder=embedder+)+```++### Citation Extraction+```python+class Citation(BaseModel):+ authors: List[str] = Field(..., description="Author names")+ title: str = Field(..., description="Paper title")+ year: Optional[int] = None+ venue: Optional[str] = None++citations = ListKnowledge(+ item_schema=Citation,+ llm_client=llm,+ embedder=embedder+)+```++## Advanced Usage++### Custom Deduplication++Subclass to implement custom deduplication:++```python+class SmartListKnowledge(ListKnowledge[Item]):+ def _merge_strategy(self, old_data, new_data):+ # Custom deduplication logic+ existing_names = {item.name for item in old_data.items}+ unique_new = [+ item for item in new_data.items+ if item.name not in existing_names+ ]+ old_data.items.extend(unique_new)+ return old_data+```++### Filtering++```python+# Filter items after extraction+engineers = [p for p in people if p.role == "Engineer"]+mit_people = [p for p in people if "MIT" in (p.organization or "")]+```++### Sorting++```python+# Sort by field+sorted_people = sorted(people.items, key=lambda p: p.name)++# Sort by multiple fields+from operator import attrgetter+sorted_people = sorted(people.items, key=attrgetter('organization', 'name'))+```++## Best Practices++1. **Required vs Optional Fields**:+ ```python+ class Item(BaseModel):+ name: str = Field(..., description="Required field") # Must have value+ details: Optional[str] = Field(None, description="Optional field")+ ```++2. **Field Descriptions**: Help LLM understand what to extract+ ```python+ date: str = Field(..., description="Date in YYYY-MM-DD format")+ ```++3. **List Fields**: Use `default_factory=list`+ ```python+ tags: List[str] = Field(default_factory=list, description="Tags")+ ```++4. **Chunk Size**: Smaller chunks for dense information+ ```python+ # For entity-rich text+ ListKnowledge(..., chunk_size=1500, chunk_overlap=150)+ ```++## Serialization++```python+# Save+people.dump("./people_knowledge")++# Load+loaded = ListKnowledge.load(+ "./people_knowledge",+ item_schema=Person,+ llm_client=llm,+ embedder=embedder+)++assert len(loaded) == len(people)+```++## Comparison with SetKnowledge++| Feature | ListKnowledge | SetKnowledge |+|---------|---------------|--------------|+| Duplicates | Allowed (basic dedup) | Prevented (by unique_key) |+| Merge | Simple append | Configurable strategies |+| Use Case | Order matters, simple lists | Uniqueness required |++**When to use List vs Set**:+- Use **List** when: Order is important, items might legitimately appear multiple times+- Use **Set** when: Uniqueness is critical, need smart deduplication++## See Also++- [SetKnowledge API](set.md) - Unique collection with deduplication+- [BaseKnowledge API](base.md) - Base class documentation+- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selectiondiff --git a/docs/api/set.md b/docs/api/set.mdnew file mode 100644index 0000000..dfca60e--- /dev/null+++ b/docs/api/set.md@@ -0,0 +1,434 @@+# SetKnowledge API++::: hyperextract.knowledge.generic.set.SetKnowledge+ options:+ show_root_heading: true+ show_source: true+ heading_level: 2++## Overview++`SetKnowledge` implements the unique collection pattern with automatic deduplication. It extracts items and ensures uniqueness based on a user-specified key field, with configurable merge strategies for handling duplicates.++## Class Signature++```python+class SetKnowledge(BaseKnowledge[ItemSetSchema[Item]], Generic[Item]):+ """Set Knowledge Pattern - extracts a unique collection of objects."""+```++## Key Characteristics++- **Extraction Target**: Unique collection of structured objects+- **Deduplication**: Based on `unique_key` field+- **Merge Strategy**: Configurable (keep_old/keep_new/field_merge/llm_merge)+- **Internal Storage**: Dict for O(1) lookup+- **External Interface**: List-like access via `items` property++## Initialization++```python+from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy+from pydantic import BaseModel, Field++class Keyword(BaseModel):+ term: str = Field(..., description="Keyword or phrase")+ category: Optional[str] = Field(None, description="Category")+ importance: Optional[str] = Field(None, description="Importance level")++keywords = SetKnowledge(+ item_schema=Keyword,+ llm_client=llm,+ embedder=embedder,+ unique_key="term", # Deduplicate by this field+ merge_strategy=MergeItemStrategy.FIELD_MERGE+)+```++### Additional Parameters++| Parameter | Type | Default | Description |+|-----------|------|---------|-------------|+| `unique_key` | `str` | Required | Field name for deduplication |+| `merge_strategy` | `MergeItemStrategy` | `KEEP_NEW` | How to handle duplicates |++## Merge Strategies++### KEEP_OLD+Keep existing item, discard new duplicate:+```python+SetKnowledge(+ ...,+ merge_strategy=MergeItemStrategy.KEEP_OLD+)+```++**Example**:+```python+# Existing: Keyword(term="AI", category="Tech", importance="High")+# New: Keyword(term="AI", category="Science", importance="Medium")+# Result: Keyword(term="AI", category="Tech", importance="High") # Old kept+```++### KEEP_NEW (Default)+Replace existing with new:+```python+SetKnowledge(+ ...,+ merge_strategy=MergeItemStrategy.KEEP_NEW+)+```++**Example**:+```python+# Existing: Keyword(term="AI", category="Tech", importance="High")+# New: Keyword(term="AI", category="Science", importance="Medium")+# Result: Keyword(term="AI", category="Science", importance="Medium") # New kept+```++### FIELD_MERGE+Merge fields - new fills old's None/empty fields:+```python+SetKnowledge(+ ...,+ merge_strategy=MergeItemStrategy.FIELD_MERGE+)+```++**Example**:+```python+# Existing: Keyword(term="AI", category="Tech", importance=None)+# New: Keyword(term="AI", category=None, importance="High")+# Result: Keyword(term="AI", category="Tech", importance="High") # Combined+```++### LLM_MERGE+Use LLM to intelligently merge:+```python+SetKnowledge(+ ...,+ merge_strategy=MergeItemStrategy.LLM_MERGE+)+```++**Example**:+```python+# Existing: Keyword(term="ML", category="AI", importance="High")+# New: Keyword(term="ML", category="Data Science", importance="Critical")+# LLM decides: Keyword(term="ML", category="AI/Data Science", importance="Critical")+```++## Properties++### `items`+```python+@property+def items(self) -> List[Item]+```++Returns list of unique items:+```python+for keyword in keywords.items:+ print(f"{keyword.term}: {keyword.category}")+```++### `unique_keys`+```python+@property+def unique_keys(self) -> List[str]+```++Returns list of unique key values:+```python+all_terms = keywords.unique_keys+print(f"Unique keywords: {', '.join(all_terms)}")+```++## Methods++### `extract()`++Extracts items with automatic deduplication:++```python+text = """+Machine learning is a key AI technology. Deep learning, a subset of+machine learning, has revolutionized AI. Neural networks are fundamental+to deep learning. Machine learning continues to evolve.+"""++keywords.extract(text)+# "machine learning" appears twice but stored once+print(f"Extracted {len(keywords)} unique keywords")+```++### `add_item()`+```python+def add_item(self, item: Item) -> bool+```++Manually add an item, applying merge strategy if duplicate:++```python+new_keyword = Keyword(term="Python", category="Language")+added = keywords.add_item(new_keyword)+print(f"Item added: {added}")+```++**Returns**: `True` if new item added, `False` if merged with existing++### `get_item()`+```python+def get_item(self, key_value: str) -> Optional[Item]+```++Retrieve item by unique key value:++```python+ai_keyword = keywords.get_item("AI")+if ai_keyword:+ print(f"Found: {ai_keyword.category}")+```++### `remove_item()`+```python+def remove_item(self, key_value: str) -> bool+```++Remove item by unique key:++```python+removed = keywords.remove_item("obsolete_term")+print(f"Removed: {removed}")+```++### `contains()`+```python+def contains(self, key_value: str) -> bool+```++Check if key exists:++```python+if keywords.contains("Python"):+ print("Python is in the set")+```++## Set Operations++`SetKnowledge` supports mathematical set operations:++### Union (`|`)+Combine two sets:+```python+combined = keywords1 | keywords2+# All unique items from both sets+```++### Intersection (`&`)+Common items:+```python+common = keywords1 & keywords2+# Only items present in both sets+```++### Difference (`-`)+Items in first but not second:+```python+unique_to_first = keywords1 - keywords2+# Items only in keywords1+```++### Symmetric Difference (`^`)+Items in either but not both:+```python+exclusive = keywords1 ^ keywords2+# Items in one set but not the other+```++### Example:+```python+tech_keywords = SetKnowledge(item_schema=Keyword, ...)+science_keywords = SetKnowledge(item_schema=Keyword, ...)++# Extract from different sources+tech_keywords.extract(tech_article)+science_keywords.extract(science_article)++# Find common keywords+common = tech_keywords & science_keywords+print(f"Keywords in both: {[k.term for k in common.items]}")++# Find tech-specific keywords+tech_only = tech_keywords - science_keywords+print(f"Tech-specific: {[k.term for k in tech_only.items]}")+```++## Indexing and Search++```python+keywords.build_index()++results = keywords.search("artificial intelligence topics", k=5)+for doc, score in results:+ print(f"Match (score {score:.3f}): {doc.page_content}")+```++## Use Cases++### Unique Entities+```python+class Entity(BaseModel):+ name: str = Field(..., description="Entity name")+ type: str = Field(..., description="Entity type")+ description: Optional[str] = None++entities = SetKnowledge(+ item_schema=Entity,+ llm_client=llm,+ embedder=embedder,+ unique_key="name",+ merge_strategy=MergeItemStrategy.FIELD_MERGE+)+```++### Topic Modeling+```python+class Topic(BaseModel):+ topic: str = Field(..., description="Topic name")+ subtopics: List[str] = Field(default_factory=list)+ relevance: Optional[str] = None++topics = SetKnowledge(+ item_schema=Topic,+ llm_client=llm,+ embedder=embedder,+ unique_key="topic",+ merge_strategy=MergeItemStrategy.LLM_MERGE # Smart merging+)+```++### Skills Extraction+```python+class Skill(BaseModel):+ skill_name: str = Field(..., description="Skill name")+ proficiency: Optional[str] = None+ years_experience: Optional[int] = None++skills = SetKnowledge(+ item_schema=Skill,+ llm_client=llm,+ embedder=embedder,+ unique_key="skill_name",+ merge_strategy=MergeItemStrategy.KEEP_NEW # Latest info+)+```++## Advanced Usage++### Custom Merge Logic++For complex merging, use `LLM_MERGE` or subclass:++```python+class SmartSetKnowledge(SetKnowledge[Item]):+ def _merge_items_llm(self, old_item: Item, new_item: Item) -> Item:+ # Custom LLM prompt for merging+ custom_prompt = f'''+ Intelligently merge these two items:+ Old: {old_item.model_dump_json()}+ New: {new_item.model_dump_json()}++ Rules:+ - Prefer more detailed information+ - Combine complementary fields+ - Resolve conflicts logically+ '''+ # Use your LLM to merge+ ...+```++### Filtering and Transformation++```python+# Filter by field+important = [k for k in keywords if k.importance == "High"]++# Group by category+from collections import defaultdict+by_category = defaultdict(list)+for k in keywords.items:+ by_category[k.category].append(k)++# Transform+terms_only = [k.term for k in keywords.items]+```++## Best Practices++1. **Choose Unique Key Carefully**:+ ```python+ # Good: Natural unique identifier+ unique_key="email" # For people+ unique_key="isbn" # For books++ # Careful: May have false duplicates+ unique_key="name" # "John Smith" appears twice+ ```++2. **Merge Strategy Selection**:+ - `KEEP_OLD`: When first mention is most authoritative+ - `KEEP_NEW`: When latest information is better+ - `FIELD_MERGE`: When information is complementary+ - `LLM_MERGE`: When conflicts need intelligent resolution++3. **Performance Considerations**:+ ```python+ # LLM_MERGE is slower (makes LLM calls)+ SetKnowledge(..., merge_strategy=MergeItemStrategy.LLM_MERGE)++ # For large datasets, consider FIELD_MERGE or KEEP_NEW+ ```++4. **Unique Key Validation**:+ ```python+ # Ensure unique_key field exists and is required+ class Item(BaseModel):+ id: str = Field(..., description="Unique ID") # Required!+ ...++ knowledge = SetKnowledge(..., unique_key="id")+ ```++## Serialization++```python+# Save+keywords.dump("./keywords_set")++# Load (preserves uniqueness)+loaded = SetKnowledge.load(+ "./keywords_set",+ item_schema=Keyword,+ llm_client=llm,+ embedder=embedder,+ unique_key="term", # Must specify again+ merge_strategy=MergeItemStrategy.FIELD_MERGE+)+```++## Comparison++| Feature | ListKnowledge | SetKnowledge |+|---------|---------------|--------------|+| Duplicates | Allowed | Prevented by unique_key |+| Merge Strategies | Simple append | 4 configurable strategies |+| Storage | List | Dict (O(1) lookup) |+| Set Operations | No | Yes (∪, ∩, -, ⊕) |+| Use Case | Simple collections | Unique items, deduplication |++## See Also++- [ListKnowledge API](list.md) - Multi-item list extraction+- [BaseKnowledge API](base.md) - Base class documentation+- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guidediff --git a/docs/api/unit.md b/docs/api/unit.mdnew file mode 100644index 0000000..164b79e--- /dev/null+++ b/docs/api/unit.md@@ -0,0 +1,279 @@+# UnitKnowledge API++::: hyperextract.knowledge.generic.unit.UnitKnowledge+ options:+ show_root_heading: true+ show_source: true+ heading_level: 2++## Overview++`UnitKnowledge` implements the single-object extraction pattern. It's designed to extract exactly one structured object per document, making it ideal for document-level information like summaries, metadata, or aggregate statistics.++## Class Signature++```python+class UnitKnowledge(BaseKnowledge[T]):+ """Unit Knowledge Pattern - extracts a single structured object from text."""+```++## Key Characteristics++- **Extraction Target**: One unique structured object per document+- **Merge Strategy**: Field-level merge - first extraction wins, subsequent fills missing fields+- **Indexing Strategy**: Each non-null field indexed independently+- **Processing**: Uses LangChain batch processing for efficient multi-chunk handling++## Initialization++Inherits all parameters from `BaseKnowledge` with same defaults.++```python+from hyperextract.knowledge.generic import UnitKnowledge+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from pydantic import BaseModel, Field++class DocumentMetadata(BaseModel):+ title: str = Field(default="", description="Document title")+ author: str = Field(default="", description="Author name")+ summary: str = Field(default="", description="Brief summary")++llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")++knowledge = UnitKnowledge(+ data_schema=DocumentMetadata,+ llm_client=llm,+ embedder=embedder,+ chunk_size=2000,+ chunk_overlap=200+)+```++## Properties++### `data`+```python+@property+def data(self) -> T+```++Returns the single extracted object.++**Example**:+```python+knowledge.extract(text)+metadata = knowledge.data+print(f"Title: {metadata.title}")+print(f"Author: {metadata.author}")+```++## Methods++### `extract()`++Extracts a single object from text. For long texts:+1. Splits into overlapping chunks+2. Extracts object from each chunk in parallel+3. Merges using first-priority strategy++**Merge Behavior**:+- First extraction: Populates all fields+- Subsequent extractions: Only fill `None` or empty fields++```python+text = """+Long document spanning multiple chunks...+Title mentioned in chunk 1...+Author mentioned in chunk 2...+Summary in chunk 3...+"""++knowledge.extract(text)+# Result: Single object with title, author, and summary+```++### `build_index()`++Builds index from all non-null fields:++```python+knowledge.build_index()+# Creates one document per field:+# - Document(page_content=title, metadata={"field": "title"})+# - Document(page_content=author, metadata={"field": "author"})+# - Document(page_content=summary, metadata={"field": "summary"})+```++### `search()`++Searches across all indexed fields:++```python+results = knowledge.search("research methodology", k=5)+for doc, score in results:+ field_name = doc.metadata["field"]+ content = doc.page_content+ print(f"Field '{field_name}' matched with score {score:.3f}")+ print(f"Content: {content}")+```++## Merge Strategy Details++The `_merge_strategy` method implements field-level merging:++```python+def _merge_strategy(self, old_data: T, new_data: T) -> T:+ """Merge strategy: Keep old values, fill only None/empty fields from new."""+```++**Example**:+```python+# Chunk 1 extracts+old_data = DocumentMetadata(+ title="AI Research",+ author="",+ summary=""+)++# Chunk 2 extracts+new_data = DocumentMetadata(+ title="Different Title", # Ignored - old has value+ author="John Smith", # Kept - old was empty+ summary="A study..." # Kept - old was empty+)++# Merged result+merged = DocumentMetadata(+ title="AI Research", # From old+ author="John Smith", # From new+ summary="A study..." # From new+)+```++## Indexing Strategy++Each field becomes a separate document:++```python+def _build_documents_for_indexing(self) -> List[Document]:+ """Build documents from non-null fields."""+ documents = []+ for field_name, value in self._data.model_dump().items():+ if value and value != [] and value != "":+ doc = Document(+ page_content=str(value),+ metadata={"field": field_name}+ )+ documents.append(doc)+ return documents+```++## Use Cases++### Document Summary+```python+class Summary(BaseModel):+ main_topic: str = Field(default="", description="Main topic")+ key_points: List[str] = Field(default_factory=list, description="Key points")+ conclusion: str = Field(default="", description="Conclusion")++summary_knowledge = UnitKnowledge(+ data_schema=Summary,+ llm_client=llm,+ embedder=embedder+)+```++### Article Metadata+```python+class Article(BaseModel):+ title: str = Field(default="", description="Article title")+ publication_date: str = Field(default="", description="Publication date")+ author: str = Field(default="", description="Author")+ category: str = Field(default="", description="Category")++article_knowledge = UnitKnowledge(+ data_schema=Article,+ llm_client=llm,+ embedder=embedder+)+```++### Sentiment Analysis+```python+class Sentiment(BaseModel):+ overall_sentiment: str = Field(default="", description="Overall sentiment")+ positive_aspects: List[str] = Field(default_factory=list)+ negative_aspects: List[str] = Field(default_factory=list)+ confidence: float = Field(default=0.0, description="Confidence score")++sentiment_knowledge = UnitKnowledge(+ data_schema=Sentiment,+ llm_client=llm,+ embedder=embedder+)+```++## Best Practices++1. **Clear Field Descriptions**: LLM uses descriptions during extraction+ ```python+ title: str = Field(default="", description="Full article title including subtitle")+ ```++2. **Appropriate Defaults**: Use empty strings for text, empty lists for sequences+ ```python+ name: str = Field(default="") # Not None+ tags: List[str] = Field(default_factory=list) # Not None+ ```++3. **Chunk Sizing**: Adjust based on where information appears+ ```python+ # For metadata (usually at start)+ UnitKnowledge(..., chunk_size=1000, chunk_overlap=100)++ # For summary (may span document)+ UnitKnowledge(..., chunk_size=3000, chunk_overlap=300)+ ```++4. **Custom Prompts**: Provide domain-specific guidance+ ```python+ custom_prompt = '''+ Extract academic paper metadata. Pay special attention to:+ - Author affiliations+ - Publication venue+ - Citation count if mentioned+ '''++ knowledge = UnitKnowledge(+ data_schema=PaperMetadata,+ llm_client=llm,+ embedder=embedder,+ prompt=custom_prompt+ )+ ```++## Serialization++```python+# Save+knowledge.dump("./document_metadata")++# Load+loaded = UnitKnowledge.load(+ "./document_metadata",+ data_schema=DocumentMetadata,+ llm_client=llm,+ embedder=embedder+)++assert loaded.data.title == knowledge.data.title+```++## See Also++- [BaseKnowledge API](base.md) - Base class documentation+- [ListKnowledge API](list.md) - For extracting multiple items+- [SetKnowledge API](set.md) - For unique collections+- [Knowledge Patterns Guide](../user-guide/knowledge-patterns.md) - Pattern selection guidediff --git a/docs/index.md b/docs/index.mdindex 000ea34..db96777 100644--- a/docs/index.md+++ b/docs/index.md@@ -1,17 +1,170 @@-# Welcome to MkDocs+# Hyper-Extract 🧠✨-For full documentation visit [mkdocs.org](https://www.mkdocs.org).+> **An Intelligent, LLM-Powered Knowledge Extraction & Evolution Framework**+>+> Transform unstructured text into structured, evolving knowledge containers with the power of Large Language Models.-## Commands+----* `mkdocs new [dir-name]` - Create a new project.-* `mkdocs serve` - Start the live-reloading docs server.-* `mkdocs build` - Build the documentation site.-* `mkdocs -h` - Print help message and exit.+## 🎯 Vision-## Project layout+**Hyper-Extract** reimagines knowledge extraction as an intelligent, self-evolving process. Rather than treating data structures as passive containers, we empower them with **LLM-driven intelligence** to actively extract, merge, evolve, and index knowledge from unstructured text.- mkdocs.yml # The configuration file.- docs/- index.md # The documentation homepage.- ... # Other markdown pages, images and other files.+Every knowledge container in Hyper-Extract follows a unified lifecycle:++1. **Extract** - Intelligent extraction from text with automatic chunking and merging+2. **Evolve** - Self-refinement through reasoning, pruning, and optimization+3. **Index** - Semantic vector search capabilities+4. **Serialize** - Persist and reload knowledge with full fidelity++---++## ✨ Key Features++### 🤖 LLM-Native Design+- **Deep LangChain Integration**: Every container is initialized with LLM and embedder instances+- **Structured Output**: Leverage `with_structured_output` for type-safe Pydantic schema extraction+- **Intelligent Merging**: First-priority merge strategy for chunk-based extraction++### 🏗️ Unified Architecture+- **Abstract Base Class**: `BaseKnowledge[T]` provides consistent interface across all knowledge types+- **Generic Type Support**: Full Python typing with `Generic[T]` for schema flexibility+- **Lifecycle Management**: Standardized `extract` → `merge` → `evolve` → `search` → `dump/load` pipeline++### 🔍 Semantic Search+- **FAISS Integration**: Fast vector similarity search for knowledge retrieval+- **Automatic Indexing**: Build and maintain indices with lazy loading and dirty tracking+- **Field-Level Search**: Search across all schema fields with metadata preservation++### 📦 Knowledge Patterns++#### ✅ Implemented+- **UnitKnowledge**: Single-object extraction for document-level information (summaries, metadata)+- **ListKnowledge**: Collection-based extraction for entities, events, and structured lists+- **SetKnowledge**: Unique collection with automatic deduplication and intelligent merge strategies++#### 🚧 In Development+- **EntityKnowledge**: Entity-centric extraction with attribute management+- **RelationKnowledge**: Relationship extraction with entity linking+- **GraphKnowledge**: Knowledge graph construction (nodes + edges)+- **HypergraphKnowledge**: Hypergraph representation for complex multi-entity relations++---++## 🛠️ Installation++### Prerequisites+- Python 3.10++- OpenAI API key (or compatible LLM provider)++### Install with uv (Recommended)+```bash+# Clone the repository+git clone https://github.com/your-username/hyper-extract.git+cd hyper-extract++# Install dependencies with uv+uv sync+```++### Install with pip+```bash+pip install -e .+```++---++## 🚀 Quick Start++### 1. Define Your Knowledge Schema++```python+from pydantic import BaseModel, Field+from typing import List++class ArticleKnowledge(BaseModel):+ """Schema for article knowledge extraction"""+ title: str = Field(default="", description="Article title")+ author: str = Field(default="", description="Author name")+ summary: str = Field(default="", description="Article summary")+ key_points: List[str] = Field(default_factory=list, description="Key points")+ topics: List[str] = Field(default_factory=list, description="Topic tags")+```++### 2. Initialize Knowledge Container++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract.knowledge.generic import UnitKnowledge++# Initialize LLM and embedder+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")++# Create knowledge container+knowledge = UnitKnowledge(+ data_schema=ArticleKnowledge,+ llm_client=llm,+ embedder=embedder+)+```++### 3. Extract Knowledge++```python+# Extract from text+text = """+'Introduction to Deep Learning' is a comprehensive textbook+written by Professor Li Ming...+"""++knowledge.extract(text)+print(knowledge.data) # Access extracted structured knowledge+```++### 4. Semantic Search++```python+# Build index+knowledge.build_index()++# Search for relevant content+results = knowledge.search("applications of deep learning", k=3)+for doc, score in results:+ print(f"Score: {score}, Content: {doc.page_content}")+```++### 5. Persistence++```python+# Save to disk+knowledge.dump("./my_knowledge")++# Load+loaded_knowledge = UnitKnowledge.load(+ "./my_knowledge",+ data_schema=ArticleKnowledge,+ llm_client=llm,+ embedder=embedder+)+```++---++## 📚 Documentation++- **[Getting Started](user-guide/getting-started.md)** - Detailed tutorial and setup guide+- **[Knowledge Patterns](user-guide/knowledge-patterns.md)** - Understanding different extraction patterns+- **[API Reference](api/base.md)** - Complete API documentation++---++## 🤝 Contributing++Contributions are welcome! Please see our contributing guidelines for details.++---++## 📄 License++This project is licensed under the MIT License.diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.mdnew file mode 100644index 0000000..cef91c0--- /dev/null+++ b/docs/user-guide/getting-started.md@@ -0,0 +1,231 @@+# Getting Started++This guide will walk you through the basics of using Hyper-Extract to extract structured knowledge from unstructured text.++## Installation++### Prerequisites++Before you begin, ensure you have:++- **Python 3.10 or higher** installed+- An **OpenAI API key** (or access to a compatible LLM provider)+- Basic understanding of Python and Pydantic++### Setting Up Your Environment++1. **Clone the repository**:+ ```bash+ git clone https://github.com/your-username/hyper-extract.git+ cd hyper-extract+ ```++2. **Install dependencies** (recommended: use `uv`):+ ```bash+ uv sync+ ```++ Or with pip:+ ```bash+ pip install -e .+ ```++3. **Configure your API key**:+ Create a `.env` file in the project root:+ ```bash+ OPENAI_API_KEY=your-api-key-here+ ```++## Core Concepts++### Knowledge Schema++A **knowledge schema** is a Pydantic model that defines the structure of information you want to extract. It should include:++- Field names and types+- Default values+- Descriptive annotations (used by the LLM during extraction)++Example:+```python+from pydantic import BaseModel, Field+from typing import List++class BookReview(BaseModel):+ """Schema for book review extraction"""+ title: str = Field(default="", description="Book title")+ author: str = Field(default="", description="Author name")+ rating: float = Field(default=0.0, description="Rating out of 5")+ pros: List[str] = Field(default_factory=list, description="Positive aspects")+ cons: List[str] = Field(default_factory=list, description="Negative aspects")+ recommendation: str = Field(default="", description="Final recommendation")+```++### Knowledge Container++A **knowledge container** is an instance of a knowledge pattern class (e.g., `UnitKnowledge`, `ListKnowledge`, `SetKnowledge`) that manages extraction, storage, and retrieval of structured knowledge.++### Extraction Pipeline++The extraction pipeline automatically:++1. **Chunks** long texts into manageable pieces+2. **Extracts** structured data from each chunk in parallel+3. **Merges** results according to the pattern's merge strategy+4. **Indexes** the data for semantic search++## Basic Usage Example++### Step 1: Define Your Schema++```python+from pydantic import BaseModel, Field+from typing import List++class ResearchPaper(BaseModel):+ """Research paper metadata extraction"""+ title: str = Field(default="", description="Paper title")+ authors: List[str] = Field(default_factory=list, description="Author names")+ abstract: str = Field(default="", description="Paper abstract")+ keywords: List[str] = Field(default_factory=list, description="Keywords")+ methodology: str = Field(default="", description="Research methodology")+ findings: List[str] = Field(default_factory=list, description="Key findings")+ conclusion: str = Field(default="", description="Main conclusion")+```++### Step 2: Initialize LLM and Embedder++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++# Initialize models+llm = ChatOpenAI(+ model="gpt-4o-mini",+ temperature=0 # Use 0 for consistent extraction+)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")+```++### Step 3: Create Knowledge Container++```python+from hyperextract.knowledge.generic import UnitKnowledge++# Create container+paper_knowledge = UnitKnowledge(+ data_schema=ResearchPaper,+ llm_client=llm,+ embedder=embedder,+ chunk_size=2000, # Characters per chunk+ chunk_overlap=200 # Overlap between chunks+)+```++### Step 4: Extract Knowledge++```python+# Your source text+paper_text = """+Title: Deep Learning for Natural Language Processing++Authors: John Smith, Jane Doe, Bob Johnson++Abstract: This paper presents a comprehensive study on applying+deep learning techniques to natural language processing tasks...++[... rest of the paper ...]+"""++# Extract knowledge+paper_knowledge.extract(paper_text)++# Access extracted data+print(paper_knowledge.data.title)+print(paper_knowledge.data.authors)+print(paper_knowledge.data.abstract)+```++### Step 5: Semantic Search++```python+# Build search index+paper_knowledge.build_index()++# Search for specific information+results = paper_knowledge.search(+ "What methodology was used?",+ k=3 # Top 3 results+)++# Display results+for doc, score in results:+ print(f"Relevance: {score:.3f}")+ print(f"Content: {doc.page_content}")+ print(f"Metadata: {doc.metadata}")+ print("---")+```++### Step 6: Save and Load++```python+# Save knowledge to disk+paper_knowledge.dump("./paper_knowledge")++# Later, load it back+loaded_knowledge = UnitKnowledge.load(+ "./paper_knowledge",+ data_schema=ResearchPaper,+ llm_client=llm,+ embedder=embedder+)++# Knowledge is fully restored+print(loaded_knowledge.data)+```++## Working with Long Documents++Hyper-Extract automatically handles long documents by:++1. **Splitting** text into overlapping chunks+2. **Processing** chunks in parallel for efficiency+3. **Merging** extracted information intelligently++You can control chunking behavior:++```python+knowledge = UnitKnowledge(+ data_schema=YourSchema,+ llm_client=llm,+ embedder=embedder,+ chunk_size=3000, # Larger chunks for more context+ chunk_overlap=300, # More overlap to avoid missing information+ max_workers=5 # Control parallelism+)+```++## Error Handling++Always wrap extraction in try-except blocks:++```python+try:+ knowledge.extract(text)+except Exception as e:+ print(f"Extraction failed: {e}")+ # Handle error appropriately+```++## Next Steps++- Learn about different [Knowledge Patterns](knowledge-patterns.md)+- Explore the [API Reference](../api/base.md)+- Check out the [examples](https://github.com/your-username/hyper-extract/tree/main/examples) folder++## Tips for Better Extraction++1. **Clear Descriptions**: Write detailed field descriptions in your schema - the LLM uses these+2. **Appropriate Defaults**: Set sensible default values for optional fields+3. **Type Hints**: Use proper Python type hints for better extraction accuracy+4. **Chunk Size**: Adjust based on your document structure (2000-4000 chars works well)+5. **Temperature**: Use 0 for consistent extraction, higher for creative tasksdiff --git a/docs/user-guide/knowledge-patterns.md b/docs/user-guide/knowledge-patterns.mdnew file mode 100644index 0000000..d34a28a--- /dev/null+++ b/docs/user-guide/knowledge-patterns.md@@ -0,0 +1,359 @@+# Knowledge Patterns++Hyper-Extract provides three fundamental knowledge patterns, each optimized for different extraction scenarios. Understanding when to use each pattern is key to effective knowledge extraction.++---++## Overview++| Pattern | Use Case | Deduplication | Merge Strategy |+|---------|----------|---------------|----------------|+| **UnitKnowledge** | Single object per document | N/A | Field-level merge |+| **ListKnowledge** | Multiple independent items | Basic | Append |+| **SetKnowledge** | Unique collection | Automatic | Configurable |++---++## UnitKnowledge Pattern++### When to Use++Use `UnitKnowledge` when you want to extract **exactly one structured object** from a document, regardless of its length.++**Perfect for**:+- Document summaries+- Article metadata (title, author, date)+- Aggregate statistics+- Overall sentiment analysis+- Document classification++### How It Works++1. **Extraction**: Processes text in chunks, extracting a schema instance from each+2. **Merging**: Uses "first-wins" strategy - first extraction populates fields, subsequent extractions only fill missing values+3. **Result**: Single consolidated object with information from entire document++### Example: Document Metadata++```python+from pydantic import BaseModel, Field+from typing import List+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract.knowledge.generic import UnitKnowledge++# Define schema+class DocumentMetadata(BaseModel):+ title: str = Field(default="", description="Document title")+ author: str = Field(default="", description="Primary author")+ publication_date: str = Field(default="", description="Publication date")+ category: str = Field(default="", description="Document category")+ summary: str = Field(default="", description="Brief summary")+ word_count_estimate: int = Field(default=0, description="Approximate word count")++# Initialize+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")++metadata = UnitKnowledge(+ data_schema=DocumentMetadata,+ llm_client=llm,+ embedder=embedder+)++# Extract+long_article = """+[Your long article text here...]+"""+metadata.extract(long_article)++# Access single consolidated object+print(metadata.data.title)+print(metadata.data.summary)+```++### Indexing Strategy++Each non-null field is indexed separately, allowing field-specific semantic search:++```python+metadata.build_index()++# Search across all fields+results = metadata.search("machine learning applications", k=5)+```++---++## ListKnowledge Pattern++### When to Use++Use `ListKnowledge` when you need to extract **multiple independent items** of the same type from a document.++**Perfect for**:+- Entity lists (people, organizations, locations)+- Event sequences+- Product mentions+- Citation extraction+- Task lists++### How It Works++1. **Extraction**: Processes text in chunks, extracting multiple items from each+2. **Merging**: Appends all items with basic deduplication (exact matches only)+3. **Result**: List of all extracted items++### Example: Entity Extraction++```python+from pydantic import BaseModel, Field+from typing import List, Optional+from hyperextract.knowledge.generic import ListKnowledge++# Define item schema+class Person(BaseModel):+ name: str = Field(..., description="Person's full name")+ role: Optional[str] = Field(None, description="Their role or title")+ organization: Optional[str] = Field(None, description="Associated organization")+ context: Optional[str] = Field(None, description="Context of mention")++# Initialize+people = ListKnowledge(+ item_schema=Person, # Note: item_schema, not data_schema+ llm_client=llm,+ embedder=embedder+)++# Extract+text = """+Dr. Sarah Johnson, the lead researcher at MIT, collaborated with+Professor Michael Chen from Stanford University. The project manager,+Emily Rodriguez from Google, coordinated the efforts...+"""+people.extract(text)++# Access list of items+for person in people.items:+ print(f"{person.name} - {person.role} at {person.organization}")++# Or use the property+print(f"Total people extracted: {len(people.items)}")+```++### List Operations++`ListKnowledge` supports Pythonic list operations:++```python+# Append new items+people.append(Person(name="John Doe", role="Engineer"))++# Extend with multiple items+new_people = [+ Person(name="Alice Smith", role="Designer"),+ Person(name="Bob Jones", role="Manager")+]+people.extend(new_people)++# Access by index+first_person = people[0]++# Iterate+for person in people:+ print(person.name)++# Length+count = len(people)++# Check membership+if some_person in people:+ print("Found!")+```++### Indexing Strategy++Each item in the list is indexed independently:++```python+people.build_index()++# Search returns relevant items with scores+results = people.search("researchers from MIT", k=3)+for doc, score in results:+ print(f"Match: {doc.page_content} (score: {score})")+```++---++## SetKnowledge Pattern++### When to Use++Use `SetKnowledge` when you need a **unique collection** with automatic deduplication based on a key field.++**Perfect for**:+- Keywords/tags (deduplicated by term)+- Unique entities (deduplicated by ID or name)+- Topic modeling (unique topics)+- Skill extraction (unique skills)+- Location mentions (unique places)++### How It Works++1. **Extraction**: Processes text in chunks, extracting multiple items+2. **Deduplication**: Uses specified `unique_key` field to identify duplicates+3. **Merging**: Configurable strategy (keep_old, keep_new, field_merge, llm_merge)+4. **Result**: Unique collection with smart duplicate handling++### Example: Keyword Extraction with Deduplication++```python+from pydantic import BaseModel, Field+from typing import Optional+from hyperextract.knowledge.generic import SetKnowledge, MergeItemStrategy++# Define item schema with unique key+class Keyword(BaseModel):+ term: str = Field(..., description="The keyword or phrase")+ category: Optional[str] = Field(None, description="Keyword category")+ importance: Optional[str] = Field(None, description="Importance level")+ context: Optional[str] = Field(None, description="Usage context")++# Initialize with unique_key+keywords = SetKnowledge(+ item_schema=Keyword,+ llm_client=llm,+ embedder=embedder,+ unique_key="term", # Deduplicate by term+ merge_strategy=MergeItemStrategy.FIELD_MERGE # Fill missing fields+)++# Extract+text = """+Machine learning is a key technology. Deep learning, a subset of+machine learning, has revolutionized AI. Neural networks are fundamental+to deep learning...+"""+keywords.extract(text)++# Access unique items+for kw in keywords.items:+ print(f"{kw.term}: {kw.category} ({kw.importance})")+```++### Merge Strategies++#### 1. KEEP_OLD+Keeps existing item, discards new duplicate:+```python+keywords = SetKnowledge(+ item_schema=Keyword,+ llm_client=llm,+ embedder=embedder,+ unique_key="term",+ merge_strategy=MergeItemStrategy.KEEP_OLD+)+```++#### 2. KEEP_NEW (Default)+Replaces existing item with new one:+```python+merge_strategy=MergeItemStrategy.KEEP_NEW+```++#### 3. FIELD_MERGE+Merges fields - new values fill old's None/empty fields:+```python+merge_strategy=MergeItemStrategy.FIELD_MERGE+```++#### 4. LLM_MERGE+Uses LLM to intelligently merge both items:+```python+merge_strategy=MergeItemStrategy.LLM_MERGE+```++### Set Operations++`SetKnowledge` supports set operations:++```python+# Union: combine two sets+combined = keywords1 | keywords2++# Intersection: common items+common = keywords1 & keywords2++# Difference: items in first but not second+unique = keywords1 - keywords2++# Symmetric difference: items in either but not both+exclusive = keywords1 ^ keywords2+```++### Indexing Strategy++Like ListKnowledge, each unique item is indexed:++```python+keywords.build_index()+results = keywords.search("AI technologies", k=5)+```++---++## Choosing the Right Pattern++### Decision Tree++```+Is it a single object per document?+├─ Yes → Use UnitKnowledge+└─ No → Are duplicates acceptable?+ ├─ Yes → Use ListKnowledge+ └─ No → Use SetKnowledge (specify unique_key)+```++### Examples by Domain++**News Article Processing**:+- `UnitKnowledge`: Article metadata, summary+- `ListKnowledge`: All mentioned events+- `SetKnowledge`: Unique entities, locations++**Research Paper Analysis**:+- `UnitKnowledge`: Paper metadata, abstract+- `ListKnowledge`: All citations+- `SetKnowledge`: Unique keywords, methodologies++**Customer Reviews**:+- `UnitKnowledge`: Overall sentiment, rating+- `ListKnowledge`: All complaints/praises+- `SetKnowledge`: Unique features mentioned++---++## Advanced: Custom Patterns++You can create custom patterns by subclassing `BaseKnowledge`:++```python+from hyperextract.knowledge.base import BaseKnowledge++class MyCustomPattern(BaseKnowledge[MySchema]):+ def _merge_strategy(self, old_data, new_data):+ # Implement custom merge logic+ pass++ def _build_documents_for_indexing(self):+ # Implement custom indexing logic+ pass+```++---++## Next Steps++- Explore [API Reference](../api/base.md) for detailed method documentation+- Check [Getting Started](getting-started.md) for setup instructions+- Review [examples](https://github.com/your-username/hyper-extract/tree/main/examples) for complete code samplesdiff --git a/docs/zh/index.md b/docs/zh/index.mdnew file mode 100644index 0000000..cc21c48--- /dev/null+++ b/docs/zh/index.md@@ -0,0 +1,169 @@+# Hyper-Extract 🧠✨++> **一个智能的、基于 LLM 的知识提取与演化框架**+>+> 借助大语言模型的力量,将非结构化文本转化为结构化、可演化的知识容器。++---++## 🎯 愿景++**Hyper-Extract** 将知识提取重新定义为一个智能的、自我演化的过程。我们不是将数据结构视为被动容器,而是赋予它们 **LLM 驱动的智能**,使其能够主动地从非结构化文本中提取、合并、演化和索引知识。++Hyper-Extract 中的每个知识容器都遵循统一的生命周期:++1. **Extract(提取)** - 使用 LLM 从文本智能提取,自动处理分块和合并+2. **Evolve(演化)** - 通过推理、剪枝和优化实现自我优化+3. **Index(索引)** - 提供语义向量搜索能力+4. **Serialize(序列化)** - 完整保存和加载知识状态++---++## ✨ 核心特性++### 🤖 LLM 原生设计+- **深度 LangChain 集成**:每个容器都使用 LLM 和嵌入模型初始化+- **结构化输出**:利用 `with_structured_output` 实现类型安全的 Pydantic 模式提取+- **智能合并**:针对分块提取的优先级合并策略++### 🏗️ 统一架构+- **抽象基类**:`BaseKnowledge[T]` 为所有知识类型提供一致的接口+- **泛型类型支持**:使用 `Generic[T]` 实现完整的 Python 类型提示+- **生命周期管理**:标准化的 `extract` → `merge` → `evolve` → `search` → `dump/load` 流程++### 🔍 语义搜索+- **FAISS 集成**:快速的向量相似度搜索+- **自动索引**:支持懒加载和脏数据跟踪+- **字段级搜索**:可以在所有 schema 字段中搜索,并保留元数据++### 📦 知识模式++#### ✅ 已实现+- **UnitKnowledge**:单对象提取,适用于文档级信息(摘要、元数据)+- **ListKnowledge**:列表提取,适用于实体、事件和结构化列表+- **SetKnowledge**:唯一集合,支持自动去重和智能合并策略++#### 🚧 开发中+- **EntityKnowledge**:实体中心的提取,支持属性管理+- **RelationKnowledge**:关系提取,支持实体链接+- **GraphKnowledge**:知识图谱构建(节点 + 边)+- **HypergraphKnowledge**:超图表示,用于复杂的多实体关系++---++## 🛠️ 安装++### 前置要求+- Python 3.10++- OpenAI API 密钥(或兼容的 LLM 提供商)++### 使用 uv 安装(推荐)+```bash+# 克隆仓库+git clone https://github.com/your-username/hyper-extract.git+cd hyper-extract++# 使用 uv 安装依赖+uv sync+```++### 使用 pip 安装+```bash+pip install -e .+```++---++## 🚀 快速开始++### 1. 定义知识 Schema++```python+from pydantic import BaseModel, Field+from typing import List++class ArticleKnowledge(BaseModel):+ """文章知识提取的 Schema"""+ title: str = Field(default="", description="文章标题")+ author: str = Field(default="", description="作者名称")+ summary: str = Field(default="", description="文章摘要")+ key_points: List[str] = Field(default_factory=list, description="关键要点")+ topics: List[str] = Field(default_factory=list, description="主题标签")+```++### 2. 初始化知识容器++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract.knowledge.generic import UnitKnowledge++# 初始化 LLM 和嵌入模型+llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)+embedder = OpenAIEmbeddings(model="text-embedding-3-small")++# 创建知识容器+knowledge = UnitKnowledge(+ data_schema=ArticleKnowledge,+ llm_client=llm,+ embedder=embedder+)+```++### 3. 提取知识++```python+# 从文本提取+text = """+《深度学习导论》是由李明教授撰写的一本综合性教材...+"""++knowledge.extract(text)+print(knowledge.data) # 访问提取的结构化知识+```++### 4. 语义搜索++```python+# 构建索引+knowledge.build_index()++# 搜索相关内容+results = knowledge.search("深度学习的主要应用", k=3)+for doc, score in results:+ print(f"相关度: {score}, 内容: {doc.page_content}")+```++### 5. 持久化++```python+# 保存到磁盘+knowledge.dump("./my_knowledge")++# 加载+loaded_knowledge = UnitKnowledge.load(+ "./my_knowledge",+ data_schema=ArticleKnowledge,+ llm_client=llm,+ embedder=embedder+)+```++---++## 📚 文档导航++- **[快速入门指南](user-guide/getting-started.md)** - 详细的入门教程+- **[知识模式指南](user-guide/knowledge-patterns.md)** - 了解不同的知识提取模式+- **[API 参考](api/base.md)** - 完整的 API 文档++---++## 🤝 贡献++欢迎贡献!请查看我们的贡献指南了解详情。++---++## 📄 许可证++本项目基于 MIT 许可证开源。diff --git a/hyperextract/config.py b/hyperextract/config.pyindex 5ff8553..e5d6112 100644--- a/hyperextract/config.py+++ b/hyperextract/config.py@@ -1,20 +1,20 @@-# src/hyperextract/config.py+# hyperextract/config.pyimport sysimport osfrom pathlib import Pathfrom dotenv import load_dotenvfrom loguru import logger-# 1. 自动加载 .env-# 寻找项目根目录的 .env 文件+# 1. Auto load .env+# Find the .env file in the project root directoryenv_path = Path(__file__).parent.parent / ".env"load_dotenv(dotenv_path=env_path)-# 2. 配置 Loguru-# 移除默认的 handler (避免重复或格式不统一)+# 2. Configure Loguru+# Remove default handler (to avoid duplication or inconsistent formatting)logger.remove()-# 添加控制台输出 (开发模式下级别可以是 DEBUG)+# Add console output (development mode level can be DEBUG)LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")logger.add(sys.stderr,@@ -22,15 +22,15 @@ logger.add(format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>",)-# 添加文件输出 (自动轮转,保存日志文件)+# Add file output (automatic rotation, save log files)logger.add("logs/hyperextract.log",- rotation="10 MB", # 文件超过 10MB 就分割- retention="10 days", # 只保留最近 10 天- level="DEBUG", # 文件里记录更详细的信息- compression="zip", # 历史日志压缩+ rotation="10 MB", # Split file when it exceeds 10MB+ retention="10 days", # Only keep recent 10 days+ level="DEBUG", # Record more detailed information in file+ compression="zip", # Compress historical logs)-# 导出 logger 供其他模块使用-__all__ = ["logger"]+# Export logger for use in other modules+__all__ = ["logger"]\ No newline at end of filediff --git a/mkdocs.yml b/mkdocs.ymlindex cc5e49d..5f74227 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -1,18 +1,22 @@-site_name: HyperExtract-site_description: Context-aware knowledge extraction library.-site_url: https://your-github-username.github.io/hyperextract/+site_name: Hyper-Extract+site_description: An intelligent, LLM-powered knowledge extraction and evolution framework+site_url: https://your-github-username.github.io/hyper-extract/+repo_url: https://github.com/your-username/hyper-extract+repo_name: hyper-extract-# 核心主题配置+# Theme configurationtheme:name: material- palette:- # 切换日间/夜间模式的按钮+ language: en+ palette:+ # Light mode- scheme: defaultprimary: tealaccent: purpletoggle:icon: material/brightness-7name: Switch to dark mode+ # Dark mode- scheme: slateprimary: tealaccent: lime@@ -20,24 +24,80 @@ theme:icon: material/brightness-4name: Switch to light modefeatures:- - content.code.copy # 代码块复制按钮- - navigation.tabs # 顶部导航栏+ - content.code.copy # Code copy button+ - content.code.annotate # Code annotations+ - navigation.tabs # Top navigation tabs+ - navigation.tabs.sticky # Sticky tabs+ - navigation.sections # Navigation sections+ - navigation.expand # Expand navigation by default+ - navigation.top # Back to top button+ - search.suggest # Search suggestions+ - search.highlight # Highlight search results+ - toc.follow # TOC follows scroll+ icon:+ repo: fontawesome/brands/github-# 插件配置+# Plugins configurationplugins:- - search+ - search:+ lang:+ - en+ - zh- mkdocstrings:default_handler: pythonhandlers:python:- paths: [src] # 告诉插件源码在哪里+ paths: [.]+ options:+ show_source: true+ show_root_heading: true+ show_root_full_path: false+ heading_level: 2+ docstring_style: google-# 导航栏结构+# Markdown extensions+markdown_extensions:+ - pymdownx.highlight:+ anchor_linenums: true+ line_spans: __span+ pygments_lang_class: true+ - pymdownx.inlinehilite+ - pymdownx.snippets+ - pymdownx.superfences+ - pymdownx.tabbed:+ alternate_style: true+ - pymdownx.details+ - admonition+ - tables+ - attr_list+ - md_in_html+ - toc:+ permalink: true++# Extra configuration+extra:+ social:+ - icon: fontawesome/brands/github+ link: https://github.com/your-username/hyper-extract+ alternate:+ - name: English+ link: /+ lang: en+ - name: 中文+ link: /zh/+ lang: zh++# Navigation structurenav:- Home: index.md- User Guide:- Getting Started: user-guide/getting-started.md- - Configuration: user-guide/configuration.md- - API Reference: # 这里用来自动生成 API 文档- - Extractor: api/extractor.md- - Strategies: api/strategies.md+ - Knowledge Patterns: user-guide/knowledge-patterns.md+ - API Reference:+ - Overview: api/index.md+ - BaseKnowledge: api/base.md+ - UnitKnowledge: api/unit.md+ - ListKnowledge: api/list.md+ - SetKnowledge: api/set.md+ - 中文文档:+ - 首页: zh/index.md