Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
docs: 更新文档结构和内容,添加新logo资源
yifanfeng97012605802ff5
diff --git a/README.md b/README.mdindex 3af4e93..3df16c4 100644--- a/README.md+++ b/README.md@@ -1,6 +1,13 @@<div align="center">-# 🚀 Hyper-Extract+<picture>+ <source media="(prefers-color-scheme: dark)" srcset="docs/assets/logo/logo-horizontal-dark.svg">+ <source media="(prefers-color-scheme: light)" srcset="docs/assets/logo/logo-horizontal.svg">+ <img alt="Hyper-Extract Logo" src="docs/assets/logo/logo-horizontal.svg" width="600">+</picture>++<br/>+<br/>**Smart Knowledge Extraction CLI — Transform documents into structured knowledge with one command.**diff --git a/README_ZH.md b/README_ZH.mdindex 2687d14..fd3a7bb 100644--- a/README_ZH.md+++ b/README_ZH.md@@ -1,6 +1,13 @@<div align="center">-# 🚀 Hyper-Extract+<picture>+ <source media="(prefers-color-scheme: dark)" srcset="docs/assets/logo/logo-horizontal-dark.svg">+ <source media="(prefers-color-scheme: light)" srcset="docs/assets/logo/logo-horizontal.svg">+ <img alt="Hyper-Extract Logo" src="docs/assets/logo/logo-horizontal.svg" width="600">+</picture>++<br/>+<br/>**智能知识提取 CLI —— 一行命令,将文档转化为结构化知识。**diff --git a/docs/assets/logo/logo-app.svg b/docs/assets/logo/logo-app.svgnew file mode 100644index 0000000..f7eb5cc--- /dev/null+++ b/docs/assets/logo/logo-app.svg@@ -0,0 +1,19 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 500 500" width="100%" height="100%">+ <!-- Dark background rounded rectangle for app icon / avatar -->+ <rect width="500" height="500" rx="40" fill="#0D0715"/>+ <g transform="translate(0, -20)">+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#7A52BD" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#B275FF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#7A52BD" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#9E5FFF" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#B275FF" stroke-width="16" stroke-linecap="round"/>++ <!-- Nodes -->+ <circle cx="250" cy="250" r="16" fill="#FFFFFF"/>+ <circle cx="350" cy="200" r="10" fill="#FFFFFF"/>+ <circle cx="350" cy="300" r="10" fill="#FFFFFF"/>+ </g>+ <text x="250" y="440" font-family="system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif" font-size="36" font-weight="900" fill="#FFFFFF" text-anchor="middle" letter-spacing="8">+ HYPER<tspan fill="#A259FF">EXTRACT</tspan>+ </text>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-horizontal-dark.svg b/docs/assets/logo/logo-horizontal-dark.svgnew file mode 100644index 0000000..57dcd95--- /dev/null+++ b/docs/assets/logo/logo-horizontal-dark.svg@@ -0,0 +1,17 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 600 120" width="100%" height="100%">+ <!-- Horizontal layout for Dark Mode environments -->+ <g transform="translate(15, -55) scale(0.45)">+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#9A77E2" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#D1ABFF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#9A77E2" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#AE7FF2" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#D1ABFF" stroke-width="16" stroke-linecap="round"/>+ <!-- Nodes fit for dark background (White) -->+ <circle cx="250" cy="250" r="16" fill="#FFFFFF"/>+ <circle cx="350" cy="200" r="10" fill="#FFFFFF"/>+ <circle cx="350" cy="300" r="10" fill="#FFFFFF"/>+ </g>+ <text x="180" y="75" font-family="system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif" font-size="50" font-weight="900" fill="#FFFFFF" letter-spacing="4">+ HYPER<tspan fill="#A259FF">EXTRACT</tspan>+ </text>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-horizontal.svg b/docs/assets/logo/logo-horizontal.svgnew file mode 100644index 0000000..5a6a2e4--- /dev/null+++ b/docs/assets/logo/logo-horizontal.svg@@ -0,0 +1,18 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 600 120" width="100%" height="100%">+ <!-- Horizontal layout for light document backgrounds -->+ <!-- Shrink icon and center vertically to left of text -->+ <g transform="translate(15, -55) scale(0.45)">+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#7A52BD" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#B275FF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#7A52BD" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#9E5FFF" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#B275FF" stroke-width="16" stroke-linecap="round"/>+ <!-- Nodes fit for light background (Dark Purple) -->+ <circle cx="250" cy="250" r="16" fill="#1C1032"/>+ <circle cx="350" cy="200" r="10" fill="#1C1032"/>+ <circle cx="350" cy="300" r="10" fill="#1C1032"/>+ </g>+ <text x="180" y="75" font-family="system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif" font-size="50" font-weight="900" fill="#181A1F" letter-spacing="4">+ HYPER<tspan fill="#A259FF">EXTRACT</tspan>+ </text>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-icon-white.svg b/docs/assets/logo/logo-icon-white.svgnew file mode 100644index 0000000..2b4f887--- /dev/null+++ b/docs/assets/logo/logo-icon-white.svg@@ -0,0 +1,12 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="135 135 230 230" width="100%" height="100%">+ <!-- Pure Monochrome White Icon Variant -->+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#FFFFFF" stroke-opacity="0.6" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#FFFFFF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#FFFFFF" stroke-opacity="0.6" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#FFFFFF" stroke-opacity="0.8" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#FFFFFF" stroke-width="16" stroke-linecap="round"/>++ <circle cx="250" cy="250" r="16" fill="#FFFFFF"/>+ <circle cx="350" cy="200" r="10" fill="#FFFFFF"/>+ <circle cx="350" cy="300" r="10" fill="#FFFFFF"/>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-icon.svg b/docs/assets/logo/logo-icon.svgnew file mode 100644index 0000000..295b403--- /dev/null+++ b/docs/assets/logo/logo-icon.svg@@ -0,0 +1,13 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="135 135 230 230" width="100%" height="100%">+ <!-- Transparent background icon only -->+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#7A52BD" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#B275FF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#7A52BD" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#9E5FFF" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#B275FF" stroke-width="16" stroke-linecap="round"/>++ <!-- Colored nodes adapted for transparent -->+ <circle cx="250" cy="250" r="16" fill="#B275FF"/>+ <circle cx="350" cy="200" r="10" fill="#B275FF"/>+ <circle cx="350" cy="300" r="10" fill="#B275FF"/>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-vertical-dark.svg b/docs/assets/logo/logo-vertical-dark.svgnew file mode 100644index 0000000..11292e9--- /dev/null+++ b/docs/assets/logo/logo-vertical-dark.svg@@ -0,0 +1,18 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 500 500" width="100%" height="100%">+ <!-- Vertical Stack layout for Dark/Black backgrounds -->+ <g transform="translate(0, -20)">+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#9A77E2" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#D1ABFF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#9A77E2" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#AE7FF2" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#D1ABFF" stroke-width="16" stroke-linecap="round"/>++ <!-- White nodes for dark backgrounds -->+ <circle cx="250" cy="250" r="16" fill="#FFFFFF"/>+ <circle cx="350" cy="200" r="10" fill="#FFFFFF"/>+ <circle cx="350" cy="300" r="10" fill="#FFFFFF"/>+ </g>+ <text x="250" y="440" font-family="system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif" font-size="42" font-weight="900" fill="#FFFFFF" text-anchor="middle" letter-spacing="8">+ HYPER<tspan fill="#A259FF">EXTRACT</tspan>+ </text>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-vertical.svg b/docs/assets/logo/logo-vertical.svgnew file mode 100644index 0000000..637bb10--- /dev/null+++ b/docs/assets/logo/logo-vertical.svg@@ -0,0 +1,18 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 500 500" width="100%" height="100%">+ <!-- Vertical Stack layout for Light backgrounds -->+ <g transform="translate(0, -20)">+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#7A52BD" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#B275FF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#7A52BD" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#9E5FFF" stroke-width="14" stroke-linecap="round"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#B275FF" stroke-width="16" stroke-linecap="round"/>++ <!-- Dark nodes for light backgrounds -->+ <circle cx="250" cy="250" r="16" fill="#1C1032"/>+ <circle cx="350" cy="200" r="10" fill="#1C1032"/>+ <circle cx="350" cy="300" r="10" fill="#1C1032"/>+ </g>+ <text x="250" y="440" font-family="system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif" font-size="42" font-weight="900" fill="#181A1F" text-anchor="middle" letter-spacing="8">+ HYPER<tspan fill="#A259FF">EXTRACT</tspan>+ </text>+</svg>\ No newline at end of filediff --git a/docs/assets/logo/logo-white.svg b/docs/assets/logo/logo-white.svgnew file mode 100644index 0000000..2a377b9--- /dev/null+++ b/docs/assets/logo/logo-white.svg@@ -0,0 +1,18 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 500 500" width="100%" height="100%">+ <!-- Pure Monochrome White Variant -->+ <!-- Excellent for image watermarks, dark photography overlays, and solid color backgrounds -->+ <g transform="translate(0, -20)">+ <polyline points="250,150 150,200 150,300 250,350" fill="none" stroke="#FFFFFF" stroke-opacity="0.6" stroke-width="15" stroke-dasharray="16, 20" stroke-linecap="round" stroke-linejoin="round"/>+ <polyline points="250,150 350,200 350,300 250,350" fill="none" stroke="#FFFFFF" stroke-width="17" stroke-linecap="round" stroke-linejoin="round"/>+ <line x1="250" y1="250" x2="160" y2="198" stroke="#FFFFFF" stroke-opacity="0.6" stroke-width="12" stroke-linecap="round" stroke-dasharray="16, 20"/>+ <line x1="250" y1="250" x2="340" y2="198" stroke="#FFFFFF" stroke-width="14" stroke-linecap="round" stroke-opacity="0.8"/>+ <line x1="250" y1="250" x2="250" y2="345" stroke="#FFFFFF" stroke-width="16" stroke-linecap="round"/>++ <circle cx="250" cy="250" r="16" fill="#FFFFFF"/>+ <circle cx="350" cy="200" r="10" fill="#FFFFFF"/>+ <circle cx="350" cy="300" r="10" fill="#FFFFFF"/>+ </g>+ <text x="250" y="440" font-family="system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif" font-size="42" font-weight="900" fill="#FFFFFF" text-anchor="middle" letter-spacing="8">+ HYPEREXTRACT+ </text>+</svg>\ No newline at end of filediff --git a/docs/en/cli/cli.md b/docs/en/cli/cli.mddeleted file mode 100644index 5af0cc1..0000000--- a/docs/en/cli/cli.md+++ /dev/null@@ -1,162 +0,0 @@-# CLI Guide--The Hyper-Extract CLI provides a powerful command-line interface for knowledge extraction.--## Installation--The CLI is included with the main package:--```bash-uv pip install hyper-extract-```--## Basic Usage--### Initialize Configuration--```bash-he config init -k <your-api-key>-```--### Parse a Document--```bash-he parse sample.txt -o output/-```--### Search Extracted Knowledge--```bash-he search output/ "What are the key events?"-```--### Feed New Documents--```bash-he feed output/ another_sample.txt-```--## Commands--### config--Manage configuration:--```bash-# Initialize-he config init -k <your-api-key>-```-# Show current config-he config show--# Update config-he config set llm.model gpt-4o-mini--# List available templates-he config list-templates-```--### parse--Extract knowledge from documents:--```bash-# Basic usage-he parse input.txt -o output/--# With template-he parse input.txt -t finance/earnings_summary -o output/--# With language detection-he parse input.txt -l zh -o output/--# Batch processing-he parse ./documents/ -o output/ --batch-```--### search--Query extracted knowledge:--```bash-# Basic search-he search output/ "What happened?"--# With filters-he search output/ "financial events" --filter type=Event--# With limit-he search output/ "companies" --limit 10-```--### feed--Add new documents to existing knowledge:--```bash-# Feed single document-he feed output/ new_doc.txt--# Feed multiple documents-he feed output/ ./new_docs/-```--### list--List available templates and presets:--```bash-# List all templates-he list--# List by category-he list --category finance--# List domain templates-he list --domain general-```--## Options--### Global Options--| Option | Description |-|--------|-------------|-| `--help` | Show help message |-| `--version` | Show version |-| `--verbose` | Enable verbose output |-| `--config PATH` | Use custom config file |--### Parse Options--| Option | Description |-|--------|-------------|-| `-o, --output PATH` | Output directory |-| `-t, --template NAME` | Template to use |-| `-l, --language CODE` | Document language (en, zh, etc.) |-| `--batch` | Enable batch processing |--## Examples--### Financial Document Extraction--```bash-he parse annual_report.pdf -t finance/earnings_summary -o ./results/-```--### Legal Document Extraction--```bash-he parse contract.txt -t legal/case_facts -o ./legal_output/-```--### Multi-language Support--```bash-he parse chinese_doc.txt -l zh -o ./output/-```--## Next Steps--- Explore [Python API](../python/python-api.md)-- Browse [Domain Templates](../templates/index.md)diff --git a/docs/en/cli/commands/build-index.md b/docs/en/cli/commands/build-index.mdnew file mode 100644index 0000000..87b309c--- /dev/null+++ b/docs/en/cli/commands/build-index.md@@ -0,0 +1,202 @@+# he build-index++Build or rebuild the vector search index for a knowledge base.++---++## Synopsis++```bash+he build-index KA_PATH [OPTIONS]+```++## Arguments++| Argument | Description |+|----------|-------------|+| `KA_PATH` | Path to knowledge base directory |++## Options++| Option | Short | Description |+|--------|-------|-------------|+| `--force` | `-f` | Force rebuild even if index exists |++---++## Description++Builds a vector search index for semantic search and chat functionality:++1. **Reads all entities/relations** — From the knowledge base data+2. **Generates embeddings** — Using the configured embedding model+3. **Builds FAISS index** — For fast similarity search+4. **Saves to disk** — In the `index/` subdirectory++**Required for**: `he search` and `he talk` commands.++---++## Examples++### Build Index++```bash+he build-index ./output/+```++**Output:**+```+Template: general/biography_graph+Language: en++Success! Index built for ./output/++Now you can:+ he search ./output/ "keyword" # Semantic search+ he talk ./output/ -i # Interactive chat+```++### Force Rebuild++If the index is corrupted or you want to rebuild:++```bash+he build-index ./output/ -f+```++---++## When to Build Index++### After Parse (Default)++By default, `he parse` builds the index automatically:++```bash+he parse doc.md -t general/biography_graph -o ./kb/ -l en+# Index is built automatically+```++Skip with `--no-index` if you don't need search/chat:++```bash+he parse doc.md -t general/biography_graph -o ./kb/ -l en --no-index+```++### After Feed++Always rebuild after feeding new documents:++```bash+he feed ./kb/ new_doc.md+he build-index ./kb/+```++### After Manual Changes++If you modify `data.json` manually:++```bash+he build-index ./kb/ -f+```++---++## Index Storage++The index is stored in the knowledge base directory:++```+./kb/+├── data.json+├── metadata.json+└── index/ # Index directory+ ├── index.faiss # FAISS vector index+ └── docstore.json # Document store mapping+```++---++## Performance++### Build Time++| Knowledge Base Size | Approximate Build Time |+|---------------------|----------------------|+| Small (< 100 items) | < 5 seconds |+| Medium (100-1000) | 5-30 seconds |+| Large (1000+) | 30+ seconds |++### Search Speed++Once built, searches are fast:++```bash+he search ./kb/ "query" # Typically < 1 second+```++---++## Best Practices++1. **Build after feeding** — Index becomes stale after `he feed`+2. **Use `--no-index` for batch** — Build once after all parsing+3. **Force rebuild if issues** — Use `-f` if search returns unexpected results+4. **Backup large indices** — The `index/` directory can be large++---++## Batch Workflow++For processing multiple documents efficiently:++```bash+# Parse all without building index+he parse doc1.md -t general/biography_graph -o ./kb/ -l en --no-index+he feed ./kb/ doc2.md+he feed ./kb/ doc3.md++# Build index once at the end+he build-index ./kb/++# Now ready for search/chat+he search ./kb/ "query"+he talk ./kb/ -q "question"+```++---++## Troubleshooting++### "Index already exists"++Use `-f` to force rebuild:++```bash+he build-index ./kb/ -f+```++### "Failed to build index"++Check:+1. Knowledge base has data: `he info ./kb/`+2. API key is configured: `he config show`+3. Sufficient disk space for index++### Search still doesn't work++Try force rebuild:++```bash+he build-index ./kb/ -f+```++---++## See Also++- [`he parse`](parse.md) — Parse with optional index building+- [`he feed`](feed.md) — Add documents (requires rebuild)+- [`he search`](search.md) — Search the index+- [`he talk`](talk.md) — Chat using the indexdiff --git a/docs/en/cli/commands/config.md b/docs/en/cli/commands/config.mdnew file mode 100644index 0000000..23c402d--- /dev/null+++ b/docs/en/cli/commands/config.md@@ -0,0 +1,233 @@+# he config++Manage Hyper-Extract configuration.++---++## Synopsis++```bash+he config [COMMAND] [OPTIONS]+```++## Commands++| Command | Description |+|---------|-------------|+| `init` | Initialize configuration |+| `show` | Display current configuration |+| `set` | Set a configuration value |+| `get` | Get a configuration value |++---++## he config init++Initialize configuration with API credentials.++```bash+he config init [OPTIONS]+```++### Options++| Option | Short | Description |+|--------|-------|-------------|+| `--key` | `-k` | OpenAI API key |+| `--base-url` | `-u` | API base URL (optional) |+| `--model` | `-m` | Default model (default: gpt-4o-mini) |+| `--embedder` | `-e` | Embedding model (default: text-embedding-3-small) |++### Examples++#### Basic Setup++```bash+he config init -k sk-your-api-key-here+```++#### With Custom Base URL++```bash+he config init -k sk-your-key -u https://api.openai.com/v1+```++#### With Custom Models++```bash+he config init -k sk-your-key -m gpt-4o -e text-embedding-3-large+```++#### Interactive Mode++```bash+he config init+# Prompts for API key interactively+```++---++## he config show++Display current configuration.++```bash+he config show+```++**Output:**+```+Configuration (from ~/.he/config.toml):++[llm]+model = "gpt-4o-mini"+api_key = "sk-...***"+base_url = "https://api.openai.com/v1"++[embedder]+model = "text-embedding-3-small"+api_key = "sk-...***"++[defaults]+language = "en"+chunk_size = 2048+max_workers = 10+```++---++## he config set++Set a configuration value.++```bash+he config set <KEY> <VALUE>+```++### Examples++```bash+# Change default model+he config set llm.model gpt-4o++# Change embedding model+he config set embedder.model text-embedding-3-large++# Set default language+he config set defaults.language zh+```++---++## he config get++Get a configuration value.++```bash+he config get <KEY>+```++### Examples++```bash+he config get llm.model+# Output: gpt-4o-mini++he config get defaults.language+# Output: en+```++---++## Configuration File++Configuration is stored at:++- **Linux/macOS**: `~/.he/config.toml`+- **Windows**: `%USERPROFILE%\.he\config.toml`++### Example Configuration++```toml+[llm]+model = "gpt-4o-mini"+api_key = "sk-your-api-key"+base_url = "https://api.openai.com/v1"++[embedder]+model = "text-embedding-3-small"+api_key = "sk-your-api-key"++[defaults]+language = "en"+chunk_size = 2048+chunk_overlap = 256+max_workers = 10+verbose = false+```++---++## Environment Variables++Configuration can also be set via environment variables (higher priority):++| Variable | Description |+|----------|-------------|+| `OPENAI_API_KEY` | API key for LLM and embedder |+| `OPENAI_BASE_URL` | Custom API base URL |+| `HE_LLM_MODEL` | Default LLM model |+| `HE_EMBEDDER_MODEL` | Default embedding model |++---++## Priority Order++Configuration is resolved in this order (highest first):++1. Command-line flags+2. Environment variables+3. Configuration file+4. Built-in defaults++---++## Troubleshooting++### "API key not found"++Initialize configuration:++```bash+he config init -k your-api-key+```++Or set environment variable:++```bash+export OPENAI_API_KEY=your-api-key+```++### "Invalid API key"++Check your API key:++```bash+he config show+# Verify the key is correct+```++### "Configuration file not found"++Run init to create it:++```bash+he config init -k your-api-key+```++---++## See Also++- [Configuration Reference](../configuration.md) — Detailed configuration options+- [Installation Guide](../../getting-started/installation.md) — Initial setupdiff --git a/docs/en/cli/commands/feed.md b/docs/en/cli/commands/feed.mdnew file mode 100644index 0000000..497eba4--- /dev/null+++ b/docs/en/cli/commands/feed.md@@ -0,0 +1,180 @@+# he feed++Add documents to an existing knowledge base incrementally.++---++## Synopsis++```bash+he feed KA_PATH INPUT [OPTIONS]+```++## Arguments++| Argument | Description |+|----------|-------------|+| `KA_PATH` | Path to existing knowledge base directory |+| `INPUT` | Input file path or `-` for stdin |++## Options++| Option | Short | Description |+|--------|-------|-------------|+| `--template` | `-t` | Override template (uses metadata if omitted) |+| `--lang` | `-l` | Override language (uses metadata if omitted) |++---++## Description++The `feed` command adds new documents to an existing knowledge base without losing existing data:++1. **Loads existing knowledge** — Reads current knowledge base state+2. **Extracts from new document** — Processes the new content+3. **Merges intelligently** — Combines new and existing data, handling duplicates+4. **Updates metadata** — Records the update timestamp++This is ideal for:+- Building knowledge bases over time+- Adding updates to existing documents+- Combining information from multiple sources++---++## Examples++### Basic Usage++```bash+# Initial extraction+he parse tesla_bio.md -t general/biography_graph -o ./tesla_kb/ -l en++# Add more content+he feed ./tesla_kb/ tesla_inventions.md+```++### Feed Multiple Documents++```bash+he feed ./kb/ doc1.md+he feed ./kb/ doc2.md+he feed ./kb/ doc3.md+```++Or use a loop:++```bash+for file in updates/*.md; do+ he feed ./kb/ "$file"+done+```++### From Stdin++```bash+cat new_content.md | he feed ./kb/ -+```++---++## Merge Behavior++The merge process handles:++| Scenario | Behavior |+|----------|----------|+| Same entity | Merged, descriptions combined |+| Same relation | Updated with latest information |+| New entities | Added to knowledge base |+| New relations | Added connecting existing/new entities |++---++## Workflow Example++### Building a Research Knowledge Base++```bash+# Day 1: Initial paper+he parse paper_v1.md -t general/concept_graph -o ./research_kb/ -l en+he show ./research_kb/++# Day 7: Updated version+he feed ./research_kb/ paper_v2.md+he show ./research_kb/++# Day 14: Related work+he feed ./research_kb/ related_work.md+he build-index ./research_kb/+he talk ./research_kb/ -q "What are the key concepts across all papers?"+```++### Incremental Biography++```bash+# Start with early life+he parse early_life.md -t general/biography_graph -o ./bio_kb/ -l en++# Add career period+he feed ./bio_kb/ career.md++# Add later years+he feed ./bio_kb/ later_years.md++# Final visualization+he show ./bio_kb/+```++---++## Verification++Check that the feed worked:++```bash+he info ./kb/+```++Look for:+- Increased node count+- Increased edge count+- Updated timestamp++---++## Best Practices++1. **Use same template** — Feeding should use compatible templates+2. **Match language** — Use consistent language for best results+3. **Rebuild index after** — `he build-index ./kb/` for search/chat+4. **Visualize changes** — `he show ./kb/` to see updates++---++## Error Handling++### "Not a valid Knowledge Abstract directory"++The directory doesn't contain a valid knowledge base. Check:++```bash+ls ./kb/+# Should contain: data.json, metadata.json+```++### "Template mismatch"++Feeding works best with the same template type. Override if needed:++```bash+he feed ./kb/ doc.md -t general/biography_graph+```++---++## See Also++- [`he parse`](parse.md) — Create new knowledge base+- [`he info`](info.md) — View knowledge base statistics+- [`he build-index`](build-index.md) — Rebuild search index after feedingdiff --git a/docs/en/cli/commands/info.md b/docs/en/cli/commands/info.mdnew file mode 100644index 0000000..06e10b7--- /dev/null+++ b/docs/en/cli/commands/info.md@@ -0,0 +1,172 @@+# he info++Display information and statistics about a knowledge base.++---++## Synopsis++```bash+he info KA_PATH+```++## Arguments++| Argument | Description |+|----------|-------------|+| `KA_PATH` | Path to knowledge base directory |++---++## Description++Shows metadata and statistics for a knowledge base:++- **Path** — Location of the knowledge base+- **Template** — Template used for extraction+- **Language** — Language code (en/zh)+- **Timestamps** — Creation and last update times+- **Statistics** — Node count, edge count+- **Index status** — Whether search index is built++---++## Examples++### Basic Usage++```bash+he info ./output/+```++**Output:**+```+Knowledge Abstract Info++Path ./output/+Template general/biography_graph+Language en+Created 2024-01-15 10:30:00+Updated 2024-01-15 10:35:22+Nodes 25+Edges 32+Index Built+```++### After Extraction++```bash+he parse tesla.md -t general/biography_graph -o ./kb/ -l en+he info ./kb/+```++### After Feed++```bash+he feed ./kb/ more_content.md+he info ./kb/+# Note the updated node/edge counts and timestamp+```++---++## Output Fields++| Field | Description |+|-------|-------------|+| `Path` | Absolute path to knowledge base |+| `Template` | Template identifier (e.g., `general/biography_graph`) |+| `Language` | Processing language (`en` or `zh`) |+| `Created` | Initial extraction timestamp |+| `Updated` | Last modification timestamp |+| `Nodes` | Number of entities/items |+| `Edges` | Number of relationships |+| `Index` | Search index status (`Built` or `Not Built`) |++---++## Use Cases++### Verify Extraction++Check that extraction worked:++```bash+he info ./kb/+# Should show Nodes > 0, Edges > 0+```++### Check Index Status++Before using search or chat:++```bash+he info ./kb/+# If Index shows "Not Built", run:+he build-index ./kb/+```++### Monitor Growth++Track knowledge base growth over time:++```bash+he info ./kb/+# Feed more documents+he feed ./kb/ update.md+he info ./kb/+# Compare node/edge counts+```++### Scripting++Use in automation scripts:++```bash+#!/bin/bash++if he info ./kb/ 2>/dev/null; then+ echo "Knowledge base exists"+ he feed ./kb/ new_doc.md+else+ echo "Creating new knowledge base"+ he parse new_doc.md -t general/biography_graph -o ./kb/ -l en+fi+```++---++## Troubleshooting++### "Knowledge base directory not found"++The directory doesn't exist or doesn't contain a knowledge base:++```bash+# Check directory exists+ls ./kb/++# Should contain:+# - data.json+# - metadata.json+```++### Missing metadata++If metadata is corrupted, template/language show as "unknown":++```bash+he info ./kb/+# Template: [yellow]unknown[/yellow]+# Language: [yellow]unknown[/yellow]+```++The knowledge base is still usable but some features may be limited.++---++## See Also++- [`he parse`](parse.md) — Create knowledge base+- [`he feed`](feed.md) — Add to knowledge base+- [`he build-index`](build-index.md) — Build search indexdiff --git a/docs/en/cli/commands/list.md b/docs/en/cli/commands/list.mdnew file mode 100644index 0000000..03265c2--- /dev/null+++ b/docs/en/cli/commands/list.md@@ -0,0 +1,149 @@+# he list++List available templates and methods.++---++## Synopsis++```bash+he list [COMMAND]+```++## Commands++| Command | Description |+|---------|-------------|+| `template` | List knowledge templates |+| `method` | List extraction methods |++---++## he list template++Display all available knowledge templates.++```bash+he list template+```++**Output:**+```+Available Templates:++Path Type Description+general/base_model model Basic model extraction+general/base_list list Basic list extraction+general/base_graph graph Basic graph extraction+general/biography_graph temporal_graph Biography graph with timeline+finance/earnings_summary model Financial earnings summary+finance/ownership_graph graph Company ownership structure+legal/contract_obligation list Contract obligations+medicine/drug_interaction graph Drug interaction network+...+```++### Filter by Type++Templates are organized by output type:++| Type | Templates |+|------|-----------|+| `model` | Structured data extraction |+| `list` | Ordered collections |+| `set` | Deduplicated collections |+| `graph` | Entity-relationship networks |+| `hypergraph` | Multi-entity relationships |+| `temporal_graph` | Time-based relationships |+| `spatial_graph` | Location-based relationships |+| `spatio_temporal_graph` | Combined time + space |++---++## he list method++Display all available extraction methods.++```bash+he list method+```++**Output:**+```+Available Methods:++Name Type Description+graph_rag graph Graph-RAG with Community detection+light_rag graph Lightweight Graph-based RAG+hyper_rag hypergraph Hypergraph-based RAG+hypergraph_rag hypergraph Advanced Hypergraph RAG+cog_rag hypergraph Cognitive RAG+itext2kg graph iText2KG: Triple-based extraction+itext2kg_star graph Enhanced iText2KG+kg_gen graph Knowledge Graph Generator+atom graph Temporal knowledge graph+```++### Method Categories++| Category | Methods | Best For |+|----------|---------|----------|+| **RAG-based** | `graph_rag`, `light_rag`, `hyper_rag` | Large documents, retrieval-enhanced |+| **Typical** | `itext2kg`, `kg_gen`, `atom` | Direct extraction, specific features |++---++## Use Cases++### Find a Template++```bash+# See all options+he list template++# Then use one+he parse doc.md -t general/biography_graph -o ./out/ -l en+```++### Choose a Method++```bash+# See available methods+he list method++# Use one instead of template+he parse doc.md -m light_rag -o ./out/+```++### Scripting++```bash+# Count available templates+he list template | wc -l++# Find templates by domain+he list template | grep finance++# Check if specific method exists+he list method | grep -q light_rag && echo "Available"+```++---++## Template vs Method++| Feature | Templates | Methods |+|---------|-----------|---------|+| **Purpose** | Domain-specific extraction | General algorithms |+| **Configuration** | Pre-configured prompts | Algorithm parameters |+| **Language** | Multi-language support | English only |+| **Use case** | Quick start, domain tasks | Research, customization |+| **Example** | `finance/earnings_summary` | `light_rag` |++---++## See Also++- [`he parse`](parse.md) — Use templates/methods+- [Template Library](../../templates/index.md) — Detailed template documentation+- [Choosing Methods](../../concepts/methods.md) — Method selection guidediff --git a/docs/en/cli/commands/parse.md b/docs/en/cli/commands/parse.mdnew file mode 100644index 0000000..9ce1759--- /dev/null+++ b/docs/en/cli/commands/parse.md@@ -0,0 +1,211 @@+# he parse++Extract knowledge from documents and save to a knowledge base.++---++## Synopsis++```bash+he parse INPUT [OPTIONS]+```++## Arguments++| Argument | Description |+|----------|-------------|+| `INPUT` | Input file path, directory, or `-` for stdin |++## Options++| Option | Short | Description |+|--------|-------|-------------|+| `--output` | `-o` | Output directory (required) |+| `--template` | `-t` | Template to use (omit for interactive selection) |+| `--method` | `-m` | Method template (e.g., `light_rag`, `graph_rag`) |+| `--lang` | `-l` | Language: `zh` or `en` (required for knowledge templates) |+| `--force` | `-f` | Force overwrite existing output |+| `--no-index` | — | Skip building search index |++---++## Examples++### Basic Usage++Extract from a single file:++```bash+he parse document.md -t general/biography_graph -o ./output/ -l en+```++### Interactive Template Selection++Omit `-t` to select from available templates:++```bash+he parse document.md -o ./output/ -l en++# You'll see:+# Select a template:+# [1] general/biography_graph+# [2] general/knowledge_graph+# [3] finance/earnings_summary+# ...+# Enter number or search keyword:+```++### Process a Directory++Extract from all `.md` and `.txt` files in a directory:++```bash+he parse ./documents/ -t general/concept_graph -o ./output/ -l en+```++Files are combined in alphabetical order before extraction.++### Using Methods Instead of Templates++Use underlying extraction methods:++```bash+he parse document.md -m light_rag -o ./output/+```++Methods always use English prompts.++### Force Overwrite++Overwrite existing output directory:++```bash+he parse document.md -t general/biography_graph -o ./output/ -l en -f+```++### Skip Index Building++Speed up extraction if you don't need search/chat:++```bash+he parse document.md -t general/biography_graph -o ./output/ -l en --no-index+```++Build index later with `he build-index`.++### Read from Stdin++```bash+cat document.md | he parse - -t general/biography_graph -o ./output/ -l en+```++---++## Output Structure++```+./output/+├── data.json # Extracted knowledge (entities, relations, etc.)+├── metadata.json # Extraction metadata+│ ├── template # Template used+│ ├── lang # Language+│ ├── created_at # Creation timestamp+│ └── updated_at # Last update timestamp+└── index/ # Vector search index (if built)+ ├── index.faiss+ └── docstore.json+```++---++## Language Support++Templates support multiple languages:++```bash+# English+he parse doc.md -t general/biography_graph -l en -o ./output/++# Chinese+he parse doc.md -t general/biography_graph -l zh -o ./output/+```++Choose the language that matches your document for best results.++---++## Common Use Cases++### Research Paper++```bash+he parse paper.md -t general/concept_graph -o ./paper_kb/ -l en+```++### Biography++```bash+he parse biography.md -t general/biography_graph -o ./bio_kb/ -l en+```++### Legal Contract++```bash+he parse contract.md -t legal/contract_obligation -o ./contract_kb/ -l en+```++### Financial Report++```bash+he parse earnings.md -t finance/earnings_summary -o ./finance_kb/ -l en+```++---++## Error Handling++### "Output directory already exists"++The output directory exists and is not empty. Solutions:++1. Use `-f` to force overwrite+2. Choose a different output path+3. Remove the existing directory first++### "Template not found"++The specified template doesn't exist. Solutions:++1. List available templates: `he list template`+2. Use interactive selection (omit `-t`)+3. Check template path spelling++### "Language is required"++Knowledge templates require a language flag. Methods don't:++```bash+# Template - requires -l+he parse doc.md -t general/biography_graph -o ./out/ -l en++# Method - no -l needed+he parse doc.md -m light_rag -o ./out/+```++---++## Best Practices++1. **Choose the right template** — Match your document type+2. **Use correct language** — Improves extraction quality+3. **Organize outputs** — Use descriptive directory names+4. **Skip index during batch** — Use `--no-index`, build once at the end++---++## See Also++- [`he feed`](feed.md) — Add documents incrementally+- [`he build-index`](build-index.md) — Build search index+- [`he list`](list.md) — List available templates+- [Template Library](../../templates/index.md)diff --git a/docs/en/cli/commands/search.md b/docs/en/cli/commands/search.mdnew file mode 100644index 0000000..0b7da7d--- /dev/null+++ b/docs/en/cli/commands/search.md@@ -0,0 +1,152 @@+# he search++Perform semantic search over a knowledge base.++---++## Synopsis++```bash+he search KA_PATH QUERY [OPTIONS]+```++## Arguments++| Argument | Description |+|----------|-------------|+| `KA_PATH` | Path to knowledge base directory |+| `QUERY` | Search query string |++## Options++| Option | Short | Description | Default |+|--------|-------|-------------|---------|+| `--top-k` | `-n` | Number of results to return | 3 |++---++## Description++Semantic search finds relevant information even when keywords don't match exactly. It uses:++1. **Vector embeddings** — Converts query and content to vectors+2. **Similarity matching** — Finds semantically similar content+3. **Ranking** — Returns most relevant results++**Requires**: Search index must be built. Run `he build-index` if needed.++---++## Examples++### Basic Search++```bash+he search ./output/ "Tesla's inventions"+```++### Get More Results++```bash+he search ./output/ "electrical engineering" -n 10+```++### Natural Language Queries++```bash+he search ./kb/ "What were the major achievements?"+he search ./kb/ "People who worked with Edison"+he search ./kb/ "Important dates in the timeline"+```++### After Building Index++```bash+# First, ensure index exists+he build-index ./output/++# Then search+he search ./output/ "innovation"+```++---++## Output Format++```+Found 3 result(s):++Result 1:+{+ "name": "Nikola Tesla",+ "type": "person",+ "description": "Serbian-American inventor, electrical engineer..."+}++Result 2:+{+ "source": "Nikola Tesla",+ "target": "Thomas Edison",+ "type": "worked_with",+ "description": "Tesla worked for Edison in 1884"+}++Result 3:+...+```++---++## How It Works++1. **Query Embedding** — Converts your query to a vector+2. **Index Search** — Finds nearest vectors in the knowledge base+3. **Result Ranking** — Returns top-k most similar items++---++## Tips for Better Search++1. **Use natural language** — "inventions in electrical engineering" vs "invention electrical"+2. **Be specific** — "Tesla's work on AC power" vs "Tesla work"+3. **Try synonyms** — If "inventions" doesn't work, try "discoveries"+4. **Increase top-k** — Use `-n 10` for broader results++---++## Comparison with `he talk`++| Feature | `he search` | `he talk` |+|---------|-------------|-----------|+| Returns | Raw entities/relations | Natural language answer |+| Use case | Find specific data | Get explanations |+| Speed | Faster | Slower (LLM generation) |+| Precision | Exact matches | Interpretive |++---++## Troubleshooting++### "Index not found"++Build the search index:++```bash+he build-index ./output/+```++### "No results found"++Try:+1. Broader query terms+2. Increase `-n` for more results+3. Different phrasing+4. Check `he info ./output/` to verify data exists++---++## See Also++- [`he talk`](talk.md) — Chat with knowledge base+- [`he build-index`](build-index.md) — Build search index+- [`he parse`](parse.md) — Extract with index buildingdiff --git a/docs/en/cli/commands/show.md b/docs/en/cli/commands/show.mdnew file mode 100644index 0000000..903fffe--- /dev/null+++ b/docs/en/cli/commands/show.md@@ -0,0 +1,140 @@+# he show++Visualize knowledge graph using OntoSight interactive viewer.++---++## Synopsis++```bash+he show KA_PATH+```++## Arguments++| Argument | Description |+|----------|-------------|+| `KA_PATH` | Path to knowledge base directory |++---++## Description++Opens an interactive visualization of your knowledge graph in the default web browser. The visualization displays:++- **Nodes** — Entities, concepts, or items (colored by type)+- **Edges** — Relationships between nodes+- **Labels** — Click on nodes/edges to see details++---++## Examples++### Basic Usage++```bash+he show ./output/+```++### After Extraction++```bash+# Extract first+he parse tesla.md -t general/biography_graph -o ./tesla_kb/ -l en++# Then visualize+he show ./tesla_kb/+```++### After Incremental Update++```bash+# Add more content+he feed ./tesla_kb/ additional_info.md++# Visualize updated graph+he show ./tesla_kb/+```++---++## Visualization Features++The OntoSight viewer provides:++### Interactive Controls++- **Zoom** — Mouse wheel or pinch gesture+- **Pan** — Click and drag background+- **Select** — Click on nodes/edges to see details+- **Filter** — Show/hide node types++### Node Display++- Size indicates importance (configurable)+- Color indicates entity type+- Label shows entity name++### Edge Display++- Thickness indicates relationship strength+- Label shows relationship type+- Direction shows relationship flow++---++## Supported Auto-Types++Visualization works with all graph-based Auto-Types:++| Auto-Type | Visualization |+|-----------|--------------|+| `AutoGraph` | ✓ Full graph visualization |+| `AutoHypergraph` | ✓ Hypergraph with multi-node edges |+| `AutoTemporalGraph` | ✓ Graph with temporal info |+| `AutoSpatialGraph` | ✓ Graph with spatial info |+| `AutoSpatioTemporalGraph` | ✓ Full context visualization |+| `AutoList` | ✓ List view |+| `AutoSet` | ✓ Set view |+| `AutoModel` | ✓ Structured view |++---++## Troubleshooting++### "Browser doesn't open"++The visualization URL is printed. Open it manually:++```+http://localhost:xxxx+```++### "Empty graph displayed"++Check that extraction worked:++```bash+he info ./kb/+# Should show Nodes > 0 and Edges > 0+```++### "Error loading visualization"++Ensure the knowledge base is valid:++```bash+# Check data file exists+ls ./kb/data.json++# Try reloading+he show ./kb/+```++---++## See Also++- [`he parse`](parse.md) — Extract knowledge+- [`he feed`](feed.md) — Add documents incrementally+- [`he info`](info.md) — View knowledge base statisticsdiff --git a/docs/en/cli/commands/talk.md b/docs/en/cli/commands/talk.mdnew file mode 100644index 0000000..5137c84--- /dev/null+++ b/docs/en/cli/commands/talk.md@@ -0,0 +1,202 @@+# he talk++Chat with your knowledge base using natural language.++---++## Synopsis++```bash+he talk KA_PATH [OPTIONS]+```++## Arguments++| Argument | Description |+|----------|-------------|+| `KA_PATH` | Path to knowledge base directory |++## Options++| Option | Short | Description |+|--------|-------|-------------|+| `--query` | `-q` | Single question to ask |+| `--top-k` | `-n` | Number of context items to retrieve | 3 |+| `--interactive` | `-i` | Enter interactive chat mode |++---++## Description++The `talk` command lets you have a conversation with your knowledge base:++1. **Retrieves relevant context** — Searches for information related to your question+2. **Generates answer** — Uses LLM to synthesize a natural language response+3. **Shows sources** — Displays which items were used to generate the answer++**Requires**: Search index must be built.++---++## Examples++### Single Question++```bash+he talk ./output/ -q "What were Tesla's major achievements?"+```++**Output:**+```+Query: What were Tesla's major achievements?+Knowledge Abstract: ./output/+Top K: 3++Nikola Tesla made numerous groundbreaking contributions to electrical+engineering. His major achievements include:++1. Development of the alternating current (AC) electrical system, which+ became the dominant method of power transmission worldwide.++2. Invention of the Tesla coil, used in radio technology and electrical+ resonant circuits.++3. Contributions to the development of X-ray imaging and wireless+ communication technologies.++Retrieved context:+1. Nikola Tesla: Serbian-American inventor, electrical engineer...+2. AC Power System: Type: invention, Description: System for alternating...+3. Tesla Coil: Type: invention, Description: Resonant transformer circuit...+```++### Interactive Mode++```bash+he talk ./output/ -i+```++**Session:**+```+Entering interactive mode. Type 'exit' or 'quit' to stop.++Knowledge Abstract: ./output/+Template: general/biography_graph+Top K: 3++> Who was Nikola Tesla?+Nikola Tesla was a Serbian-American inventor, electrical engineer,+mechanical engineer, and futurist best known for his contributions+to the design of the modern alternating current (AC) electricity+supply system.++> When was he born?+Nikola Tesla was born on July 10, 1856, in Smiljan, Croatia.++> What about his relationship with Edison?+Tesla worked for Thomas Edison in New York City in 1884. They had+a contentious relationship due to differing views on direct current+(DC) versus alternating current (AC) power systems.++> exit+Goodbye!++Other useful commands:+ he show ./output/ # Visualize+ he search ./output/ "keyword" # Search+ he info ./output/ # View info+```++### With More Context++Increase context for complex questions:++```bash+he talk ./output/ -q "Explain the War of Currents" -n 10+```++---++## Interactive Commands++In interactive mode (`-i`):++| Command | Action |+|---------|--------|+| `exit`, `quit`, `q` | Exit interactive mode |+| `help` | Show available commands |++---++## Use Cases++### Research Assistant++```bash+he talk ./paper_kb/ -q "Summarize the main contributions of this paper"+```++### Legal Analysis++```bash+he talk ./contract_kb/ -q "What are the termination conditions?"+```++### Historical Research++```bash+he talk ./bio_kb/ -q "What events led to Tesla's financial difficulties?"+```++---++## How It Works++1. **Semantic Search** — Finds relevant items in the knowledge base+2. **Context Assembly** — Combines retrieved items into context+3. **LLM Generation** — Generates answer using the context+4. **Source Attribution** — Shows which items informed the answer++---++## Tips for Better Answers++1. **Be specific** — "What inventions did Tesla create?" vs "Tell me about Tesla"+2. **Ask follow-ups** — Build context through conversation+3. **Adjust top-k** — Use `-n 5` or higher for complex questions+4. **Check sources** — Review "Retrieved context" for accuracy++---++## Comparison with `he search`++| Feature | `he search` | `he talk` |+|---------|-------------|-----------|+| Output | Raw entities/relations | Natural language |+| Best for | Finding specific data | Understanding/explaining |+| Speed | Fast | Slower (LLM call) |+| Source visibility | Direct | Cited in context |++---++## Troubleshooting++### "Index not found"++```bash+he build-index ./output/+```++### "No relevant information"++- Try rephrasing your question+- Increase context with `-n 10`+- Verify knowledge base has relevant data: `he info ./output/`++---++## See Also++- [`he search`](search.md) — Search for specific items+- [`he build-index`](build-index.md) — Build search index+- [`he show`](show.md) — Visualize knowledge graphdiff --git a/docs/en/cli/config.md b/docs/en/cli/config.mddeleted file mode 100644index 092162e..0000000--- a/docs/en/cli/config.md+++ /dev/null@@ -1,83 +0,0 @@-# CLI Configuration--## Configuration File--The CLI configuration is stored in `~/.he/config.toml`.--## Initialize Configuration--```bash-# Initialize with API Key (recommended for first-time users)-he config init -k YOUR_OPENAI_API_KEY--# Interactive setup (will prompt for LLM and Embedder settings)-he config init-```--## View Configuration--```bash-# Show current configuration-he config show-```--## Configure LLM--```bash-# Set LLM API key and model-he config llm --api-key YOUR_KEY --model gpt-4o-mini--# Show current LLM configuration-he config llm --show--# Set custom base URL-he config llm --api-key YOUR_KEY --base-url https://api.openai.com/v1--# Clear LLM configuration-he config llm --unset-```--## Configure Embedder--```bash-# Set Embedder API key and model-he config embedder --api-key YOUR_KEY --model text-embedding-3-small--# Show current Embedder configuration-he config embedder --show--# Set custom base URL-he config embedder --api-key YOUR_KEY --base-url https://api.openai.com/v1--# Clear Embedder configuration-he config embedder --unset-```--## Configuration Options--### LLM Options--| Option | Flag | Description | Default |-|--------|------|-------------|---------|-| API Key | `--api-key`, `-k` | LLM API key | Required |-| Model | `--model`, `-m` | LLM model name | `gpt-4o-mini` |-| Base URL | `--base-url`, `-u` | Custom API base URL | OpenAI default |--### Embedder Options--| Option | Flag | Description | Default |-|--------|------|-------------|---------|-| API Key | `--api-key`, `-k` | Embedder API key | Required |-| Model | `--model`, `-m` | Embedder model name | `text-embedding-3-small` |-| Base URL | `--base-url`, `-u` | Custom API base URL | OpenAI default |--## Environment Variables--You can also use environment variables as an alternative:--```bash-export OPENAI_API_KEY=your-api-key-export OPENAI_BASE_URL=https://api.openai.com/v1-```--Environment variables are used when not set in the configuration file.diff --git a/docs/en/cli/configuration.md b/docs/en/cli/configuration.mdnew file mode 100644index 0000000..b3d1043--- /dev/null+++ b/docs/en/cli/configuration.md@@ -0,0 +1,224 @@+# CLI Configuration Reference++Complete reference for Hyper-Extract CLI configuration.++---++## Configuration File Location++- **Linux/macOS**: `~/.he/config.toml`+- **Windows**: `%USERPROFILE%\.he\config.toml`++---++## Configuration Sections++### [llm] — Language Model Settings++| Key | Description | Default |+|-----|-------------|---------|+| `model` | LLM model name | `gpt-4o-mini` |+| `api_key` | OpenAI API key | *required* |+| `base_url` | API base URL | `https://api.openai.com/v1` |++**Example:**+```toml+[llm]+model = "gpt-4o"+api_key = "sk-your-api-key"+base_url = "https://api.openai.com/v1"+```++**Supported Models:**+- `gpt-4o-mini` — Fast, cost-effective+- `gpt-4o` — High quality+- `gpt-4-turbo` — Best quality+- Other OpenAI-compatible models++---++### [embedder] — Embedding Model Settings++| Key | Description | Default |+|-----|-------------|---------|+| `model` | Embedding model name | `text-embedding-3-small` |+| `api_key` | API key (defaults to llm.api_key) | — |++**Example:**+```toml+[embedder]+model = "text-embedding-3-large"+```++**Supported Models:**+- `text-embedding-3-small` — Fast, cost-effective+- `text-embedding-3-large` — Better quality+- `text-embedding-ada-002` — Legacy++---++### [defaults] — Default Behavior Settings++| Key | Description | Default |+|-----|-------------|---------|+| `language` | Default language (`en` or `zh`) | `en` |+| `chunk_size` | Text chunk size for processing | `2048` |+| `chunk_overlap` | Overlap between chunks | `256` |+| `max_workers` | Parallel processing workers | `10` |+| `verbose` | Enable verbose output | `false` |++**Example:**+```toml+[defaults]+language = "en"+chunk_size = 4096+chunk_overlap = 512+max_workers = 5+verbose = true+```++---++## Environment Variables++All configuration options can be set via environment variables:++| Variable | Maps To | Example |+|----------|---------|---------|+| `OPENAI_API_KEY` | `llm.api_key`, `embedder.api_key` | `sk-...` |+| `OPENAI_BASE_URL` | `llm.base_url` | `https://api.openai.com/v1` |+| `HE_LLM_MODEL` | `llm.model` | `gpt-4o` |+| `HE_EMBEDDER_MODEL` | `embedder.model` | `text-embedding-3-large` |+| `HE_DEFAULT_LANGUAGE` | `defaults.language` | `zh` |+| `HE_CHUNK_SIZE` | `defaults.chunk_size` | `4096` |+| `HE_VERBOSE` | `defaults.verbose` | `true` |++---++## Complete Example++```toml+# ~/.he/config.toml++[llm]+model = "gpt-4o-mini"+api_key = "sk-your-api-key-here"+base_url = "https://api.openai.com/v1"++[embedder]+model = "text-embedding-3-small"+# api_key defaults to llm.api_key++[defaults]+language = "en"+chunk_size = 2048+chunk_overlap = 256+max_workers = 10+verbose = false+```++---++## Configuration Priority++Settings are resolved in this order (highest priority first):++1. **Command-line flags** (e.g., `-l zh`)+2. **Environment variables** (e.g., `OPENAI_API_KEY`)+3. **Configuration file** (`~/.he/config.toml`)+4. **Built-in defaults**++---++## Managing Configuration++### Initialize++```bash+he config init -k your-api-key+```++### View Current Settings++```bash+he config show+```++### Update Settings++```bash+# Update single value+he config set llm.model gpt-4o++# Or edit file directly+nano ~/.he/config.toml+```++### Reset to Defaults++```bash+rm ~/.he/config.toml+he config init -k your-api-key+```++---++## Advanced Configuration++### Custom API Endpoint++For OpenAI-compatible APIs (Azure, local models, etc.):++```toml+[llm]+model = "your-model-name"+api_key = "your-key"+base_url = "https://your-api-endpoint.com/v1"++[embedder]+model = "your-embedding-model"+```++### Different Keys for LLM and Embedder++```toml+[llm]+model = "gpt-4o-mini"+api_key = "sk-llm-key"++[embedder]+model = "text-embedding-3-small"+api_key = "sk-embedder-key"+```++---++## Troubleshooting++### Permission Denied++```bash+# Fix permissions+chmod 600 ~/.he/config.toml+```++### Invalid TOML Format++Validate your configuration:++```bash+# Using Python+python -c "import tomllib; tomllib.load(open('~/.he/config.toml', 'rb'))"+```++Common issues:+- Missing quotes around strings+- Trailing commas+- Incorrect section headers++---++## See Also++- [`he config`](commands/config.md) — Configuration commands+- [Installation Guide](../getting-started/installation.md) — Initial setupdiff --git a/docs/en/cli/index.md b/docs/en/cli/index.mdindex 0adf21e..3ce68c2 100644--- a/docs/en/cli/index.md+++ b/docs/en/cli/index.md@@ -1,50 +1,201 @@-# CLI+# CLI Guide-Command-line interface for Hyper-Extract.+The Hyper-Extract CLI (`he`) provides a powerful, easy-to-use interface for knowledge extraction directly from your terminal.++---## Installation```bash-uv pip install hyper-extract+pip install hyper-extract+```++Verify installation:++```bash+he --version+```++---++## Quick Command Reference++| Command | Purpose | Common Flags |+|---------|---------|--------------|+| `he parse` | Extract knowledge from documents | `-t` template, `-o` output, `-l` language |+| `he show` | Visualize knowledge graph | — |+| `he search` | Semantic search in knowledge base | `-n` top-k results |+| `he talk` | Chat with knowledge base | `-i` interactive, `-q` query |+| `he feed` | Add documents incrementally | — |+| `he info` | Show knowledge base statistics | — |+| `he build-index` | Build/rebuild search index | `-f` force rebuild |+| `he list` | List templates and methods | `template` or `method` |+| `he config` | Manage configuration | `init`, `show`, `set` |++---++## Complete Workflow++The typical workflow for extracting and interacting with knowledge:++```mermaid+graph LR+ A[Document] -->|he parse| B[Knowledge Base]+ B -->|he show| C[Visualize]+ B -->|he search| D[Search]+ B -->|he talk| E[Chat]+ B -->|he feed| B[Update]```-## Quick Start+1. **Parse** — Extract knowledge from documents+2. **Show** — Visualize the knowledge graph+3. **Search** — Find specific information+4. **Talk** — Have a conversation about the content+5. **Feed** — Add more documents incrementally++→ [Detailed Workflow Walkthrough](workflow.md)++---++## Getting Started++### 1. Configure API Key```bash-# Configure API Keyhe config init -k YOUR_OPENAI_API_KEY+```-# Extract knowledge-he parse document.txt -t general/biography_graph -o ./output/+### 2. Extract Knowledge-# Query the knowledge base-he search ./output/ "What are the key events?"+```bash+he parse document.md -t general/biography_graph -o ./output/ -l en+```++### 3. Visualize-# Visualize the knowledge graph+```bashhe show ./output/```-## Core Commands+---++## Commands in Detail++### Knowledge Extraction++- **[`he parse`](commands/parse.md)** — Extract knowledge from documents+- **[`he feed`](commands/feed.md)** — Add documents to existing knowledge base++### Exploration++- **[`he show`](commands/show.md)** — Visualize knowledge graph+- **[`he search`](commands/search.md)** — Semantic search+- **[`he talk`](commands/talk.md)** — Chat with knowledge base+- **[`he info`](commands/info.md)** — View knowledge base statistics-| Command | Purpose |-|---------|---------|-| `he config init` | Configure API Key |-| `he parse` | Extract knowledge |-| `he search` | Query knowledge base |-| `he show` | Visualize graph |-| `he feed` | Incrementally supplement |-| `he list` | List templates |+### Management++- **[`he build-index`](commands/build-index.md)** — Build search index+- **[`he list`](commands/list.md)** — List available templates/methods+- **[`he config`](commands/config.md)** — Configuration management++---## Configuration-[→ CLI Configuration Guide](./config.md)+The CLI stores configuration in `~/.he/config.toml`.++→ [Configuration Reference](configuration.md)++---++## Template vs Method++Hyper-Extract offers two ways to extract knowledge:++### Templates (Recommended for Most Users)++Domain-specific, ready-to-use configurations:++```bash+he parse doc.md -t general/biography_graph -l en+```++### Methods (Advanced)++Underlying extraction algorithms:++```bash+he parse doc.md -m light_rag+```++→ [Learn when to use each](../concepts/architecture.md)++---++## Language Support++Templates support multiple languages:++```bash+# English+he parse doc.md -t general/biography_graph -l en++# Chinese+he parse doc.md -t general/biography_graph -l zh+```++Method templates always use English prompts.++---++## Examples by Use Case++### Research++```bash+# Extract from a research paper+he parse paper.md -t general/concept_graph -o ./paper_kb/ -l en++# Ask questions about it+he talk ./paper_kb/ -q "What are the main contributions?"+```++### Biography Analysis++```bash+# Extract from a biography+he parse biography.md -t general/biography_graph -o ./bio_kb/ -l en++# Visualize life events+he show ./bio_kb/+```++### Legal Document Analysis++```bash+# Extract contract obligations+he parse contract.md -t legal/contract_obligation -o ./contract_kb/ -l en++# Search for specific clauses+he search ./contract_kb/ "termination conditions"+```++---++## Tips and Best Practices-## Detailed Guide+1. **Use templates for domain-specific tasks** — They're optimized for specific use cases+2. **Build the index** — Required for search and chat functionality+3. **Feed incrementally** — Add documents over time without reprocessing+4. **Choose the right language** — Improves extraction quality for non-English documents-[→ CLI Command Details](./cli.md)+----## Usage Examples+## Getting Help-- [Finance Domain Templates](../templates/finance.md)-- [Legal Domain Templates](../templates/legal.md)-- [Medical Domain Templates](../templates/medicine.md)+- View help for any command: `he <command> --help`+- List all templates: `he list template`+- List all methods: `he list method`+- [FAQ](../resources/faq.md)+- [Troubleshooting](../resources/troubleshooting.md)diff --git a/docs/en/cli/workflow.md b/docs/en/cli/workflow.mdnew file mode 100644index 0000000..e722404--- /dev/null+++ b/docs/en/cli/workflow.md@@ -0,0 +1,352 @@+# CLI Workflow Guide++This guide walks you through a complete knowledge extraction workflow using real-world examples.++---++## Scenario: Research Paper Analysis++Imagine you're a researcher who wants to extract and interact with knowledge from academic papers. Here's how to do it with Hyper-Extract.++---++## Step 1: Prepare Your Document++First, ensure your document is in a supported format (`.md`, `.txt`):++```bash+# Example: Convert PDF to text (using external tools)+pdftotext paper.pdf paper.md+```++Or create a sample document:++```bash+cat > transformer_paper.md << 'EOF'+# Attention Is All You Need++## Abstract+The dominant sequence transduction models are based on complex recurrent or+convolutional neural networks that include an encoder and a decoder. The best+performing models also connect the encoder and decoder through an attention+mechanism. We propose a new simple network architecture, the Transformer,+based solely on attention mechanisms.++## Authors+- Ashish Vaswani+- Noam Shazeer+- Niki Parmar+- Jakob Uszkoreit+- Llion Jones+- Aidan N. Gomez+- Lukasz Kaiser+- Illia Polosukhin++## Key Innovation+The Transformer eschews recurrence and instead relies entirely on an attention+mechanism to draw global dependencies between input and output sequences.++## Performance+On the WMT 2014 English-to-German translation task, our model achieves a+BLEU score of 28.4, outperforming all existing models.+EOF+```++---++## Step 2: Extract Knowledge++Use the `parse` command to extract knowledge:++```bash+he parse transformer_paper.md -t general/concept_graph -o ./transformer_kb/ -l en+```++**What happens:**+1. The document is read and chunked (if long)+2. LLM extracts entities and relationships+3. Results are saved to `./transformer_kb/`++**Output structure:**+```+./transformer_kb/+├── data.json # Extracted knowledge+├── metadata.json # Extraction metadata+└── index/ # Vector search index (if built)+```++---++## Step 3: Explore the Knowledge++### View Statistics++```bash+he info ./transformer_kb/+```++**Output:**+```+Knowledge Abstract Info++Path ./transformer_kb/+Template general/concept_graph+Language en+Created 2024-01-15 10:30:00+Updated 2024-01-15 10:30:00+Nodes 12+Edges 15+Index Built+```++### Visualize++```bash+he show ./transformer_kb/+```++This opens an interactive graph in your browser showing:+- **Nodes**: Authors, concepts, models, metrics+- **Edges**: Relationships between them++---++## Step 4: Search for Information++### Semantic Search++Find information even if keywords don't match exactly:++```bash+he search ./transformer_kb/ "neural network architecture"+```++**Results:**+```+Found 3 result(s):++Result 1:+{+ "name": "Transformer",+ "type": "model",+ "description": "A neural network architecture based solely on attention mechanisms"+}++Result 2:+{+ "name": "Attention Mechanism",+ "type": "concept",+ "description": "Mechanism to draw global dependencies between sequences"+}+...+```++### Specific Queries++```bash+he search ./transformer_kb/ "performance metrics" -n 5+```++---++## Step 5: Chat with Your Knowledge++### Single Question++```bash+he talk ./transformer_kb/ -q "What is the main contribution of this paper?"+```++**Response:**+```+The main contribution is the introduction of the Transformer architecture,+which achieves state-of-the-art results on machine translation tasks while+being more parallelizable and requiring significantly less training time than+recurrent or convolutional architectures.+```++### Interactive Mode++```bash+he talk ./transformer_kb/ -i+```++**Session:**+```+Entering interactive mode. Type 'exit' or 'quit' to stop.++> Who are the authors?+The paper was authored by Ashish Vaswani, Noam Shazeer, Niki Parmar,+Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and+Illia Polosukhin.++> What BLEU score did they achieve?+The model achieved a BLEU score of 28.4 on the WMT 2014 English-to-German+translation task.++> exit+Goodbye!+```++---++## Step 6: Expand Your Knowledge Base++### Add Another Document++```bash+# Download another paper+curl -o bert_paper.md https://example.com/bert.md++# Add to existing knowledge base+he feed ./transformer_kb/ bert_paper.md+```++### Verify the Update++```bash+he info ./transformer_kb/+```++Notice the increased node/edge count and updated timestamp.++---++## Step 7: Rebuild Index (If Needed)++After adding documents, rebuild the search index:++```bash+he build-index ./transformer_kb/+```++Or force a complete rebuild:++```bash+he build-index ./transformer_kb/ -f+```++---++## Complete Script Example++Here's a bash script that automates the entire workflow:++```bash+#!/bin/bash++# Configuration+INPUT_FILE="paper.md"+OUTPUT_DIR="./paper_kb/"+TEMPLATE="general/concept_graph"+LANGUAGE="en"++echo "=== Hyper-Extract Workflow ==="+echo++# Step 1: Parse+echo "Step 1: Extracting knowledge..."+he parse "$INPUT_FILE" -t "$TEMPLATE" -o "$OUTPUT_DIR" -l "$LANGUAGE"+echo++# Step 2: Info+echo "Step 2: Knowledge base info:"+he info "$OUTPUT_DIR"+echo++# Step 3: Search example+echo "Step 3: Sample search:"+he search "$OUTPUT_DIR" "main contributions" -n 2+echo++# Step 4: Open visualization+echo "Step 4: Opening visualization..."+he show "$OUTPUT_DIR"++echo "=== Workflow Complete ==="+```++---++## Advanced: Batch Processing++Process multiple documents at once:++```bash+# Create output directories+mkdir -p ./kb/paper1 ./kb/paper2 ./kb/paper3++# Process each+he parse papers/paper1.md -t general/concept_graph -o ./kb/paper1/ -l en+he parse papers/paper2.md -t general/concept_graph -o ./kb/paper2/ -l en+he parse papers/paper3.md -t general/concept_graph -o ./kb/paper3/ -l en++# Or use a loop+for file in papers/*.md; do+ name=$(basename "$file" .md)+ he parse "$file" -t general/concept_graph -o "./kb/$name/" -l en+done+```++---++## Common Patterns++### Pattern 1: Continuous Knowledge Building++```bash+# Initial extraction+he parse initial_doc.md -t general/biography_graph -o ./kb/ -l en++# Weekly updates+he feed ./kb/ week1_update.md+he feed ./kb/ week2_update.md+he feed ./kb/ week3_update.md++# Monthly rebuild+he build-index ./kb/ -f+```++### Pattern 2: Multi-Domain Project++```bash+# Technical documentation+he parse api_docs.md -t general/concept_graph -o ./project_kb/tech/ -l en++# Legal contracts+he parse contract.pdf -t legal/contract_obligation -o ./project_kb/legal/ -l en++# Financial reports+he parse q4_report.md -t finance/earnings_summary -o ./project_kb/finance/ -l en+```++### Pattern 3: Compare Documents++```bash+# Extract two versions+he parse draft_v1.md -t general/concept_graph -o ./kb/v1/ -l en+he parse draft_v2.md -t general/concept_graph -o ./kb/v2/ -l en++# Compare via chat+he talk ./kb/v1/ -q "What are the main topics?"+he talk ./kb/v2/ -q "What are the main topics?"+```++---++## Troubleshooting Tips++| Issue | Solution |+|-------|----------|+| Extraction is slow | Long documents are chunked; use `--no-index` to skip indexing during parse |+| Search returns nothing | Ensure index is built: `he build-index ./kb/` |+| Template not found | List available: `he list template` |+| Out of memory | Reduce chunk size in config or process smaller documents |++---++## Next Steps++- Learn about [all CLI commands](commands/parse.md)+- Explore the [Template Library](../templates/index.md)+- Read about [choosing the right Auto-Type](../concepts/autotypes.md)diff --git a/docs/en/concepts/architecture.md b/docs/en/concepts/architecture.mdnew file mode 100644index 0000000..de4d32e--- /dev/null+++ b/docs/en/concepts/architecture.md@@ -0,0 +1,231 @@+# Architecture++Deep dive into Hyper-Extract's system design and data flow.++---++## System Overview++```mermaid+graph LR+ A[Input Text] --> B[Text Processor]+ B --> C[LLM Extractor]+ C --> D[Merger]+ D --> E[Auto-Type Instance]+ E --> F[Operations]++ subgraph "Text Processor"+ B1[Chunking] --> B2[Parallel Processing]+ end++ subgraph "LLM Extractor"+ C1[Prompt] --> C2[LLM Call]+ C2 --> C3[Structured Output]+ end++ subgraph "Operations"+ F1[Search]+ F2[Chat]+ F3[Visualize]+ F4[Save/Load]+ end+```++---++## Data Flow++### 1. Input Processing++```python+# Input: Raw text+text = "Nikola Tesla was an inventor..."++# If text > chunk_size, split into chunks+chunks = [+ "Nikola Tesla was an inventor...",+ "He developed the AC system...",+ # ...+]+```++**Chunking Strategy:**+- Default size: 2048 characters+- Overlap: 256 characters+- Separators: Paragraphs, sentences, words++### 2. Extraction++```python+# Each chunk processed in parallel+for chunk in chunks:+ # LLM extracts structured data+ data = llm.extract(chunk, schema=AutoTypeSchema)+```++**Process:**+1. Format prompt with chunk+2. Call LLM with structured output+3. Parse response into Pydantic model++### 3. Merging++```python+# Merge results from all chunks+final_data = merge(chunk_results)+```++**Merge Operations:**+- Deduplicate entities+- Combine relations+- Resolve conflicts++### 4. Result++```python+# Auto-Type instance with extracted data+result = AutoTypeInstance(data=final_data)+```++---++## Component Details++### Template Engine++```+Template YAML → Parser → Configuration → Factory → Auto-Type Instance+```++Components:+- **Gallery** — Template discovery and listing+- **Parser** — YAML parsing and validation+- **Factory** — Instance creation++### Auto-Type Base Class++```python+class BaseAutoType:+ # Core functionality+ def parse(text) -> AutoType+ def feed_text(text) -> AutoType++ # Query+ def build_index()+ def search(query) -> List[Item]+ def chat(query) -> Response++ # Persistence+ def dump(path)+ def load(path)++ # Visualization+ def show()+```++### Method Registry++```python+# Methods are registered at import+_METHOD_REGISTRY = {+ "light_rag": {+ "class": Light_RAG,+ "type": "graph",+ "description": "..."+ },+ # ...+}+```++---++## Extension Points++### Custom Templates++Create domain-specific templates:++```yaml+# my_template.yaml+name: custom_extraction+type: graph+output:+ entities:+ fields:+ - name: name+ type: str+# ...+```++### Custom Auto-Types++Extend base classes:++```python+from hyperextract.types import AutoGraph++class MyCustomGraph(AutoGraph):+ def _default_prompt(self):+ return "Custom prompt..."+```++### Custom Methods++Implement extraction algorithms:++```python+from hyperextract.methods import register_method++class MyMethod:+ def extract(self, text):+ # Custom extraction logic+ pass++register_method("my_method", MyMethod, "graph", "Description")+```++---++## Performance Considerations++### Chunking++| Document Size | Chunks | Processing Time |+|---------------|--------|-----------------|+| < 2KB | 1 | Fast |+| 2-10KB | 2-5 | Medium |+| 10-50KB | 5-25 | Slow |+| > 50KB | 25+ | Very Slow |++### Parallelization++```python+# Default: 10 concurrent workers+# Adjustable via config+max_workers = 10+```++### Memory Usage++```python+# Index memory estimate+# ~1KB per entity/relation+```++---++## Design Principles++1. **Type Safety** — Pydantic schemas throughout+2. **Extensibility** — Plugin architecture for methods+3. **Usability** — Templates for common tasks+4. **Performance** — Chunking and parallelization+5. **Interoperability** — Standard formats (JSON, YAML)++---++## See Also++- [Auto-Types](autotypes.md)+- [Methods](methods.md)+- [Template Format](templates-format.md)diff --git a/docs/en/concepts/auto-types.md b/docs/en/concepts/auto-types.mddeleted file mode 100644index a5882d7..0000000--- a/docs/en/concepts/auto-types.md+++ /dev/null@@ -1,91 +0,0 @@-# AutoTypes--AutoTypes define the **output data structure** for knowledge extraction. There are 8 types:--## Record Types--### AutoModel--Structured data extraction with Pydantic models.--```python-from pydantic import BaseModel, Field-from hyperextract import AutoModel--class Summary(BaseModel):- title: str = Field(description="Title")- summary: str = Field(description="Summary")--model = AutoModel(data_schema=Summary, llm_client=llm)-```--### AutoList--Extract lists of items.--```python-from hyperextract.types import AutoList--lst = AutoList[str](item_schema=str, llm_client=llm)-lst.feed_text(text)-```--### AutoSet--Extract unique items (deduplicated).--```python-from hyperextract.types import AutoSet--s = AutoSet[str](item_schema=str, llm_client=llm)-s.feed_text(text)-```--## Graph Types--### AutoGraph--Entity-relationship extraction.--```python-from hyperextract.types import AutoGraph--graph = AutoGraph(node_schema=Entity, edge_schema=Relation, llm_client=llm)-graph.feed_text(text)-```--### AutoHypergraph--Multi-entity relationships (n-ary relations).--```python-from hyperextract.types import AutoHypergraph-```--### AutoTemporalGraph--Time-based relationships.--```python-from hyperextract.types import AutoTemporalGraph-```--### AutoSpatialGraph--Location-based relationships.--```python-from hyperextract.types import AutoSpatialGraph-```--### AutoSpatioTemporalGraph--Both time and location.--```python-from hyperextract.types import AutoSpatioTemporalGraph-```--## Quick Links--- [Domain Templates](../templates/index.md) - Ready-to-use templatesdiff --git a/docs/en/concepts/autotypes.md b/docs/en/concepts/autotypes.mdnew file mode 100644index 0000000..21d6129--- /dev/null+++ b/docs/en/concepts/autotypes.md@@ -0,0 +1,362 @@+# Auto-Types++Complete guide to the 8 knowledge structure types.++---++## Overview++Auto-Types are intelligent data structures that:+- Define extraction output format+- Provide type-safe schemas+- Include built-in operations (search, visualize, etc.)+- Support serialization++---++## The 8 Auto-Types++```mermaid+graph TD+ A[Auto-Types] --> B[Scalar Types]+ A --> C[Graph Types]+ A --> D[Temporal Types]++ B --> B1[AutoModel]+ B --> B2[AutoList]+ B --> B3[AutoSet]++ C --> C1[AutoGraph]+ C --> C2[AutoHypergraph]++ D --> D1[AutoTemporalGraph]+ D --> D2[AutoSpatialGraph]+ D --> D3[AutoSpatioTemporalGraph]+```++---++## Scalar Types++### AutoModel++**Purpose**: Single structured object++**Use When**:+- Extracting a summary/report+- Structured data with known fields+- Single entity with many attributes++**Example Output**:+```python+{+ "company_name": "Tesla Inc",+ "revenue": 81.46,+ "eps": 4.07,+ "employees": 127855+}+```++**Common Templates**:+- `finance/earnings_summary`+- `general/base_model`++---++### AutoList++**Purpose**: Ordered collection++**Use When**:+- Sequence matters+- Ranked items+- Timeline events (simple)++**Example Output**:+```python+{+ "items": [+ {"name": "AC Motor", "year": 1888},+ {"name": "Tesla Coil", "year": 1891},+ {"name": "Radio", "year": 1898}+ ]+}+```++**Common Templates**:+- `general/base_list`+- `legal/compliance_list`++---++### AutoSet++**Purpose**: Deduplicated collection++**Use When**:+- Unique items only+- Tags/categories+- Membership testing++**Example Output**:+```python+{+ "items": [+ "Electrical Engineering",+ "Physics",+ "Invention",+ "Renewable Energy"+ ]+}+```++**Common Templates**:+- `general/base_set`+- `finance/risk_factor_set`++---++## Graph Types++### AutoGraph++**Purpose**: Entity-relationship network++**Use When**:+- People, organizations, concepts+- Binary relationships+- Knowledge graphs++**Example Output**:+```python+{+ "entities": [+ {"name": "Tesla", "type": "person"},+ {"name": "Edison", "type": "person"},+ {"name": "AC Motor", "type": "invention"}+ ],+ "relations": [+ {"source": "Tesla", "target": "AC Motor", "type": "invented"},+ {"source": "Tesla", "target": "Edison", "type": "rivals"}+ ]+}+```++**Common Templates**:+- `general/knowledge_graph`+- `general/biography_graph`++---++### AutoHypergraph++**Purpose**: Multi-entity relationships++**Use When**:+- Relationships involve 3+ entities+- Complex interactions+- N-ary associations++**Example Output**:+```python+{+ "entities": [...],+ "hyperedges": [+ {+ "entities": ["Tesla", "Westinghouse", "Niagara"],+ "type": "collaboration",+ "description": "Power plant project"+ }+ ]+}+```++**Common Templates**:+- `general/base_hypergraph`++---++## Temporal Types++### AutoTemporalGraph++**Purpose**: Graph + time information++**Use When**:+- Timeline is important+- Event sequences+- Historical analysis++**Example Output**:+```python+{+ "entities": [...],+ "relations": [+ {+ "source": "Tesla",+ "target": "AC Motor",+ "type": "invented",+ "time": "1888"+ },+ {+ "source": "Tesla",+ "target": "Wardenclyffe Tower",+ "type": "built",+ "time": "1901-1902"+ }+ ]+}+```++**Common Templates**:+- `general/base_temporal_graph`+- `finance/event_timeline`++---++### AutoSpatialGraph++**Purpose**: Graph + location information++**Use When**:+- Geographic data+- Location-based analysis+- Mapping++**Example Output**:+```python+{+ "entities": [+ {+ "name": "Colorado Springs",+ "type": "location",+ "coordinates": "38.8339,-104.8214"+ }+ ],+ "relations": [+ {+ "source": "Tesla",+ "target": "Colorado Springs",+ "type": "conducted_experiments",+ "location": "Colorado Springs"+ }+ ]+}+```++**Common Templates**:+- `general/base_spatial_graph`++---++### AutoSpatioTemporalGraph++**Purpose**: Graph + time + space++**Use When**:+- Full context needed+- Historical geography+- Event analysis with when and where++**Example Output**:+```python+{+ "entities": [...],+ "relations": [+ {+ "source": "Tesla",+ "target": "AC Motor",+ "type": "demonstrated",+ "time": "1888",+ "location": "Pittsburgh",+ "description": "Demonstration at Westinghouse"+ }+ ]+}+```++**Common Templates**:+- `general/base_spatio_temporal_graph`++---++## Selection Guide++### Decision Tree++```+What do you need to extract?+│+├─ Single structured object → AutoModel+│+├─ Collection of items+│ ├─ Ordered/Ranked → AutoList+│ └─ Unique/Tags → AutoSet+│+└─ Relationships+ ├─ Simple (binary)+ │ ├─ With time → AutoTemporalGraph+ │ ├─ With space → AutoSpatialGraph+ │ ├─ With both → AutoSpatioTemporalGraph+ │ └─ Neither → AutoGraph+ │+ └─ Complex (multi-entity)+ └─ AutoHypergraph+```++### By Use Case++| Use Case | Recommended Type |+|----------|------------------|+| Company report | AutoModel |+| Top 10 list | AutoList |+| Tags/keywords | AutoSet |+| People network | AutoGraph |+| Project teams | AutoHypergraph |+| Biography timeline | AutoTemporalGraph |+| Travel log | AutoSpatialGraph |+| Historical events | AutoSpatioTemporalGraph |++---++## Common Operations++All Auto-Types support:++```python+# Extraction+result = ka.parse(text)++# Incremental update+result.feed_text(more_text)++# Search (requires index)+result.build_index()+results = result.search("query")++# Chat (requires index)+response = result.chat("question")++# Visualization+result.show()++# Persistence+result.dump("./path/")+result.load("./path/")++# Check empty+if result.empty():+ print("No data")++# Clear+result.clear()+result.clear_index()+```++---++## See Also++- [Working with Auto-Types](../python/guides/working-with-autotypes.md)+- [Template Library](../templates/index.md)+- [Methods](methods.md)diff --git a/docs/en/concepts/index.md b/docs/en/concepts/index.mdindex d11c261..c34f8d4 100644--- a/docs/en/concepts/index.md+++ b/docs/en/concepts/index.md@@ -1,18 +1,86 @@# Concepts-Hyper-Extract is built on a **three-layer architecture**:+Understand the fundamental concepts behind Hyper-Extract.-- **Auto-Types** define the data structures for knowledge extraction-- **Methods** provide extraction algorithms built on Auto-Types-- **Templates** offer domain-specific configurations with ready-to-use prompts+----## Auto-Types+## Three-Layer Architecture-Auto-Types are the foundation - they define the output data structure for knowledge extraction.+Hyper-Extract is built on a three-layer architecture:-[→ Learn more about Auto-Types](./auto-types.md)+```mermaid+graph TD+ subgraph "Layer 3: Interface"+ A[CLI] --> B[Python SDK]+ B --> C[Template API]+ end++ subgraph "Layer 2: Methods"+ C --> D[RAG-Based]+ C --> E[Typical]+ end++ subgraph "Layer 1: Data"+ D --> F[Auto-Types]+ E --> F+ end++ F --> G[Structured Knowledge]+```++| Layer | Purpose | Components |+|-------|---------|------------|+| **Auto-Types** | Define data structures | 8 type classes |+| **Methods** | Extraction algorithms | RAG + Typical methods |+| **Templates** | Domain-specific configs | 80+ preset templates |++---++## Core Concepts++### [Auto-Types](autotypes.md)++The 8 data structure types that define extraction output:++- **Scalar Types**: AutoModel, AutoList, AutoSet+- **Graph Types**: AutoGraph, AutoHypergraph+- **Temporal Types**: AutoTemporalGraph, AutoSpatialGraph, AutoSpatioTemporalGraph++→ [Learn about Auto-Types](autotypes.md)++### [Methods](methods.md)++The extraction algorithms:++- **RAG-Based**: GraphRAG, LightRAG, Hyper-RAG+- **Typical**: iText2KG, KG-Gen, Atom++→ [Learn about Methods](methods.md)++### [Template Format](templates-format.md)++The YAML format for defining extraction templates:++- Schema definition+- Prompt engineering+- Guidelines and rules++→ [Learn about Template Format](templates-format.md)++### [Architecture](architecture.md)++Deep dive into the system design:++- Data flow+- Processing pipeline+- Extension points++→ [Learn about Architecture](architecture.md)++---## Quick Links-- [Auto-Types](./auto-types.md) - 8 strong-typed knowledge structures-- [Templates](../templates/index.md) - Ready-to-use templates+- [CLI Documentation](../cli/index.md) — Terminal usage+- [Python SDK](../python/index.md) — Programmatic usage+- [Template Library](../templates/index.md) — Available templatesdiff --git a/docs/en/concepts/methods.md b/docs/en/concepts/methods.mdnew file mode 100644index 0000000..e876037--- /dev/null+++ b/docs/en/concepts/methods.md@@ -0,0 +1,300 @@+# Methods++Guide to extraction algorithms and when to use each.++---++## Overview++Methods are the underlying algorithms that extract knowledge from text. They determine:+- How text is processed+- How entities and relations are identified+- The quality and style of extraction++---++## Method Categories++### RAG-Based Methods++**Retrieval-Augmented Generation** combines information retrieval with text generation.++**Best for**: Large documents, complex queries++```mermaid+graph LR+ A[Document] --> B[Chunk & Index]+ B --> C[Retrieve Relevant]+ C --> D[Generate Answer]+ D --> E[Extract Structure]+```++### Typical Methods++**Direct extraction** without retrieval.++**Best for**: Smaller documents, direct extraction++```mermaid+graph LR+ A[Document] --> B[LLM Processing]+ B --> C[Extract Structure]+```++---++## RAG-Based Methods++### light_rag++**Description**: Lightweight Graph-based RAG++**Characteristics**:+- Fastest RAG method+- Binary edges (source → target)+- Good balance of speed/quality+- Suitable for most use cases++**Best for**:+- General-purpose extraction+- Medium to large documents+- Quick results++**Usage**:+```python+ka = Template.create("method/light_rag")+```++---++### graph_rag++**Description**: Graph-RAG with Community Detection++**Characteristics**:+- Community detection for organization+- Hierarchical summaries+- Best for very large documents+- Slower but thorough++**Best for**:+- Very large documents (books, long papers)+- Complex topic structures+- Research papers++**Usage**:+```python+ka = Template.create("method/graph_rag")+```++---++### hyper_rag++**Description**: Hypergraph-based RAG++**Characteristics**:+- N-ary hyperedges (2+ entities)+- Captures complex relationships+- Richer graph structure++**Best for**:+- Multi-party relationships+- Project collaborations+- Complex organizational structures++**Usage**:+```python+ka = Template.create("method/hyper_rag")+```++---++### hypergraph_rag++**Description**: Advanced Hypergraph RAG++**Characteristics**:+- Enhanced hypergraph capabilities+- Advanced relationship modeling++**Best for**:+- Complex hypergraph scenarios+- Advanced relationship analysis++**Usage**:+```python+ka = Template.create("method/hypergraph_rag")+```++---++### cog_rag++**Description**: Cognitive RAG++**Characteristics**:+- Cognitive retrieval mechanisms+- Reasoning-focused++**Best for**:+- Reasoning tasks+- Question-answering systems++**Usage**:+```python+ka = Template.create("method/cog_rag")+```++---++## Typical Methods++### itext2kg++**Description**: High-quality triple-based extraction++**Characteristics**:+- Optimized for triple quality+- Iterative refinement+- Good for knowledge base construction++**Best for**:+- Knowledge graph construction+- High-quality requirements+- Triple extraction++**Usage**:+```python+ka = Template.create("method/itext2kg")+```++---++### itext2kg_star++**Description**: Enhanced iText2KG++**Characteristics**:+- Improved extraction quality+- Better handling of complex cases+- Enhanced entity linking++**Best for**:+- When quality is critical+- Complex extraction scenarios+- Production systems++**Usage**:+```python+ka = Template.create("method/itext2kg_star")+```++---++### kg_gen++**Description**: Knowledge Graph Generator++**Characteristics**:+- Configurable generation+- Flexible schema+- Fast processing++**Best for**:+- Custom schemas+- Rapid prototyping+- Flexible requirements++**Usage**:+```python+ka = Template.create("method/kg_gen")+```++---++### atom++**Description**: Temporal knowledge graph with evidence++**Characteristics**:+- Temporal fact extraction+- Evidence attribution+- Confidence scoring++**Best for**:+- Temporal analysis+- Fact verification+- Timeline extraction++**Usage**:+```python+ka = Template.create("method/atom")+```++---++## Selection Guide++### By Document Size++| Size | Recommended |+|------|-------------|+| Small (< 1K words) | itext2kg, kg_gen |+| Medium (1-10K) | light_rag, itext2kg_star |+| Large (10-50K) | light_rag, graph_rag |+| Very Large (> 50K) | graph_rag |++### By Use Case++| Use Case | Recommended |+|----------|-------------|+| Quick extraction | light_rag |+| Best quality | itext2kg_star |+| Large documents | graph_rag |+| Complex relationships | hyper_rag |+| Temporal facts | atom |+| Knowledge bases | itext2kg |++### By Priority++| Priority | Recommended |+|----------|-------------|+| Speed | light_rag |+| Quality | itext2kg_star |+| Cost | light_rag |+| Completeness | graph_rag |++---++## Comparison Table++| Method | Speed | Quality | Memory | Best For |+|--------|-------|---------|--------|----------|+| light_rag | ⭐⭐⭐ | ⭐⭐ | ⭐⭐ | General use |+| graph_rag | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ | Large docs |+| hyper_rag | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | Complex relations |+| itext2kg | ⭐⭐⭐ | ⭐⭐⭐ | ⭐ | Quality focused |+| atom | ⭐⭐ | ⭐⭐⭐ | ⭐⭐ | Temporal data |+| kg_gen | ⭐⭐⭐ | ⭐⭐ | ⭐ | Flexibility |++---++## Listing Available Methods++```python+from hyperextract.methods import list_methods++methods = list_methods()+for name, info in methods.items():+ print(f"{name}: {info['description']}")+ print(f" Type: {info['type']}")+```++---++## See Also++- [Choosing Methods Guide](../python/guides/choosing-methods.md)+- [Templates](../templates/index.md)+- [Auto-Types](autotypes.md)diff --git a/docs/en/concepts/templates-format.md b/docs/en/concepts/templates-format.mdnew file mode 100644index 0000000..62375a7--- /dev/null+++ b/docs/en/concepts/templates-format.md@@ -0,0 +1,309 @@+# Template Format++Complete reference for YAML template structure.++---++## Overview++Templates are YAML files that define:+- **Output schema** — What to extract+- **Prompts** — How to instruct the LLM+- **Guidelines** — Extraction rules+- **Configuration** — Display and identifiers++---++## Template Structure++```yaml+language: [zh, en] # Supported languages++name: template_name # Template identifier+type: graph # Auto-Type+tags: [domain, category] # Classification tags++description: # Human-readable description+ zh: "中文描述"+ en: "English description"++output: # Output schema definition+ description:+ zh: "输出描述"+ en: "Output description"+ entities: # Entity schema+ description:+ zh: "实体描述"+ en: "Entity description"+ fields:+ - name: field_name+ type: str+ description:+ zh: "字段描述"+ en: "Field description"+ required: true+ relations: # Relation schema (for graphs)+ # Same structure as entities++guideline: # LLM instructions+ target:+ zh: "提取目标"+ en: "Extraction target"+ rules_for_entities:+ zh: ["规则1", "规则2"]+ en: ["Rule 1", "Rule 2"]+ rules_for_relations:+ zh: ["规则1"]+ en: ["Rule 1"]++identifiers: # ID generation rules+ entity_id: name # Field used as entity ID+ relation_id: "{source}|{type}|{target}" # Relation ID format+ relation_members:+ source: source+ target: target++display: # Visualization settings+ entity_label: "{name} ({type})"+ relation_label: "{type}"+```++---++## Field Types++| Type | Description | Example |+|------|-------------|---------|+| `str` | String | `"Hello"` |+| `int` | Integer | `42` |+| `float` | Float | `3.14` |+| `bool` | Boolean | `true` |+| `list[str]` | List of strings | `["a", "b"]` |+| `datetime` | Date/time | `"2024-01-01"` |++---++## Complete Example++```yaml+language: [zh, en]++name: biography_graph+type: temporal_graph+tags: [general, biography, life, events, timeline]++description:+ zh: '传记图模板 - 从人物传记、回忆录、年谱中提取实体和关系'+ en: 'Biography Graph Template - Extract entities and relationships from biographies'++output:+ description:+ zh: '人物传记中的实体和关系网络'+ en: "Entity and relationship network from biographies"++ entities:+ description:+ zh: '传记中的实体'+ en: 'Entities in the biography'+ fields:+ - name: name+ type: str+ description:+ zh: '实体名称'+ en: 'Entity name'+ required: true++ - name: type+ type: str+ description:+ zh: '实体类型:人物、地点、组织、作品、事件、成就等'+ en: 'Entity type: person, location, organization, work, event, achievement'+ required: true++ - name: description+ type: str+ description:+ zh: '实体描述'+ en: 'Entity description'+ required: false++ relations:+ description:+ zh: '实体之间的关系'+ en: 'Relationships between entities'+ fields:+ - name: source+ type: str+ description:+ zh: '关系起点实体'+ en: 'Source entity'+ required: true++ - name: target+ type: str+ description:+ zh: '关系终点实体'+ en: 'Target entity'+ required: true++ - name: type+ type: str+ description:+ zh: '关系类型'+ en: 'Relation type'+ required: true++ - name: time+ type: str+ description:+ zh: '关系对应的时间'+ en: 'Time associated with the relation'+ required: false++guideline:+ target:+ zh: '你是一位专业的传记分析师...'+ en: 'You are a professional biography analyst...'++ rules_for_entities:+ zh:+ - '提取传记主人公作为核心实体'+ - '同一实体在全文中保持命名一致'+ en:+ - 'Extract the biographical subject as the core entity'+ - 'Maintain consistent naming for the same entity'++ rules_for_relations:+ zh:+ - '仅在文本明确表达两个实体之间存在关系时创建关系'+ en:+ - 'Create relationships only when explicitly expressed'++ rules_for_time:+ zh:+ - '绝对日期:保持原文格式'+ - '相对时间:转换为绝对时间'+ en:+ - 'Absolute dates: Keep as-is'+ - 'Relative time: Convert to absolute'++identifiers:+ entity_id: name+ relation_id: '{source}|{type}|{target}'+ relation_members:+ source: source+ target: target+ time_field: time++display:+ entity_label: '{name}'+ relation_label: '{type}'+```++---++## Auto-Type Specific Sections++### For Lists/Sets++```yaml+output:+ items: # Instead of entities/relations+ description:+ zh: "列表项"+ en: "List items"+ fields:+ - name: value+ type: str+```++### For Models++```yaml+output:+ fields: # Direct fields (no entities)+ - name: company_name+ type: str+ - name: revenue+ type: float+```++### For Temporal Graphs++```yaml+guideline:+ rules_for_time: # Time extraction rules+ zh: ["规则1", "规则2"]+ en: ["Rule 1", "Rule 2"]++identifiers:+ time_field: time # Which field contains time+```++### For Spatial Graphs++```yaml+output:+ entities:+ fields:+ - name: location # Location field+ type: str+ - name: coordinates # Optional coordinates+ type: str++guideline:+ rules_for_location: # Location extraction rules+ zh: ["规则1"]+ en: ["Rule 1"]+```++---++## Template Variables++Use placeholders in guidelines:++```yaml+guideline:+ target:+ zh: '观察时间基准:{observation_time}'+ en: 'Observation time baseline: {observation_time}'+```++Passed at runtime:++```python+ka = Template.create(+ "template_name",+ "en",+ observation_time="2024-01-01"+)+```++---++## Validation++Templates are validated for:+- Required fields present+- Type definitions valid+- Language codes supported+- Schema consistency++---++## Best Practices++1. **Clear descriptions** — Help users choose the right template+2. **Specific rules** — Guide LLM to better extraction+3. **Consistent naming** — Use clear, consistent field names+4. **Bilingual support** — Provide both languages when possible+5. **Required vs optional** — Mark fields appropriately++---++## See Also++- [Template Library](../templates/index.md)+- [Creating Custom Templates](../python/guides/custom-templates.md)+- [Auto-Types](autotypes.md)diff --git a/docs/en/getting-started/cli-quickstart.md b/docs/en/getting-started/cli-quickstart.mdnew file mode 100644index 0000000..f6fee7f--- /dev/null+++ b/docs/en/getting-started/cli-quickstart.md@@ -0,0 +1,178 @@+# CLI Quickstart++Get your first knowledge extraction running in 5 minutes using the terminal.++---++## Prerequisites++- [Hyper-Extract installed](installation.md)+- API key configured (via `he config init` or environment variable)+- A text file to extract from (we'll use an example)++---++## Step 1: Download Sample Document++```bash+# Download a sample biography+curl -o tesla.md https://raw.githubusercontent.com/yifanfeng97/hyper-extract/main/examples/en/tesla.md+```++Or create a simple test file:++```bash+cat > sample.txt << 'EOF'+Nikola Tesla was a Serbian-American inventor, electrical engineer,+mechanical engineer, and futurist. He is best known for his+contributions to the design of the modern alternating current+(AC) electricity supply system.++Born: July 10, 1856, Smiljan, Croatia+Died: January 7, 1943, New York City, NY+EOF+```++---++## Step 2: Extract Knowledge++Run the `parse` command to extract knowledge:++```bash+he parse tesla.md -t general/biography_graph -o ./output/ -l en+```++What this does:+- `-t general/biography_graph` — Use the biography graph template+- `-o ./output/` — Save results to the output directory+- `-l en` — Process in English++**Output:**+```+Input: tesla.md+Output: ./output/+Template: general/biography_graph+Language: en+Build Index: Yes++Template resolved: Biography Graph Template+✓ Knowledge extracted to ./output/++What's next?+ he show ./output/ # Visualize knowledge graph+ he search ./output/ "keyword" # Semantic search+ he talk ./output/ -i # Interactive chat+```++---++## Step 3: Visualize the Knowledge Graph++```bash+he show ./output/+```++This opens an interactive visualization in your browser, showing:+- **Entities** (people, places, events) as nodes+- **Relationships** as edges connecting the nodes++---++## Step 4: Search Your Knowledge Base++```bash+he search ./output/ "What were Tesla's major inventions?"+```++**Output:**+```+Found 3 result(s):++Result 1:+{+ "name": "Nikola Tesla",+ "type": "person",+ "description": "Serbian-American inventor..."+}+...+```++---++## Step 5: Chat with Your Knowledge++Interactive mode:++```bash+he talk ./output/ -i+```++Or ask a single question:++```bash+he talk ./output/ -q "Summarize Tesla's career in three sentences"+```++---++## Step 6: Incrementally Add Knowledge++Got more documents? Add them without reprocessing:++```bash+he feed ./output/ additional_document.md+```++Then visualize the updated knowledge:++```bash+he show ./output/+```++---++## Complete Workflow++Here's the typical workflow:++```bash+# 1. Extract knowledge+he parse document.md -t general/biography_graph -o ./output/ -l en++# 2. Visualize+he show ./output/++# 3. Search+he search ./output/ "your query"++# 4. Chat+he talk ./output/ -i++# 5. Add more documents+he feed ./output/ another_document.md++# 6. Rebuild index if needed+he build-index ./output/+```++---++## What's Next?++- [CLI Workflow Guide](../cli/workflow.md) — Complete workflow walkthrough+- [All CLI Commands](../cli/index.md) — Detailed command reference+- [Template Library](../templates/index.md) — Find templates for your use case++---++## Troubleshooting++**"No API key found"**+→ Run `he config init -k YOUR_API_KEY`++**"Template not found"**+→ List available templates with `he list template`++**"Output directory already exists"**+→ Add `-f` flag to force overwrite, or choose a different output pathdiff --git a/docs/en/getting-started/index.md b/docs/en/getting-started/index.mdnew file mode 100644index 0000000..c871a51--- /dev/null+++ b/docs/en/getting-started/index.md@@ -0,0 +1,53 @@+# Getting Started++Welcome to Hyper-Extract! This guide will help you get up and running quickly, whether you prefer using the command line or Python.++---++## Choose Your Path++<div class="grid cards" markdown>++- :material-console-line:{ .lg .middle } __I want to use the terminal__++ ---++ The CLI is perfect if you:++ - Want to process documents without writing code+ - Need to batch process files+ - Prefer terminal workflows++ **Start here:** [:octicons-arrow-right-24: Installation](installation.md) → [:octicons-arrow-right-24: CLI Quickstart](cli-quickstart.md)++- :material-language-python:{ .lg .middle } __I want to use Python__++ ---++ The Python SDK is perfect if you:++ - Are building an application+ - Need to integrate with existing code+ - Want full programmatic control++ **Start here:** [:octicons-arrow-right-24: Installation](installation.md) → [:octicons-arrow-right-24: Python Quickstart](python-quickstart.md)++</div>++---++## What's Next?++After completing the quickstart:++1. **Explore the [CLI Guide](../cli/index.md)** — Learn the complete workflow+2. **Browse the [Template Library](../templates/index.md)** — Find templates for your domain+3. **Read [Core Concepts](../concepts/index.md)** — Understand how it all works++---++## Need Help?++- **Questions?** Check the [FAQ](../resources/faq.md)+- **Issues?** See [Troubleshooting](../resources/troubleshooting.md)+- **Found a bug?** [Report it on GitHub](https://github.com/yifanfeng97/hyper-extract/issues)diff --git a/docs/en/getting-started/installation.md b/docs/en/getting-started/installation.mdnew file mode 100644index 0000000..ade68f3--- /dev/null+++ b/docs/en/getting-started/installation.md@@ -0,0 +1,120 @@+# Installation++Hyper-Extract requires **Python 3.11+**.++---++## Install from PyPI++=== "pip"++ ```bash+ pip install hyper-extract+ ```++=== "uv (recommended)"++ ```bash+ uv pip install hyper-extract+ ```++=== "conda"++ ```bash+ pip install hyper-extract+ ```++---++## Verify Installation++```bash+he --version+```++You should see something like:++```+Hyper-Extract CLI version 0.1.0+```++---++## Optional Dependencies++=== "Anthropic Claude Support"++ ```bash+ pip install hyper-extract[anthropic]+ ```++=== "Google Gemini Support"++ ```bash+ pip install hyper-extract[google]+ ```++=== "All Optional Dependencies"++ ```bash+ pip install hyper-extract[all]+ ```++---++## Configuration++Before using Hyper-Extract, you need to configure your LLM API credentials.++### Option 1: Using CLI (Recommended)++```bash+he config init -k YOUR_OPENAI_API_KEY+```++This creates a configuration file at `~/.he/config.toml`.++### Option 2: Using Environment Variables++```bash+export OPENAI_API_KEY=your-api-key+export OPENAI_BASE_URL=https://api.openai.com/v1 # Optional+```++### Option 3: Using .env File (Python Only)++Create a `.env` file in your project root:++```+OPENAI_API_KEY=your-api-key+OPENAI_BASE_URL=https://api.openai.com/v1+```++Then load it in your Python code:++```python+from dotenv import load_dotenv+load_dotenv()+```++---++## Development Installation++If you want to contribute or modify the source code:++```bash+# Clone the repository+git clone https://github.com/yifanfeng97/hyper-extract.git+cd hyper-extract++# Install in editable mode with dev dependencies+pip install -e ".[dev]"+```++---++## What's Next?++- [:octicons-arrow-right-24: CLI Quickstart](cli-quickstart.md) — Your first extraction from the terminal+- [:octicons-arrow-right-24: Python Quickstart](python-quickstart.md) — Your first extraction with Pythondiff --git a/docs/en/getting-started/python-quickstart.md b/docs/en/getting-started/python-quickstart.mdnew file mode 100644index 0000000..dc82508--- /dev/null+++ b/docs/en/getting-started/python-quickstart.md@@ -0,0 +1,245 @@+# Python Quickstart++Get your first knowledge extraction running in 5 minutes using Python.++---++## Prerequisites++- [Hyper-Extract installed](installation.md)+- API key configured (via `.env` file or environment variable)++---++## Step 1: Set Up Your Project++Create a new directory and set up your environment:++```bash+mkdir my_extraction_project+cd my_extraction_project++# Create virtual environment (optional but recommended)+python -m venv venv+source venv/bin/activate # On Windows: venv\Scripts\activate++# Install hyper-extract+pip install hyper-extract+```++---++## Step 2: Configure API Key++Create a `.env` file:++```bash+echo "OPENAI_API_KEY=your-api-key" > .env+```++Or set it in your code:++```python+import os+os.environ["OPENAI_API_KEY"] = "your-api-key"+```++---++## Step 3: Create Your First Extraction Script++Create a file `extract.py`:++```python+from dotenv import load_dotenv++# Load API key from .env file+load_dotenv()++from hyperextract import Template++# Create a template instance+ka = Template.create("general/biography_graph", language="en")++# Sample text+text = """+Nikola Tesla was a Serbian-American inventor, electrical engineer,+mechanical engineer, and futurist. He is best known for his+contributions to the design of the modern alternating current+(AC) electricity supply system.++Born: July 10, 1856, Smiljan, Croatia+Died: January 7, 1943, New York City, NY+"""++# Extract knowledge+result = ka.parse(text)++# Access the extracted data+print(f"Entities: {len(result.data.entities)}")+print(f"Relations: {len(result.data.relations)}")++# Print first entity+if result.data.entities:+ entity = result.data.entities[0]+ print(f"\nFirst entity: {entity.name} ({entity.type})")++# Visualize+result.show()+```++---++## Step 4: Run the Script++```bash+python extract.py+```++You should see:++```+Entities: 5+Relations: 4++First entity: Nikola Tesla (person)+```++And a browser window will open showing the interactive knowledge graph.++---++## Step 5: Working with Results++Access different parts of the extraction:++```python+# Iterate over all entities+for entity in result.data.entities:+ print(f"- {entity.name}: {entity.description}")++# Iterate over all relations+for relation in result.data.relations:+ print(f"- {relation.source} --{relation.type}--> {relation.target}")++# Search within the knowledge base+result.build_index()+search_results = result.search("inventions", top_k=3)+for item in search_results:+ print(item)+```++---++## Step 6: Save and Load++Save your knowledge base for later use:++```python+# Save to disk+result.dump("./my_knowledge_base/")++# Load it back later+new_ka = Template.create("general/biography_graph", language="en")+new_ka.load("./my_knowledge_base/")+```++---++## Step 7: Incremental Updates++Add more text without losing existing knowledge:++```python+additional_text = """+Tesla worked for Thomas Edison in New York City in 1884.+They had a contentious relationship due to differing views on DC vs AC power.+"""++# Feed adds to existing knowledge+result.feed_text(additional_text)++# Visualize the updated graph+result.show()+```++---++## Complete Example++Here's a complete, production-ready script:++```python+"""Extract knowledge from a document and interact with it."""++import os+from pathlib import Path+from dotenv import load_dotenv+from hyperextract import Template++# Configuration+load_dotenv()++INPUT_FILE = "document.txt"+OUTPUT_DIR = "./output/"+TEMPLATE = "general/biography_graph"+LANGUAGE = "en"+++def main():+ # Create template+ print(f"Creating template: {TEMPLATE}")+ ka = Template.create(TEMPLATE, language=LANGUAGE)++ # Read document+ print(f"Reading: {INPUT_FILE}")+ text = Path(INPUT_FILE).read_text(encoding="utf-8")++ # Extract knowledge+ print("Extracting knowledge...")+ result = ka.parse(text)++ # Print summary+ print(f"\nExtraction complete:")+ print(f" - Entities: {len(result.data.entities)}")+ print(f" - Relations: {len(result.data.relations)}")++ # Build search index+ print("Building search index...")+ result.build_index()++ # Save to disk+ print(f"Saving to: {OUTPUT_DIR}")+ result.dump(OUTPUT_DIR)++ # Interactive visualization+ print("Opening visualization...")+ result.show()++ print("\nDone!")+++if __name__ == "__main__":+ main()+```++---++## What's Next?++- [Python SDK Overview](../python/index.md) — Complete API reference+- [Working with Templates](../python/guides/using-templates.md) — Template usage guide+- [Auto-Types Guide](../concepts/autotypes.md) — Choose the right data structure++---++## Troubleshooting++**"No module named 'hyperextract'"**+→ Run `pip install hyper-extract`++**"API key not found"**+→ Check your `.env` file or set `OPENAI_API_KEY` environment variable++**"Template not found"**+→ Use `Template.list()` to see available templatesdiff --git a/docs/en/index.md b/docs/en/index.mdindex e943856..6cf5fe6 100644--- a/docs/en/index.md+++ b/docs/en/index.md@@ -1,167 +1,216 @@-# 🚀 Hyper-Extract--> **"Stop reading. Start understanding."**+<div align="center">+ <picture>+ <source media="(prefers-color-scheme: dark)" srcset="../assets/logo/logo-horizontal-dark.svg">+ <source media="(prefers-color-scheme: light)" srcset="../assets/logo/logo-horizontal.svg">+ <img alt="Hyper-Extract Logo" src="../assets/logo/logo-horizontal.svg" width="500">+ </picture>+</div>++<br/>++> **Transform documents into structured knowledge with one command.**+>> *"告别文档焦虑,让信息一目了然"*-**Smart Knowledge Extraction CLI — Transform documents into structured knowledge with one command.**+**Hyper-Extract** is an intelligent, LLM-powered knowledge extraction framework. It transforms unstructured text into persistent, predictable, and strongly-typed knowledge structures—from simple lists to complex knowledge graphs, hypergraphs, and spatio-temporal graphs.-+----[](https://python.org)-[](LICENSE)-[]()+## ⚡ 5-Minute Quick Start-## What is Hyper-Extract?+=== "CLI (Terminal)"-Hyper-Extract is an intelligent, LLM-powered knowledge extraction and evolution framework. It radically simplifies transforming highly unstructured texts into persistent, predictable, and strongly-typed knowledge summaries. It effortlessly extracts information into a wide spectrum of formats—ranging from simple **Collections** (Lists/Sets) and **Pydantic Models**, to complex **Knowledge Graphs**, **Hypergraphs**, and even **Spatio-Temporal Graphs**.+ ```bash+ # 1. Install+ pip install hyper-extract-## ✨ Core Features+ # 2. Configure API Key+ he config init -k YOUR_OPENAI_API_KEY-- 🔷 **8 Auto-Types:** From basic `AutoModel`/`AutoList` to advanced `AutoGraph`, `AutoHypergraph`, and `AutoSpatioTemporalGraph`.-- 🧠 **10+ Extraction Engines:** Out-of-the-box support for cutting-edge retrieval paradigms like `GraphRAG`, `LightRAG`, `Hyper-RAG`, and `KG-Gen`.-- 📝 **Declarative YAML Templates:** Zero-code extraction definition. Includes 80+ presets across 6 domains.-- 🔄 **Incremental Evolution:** Feed new documents on the fly to continuously map out and expand the extracted knowledge.+ # 3. Extract knowledge from a document+ he parse tesla.md -t general/biography_graph -o ./output/ -l en-## ⚡ Quick Start+ # 4. Visualize the knowledge graph+ he show ./output/+ ```-### Installation+=== "Python"-```bash-uv pip install hyper-extract-```+ ```python+ from hyperextract import Template-### The Command Line Way+ # 1. Create a template+ ka = Template.create("general/biography_graph", "en")-```bash-# Configure OpenAI API Key-he config init -k YOUR_OPENAI_API_KEY+ # 2. Extract knowledge+ with open("tesla.md") as f:+ result = ka.parse(f.read())-# Extract knowledge-he parse examples/en/tesla.md -t general/biography_graph -o ./output/ -l en+ # 3. Visualize+ ka.show(result)+ ```-# Query the knowledge base-he search ./output/ "What are Tesla's major achievements?"+**→ Ready to dive deeper?** Check out the [Getting Started Guide](getting-started/index.md) or jump to [CLI](cli/index.md) / [Python SDK](python/index.md) documentation.-# Visualize the knowledge graph-he show ./output/+----# Incrementally supplement knowledge-he feed ./output/ examples/en/tesla_question.md+## ✨ What Makes Hyper-Extract Different?-# Show the updated knowledge graph-he show ./output/-```+<div class="grid cards" markdown>-### The Python API Way+- :material-shape:{ .lg .middle } **8 Auto-Types**-#### Installation+ ----```bash-# Clone the repository-git clone https://github.com/yifanfeng97/hyper-extract.git-cd hyper-extract+ From simple `AutoList`/`AutoModel` to advanced `AutoGraph`, `AutoHypergraph`, and `AutoSpatioTemporalGraph`. Pick the right structure for your data.-# Install dependencies-uv sync-```+- :material-brain:{ .lg .middle } **10+ Extraction Engines**-#### Configuration+ ----```bash-# Copy the example env file-cp .env.example .env+ Built-in support for GraphRAG, LightRAG, Hyper-RAG, KG-Gen, iText2KG, and more. Choose the best method for your use case.-# Edit .env with your API key and base URL-# OPENAI_API_KEY=your-api-key-# OPENAI_BASE_URL=https://api.openai.com/v1-```+- :material-file-document:{ .lg .middle } **80+ Domain Templates**-#### Usage+ ----```python-import os-from dotenv import load_dotenv+ Ready-to-use templates for Finance, Legal, Medical, TCM, and Industry. Zero configuration needed.-# Load environment variables from .env file-load_dotenv()+- :material-sync:{ .lg .middle } **Incremental Evolution**-from hyperextract import Template+ ----# Create a template-ka = Template.create("general/biography_graph")+ Feed new documents to continuously expand your knowledge base. No need to reprocess everything.-# Parse a document-with open("examples/en/tesla.md", "r") as f:- text = f.read()-result = ka.parse(text)+</div>-# Visualize the knowledge graph-ka.show(result)+----# Incrementally supplement knowledge-with open("examples/en/tesla_question.md", "r") as f:- new_text = f.read()-ka.feed(result, new_text)+## 🎯 Choose Your Path-# Show the updated knowledge graph-ka.show(result)-```+<div class="grid cards" markdown>++- :material-console:{ .lg .middle } __CLI User__++ ---++ Process documents directly from your terminal. Perfect for:++ - Quick knowledge extraction+ - Batch document processing+ - Building knowledge bases without coding++ [:octicons-arrow-right-24: CLI Guide](cli/index.md)++- :material-language-python:{ .lg .middle } __Python Developer__-## 🧩 The 8 Auto-Types+ ----Our framework embraces complexity without making you write boilerplate code.+ Integrate into your Python applications. Perfect for:++ - Custom extraction pipelines+ - Integration with existing workflows+ - Building AI-powered applications++ [:octicons-arrow-right-24: Python SDK](python/index.md)-+- :material-school:{ .lg .middle } __Want to Learn More?__-| Type | Description |-|------|-------------|-| AutoModel | Pydantic model extraction |-| AutoList | List/collection extraction |-| AutoSet | Unique set extraction |-| AutoGraph | Knowledge graph extraction |-| AutoHypergraph | Hypergraph extraction |-| AutoTemporalGraph | Temporal knowledge graph |-| AutoSpatialGraph | Spatial knowledge graph |-| AutoSpatioTemporalGraph | Spatio-temporal knowledge graph |+ ----### Example: AutoGraph Visualization+ Understand the concepts and architecture:++ - How Auto-Types work+ - Choosing extraction methods+ - Creating custom templates++ [:octicons-arrow-right-24: Concepts](concepts/index.md)-Here is the knowledge graph visualization after `AutoGraph` extraction:+</div>-+----## 🛠️ Architecture+## 🧩 The 8 Auto-Types at a Glance++| Type | Use Case | Example Output |+|------|----------|----------------|+| **AutoModel** | Structured summaries | A pydantic model with specific fields |+| **AutoList** | Collections of items | A list of entities or facts |+| **AutoSet** | Deduplicated collections | A set of unique items |+| **AutoGraph** | Entity-relationship networks | Knowledge graph with nodes and edges |+| **AutoHypergraph** | Multi-entity relationships | Hyperedges connecting multiple nodes |+| **AutoTemporalGraph** | Time-based relationships | Graph with temporal information |+| **AutoSpatialGraph** | Location-based relationships | Graph with geographic data |+| **AutoSpatioTemporalGraph** | Time + Space combined | Full context with when and where |++→ [Learn which Auto-Type to choose](concepts/autotypes.md)++---++## 🏗️ Architecture OverviewHyper-Extract follows a **three-layer architecture**:-- **Auto-Types** define the data structures for knowledge extraction. With 8 strong-typed structures (AutoModel, AutoList, AutoSet, AutoGraph, AutoHypergraph, AutoTemporalGraph, AutoSpatialGraph, AutoSpatioTemporalGraph), they serve as the output format for all extractions.+```mermaid+graph TD+ A[Your Document] --> B[Template / Method]+ B --> C[Auto-Type]+ C --> D[Structured Knowledge]++ subgraph "Layer 3: Templates"+ B+ end++ subgraph "Layer 2: Methods"+ B+ end++ subgraph "Layer 1: Auto-Types"+ C+ end+```++1. **Auto-Types** — Define the output data structure (8 types)+2. **Methods** — Provide extraction algorithms (RAG-based and Typical)+3. **Templates** — Offer domain-specific, ready-to-use configurations++You can use Hyper-Extract at any level: pick a template for quick results, choose a method for more control, or work directly with Auto-Types for full customization.++---++## 📊 Comparison with Other Tools++| Feature | GraphRAG | LightRAG | KG-Gen | **Hyper-Extract** |+|---------|:--------:|:--------:|:------:|:-----------------:|+| Knowledge Graph | ✅ | ✅ | ✅ | ✅ |+| Temporal Graph | ✅ | ❌ | ❌ | ✅ |+| Spatial Graph | ❌ | ❌ | ❌ | ✅ |+| Hypergraph | ❌ | ❌ | ❌ | ✅ |+| Domain Templates | ❌ | ❌ | ❌ | ✅ |+| CLI Tool | ✅ | ❌ | ❌ | ✅ |+| Multi-language | ✅ | ❌ | ❌ | ✅ |-- **Methods** provide extraction algorithms built on Auto-Types. This includes Typical methods (KG-Gen, iText2KG, iText2KG*) and RAG-based methods (GraphRAG, LightRAG, Hyper-RAG, HypergraphRAG, Cog-RAG).+----- **Templates** offer domain-specific configurations with ready-to-use prompts and data structures. Covering 6 domains (Finance, Legal, Medical, TCM, Industry, General) with 80+ preset templates, users can extract knowledge without dealing with Auto-Types or Methods directly.+## 📚 Documentation Structure-Use via **CLI** (`he parse`, `he search`, `he show`...) or **Python API** (`Template.create()`).+- **[Getting Started](getting-started/index.md)** — Installation and your first extraction+- **[CLI Guide](cli/index.md)** — Complete terminal workflow documentation+- **[Python SDK](python/index.md)** — API reference and developer guides+- **[Concepts](concepts/index.md)** — Understanding the architecture+- **[Templates](templates/index.md)** — Domain-specific extraction templates+- **[Resources](resources/index.md)** — FAQ, troubleshooting, and contributing-+----## 📈 Comparison with Other Libraries+## 🤝 Contributing-| Feature | GraphRAG | LightRAG | KG-Gen | ATOM | **Hyper-Extract** |-| ---------------- | :------: | :------: | :----: | :--: | :---------------: |-| Knowledge Graph | ✅ | ✅ | ✅ | ✅ | ✅ |-| Temporal Graph | ✅ | ❌ | ❌ | ✅ | ✅ |-| Spatial Graph | ❌ | ❌ | ❌ | ❌ | ✅ |-| Hypergraph | ❌ | ❌ | ❌ | ❌ | ✅ |-| Domain Templates | ❌ | ❌ | ❌ | ❌ | ✅ |-| CLI Tool | ✅ | ❌ | ❌ | ❌ | ✅ |-| Multi-language | ✅ | ❌ | ❌ | ❌ | ✅ |+Contributions are welcome! Whether it's bug reports, feature requests, or documentation improvements, please feel free to submit an issue or pull request.-## 📚 Documentation+[:fontawesome-brands-github: GitHub Repository](https://github.com/yifanfeng97/hyper-extract){ .md-button .md-button--primary }-- [Getting Started](cli/index.md) - Get up and running quickly-- [Domain Templates](templates/index.md) - Ready-to-use templates+----## 🤝 Contributing & License+## 📄 License-Contributions are welcome! Please submit Issues and PRs.-Licensed under **Apache-2.0**.+Hyper-Extract is licensed under the [Apache-2.0 License](https://github.com/yifanfeng97/hyper-extract/blob/main/LICENSE).diff --git a/docs/en/python/api-reference/autotypes.md b/docs/en/python/api-reference/autotypes.mdnew file mode 100644index 0000000..1c13560--- /dev/null+++ b/docs/en/python/api-reference/autotypes.md@@ -0,0 +1,68 @@+# Auto-Types API++## BaseAutoType++::: hyperextract.types.BaseAutoType+ options:+ members:+ - parse+ - feed_text+ - search+ - chat+ - build_index+ - dump+ - load+ - show+ - empty+ - clear+ - data+ - data_schema+ - metadata++## AutoModel++::: hyperextract.types.AutoModel+ options:+ members: true++## AutoList++::: hyperextract.types.AutoList+ options:+ members: true++## AutoSet++::: hyperextract.types.AutoSet+ options:+ members: true++## AutoGraph++::: hyperextract.types.AutoGraph+ options:+ members: true++## AutoHypergraph++::: hyperextract.types.AutoHypergraph+ options:+ members: true++## AutoTemporalGraph++::: hyperextract.types.AutoTemporalGraph+ options:+ members: true++## AutoSpatialGraph++::: hyperextract.types.AutoSpatialGraph+ options:+ members: true++## AutoSpatioTemporalGraph++::: hyperextract.types.AutoSpatioTemporalGraph+ options:+ members: truediff --git a/docs/en/python/api-reference/methods.md b/docs/en/python/api-reference/methods.mdnew file mode 100644index 0000000..3077767--- /dev/null+++ b/docs/en/python/api-reference/methods.md@@ -0,0 +1,136 @@+# Methods API++API reference for extraction methods.++---++## Registry Functions++::: hyperextract.methods.registry+ options:+ members:+ - register_method+ - get_method+ - list_methods+ - get_method_cfg+ - list_method_cfgs++---++## Method Configuration++::: hyperextract.methods.registry.MethodCfg+ options:+ members: true++---++## RAG-Based Methods++### LightRAG++::: hyperextract.methods.rag.Light_RAG+ options:+ members: true++### GraphRAG++::: hyperextract.methods.rag.Graph_RAG+ options:+ members: true++### HyperRAG++::: hyperextract.methods.rag.Hyper_RAG+ options:+ members: true++### HyperGraphRAG++::: hyperextract.methods.rag.HyperGraph_RAG+ options:+ members: true++### CogRAG++::: hyperextract.methods.rag.Cog_RAG+ options:+ members: true++---++## Typical Methods++### iText2KG++::: hyperextract.methods.typical.iText2KG+ options:+ members: true++### iText2KG_Star++::: hyperextract.methods.typical.iText2KG_Star+ options:+ members: true++### KG_Gen++::: hyperextract.methods.typical.KG_Gen+ options:+ members: true++### Atom++::: hyperextract.methods.typical.Atom+ options:+ members: true++---++## Using Methods Directly++```python+from hyperextract.methods import Light_RAG+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++# Create method instance+llm = ChatOpenAI()+emb = OpenAIEmbeddings()++method = Light_RAG(+ llm_client=llm,+ embedder=emb+)++# Extract knowledge+result = method.parse(text)+```++---++## Registering Custom Methods++```python+from hyperextract.methods import register_method++class MyCustomMethod:+ def __init__(self, llm_client, embedder, **kwargs):+ self.llm_client = llm_client+ self.embedder = embedder++ def parse(self, text):+ # Your extraction logic+ pass++# Register the method+register_method(+ name="my_method",+ method_class=MyCustomMethod,+ autotype="graph",+ description="My custom extraction method"+)++# Use via Template API+from hyperextract import Template+ka = Template.create("method/my_method")+```diff --git a/docs/en/python/api-reference/template.md b/docs/en/python/api-reference/template.mdnew file mode 100644index 0000000..5fe4927--- /dev/null+++ b/docs/en/python/api-reference/template.md@@ -0,0 +1,8 @@+# Template API++::: hyperextract.utils.template_engine.Template+ options:+ members:+ - create+ - get+ - listdiff --git a/docs/en/python/config.md b/docs/en/python/config.mddeleted file mode 100644index 3f3b8ee..0000000--- a/docs/en/python/config.md+++ /dev/null@@ -1,74 +0,0 @@-# Python SDK Configuration--## Environment Variables--Create a `.env` file in your project root:--```bash-# Copy the example env file-cp .env.example .env-```--Edit `.env` with your configuration:--```bash-OPENAI_API_KEY=your-api-key-OPENAI_BASE_URL=https://api.openai.com/v1-```--## Loading Environment Variables--Use `python-dotenv` to load environment variables:--```bash-pip install python-dotenv-```--```python-from dotenv import load_dotenv--# Load environment variables from .env file-load_dotenv()-```--## Programmatic Configuration--You can also set environment variables programmatically:--```python-import os--os.environ["OPENAI_API_KEY"] = "your-api-key"-os.environ["OPENAI_BASE_URL"] = "https://api.openai.com/v1"-```--## Configuration Options--### LLM Options--| Option | Environment Variable | Default |-|--------|---------------------|---------|-| API Key | `OPENAI_API_KEY` | Required |-| Base URL | `OPENAI_BASE_URL` | OpenAI default |-| Model | `LLM_MODEL` | `gpt-4o-mini` |-| Temperature | `LLM_TEMPERATURE` | `0.0` |--### Embedder Options--| Option | Environment Variable | Default |-|--------|---------------------|---------|-| Model | `EMBEDDER_MODEL` | `text-embedding-3-small` |-| Batch Size | `EMBEDDER_BATCH_SIZE` | `100` |--## Example .env File--```bash-# Required-OPENAI_API_KEY=sk-your-api-key--# Optional-OPENAI_BASE_URL=https://api.openai.com/v1-LLM_MODEL=gpt-4o-mini-LLM_TEMPERATURE=0.0-EMBEDDER_MODEL=text-embedding-3-small-```diff --git a/docs/en/python/core-concepts.md b/docs/en/python/core-concepts.mdnew file mode 100644index 0000000..fca1669--- /dev/null+++ b/docs/en/python/core-concepts.md@@ -0,0 +1,286 @@+# Core Concepts++Understanding the three pillars of Hyper-Extract: Templates, Auto-Types, and Methods.++---++## Architecture Overview++```mermaid+graph TD+ A[Your Text] --> B{Choose Approach}+ B -->|Quick Start| C[Template]+ B -->|More Control| D[Method]+ C --> E[Auto-Type]+ D --> E+ E --> F[Structured Knowledge]+```++---++## Templates++**Templates** are pre-configured extraction setups for specific domains and tasks.++### What is a Template?++A template combines:+- **Auto-Type** — Output data structure+- **Prompts** — LLM instructions+- **Schema** — Field definitions+- **Guidelines** — Extraction rules++### Using Templates++```python+from hyperextract import Template++# Create from preset+ka = Template.create("general/biography_graph", language="en")++# Create from file+ka = Template.create("/path/to/custom.yaml", language="en")++# List available+templates = Template.list()+for path, cfg in templates.items():+ print(f"{path}: {cfg.description}")+```++### When to Use Templates++✅ **Use templates when:**+- You want quick results+- Working with common document types+- Need domain-specific extraction+- Don't want to configure prompts++❌ **Don't use templates when:**+- You need full control over extraction+- Implementing custom algorithms+- Research/experimentation++---++## Auto-Types++**Auto-Types** define the output data structure for extracted knowledge.++### The 8 Auto-Types++| Type | Description | Best For |+|------|-------------|----------|+| `AutoModel` | Single structured object | Summaries, reports |+| `AutoList` | Ordered collection | Sequences, ranked items |+| `AutoSet` | Deduplicated collection | Unique items, tags |+| `AutoGraph` | Entity-Relation graph | Knowledge graphs |+| `AutoHypergraph` | Multi-entity edges | Complex relationships |+| `AutoTemporalGraph` | Graph + time | Timelines, histories |+| `AutoSpatialGraph` | Graph + space | Maps, locations |+| `AutoSpatioTemporalGraph` | Graph + time + space | Full context |++### Working with Auto-Types++```python+from hyperextract import Template++# Graph extraction+ka = Template.create("general/knowledge_graph", "en")+result = ka.parse(text)++# Access graph data+for entity in result.data.entities:+ print(f"Entity: {entity.name}")++for relation in result.data.relations:+ print(f"{relation.source} --{relation.type}--> {relation.target}")+```++### Choosing an Auto-Type++```mermaid+graph TD+ A[What do you want to extract?] --> B{Structured data?}+ B -->|Yes| C[AutoModel]+ B -->|No| D{Relationships?}+ D -->|No| E{Ordered?}+ E -->|Yes| F[AutoList]+ E -->|No| G[AutoSet]+ D -->|Yes| H{Multi-entity edges?}+ H -->|Yes| I[AutoHypergraph]+ H -->|No| J{Time data?}+ J -->|Yes| K{Space data?}+ K -->|Yes| L[AutoSpatioTemporalGraph]+ K -->|No| M[AutoTemporalGraph]+ J -->|No| N{Space data?}+ N -->|Yes| O[AutoSpatialGraph]+ N -->|No| P[AutoGraph]+```++---++## Methods++**Methods** are the underlying extraction algorithms.++### Types of Methods++#### RAG-Based Methods++Use retrieval-augmented generation for large documents:++| Method | Description |+|--------|-------------|+| `graph_rag` | Community-based retrieval |+| `light_rag` | Lightweight graph RAG |+| `hyper_rag` | Hypergraph RAG |+| `hypergraph_rag` | Advanced hypergraph |+| `cog_rag` | Cognitive RAG |++#### Typical Methods++Direct extraction approaches:++| Method | Description |+|--------|-------------|+| `itext2kg` | Triple-based extraction |+| `itext2kg_star` | Enhanced iText2KG |+| `kg_gen` | Knowledge graph generator |+| `atom` | Temporal with evidence |++### Using Methods++```python+from hyperextract import Template++# Create from method+ka = Template.create("method/light_rag")++# Or via path+ka = Template.create("method/graph_rag")+```++### Template vs Method++| Aspect | Template | Method |+|--------|----------|--------|+| **Ease of use** | ⭐⭐⭐ | ⭐⭐ |+| **Flexibility** | ⭐⭐ | ⭐⭐⭐ |+| **Domain fit** | ⭐⭐⭐ | ⭐⭐ |+| **Configuration** | Minimal | More |+| **Language** | Multi | English |++---++## The Extraction Pipeline++### 1. Text Input++```python+text = "Your document content here..."+```++### 2. Chunking (if needed)++Long documents are automatically split:+- Default chunk size: 2048 characters+- Overlap: 256 characters+- Parallel processing with multiple workers++### 3. LLM Extraction++Each chunk is processed by the LLM:+- Structured output using Pydantic schemas+- Concurrent processing for speed+- Merge results from all chunks++### 4. Result++```python+result = ka.parse(text)++# result contains:+# - result.data: Extracted knowledge+# - result.metadata: Extraction info+# - Methods for search, chat, save+```++---++## Putting It All Together++### Template Approach (Recommended)++```python+from hyperextract import Template++# 1. Choose template+ka = Template.create("general/biography_graph", "en")++# 2. Extract+result = ka.parse(text)++# 3. Use+result.show()+```++### Method Approach (Advanced)++```python+from hyperextract import Template++# 1. Choose method+ka = Template.create("method/light_rag")++# 2. Extract+result = ka.parse(text)++# 3. Use+result.build_index()+results = result.search("query")+```++### Direct Auto-Type (Full Control)++```python+from hyperextract import AutoGraph+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++# 1. Define schema+from pydantic import BaseModel++class Entity(BaseModel):+ name: str+ type: str++class Relation(BaseModel):+ source: str+ target: str+ type: str++class MyGraph(BaseModel):+ entities: list[Entity]+ relations: list[Relation]++# 2. Create Auto-Type+llm = ChatOpenAI()+emb = OpenAIEmbeddings()++ka = AutoGraph(+ data_schema=MyGraph,+ llm_client=llm,+ embedder=emb+)++# 3. Extract+result = ka.parse(text)+```++---++## Next Steps++- [Using Templates](guides/using-templates.md)+- [Choosing Methods](guides/choosing-methods.md)+- [Working with Auto-Types](guides/working-with-autotypes.md)+- [Auto-Types Reference](../concepts/autotypes.md)diff --git a/docs/en/python/guides/choosing-methods.md b/docs/en/python/guides/choosing-methods.mdnew file mode 100644index 0000000..f96e53a--- /dev/null+++ b/docs/en/python/guides/choosing-methods.md@@ -0,0 +1,282 @@+# Choosing Methods++Understand extraction methods and when to use each.++---++## What Are Methods?++Methods are the underlying algorithms that power knowledge extraction. While templates provide easy-to-use, domain-specific configurations, methods give you direct access to the extraction algorithms.++---++## Method Categories++### RAG-Based Methods++**Retrieval-Augmented Generation** methods excel at processing large documents by combining retrieval with generation.++| Method | Best For | Key Feature |+|--------|----------|-------------|+| `light_rag` | General use | Fast, lightweight |+| `graph_rag` | Large documents | Community detection |+| `hyper_rag` | Complex relationships | N-ary hyperedges |+| `hypergraph_rag` | Advanced scenarios | Enhanced hypergraph |+| `cog_rag` | Reasoning tasks | Cognitive retrieval |++### Typical Methods++**Direct extraction** methods that process text without retrieval.++| Method | Best For | Key Feature |+|--------|----------|-------------|+| `itext2kg` | Quality-focused | High-quality triples |+| `itext2kg_star` | Enhanced quality | Improved iText2KG |+| `kg_gen` | Flexibility | Configurable generation |+| `atom` | Temporal data | Evidence attribution |++---++## Using Methods++### Basic Usage++```python+from hyperextract import Template++# Create from method+ka = Template.create("method/light_rag")++# Extract+result = ka.parse(text)+```++### With Custom Clients++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract import Template++llm = ChatOpenAI(model="gpt-4o")+emb = OpenAIEmbeddings(model="text-embedding-3-large")++ka = Template.create(+ "method/graph_rag",+ llm_client=llm,+ embedder=emb+)+```++---++## Method Selection Guide++### Decision Tree++```mermaid+graph TD+ A[Select Method] --> B{Document size?}+ B -->|Large| C[RAG-Based]+ B -->|Small/Medium| D{Need evidence?}+ D -->|Yes| E[atom]+ D -->|No| F{Prioritize quality?}+ F -->|Yes| G[iText2KG]+ F -->|No| H[kg_gen]++ C --> I{Relationship complexity?}+ I -->|Simple| J[light_rag]+ I -->|Standard| K[graph_rag]+ I -->|Complex| L[hyper_rag]+```++### By Use Case++#### Quick Extraction (Small Documents)++```python+# Fast and simple+ka = Template.create("method/kg_gen")+```++#### High-Quality Results++```python+# Best extraction quality+ka = Template.create("method/itext2kg_star")+```++#### Large Documents++```python+# Efficient processing+ka = Template.create("method/light_rag")+```++#### Complex Relationships++```python+# Multi-entity relationships+ka = Template.create("method/hyper_rag")+```++#### Temporal Analysis++```python+# Time-based with evidence+ka = Template.create("method/atom")+```++---++## RAG vs Typical Comparison++| Feature | RAG-Based | Typical |+|---------|-----------|---------|+| **Document size** | Large (10k+ words) | Small-Medium |+| **Speed** | Slower (retrieval step) | Faster |+| **Memory** | Higher | Lower |+| **Quality** | Good for large docs | Better for small docs |+| **Context handling** | Excellent | Good |+| **Use case** | Books, papers, reports | Articles, summaries |++---++## Method Details++### light_rag++Best for: General-purpose, fast extraction++```python+ka = Template.create("method/light_rag")++# Characteristics:+# - Fastest RAG method+# - Binary edges (source-target)+# - Good balance of speed/quality+```++### graph_rag++Best for: Large documents with community structure++```python+ka = Template.create("method/graph_rag")++# Characteristics:+# - Community detection+# - Hierarchical summaries+# - Best for very large documents+```++### hyper_rag++Best for: Complex multi-entity relationships++```python+ka = Template.create("method/hyper_rag")++# Characteristics:+# - Hyperedges (connect 2+ entities)+# - Captures complex relationships+# - Richer graph structure+```++### itext2kg++Best for: High-quality triple extraction++```python+ka = Template.create("method/itext2kg")++# Characteristics:+# - Optimized for triple quality+# - Iterative refinement+# - Good for knowledge base construction+```++### atom++Best for: Temporal analysis with evidence++```python+ka = Template.create("method/atom")++# Characteristics:+# - Temporal fact extraction+# - Evidence attribution+# - Confidence scoring+```++---++## Listing Methods++```python+from hyperextract import Template+from hyperextract.methods import list_methods++# List all methods+methods = list_methods()+for name, info in methods.items():+ print(f"{name}: {info['description']}")+ print(f" Type: {info['type']}")+```++---++## Method Configuration++Methods can be configured through parameters:++```python+from hyperextract import Template++# Most methods use default configuration+# Advanced users can create method instances directly+from hyperextract.methods import Light_RAG+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++llm = ChatOpenAI()+emb = OpenAIEmbeddings()++ka = Light_RAG(+ llm_client=llm,+ embedder=emb,+ # Method-specific parameters+ chunk_size=1024,+ max_workers=5+)+```++---++## Best Practices++1. **Start with light_rag** — Good default for most cases+2. **Use itext2kg for quality** — When extraction quality is critical+3. **Try hyper_rag for complex data** — When relationships are multi-faceted+4. **Consider atom for temporal data** — When time is important+5. **Benchmark on your data** — Methods perform differently on different content++---++## When to Use Templates vs Methods++| Scenario | Recommendation |+|----------|----------------|+| Quick start | Templates |+| Domain-specific task | Templates |+| Production system | Templates |+| Research/Experimentation | Methods |+| Custom requirements | Methods |+| Performance optimization | Methods |+| Full control | Methods |++---++## See Also++- [Using Templates](using-templates.md)+- [Template Library](../../templates/index.md)+- [Methods Concept Doc](../../concepts/methods.md)diff --git a/docs/en/python/guides/incremental-updates.md b/docs/en/python/guides/incremental-updates.mdnew file mode 100644index 0000000..d6dbdf5--- /dev/null+++ b/docs/en/python/guides/incremental-updates.md@@ -0,0 +1,278 @@+# Incremental Updates++Add new information to existing knowledge bases without reprocessing.++---++## Overview++The `feed_text()` method allows you to incrementally add documents to an existing knowledge base:++1. **Preserves existing data** — Won't overwrite what's already there+2. **Intelligent merging** — Handles duplicates and conflicts+3. **Updates metadata** — Tracks when updates occurred++---++## Basic Usage++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")++# Initial extraction+result = ka.parse(initial_text)+print(f"Initial: {len(result.data.entities)} entities")++# Add more content+result.feed_text(additional_text)+print(f"After feed: {len(result.data.entities)} entities")+```++---++## Use Cases++### Building Knowledge Over Time++```python+ka = Template.create("general/biography_graph", "en")+kb = ka.parse(early_life_text)++# Add career information+kb.feed_text(career_text)++# Add later years+kb.feed_text(later_years_text)++# Final result combines all periods+kb.show()+```++### Processing Multiple Documents++```python+ka = Template.create("general/concept_graph", "en")+kb = ka.parse(documents[0])++for doc in documents[1:]:+ kb.feed_text(doc)+ print(f"Added document, now {len(kb.data.entities)} entities")+```++### Updating with New Information++```python+# Original extraction+kb = ka.parse(original_paper)++# Add corrections/updates+kb.feed_text(corrections)++# Save updated version+kb.dump("./updated_kb/")+```++---++## How Merging Works++### Entity Merging++```python+# If same entity appears in both texts+# Result: Merged descriptions, fields combined+```++### Relation Merging++```python+# If same relation appears+# Result: Updated with latest information+```++### Duplicate Handling++```python+# Exact duplicates are detected and merged+# Near-duplicates (similar names) may create separate entries+```++---++## Best Practices++### 1. Same Template Type++Ensure you're using compatible Auto-Types:++```python+# Good: Same template type+kb = ka.parse(text1) # biography_graph+kb.feed_text(text2) # same type++# Be careful: Mixing types may cause issues+```++### 2. Rebuild Index After Feeding++```python+kb.feed_text(new_text)+kb.build_index() # Required for search/chat+```++### 3. Save Intermediate States++```python+# Save after major updates+kb.feed_text(chapter1)+kb.dump("./kb_v1/")++kb.feed_text(chapter2)+kb.dump("./kb_v2/")+```++### 4. Monitor Growth++```python+initial_count = len(kb.data.entities)+kb.feed_text(new_text)+new_count = len(kb.data.entities)++print(f"Added {new_count - initial_count} new entities")+```++---++## Complete Example++```python+"""Build a knowledge base incrementally from multiple sources."""++from hyperextract import Template+from pathlib import Path++def build_knowledge_base(source_dir, output_dir):+ ka = Template.create("general/biography_graph", "en")++ # Get all text files+ files = sorted(Path(source_dir).glob("*.md"))++ if not files:+ print("No files found")+ return++ # Initial extraction from first file+ print(f"Processing {files[0].name}...")+ kb = ka.parse(files[0].read_text())++ # Feed remaining files+ for file in files[1:]:+ print(f"Adding {file.name}...")+ kb.feed_text(file.read_text())+ print(f" Now {len(kb.data.entities)} entities")++ # Build index for search/chat+ print("Building search index...")+ kb.build_index()++ # Save+ print(f"Saving to {output_dir}...")+ kb.dump(output_dir)++ print("Done!")+ return kb++# Usage+kb = build_knowledge_base("./sources/", "./combined_kb/")+```++---++## Comparison: Parse vs Feed++| Operation | Use When | Result |+|-----------|----------|--------|+| `parse()` | Starting fresh | New knowledge base |+| `feed_text()` | Adding to existing | Updated knowledge base |++### Chaining Operations++```python+# Parse returns new instance+result1 = ka.parse(text1)+result2 = ka.parse(text2) # Independent of result1++# Feed modifies existing+result1.feed_text(text2) # result1 is updated+```++---++## Limitations++### 1. Memory Usage++Large knowledge bases consume memory:++```python+# Monitor size+import sys+size = sys.getsizeof(kb.data)+print(f"Knowledge base size: {size} bytes")+```++### 2. Merge Quality++Merging isn't perfect:+- Similar but not identical entities may not merge+- Very large knowledge bases may slow down++### 3. Index Staleness++Always rebuild after feeding:++```python+kb.feed_text(text)+kb.build_index() # Don't forget!+```++---++## Troubleshooting++### "Memory error"++Process in smaller batches:++```python+for batch in chunks(documents, batch_size=5):+ for doc in batch:+ kb.feed_text(doc)+ kb.dump(f"./kb_checkpoint/") # Save periodically+```++### "Duplicate entities"++Normalize entity names in your text:++```python+# Instead of "Nikola Tesla" and "Tesla"+# Use consistent naming+```++### "Index out of date"++```python+# Forgot to rebuild?+kb.build_index()+```++---++## See Also++- [Saving and Loading](saving-loading.md)+- [Search and Chat](search-and-chat.md)+- [Working with Auto-Types](working-with-autotypes.md)diff --git a/docs/en/python/guides/saving-loading.md b/docs/en/python/guides/saving-loading.mdnew file mode 100644index 0000000..6353d17--- /dev/null+++ b/docs/en/python/guides/saving-loading.md@@ -0,0 +1,347 @@+# Saving and Loading++Persist knowledge bases to disk and restore them later.++---++## Overview++Hyper-Extract provides serialization for:+- **Data** — Extracted entities and relations+- **Metadata** — Extraction settings and timestamps+- **Index** — Vector search index (optional)++---++## Saving Knowledge Bases++### Basic Save++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")+result = ka.parse(text)++# Save to directory+result.dump("./my_kb/")+```++### Save Structure++```+./my_kb/+├── data.json # Extracted knowledge+├── metadata.json # Extraction info+└── index/ # Search index (if built)+ ├── index.faiss+ └── docstore.json+```++### Before Saving++```python+# Build index first (if needed)+result.build_index()++# Then save+result.dump("./my_kb/")+```++---++## Loading Knowledge Bases++### Basic Load++```python+from hyperextract import Template++# Create template (must match original)+ka = Template.create("general/biography_graph", "en")++# Load saved data+ka.load("./my_kb/")++# Use+print(f"Loaded {len(ka.data.entities)} entities")+```++### Verify Loaded Data++```python+ka.load("./my_kb/")++# Check it's not empty+if ka.empty():+ print("Warning: No data loaded")+else:+ print(f"Entities: {len(ka.data.entities)}")+ print(f"Relations: {len(ka.data.relations)}")+```++---++## Use Cases++### Long-term Storage++```python+# Extract and save+ka = Template.create("general/concept_graph", "en")+result = ka.parse(research_paper)+result.build_index()+result.dump("./research_paper_kb/")++# Use weeks later+ka2 = Template.create("general/concept_graph", "en")+ka2.load("./research_paper_kb/")+response = ka2.chat("What are the main findings?")+```++### Sharing Knowledge Bases++```python+# Save to shared location+result.dump("/shared/kb/project_x/")++# Others can load+ka.load("/shared/kb/project_x/")+```++### Backup and Versioning++```python+from datetime import datetime++# Save with timestamp+timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")+result.dump(f"./backups/kb_{timestamp}/")++# Or versioned+result.dump("./kb_v1/")+# ... later updates ...+result.dump("./kb_v2/")+```++---++## Working with the Filesystem++### Checking for Existing KB++```python+from pathlib import Path++kb_path = Path("./my_kb/")++if kb_path.exists() and (kb_path / "data.json").exists():+ print("Knowledge base exists, loading...")+ ka.load(kb_path)+else:+ print("Creating new knowledge base...")+ result = ka.parse(text)+ result.dump(kb_path)+```++### Listing Saved KBs++```python+import os++kb_dirs = [d for d in os.listdir("./") if os.path.isdir(d) and "_kb" in d]+print("Available knowledge bases:", kb_dirs)+```++### Moving/Copying++```python+import shutil++# Copy knowledge base+shutil.copytree("./kb_v1/", "./kb_backup/")++# Move knowledge base+shutil.move("./old_location/", "./new_location/")+```++---++## Metadata++### Accessing Metadata++```python+result.dump("./my_kb/")++# Metadata is saved automatically+# Contents:+# - template: Template used+# - lang: Language+# - created_at: Creation timestamp+# - updated_at: Last update timestamp+```++### Custom Metadata++```python+# Add custom metadata+result.metadata["project"] = "Research Project X"+result.metadata["version"] = "1.0"++result.dump("./my_kb/")+```++---++## Complete Example++```python+"""Manage knowledge bases with save/load."""++from hyperextract import Template+from pathlib import Path+import json++class KnowledgeBaseManager:+ def __init__(self, storage_dir="./knowledge_bases/"):+ self.storage_dir = Path(storage_dir)+ self.storage_dir.mkdir(exist_ok=True)++ def save(self, kb, name):+ """Save knowledge base."""+ kb_path = self.storage_dir / name+ kb.dump(kb_path)+ print(f"Saved to {kb_path}")+ return kb_path++ def load(self, name, template="general/biography_graph", lang="en"):+ """Load knowledge base."""+ kb_path = self.storage_dir / name++ if not kb_path.exists():+ raise FileNotFoundError(f"Knowledge base not found: {name}")++ ka = Template.create(template, lang)+ ka.load(kb_path)+ return ka++ def list(self):+ """List available knowledge bases."""+ return [d.name for d in self.storage_dir.iterdir() if d.is_dir()]++ def info(self, name):+ """Get knowledge base info."""+ kb_path = self.storage_dir / name+ meta_path = kb_path / "metadata.json"++ if meta_path.exists():+ return json.loads(meta_path.read_text())+ return None++# Usage+manager = KnowledgeBaseManager()++# Save+ka = Template.create("general/biography_graph", "en")+result = ka.parse(text)+manager.save(result, "tesla_biography")++# List+print(manager.list()) # ['tesla_biography']++# Load+kb = manager.load("tesla_biography")+print(kb.chat("What did Tesla invent?"))+```++---++## Best Practices++### 1. Match Template on Load++```python+# Save with template X+ka = Template.create("general/biography_graph", "en")++# Load with same template+ka2 = Template.create("general/biography_graph", "en")+ka2.load("./kb/")+```++### 2. Build Index After Loading++```python+ka.load("./kb/")++# Index may need rebuild+if not ka.has_index(): # Check if index loaded+ ka.build_index()+```++### 3. Validate Before Use++```python+try:+ ka.load("./kb/")+ if ka.empty():+ print("Warning: Empty knowledge base")+except FileNotFoundError:+ print("Knowledge base not found")+```++### 4. Use Descriptive Names++```python+# Good+result.dump("./kb/tesla_2024_01_15/")++# Avoid+result.dump("./kb/temp/")+```++---++## Troubleshooting++### "File not found"++```python+from pathlib import Path++kb_path = Path("./my_kb/")+if not kb_path.exists():+ print(f"Directory not found: {kb_path}")+ print(f"Available: {list(Path('.').glob('*/'))}")+```++### "Corrupted data"++```python+# Check data file+import json+data = json.load(open("./my_kb/data.json"))+print(f"Keys: {data.keys()}")+```++### "Index not loaded"++```python+ka.load("./kb/")++# Check if index exists+if (Path("./kb/") / "index").exists():+ print("Index directory exists")+else:+ print("No index, building...")+ ka.build_index()+```++---++## See Also++- [Incremental Updates](incremental-updates.md)+- [Search and Chat](search-and-chat.md)+- [CLI `he parse` command](../../cli/commands/parse.md)diff --git a/docs/en/python/guides/search-and-chat.md b/docs/en/python/guides/search-and-chat.mdnew file mode 100644index 0000000..10c89e3--- /dev/null+++ b/docs/en/python/guides/search-and-chat.md@@ -0,0 +1,273 @@+# Search and Chat++Query your knowledge base using semantic search and conversational AI.++---++## Overview++After extracting knowledge, you can interact with it in two ways:++- **Search** — Find specific entities and relations+- **Chat** — Have natural language conversations++Both require a **search index** to be built first.++---++## Building the Index++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")+result = ka.parse(text)++# Build search index+result.build_index()+```++**Note:** Index building is required before search/chat operations.++---++## Semantic Search++### Basic Search++```python+# Search for relevant items+results = result.search("inventions", top_k=5)++for item in results:+ print(item)+```++### Search Parameters++```python+results = result.search(+ query="electrical engineering achievements",+ top_k=10 # Number of results+)+```++### Working with Results++```python+results = result.search("Nobel Prize")++for item in results:+ # Check item type+ if hasattr(item, 'name'): # Entity+ print(f"Entity: {item.name}")+ elif hasattr(item, 'source'): # Relation+ print(f"Relation: {item.source} -> {item.target}")+```++### Search Use Cases++```python+# Find specific people+people = result.search("scientists who worked with Tesla", top_k=10)++# Find concepts+concepts = result.search("alternating current system", top_k=5)++# Find events+events = result.search("important dates in Tesla's life", top_k=10)+```++---++## Chat Interface++### Single Question++```python+# Ask a question+response = result.chat("What were Tesla's major achievements?")++print(response.content)+```++### Accessing Retrieved Context++```python+response = result.chat("What did Tesla invent?")++print(response.content)++# Access items used to generate response+if "retrieved_items" in response.additional_kwargs:+ items = response.additional_kwargs["retrieved_items"]+ print(f"Based on {len(items)} items")+```++### Chat Parameters++```python+response = result.chat(+ query="Explain the War of Currents",+ top_k=10 # More context for complex questions+)+```++### Chat Use Cases++```python+# Summarization+summary = result.chat("Summarize Tesla's career in 3 sentences")++# Explanation+explanation = result.chat("What is the significance of the Tesla coil?")++# Comparison+comparison = result.chat("How did Tesla's approach differ from Edison's?")++# Timeline+timeline = result.chat("What happened in Tesla's life between 1880-1890?")+```++---++## Search vs Chat++| Feature | Search | Chat |+|---------|--------|------|+| **Returns** | Raw entities/relations | Natural language answer |+| **Speed** | Fast | Slower (LLM call) |+| **Use for** | Finding specific data | Understanding/explaining |+| **Precision** | Exact matches | Interpretive |+| **Output** | Structured | Free text |++### When to Use Each++**Use Search when:**+- You need specific entities+- Building reports or summaries+- Exporting data+- Fast lookup needed++**Use Chat when:**+- Explaining concepts+- Answering complex questions+- Summarizing content+- Interactive exploration++---++## Advanced Patterns++### Search then Chat++```python+# First, search for specific items+items = result.search("wireless technology", top_k=5)++# Then, ask about them+if items:+ context = ", ".join([item.name for item in items if hasattr(item, 'name')])+ response = result.chat(f"Explain the significance of {context}")+ print(response.content)+```++### Iterative Exploration++```python+# Start broad+response = result.chat("What are the main topics in this document?")+print(response.content)++# Drill down+response = result.chat("Tell me more about the Tesla coil")+print(response.content)++# Specific question+response = result.chat("How does the Tesla coil work?")+print(response.content)+```++### Building a Research Assistant++```python+class ResearchAssistant:+ def __init__(self, kb_path):+ self.ka = Template.create("general/concept_graph", "en")+ self.ka.load(kb_path)+ self.ka.build_index()++ def ask(self, question):+ response = self.ka.chat(question)+ return response.content++ def find(self, query, n=5):+ return self.ka.search(query, top_k=n)++ def summarize(self):+ return self.ask("Summarize the main points of this paper")++# Usage+assistant = ResearchAssistant("./paper_kb/")+print(assistant.summarize())+print(assistant.ask("What are the limitations?"))+```++---++## Best Practices++### Search Tips++1. **Use natural language** — "wireless transmission" not "wireless"+2. **Be specific** — "Tesla's patents" not "Tesla"+3. **Increase top_k for broad queries** — `top_k=10` or more+4. **Filter results by type** — Check `hasattr(item, 'name')`++### Chat Tips++1. **Ask clear questions** — Specific questions get better answers+2. **Use context** — Build understanding progressively+3. **Adjust top_k** — Complex questions need more context+4. **Check sources** — Review `retrieved_items` for accuracy++---++## Troubleshooting++### "Index not built"++```python+# Error: Need to build index first+result.build_index()+```++### "No results found"++```python+# Try different phrasing+results = result.search("inventions") # Try synonyms+results = result.search("discoveries")+results = result.search("contributions")++# Increase top_k+results = result.search("Tesla", top_k=20)+```++### "Irrelevant chat responses"++```python+# Increase context+response = result.chat(question, top_k=10)++# Or rephrase question+response = result.chat("Be more specific: ...")+```++---++## See Also++- [Working with Auto-Types](working-with-autotypes.md)+- [Incremental Updates](incremental-updates.md)+- [Saving and Loading](saving-loading.md)diff --git a/docs/en/python/guides/using-templates.md b/docs/en/python/guides/using-templates.mdnew file mode 100644index 0000000..7566868--- /dev/null+++ b/docs/en/python/guides/using-templates.md@@ -0,0 +1,292 @@+# Using Templates++Learn how to use and customize templates for knowledge extraction.++---++## What Are Templates?++Templates are pre-configured extraction setups that combine:+- **Auto-Type** — Output data structure+- **Prompts** — Instructions for the LLM+- **Schema** — Field definitions and types+- **Guidelines** — Extraction rules and constraints++---++## Built-in Templates++Hyper-Extract includes 80+ templates across 6 domains:++| Domain | Templates | Use Cases |+|--------|-----------|-----------|+| `general` | Base types, biography, concept graph | Common extraction tasks |+| `finance` | Earnings, risk, ownership, timeline | Financial analysis |+| `legal` | Contracts, cases, compliance | Legal document processing |+| `medicine` | Anatomy, drugs, treatments | Medical text analysis |+| `tcm` | Herbs, formulas, syndromes | Traditional Chinese Medicine |+| `industry` | Equipment, safety, operations | Industrial documentation |++---++## Creating Templates++### From Preset++```python+from hyperextract import Template++# Basic usage+ka = Template.create("general/biography_graph", language="en")++# With custom clients+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++llm = ChatOpenAI(model="gpt-4o")+emb = OpenAIEmbeddings()++ka = Template.create(+ "general/biography_graph",+ language="en",+ llm_client=llm,+ embedder=emb+)+```++### From File++```python+# Load custom YAML template+ka = Template.create(+ "/path/to/my_template.yaml",+ language="en"+)+```++### From Method++```python+# Use extraction method instead of template+ka = Template.create("method/light_rag")+```++---++## Listing Templates++### All Templates++```python+from hyperextract import Template++templates = Template.list()++for path, cfg in templates.items():+ print(f"{path}: {cfg.description}")+```++### Filtered Listing++```python+# By type+graphs = Template.list(filter_by_type="graph")++# By tag+biography = Template.list(filter_by_tag="biography")++# By language+zh_templates = Template.list(filter_by_language="zh")++# Combined+results = Template.list(+ filter_by_type="graph",+ filter_by_language="en"+)+```++### Search Templates++```python+# Search in name and description+results = Template.list(filter_by_query="financial")+```++---++## Template Configuration++### Getting Template Info++```python+from hyperextract import Template++# Get template configuration+cfg = Template.get("general/biography_graph")++print(cfg.name) # biography_graph+print(cfg.type) # temporal_graph+print(cfg.description) # Template description+```++### Template Properties++```python+cfg = Template.get("general/biography_graph")++# Properties+print(cfg.name) # Template name+print(cfg.type) # Auto-Type (graph, list, etc.)+print(cfg.description) # Human-readable description+print(cfg.tags) # Associated tags+print(cfg.language) # Supported languages+```++---++## Common Template Patterns++### Pattern 1: Biography Analysis++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")++with open("biography.md") as f:+ result = ka.parse(f.read())++# Access timeline data+for relation in result.data.relations:+ if hasattr(relation, 'time'):+ print(f"{relation.time}: {relation.source} -> {relation.target}")++result.show() # Visualize life timeline+```++### Pattern 2: Financial Report++```python+from hyperextract import Template++ka = Template.create("finance/earnings_summary", "en")++report = ka.parse(earnings_text)++# Access financial data+print(f"Revenue: {report.data.revenue}")+print(f"EPS: {report.data.eps}")+print(f"Growth: {report.data.yoy_growth}%")+```++### Pattern 3: Legal Contract++```python+from hyperextract import Template++ka = Template.create("legal/contract_obligation", "en")++contract = ka.parse(contract_text)++# List obligations+for obligation in contract.data.obligations:+ print(f"Party: {obligation.party}")+ print(f"Obligation: {obligation.description}")+ print(f"Deadline: {obligation.deadline}")+```++### Pattern 4: Research Paper++```python+from hyperextract import Template++ka = Template.create("general/concept_graph", "en")++paper = ka.parse(paper_text)++# Build searchable knowledge base+paper.build_index()++# Query findings+response = paper.chat("What are the main contributions?")+print(response.content)+```++---++## Multi-Language Support++Templates support both English and Chinese:++```python+# English document+ka_en = Template.create("general/biography_graph", "en")+result_en = ka_en.parse(english_text)++# Chinese document+ka_zh = Template.create("general/biography_graph", "zh")+result_zh = ka_zh.parse(chinese_text)+```++**Note:** Use the language that matches your document for best results.++---++## Template Caching++Templates are loaded and cached for efficiency:++```python+# First call loads template+ka1 = Template.create("general/biography_graph", "en")++# Second call uses cached version+ka2 = Template.create("general/biography_graph", "en")++# Both are independent instances with same configuration+```++---++## Error Handling++### Template Not Found++```python+from hyperextract import Template++try:+ ka = Template.create("nonexistent/template")+except ValueError as e:+ print(f"Template not found: {e}")++ # List available+ available = Template.list()+ print("Available templates:", list(available.keys()))+```++### Language Not Supported++```python+cfg = Template.get("general/biography_graph")+print(cfg.language) # Check supported languages++# Will raise error if language not supported+ka = Template.create("general/biography_graph", "fr") # May fail+```++---++## Best Practices++1. **Choose domain-specific templates** — Better results than generic ones+2. **Match language to document** — Improves extraction quality+3. **Cache template instances** — Reuse for multiple documents+4. **Check template output schema** — Know what fields to expect++---++## See Also++- [Template Library](../../templates/index.md) — Browse all templates+- [Creating Custom Templates](custom-templates.md) — Write your own+- [Choosing Methods](choosing-methods.md) — When to use methods insteaddiff --git a/docs/en/python/guides/working-with-autotypes.md b/docs/en/python/guides/working-with-autotypes.mdnew file mode 100644index 0000000..ca67388--- /dev/null+++ b/docs/en/python/guides/working-with-autotypes.md@@ -0,0 +1,319 @@+# Working with Auto-Types++Extract, manipulate, and utilize structured knowledge data.++---++## What Are Auto-Types?++Auto-Types are intelligent data structures that automatically extract and organize knowledge from text. They provide:++- **Type-safe schemas** — Pydantic-based validation+- **LLM-powered extraction** — Automatic content processing+- **Built-in operations** — Search, merge, visualize+- **Serialization** — Save/load to disk++---++## The 8 Auto-Types++### Scalar Types++#### AutoModel++Single structured object extraction.++```python+from hyperextract import Template++ka = Template.create("finance/earnings_summary", "en")+result = ka.parse(report_text)++# Access fields directly+print(result.data.company_name)+print(result.data.revenue)+print(result.data.eps)+```++#### AutoList++Ordered collection extraction.++```python+from hyperextract import Template++ka = Template.create("general/base_list", "en")+result = ka.parse(text)++# Iterate over items+for item in result.data.items:+ print(item)++# Access by index+first = result.data.items[0]+```++#### AutoSet++Deduplicated collection extraction.++```python+from hyperextract import Template++ka = Template.create("general/base_set", "en")+result = ka.parse(text)++# Items are unique+for item in result.data.items:+ print(item)+```++### Graph Types++#### AutoGraph++Entity-relationship graph extraction.++```python+from hyperextract import Template++ka = Template.create("general/knowledge_graph", "en")+result = ka.parse(text)++# Access entities+for entity in result.data.entities:+ print(f"{entity.name} ({entity.type})")+ print(f" Description: {entity.description}")++# Access relations+for relation in result.data.relations:+ print(f"{relation.source} --{relation.type}--> {relation.target}")+```++#### AutoHypergraph++Multi-entity relationship extraction.++```python+from hyperextract import Template++ka = Template.create("general/base_hypergraph", "en")+result = ka.parse(text)++# Hyperedges connect multiple entities+for hyperedge in result.data.hyperedges:+ print(f"Type: {hyperedge.type}")+ print(f"Entities: {hyperedge.entities}")+```++#### AutoTemporalGraph++Time-based graph extraction.++```python+from hyperextract import Template++ka = Template.create("general/base_temporal_graph", "en")+result = ka.parse(text)++# Relations include time information+for relation in result.data.relations:+ print(f"{relation.source} --{relation.type}--> {relation.target}")+ if hasattr(relation, 'time'):+ print(f" Time: {relation.time}")+```++#### AutoSpatialGraph++Location-based graph extraction.++```python+from hyperextract import Template++ka = Template.create("general/base_spatial_graph", "en")+result = ka.parse(text)++# Entities/relation include location+for entity in result.data.entities:+ if hasattr(entity, 'location'):+ print(f"{entity.name} at {entity.location}")+```++#### AutoSpatioTemporalGraph++Combined time and space extraction.++```python+from hyperextract import Template++ka = Template.create("general/base_spatio_temporal_graph", "en")+result = ka.parse(text)++# Full context: who, what, when, where+for relation in result.data.relations:+ print(f"Event: {relation.type}")+ if hasattr(relation, 'time'):+ print(f" When: {relation.time}")+ if hasattr(relation, 'location'):+ print(f" Where: {relation.location}")+```++---++## Common Operations++### Checking if Empty++```python+if result.empty():+ print("No data extracted")+else:+ print(f"Extracted {len(result.data.entities)} entities")+```++### Clearing Data++```python+# Clear all data+result.clear()++# Clear only index+result.clear_index()+```++### Merging Instances++```python+# Two separate extractions+result1 = ka.parse(text1)+result2 = ka.parse(text2)++# Merge into new instance+combined = result1 + result2+```++---++## Accessing Data++### Property Access++```python+result = ka.parse(text)++# Direct property access (Pydantic model)+entities = result.data.entities+relations = result.data.relations++# Dictionary conversion+data_dict = result.data.model_dump()+```++### JSON Export++```python+import json++# Export to JSON+json_data = result.data.model_dump_json(indent=2)++# Save to file+with open("output.json", "w") as f:+ f.write(json_data)+```++---++## Working with Results++### Iteration Patterns++```python+# Iterate entities+for entity in result.data.entities:+ print(f"Name: {entity.name}")+ print(f"Type: {entity.type}")+ if hasattr(entity, 'description'):+ print(f"Description: {entity.description}")++# Iterate relations with filtering+for relation in result.data.relations:+ if relation.type == "worked_with":+ print(f"{relation.source} worked with {relation.target}")+```++### Filtering++```python+# Filter entities by type+people = [e for e in result.data.entities if e.type == "person"]+organizations = [e for e in result.data.entities if e.type == "organization"]++# Filter relations+inventions = [r for r in result.data.relations if "invent" in r.type.lower()]+```++### Statistics++```python+# Basic counts+entity_count = len(result.data.entities)+relation_count = len(result.data.relations)++# Type distribution+from collections import Counter+entity_types = Counter(e.type for e in result.data.entities)+relation_types = Counter(r.type for r in result.data.relations)++print(f"Entities: {entity_types}")+print(f"Relations: {relation_types}")+```++---++## Advanced Usage++### Custom Schema Access++```python+# Access the schema+schema = result.data_schema++print(schema.model_fields) # Available fields+```++### Raw Data Access++```python+# Access internal data if needed+internal_data = result._data+```++### Type Checking++```python+from hyperextract import AutoGraph, AutoList++# Check instance type+if isinstance(result, AutoGraph):+ print("Graph extraction")+elif isinstance(result, AutoList):+ print("List extraction")+```++---++## Best Practices++1. **Check hasattr before optional fields** — Schema fields may vary+2. **Handle empty results** — Always check `empty()`+3. **Use model_dump for serialization** — Proper JSON conversion+4. **Leverage type hints** — IDE support for autocomplete++---++## See Also++- [Search and Chat](search-and-chat.md)+- [Saving and Loading](saving-loading.md)+- [Auto-Types Reference](../../concepts/autotypes.md)diff --git a/docs/en/python/index.md b/docs/en/python/index.mdindex 8ae8ce7..c1f525f 100644--- a/docs/en/python/index.md+++ b/docs/en/python/index.md@@ -1,6 +1,8 @@# Python SDK-Python API for Hyper-Extract.+Build knowledge extraction pipelines with the Hyper-Extract Python API.++---## Installation@@ -8,48 +10,226 @@ Python API for Hyper-Extract.pip install hyper-extract```-## Quick Start+For development:++```bash+pip install hyper-extract[dev]+```++---++## Quick Example```python-from dotenv import load_dotenv+from hyperextract import Template++# Create template+ka = Template.create("general/biography_graph", language="en")-# Load environment variables from .env file-load_dotenv()+# Extract knowledge+with open("document.md") as f:+ result = ka.parse(f.read())+# Access data+print(f"Entities: {len(result.data.entities)}")+print(f"Relations: {len(result.data.relations)}")++# Visualize+result.show()+```++---++## Core Classes++### Template++The main entry point for template-based extraction.++```pythonfrom hyperextract import Template-# Create a template-ka = Template.create("general/biography_graph", "en")+# Create from preset+ka = Template.create("general/biography_graph", language="en")-# Parse a document-with open("examples/en/tesla.md", "r") as f:- text = f.read()-result = ka.parse(text)+# Create from custom YAML+ka = Template.create("/path/to/custom_template.yaml", language="en")++# Create from method+ka = Template.create("method/light_rag")+```++### Auto-Types++Eight data structure types for different extraction needs:++| Class | Use Case |+|-------|----------|+| `AutoModel` | Structured data models |+| `AutoList` | Ordered collections |+| `AutoSet` | Deduplicated collections |+| `AutoGraph` | Entity-relationship graphs |+| `AutoHypergraph` | Multi-entity relationships |+| `AutoTemporalGraph` | Time-based relationships |+| `AutoSpatialGraph` | Location-based relationships |+| `AutoSpatioTemporalGraph` | Combined time + space |++---++## API Overview++### Template API++```python+# Create+ka = Template.create(template_path, language="en")++# Extract+result = ka.parse(text) # New extraction+result.feed_text(text) # Add to existing++# Query+result.build_index() # Build search index+results = result.search(query) # Semantic search+response = result.chat(query) # Chat with knowledge++# Persist+result.dump("./output/") # Save to disk+result.load("./output/") # Load from disk++# Visualize+result.show() # Interactive visualization+```++---++## Documentation Structure++- **[Quickstart](quickstart.md)** — Your first Python extraction+- **[Core Concepts](core-concepts.md)** — Template, AutoType, Method explained+- **Guides:**+ - [Using Templates](guides/using-templates.md)+ - [Choosing Methods](guides/choosing-methods.md)+ - [Working with Auto-Types](guides/working-with-autotypes.md)+ - [Search and Chat](guides/search-and-chat.md)+ - [Incremental Updates](guides/incremental-updates.md)+ - [Saving and Loading](guides/saving-loading.md)+- **API Reference:**+ - [Template](api-reference/template.md)+ - [Auto-Types](api-reference/autotypes.md)+ - [Methods](api-reference/methods.md)++---++## Examples by Use Case++### Research Paper Analysis++```python+from hyperextract import Template-# Access results-print(result.entities)-print(result.relations)+ka = Template.create("general/concept_graph", language="en")++with open("paper.md") as f:+ paper = ka.parse(f.read())++# Build searchable knowledge base+paper.build_index()++# Ask questions+response = paper.chat("What are the main contributions?")+print(response.content)```-## Core APIs+### Financial Report Extraction++```python+from hyperextract import Template-| API | Purpose |-|-----|---------|-| `Template.create()` | Create template instance |-| `ka.parse()` | Parse document |-| `ka.feed()` | Incrementally supplement |-| `ka.search()` | Query knowledge base |+ka = Template.create("finance/earnings_summary", language="en")++report = ka.parse(earnings_text)+print(report.data.revenue)+print(report.data.eps)+```++### Building a Knowledge Base++```python+from hyperextract import Template++ka = Template.create("general/knowledge_graph", language="en")++# Initial extraction+kb = ka.parse(doc1_text)++# Add more documents+kb.feed_text(doc2_text)+kb.feed_text(doc3_text)++# Save for later+kb.dump("./my_kb/")+```++---## Configuration-[→ Python SDK Configuration Guide](./config.md)+### Environment Variables++```python+import os++os.environ["OPENAI_API_KEY"] = "your-api-key"+os.environ["OPENAI_BASE_URL"] = "https://api.openai.com/v1"+```++### Using .env File++```python+from dotenv import load_dotenv+load_dotenv() # Loads from .env file+```++### Custom Clients++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract import Template++llm = ChatOpenAI(model="gpt-4o")+embedder = OpenAIEmbeddings(model="text-embedding-3-large")-## Detailed Guide+ka = Template.create(+ "general/biography_graph",+ language="en",+ llm_client=llm,+ embedder=embedder+)+```++---++## Type Hints++Hyper-Extract is fully typed for IDE support:++```python+from hyperextract import Template, AutoGraph++ka: AutoGraph = Template.create("general/knowledge_graph", "en")+result = ka.parse(text)++# IDE autocomplete works+entities = result.data.entities+relations = result.data.relations+```-[→ Python API Details](./python-api.md)+----## Usage Examples+## Next Steps-- [Finance Domain Templates](../templates/finance.md)-- [Legal Domain Templates](../templates/legal.md)-- [Medical Domain Templates](../templates/medicine.md)+- Follow the [Quickstart](quickstart.md)+- Learn [Core Concepts](core-concepts.md)+- Browse [Guides](guides/using-templates.md)+- Read [API Reference](api-reference/template.md)diff --git a/docs/en/python/python-api.md b/docs/en/python/python-api.mddeleted file mode 100644index ba351ee..0000000--- a/docs/en/python/python-api.md+++ /dev/null@@ -1,280 +0,0 @@-# Python API--## Installation--```bash-pip install hyperextract-```--## Configuration--Set up your API key using dotenv:--```python-from dotenv import load_dotenv-load_dotenv()-```--Create a `.env` file:-```bash-OPENAI_API_KEY=your-api-key-```--## Quick Start--```python-from hyperextract import Template--ka = Template.create("general/biography_graph", "en")-result = ka.parse(document_text)-```-------## AutoTypes--AutoTypes define the **output data structure** for knowledge extraction. There are 8 types:--### AutoModel - Structured Data--Extract structured summaries with AutoModel:--```python-from pydantic import BaseModel, Field-from hyperextract import AutoModel-from langchain_openai import ChatOpenAI--class BiographySummary(BaseModel):- title: str = Field(description="Title of the biography")- subject: str = Field(description="Main subject name")- summary: str = Field(description="Brief summary")--llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)--model = AutoModel(- data_schema=BiographySummary,- llm_client=llm,-)-model.feed_text(text)-print(model.data.title)-```--### AutoList - List Data--Extract lists of items:--```python-from hyperextract.types import AutoList--llm = ChatOpenAI(model="gpt-4o-mini")-embedder = OpenAIEmbeddings(model="text-embedding-3-small")--lst = AutoList[str](- item_schema=str,- llm_client=llm,- embedder=embedder,-)-lst.feed_text(text)-print(lst.items) # List of extracted items-```--### AutoSet - Deduplicated List--Extract unique items (deduplicated):--```python-from hyperextract.types import AutoSet--s = AutoSet[str](- item_schema=str,- llm_client=llm,- embedder=embedder,-)-s.feed_text(text)-print(s.items) # Deduplicated list-```--### AutoGraph - Knowledge Graph--Extract entities and relationships:--```python-from pydantic import BaseModel, Field-from hyperextract.types import AutoGraph--class Entity(BaseModel):- name: str = Field(description="Entity name")- type: str = Field(description="Entity type")--class Relation(BaseModel):- source: str = Field(description="Source entity")- target: str = Field(description="Target entity")- type: str = Field(description="Relation type")--graph = AutoGraph[Entity, Relation](- node_schema=Entity,- edge_schema=Relation,- node_key_extractor=lambda x: x.name,- edge_key_extractor=lambda x: f"{x.source}-{x.type}-{x.target}",- llm_client=llm,- embedder=embedder,-)-graph.feed_text(text)-print(f"Nodes: {len(graph.nodes)}, Edges: {len(graph.edges)}")-```--### AutoHypergraph - Hypergraph--For multi-entity relationships (n-ary relations):--```python-from hyperextract.types import AutoHypergraph-# See examples/en/autotypes/hypergraph_demo.py-```--### AutoTemporalGraph - Temporal Graph--For time-based relationships:--```python-from hyperextract.types import AutoTemporalGraph-# See examples/en/autotypes/temporal_graph_demo.py-```--### AutoSpatialGraph - Spatial Graph--For location-based relationships:--```python-from hyperextract.types import AutoSpatialGraph-# See examples/en/autotypes/spatial_graph_demo.py-```--### AutoSpatioTemporalGraph - Spatio-Temporal Graph--For both time and location:--```python-from hyperextract.types import AutoSpatioTemporalGraph-# See examples/en/autotypes/spatio_temporal_graph_demo.py-```-------## Methods vs Templates--Hyper-Extract provides two ways to extract knowledge:--### Methods (Low-level)--Methods give you **full control** over the extraction process. Use them when you need:-- Custom extraction logic-- Specific algorithm tuning-- Advanced use cases--```python-from hyperextract.methods.typical import KG_Gen--ka = KG_Gen(llm_client=llm, embedder=embedder)-ka.feed_text(text)-result = ka.chat("What are the main achievements?")-```--Available Methods:-- **Typical**: KG_Gen, iText2KG, iText2KG*-- **RAG-Based**: GraphRAG, LightRAG, HyperRAG, HypergraphRAG, CogRAG--### Templates (High-level)--Templates are **pre-configured** schemas for common use cases. Use them when you want:-- Quick results-- Domain-specific extraction-- Zero-code setup--```python-from hyperextract import Template--ka = Template.create("finance/earnings_summary", "en")-result = ka.parse(text)-```--### When to Use Which?--| Scenario | Recommendation |-|----------|----------------|-| Quick prototyping | Templates |-| Domain-specific extraction | Templates |-| Custom extraction logic | Methods |-| Real-time interaction | Methods |-| Batch processing | Both work |-------## Advanced Usage--### Batch Processing--```python-from hyperextract import Template--ka = Template.create("general/biography_graph", "en")-documents = ["doc1.txt", "doc2.txt", "doc3.txt"]--results = []-for doc in documents:- with open(doc) as f:- result = ka.parse(f.read())- results.append(result)-```--### Incremental Supplement (feed)--```python-ka = Template.create("general/biography_graph", "en")-result = ka.parse(text1)--# Add more knowledge-ka.feed(result, text2)-```--### Knowledge Q&A (chat)--```python-ka.build_index()-answer = ka.chat("What are the key events?")-print(answer.content)-```--### Error Handling--```python-from hyperextract import Template--try:- ka = Template.create("template_name", "en")- result = ka.parse(text)-except Exception as e:- print(f"Error: {e}")- # Handle error-```--### Result Export--```python-# Export to JSON-json_output = result.to_json()--# Export to Dict-dict_output = result.to_dict()--# Export to Triples-triples = result.to_triples()-```-------## Next Steps--- [Preset Templates](../templates/index.md)-- [Domain Templates](../templates/index.md)-- [Explore Examples](https://github.com/yifanfeng97/hyper-extract/tree/main/examples/en)diff --git a/docs/en/python/quickstart.md b/docs/en/python/quickstart.mdnew file mode 100644index 0000000..88e1a62--- /dev/null+++ b/docs/en/python/quickstart.md@@ -0,0 +1,173 @@+# Python Quickstart++Get started with the Hyper-Extract Python SDK in 5 minutes.++---++## Prerequisites++- Python 3.11++- OpenAI API key++## Installation++```bash+pip install hyper-extract+```++## Basic Usage++### 1. Configure API Key++```python+import os+os.environ["OPENAI_API_KEY"] = "your-api-key"+```++Or use a `.env` file:++```python+from dotenv import load_dotenv+load_dotenv()+```++### 2. Extract Knowledge++```python+from hyperextract import Template++# Create a template+ka = Template.create("general/biography_graph", language="en")++# Your text+text = """+Marie Curie was a Polish-French physicist and chemist who conducted+pioneering research on radioactivity. She was the first woman to win+a Nobel Prize and the only person to win Nobel Prizes in two different+scientific fields.+"""++# Extract+result = ka.parse(text)++# Access results+print(f"Entities: {len(result.data.entities)}")+print(f"Relations: {len(result.data.relations)}")++# Print first entity+if result.data.entities:+ entity = result.data.entities[0]+ print(f"\nFirst: {entity.name} ({entity.type})")+ print(f"Description: {entity.description}")+```++**Output:**+```+Entities: 5+Relations: 4++First: Marie Curie (person)+Description: Polish-French physicist and chemist+```++### 3. Visualize++```python+# Open interactive visualization+result.show()+```++### 4. Search++```python+# Build search index+result.build_index()++# Search+results = result.search("Nobel Prize", top_k=3)+for item in results:+ print(item)+```++### 5. Chat++```python+# Ask questions+response = result.chat("What did Marie Curie discover?")+print(response.content)+```++### 6. Save and Load++```python+# Save to disk+result.dump("./curie_kb/")++# Load later+new_ka = Template.create("general/biography_graph", language="en")+new_ka.load("./curie_kb/")+```++---++## Complete Example++```python+"""Complete example: Extract, explore, and save knowledge."""++from dotenv import load_dotenv+load_dotenv()++from hyperextract import Template++def main():+ # Create template+ print("Creating template...")+ ka = Template.create("general/biography_graph", language="en")++ # Sample text+ text = """+ Ada Lovelace was an English mathematician and writer, chiefly known+ for her work on Charles Babbage's early mechanical general-purpose+ computer, the Analytical Engine. She is often regarded as the first+ computer programmer.+ """++ # Extract knowledge+ print("Extracting knowledge...")+ result = ka.parse(text)++ # Display results+ print(f"\nExtraction Results:")+ print(f" Entities: {len(result.data.entities)}")+ print(f" Relations: {len(result.data.relations)}")++ # List entities+ print("\nEntities found:")+ for entity in result.data.entities:+ print(f" - {entity.name} ({entity.type})")++ # Build index and search+ print("\nBuilding search index...")+ result.build_index()++ search_results = result.search("computer programming", top_k=2)+ print(f"\nSearch results: {len(search_results)} items")++ # Save+ print("\nSaving knowledge base...")+ result.dump("./ada_kb/")++ print("\nDone! Try: he show ./ada_kb/")++if __name__ == "__main__":+ main()+```++---++## Next Steps++- Learn about [Core Concepts](core-concepts.md)+- Read [Using Templates Guide](guides/using-templates.md)+- Explore [API Reference](api-reference/template.md)diff --git a/docs/en/resources/contributing.md b/docs/en/resources/contributing.mdindex 60793bc..84b4556 100644--- a/docs/en/resources/contributing.md+++ b/docs/en/resources/contributing.md@@ -1,115 +1,211 @@# Contributing-Thank you for your interest in contributing to Hyper-Extract!+How to contribute to Hyper-Extract.++---## Ways to Contribute-### Report Bugs+- **Report bugs** — Open GitHub issues+- **Request features** — Suggest new functionality+- **Improve documentation** — Fix typos, add examples+- **Add templates** — Share domain-specific templates+- **Submit code** — Fix bugs or add features-Report bugs by opening a GitHub issue with:-- Clear description of the bug-- Steps to reproduce-- Expected vs actual behavior-- Environment information (Python version, OS, etc.)+----### Suggest Features+## Getting Started-We welcome feature suggestions! Please:-1. Check existing issues to avoid duplicates-2. Describe the use case clearly-3. Explain the expected behavior+### 1. Fork the Repository-### Contribute Code+```bash+git clone https://github.com/your-username/hyper-extract.git+cd hyper-extract+```-#### Development Setup+### 2. Set Up Development Environment-1. Fork the repository-2. Clone your fork:- ```bash- git clone https://github.com/yifanfeng97/hyper-extract.git- cd hyper-extract- ```+```bash+# Create virtual environment+python -m venv venv+source venv/bin/activate # Windows: venv\Scripts\activate-3. Create a virtual environment:- ```bash- python -m venv env- source env/bin/activate # Windows: env\Scripts\activate- ```+# Install in editable mode+pip install -e ".[dev]"+```-4. Install development dependencies:- ```bash- pip install -e ".[dev]"- ```+### 3. Run Tests-5. Install pre-commit hooks:- ```bash- pre-commit install- ```+```bash+pytest+```-#### Development Workflow+----1. Create a feature branch:- ```bash- git checkout -b feature/your-feature-name- ```+## Development Guidelines-2. Make your changes-3. Run tests:- ```bash- pytest tests/- ```+### Code Style-4. Run linting:- ```bash- ruff check .- ```+- Follow PEP 8+- Use type hints+- Write docstrings (Google style)++### Example++```python+def process_data(text: str, max_length: int = 1000) -> dict:+ """Process input text and return structured data.++ Args:+ text: Input text to process+ max_length: Maximum length to process++ Returns:+ Dictionary containing processed data++ Raises:+ ValueError: If text is empty+ """+ if not text:+ raise ValueError("Text cannot be empty")++ # Processing logic+ return {"result": text[:max_length]}+```++### Testing++Write tests for new features:++```python+def test_new_feature():+ result = new_feature("input")+ assert result["status"] == "success"+```++Run tests:+```bash+pytest tests/test_new_feature.py -v+```++---++## Adding Templates++### Template Structure-5. Commit your changes:+Create a YAML file in `hyperextract/templates/presets/<domain>/`:++```yaml+language: [zh, en]++name: my_template+type: graph+tags: [domain, category]++description:+ zh: "中文描述"+ en: "English description"++output:+ # Schema definition++guideline:+ # LLM instructions++identifiers:+ # ID rules++display:+ # Visualization settings+```++### Testing Templates++```python+from hyperextract import Template++# Test your template+ka = Template.create("domain/my_template", "en")+result = ka.parse(test_text)++# Verify output+assert len(result.data.entities) > 0+```++### Submitting Templates++1. Add template YAML file+2. Add test case+3. Update documentation+4. Submit PR with description++---++## Documentation++### Building Docs++```bash+# Install docs dependencies+pip install mkdocs mkdocs-material mkdocstrings[python]++# Serve locally+mkdocs serve++# Build+mkdocs build+```++### Documentation Guidelines++- Write clear, concise instructions+- Include code examples+- Test all examples+- Use markdown formatting++---++## Pull Request Process++1. **Create a branch**:```bash- git commit -m "Add your feature"+ git checkout -b feature/my-feature```-6. Push and create a Pull Request+2. **Make changes** with clear commit messages-#### Code Style--- Follow PEP 8-- Use type hints-- Write docstrings-- Add tests for new features+3. **Add tests** for new functionality-### Contribute Documentation+4. **Update documentation** as needed-Documentation improvements are always welcome:-- Fix typos or unclear explanations-- Add examples-- Improve translations-- Update outdated content+5. **Submit PR** with:+ - Clear description+ - What changed and why+ - Testing performed+ - Screenshots (if UI changes)-### Translation Contributions+----We welcome translation contributions:-1. Check the existing translations-2. Submit translations via PR+## Code Review-## Pull Request Guidelines+All submissions require review. We'll check:-1. Reference related issues-2. Include tests for new features-3. Update documentation as needed-4. Follow the code style guidelines-5. Ensure all tests pass+- Code quality and style+- Test coverage+- Documentation+- Backward compatibility-## Development Resources+----- [GitHub Repository](https://github.com/yifanfeng97/hyper-extract)-- [Issue Tracker](https://github.com/yifanfeng97/hyper-extract/issues)-- [Discussions](https://github.com/yifanfeng97/hyper-extract/discussions)+## Questions?-## Code of Conduct+- Open a [GitHub Discussion](https://github.com/yifanfeng97/hyper-extract/discussions)+- Comment on existing issues+- Email: [email protected]-Please be respectful and constructive in all interactions. We follow the [Contributor Covenant](https://www.contributor-covenant.org/).+---## License-By contributing, you agree that your contributions will be licensed under the Apache 2.0 License.+By contributing, you agree that your contributions will be licensed under the Apache-2.0 License.diff --git a/docs/en/resources/faq.md b/docs/en/resources/faq.mdindex 683d32e..35eb182 100644--- a/docs/en/resources/faq.md+++ b/docs/en/resources/faq.md@@ -1,106 +1,262 @@-# FAQ+# Frequently Asked Questions-Frequently asked questions about Hyper-Extract.+Common questions about Hyper-Extract.++---## General### What is Hyper-Extract?-Hyper-Extract is an intelligent, LLM-powered knowledge extraction framework that transforms unstructured documents into structured knowledge representations.+Hyper-Extract is an LLM-powered knowledge extraction framework that transforms unstructured text into structured knowledge graphs, lists, models, and more.++### What can I use it for?-### What types of documents can Hyper-Extract process?+- Research paper analysis+- Knowledge base construction+- Document processing+- Information extraction+- Question-answering systems-Hyper-Extract can process various document types including:-- Plain text files-- PDFs-- Markdown files-- HTML documents-- Word documents+### Is it free?-### What languages are supported?+The software is open-source (Apache-2.0). You need to provide your own OpenAI API key for LLM calls.-Hyper-Extract supports multiple languages including English, Chinese, Japanese, Korean, and more.+---## Installation-### How do I install Hyper-Extract?+### What are the requirements?++- Python 3.11++- OpenAI API key++### How do I install it?++```bash+pip install hyper-extract+```++### Installation fails with "No module named 'hyperextract'"++Try:+```bash+pip install --upgrade hyper-extract+```+Or use a virtual environment:```bash+python -m venv venv+source venv/bin/activate # Windows: venv\Scripts\activatepip install hyper-extract```-### What are the system requirements?+---++## Configuration++### Where do I set my API key?++**Option 1**: CLI+```bash+he config init -k YOUR_API_KEY+```-- Python 3.9 or higher-- OpenAI API key (or compatible LLM API)+**Option 2**: Environment variable+```bash+export OPENAI_API_KEY=your-api-key+```++**Option 3**: .env file+```+OPENAI_API_KEY=your-api-key+```++### Can I use a different LLM provider?++Yes, set the base URL:+```bash+he config set llm.base_url https://your-provider.com/v1+```++### Which models are supported?++- OpenAI models (gpt-4o, gpt-4o-mini, etc.)+- Any OpenAI-compatible API++---## Usage-### How do I extract knowledge from a document?+### Which template should I use?++See the [How to Choose](../templates/how-to-choose.md) guide or use:+```bash+he list template+```++### How do I process a PDF?-Using CLI:+Convert to text first:```bash-he parse document.txt -o output/+pdftotext document.pdf document.txt+he parse document.txt -t general/knowledge_graph -l en```-Using Python API:+### Can I process multiple documents?++**Option 1**: Feed incrementally+```bash+he parse doc1.md -t general/graph -o ./kb/ -l en+he feed ./kb/ doc2.md+he feed ./kb/ doc3.md+```++**Option 2**: Process directory+```bash+he parse ./docs/ -t general/graph -o ./kb/ -l en+```++### How do I extract in Chinese?++```bash+he parse doc.md -t general/biography_graph -l zh+```++---++## Performance++### Why is extraction slow?++- Long documents are chunked and processed in parallel+- Each chunk requires an LLM call+- Consider using `--no-index` during batch processing++### How can I speed it up?++1. Use smaller chunk sizes+2. Reduce `max_workers` if hitting rate limits+3. Process documents in parallel (manually)++### Memory issues with large documents?++Process in smaller batches:```python-from hyperextract import Template-ka = Template.create("general/biography")-result = ka.parse(document_text)+for batch in chunks(documents, 5):+ for doc in batch:+ kb.feed_text(doc)+ kb.dump("./checkpoint/")+```++---++## Results++### Where is my data stored?++```+./output/+├── data.json # Extracted knowledge+├── metadata.json # Extraction info+└── index/ # Search index+```++### How do I visualize results?++```bash+he show ./output/+```++Or in Python:+```python+result.show()```-### How do I choose the right template?+### Can I export to other formats?++```python+import json-Choose a template based on:-1. Document type (finance, legal, medical, etc.)-2. Extraction goal (entities, relationships, lists, etc.)-3. Output format needed (graph, list, model, etc.)+# To JSON+json_data = result.data.model_dump_json()-### How do I create a custom template?+# To dict+data_dict = result.data.model_dump()+```-See the [Domain Templates](../templates/index.md) for creating custom templates.+---## Troubleshooting-### API Key Issues+### "API key not found"++Run:+```bash+he config init -k YOUR_API_KEY+```++### "Template not found"++List available templates:+```bash+he list template+```-**Error: "API key not found"**+### "Index not found" error-Solution: Set your API key using:+Build the index:```bash-he config init -k <your-api-key>+he build-index ./output/```-### Extraction Quality+### Search returns no results-**Low quality extractions**+Try:+- Different search terms+- Increase `top_k`: `he search ./kb/ "query" -n 10`+- Check if index is built: `he info ./kb/`-Tips for better results:-1. Use domain-specific templates-2. Provide clear extraction instructions-3. Use longer context windows-4. Adjust temperature settings+----### Performance+## Advanced-**Slow extraction times**+### Can I create custom templates?-Optimization tips:-1. Use `light_rag` method for faster extraction-2. Batch process multiple documents-3. Use appropriate batch sizes+Yes! See [Custom Templates](../python/guides/custom-templates.md).-## Contributing+### Can I use my own extraction method?-### How can I contribute?+Yes, implement and register:+```python+from hyperextract.methods import register_method++class MyMethod:+ def extract(self, text):+ # Your logic+ pass++register_method("my_method", MyMethod, "graph", "Description")+```++### How do I integrate with my application?++```python+from hyperextract import Template++class MyApp:+ def __init__(self):+ self.ka = Template.create("general/graph", "en")++ def process_document(self, text):+ return self.ka.parse(text)+```-See the [Contributing Guide](contributing.md) for information on how to contribute.+----### How do I report bugs?+## Getting More Help-Please open an issue on our GitHub repository with:-- Detailed description of the bug-- Steps to reproduce-- Expected vs actual behavior-- Environment information+- [GitHub Issues](https://github.com/yifanfeng97/hyper-extract/issues)+- [Troubleshooting Guide](troubleshooting.md)+- [CLI Documentation](../cli/index.md)+- [Python SDK](../python/index.md)diff --git a/docs/en/resources/index.md b/docs/en/resources/index.mdindex 1f9a9c8..d91569f 100644--- a/docs/en/resources/index.md+++ b/docs/en/resources/index.md@@ -1,15 +1,57 @@# Resources-Additional resources and information.+Help and reference materials.-## FAQ+----Frequently asked questions about Hyper-Extract.+## Getting Help-## Troubleshooting+### FAQ-Solutions to common issues.+Common questions and answers.++→ [FAQ](faq.md)++### Troubleshooting++Solutions to common problems.++→ [Troubleshooting](troubleshooting.md)++### GitHub++Report issues, request features, or contribute.++→ [GitHub Repository](https://github.com/yifanfeng97/hyper-extract)++---## Contributing-Guidelines for contributing to the project.+Want to contribute to Hyper-Extract?++→ [Contributing Guide](contributing.md)++---++## Quick Links++- [Getting Started](../getting-started/index.md)+- [CLI Documentation](../cli/index.md)+- [Python SDK](../python/index.md)+- [Template Library](../templates/index.md)+- [Core Concepts](../concepts/index.md)++---++## Community++- **GitHub Discussions**: Ask questions and share ideas+- **Issues**: Report bugs and request features+- **Pull Requests**: Contribute code and documentation++---++## License++Hyper-Extract is licensed under the [Apache-2.0 License](https://github.com/yifanfeng97/hyper-extract/blob/main/LICENSE).diff --git a/docs/en/resources/troubleshooting.md b/docs/en/resources/troubleshooting.mdindex f6199a9..cc00b10 100644--- a/docs/en/resources/troubleshooting.md+++ b/docs/en/resources/troubleshooting.md@@ -1,149 +1,315 @@# Troubleshooting-Common issues and solutions for Hyper-Extract.+Solutions to common issues.++---## Installation Issues-### Package Installation Fails+### pip install fails-**Problem**: `pip install hyper-extract` fails+**Problem**: Installation fails with errors**Solutions**:-1. Upgrade pip:+1. Upgrade pip: `pip install --upgrade pip`+2. Use Python 3.11+: `python --version`+3. Install in virtual environment:```bash- pip install --upgrade pip+ python -m venv venv+ source venv/bin/activate+ pip install hyper-extract```-2. Use a virtual environment:+### ImportError: No module named 'hyperextract'++**Problem**: Can't import after installation++**Solutions**:+1. Check Python version: `python --version` (need 3.11+)+2. Verify installation: `pip list | grep hyper`+3. Check virtual environment is activated+4. Reinstall: `pip install --force-reinstall hyper-extract`++---++## Configuration Issues++### API Key Not Found++**Error**: `No API key configured`++**Solutions**:++1. **CLI Configuration** (recommended):```bash- python -m venv env- source env/bin/activate # On Windows: env\Scripts\activate- pip install hyper-extract+ he config init -k YOUR_API_KEY+ ```++2. **Environment Variable**:+ ```bash+ export OPENAI_API_KEY=your-api-key+ ```++3. **Verify Configuration**:+ ```bash+ he config show+ ```++### Invalid API Key++**Error**: `Authentication failed`++**Solutions**:+1. Verify key is correct+2. Check for extra spaces+3. Try regenerating key in OpenAI dashboard+4. Check if key has available credits++---++## Runtime Issues++### Template Not Found++**Error**: `Template 'xxx' not found`++**Solutions**:++1. **List available templates**:+ ```bash+ he list template+ ```++2. **Check spelling**:+ ```bash+ # Correct+ he parse doc.md -t general/biography_graph++ # Incorrect+ he parse doc.md -t general/biography```-### Dependency Conflicts+3. **Use Python to search**:+ ```python+ from hyperextract import Template+ templates = Template.list(filter_by_query="bio")+ ```-**Problem**: Dependency conflicts with other packages+### Language Required-**Solution**: Create a clean virtual environment:+**Error**: `--lang is required`++**Solution**:```bash-python -m venv hyper-extract-env-source hyper-extract-env/bin/activate-pip install hyper-extract+# Add language flag+he parse doc.md -t general/biography_graph -o ./out/ -l en```-## API Issues+Note: Method templates don't require language.-### Invalid API Key+### Output Directory Exists-**Problem**: "Invalid API key" error+**Error**: `Output directory already exists`**Solutions**:-1. Verify your API key is correct-2. Check key has not expired-3. Ensure sufficient credits/quota-### Rate Limiting+1. **Force overwrite**:+ ```bash+ he parse doc.md -t general/graph -o ./out/ -l en -f+ ```-**Problem**: "Rate limit exceeded" error+2. **Use different directory**:+ ```bash+ he parse doc.md -t general/graph -o ./out2/ -l en+ ```-**Solutions**:-1. Add delay between requests-2. Use batch processing-3. Reduce `max_workers` in configuration+3. **Remove existing**:+ ```bash+ rm -rf ./out/+ he parse doc.md -t general/graph -o ./out/ -l en+ ```++---++## Index and Search Issues++### Index Not Found++**Error**: `Search index not built`++**Solution**:+```bash+he build-index ./output/+```-### Timeout Errors+### Search Returns Empty-**Problem**: Requests timeout+**Problem**: `he search` finds no results**Solutions**:-1. Increase timeout in configuration:- ```yaml- extraction:- timeout: 120++1. **Verify index exists**:+ ```bash+ he info ./output/+ # Should show: Index: Built```-2. Use smaller document chunks-3. Check network connectivity-## Extraction Issues+2. **Try different query**:+ ```bash+ he search ./output/ "different keywords"+ ```++3. **Increase top_k**:+ ```bash+ he search ./output/ "query" -n 10+ ```-### Empty Results+4. **Check data exists**:+ ```bash+ he info ./output/+ # Should show Nodes > 0+ ```-**Problem**: Extraction returns no results+### Chat Fails-**Solutions**:-1. Verify document contains extractable content-2. Check template matches document type-3. Review extraction logs for errors+**Error**: `Chat failed: index not found`-### Incorrect Extractions+**Solution**:+```bash+he build-index ./output/+he talk ./output/ -q "your question"+```-**Problem**: Extracted data is incorrect or incomplete+----**Solutions**:-1. Use more specific templates-2. Add extraction hints in template description-3. Try different extraction methods-4. Adjust confidence thresholds+## Performance Issues-### Memory Issues+### Extraction is Very Slow-**Problem**: Out of memory errors+**Problem**: Taking too long to process**Solutions**:-1. Process documents in smaller batches-2. Reduce batch size-3. Clear cache periodically-## Performance Issues+1. **Skip index during batch**:+ ```bash+ he parse doc.md -t general/graph -o ./out/ -l en --no-index+ ```-### Slow Extraction+2. **Reduce chunk size** (Python):+ ```python+ ka = Template.create("general/graph", "en")+ ka.chunk_size = 1024 # Default: 2048+ ```-**Problem**: Extraction is very slow+3. **Reduce workers** (if hitting rate limits):+ ```python+ ka = Template.create("general/graph", "en")+ ka.max_workers = 5 # Default: 10+ ```++### Out of Memory++**Problem**: Process killed or memory error**Solutions**:-1. Use faster extraction methods:++1. **Process smaller chunks**:+ ```python+ for chunk in split_document(text, chunk_size=1000):+ result.feed_text(chunk)+ ```++2. **Save intermediate results**:```python- result = atom.extract(text) # Fastest+ for i, doc in enumerate(documents):+ result.feed_text(doc)+ if i % 5 == 0:+ result.dump(f"./checkpoint_{i}/")```-2. Enable parallel processing:+3. **Don't build index for intermediate steps**:```python- results = ka.batch_parse(docs, parallel=True)+ # Build only at the end+ result.build_index()```-3. Use lighter models for faster processing+----## CLI Issues+## Data Issues-### Command Not Found+### No Entities Extracted-**Problem**: `he` command not found+**Problem**: Empty result**Solutions**:-1. Reinstall the package:++1. **Check input text**:```bash- pip uninstall hyper-extract- pip install hyper-extract+ wc -l document.md # Check not empty```-2. Use Python module directly:+2. **Try different template**:```bash- python -m hyperextract parse document.txt+ he parse doc.md -t general/base_model -l en```-### Configuration Not Found+3. **Check language**:+ ```bash+ # Wrong language+ he parse chinese_doc.md -t general/graph -l en++ # Correct+ he parse chinese_doc.md -t general/graph -l zh+ ```-**Problem**: CLI can't find configuration+### Corrupted Knowledge Base-**Solution**: Initialize configuration:-```bash-he config init -k <your-api-key>-```+**Problem**: Can't load or errors reading++**Solutions**:++1. **Check file structure**:+ ```bash+ ls ./kb/+ # Should have: data.json, metadata.json+ ```++2. **Validate JSON**:+ ```bash+ python -c "import json; json.load(open('./kb/data.json'))"+ ```++3. **Re-extract**:+ ```bash+ rm -rf ./kb/+ he parse doc.md -t general/graph -o ./kb/ -l en+ ```++---## Still Having Issues?-If you're still experiencing issues:-1. Check the [FAQ](faq.md)-2. Search existing [GitHub Issues](https://github.com/yifanfeng97/hyper-extract/issues)-3. Create a new issue with detailed information+1. **Check logs** — Look for detailed error messages+2. **Update to latest** — `pip install --upgrade hyper-extract`+3. **Check GitHub Issues** — [github.com/yifanfeng97/hyper-extract/issues](https://github.com/yifanfeng97/hyper-extract/issues)+4. **Create new issue** — Include error messages and reproduction steps++---++## Debug Mode++Enable verbose output:++```python+import logging+logging.basicConfig(level=logging.DEBUG)++from hyperextract import Template+ka = Template.create("general/graph", "en")+```++Or in CLI config:+```toml+[defaults]+verbose = true+```diff --git a/docs/en/templates/browse.md b/docs/en/templates/browse.mdnew file mode 100644index 0000000..7bdb3d5--- /dev/null+++ b/docs/en/templates/browse.md@@ -0,0 +1,192 @@+# Browse All Templates++Complete list of available templates.++---++## Statistics++- **Total Templates**: 80++- **Categories**: 6 domains+- **Auto-Types**: 8 types+- **Languages**: Chinese, English++---++## By Domain++### General Templates++| Template | Type | Description |+|----------|------|-------------|+| `general/base_model` | model | Basic structured model |+| `general/base_list` | list | Basic ordered list |+| `general/base_set` | set | Basic unique set |+| `general/base_graph` | graph | Basic knowledge graph |+| `general/base_hypergraph` | hypergraph | Basic hypergraph |+| `general/base_temporal_graph` | temporal_graph | Basic temporal graph |+| `general/base_spatial_graph` | spatial_graph | Basic spatial graph |+| `general/base_spatio_temporal_graph` | spatio_temporal_graph | Basic spatio-temporal graph |+| `general/biography_graph` | temporal_graph | Person's life timeline |+| `general/concept_graph` | graph | Research concept extraction |+| `general/doc_structure` | model | Document outline structure |+| `general/workflow_graph` | graph | Process workflows |++### Finance Templates++| Template | Type | Description |+|----------|------|-------------|+| `finance/earnings_summary` | model | Quarterly/annual earnings |+| `finance/event_timeline` | temporal_graph | Financial events |+| `finance/ownership_graph` | graph | Company ownership structure |+| `finance/risk_factor_set` | set | Risk factors |+| `finance/sentiment_model` | model | Sentiment analysis |++### Legal Templates++| Template | Type | Description |+|----------|------|-------------|+| `legal/case_citation` | graph | Legal case precedents |+| `legal/case_fact_timeline` | temporal_graph | Case chronology |+| `legal/compliance_list` | list | Compliance requirements |+| `legal/contract_obligation` | list | Contract obligations |+| `legal/defined_term_set` | set | Defined terms |++### Medical Templates++| Template | Type | Description |+|----------|------|-------------|+| `medicine/anatomy_graph` | graph | Anatomical structures |+| `medicine/discharge_instruction` | model | Patient discharge summary |+| `medicine/drug_interaction` | graph | Drug interactions |+| `medicine/hospital_timeline` | temporal_graph | Hospital stay timeline |+| `medicine/treatment_map` | graph | Treatment protocols |++### TCM (Traditional Chinese Medicine) Templates++| Template | Type | Description |+|----------|------|-------------|+| `tcm/formula_composition` | graph | Herbal formula composition |+| `tcm/herb_property` | model | Herb properties |+| `tcm/herb_relation` | graph | Herb relationships |+| `tcm/meridian_graph` | graph | Meridian pathways |+| `tcm/syndrome_reasoning` | graph | Syndrome differentiation |++### Industry Templates++| Template | Type | Description |+|----------|------|-------------|+| `industry/emergency_response` | graph | Emergency procedures |+| `industry/equipment_topology` | graph | Equipment connections |+| `industry/failure_case` | temporal_graph | Failure analysis |+| `industry/operation_flow` | graph | Operational workflows |+| `industry/safety_control` | graph | Safety control systems |++---++## By Auto-Type++### Model Templates++| Template | Domain | Description |+|----------|--------|-------------|+| `general/base_model` | general | Basic structured model |+| `finance/earnings_summary` | finance | Financial summary |+| `finance/sentiment_model` | finance | Sentiment analysis |+| `medicine/discharge_instruction` | medical | Discharge summary |+| `tcm/herb_property` | tcm | Herb properties |++### List Templates++| Template | Domain | Description |+|----------|--------|-------------|+| `general/base_list` | general | Basic list |+| `legal/compliance_list` | legal | Compliance requirements |+| `legal/contract_obligation` | legal | Contract obligations |++### Set Templates++| Template | Domain | Description |+|----------|--------|-------------|+| `general/base_set` | general | Basic set |+| `finance/risk_factor_set` | finance | Risk factors |+| `legal/defined_term_set` | legal | Defined terms |++### Graph Templates++| Template | Domain | Description |+|----------|--------|-------------|+| `general/base_graph` | general | Basic graph |+| `general/knowledge_graph` | general | Knowledge graph |+| `general/concept_graph` | general | Concept graph |+| `finance/ownership_graph` | finance | Ownership structure |+| `legal/case_citation` | legal | Case precedents |+| `medicine/anatomy_graph` | medical | Anatomy |+| `tcm/herb_relation` | tcm | Herb relationships |++### Temporal Graph Templates++| Template | Domain | Description |+|----------|--------|-------------|+| `general/biography_graph` | general | Biography timeline |+| `finance/event_timeline` | finance | Financial events |+| `legal/case_fact_timeline` | legal | Case timeline |+| `medicine/hospital_timeline` | medical | Hospital timeline |++---++## By Language++### English Only++Methods (not templates):+- `method/light_rag`+- `method/graph_rag`+- `method/hyper_rag`+- `method/itext2kg`+- etc.++### Bilingual (zh, en)++All templates support both languages:+- `general/biography_graph`+- `finance/earnings_summary`+- `legal/contract_obligation`+- etc.++---++## Using This List++### CLI++```bash+# List all+he list template++# Filter by domain+he list template | grep finance+```++### Python++```python+from hyperextract import Template++# Get all+templates = Template.list()++# Filter by type+graphs = Template.list(filter_by_type="graph")++# Filter by tag+finance = Template.list(filter_by_tag="finance")+```++---++## See Also++- [How to Choose](how-to-choose.md)+- [Template Library](index.md)+- [Domain-Specific Guides](general/index.md)diff --git a/docs/en/templates/finance.md b/docs/en/templates/finance.mdindex caf8be4..52c2e6a 100644--- a/docs/en/templates/finance.md+++ b/docs/en/templates/finance.md@@ -1,230 +1,237 @@# Finance Templates-Hyper-Extract provides specialized templates for financial document extraction.+Financial document analysis and extraction.-## Available Templates+----### Earnings Summary+## Overview-Extract key financial metrics from earnings reports:+Finance templates are optimized for extracting structured information from financial documents.-```yaml-language: en+----name: Earnings Summary-type: model-tags: [finance]+## Templates-description: 'Extract key financial metrics from earnings reports.'+### earnings_summary-output:- fields:- - name: company_name- type: str- description: 'Company name'- - name: revenue- type: str- description: 'Revenue amount'- - name: quarter- type: str- description: 'Quarter'+**Type**: model-guideline:- target: 'Extract key financial information.'- rules:- - 'Extract company name and financial data'- - 'Use numbers from the source text'+**Purpose**: Extract key metrics from earnings reports-display:- label: '{company_name}'+**Best for**:+- Quarterly earnings (10-Q)+- Annual reports (10-K)+- Earnings call transcripts++**Fields**:+| Field | Type | Description |+|-------|------|-------------|+| `company_name` | str | Company name |+| `quarter` | str | Fiscal quarter |+| `revenue` | float | Total revenue |+| `net_income` | float | Net income |+| `eps` | float | Earnings per share |+| `yoy_growth` | float | Year-over-year growth |++**Example:**+```bash+he parse 10q.md -t finance/earnings_summary -l en+```++**Python:**+```python+ka = Template.create("finance/earnings_summary", "en")+result = ka.parse(earnings_text)++print(f"Revenue: ${result.data.revenue}B")+print(f"EPS: ${result.data.eps}")```-### Risk Factors+----Extract risk factors from financial documents:+### ownership_graph-```yaml-language: en+**Type**: graph-name: Risk Factor List-type: list-tags: [finance]+**Purpose**: Extract company ownership structures-description: 'Extract risk factor list.'+**Best for**:+- Shareholder reports+- Proxy statements+- Corporate structures-output:- fields:- - name: category- type: str- description: 'Risk category'- - name: description- type: str- description: 'Risk description'+**Entities**:+- Companies+- Shareholders+- Subsidiaries-guideline:- target: 'Extract risk factors.'- rules:- - 'Extract all risk factors'- - 'Each risk factor includes category and description'+**Relations**:+- `owns` — Ownership relationships+- `controls` — Control relationships+- `subsidiary_of` — Subsidiary relationships-display:- label: '{category}'+**Example:**+```bash+he parse proxy.md -t finance/ownership_graph -l en```-### Ownership Graph--Extract ownership relationships:--```yaml-language: en--name: Ownership Graph-type: graph-tags: [finance]--description: 'Extract ownership relationships.'--output:- entities:- fields:- - name: name- type: str- description: 'Entity name'- - name: type- type: str- description: 'Entity type (company/person/organization)'- relations:- fields:- - name: source- type: str- description: 'Owner'- - name: target- type: str- description: 'Owned entity'- - name: type- type: str- description: 'Relation type'--guideline:- target: 'Extract ownership relationships.'- rules_for_entities:- - 'Extract companies and individuals'- - 'Maintain consistent naming'- rules_for_relations:- - 'Create relations only when explicitly expressed'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name} ({type})'- relation_label: '{type}'+**Visualization:**+```python+result.show() # Shows ownership network```-### Event Timeline--Extract financial events and their temporal relationships:--```yaml-language: en--name: Event Timeline-type: temporal_graph-tags: [finance]--description: 'Extract financial events and their temporal relationships.'--output:- entities:- fields:- - name: name- type: str- description: 'Event name'- - name: type- type: str- description: 'Event type'- relations:- fields:- - name: source- type: str- description: 'Source event'- - name: target- type: str- description: 'Target event'- - name: type- type: str- description: 'Relation type'- - name: time- type: str- description: 'Time'- required: false--guideline:- target: 'Extract financial events and temporal relationships.'- rules_for_entities:- - 'Extract key financial events'- rules_for_relations:- - 'Extract temporal relationships between events'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}|{time}'- relation_members:- source: source- target: target- time_field: time--display:- entity_label: '{name} ({type})'- relation_label: '{type}@{time}'+---++### event_timeline++**Type**: temporal_graph++**Purpose**: Extract financial events with dates++**Best for**:+- Corporate event histories+- M&A timelines+- Market events++**Features**:+- Event dates+- Event types (merger, acquisition, IPO, etc.)+- Related entities++**Example:**+```bash+he parse history.md -t finance/event_timeline -l en```-## Usage Examples+----### CLI+### risk_factor_set+**Type**: set++**Purpose**: Extract unique risk factors++**Best for**:+- Risk factor sections+- Due diligence reports+- Risk assessments++**Example:**```bash-# Extract earnings summary-he parse earnings_report.pdf -t finance/earnings_summary -o output/+he parse 10k.md -t finance/risk_factor_set -l en+```-# Extract risk factors-he parse 10k_filing.txt -t finance/risk_factor_set -o output/+**Output:**+```python+{+ "items": [+ "Market volatility",+ "Regulatory changes",+ "Competition",+ "Supply chain risks"+ ]+}+```-# Extract ownership relationships-he parse ownership.txt -t finance/ownership_graph -o output/+---++### sentiment_model++**Type**: model++**Purpose**: Extract sentiment indicators++**Best for**:+- News articles+- Analyst reports+- Social media sentiment++**Fields**:+| Field | Type | Description |+|-------|------|-------------|+| `sentiment` | str | Overall sentiment (positive/negative/neutral) |+| `confidence` | float | Confidence score |+| `key_points` | list[str] | Key sentiment indicators |++**Example:**+```bash+he parse article.md -t finance/sentiment_model -l en```-### Python API+---++## Use Cases++### Use Case 1: Quarterly Report Analysis```pythonfrom hyperextract import Template-# Load finance template+# Extract earningska = Template.create("finance/earnings_summary", "en")+q1 = ka.parse(q1_report)+q2 = ka.parse(q2_report)-# Extract from document-result = ka.parse(earnings_text)+# Compare+print(f"Q1 Revenue: ${q1.data.revenue}B")+print(f"Q2 Revenue: ${q2.data.revenue}B")+print(f"Growth: {(q2.data.revenue - q1.data.revenue) / q1.data.revenue * 100:.1f}%")+```++### Use Case 2: Ownership Structure-# Access results-print(result.fields)+```python+ka = Template.create("finance/ownership_graph", "en")+ownership = ka.parse(proxy_statement)++# Find major shareholders+shareholders = [+ e for e in ownership.data.entities+ if e.type == "shareholder"+]++for sh in sorted(shareholders, key=lambda x: x.stake, reverse=True)[:5]:+ print(f"{sh.name}: {sh.stake}%")```-## Supported Document Types+### Use Case 3: Risk Assessment++```python+ka = Template.create("finance/risk_factor_set", "en")+risk_factors = ka.parse(risk_section)++# Categorize risks+for risk in risk_factors.data.items:+ if "regulatory" in risk.lower():+ print(f"Regulatory Risk: {risk}")+ elif "market" in risk.lower():+ print(f"Market Risk: {risk}")+```++---++## Data Sources++These templates work well with:+- SEC filings (10-K, 10-Q, 8-K)+- Earnings call transcripts+- Investor presentations+- Analyst reports+- Financial news++---++## Tips-- Annual reports-- Quarterly earnings-- 10-K/10-Q filings-- Equity research reports-- Financial news articles-- IPO prospectuses-- Supply chain analysis+1. **Use earnings_summary for quick metrics** — Fast extraction of key numbers+2. **Use ownership_graph for structure** — Visualize corporate hierarchies+3. **Use event_timeline for history** — Track corporate events+4. **Combine templates** — Use multiple templates for comprehensive analysis-## Next Steps+----- [Legal Templates](legal.md)-- [Medicine Templates](medicine.md)+## See Also+- [Browse All Templates](browse.md)+- [How to Choose](how-to-choose.md)+- [General Templates](general/index.md)diff --git a/docs/en/templates/general/index.md b/docs/en/templates/general/index.mdnew file mode 100644index 0000000..f525ff1--- /dev/null+++ b/docs/en/templates/general/index.md@@ -0,0 +1,278 @@+# General Templates++Base types and common extraction tasks.++---++## Overview++General templates provide fundamental extraction capabilities suitable for a wide range of documents.++---++## Base Types++### base_model++**Purpose**: Basic structured data extraction++**Output**: Single structured object++**Use for**: Simple summaries, form-like data++**Example:**+```bash+he parse doc.md -t general/base_model -l en+```++**Output Schema**:+```python+{+ "field1": "value1",+ "field2": "value2"+}+```++---++### base_list++**Purpose**: Ordered collection extraction++**Output**: List of items++**Use for**: Sequences, rankings++**Example:**+```bash+he parse doc.md -t general/base_list -l en+```++**Output Schema**:+```python+{+ "items": ["item1", "item2", "item3"]+}+```++---++### base_set++**Purpose**: Unique collection extraction++**Output**: Set of unique items++**Use for**: Tags, categories, keywords++**Example:**+```bash+he parse doc.md -t general/base_set -l en+```++**Output Schema**:+```python+{+ "items": {"unique1", "unique2", "unique3"}+}+```++---++### base_graph++**Purpose**: Basic entity-relationship extraction++**Output**: Graph with entities and relations++**Use for**: Knowledge graphs, networks++**Example:**+```bash+he parse doc.md -t general/base_graph -l en+```++**Output Schema**:+```python+{+ "entities": [+ {"name": "Entity1", "type": "type1"},+ {"name": "Entity2", "type": "type2"}+ ],+ "relations": [+ {"source": "Entity1", "target": "Entity2", "type": "relates_to"}+ ]+}+```++---++## Specialized Templates++### biography_graph++**Type**: temporal_graph++**Purpose**: Extract person's life events and timeline++**Best for**: Biographies, profiles, memoirs++**Example Use:**+```bash+he parse tesla_bio.md -t general/biography_graph -l en+```++**Features**:+- Extracts people, places, events+- Captures temporal relationships+- Timeline visualization++**Example Output**:+```python+{+ "entities": [+ {"name": "Nikola Tesla", "type": "person"},+ {"name": "AC Motor", "type": "invention"}+ ],+ "relations": [+ {+ "source": "Nikola Tesla",+ "target": "AC Motor",+ "type": "invented",+ "time": "1888"+ }+ ]+}+```++---++### concept_graph++**Type**: graph++**Purpose**: Extract concepts and their relationships++**Best for**: Research papers, technical documents, articles++**Example Use:**+```bash+he parse paper.md -t general/concept_graph -l en+```++**Features**:+- Concept identification+- Relationship mapping+- Hierarchical structures++---++### doc_structure++**Type**: model++**Purpose**: Extract document structure and outline++**Best for**: Long documents, reports, books++**Example Use:**+```bash+he parse report.md -t general/doc_structure -l en+```++**Features**:+- Section identification+- Heading hierarchy+- Key points per section++---++### workflow_graph++**Type**: graph++**Purpose**: Extract process workflows++**Best for**: Procedures, processes, workflows++**Example Use:**+```bash+he parse procedure.md -t general/workflow_graph -l en+```++**Features**:+- Step identification+- Flow relationships+- Decision points++---++## When to Use General Templates++### Use When:+- Document doesn't fit specific domain+- Need basic extraction+- Exploring/document type unknown+- Building custom workflows++### Don't Use When:+- Domain-specific template available (use that instead)+- Need specialized fields (create custom template)++---++## Examples++### Example 1: Simple Summary++```python+from hyperextract import Template++ka = Template.create("general/base_model", "en")+result = ka.parse("""+Meeting Notes:+Date: January 15, 2024+Attendees: Alice, Bob, Carol+Topic: Q1 Planning+Decision: Launch product in March+""")++print(result.data)+```++### Example 2: Biography Analysis++```python+ka = Template.create("general/biography_graph", "en")+result = ka.parse(biography_text)++# Visualize life events+result.show()++# Ask questions+result.build_index()+response = result.chat("What were the major achievements?")+print(response.content)+```++### Example 3: Research Paper++```python+ka = Template.create("general/concept_graph", "en")+result = ka.parse(paper_text)++# Get concept map+for entity in result.data.entities:+ print(f"Concept: {entity.name}")++for relation in result.data.relations:+ print(f"{relation.source} → {relation.target}")+```++---++## See Also++- [Browse All Templates](../browse.md)+- [How to Choose](../how-to-choose.md)+- [Auto-Types](../../concepts/autotypes.md)diff --git a/docs/en/templates/how-to-choose.md b/docs/en/templates/how-to-choose.mdnew file mode 100644index 0000000..4165f56--- /dev/null+++ b/docs/en/templates/how-to-choose.md@@ -0,0 +1,264 @@+# How to Choose a Template++Decision guide for selecting the right template.++---++## Quick Decision Tree++```mermaid+graph TD+ A[What do you want to extract?] --> B{Structured summary?}+ B -->|Yes| C[AutoModel]+ B -->|No| D{Relationships?}+ D -->|No| E{Collection?}+ E -->|Ordered| F[AutoList]+ E -->|Unique| G[AutoSet]+ D -->|Yes| H{Binary relations?}+ H -->|No| I[AutoHypergraph]+ H -->|Yes| J{Time data?}+ J -->|Yes| K{Space data?}+ K -->|Yes| L[AutoSpatioTemporalGraph]+ K -->|No| M[AutoTemporalGraph]+ J -->|No| N{Space data?}+ N -->|Yes| O[AutoSpatialGraph]+ N -->|No| P[AutoGraph]+```++---++## By Use Case++### People & Biographies++| Template | Output | Best For |+|----------|--------|----------|+| `general/biography_graph` | Temporal graph | Life stories, profiles |+| `general/base_model` | Model | Person summaries |++**Example:**+```bash+he parse bio.md -t general/biography_graph -l en -o ./out/+```++### Research & Documents++| Template | Output | Best For |+|----------|--------|----------|+| `general/concept_graph` | Graph | Research papers |+| `general/knowledge_graph` | Graph | Technical documents |+| `general/doc_structure` | Model | Document outlines |++### Financial++| Template | Output | Best For |+|----------|--------|----------|+| `finance/earnings_summary` | Model | Earnings reports |+| `finance/ownership_graph` | Graph | Company structures |+| `finance/event_timeline` | Temporal | Financial events |+| `finance/risk_factor_set` | Set | Risk assessments |++### Legal++| Template | Output | Best For |+|----------|--------|----------|+| `legal/contract_obligation` | List | Contract terms |+| `legal/case_citation` | Graph | Legal precedents |+| `legal/case_fact_timeline` | Temporal | Case chronologies |++### Medical++| Template | Output | Best For |+|----------|--------|----------|+| `medicine/anatomy_graph` | Graph | Anatomy texts |+| `medicine/drug_interaction` | Graph | Drug information |+| `medicine/treatment_map` | Graph | Treatment protocols |++---++## By Output Type++### I need a summary/report → AutoModel++```python+# Financial summary+ka = Template.create("finance/earnings_summary", "en")++# Patient discharge+ka = Template.create("medicine/discharge_instruction", "en")+```++### I need a list → AutoList++```python+# Compliance checklist+ka = Template.create("legal/compliance_list", "en")++# Symptoms+ka = Template.create("medicine/symptom_list", "en")+```++### I need unique items → AutoSet++```python+# Risk factors+ka = Template.create("finance/risk_factor_set", "en")++# Key terms+ka = Template.create("legal/defined_term_set", "en")+```++### I need a network → AutoGraph++```python+# General knowledge+ka = Template.create("general/knowledge_graph", "en")++# Company ownership+ka = Template.create("finance/ownership_graph", "en")+```++### I need a timeline → AutoTemporalGraph++```python+# Biography+ka = Template.create("general/biography_graph", "en")++# Event sequence+ka = Template.create("finance/event_timeline", "en")+```++---++## By Document Language++### English Documents++All templates support English:++```bash+he parse doc.md -t general/biography_graph -l en+```++### Chinese Documents++All templates support Chinese:++```bash+he parse doc.md -t general/biography_graph -l zh+```++**Tip:** Use the language matching your document for best results.++---++## By Document Size++### Short Documents (< 1000 words)++Any template works well. Consider:+- AutoModel for summaries+- AutoGraph for relationships++### Medium Documents (1000-5000 words)++Most templates work well:+- Biography graphs+- Concept graphs+- Knowledge graphs++### Long Documents (> 5000 words)++Consider chunking or using RAG methods:+- Use with `method/light_rag`+- Or split document into sections++---++## Examples by Scenario++### Scenario 1: Research Paper Analysis++**Need:** Extract key concepts and their relationships++**Solution:**+```bash+he parse paper.md -t general/concept_graph -l en -o ./paper_kb/+```++### Scenario 2: Financial Report++**Need:** Extract earnings metrics++**Solution:**+```bash+he parse 10k.md -t finance/earnings_summary -l en -o ./earnings/+```++### Scenario 3: Legal Contract++**Need:** Extract obligations and deadlines++**Solution:**+```bash+he parse contract.md -t legal/contract_obligation -l en -o ./contract/+```++### Scenario 4: Medical Case++**Need:** Track patient timeline++**Solution:**+```bash+he parse case.md -t medicine/hospital_timeline -l en -o ./case/+```++---++## When to Use Methods Instead++Templates don't fit your need? Use methods directly:++```python+# For large documents+ka = Template.create("method/graph_rag")++# For specific algorithms+ka = Template.create("method/itext2kg")+```++→ [Choosing Methods](../python/guides/choosing-methods.md)++---++## Getting Help++### List All Templates++```bash+he list template+```++### Search Templates++```python+from hyperextract import Template++# Search by keyword+results = Template.list(filter_by_query="finance")+```++### Check Template Details++```python+cfg = Template.get("general/biography_graph")+print(cfg.description)+```++---++## See Also++- [Browse All Templates](browse.md)+- [Template Library](index.md)+- [Custom Templates](../python/guides/custom-templates.md)diff --git a/docs/en/templates/index.md b/docs/en/templates/index.mdindex a2f3fd4..623accf 100644--- a/docs/en/templates/index.md+++ b/docs/en/templates/index.md@@ -1,67 +1,211 @@-# Templates+# Template Library-Hyper-Extract includes specialized templates for various industries and use cases.+Browse and choose from 80+ domain-specific extraction templates.-## Available Domains+----- [Finance](domain/finance.md) - Financial document extraction-- [Legal](domain/legal.md) - Legal document extraction-- [Medicine](domain/medicine.md) - Medical document extraction-- [TCM](domain/tcm.md) - Traditional Chinese Medicine templates+## What Are Templates?-## Quick Start+Templates are pre-configured extraction setups that combine:+- **Auto-Type** — Output data structure+- **Prompts** — Optimized for specific domains+- **Schema** — Field definitions for your use case+- **Guidelines** — Extraction rules-### Using a Domain Template+---++## Template Categories++<div class="grid cards" markdown>++- :material-folder:{ .lg .middle } __General__++ ---++ Base types and common extraction tasks++ - Biography graphs+ - Knowledge graphs+ - Concept extraction++ [:octicons-arrow-right-24: Browse](general/index.md)++- :material-chart-line:{ .lg .middle } __Finance__++ ---++ Financial document analysis++ - Earnings summaries+ - Risk factors+ - Ownership structures++ [:octicons-arrow-right-24: Browse](finance.md)++- :material-scale-balance:{ .lg .middle } __Legal__++ ---++ Legal document processing++ - Contract obligations+ - Case citations+ - Compliance lists++ [:octicons-arrow-right-24: Browse](legal.md)++- :material-hospital:{ .lg .middle } __Medical__++ ---++ Medical text analysis++ - Anatomy graphs+ - Drug interactions+ - Treatment plans++ [:octicons-arrow-right-24: Browse](medicine.md)++- :material-leaf:{ .lg .middle } __TCM__++ ---++ Traditional Chinese Medicine++ - Herb properties+ - Formula composition+ - Meridian graphs++ [:octicons-arrow-right-24: Browse](tcm.md)++- :material-factory:{ .lg .middle } __Industry__++ ---++ Industrial documentation++ - Equipment topology+ - Safety controls+ - Operation flows++ [:octicons-arrow-right-24: Browse](industry.md)++</div>++---++## Quick Selection Guide++| I want to extract... | Use Template |+|---------------------|--------------|+| Person's life story | `general/biography_graph` |+| Research paper concepts | `general/concept_graph` |+| Company earnings | `finance/earnings_summary` |+| Contract terms | `legal/contract_obligation` |+| Medical symptoms | `medicine/symptom_list` |+| Chinese herbs | `tcm/herb_property` |+| Equipment specs | `industry/equipment_topology` |++---++## How to Choose++### By Document Type++| Document | Template Category |+|----------|-------------------|+| Biographies, profiles | General |+| Financial reports, 10-K | Finance |+| Contracts, legal briefs | Legal |+| Medical records, papers | Medical |+| TCM texts, prescriptions | TCM |+| Manuals, procedures | Industry |++### By Output Type++| Need | Auto-Type | Example Templates |+|------|-----------|-------------------|+| Summary report | AutoModel | `earnings_summary`, `discharge_instruction` |+| List of items | AutoList | `compliance_list`, `symptom_list` |+| Unique items | AutoSet | `risk_factor_set`, `defined_term_set` |+| Network/Graph | AutoGraph | `knowledge_graph`, `ownership_graph` |+| Timeline | AutoTemporalGraph | `event_timeline`, `biography_graph` |++→ [Complete Selection Guide](how-to-choose.md)++---++## Using Templates++### CLI```bash-# CLI-he parse document.txt -t finance/earnings_summary+# List available+he list template-# Python API-ka = Template.create("finance/earnings_summary")-result = ka.parse(document_text)+# Use a template+he parse doc.md -t general/biography_graph -o ./out/ -l en```-## Template Categories+### Python-### Finance Templates+```python+from hyperextract import Template-Extract financial information from various document types:+# Create from preset+ka = Template.create("general/biography_graph", "en")-- Earnings reports-- Financial news-- IPO prospectuses-- Equity research-- Supply chain analysis+# List all+all_templates = Template.list()++# Filter by domain+finance = Template.list(filter_by_tag="finance")+```++---++## Template Properties++Each template has:++- **Name** — Unique identifier+- **Type** — Auto-Type (graph, list, model, etc.)+- **Tags** — Domain categorization+- **Languages** — Supported languages (zh, en)+- **Description** — What it extracts++### Viewing Template Details++```python+from hyperextract import Template++# Get template info+cfg = Template.get("general/biography_graph")++print(cfg.name) # biography_graph+print(cfg.type) # temporal_graph+print(cfg.tags) # ['general', 'biography']+print(cfg.description) # Description text+```-### Legal Templates+----Extract legal information:+## Browse All Templates-- Court judgments-- Contracts-- Compliance filings-- Legal treatises+→ [Complete Template Browser](browse.md)-### Medicine Templates+----Extract medical information:+## Creating Custom Templates-- Clinical guidelines-- Discharge summaries-- Package inserts-- Pathology reports-- Textbooks+Need something specific? Create your own template:-### TCM Templates+→ [Custom Templates Guide](../python/guides/custom-templates.md)-Traditional Chinese Medicine specific templates:+----- Herb properties-- Formula composition-- Meridian graphs-- Syndrome reasoning+## Template Format Reference-## Quick Links+Learn about the YAML structure:-- [Concepts](../concepts/index.md) - Understand AutoTypes and Methods+→ [Template Format](../concepts/templates-format.md)diff --git a/docs/en/templates/industry.md b/docs/en/templates/industry.mdnew file mode 100644index 0000000..1db7209--- /dev/null+++ b/docs/en/templates/industry.md@@ -0,0 +1,227 @@+# Industry Templates++Industrial documentation and operational analysis.++---++## Overview++Industry templates are designed for extracting information from industrial documentation, including equipment specifications, safety procedures, and operational workflows.++---++## Templates++### equipment_topology++**Type**: graph++**Purpose**: Extract equipment connections and topology++**Best for**:+- System diagrams+- P&IDs (Piping and Instrumentation Diagrams)+- Equipment manuals+- Network layouts++**Entities**:+- Equipment (pumps, valves, tanks, etc.)+- Sensors+- Control systems++**Relations**:+- `connected_to` — Physical connections+- `controlled_by` — Control relationships+- `monitors` — Monitoring relationships++**Example:**+```bash+he parse system_diagram.md -t industry/equipment_topology -l en+```++**Visualization:**+```python+result.show() # Shows equipment network+```++---++### safety_control++**Type**: graph++**Purpose**: Extract safety control systems++**Best for**:+- Safety manuals+- HAZOP studies+- Safety system designs+- Emergency procedures++**Entities**:+- Safety functions+- Hazards+- Safeguards+- Sensors+- Actuators++**Relations**:+- `protects_against` — Safeguard-hazard relationship+- `triggers` — Sensor-action relationship+- `implements` — System-function relationship++**Example:**+```bash+he parse safety_manual.md -t industry/safety_control -l en+```++---++### operation_flow++**Type**: graph++**Purpose**: Extract operational procedures++**Best for**:+- Operating procedures (SOPs)+- Batch records+- Process descriptions+- Work instructions++**Features**:+- Step sequences+- Decision points+- Parallel operations+- Dependencies++**Example:**+```bash+he parse sop.md -t industry/operation_flow -l en+```++---++### emergency_response++**Type**: graph++**Purpose**: Extract emergency response procedures++**Best for**:+- Emergency response plans+- Incident response guides+- Crisis management protocols++**Entities**:+- Emergency types+- Response actions+- Responsible parties+- Resources++**Relations**:+- `responds_to` — Action-emergency relationship+- `requires` — Resource requirements+- `notifies` — Notification chain++**Example:**+```bash+he parse emergency_plan.md -t industry/emergency_response -l en+```++---++### failure_case++**Type**: temporal_graph++**Purpose**: Extract failure analysis cases++**Best for**:+- Root cause analysis reports+- Incident investigations+- Failure mode documentation++**Features**:+- Timeline of events+- Contributing factors+- Root causes+- Corrective actions++**Example:**+```bash+he parse incident_report.md -t industry/failure_case -l en+```++---++## Use Cases++### System Documentation++```python+from hyperextract import Template++ka = Template.create("industry/equipment_topology", "en")+system = ka.parse(pandid_description)++# Find all pumps+pumps = [e for e in system.data.entities if "pump" in e.type.lower()]++for pump in pumps:+ # Find connections+ connections = [+ r for r in system.data.relations+ if r.source == pump.name or r.target == pump.name+ ]+ print(f"{pump.name}: {len(connections)} connections")+```++### Safety Analysis++```python+ka = Template.create("industry/safety_control", "en")+safety = ka.parse(hazop_report)++# Map hazards to safeguards+hazards = [e for e in safety.data.entities if e.type == "hazard"]++for hazard in hazards:+ safeguards = [+ r.source for r in safety.data.relations+ if r.target == hazard.name and r.type == "protects_against"+ ]+ print(f"{hazard.name}:")+ for sg in safeguards:+ print(f" Protected by: {sg}")+```++### Procedure Documentation++```python+ka = Template.create("industry/operation_flow", "en")+procedure = ka.parse(sop_document)++# Visualize workflow+procedure.show()++# Search for specific steps+procedure.build_index()+results = procedure.search("startup sequence")+```++---++## Tips++1. **equipment_topology for system design** — Document equipment networks+2. **safety_control for HAZOP** — Extract safety systems+3. **operation_flow for SOPs** — Document procedures+4. **failure_case for RCA** — Analyze incidents++---++## See Also++- [Browse All Templates](browse.md)+- [General Templates](general/index.md)diff --git a/docs/en/templates/legal.md b/docs/en/templates/legal.mdindex 95bd9f9..25008c1 100644--- a/docs/en/templates/legal.md+++ b/docs/en/templates/legal.md@@ -1,230 +1,223 @@# Legal Templates-Hyper-Extract provides specialized templates for legal document extraction.--## Available Templates--### Case Facts--Extract facts from court judgments:--```yaml-language: en--name: Case Fact Timeline-type: temporal_graph-tags: [legal]--description: 'Extract case facts from court judgments.'--output:- entities:- fields:- - name: name- type: str- description: 'Entity name'- - name: type- type: str- description: 'Entity type (party/event/fact)'- relations:- fields:- - name: source- type: str- description: 'Source entity'- - name: target- type: str- description: 'Target entity'- - name: type- type: str- description: 'Relation type'- - name: time- type: str- description: 'Time'- required: false--guideline:- target: 'Extract case facts.'- rules_for_entities:- - 'Extract parties and events'- rules_for_relations:- - 'Extract fact relationships'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}|{time}'- relation_members:- source: source- target: target- time_field: time--display:- entity_label: '{name} ({type})'- relation_label: '{type}@{time}'-```+Legal document processing and analysis.++---++## Overview++Legal templates are designed for extracting structured information from legal documents.++---++## Templates-### Contract Terms+### contract_obligation-Extract contract obligations:+**Type**: list-```yaml-language: en+**Purpose**: Extract contract obligations and terms-name: Contract Terms List-type: list-tags: [legal]+**Best for**:+- Service agreements+- Employment contracts+- Purchase agreements+- License agreements-description: 'Extract contract terms.'+**Fields**:+| Field | Type | Description |+|-------|------|-------------|+| `party` | str | Obligated party |+| `obligation` | str | Description of obligation |+| `deadline` | str | Due date/timeline |+| `conditions` | str | Conditions or caveats |-output:- fields:- - name: clause_type- type: str- description: 'Clause type'- - name: parties- type: str- description: 'Parties involved'- - name: obligations- type: str- description: 'Obligation content'+**Example:**+```bash+he parse contract.md -t legal/contract_obligation -l en+```-guideline:- target: 'Extract contract terms.'- rules:- - 'Extract all contract terms'- - 'Each term includes type, parties, and obligations'+**Python:**+```python+ka = Template.create("legal/contract_obligation", "en")+result = ka.parse(contract_text)-display:- label: '{clause_type}'+for obl in result.data.items:+ print(f"{obl.party}: {obl.obligation}")+ if obl.deadline:+ print(f" Due: {obl.deadline}")```-### Compliance Requirements+----Extract compliance requirements:+### case_citation-```yaml-language: en+**Type**: graph-name: Compliance Requirements List-type: list-tags: [legal]+**Purpose**: Extract legal case citations and relationships-description: 'Extract compliance requirements.'+**Best for**:+- Legal briefs+- Court opinions+- Research memos+- Case law analysis-output:- fields:- - name: regulation- type: str- description: 'Regulation name'- - name: requirement- type: str- description: 'Compliance requirement'- - name: deadline- type: str- description: 'Deadline'- required: false+**Entities**:+- Cases+- Statutes+- Regulations+- Courts-guideline:- target: 'Extract compliance requirements.'- rules:- - 'Extract all compliance requirements'+**Relations**:+- `cited_by` — Citation relationships+- `overruled` — Overruling relationships+- `distinguished` — Distinction relationships-display:- label: '{regulation}'+**Example:**+```bash+he parse brief.md -t legal/case_citation -l en```-### Case Citation--Extract case citation relationships:--```yaml-language: en--name: Case Citation Graph-type: graph-tags: [legal]--description: 'Extract case citation relationships.'--output:- entities:- fields:- - name: name- type: str- description: 'Case name'- - name: type- type: str- description: 'Case type'- relations:- fields:- - name: source- type: str- description: 'Citing case'- - name: target- type: str- description: 'Cited case'- - name: type- type: str- description: 'Citation type'--guideline:- target: 'Extract case citation relationships.'- rules_for_entities:- - 'Extract case names'- rules_for_relations:- - 'Extract citation relationships between cases'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'+---++### case_fact_timeline++**Type**: temporal_graph++**Purpose**: Extract chronological case facts++**Best for**:+- Case summaries+- Investigative reports+- Litigation timelines++**Features**:+- Event dates+- Fact descriptions+- Related parties++**Example:**+```bash+he parse case_summary.md -t legal/case_fact_timeline -l en```-## Usage Examples+---++### compliance_list-### CLI+**Type**: list+**Purpose**: Extract compliance requirements++**Best for**:+- Regulatory documents+- Compliance manuals+- Policy documents++**Fields**:+| Field | Type | Description |+|-------|------|-------------|+| `requirement` | str | Compliance requirement |+| `regulation` | str | Source regulation |+| `priority` | str | Priority level |++**Example:**```bash-# Extract case facts-he parse judgment.txt -t legal/case_fact_timeline -o output/+he parse compliance.md -t legal/compliance_list -l en+```-# Extract contract terms-he parse contract.pdf -t legal/contract_obligation -o output/+----# Extract compliance requirements-he parse compliance.txt -t legal/compliance_list -o output/+### defined_term_set++**Type**: set++**Purpose**: Extract defined terms and their meanings++**Best for**:+- Contracts with definitions sections+- Technical legal documents+- Glossary extraction++**Example:**+```bash+he parse agreement.md -t legal/defined_term_set -l en+```++**Output:**+```python+{+ "items": [+ "Confidential Information: means any and all non-public...",+ "Effective Date: means the date of execution...",+ "Party: means either signatory..."+ ]+}```-### Python API+---++## Use Cases++### Contract Review```pythonfrom hyperextract import Template-# Load legal template-ka = Template.create("legal/case_fact_timeline", "en")+ka = Template.create("legal/contract_obligation", "en")+obligations = ka.parse(contract)++# Find all deadlines+deadlines = [o for o in obligations.data.items if o.deadline]+for d in sorted(deadlines, key=lambda x: x.deadline):+ print(f"{d.deadline}: {d.party} - {d.obligation}")+```-# Extract from document-result = ka.parse(judgment_text)+### Case Law Analysis-# Access results-print(result.entities)+```python+ka = Template.create("legal/case_citation", "en")+case_graph = ka.parse(brief)++# Find most cited cases+citations = {}+for rel in case_graph.data.relations:+ if rel.type == "cited_by":+ citations[rel.target] = citations.get(rel.target, 0) + 1++top_cases = sorted(citations.items(), key=lambda x: x[1], reverse=True)```-## Supported Document Types+### Due Diligence-- Court judgments-- Legal contracts-- Compliance filings-- Legal treatises-- Regulatory documents+```python+# Extract obligations+ka = Template.create("legal/contract_obligation", "en")+obligations = ka.parse(agreement)++# Extract risks+ka2 = Template.create("finance/risk_factor_set", "en")+risks = ka2.parse(agreement)++# Analyze+high_risk_obligations = [+ o for o in obligations.data.items+ if any(r in o.obligation.lower() for r in risks.data.items)+]+```-## Next Steps+----- [Finance Templates](finance.md)-- [Medicine Templates](medicine.md)+## Tips++1. **contract_obligation for deadlines** — Track contractual obligations+2. **case_citation for research** — Build citation networks+3. **defined_term_set for clarity** — Extract key definitions+4. **Combine with search** — Use `he search` to find specific clauses++---+## See Also++- [Browse All Templates](browse.md)+- [Finance Templates](finance.md)diff --git a/docs/en/templates/medicine.md b/docs/en/templates/medicine.mdindex c9643f8..6d2990b 100644--- a/docs/en/templates/medicine.md+++ b/docs/en/templates/medicine.md@@ -1,211 +1,226 @@-# Medicine Templates--Hyper-Extract provides specialized templates for medical document extraction.--## Available Templates--### Drug Interactions--Extract drug interaction information:--```yaml-language: en--name: Drug Interaction Graph-type: graph-tags: [medicine]--description: 'Extract drug interaction information.'--output:- entities:- fields:- - name: name- type: str- description: 'Drug name'- - name: type- type: str- description: 'Drug type'- relations:- fields:- - name: source- type: str- description: 'Source drug'- - name: target- type: str- description: 'Target drug'- - name: type- type: str- description: 'Interaction type'--guideline:- target: 'Extract drug interactions.'- rules_for_entities:- - 'Extract drug names'- rules_for_relations:- - 'Extract interactions between drugs'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'+# Medical Templates++Medical text analysis and extraction.++---++## Overview++Medical templates are designed for extracting clinical information from medical texts.++**Disclaimer**: These templates are for research and analysis purposes only. Not for clinical decision-making without professional review.++---++## Templates++### anatomy_graph++**Type**: graph++**Purpose**: Extract anatomical structures and relationships++**Best for**:+- Anatomy textbooks+- Surgical reports+- Medical education materials++**Entities**:+- Body parts+- Organs+- Systems+- Structures++**Relations**:+- `part_of` — Anatomical hierarchy+- `connected_to` — Structural connections+- `supplies` — Blood/nerve supply++**Example:**+```bash+he parse anatomy.md -t medicine/anatomy_graph -l en```-### Treatment Plan--Extract treatment plan details:--```yaml-language: en--name: Treatment Plan Timeline-type: temporal_graph-tags: [medicine]--description: 'Extract treatment plan details.'--output:- entities:- fields:- - name: name- type: str- description: 'Entity name'- - name: type- type: str- description: 'Type (treatment/medication/procedure)'- relations:- fields:- - name: source- type: str- description: 'Source entity'- - name: target- type: str- description: 'Target entity'- - name: type- type: str- description: 'Relation type'- - name: time- type: str- description: 'Time'- required: false--guideline:- target: 'Extract treatment plan.'- rules_for_entities:- - 'Extract treatment plans and medications'- rules_for_relations:- - 'Extract treatment steps'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}|{time}'- relation_members:- source: source- target: target- time_field: time--display:- entity_label: '{name} ({type})'- relation_label: '{type}@{time}'+---++### discharge_instruction++**Type**: model++**Purpose**: Extract discharge summary information++**Best for**:+- Discharge summaries+- Transfer notes+- After-visit summaries++**Fields**:+| Field | Type | Description |+|-------|------|-------------|+| `diagnosis` | str | Primary diagnosis |+| `procedures` | list[str] | Procedures performed |+| `medications` | list[str] | Prescribed medications |+| `follow_up` | str | Follow-up instructions |+| `warnings` | list[str] | Warning signs |++**Example:**+```bash+he parse discharge.md -t medicine/discharge_instruction -l en```-### Anatomy Graph--Extract anatomy structure relationships:--```yaml-language: en--name: Anatomy Structure Graph-type: graph-tags: [medicine]--description: 'Extract anatomy structure relationships.'--output:- entities:- fields:- - name: name- type: str- description: 'Anatomy structure name'- - name: type- type: str- description: 'Structure type'- relations:- fields:- - name: source- type: str- description: 'Source structure'- - name: target- type: str- description: 'Target structure'- - name: type- type: str- description: 'Relation type'--guideline:- target: 'Extract anatomy structure relationships.'- rules_for_entities:- - 'Extract anatomy structures'- rules_for_relations:- - 'Extract relationships between structures'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'+**Python:**+```python+ka = Template.create("medicine/discharge_instruction", "en")+summary = ka.parse(discharge_text)++print(f"Diagnosis: {summary.data.diagnosis}")+print(f"Follow-up: {summary.data.follow_up}")```-## Usage Examples+---++### drug_interaction++**Type**: graph-### CLI+**Purpose**: Extract drug interactions and relationships+**Best for**:+- Drug databases+- Pharmacy references+- Medication reconciliation++**Entities**:+- Drugs+- Drug classes+- Effects++**Relations**:+- `interacts_with` — Drug interactions+- `contraindicated_with` — Contraindications+- `synergizes_with` — Synergistic effects++**Example:**```bash-# Extract drug interactions-he parse clinical_note.txt -t medicine/drug_interaction -o output/+he parse drug_ref.md -t medicine/drug_interaction -l en+```++---++### hospital_timeline++**Type**: temporal_graph++**Purpose**: Extract hospital stay timeline-# Extract treatment plan-he parse discharge_summary.pdf -t medicine/treatment_map -o output/+**Best for**:+- Hospital course notes+- Progress notes+- Transfer summaries++**Features**:+- Admission/discharge dates+- Procedures and events+- Timeline visualization++**Example:**+```bash+he parse hospital_notes.md -t medicine/hospital_timeline -l en+```++---++### treatment_map++**Type**: graph++**Purpose**: Extract treatment protocols and pathways++**Best for**:+- Treatment guidelines+- Care pathways+- Protocol documents++**Example:**+```bash+he parse protocol.md -t medicine/treatment_map -l en```-### Python API+---++## Use Cases++### Discharge Summary Analysis```pythonfrom hyperextract import Template-# Load medicine template+ka = Template.create("medicine/discharge_instruction", "en")+discharges = []++for file in discharge_files:+ summary = ka.parse(file.read_text())+ discharges.append(summary.data)++# Analyze common diagnoses+from collections import Counter+diagnoses = Counter(d.diagnosis for d in discharges)+print(diagnoses.most_common(10))+```++### Drug Interaction Network++```pythonka = Template.create("medicine/drug_interaction", "en")+network = ka.parse(drug_database)-# Extract from document-result = ka.parse(clinical_text)+# Find interactions for specific drug+drug = "Warfarin"+interactions = [+ r for r in network.data.relations+ if r.source == drug or r.target == drug+]-# Access results-print(result.entities)+for i in interactions:+ print(f"{i.source} {i.type} {i.target}")```-## Supported Document Types+### Anatomy Education-- Clinical guidelines-- Discharge summaries-- Package inserts-- Pathology reports-- Medical textbooks-- Clinical notes+```python+ka = Template.create("medicine/anatomy_graph", "en")+anatomy = ka.parse(textbook_chapter)-## Next Steps+# Visualize+anatomy.show()-- [TCM Templates](tcm.md)-- [Finance Templates](finance.md)+# Search+anatomy.build_index()+results = anatomy.search("nerves in hand")+```++---+## Tips++1. **discharge_instruction for summaries** — Quick extraction of key discharge info+2. **drug_interaction for safety** — Build interaction checking tools+3. **anatomy_graph for education** — Create interactive anatomy maps+4. **hospital_timeline for courses** — Track patient journeys++---++## Data Privacy++When working with medical data:+- Ensure HIPAA compliance+- De-identify data before processing+- Use secure environments+- Follow institutional policies++---++## See Also++- [Browse All Templates](browse.md)+- [TCM Templates](tcm.md)diff --git a/docs/en/templates/tcm.md b/docs/en/templates/tcm.mdindex b9167da..8ea8b3b 100644--- a/docs/en/templates/tcm.md+++ b/docs/en/templates/tcm.md@@ -1,228 +1,224 @@# TCM Templates-Hyper-Extract provides specialized templates for Traditional Chinese Medicine (TCM) document extraction.--## Available Templates--### Herb Properties--Extract properties of medicinal herbs:--```yaml-language: en--name: Herb Properties-type: model-tags: [tcm]--description: 'Extract herb property information.'--output:- fields:- - name: herb_name- type: str- description: 'Herb name'- - name: properties- type: str- description: 'Properties'- - name: channels- type: str- description: 'Meridian channels'- - name: indications- type: str- description: 'Indications'--guideline:- target: 'Extract herb properties.'- rules:- - 'Extract herb name and properties'- - 'Keep original expressions'--display:- label: '{herb_name}'-```+Traditional Chinese Medicine text analysis.++---++## Overview++TCM (Traditional Chinese Medicine) templates are designed for extracting knowledge from TCM texts, including herbal medicine, formulas, and syndrome differentiation.++---++## Templates-### Formula Composition+### herb_property-Extract herbal formula composition:+**Type**: model-```yaml-language: en+**Purpose**: Extract herb properties and characteristics-name: Formula Composition List-type: list-tags: [tcm]+**Best for**:+- Materia medica texts+- Herb databases+- Prescription references-description: 'Extract formula composition.'+**Fields**:+| Field | Type | Description |+|-------|------|-------------|+| `name` | str | Herb name (Chinese/pinyin) |+| `properties` | list[str] | Nature (hot, warm, cool, cold, neutral) |+| `flavors` | list[str] | Five flavors (sour, bitter, sweet, pungent, salty) |+| `meridians` | list[str] | Entered meridians |+| `functions` | list[str] | Main functions |+| `indications` | list[str] | Clinical indications |-output:- fields:- - name: herb_name- type: str- description: 'Herb name'- - name: dosage- type: str- description: 'Dosage'- - name: role- type: str- description: 'Role (jun/chen/zuo/shi)'- required: false+**Example:**+```bash+he parse herb_text.md -t tcm/herb_property -l zh+```-guideline:- target: 'Extract formula composition.'- rules:- - 'Extract all herbs in the formula'- - 'Record dosage and role'+**Python:**+```python+ka = Template.create("tcm/herb_property", "zh")+herb = ka.parse(huang_qi_text)-display:- label: '{herb_name}'+print(f"Name: {herb.data.name}")+print(f"Nature: {', '.join(herb.data.properties)}")+print(f"Functions: {', '.join(herb.data.functions)}")```-### Meridian Graph--Extract meridian relationships:--```yaml-language: en--name: Meridian Relationship Graph-type: graph-tags: [tcm]--description: 'Extract meridian relationships.'--output:- entities:- fields:- - name: name- type: str- description: 'Entity name (herb/meridian)'- - name: type- type: str- description: 'Type'- relations:- fields:- - name: source- type: str- description: 'Source entity'- - name: target- type: str- description: 'Target entity'- - name: type- type: str- description: 'Relation type'--guideline:- target: 'Extract meridian relationships.'- rules_for_entities:- - 'Extract herbs and meridians'- rules_for_relations:- - 'Extract relationships of herbs entering meridians'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'+---++### formula_composition++**Type**: graph++**Purpose**: Extract herbal formula compositions++**Best for**:+- Formula collections (方剂学)+- Prescription texts+- Classic texts (伤寒论, 金匮要略)++**Entities**:+- Herbs (药物)+- Formulas (方剂)+- Conditions (病症)++**Relations**:+- `contains` — Formula contains herb+- `treats` — Formula treats condition+- `modifies` — Modified formula relationship++**Example:**+```bash+he parse formula_book.md -t tcm/formula_composition -l zh```-### Syndrome Reasoning--Extract syndrome diagnosis and reasoning:--```yaml-language: en--name: Syndrome Reasoning Hypergraph-type: hypergraph-tags: [tcm]--description: 'Extract syndrome diagnosis and reasoning.'--output:- entities:- fields:- - name: name- type: str- description: 'Entity name'- - name: type- type: str- description: 'Type (syndrome/symptom/treatment)'- relations:- fields:- - name: syndrome- type: str- description: 'Syndrome name'- - name: symptoms- type: list- description: 'Related symptoms'- - name: treatment- type: str- description: 'Treatment method'- required: false--guideline:- target: 'Extract syndrome diagnosis and reasoning.'- rules_for_entities:- - 'Extract syndromes and symptoms'- rules_for_relations:- - 'Extract relationships between syndrome and symptoms/treatment'--identifiers:- entity_id: name- relation_id: '{syndrome}'- relation_members: symptoms--display:- entity_label: '{name}'- relation_label: '{syndrome}'+---++### herb_relation++**Type**: graph++**Purpose**: Extract herb-to-herb relationships++**Best for**:+- Herb pairing references (药对)+- Compatibility texts (十八反, 十九畏)+- Combination guides++**Relations**:+- `pairs_with` — Common pairing+- `synergizes_with` — Synergistic effect+- `incompatible_with` — Contraindicated combination++**Example:**+```bash+he parse herb_pairs.md -t tcm/herb_relation -l zh```-## Usage Examples+----### CLI+### meridian_graph+**Type**: graph++**Purpose**: Extract meridian pathway information++**Best for**:+- Acupuncture texts+- Meridian studies+- Point location references++**Entities**:+- Meridians (经脉)+- Acupoints (穴位)+- Organs (脏腑)++**Relations**:+- `connects_to` — Meridian connections+- `belongs_to` — Point to meridian+- `influences` — Meridian-organ relationship++**Example:**```bash-# Extract herb properties-he parse herbal_compendium.txt -t tcm/herb_property -o output/+he parse acupuncture.md -t tcm/meridian_graph -l zh+```++---++### syndrome_reasoning++**Type**: graph++**Purpose**: Extract syndrome differentiation logic++**Best for**:+- Diagnostics texts (中医诊断学)+- Case studies+- Syndrome differentiation guides++**Entities**:+- Symptoms (症状)+- Syndromes (证型)+- Patterns (病机)-# Extract formula composition-he parse prescription.txt -t tcm/formula_composition -o output/+**Relations**:+- `indicates` — Symptom indicates syndrome+- `differentiates_from` — Differential diagnosis+- `leads_to` — Pattern progression-# Extract meridian relationships-he parse meridian.txt -t tcm/meridian_graph -o output/+**Example:**+```bash+he parse diagnostics.md -t tcm/syndrome_reasoning -l zh```-### Python API+---++## Use Cases++### Herb Database Building```pythonfrom hyperextract import Template-# Load TCM template-ka = Template.create("tcm/herb_property", "en")+ka = Template.create("tcm/herb_property", "zh")+herb_db = {}++for herb_file in herb_files:+ herb = ka.parse(herb_file.read_text())+ herb_db[herb.data.name] = herb.data++# Query by function+for name, data in herb_db.items():+ if "补气" in data.functions:+ print(f"{name}: Qi tonic herb")+```++### Formula Analysis++```python+ka = Template.create("tcm/formula_composition", "zh")+formula = ka.parse(si_jun_zi_tang_text)++# List herbs+print("Formula composition:")+for relation in formula.data.relations:+ if relation.type == "contains":+ print(f" - {relation.target}")+```-# Extract from document-result = ka.parse(herbal_text)+### Syndrome Study-# Access results-print(result.fields)+```python+ka = Template.create("tcm/syndrome_reasoning", "zh")+syndromes = ka.parse(diagnostics_text)++# Find what symptom indicates what syndrome+symptom = "舌淡苔白"+related = [+ r for r in syndromes.data.relations+ if r.source == symptom and r.type == "indicates"+]++for r in related:+ print(f"{symptom} indicates {r.target}")```-## Supported Document Types+---++## Tips-- Herbal compendiums-- Prescription forms-- Medical case records-- Meridian treatises-- Formula pharmacopoeias+1. **Use Chinese language (`-l zh`)** — Better extraction from TCM texts+2. **herb_property for materia medica** — Build herb databases+3. **formula_composition for方剂学** — Study classic formulas+4. **meridian_graph for acupuncture** — Map meridian pathways-## Next Steps+----- [Medicine Templates](medicine.md)-- [Finance Templates](finance.md)+## See Also+- [Browse All Templates](browse.md)+- [Medical Templates](medicine.md)diff --git a/docs/en/tutorials/document-analysis/index.md b/docs/en/tutorials/document-analysis/index.mdnew file mode 100644index 0000000..877aa1a--- /dev/null+++ b/docs/en/tutorials/document-analysis/index.md@@ -0,0 +1,573 @@+# Document Analysis Tutorial++Process large collections of documents efficiently.++---++## What You'll Learn++This tutorial covers strategies for processing large document collections:+- Batch processing techniques+- Parallelization strategies+- Error handling and recovery+- Performance optimization+- Memory management++---++## Use Cases++### Use Case 1: Document Archive Processing++Process thousands of historical documents:+- Legal document archives+- Medical records+- Research paper collections++### Use Case 2: Real-time Document Pipeline++Continuous processing of incoming documents:+- News feed analysis+- Social media monitoring+- Email processing++### Use Case 3: Large-scale Analysis++One-time processing of massive datasets:+- Corporate document repositories+- Government archives+- Academic databases++---++## Architecture++```+document-analysis/+├── input/ # Input documents+│ ├── pending/ # Documents to process+│ ├── processing/ # Currently processing+│ └── completed/ # Successfully processed+├── output/ # Extracted knowledge bases+│ ├── batch_001/+│ ├── batch_002/+│ └── combined/+├── logs/ # Processing logs+├── batch_processor.py # Main processor+└── config.yaml # Configuration+```++---++## Strategies++### Strategy 1: Simple Batching++Process documents in fixed-size batches.++### Strategy 2: Parallel Processing++Use multiple workers to process documents concurrently.++### Strategy 3: Streaming++Process documents as they arrive without waiting for batches.++---++## Complete Example++```python+"""Batch Document Processor."""++import os+import json+import logging+from pathlib import Path+from datetime import datetime+from concurrent.futures import ProcessPoolExecutor, as_completed+from dataclasses import dataclass+from typing import List, Optional++from hyperextract import Template++# Setup logging+logging.basicConfig(+ level=logging.INFO,+ format='%(asctime)s - %(levelname)s - %(message)s',+ handlers=[+ logging.FileHandler('logs/processor.log'),+ logging.StreamHandler()+ ]+)+logger = logging.getLogger(__name__)+++@dataclass+class ProcessingResult:+ document: str+ success: bool+ output_path: Optional[str]+ error: Optional[str]+ processing_time: float+++class BatchDocumentProcessor:+ """Process large collections of documents."""++ def __init__(+ self,+ template: str = "general/knowledge_graph",+ language: str = "en",+ batch_size: int = 10,+ max_workers: int = 4+ ):+ self.template = template+ self.language = language+ self.batch_size = batch_size+ self.max_workers = max_workers++ # Ensure directories exist+ for dir_name in ['input/pending', 'input/processing', 'input/completed',+ 'output', 'logs']:+ Path(dir_name).mkdir(parents=True, exist_ok=True)++ def get_pending_documents(self) -> List[Path]:+ """Get list of pending documents."""+ pending_dir = Path("input/pending")+ docs = []+ docs.extend(pending_dir.glob("**/*.md"))+ docs.extend(pending_dir.glob("**/*.txt"))+ return sorted(docs)++ def process_single_document(self, doc_path: Path) -> ProcessingResult:+ """Process a single document."""+ import time+ start_time = time.time()++ try:+ # Move to processing+ processing_path = Path("input/processing") / doc_path.name+ doc_path.rename(processing_path)++ # Read document+ logger.info(f"Processing: {doc_path.name}")+ text = processing_path.read_text(encoding="utf-8")++ # Extract knowledge+ ka = Template.create(self.template, self.language)+ result = ka.parse(text)++ # Build index+ result.build_index()++ # Save output+ output_name = doc_path.stem+ output_path = Path("output") / output_name+ result.dump(str(output_path))++ # Move to completed+ completed_path = Path("input/completed") / doc_path.name+ processing_path.rename(completed_path)++ processing_time = time.time() - start_time+ logger.info(f"✓ Completed: {doc_path.name} ({processing_time:.2f}s)")++ return ProcessingResult(+ document=str(doc_path),+ success=True,+ output_path=str(output_path),+ error=None,+ processing_time=processing_time+ )++ except Exception as e:+ processing_time = time.time() - start_time+ logger.error(f"✗ Failed: {doc_path.name} - {str(e)}")++ # Move back to pending for retry+ if processing_path.exists():+ processing_path.rename(doc_path)++ return ProcessingResult(+ document=str(doc_path),+ success=False,+ output_path=None,+ error=str(e),+ processing_time=processing_time+ )++ def process_batch(self, documents: List[Path]) -> List[ProcessingResult]:+ """Process a batch of documents."""+ results = []++ # Sequential processing (for memory efficiency)+ for doc in documents:+ result = self.process_single_document(doc)+ results.append(result)++ return results++ def process_parallel(self, documents: List[Path]) -> List[ProcessingResult]:+ """Process documents in parallel."""+ results = []++ with ProcessPoolExecutor(max_workers=self.max_workers) as executor:+ # Submit all tasks+ future_to_doc = {+ executor.submit(self.process_single_document, doc): doc+ for doc in documents+ }++ # Collect results as they complete+ for future in as_completed(future_to_doc):+ result = future.result()+ results.append(result)++ return results++ def run(self, parallel: bool = False):+ """Run the batch processor."""+ # Get pending documents+ documents = self.get_pending_documents()++ if not documents:+ logger.info("No pending documents to process")+ return []++ logger.info(f"Found {len(documents)} documents to process")++ # Process in batches+ all_results = []++ for i in range(0, len(documents), self.batch_size):+ batch = documents[i:i + self.batch_size]+ logger.info(f"Processing batch {i//self.batch_size + 1}: {len(batch)} documents")++ if parallel:+ results = self.process_parallel(batch)+ else:+ results = self.process_batch(batch)++ all_results.extend(results)++ # Log batch summary+ successes = sum(1 for r in results if r.success)+ logger.info(f"Batch complete: {successes}/{len(batch)} successful")++ # Final summary+ total_success = sum(1 for r in all_results if r.success)+ logger.info(f"Processing complete: {total_success}/{len(documents)} successful")++ # Save results log+ self._save_results_log(all_results)++ return all_results++ def _save_results_log(self, results: List[ProcessingResult]):+ """Save processing results to log file."""+ log_data = [+ {+ "document": r.document,+ "success": r.success,+ "output_path": r.output_path,+ "error": r.error,+ "processing_time": r.processing_time,+ "timestamp": datetime.now().isoformat()+ }+ for r in results+ ]++ log_file = f"logs/results_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"+ with open(log_file, "w") as f:+ json.dump(log_data, f, indent=2)++ logger.info(f"Results log saved: {log_file}")++ def combine_results(self, output_name: str = "combined"):+ """Combine all individual results into one knowledge base."""+ logger.info("Combining results...")++ output_dirs = [d for d in Path("output").iterdir() if d.is_dir()]++ if not output_dirs:+ logger.warning("No output directories found")+ return++ # Load first result+ ka = Template.create(self.template, self.language)+ combined = ka.load(str(output_dirs[0]))++ # Merge remaining+ for output_dir in output_dirs[1:]:+ kb = Template.create(self.template, self.language)+ kb.load(str(output_dir))++ # Merge entities and relations+ for entity in kb.data.entities:+ if entity not in combined.data.entities:+ combined.data.entities.append(entity)++ for relation in kb.data.relations:+ if relation not in combined.data.relations:+ combined.data.relations.append(relation)++ # Rebuild index and save+ combined.build_index()+ combined_path = Path("output") / output_name+ combined.dump(str(combined_path))++ logger.info(f"Combined knowledge base saved: {combined_path}")+ return combined+++# CLI Interface+if __name__ == "__main__":+ import argparse++ parser = argparse.ArgumentParser(description="Batch Document Processor")+ parser.add_argument("--template", default="general/knowledge_graph", help="Template to use")+ parser.add_argument("--language", default="en", help="Document language")+ parser.add_argument("--batch-size", type=int, default=10, help="Batch size")+ parser.add_argument("--parallel", action="store_true", help="Enable parallel processing")+ parser.add_argument("--combine", action="store_true", help="Combine results after processing")++ args = parser.parse_args()++ processor = BatchDocumentProcessor(+ template=args.template,+ language=args.language,+ batch_size=args.batch_size+ )++ # Process documents+ results = processor.run(parallel=args.parallel)++ # Combine if requested+ if args.combine:+ processor.combine_results()+```++---++## Usage++### Setup++```bash+# Create directory structure+mkdir -p input/pending input/processing input/completed output logs++# Add documents to process+cp /path/to/documents/*.md input/pending/+```++### Run++```bash+# Sequential processing+python batch_processor.py++# Parallel processing+python batch_processor.py --parallel --max-workers 4++# With custom template+python batch_processor.py --template finance/earnings_summary++# Process and combine results+python batch_processor.py --combine+```++---++## Performance Tips++### 1. Batch Size++```python+# Small batches for memory-constrained environments+processor = BatchDocumentProcessor(batch_size=5)++# Larger batches for I/O bound processing+processor = BatchDocumentProcessor(batch_size=50)+```++### 2. Parallel Workers++```python+# CPU-bound: Match CPU cores+processor = BatchDocumentProcessor(max_workers=os.cpu_count())++# API-bound: Limit to avoid rate limits+processor = BatchDocumentProcessor(max_workers=5)+```++### 3. Memory Management++```python+# Process in smaller chunks+def process_large_dataset(documents, chunk_size=100):+ for i in range(0, len(documents), chunk_size):+ chunk = documents[i:i + chunk_size]+ results = processor.process_batch(chunk)++ # Save intermediate results+ save_checkpoint(results, i)++ # Clear memory+ gc.collect()+```++---++## Error Handling++### Retry Logic++```python+from tenacity import retry, stop_after_attempt, wait_exponential++@retry(+ stop=stop_after_attempt(3),+ wait=wait_exponential(multiplier=1, min=4, max=10)+)+def process_with_retry(doc_path):+ return processor.process_single_document(doc_path)+```++### Failed Document Queue++```python+def process_with_fallback(documents):+ failed = []++ for doc in documents:+ result = processor.process_single_document(doc)+ if not result.success:+ failed.append(doc)++ # Retry failed documents with different settings+ for doc in failed:+ # Try with smaller chunks+ result = processor.process_single_document(+ doc,+ chunk_size=1024 # Smaller chunks+ )+```++---++## Monitoring++### Progress Tracking++```python+from tqdm import tqdm++def process_with_progress(documents):+ results = []++ with tqdm(total=len(documents), desc="Processing") as pbar:+ for doc in documents:+ result = processor.process_single_document(doc)+ results.append(result)+ pbar.update(1)++ # Update description+ success_rate = sum(1 for r in results if r.success) / len(results)+ pbar.set_postfix({"success_rate": f"{success_rate:.1%}"})++ return results+```++### Metrics Collection++```python+import time+from dataclasses import asdict++def collect_metrics(results: List[ProcessingResult]):+ metrics = {+ "total_documents": len(results),+ "successful": sum(1 for r in results if r.success),+ "failed": sum(1 for r in results if not r.success),+ "avg_processing_time": sum(r.processing_time for r in results) / len(results),+ "total_time": sum(r.processing_time for r in results),+ "errors": [r.error for r in results if r.error]+ }++ with open("logs/metrics.json", "w") as f:+ json.dump(metrics, f, indent=2)++ return metrics+```++---++## Advanced: Streaming Processor++For real-time document processing:++```python+import asyncio+from watchdog.observers import Observer+from watchdog.events import FileSystemEventHandler++class StreamingProcessor(FileSystemEventHandler):+ """Process documents as they arrive."""++ def __init__(self):+ self.processor = BatchDocumentProcessor()+ self.queue = asyncio.Queue()++ def on_created(self, event):+ if not event.is_directory:+ self.queue.put_nowait(event.src_path)++ async def process_queue(self):+ while True:+ file_path = await self.queue.get()+ result = self.processor.process_single_document(Path(file_path))++ if result.success:+ logger.info(f"Processed: {file_path}")+ else:+ logger.error(f"Failed: {file_path} - {result.error}")++ self.queue.task_done()++ def start(self, watch_dir: str = "input/pending"):+ # Start file watcher+ observer = Observer()+ observer.schedule(self, watch_dir, recursive=True)+ observer.start()++ # Start processing loop+ asyncio.run(self.process_queue())+```++---++## Summary++You now have a complete batch processing system that can:++✓ Process large document collections+✓ Handle errors and retries+✓ Run in parallel for speed+✓ Monitor progress and collect metrics+✓ Combine results into unified knowledge bases++### Next Steps++- Deploy as a service+- Add REST API interface+- Implement real-time streaming+- Scale with distributed processing++---++## See Also++- [Research Assistant Tutorial](../research-assistant/index.md)+- [Knowledge Base Tutorial](../knowledge-base/index.md)diff --git a/docs/en/tutorials/index.md b/docs/en/tutorials/index.mdnew file mode 100644index 0000000..eee941b--- /dev/null+++ b/docs/en/tutorials/index.md@@ -0,0 +1,65 @@+# Tutorials++Hands-on tutorials for real-world use cases.++---++## Available Tutorials++<div class="grid cards" markdown>++- :material-school:{ .lg .middle } __Research Assistant__++ ---++ Build an interactive research assistant from academic papers.++ - Extract knowledge from research papers+ - Enable semantic search+ - Build a Q&A system++ [:octicons-arrow-right-24: Start Tutorial](research-assistant/index.md)++- :material-database:{ .lg .middle } __Knowledge Base__++ ---++ Build and maintain a domain-specific knowledge base.++ - Incremental document ingestion+ - Version management+ - Query and update workflows++ [:octicons-arrow-right-24: Start Tutorial](knowledge-base/index.md)++- :material-file-document-multiple:{ .lg .middle } __Document Analysis__++ ---++ Process large collections of documents.++ - Batch processing strategies+ - Parallelization techniques+ - Performance optimization++ [:octicons-arrow-right-24: Start Tutorial](document-analysis/index.md)++</div>++---++## Prerequisites++All tutorials assume you have:+- Hyper-Extract installed+- API key configured+- Basic familiarity with CLI or Python SDK++---++## Getting Help++Stuck on a tutorial?+- Check the [FAQ](../resources/faq.md)+- Review [Troubleshooting](../resources/troubleshooting.md)+- Open an issue on [GitHub](https://github.com/yifanfeng97/hyper-extract/issues)diff --git a/docs/en/tutorials/knowledge-base/index.md b/docs/en/tutorials/knowledge-base/index.mdnew file mode 100644index 0000000..bb4094a--- /dev/null+++ b/docs/en/tutorials/knowledge-base/index.md@@ -0,0 +1,78 @@+# Knowledge Base Tutorial++Build and maintain a domain-specific knowledge base.++---++## What You'll Build++A production-ready knowledge base that:+- Ingests documents incrementally+- Supports versioning and backups+- Provides query and search capabilities+- Handles updates and modifications++---++## Tutorial Overview++| Step | Topic | What You'll Learn |+|------|-------|-------------------|+| 1 | [Setup](step1-setup.md) | Project structure and template selection |+| 2 | [Ingest](step2-ingest.md) | Add documents to your knowledge base |+| 3 | [Query](step3-query.md) | Search, update, and maintain |++---++## Use Cases++### Use Case 1: Company Knowledge Base++Centralize company documentation:+- Policies and procedures+- Technical documentation+- Meeting notes and decisions++### Use Case 2: Research Knowledge Base++Build a personal research database:+- Academic papers+- Notes and annotations+- Cross-paper connections++### Use Case 3: Legal Document Repository++Manage legal documents:+- Contracts and agreements+- Case files+- Regulatory documents++---++## Architecture++```+knowledge-base/+├── config.yaml # Configuration+├── documents/ # Source documents+├── kb/ # Knowledge base storage+│ ├── v1/ # Version 1+│ ├── v2/ # Version 2+│ └── current/ # Current version+├── backups/ # Backups+└── kb_manager.py # Management script+```++---++## Prerequisites++- Hyper-Extract installed+- API key configured+- Collection of documents to ingest++---++## Next Steps++→ [Step 1: Setup](step1-setup.md)diff --git a/docs/en/tutorials/knowledge-base/step1-setup.md b/docs/en/tutorials/knowledge-base/step1-setup.mdnew file mode 100644index 0000000..4f02378--- /dev/null+++ b/docs/en/tutorials/knowledge-base/step1-setup.md@@ -0,0 +1,225 @@+# Step 1: Setup++Project structure and template selection.++---++## Goal++Set up a robust project structure for maintaining a domain-specific knowledge base.++---++## Project Structure++Create the following directory structure:++```bash+mkdir -p my_knowledge_base/{documents,kb,backups,logs}+cd my_knowledge_base+```++Structure:+```+my_knowledge_base/+├── config.yaml # Project configuration+├── documents/ # Source documents+│ ├── raw/ # Original files+│ └── processed/ # Processing log+├── kb/ # Knowledge base versions+│ └── current/ # Current version (symlink)+├── backups/ # Version backups+├── logs/ # Operation logs+└── kb_manager.py # Management script+```++---++## Configuration++### config.yaml++```yaml+# Knowledge Base Configuration+name: "Company Knowledge Base"+domain: "legal" # or finance, medical, etc.++# Template settings+template: "legal/contract_obligation" # Choose based on domain+language: "en"++# Processing settings+chunk_size: 2048+max_workers: 10++# Versioning+version_format: "v{major}.{minor}"+auto_backup: true++# Paths+documents_dir: "./documents/raw"+kb_dir: "./kb"+backup_dir: "./backups"+```++### Template Selection Guide++| Domain | Recommended Template | Auto-Type |+|--------|---------------------|-----------|+| Legal | `legal/contract_obligation` | list |+| Finance | `finance/earnings_summary` | model |+| Medical | `medicine/anatomy_graph` | graph |+| General | `general/knowledge_graph` | graph |+| Research | `general/concept_graph` | graph |++---++## Management Script++### kb_manager.py (Starter)++```python+"""Knowledge Base Manager."""++import yaml+import json+from pathlib import Path+from datetime import datetime+from dataclasses import dataclass+from typing import Optional++from hyperextract import Template++@dataclass+class KBConfig:+ name: str+ domain: str+ template: str+ language: str+ documents_dir: str+ kb_dir: str+ backup_dir: str++class KnowledgeBaseManager:+ def __init__(self, config_path: str = "config.yaml"):+ self.config = self._load_config(config_path)+ self.ka = None+ self.current_kb = None++ def _load_config(self, path: str) -> KBConfig:+ """Load configuration from YAML."""+ with open(path) as f:+ data = yaml.safe_load(f)+ return KBConfig(**data)++ def initialize(self):+ """Initialize knowledge base."""+ print(f"Initializing: {self.config.name}")++ # Create directories+ Path(self.config.documents_dir).mkdir(parents=True, exist_ok=True)+ Path(self.config.kb_dir).mkdir(parents=True, exist_ok=True)+ Path(self.config.backup_dir).mkdir(parents=True, exist_ok=True)++ # Create template+ self.ka = Template.create(+ self.config.template,+ language=self.config.language+ )++ print("✓ Initialization complete")++ def get_version_path(self, version: Optional[str] = None) -> Path:+ """Get path for a specific version."""+ if version is None:+ version = datetime.now().strftime("v%Y%m%d_%H%M%S")+ return Path(self.config.kb_dir) / version++ def save_version(self, kb_instance, version: Optional[str] = None):+ """Save knowledge base version."""+ path = self.get_version_path(version)+ kb_instance.dump(str(path))++ # Update current symlink+ current_link = Path(self.config.kb_dir) / "current"+ if current_link.exists():+ current_link.unlink()+ current_link.symlink_to(path, target_is_directory=True)++ print(f"✓ Saved version: {path.name}")+ return path++# Usage+if __name__ == "__main__":+ manager = KnowledgeBaseManager()+ manager.initialize()+```++---++## Initialization++### 1. Create Config++```bash+cat > config.yaml << 'EOF'+name: "My Knowledge Base"+domain: "general"+template: "general/knowledge_graph"+language: "en"+documents_dir: "./documents/raw"+kb_dir: "./kb"+backup_dir: "./backups"+EOF+```++### 2. Run Setup++```bash+python kb_manager.py+```++Expected output:+```+Initializing: My Knowledge Base+✓ Initialization complete+```++### 3. Verify Structure++```bash+tree my_knowledge_base/+```++---++## Template Testing++Before ingesting documents, test your template:++```python+# test_template.py+from hyperextract import Template++# Load config+import yaml+with open("config.yaml") as f:+ config = yaml.safe_load(f)++# Test template+ka = Template.create(config["template"], config["language"])++# Test with sample text+test_text = "This is a test document for template validation."+result = ka.parse(test_text)++print(f"Template: {config['template']}")+print(f"Entities: {len(result.data.entities)}")+print(f"Relations: {len(result.data.relations)}")+```++---++## Next Step++→ [Step 2: Ingest Documents](step2-ingest.md)diff --git a/docs/en/tutorials/knowledge-base/step2-ingest.md b/docs/en/tutorials/knowledge-base/step2-ingest.mdnew file mode 100644index 0000000..ccd3ca7--- /dev/null+++ b/docs/en/tutorials/knowledge-base/step2-ingest.md@@ -0,0 +1,268 @@+# Step 2: Ingest++Add documents to your knowledge base.++---++## Goal++Ingest documents into the knowledge base with proper version control.++---++## Document Preparation++### Supported Formats++- Markdown (.md)+- Text (.txt)+- Convert PDFs: `pdftotext document.pdf document.txt`++### Organize Documents++```+documents/+├── raw/+│ ├── 2024/+│ │ ├── 01/+│ │ │ ├── doc1.md+│ │ │ └── doc2.md+│ │ └── 02/+│ │ └── doc3.md+│ └── archive/+└── processed/ # Processing log+```++---++## Ingestion Methods++### Method 1: Initial Ingestion++First batch of documents:++```python+def initial_ingest(self, documents_dir: str):+ """Initial ingestion of documents."""+ print("Starting initial ingestion...")++ # Get all documents+ docs = list(Path(documents_dir).glob("**/*.md"))+ docs.extend(Path(documents_dir).glob("**/*.txt"))++ print(f"Found {len(docs)} documents")++ # Parse first document+ print(f"Processing: {docs[0].name}")+ text = docs[0].read_text(encoding="utf-8")+ kb = self.ka.parse(text)++ # Feed remaining documents+ for doc in docs[1:]:+ print(f"Adding: {doc.name}")+ text = doc.read_text(encoding="utf-8")+ kb.feed_text(text)++ # Build index+ print("Building search index...")+ kb.build_index()++ # Save version+ version_path = self.save_version(kb, "v1.0")++ # Log processing+ self._log_processing(docs, version_path)++ print(f"✓ Ingested {len(docs)} documents")+ print(f"✓ Knowledge base: {version_path}")++ return kb+```++### Method 2: Incremental Updates++Add new documents to existing KB:++```python+def add_documents(self, document_paths: list[str]):+ """Add new documents to existing knowledge base."""+ print("Loading current knowledge base...")++ # Load current version+ current_path = Path(self.config.kb_dir) / "current"+ kb = Template.create(self.config.template, self.config.language)+ kb.load(current_path)++ # Add new documents+ for path in document_paths:+ doc_path = Path(path)+ print(f"Adding: {doc_path.name}")++ text = doc_path.read_text(encoding="utf-8")+ kb.feed_text(text)++ # Rebuild index+ print("Rebuilding search index...")+ kb.build_index()++ # Save new version+ version = self._get_next_version()+ version_path = self.save_version(kb, version)++ print(f"✓ Added {len(document_paths)} documents")+ print(f"✓ New version: {version}")++ return kb++def _get_next_version(self) -> str:+ """Generate next version number."""+ current = Path(self.config.kb_dir) / "current"+ if not current.exists():+ return "v1.0"++ # Parse current version+ current_target = current.readlink().name+ if current_target.startswith("v"):+ try:+ parts = current_target[1:].split(".")+ major = int(parts[0])+ minor = int(parts[1])+ return f"v{major}.{minor + 1}"+ except:+ pass++ return datetime.now().strftime("v%Y%m%d_%H%M%S")+```++---++## Complete Ingestion Script++```python+"""Step 2: Document Ingestion."""++import argparse+from pathlib import Path+from kb_manager import KnowledgeBaseManager++def main():+ parser = argparse.ArgumentParser(description="Ingest documents into KB")+ parser.add_argument("--initial", action="store_true", help="Initial ingestion")+ parser.add_argument("--add", nargs="+", help="Add specific documents")+ parser.add_argument("--dir", default="./documents/raw", help="Documents directory")+ args = parser.parse_args()++ # Initialize manager+ manager = KnowledgeBaseManager()+ manager.initialize()++ if args.initial:+ # Initial ingestion+ kb = manager.initial_ingest(args.dir)++ # Print stats+ print("\nKnowledge Base Stats:")+ print(f" Entities: {len(kb.data.entities)}")+ print(f" Relations: {len(kb.data.relations)}")++ elif args.add:+ # Add specific documents+ kb = manager.add_documents(args.add)++ print("\nKnowledge Base Stats:")+ print(f" Entities: {len(kb.data.entities)}")+ print(f" Relations: {len(kb.data.relations)}")++if __name__ == "__main__":+ main()+```++### Usage++```bash+# Initial ingestion+python step2_ingest.py --initial++# Add specific documents+python step2_ingest.py --add documents/raw/2024/02/new_doc.md++# Add multiple documents+python step2_ingest.py --add doc1.md doc2.md doc3.md+```++---++## Processing Log++Track what was ingested:++```python+def _log_processing(self, documents: list[Path], version_path: Path):+ """Log processing details."""+ log_entry = {+ "timestamp": datetime.now().isoformat(),+ "version": version_path.name,+ "documents": [str(d) for d in documents],+ "document_count": len(documents)+ }++ log_file = Path("logs") / "ingestions.jsonl"+ log_file.parent.mkdir(exist_ok=True)++ with open(log_file, "a") as f:+ f.write(json.dumps(log_entry) + "\n")+```++---++## Best Practices++### 1. Batch Size++Process documents in batches:++```python+BATCH_SIZE = 10++for i in range(0, len(docs), BATCH_SIZE):+ batch = docs[i:i + BATCH_SIZE]+ for doc in batch:+ kb.feed_text(doc.read_text())++ # Save checkpoint+ if i % (BATCH_SIZE * 5) == 0:+ kb.dump(f"./kb/checkpoint_{i}/")+```++### 2. Error Handling++```python+try:+ text = doc.read_text(encoding="utf-8")+ kb.feed_text(text)+except Exception as e:+ print(f"Error processing {doc}: {e}")+ # Log error, continue with next+ continue+```++### 3. Validation++```python+def validate_ingestion(self, kb):+ """Validate knowledge base after ingestion."""+ assert not kb.empty(), "Knowledge base is empty"+ assert len(kb.data.entities) > 0, "No entities extracted"++ # Try to build index+ try:+ kb.build_index()+ except Exception as e:+ raise ValueError(f"Failed to build index: {e}")+```++---++## Next Step++→ [Step 3: Query and Maintain](step3-query.md)diff --git a/docs/en/tutorials/knowledge-base/step3-query.md b/docs/en/tutorials/knowledge-base/step3-query.mdnew file mode 100644index 0000000..8feec31--- /dev/null+++ b/docs/en/tutorials/knowledge-base/step3-query.md@@ -0,0 +1,354 @@+# Step 3: Query++Search, update, and maintain your knowledge base.++---++## Goal++Query the knowledge base and implement maintenance workflows.++---++## Query Interface++### Search Functionality++```python+def search(self, query: str, top_k: int = 5):+ """Search the knowledge base."""+ current_path = Path(self.config.kb_dir) / "current"++ kb = Template.create(self.config.template, self.config.language)+ kb.load(current_path)++ results = kb.search(query, top_k=top_k)+ return results+```++### Q&A Functionality++```python+def ask(self, question: str, top_k: int = 5):+ """Ask a question about the knowledge base."""+ current_path = Path(self.config.kb_dir) / "current"++ kb = Template.create(self.config.template, self.config.language)+ kb.load(current_path)++ response = kb.chat(question, top_k=top_k)+ return response.content+```++---++## Interactive Query Shell++```python+"""Step 3: Interactive Query Interface."""++import cmd+from kb_manager import KnowledgeBaseManager++class KBQueryShell(cmd.Cmd):+ intro = """+Knowledge Base Query Shell+==========================+Type 'help' for commands, 'quit' to exit++Commands:+ search <query> - Semantic search+ ask <question> - Ask a question+ stats - Show KB statistics+ versions - List versions+ backup - Create backup+"""+ prompt = "kb> "++ def __init__(self):+ super().__init__()+ self.manager = KnowledgeBaseManager()+ self.kb = None+ self._load_kb()++ def _load_kb(self):+ """Load current knowledge base."""+ current_path = Path(self.manager.config.kb_dir) / "current"+ if current_path.exists():+ self.kb = Template.create(+ self.manager.config.template,+ self.manager.config.language+ )+ self.kb.load(current_path)+ print(f"Loaded: {self.manager.config.name}")+ else:+ print("Warning: No knowledge base found")++ def do_search(self, arg):+ """Search the knowledge base: search <query>"""+ if not arg:+ print("Usage: search <query>")+ return++ if not self.kb:+ print("No knowledge base loaded")+ return++ results = self.kb.search(arg, top_k=5)++ print(f"\nFound {len(results)} results:\n")+ for i, item in enumerate(results, 1):+ if hasattr(item, 'name'):+ print(f"{i}. [{item.type}] {item.name}")+ elif hasattr(item, 'source'):+ print(f"{i}. {item.source} -> {item.target}")+ print()++ def do_ask(self, arg):+ """Ask a question: ask <question>"""+ if not arg:+ print("Usage: ask <question>")+ return++ if not self.kb:+ print("No knowledge base loaded")+ return++ print("\nThinking...")+ answer = self.kb.chat(arg).content+ print(f"\n{answer}\n")++ def do_stats(self, arg):+ """Show knowledge base statistics."""+ if not self.kb:+ print("No knowledge base loaded")+ return++ print("\nKnowledge Base Statistics:")+ print(f" Entities: {len(self.kb.data.entities)}")+ print(f" Relations: {len(self.kb.data.relations)}")+ print(f" Template: {self.manager.config.template}")++ # Entity types+ from collections import Counter+ types = Counter(e.type for e in self.kb.data.entities)+ print("\nEntity Types:")+ for t, count in types.most_common():+ print(f" {t}: {count}")+ print()++ def do_versions(self, arg):+ """List all versions."""+ kb_dir = Path(self.manager.config.kb_dir)+ versions = [d.name for d in kb_dir.iterdir() if d.is_dir()]++ print("\nVersions:")+ for v in sorted(versions):+ marker = " (current)" if v == "current" else ""+ print(f" {v}{marker}")+ print()++ def do_backup(self, arg):+ """Create backup of current version."""+ import shutil++ current = Path(self.manager.config.kb_dir) / "current"+ if not current.exists():+ print("No current version to backup")+ return++ timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")+ backup_path = Path(self.manager.config.backup_dir) / f"backup_{timestamp}"++ shutil.copytree(current, backup_path)+ print(f"✓ Backup created: {backup_path}\n")++ def do_quit(self, arg):+ """Exit the shell."""+ print("Goodbye!")+ return True++ do_exit = do_quit++if __name__ == "__main__":+ KBQueryShell().cmdloop()+```++---++## Maintenance Operations++### Backup Strategy++```python+def create_backup(self, name: Optional[str] = None):+ """Create a backup of current knowledge base."""+ import shutil++ current = Path(self.config.kb_dir) / "current"+ if not current.exists():+ raise ValueError("No current version to backup")++ if name is None:+ name = datetime.now().strftime("backup_%Y%m%d_%H%M%S")++ backup_path = Path(self.config.backup_dir) / name+ shutil.copytree(current, backup_path)++ print(f"✓ Backup created: {backup_path}")+ return backup_path++def restore_backup(self, backup_name: str):+ """Restore from backup."""+ import shutil++ backup_path = Path(self.config.backup_dir) / backup_name+ if not backup_path.exists():+ raise ValueError(f"Backup not found: {backup_name}")++ # Save current as version first+ current = Path(self.config.kb_dir) / "current"+ if current.exists():+ self._save_current_as_version("pre-restore")++ # Restore backup+ if current.exists():+ shutil.rmtree(current)+ shutil.copytree(backup_path, current)++ print(f"✓ Restored from: {backup_path}")+```++### Version Management++```python+def list_versions(self):+ """List all versions."""+ kb_dir = Path(self.config.kb_dir)+ versions = []++ for item in kb_dir.iterdir():+ if item.is_dir() and item.name != "current":+ # Get version info+ metadata_file = item / "metadata.json"+ if metadata_file.exists():+ import json+ with open(metadata_file) as f:+ metadata = json.load(f)+ versions.append({+ "name": item.name,+ "created": metadata.get("created_at", "unknown"),+ "updated": metadata.get("updated_at", "unknown")+ })++ return sorted(versions, key=lambda x: x["name"])++def rollback(self, version: str):+ """Rollback to a specific version."""+ version_path = Path(self.config.kb_dir) / version+ if not version_path.exists():+ raise ValueError(f"Version not found: {version}")++ # Update current symlink+ current_link = Path(self.config.kb_dir) / "current"+ if current_link.exists():+ current_link.unlink()+ current_link.symlink_to(version_path, target_is_directory=True)++ print(f"✓ Rolled back to: {version}")+```++### Cleanup++```python+def cleanup_old_versions(self, keep: int = 10):+ """Remove old versions, keeping only the most recent."""+ versions = self.list_versions()++ if len(versions) <= keep:+ print(f"No cleanup needed ({len(versions)} versions)")+ return++ to_remove = versions[:-keep]+ for v in to_remove:+ version_path = Path(self.config.kb_dir) / v["name"]+ shutil.rmtree(version_path)+ print(f"Removed: {v['name']}")++ print(f"✓ Cleaned up {len(to_remove)} old versions")+```++---++## Export and Reporting++### Export to JSON++```python+def export_to_json(self, output_file: str = "kb_export.json"):+ """Export knowledge base to JSON."""+ data = {+ "config": {+ "name": self.config.name,+ "template": self.config.template,+ "language": self.config.language+ },+ "data": self.kb.data.model_dump(),+ "metadata": self.kb.metadata+ }++ with open(output_file, "w") as f:+ json.dump(data, f, indent=2, default=str)++ print(f"✓ Exported to: {output_file}")+```++### Generate Report++```python+def generate_report(self):+ """Generate knowledge base report."""+ from collections import Counter++ report = []+ report.append("# Knowledge Base Report")+ report.append(f"\nName: {self.config.name}")+ report.append(f"Template: {self.config.template}")+ report.append(f"\nStatistics:")+ report.append(f"- Entities: {len(self.kb.data.entities)}")+ report.append(f"- Relations: {len(self.kb.data.relations)}")++ # Entity types+ types = Counter(e.type for e in self.kb.data.entities)+ report.append("\n## Entity Types")+ for t, count in types.most_common():+ report.append(f"- {t}: {count}")++ return "\n".join(report)+```++---++## Summary++You now have a complete knowledge base system with:++✓ Document ingestion with versioning+✓ Search and Q&A capabilities+✓ Backup and restore functionality+✓ Maintenance operations+✓ Export and reporting++### Next Steps++- Schedule regular backups+- Monitor knowledge base growth+- Set up automated ingestion pipelines+- Integrate with your applications++---++## See Also++- [Research Assistant Tutorial](../research-assistant/index.md)+- [Document Analysis Tutorial](../document-analysis/index.md)diff --git a/docs/en/tutorials/research-assistant/index.md b/docs/en/tutorials/research-assistant/index.mdnew file mode 100644index 0000000..6d195fb--- /dev/null+++ b/docs/en/tutorials/research-assistant/index.md@@ -0,0 +1,74 @@+# Research Assistant Tutorial++Build an interactive research assistant from academic papers.++---++## What You'll Build++By the end of this tutorial, you'll have a research assistant that can:+- Extract structured knowledge from research papers+- Answer questions about the paper's content+- Find relevant sections based on semantic search+- Visualize concept relationships++---++## Tutorial Overview++| Step | Topic | What You'll Learn |+|------|-------|-------------------|+| 1 | [Extract Knowledge](step1-extract.md) | Parse a research paper and extract concepts |+| 2 | [Semantic Search](step2-search.md) | Build a searchable knowledge base |+| 3 | [Q&A System](step3-chat.md) | Create an interactive Q&A interface |++---++## Prerequisites++- Hyper-Extract installed: `pip install hyper-extract`+- OpenAI API key configured+- A research paper (PDF or text format)++---++## Example Use Cases++### Use Case 1: Paper Review++Quickly understand a new paper by asking questions:+- "What are the main contributions?"+- "How does this compare to prior work?"+- "What are the limitations?"++### Use Case 2: Literature Review++Build a knowledge base from multiple papers:+- Extract concepts from 10+ papers+- Search across all papers+- Find connections between works++### Use Case 3: Teaching Assistant++Help students understand complex papers:+- Visualize concept maps+- Answer student questions+- Generate summaries++---++## Project Structure++```+research-assistant/+├── paper.md # Your research paper+├── knowledge_base/ # Extracted knowledge+├── research_assistant.py # Main application+└── requirements.txt # Dependencies+```++---++## Next Steps++Ready to start? Begin with [Step 1: Extract Knowledge](step1-extract.md).diff --git a/docs/en/tutorials/research-assistant/step1-extract.md b/docs/en/tutorials/research-assistant/step1-extract.mdnew file mode 100644index 0000000..a5f742b--- /dev/null+++ b/docs/en/tutorials/research-assistant/step1-extract.md@@ -0,0 +1,193 @@+# Step 1: Extract Knowledge++Parse a research paper and extract structured concepts.++---++## Goal++Extract entities, relationships, and concepts from a research paper into a structured knowledge graph.++---++## Preparation++### 1. Get a Research Paper++Download a paper or use your own. For this tutorial, we'll use a sample:++```bash+# Download a sample paper (or use your own)+curl -o paper.md https://arxiv.org/abs/1706.03762 # Attention Is All You Need+```++### 2. Convert to Text (if needed)++If you have a PDF:++```bash+pdftotext paper.pdf paper.md+```++---++## Extract Using CLI++### Basic Extraction++```bash+he parse paper.md -t general/concept_graph -o ./paper_kb/ -l en+```++**What this does:**+- Reads the paper+- Extracts concepts and their relationships+- Saves to `./paper_kb/`++### Verify Extraction++```bash+he info ./paper_kb/+```++Expected output:+```+Knowledge Abstract Info++Path ./paper_kb/+Template general/concept_graph+Language en+Nodes 25+Edges 32+Index Built+```++### Visualize++```bash+he show ./paper_kb/+```++---++## Extract Using Python++### Script++```python+"""Step 1: Extract knowledge from research paper."""++from dotenv import load_dotenv+load_dotenv()++from hyperextract import Template+from pathlib import Path++# Configuration+PAPER_FILE = "paper.md"+OUTPUT_DIR = "./paper_kb/"++def main():+ # Create template+ print("Creating concept extraction template...")+ ka = Template.create("general/concept_graph", language="en")++ # Read paper+ print(f"Reading: {PAPER_FILE}")+ text = Path(PAPER_FILE).read_text(encoding="utf-8")++ # Extract knowledge+ print("Extracting concepts and relationships...")+ result = ka.parse(text)++ # Display results+ print(f"\nExtraction Complete:")+ print(f" Entities: {len(result.data.entities)}")+ print(f" Relations: {len(result.data.relations)}")++ # Show sample entities+ print("\nSample Concepts:")+ for entity in result.data.entities[:5]:+ print(f" - {entity.name} ({entity.type})")++ # Save+ print(f"\nSaving to: {OUTPUT_DIR}")+ result.dump(OUTPUT_DIR)++ # Build index for next step+ print("Building search index...")+ result.build_index()+ result.dump(OUTPUT_DIR)++ print("\n✓ Step 1 complete!")+ print(f" Knowledge base: {OUTPUT_DIR}")+ print(f"\nNext: Run 'python step2_search.py'")++if __name__ == "__main__":+ main()+```++### Run++```bash+python step1_extract.py+```++---++## Understanding the Output++### What Was Extracted?++The concept graph template extracts:++**Entities:**+- Concepts (models, algorithms, techniques)+- Authors+- Datasets+- Metrics++**Relations:**+- `uses` — Concept uses another+- `improves_upon` — Improvement relationships+- `evaluated_on` — Evaluation datasets+- `achieves` — Results/metrics++### Example Output++```python+# Entities+[+ {"name": "Transformer", "type": "model"},+ {"name": "Attention Mechanism", "type": "concept"},+ {"name": "BLEU Score", "type": "metric"}+]++# Relations+[+ {"source": "Transformer", "target": "Attention Mechanism", "type": "uses"},+ {"source": "Transformer", "target": "BLEU Score", "type": "achieves"}+]+```++---++## Troubleshooting++### "No entities extracted"++- Check paper is not empty: `wc -l paper.md`+- Try different template: `general/knowledge_graph`+- Check language setting matches document++### "Extraction is slow"++- Long papers are chunked automatically+- Each chunk requires an LLM call+- Consider using `--no-index` and building later++---++## Next Step++→ [Step 2: Semantic Search](step2-search.md)diff --git a/docs/en/tutorials/research-assistant/step2-search.md b/docs/en/tutorials/research-assistant/step2-search.mdnew file mode 100644index 0000000..ef9e535--- /dev/null+++ b/docs/en/tutorials/research-assistant/step2-search.md@@ -0,0 +1,184 @@+# Step 2: Semantic Search++Build a searchable knowledge base.++---++## Goal++Enable semantic search over the extracted knowledge to find relevant concepts and sections.++---++## Why Semantic Search?++Unlike keyword search, semantic search:+- Understands context and meaning+- Finds relevant content even with different words+- Handles synonyms and related concepts++---++## Using CLI++### Basic Search++```bash+he search ./paper_kb/ "attention mechanism"+```++### Natural Language Queries++```bash+he search ./paper_kb/ "What are the main contributions?"+```++### Get More Results++```bash+he search ./paper_kb/ "transformer architecture" -n 10+```++---++## Using Python++### Script++```python+"""Step 2: Semantic search over knowledge base."""++from dotenv import load_dotenv+load_dotenv()++from hyperextract import Template++KB_DIR = "./paper_kb/"++def main():+ # Load knowledge base+ print("Loading knowledge base...")+ ka = Template.create("general/concept_graph", language="en")+ ka.load(KB_DIR)++ # Ensure index is built+ print("Building search index...")+ ka.build_index()++ # Interactive search loop+ print("\n" + "="*50)+ print("Semantic Search Interface")+ print("="*50)+ print("Type your query (or 'quit' to exit)\n")++ while True:+ query = input("Search: ").strip()++ if query.lower() in ['quit', 'exit', 'q']:+ break++ if not query:+ continue++ # Search+ results = ka.search(query, top_k=5)++ # Display results+ print(f"\nFound {len(results)} results:\n")++ for i, item in enumerate(results, 1):+ if hasattr(item, 'name'): # Entity+ print(f"{i}. [{item.type}] {item.name}")+ if hasattr(item, 'description'):+ print(f" {item.description[:100]}...")+ elif hasattr(item, 'source'): # Relation+ print(f"{i}. {item.source} --{item.type}--> {item.target}")+ print()++ print("-" * 50)++ print("\n✓ Step 2 complete!")+ print("Next: Run 'python step3_chat.py'")++if __name__ == "__main__":+ main()+```++### Run++```bash+python step2_search.py+```++**Example Session:**+```+Search: transformer architecture++Found 3 results:++1. [model] Transformer+ A neural network architecture based on attention...++2. [concept] Self-Attention+ Mechanism allowing the model to weigh input tokens...++3. [concept] Multi-Head Attention+ Multiple attention heads operating in parallel...++--------------------------------------------------+Search: quit+```++---++## Search Tips++### Effective Queries++| Query Type | Example | Use When |+|------------|---------|----------|+| Concept | "attention mechanism" | Finding specific concepts |+| Question | "What are the main contributions?" | Broad exploration |+| Comparison | "transformer vs rnn" | Finding comparisons |+| Results | "bleu score results" | Finding specific data |++### Improving Results++1. **Be specific**: "transformer encoder" vs "transformer"+2. **Use natural language**: "How does attention work?"+3. **Try synonyms**: "attention" → "self-attention" → "query-key-value"+4. **Increase top_k**: `top_k=10` for broader results++---++## Building Search into Your App++```python+class ResearchSearch:+ def __init__(self, kb_path):+ self.ka = Template.create("general/concept_graph", "en")+ self.ka.load(kb_path)+ self.ka.build_index()++ def find_concepts(self, query, n=5):+ """Find concepts related to query."""+ results = self.ka.search(query, top_k=n)+ return [r for r in results if hasattr(r, 'name')]++ def find_relationships(self, concept):+ """Find relationships for a concept."""+ return [+ r for r in self.ka.data.relations+ if r.source == concept or r.target == concept+ ]++# Usage+search = ResearchSearch("./paper_kb/")+concepts = search.find_concepts("attention", n=10)+```++---++## Next Step++→ [Step 3: Q&A System](step3-chat.md)diff --git a/docs/en/tutorials/research-assistant/step3-chat.md b/docs/en/tutorials/research-assistant/step3-chat.mdnew file mode 100644index 0000000..a15a5f4--- /dev/null+++ b/docs/en/tutorials/research-assistant/step3-chat.md@@ -0,0 +1,298 @@+# Step 3: Q&A System++Create an interactive Q&A interface.++---++## Goal++Build a conversational interface that answers questions about the research paper using natural language.++---++## How It Works++1. User asks a question+2. System retrieves relevant context from knowledge base+3. LLM generates answer based on retrieved context+4. Response includes citations from the paper++---++## Using CLI++### Single Question++```bash+he talk ./paper_kb/ -q "What are the main contributions of this paper?"+```++### Interactive Mode++```bash+he talk ./paper_kb/ -i+```++**Example Session:**+```+Entering interactive mode. Type 'exit' or 'quit' to stop.++> What is the Transformer architecture?++The Transformer is a novel neural network architecture introduced in+this paper that relies entirely on attention mechanisms, dispensing+with recurrence and convolutions entirely...++> How does it compare to RNNs?++Unlike RNNs, which process sequences sequentially, the Transformer+processes all positions in parallel, making it more efficient for+training on large datasets...++> exit+```++---++## Using Python++### Script++```python+"""Step 3: Interactive Q&A system."""++from dotenv import load_dotenv+load_dotenv()++from hyperextract import Template++KB_DIR = "./paper_kb/"++class ResearchAssistant:+ def __init__(self, kb_path):+ print("Loading research assistant...")+ self.ka = Template.create("general/concept_graph", "en")+ self.ka.load(kb_path)+ self.ka.build_index()+ print("✓ Ready!\n")++ def ask(self, question, top_k=5):+ """Ask a question about the paper."""+ response = self.ka.chat(question, top_k=top_k)+ return response.content++ def interactive(self):+ """Run interactive Q&A session."""+ print("="*60)+ print("Research Assistant - Ask me about the paper!")+ print("="*60)+ print("Type 'quit' to exit\n")++ while True:+ question = input("\nQ: ").strip()++ if question.lower() in ['quit', 'exit', 'q']:+ break++ if not question:+ continue++ print("\nThinking...")+ answer = self.ask(question)+ print(f"\nA: {answer}\n")+ print("-"*60)++ print("\nGoodbye!")++def main():+ assistant = ResearchAssistant(KB_DIR)+ assistant.interactive()++if __name__ == "__main__":+ main()+```++### Run++```bash+python step3_chat.py+```++---++## Example Questions++### Understanding the Paper++- "What problem does this paper solve?"+- "What are the main contributions?"+- "What is the key innovation?"++### Technical Details++- "Explain the attention mechanism"+- "What is multi-head attention?"+- "How is positional encoding handled?"++### Results and Evaluation++- "What datasets were used?"+- "What were the main results?"+- "How does it compare to baseline methods?"++### Limitations and Future Work++- "What are the limitations?"+- "What future work is suggested?"++---++## Advanced Features++### Citation Tracking++```python+def ask_with_citations(assistant, question):+ """Get answer with source citations."""+ response = assistant.ka.chat(question, top_k=5)++ print(f"Answer: {response.content}\n")++ # Show retrieved sources+ if "retrieved_items" in response.additional_kwargs:+ print("Sources:")+ for item in response.additional_kwargs["retrieved_items"]:+ if hasattr(item, 'name'):+ print(f" - {item.name}")+```++### Question Suggestions++```python+def suggest_questions(assistant):+ """Suggest questions based on paper content."""+ suggestions = [+ "What are the main contributions?",+ "What problem does this solve?",+ "What are the key results?",+ "How does this compare to prior work?",+ ]+ return suggestions+```++### Exporting Q&A History++```python+def export_history(questions_answers, filename="qa_history.json"):+ """Export Q&A session to file."""+ import json+ with open(filename, 'w') as f:+ json.dump(questions_answers, f, indent=2)+```++---++## Complete Application++Putting it all together:++```python+"""Complete Research Assistant Application."""++from hyperextract import Template+from pathlib import Path++class ResearchAssistantApp:+ def __init__(self, paper_path, kb_dir="./paper_kb/"):+ self.paper_path = paper_path+ self.kb_dir = kb_dir+ self.ka = None++ # Load or create knowledge base+ if Path(kb_dir).exists():+ self._load_kb()+ else:+ self._create_kb()++ def _create_kb(self):+ """Create knowledge base from paper."""+ print("Creating knowledge base...")+ ka = Template.create("general/concept_graph", "en")+ text = Path(self.paper_path).read_text()+ self.ka = ka.parse(text)+ self.ka.build_index()+ self.ka.dump(self.kb_dir)+ print("✓ Knowledge base created\n")++ def _load_kb(self):+ """Load existing knowledge base."""+ print("Loading knowledge base...")+ self.ka = Template.create("general/concept_graph", "en")+ self.ka.load(self.kb_dir)+ print("✓ Knowledge base loaded\n")++ def search(self, query):+ """Search the paper."""+ return self.ka.search(query)++ def ask(self, question):+ """Ask a question."""+ return self.ka.chat(question).content++ def visualize(self):+ """Open visualization."""+ self.ka.show()++ def run(self):+ """Run interactive session."""+ print("="*60)+ print("Research Assistant")+ print("Commands: search, ask, visualize, quit")+ print("="*60)++ while True:+ cmd = input("\n> ").strip().lower()++ if cmd == "quit":+ break+ elif cmd == "search":+ query = input("Search query: ")+ results = self.search(query)+ for r in results[:5]:+ print(f" - {r.name if hasattr(r, 'name') else r}")+ elif cmd == "ask":+ question = input("Question: ")+ answer = self.ask(question)+ print(f"\n{answer}")+ elif cmd == "visualize":+ self.visualize()++if __name__ == "__main__":+ app = ResearchAssistantApp("paper.md")+ app.run()+```++---++## Summary++Congratulations! You've built a complete research assistant that can:++✓ Extract knowledge from research papers+✓ Search using semantic queries+✓ Answer questions in natural language+✓ Visualize concept relationships++### Next Steps++- Try with different papers+- Experiment with different templates+- Build a web interface using Streamlit or Flask+- Process multiple papers and compare them++---++## See Also++- [Knowledge Base Tutorial](../knowledge-base/index.md)+- [Document Analysis Tutorial](../document-analysis/index.md)diff --git a/docs/zh/cli/cli.md b/docs/zh/cli/cli.mddeleted file mode 100644index 84c6ea8..0000000--- a/docs/zh/cli/cli.md+++ /dev/null@@ -1,162 +0,0 @@-# CLI 指南--Hyper-Extract CLI 提供了强大的命令行界面用于知识提取。--## 安装--CLI 已包含在主包中:--```bash-uv pip install hyper-extract-```--## 基本用法--### 初始化配置--```bash-he config init -k <your-api-key>-```--### 解析文档--```bash-he parse sample.txt -o output/-```--### 搜索提取的知识--```bash-he search output/ "关键事件有哪些?"-```--### 添加新文档--```bash-he feed output/ another_sample.txt-```--## 命令--### config--管理配置:--```bash-# 初始化-he config init -k <your-api-key>--# 显示当前配置-he config show--# 更新配置-he config set llm.model gpt-4o-mini--# 列出可用模板-he config list-templates-```--### parse--从文档中提取知识:--```bash-# 基本用法-he parse input.txt -o output/--# 使用模板-he parse input.txt -t finance/earnings_summary -o output/--# 指定语言-he parse input.txt -l zh -o output/--# 批量处理-he parse ./documents/ -o output/ --batch-```--### search--查询提取的知识:--```bash-# 基本搜索-he search output/ "发生了什么?"--# 使用过滤器-he search output/ "财务事件" --filter type=Event--# 限制结果数量-he search output/ "公司" --limit 10-```--### feed--向现有知识库添加新文档:--```bash-# 添加单个文档-he feed output/ new_doc.txt--# 添加多个文档-he feed output/ ./new_docs/-```--### list--列出可用模板和预设:--```bash-# 列出所有模板-he list--# 按类别列出-he list --category finance--# 列出领域模板-he list --domain general-```--## 选项--### 全局选项--| 选项 | 描述 |-|------|------|-| `--help` | 显示帮助信息 |-| `--version` | 显示版本号 |-| `--verbose` | 启用详细输出 |-| `--config PATH` | 使用自定义配置文件 |--### Parse 选项--| 选项 | 描述 |-|------|------|-| `-o, --output PATH` | 输出目录 |-| `-t, --template NAME` | 要使用的模板 |-| `-l, --language CODE` | 文档语言(en、zh 等)|-| `--batch` | 启用批量处理 |--## 示例--### 金融文档提取--```bash-he parse annual_report.pdf -t finance/earnings_summary -o ./results/-```--### 法律文档提取--```bash-he parse contract.txt -t legal/case_facts -o ./legal_output/-```--### 多语言支持--```bash-he parse chinese_doc.txt -l zh -o ./output/-```--## 下一步--- 探索 [Python API](../python/python-api.md)-- 浏览 [领域模板](../templates/index.md)diff --git a/docs/zh/cli/commands/build-index.md b/docs/zh/cli/commands/build-index.mdnew file mode 100644index 0000000..cea1e8f--- /dev/null+++ b/docs/zh/cli/commands/build-index.md@@ -0,0 +1,202 @@+# he build-index++为知识库构建或重建向量搜索索引。++---++## 概要++```bash+he build-index KA_PATH [OPTIONS]+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `KA_PATH` | 知识库目录的路径 |++## 选项++| 选项 | 简写 | 描述 |+|--------|-------|-------------|+| `--force` | `-f` | 即使索引存在也强制重建 |++---++## 描述++为语义搜索和聊天功能构建向量搜索索引:++1. **读取所有实体/关系** — 从知识库数据中+2. **生成嵌入** — 使用配置的嵌入模型+3. **构建 FAISS 索引** — 用于快速相似度搜索+4. **保存到磁盘** — 在 `index/` 子目录中++**必需用于**:`he search` 和 `he talk` 命令。++---++## 示例++### 构建索引++```bash+he build-index ./output/+```++**输出:**+```+Template: general/biography_graph+Language: en++Success! Index built for ./output/++Now you can:+ he search ./output/ "keyword" # Semantic search+ he talk ./output/ -i # Interactive chat+```++### 强制重建++如果索引损坏或您想重建:++```bash+he build-index ./output/ -f+```++---++## 何时构建索引++### Parse 后(默认)++默认情况下,`he parse` 会自动构建索引:++```bash+he parse doc.md -t general/biography_graph -o ./kb/ -l en+# 索引自动构建+```++如果不需要搜索/聊天,使用 `--no-index` 跳过:++```bash+he parse doc.md -t general/biography_graph -o ./kb/ -l en --no-index+```++### Feed 后++喂养新文档后始终重建:++```bash+he feed ./kb/ new_doc.md+he build-index ./kb/+```++### 手动更改后++如果您手动修改了 `data.json`:++```bash+he build-index ./kb/ -f+```++---++## 索引存储++索引存储在知识库目录中:++```+./kb/+├── data.json+├── metadata.json+└── index/ # 索引目录+ ├── index.faiss # FAISS 向量索引+ └── docstore.json # 文档存储映射+```++---++## 性能++### 构建时间++| 知识库大小 | 预计构建时间 |+|---------------------|----------------------|+| 小型(< 100 项) | < 5 秒 |+| 中型(100-1000) | 5-30 秒 |+| 大型(1000+) | 30+ 秒 |++### 搜索速度++一旦构建完成,搜索非常快:++```bash+he search ./kb/ "query" # 通常 < 1 秒+```++---++## 最佳实践++1. **Feed 后构建** — `he feed` 后索引会变旧+2. **批量处理使用 `--no-index`** — 所有解析后统一构建一次+3. **有问题时强制重建** — 如果搜索返回意外结果,使用 `-f`+4. **备份大型索引** — `index/` 目录可能很大++---++## 批量工作流程++高效处理多个文档:++```bash+# 解析所有文件,不构建索引+he parse doc1.md -t general/biography_graph -o ./kb/ -l en --no-index+he feed ./kb/ doc2.md+he feed ./kb/ doc3.md++# 最后统一构建索引+he build-index ./kb/++# 现在可以进行搜索/聊天+he search ./kb/ "query"+he talk ./kb/ -q "question"+```++---++## 故障排除++### "索引已存在"++使用 `-f` 强制重建:++```bash+he build-index ./kb/ -f+```++### "索引构建失败"++检查:+1. 知识库有数据:`he info ./kb/`+2. API 密钥已配置:`he config show`+3. 有足够的磁盘空间存储索引++### 搜索仍然不工作++尝试强制重建:++```bash+he build-index ./kb/ -f+```++---++## 另请参见++- [`he parse`](parse.md) — 可选索引构建的解析+- [`he feed`](feed.md) — 添加文档(需要重建)+- [`he search`](search.md) — 搜索索引+- [`he talk`](talk.md) — 使用索引聊天diff --git a/docs/zh/cli/commands/config.md b/docs/zh/cli/commands/config.mdnew file mode 100644index 0000000..6f192c1--- /dev/null+++ b/docs/zh/cli/commands/config.md@@ -0,0 +1,233 @@+# he config++管理 Hyper-Extract 配置。++---++## 概要++```bash+he config [COMMAND] [OPTIONS]+```++## 命令++| 命令 | 描述 |+|---------|-------------|+| `init` | 初始化配置 |+| `show` | 显示当前配置 |+| `set` | 设置配置值 |+| `get` | 获取配置值 |++---++## he config init++使用 API 凭证初始化配置。++```bash+he config init [OPTIONS]+```++### 选项++| 选项 | 简写 | 描述 |+|--------|-------|-------------|+| `--key` | `-k` | OpenAI API 密钥 |+| `--base-url` | `-u` | API 基础 URL(可选) |+| `--model` | `-m` | 默认模型(默认:gpt-4o-mini) |+| `--embedder` | `-e` | 嵌入模型(默认:text-embedding-3-small) |++### 示例++#### 基本设置++```bash+he config init -k sk-your-api-key-here+```++#### 使用自定义基础 URL++```bash+he config init -k sk-your-key -u https://api.openai.com/v1+```++#### 使用自定义模型++```bash+he config init -k sk-your-key -m gpt-4o -e text-embedding-3-large+```++#### 交互模式++```bash+he config init+# 交互式提示输入 API 密钥+```++---++## he config show++显示当前配置。++```bash+he config show+```++**输出:**+```+Configuration (from ~/.he/config.toml):++[llm]+model = "gpt-4o-mini"+api_key = "sk-...***"+base_url = "https://api.openai.com/v1"++[embedder]+model = "text-embedding-3-small"+api_key = "sk-...***"++[defaults]+language = "en"+chunk_size = 2048+max_workers = 10+```++---++## he config set++设置配置值。++```bash+he config set <KEY> <VALUE>+```++### 示例++```bash+# 更改默认模型+he config set llm.model gpt-4o++# 更改嵌入模型+he config set embedder.model text-embedding-3-large++# 设置默认语言+he config set defaults.language zh+```++---++## he config get++获取配置值。++```bash+he config get <KEY>+```++### 示例++```bash+he config get llm.model+# Output: gpt-4o-mini++he config get defaults.language+# Output: en+```++---++## 配置文件++配置存储在:++- **Linux/macOS**: `~/.he/config.toml`+- **Windows**: `%USERPROFILE%\.he\config.toml`++### 配置示例++```toml+[llm]+model = "gpt-4o-mini"+api_key = "sk-your-api-key"+base_url = "https://api.openai.com/v1"++[embedder]+model = "text-embedding-3-small"+api_key = "sk-your-api-key"++[defaults]+language = "en"+chunk_size = 2048+chunk_overlap = 256+max_workers = 10+verbose = false+```++---++## 环境变量++也可以通过环境变量设置配置(更高优先级):++| 变量 | 描述 |+|----------|-------------|+| `OPENAI_API_KEY` | LLM 和嵌入器的 API 密钥 |+| `OPENAI_BASE_URL` | 自定义 API 基础 URL |+| `HE_LLM_MODEL` | 默认 LLM 模型 |+| `HE_EMBEDDER_MODEL` | 默认嵌入模型 |++---++## 优先级顺序++配置按以下顺序解析(最高优先):++1. 命令行参数+2. 环境变量+3. 配置文件+4. 内置默认值++---++## 故障排除++### "未找到 API 密钥"++初始化配置:++```bash+he config init -k your-api-key+```++或设置环境变量:++```bash+export OPENAI_API_KEY=your-api-key+```++### "API 密钥无效"++检查您的 API 密钥:++```bash+he config show+# 验证密钥是否正确+```++### "未找到配置文件"++运行 init 创建:++```bash+he config init -k your-api-key+```++---++## 另请参见++- [配置参考](../configuration.md) — 详细配置选项+- [安装指南](../../getting-started/installation.md) — 初始设置diff --git a/docs/zh/cli/commands/feed.md b/docs/zh/cli/commands/feed.mdnew file mode 100644index 0000000..3dc53a4--- /dev/null+++ b/docs/zh/cli/commands/feed.md@@ -0,0 +1,180 @@+# he feed++增量地向现有知识库添加文档。++---++## 概要++```bash+he feed KA_PATH INPUT [OPTIONS]+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `KA_PATH` | 现有知识库目录的路径 |+| `INPUT` | 输入文件路径或 `-` 表示标准输入 |++## 选项++| 选项 | 简写 | 描述 |+|--------|-------|-------------|+| `--template` | `-t` | 覆盖模板(如果省略则使用元数据) |+| `--lang` | `-l` | 覆盖语言(如果省略则使用元数据) |++---++## 描述++`feed` 命令向现有知识库添加新文档,不会丢失现有数据:++1. **加载现有知识** — 读取当前知识库状态+2. **从新文档提取** — 处理新内容+3. **智能合并** — 合并新旧数据,处理重复+4. **更新元数据** — 记录更新时间戳++这非常适合:+- 随着时间构建知识库+- 向现有文档添加更新+- 组合多个来源的信息++---++## 示例++### 基本用法++```bash+# 初始提取+he parse tesla_bio.md -t general/biography_graph -o ./tesla_kb/ -l en++# 添加更多内容+he feed ./tesla_kb/ tesla_inventions.md+```++### 喂养多个文档++```bash+he feed ./kb/ doc1.md+he feed ./kb/ doc2.md+he feed ./kb/ doc3.md+```++或使用循环:++```bash+for file in updates/*.md; do+ he feed ./kb/ "$file"+done+```++### 从标准输入++```bash+cat new_content.md | he feed ./kb/ -+```++---++## 合并行为++合并过程处理:++| 场景 | 行为 |+|----------|----------|+| 相同实体 | 合并,描述组合 |+| 相同关系 | 使用最新信息更新 |+| 新实体 | 添加到知识库 |+| 新关系 | 添加连接现有/新实体 |++---++## 工作流程示例++### 构建研究知识库++```bash+# 第 1 天:初始论文+he parse paper_v1.md -t general/concept_graph -o ./research_kb/ -l en+he show ./research_kb/++# 第 7 天:更新版本+he feed ./research_kb/ paper_v2.md+he show ./research_kb/++# 第 14 天:相关工作+he feed ./research_kb/ related_work.md+he build-index ./research_kb/+he talk ./research_kb/ -q "所有论文中的关键概念是什么?"+```++### 增量传记++```bash+# 从早年生活开始+he parse early_life.md -t general/biography_graph -o ./bio_kb/ -l en++# 添加职业生涯+he feed ./bio_kb/ career.md++# 添加晚年+he feed ./bio_kb/ later_years.md++# 最终可视化+he show ./bio_kb/+```++---++## 验证++检查 feed 是否成功:++```bash+he info ./kb/+```++查看:+- 节点数增加+- 边数增加+- 时间戳更新++---++## 最佳实践++1. **使用相同模板** — Feed 应使用兼容的模板+2. **匹配语言** — 为获得最佳效果,使用一致的语言+3. **之后重建索引** — 搜索/聊天需要 `he build-index ./kb/`+4. **可视化更改** — 使用 `he show ./kb/` 查看更新++---++## 错误处理++### "不是有效的知识库目录"++目录不包含有效的知识库。检查:++```bash+ls ./kb/+# 应该包含:data.json, metadata.json+```++### "模板不匹配"++Feed 最适合使用相同类型的模板。如有需要可以覆盖:++```bash+he feed ./kb/ doc.md -t general/biography_graph+```++---++## 另请参见++- [`he parse`](parse.md) — 创建新知识库+- [`he info`](info.md) — 查看知识库统计信息+- [`he build-index`](build-index.md) — Feed 后重建搜索索引diff --git a/docs/zh/cli/commands/info.md b/docs/zh/cli/commands/info.mdnew file mode 100644index 0000000..cec265d--- /dev/null+++ b/docs/zh/cli/commands/info.md@@ -0,0 +1,172 @@+# he info++显示知识库的信息和统计信息。++---++## 概要++```bash+he info KA_PATH+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `KA_PATH` | 知识库目录的路径 |++---++## 描述++显示知识库的元数据和统计信息:++- **路径** — 知识库的位置+- **模板** — 用于提取的模板+- **语言** — 语言代码(en/zh)+- **时间戳** — 创建和最后更新时间+- **统计信息** — 节点数、边数+- **索引状态** — 搜索索引是否已构建++---++## 示例++### 基本用法++```bash+he info ./output/+```++**输出:**+```+Knowledge Abstract Info++Path ./output/+Template general/biography_graph+Language en+Created 2024-01-15 10:30:00+Updated 2024-01-15 10:35:22+Nodes 25+Edges 32+Index Built+```++### 提取后++```bash+he parse tesla.md -t general/biography_graph -o ./kb/ -l en+he info ./kb/+```++### Feed 后++```bash+he feed ./kb/ more_content.md+he info ./kb/+# 注意更新的节点/边数和时间戳+```++---++## 输出字段++| 字段 | 描述 |+|-------|-------------|+| `Path` | 知识库的绝对路径 |+| `Template` | 模板标识符(例如 `general/biography_graph`) |+| `Language` | 处理语言(`en` 或 `zh`) |+| `Created` | 初始提取时间戳 |+| `Updated` | 最后修改时间戳 |+| `Nodes` | 实体/项目数量 |+| `Edges` | 关系数量 |+| `Index` | 搜索索引状态(`Built` 或 `Not Built`) |++---++## 用例++### 验证提取++检查提取是否成功:++```bash+he info ./kb/+# 应该显示 Nodes > 0, Edges > 0+```++### 检查索引状态++在使用搜索或聊天之前:++```bash+he info ./kb/+# 如果索引显示 "Not Built",运行:+he build-index ./kb/+```++### 监控增长++跟踪知识库随时间的增长:++```bash+he info ./kb/+# 喂养更多文档+he feed ./kb/ update.md+he info ./kb/+# 比较节点/边数+```++### 脚本++在自动化脚本中使用:++```bash+#!/bin/bash++if he info ./kb/ 2>/dev/null; then+ echo "Knowledge base exists"+ he feed ./kb/ new_doc.md+else+ echo "Creating new knowledge base"+ he parse new_doc.md -t general/biography_graph -o ./kb/ -l en+fi+```++---++## 故障排除++### "未找到知识库目录"++目录不存在或不包含知识库:++```bash+# 检查目录存在+ls ./kb/++# 应该包含:+# - data.json+# - metadata.json+```++### 元数据缺失++如果元数据损坏,模板/语言显示为 "unknown":++```bash+he info ./kb/+# Template: [yellow]unknown[/yellow]+# Language: [yellow]unknown[/yellow]+```++知识库仍然可用,但某些功能可能受限。++---++## 另请参见++- [`he parse`](parse.md) — 创建知识库+- [`he feed`](feed.md) — 添加到知识库+- [`he build-index`](build-index.md) — 构建搜索索引diff --git a/docs/zh/cli/commands/list.md b/docs/zh/cli/commands/list.mdnew file mode 100644index 0000000..b2acb33--- /dev/null+++ b/docs/zh/cli/commands/list.md@@ -0,0 +1,149 @@+# he list++列出可用的模板和方法。++---++## 概要++```bash+he list [COMMAND]+```++## 命令++| 命令 | 描述 |+|---------|-------------|+| `template` | 列出知识模板 |+| `method` | 列出提取方法 |++---++## he list template++显示所有可用的知识模板。++```bash+he list template+```++**输出:**+```+Available Templates:++Path Type Description+general/base_model model Basic model extraction+general/base_list list Basic list extraction+general/base_graph graph Basic graph extraction+general/biography_graph temporal_graph Biography graph with timeline+finance/earnings_summary model Financial earnings summary+finance/ownership_graph graph Company ownership structure+legal/contract_obligation list Contract obligations+medicine/drug_interaction graph Drug interaction network+...+```++### 按类型筛选++模板按输出类型组织:++| 类型 | 模板 |+|------|-----------|+| `model` | 结构化数据提取 |+| `list` | 有序集合 |+| `set` | 去重集合 |+| `graph` | 实体关系网络 |+| `hypergraph` | 多实体关系 |+| `temporal_graph` | 基于时间的关系 |+| `spatial_graph` | 基于位置的关系 |+| `spatio_temporal_graph` | 时间 + 空间组合 |++---++## he list method++显示所有可用的提取方法。++```bash+he list method+```++**输出:**+```+Available Methods:++Name Type Description+graph_rag graph Graph-RAG with Community detection+light_rag graph Lightweight Graph-based RAG+hyper_rag hypergraph Hypergraph-based RAG+hypergraph_rag hypergraph Advanced Hypergraph RAG+cog_rag hypergraph Cognitive RAG+itext2kg graph iText2KG: Triple-based extraction+itext2kg_star graph Enhanced iText2KG+kg_gen graph Knowledge Graph Generator+atom graph Temporal knowledge graph+```++### 方法类别++| 类别 | 方法 | 最佳用途 |+|----------|---------|----------|+| **基于 RAG** | `graph_rag`, `light_rag`, `hyper_rag` | 大型文档、检索增强 |+| **典型方法** | `itext2kg`, `kg_gen`, `atom` | 直接提取、特定功能 |++---++## 用例++### 查找模板++```bash+# 查看所有选项+he list template++# 然后使用一个+he parse doc.md -t general/biography_graph -o ./out/ -l en+```++### 选择方法++```bash+# 查看可用方法+he list method++# 使用一个而不是模板+he parse doc.md -m light_rag -o ./out/+```++### 脚本++```bash+# 统计可用模板数量+he list template | wc -l++# 按领域筛选模板+he list template | grep finance++# 检查特定方法是否存在+he list method | grep -q light_rag && echo "Available"+```++---++## 模板 vs 方法++| 功能 | 模板 | 方法 |+|---------|-----------|---------|+| **目的** | 特定领域提取 | 通用算法 |+| **配置** | 预配置提示 | 算法参数 |+| **语言** | 多语言支持 | 仅英文 |+| **用例** | 快速开始、领域任务 | 研究、定制 |+| **示例** | `finance/earnings_summary` | `light_rag` |++---++## 另请参见++- [`he parse`](parse.md) — 使用模板/方法+- [模板库](../../templates/index.md) — 详细模板文档+- [选择方法](../../concepts/methods.md) — 方法选择指南diff --git a/docs/zh/cli/commands/parse.md b/docs/zh/cli/commands/parse.mdnew file mode 100644index 0000000..b59831b--- /dev/null+++ b/docs/zh/cli/commands/parse.md@@ -0,0 +1,211 @@+# he parse++从文档中提取知识并保存到知识库。++---++## 概要++```bash+he parse INPUT [OPTIONS]+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `INPUT` | 输入文件路径、目录或 `-` 表示标准输入 |++## 选项++| 选项 | 简写 | 描述 |+|--------|-------|-------------|+| `--output` | `-o` | 输出目录(必填) |+| `--template` | `-t` | 要使用的模板(省略以进行交互式选择) |+| `--method` | `-m` | 方法模板(例如 `light_rag`、`graph_rag`) |+| `--lang` | `-l` | 语言:`zh` 或 `en`(知识模板必填) |+| `--force` | `-f` | 强制覆盖现有输出 |+| `--no-index` | — | 跳过构建搜索索引 |++---++## 示例++### 基本用法++从单个文件提取:++```bash+he parse document.md -t general/biography_graph -o ./output/ -l en+```++### 交互式模板选择++省略 `-t` 以从可用模板中选择:++```bash+he parse document.md -o ./output/ -l en++# 您将看到:+# Select a template:+# [1] general/biography_graph+# [2] general/knowledge_graph+# [3] finance/earnings_summary+# ...+# Enter number or search keyword:+```++### 处理目录++从目录中的所有 `.md` 和 `.txt` 文件提取:++```bash+he parse ./documents/ -t general/concept_graph -o ./output/ -l en+```++文件按字母顺序组合后进行提取。++### 使用方法而非模板++使用底层提取方法:++```bash+he parse document.md -m light_rag -o ./output/+```++方法始终使用英文提示。++### 强制覆盖++覆盖现有输出目录:++```bash+he parse document.md -t general/biography_graph -o ./output/ -l en -f+```++### 跳过索引构建++如果您不需要搜索/聊天,可以加快提取速度:++```bash+he parse document.md -t general/biography_graph -o ./output/ -l en --no-index+```++之后使用 `he build-index` 构建索引。++### 从标准输入读取++```bash+cat document.md | he parse - -t general/biography_graph -o ./output/ -l en+```++---++## 输出结构++```+./output/+├── data.json # 提取的知识(实体、关系等)+├── metadata.json # 提取元数据+│ ├── template # 使用的模板+│ ├── lang # 语言+│ ├── created_at # 创建时间戳+│ └── updated_at # 最后更新时间戳+└── index/ # 向量搜索索引(如已构建)+ ├── index.faiss+ └── docstore.json+```++---++## 语言支持++模板支持多种语言:++```bash+# 英文+he parse doc.md -t general/biography_graph -l en -o ./output/++# 中文+he parse doc.md -t general/biography_graph -l zh -o ./output/+```++选择与文档匹配的语言以获得最佳效果。++---++## 常见用例++### 研究论文++```bash+he parse paper.md -t general/concept_graph -o ./paper_kb/ -l en+```++### 传记++```bash+he parse biography.md -t general/biography_graph -o ./bio_kb/ -l en+```++### 法律合同++```bash+he parse contract.md -t legal/contract_obligation -o ./contract_kb/ -l en+```++### 财务报告++```bash+he parse earnings.md -t finance/earnings_summary -o ./finance_kb/ -l en+```++---++## 错误处理++### "输出目录已存在"++输出目录存在且不为空。解决方案:++1. 使用 `-f` 强制覆盖+2. 选择不同的输出路径+3. 先删除现有目录++### "模板未找到"++指定的模板不存在。解决方案:++1. 列出可用模板:`he list template`+2. 使用交互式选择(省略 `-t`)+3. 检查模板路径拼写++### "需要语言"++知识模板需要语言参数。方法不需要:++```bash+# 模板 - 需要 -l+he parse doc.md -t general/biography_graph -o ./out/ -l en++# 方法 - 不需要 -l+he parse doc.md -m light_rag -o ./out/+```++---++## 最佳实践++1. **选择正确的模板** — 匹配您的文档类型+2. **使用正确的语言** — 提高提取质量+3. **组织输出** — 使用描述性目录名+4. **批量处理时跳过索引** — 使用 `--no-index`,最后统一构建++---++## 另请参见++- [`he feed`](feed.md) — 增量添加文档+- [`he build-index`](build-index.md) — 构建搜索索引+- [`he list`](list.md) — 列出可用模板+- [模板库](../../templates/index.md)diff --git a/docs/zh/cli/commands/search.md b/docs/zh/cli/commands/search.mdnew file mode 100644index 0000000..a1aadca--- /dev/null+++ b/docs/zh/cli/commands/search.md@@ -0,0 +1,152 @@+# he search++对知识库执行语义搜索。++---++## 概要++```bash+he search KA_PATH QUERY [OPTIONS]+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `KA_PATH` | 知识库目录的路径 |+| `QUERY` | 搜索查询字符串 |++## 选项++| 选项 | 简写 | 描述 | 默认值 |+|--------|-------|-------------|---------|+| `--top-k` | `-n` | 返回结果数量 | 3 |++---++## 描述++语义搜索即使在关键词不完全匹配时也能找到相关信息。它使用:++1. **向量嵌入** — 将查询和内容转换为向量+2. **相似度匹配** — 找到语义相似的内容+3. **排名** — 返回最相关的结果++**要求**:必须构建搜索索引。如有需要,运行 `he build-index`。++---++## 示例++### 基本搜索++```bash+he search ./output/ "Tesla's inventions"+```++### 获取更多结果++```bash+he search ./output/ "electrical engineering" -n 10+```++### 自然语言查询++```bash+he search ./kb/ "主要成就是什么?"+he search ./kb/ "与爱迪生一起工作的人"+he search ./kb/ "时间线上的重要日期"+```++### 构建索引后++```bash+# 首先,确保索引存在+he build-index ./output/++# 然后搜索+he search ./output/ "innovation"+```++---++## 输出格式++```+Found 3 result(s):++Result 1:+{+ "name": "Nikola Tesla",+ "type": "person",+ "description": "Serbian-American inventor, electrical engineer..."+}++Result 2:+{+ "source": "Nikola Tesla",+ "target": "Thomas Edison",+ "type": "worked_with",+ "description": "Tesla worked for Edison in 1884"+}++Result 3:+...+```++---++## 工作原理++1. **查询嵌入** — 将您的查询转换为向量+2. **索引搜索** — 在知识库中找到最近的向量+3. **结果排名** — 返回 top-k 最相似的项目++---++## 更好搜索的技巧++1. **使用自然语言** — "电气工程方面的发明" vs "发明 电气"+2. **要具体** — "特斯拉在交流电方面的工作" vs "特斯拉 工作"+3. **尝试同义词** — 如果 "发明" 不行,尝试 "发现"+4. **增加 top-k** — 使用 `-n 10` 获取更广泛的结果++---++## 与 `he talk` 的比较++| 功能 | `he search` | `he talk` |+|---------|-------------|-----------|+| 返回 | 原始实体/关系 | 自然语言答案 |+| 用例 | 查找特定数据 | 获取解释 |+| 速度 | 更快 | 更慢(LLM 生成) |+| 精度 | 精确匹配 | 解释性 |++---++## 故障排除++### "未找到索引"++构建搜索索引:++```bash+he build-index ./output/+```++### "未找到结果"++尝试:+1. 更广泛的查询词+2. 增加 `-n` 获取更多结果+3. 不同的措辞+4. 检查 `he info ./output/` 验证数据存在++---++## 另请参见++- [`he talk`](talk.md) — 与知识库对话+- [`he build-index`](build-index.md) — 构建搜索索引+- [`he parse`](parse.md) — 带索引构建的提取diff --git a/docs/zh/cli/commands/show.md b/docs/zh/cli/commands/show.mdnew file mode 100644index 0000000..1162ce9--- /dev/null+++ b/docs/zh/cli/commands/show.md@@ -0,0 +1,140 @@+# he show++使用 OntoSight 交互式查看器可视化知识图谱。++---++## 概要++```bash+he show KA_PATH+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `KA_PATH` | 知识库目录的路径 |++---++## 描述++在默认 Web 浏览器中打开知识图谱的交互式可视化。可视化显示:++- **节点** — 实体、概念或项目(按类型着色)+- **边** — 节点之间的关系+- **标签** — 点击节点/边查看详情++---++## 示例++### 基本用法++```bash+he show ./output/+```++### 提取后++```bash+# 先提取+he parse tesla.md -t general/biography_graph -o ./tesla_kb/ -l en++# 然后可视化+he show ./tesla_kb/+```++### 增量更新后++```bash+# 添加更多内容+he feed ./tesla_kb/ additional_info.md++# 可视化更新后的图谱+he show ./tesla_kb/+```++---++## 可视化功能++OntoSight 查看器提供:++### 交互控制++- **缩放** — 鼠标滚轮或捏合手势+- **平移** — 点击并拖动背景+- **选择** — 点击节点/边查看详情+- **筛选** — 显示/隐藏节点类型++### 节点显示++- 大小表示重要性(可配置)+- 颜色表示实体类型+- 标签显示实体名称++### 边显示++- 粗细表示关系强度+- 标签显示关系类型+- 方向显示关系流向++---++## 支持的自动类型++可视化适用于所有基于图谱的自动类型:++| 自动类型 | 可视化 |+|-----------|--------------|+| `AutoGraph` | ✓ 完整图谱可视化 |+| `AutoHypergraph` | ✓ 带多节点边的超图 |+| `AutoTemporalGraph` | ✓ 带时间信息的图谱 |+| `AutoSpatialGraph` | ✓ 带空间信息的图谱 |+| `AutoSpatioTemporalGraph` | ✓ 完整上下文可视化 |+| `AutoList` | ✓ 列表视图 |+| `AutoSet` | ✓ 集合视图 |+| `AutoModel` | ✓ 结构化视图 |++---++## 故障排除++### "浏览器未打开"++会打印可视化 URL。手动打开:++```+http://localhost:xxxx+```++### "显示空图谱"++检查提取是否成功:++```bash+he info ./kb/+# 应该显示 Nodes > 0 和 Edges > 0+```++### "加载可视化出错"++确保知识库有效:++```bash+# 检查数据文件存在+ls ./kb/data.json++# 尝试重新加载+he show ./kb/+```++---++## 另请参见++- [`he parse`](parse.md) — 提取知识+- [`he feed`](feed.md) — 增量添加文档+- [`he info`](info.md) — 查看知识库统计信息diff --git a/docs/zh/cli/commands/talk.md b/docs/zh/cli/commands/talk.mdnew file mode 100644index 0000000..ec77e52--- /dev/null+++ b/docs/zh/cli/commands/talk.md@@ -0,0 +1,201 @@+# he talk++使用自然语言与您的知识库对话。++---++## 概要++```bash+he talk KA_PATH [OPTIONS]+```++## 参数++| 参数 | 描述 |+|----------|-------------|+| `KA_PATH` | 知识库目录的路径 |++## 选项++| 选项 | 简写 | 描述 |+|--------|-------|-------------|+| `--query` | `-q` | 要问的单个问题 |+| `--top-k` | `-n` | 要检索的上下文项目数量 | 3 |+| `--interactive` | `-i` | 进入交互式聊天模式 |++---++## 描述++`talk` 命令让您与知识库进行对话:++1. **检索相关上下文** — 搜索与您问题相关的信息+2. **生成答案** — 使用 LLM 综合自然语言响应+3. **显示来源** — 显示用于生成答案的项目++**要求**:必须构建搜索索引。++---++## 示例++### 单个问题++```bash+he talk ./output/ -q "特斯拉的主要成就是什么?"+```++**输出:**+```+Query: 特斯拉的主要成就是什么?+Knowledge Abstract: ./output/+Top K: 3++Nikola Tesla made numerous groundbreaking contributions to electrical+engineering. His major achievements include:++1. Development of the alternating current (AC) electrical system, which+ became the dominant method of power transmission worldwide.++2. Invention of the Tesla coil, used in radio technology and electrical+ resonant circuits.++3. Contributions to the development of X-ray imaging and wireless+ communication technologies.++Retrieved context:+1. Nikola Tesla: Serbian-American inventor, electrical engineer...+2. AC Power System: Type: invention, Description: System for alternating...+3. Tesla Coil: Type: invention, Description: Resonant transformer circuit...+```++### 交互模式++```bash+he talk ./output/ -i+```++**会话:**+```+Entering interactive mode. Type 'exit' or 'quit' to stop.++Knowledge Abstract: ./output/+Template: general/biography_graph+Top K: 3++> 谁是 Nikola Tesla?+Nikola Tesla was a Serbian-American inventor, electrical engineer,+mechanical engineer, and futurist best known for his contributions+to the design of the modern alternating current (AC) electricity+supply system.++> 他什么时候出生的?+Nikola Tesla 于 1856 年 7 月 10 日出生在克罗地亚的 Smiljan。++> 他与爱迪生的关系如何?+Tesla 于 1884 年在纽约市为 Thomas Edison 工作。由于在直流电(DC)+和交流电(AC)电力系统方面的不同观点,他们的关系存在争议。++> exit+Goodbye!++Other useful commands:+ he show ./output/ # 可视化+ he search ./output/ "keyword" # 搜索+ he info ./output/ # 查看信息+```++### 使用更多上下文++为复杂问题增加上下文:++```bash+he talk ./output/ -q "解释电流战争" -n 10+```++---++## 交互命令++在交互模式(`-i`)中:++| 命令 | 操作 |+|---------|--------|+| `exit`, `quit`, `q` | 退出交互模式 |+| `help` | 显示可用命令 |++---++## 用例++### 研究助手++```bash+he talk ./paper_kb/ -q "总结本文的主要贡献"+```++### 法律分析++```bash+he talk ./contract_kb/ -q "终止条件是什么?"+```++### 历史研究++```bash+he talk ./bio_kb/ -q "哪些事件导致了特斯拉的经济困难?"+```++---++## 工作原理++1. **语义搜索** — 在知识库中找到相关项目+2. **上下文组装** — 将检索到的项目组合成上下文+3. **LLM 生成** — 使用上下文生成答案+4. **来源归属** — 显示哪些项目提供了答案++---++## 更好答案的技巧++1. **要具体** — "特斯拉创造了什么发明?" vs "告诉我关于特斯拉的事"+2. **追问** — 通过对话构建上下文+3. **调整 top-k** — 对于复杂问题使用 `-n 5` 或更高+4. **检查来源** — 审查 "Retrieved context" 的准确性++---++## 与 `he search` 的比较++| 功能 | `he search` | `he talk` |+|---------|-------------|-----------|+| 输出 | 原始实体/关系 | 自然语言 |+| 最适合 | 查找特定数据 | 理解/解释 |+| 速度 | 快 | 更慢(LLM 调用) |+| 来源可见性 | 直接 | 在上下文中引用 |++---++## 故障排除++### "未找到索引"++```bash+he build-index ./output/+```++### "没有相关信息"++- 尝试重新表述您的问题+- 使用 `-n 10` 增加上下文+- 验证知识库有相关数据:`he info ./output/`++---++## 另请参见++- [`he search`](search.md) — 搜索特定项目+- [`he build-index`](build-index.md) — 构建搜索索引+- [`he show`](show.md) — 可视化知识图谱diff --git a/docs/zh/cli/config.md b/docs/zh/cli/config.mddeleted file mode 100644index 6520c48..0000000--- a/docs/zh/cli/config.md+++ /dev/null@@ -1,83 +0,0 @@-# CLI 配置--## 配置文件--CLI 配置文件位于 `~/.he/config.toml`。--## 初始化配置--```bash-# 使用 API Key 初始化(推荐首次用户)-he config init -k YOUR_OPENAI_API_KEY--# 交互式设置(会提示输入 LLM 和 Embedder 配置)-he config init-```--## 查看配置--```bash-# 显示当前配置-he config show-```--## 配置 LLM--```bash-# 设置 LLM API key 和模型-he config llm --api-key YOUR_KEY --model gpt-4o-mini--# 显示当前 LLM 配置-he config llm --show--# 设置自定义 base URL-he config llm --api-key YOUR_KEY --base-url https://api.openai.com/v1--# 清除 LLM 配置-he config llm --unset-```--## 配置 Embedder--```bash-# 设置 Embedder API key 和模型-he config embedder --api-key YOUR_KEY --model text-embedding-3-small--# 显示当前 Embedder 配置-he config embedder --show--# 设置自定义 base URL-he config embedder --api-key YOUR_KEY --base-url https://api.openai.com/v1--# 清除 Embedder 配置-he config embedder --unset-```--## 配置选项--### LLM 选项--| 选项 | 标志 | 描述 | 默认值 |-|------|------|------|--------|-| API Key | `--api-key`, `-k` | LLM API key | 必填 |-| Model | `--model`, `-m` | LLM 模型名称 | `gpt-4o-mini` |-| Base URL | `--base-url`, `-u` | 自定义 API base URL | OpenAI 默认 |--### Embedder 选项--| 选项 | 标志 | 描述 | 默认值 |-|------|------|------|--------|-| API Key | `--api-key`, `-k` | Embedder API key | 必填 |-| Model | `--model`, `-m` | Embedder 模型名称 | `text-embedding-3-small` |-| Base URL | `--base-url`, `-u` | 自定义 API base URL | OpenAI 默认 |--## 环境变量--你也可以使用环境变量作为替代方案:--```bash-export OPENAI_API_KEY=your-api-key-export OPENAI_BASE_URL=https://api.openai.com/v1-```--当配置文件中未设置时,将使用环境变量。diff --git a/docs/zh/cli/configuration.md b/docs/zh/cli/configuration.mdnew file mode 100644index 0000000..5de5f63--- /dev/null+++ b/docs/zh/cli/configuration.md@@ -0,0 +1,224 @@+# CLI 配置参考++Hyper-Extract CLI 配置的完整参考。++---++## 配置文件位置++- **Linux/macOS**: `~/.he/config.toml`+- **Windows**: `%USERPROFILE%\.he\config.toml`++---++## 配置部分++### [llm] — 语言模型设置++| 键 | 描述 | 默认值 |+|-----|-------------|---------|+| `model` | LLM 模型名称 | `gpt-4o-mini` |+| `api_key` | OpenAI API 密钥 | *必填* |+| `base_url` | API 基础 URL | `https://api.openai.com/v1` |++**示例:**+```toml+[llm]+model = "gpt-4o"+api_key = "sk-your-api-key"+base_url = "https://api.openai.com/v1"+```++**支持的模型:**+- `gpt-4o-mini` — 快速、成本效益高+- `gpt-4o` — 高质量+- `gpt-4-turbo` — 最佳质量+- 其他 OpenAI 兼容模型++---++### [embedder] — 嵌入模型设置++| 键 | 描述 | 默认值 |+|-----|-------------|---------|+| `model` | 嵌入模型名称 | `text-embedding-3-small` |+| `api_key` | API 密钥(默认为 llm.api_key) | — |++**示例:**+```toml+[embedder]+model = "text-embedding-3-large"+```++**支持的模型:**+- `text-embedding-3-small` — 快速、成本效益高+- `text-embedding-3-large` — 更好的质量+- `text-embedding-ada-002` — 传统模型++---++### [defaults] — 默认行为设置++| 键 | 描述 | 默认值 |+|-----|-------------|---------|+| `language` | 默认语言(`en` 或 `zh`) | `en` |+| `chunk_size` | 处理文本的块大小 | `2048` |+| `chunk_overlap` | 块之间的重叠 | `256` |+| `max_workers` | 并行处理工作线程数 | `10` |+| `verbose` | 启用详细输出 | `false` |++**示例:**+```toml+[defaults]+language = "en"+chunk_size = 4096+chunk_overlap = 512+max_workers = 5+verbose = true+```++---++## 环境变量++所有配置选项都可以通过环境变量设置:++| 变量 | 映射到 | 示例 |+|----------|---------|---------|+| `OPENAI_API_KEY` | `llm.api_key`, `embedder.api_key` | `sk-...` |+| `OPENAI_BASE_URL` | `llm.base_url` | `https://api.openai.com/v1` |+| `HE_LLM_MODEL` | `llm.model` | `gpt-4o` |+| `HE_EMBEDDER_MODEL` | `embedder.model` | `text-embedding-3-large` |+| `HE_DEFAULT_LANGUAGE` | `defaults.language` | `zh` |+| `HE_CHUNK_SIZE` | `defaults.chunk_size` | `4096` |+| `HE_VERBOSE` | `defaults.verbose` | `true` |++---++## 完整示例++```toml+# ~/.he/config.toml++[llm]+model = "gpt-4o-mini"+api_key = "sk-your-api-key-here"+base_url = "https://api.openai.com/v1"++[embedder]+model = "text-embedding-3-small"+# api_key 默认为 llm.api_key++[defaults]+language = "en"+chunk_size = 2048+chunk_overlap = 256+max_workers = 10+verbose = false+```++---++## 配置优先级++设置按以下顺序解析(最高优先级优先):++1. **命令行参数**(例如 `-l zh`)+2. **环境变量**(例如 `OPENAI_API_KEY`)+3. **配置文件**(`~/.he/config.toml`)+4. **内置默认值**++---++## 管理配置++### 初始化++```bash+he config init -k your-api-key+```++### 查看当前设置++```bash+he config show+```++### 更新设置++```bash+# 更新单个值+he config set llm.model gpt-4o++# 或直接编辑文件+nano ~/.he/config.toml+```++### 重置为默认值++```bash+rm ~/.he/config.toml+he config init -k your-api-key+```++---++## 高级配置++### 自定义 API 端点++对于 OpenAI 兼容 API(Azure、本地模型等):++```toml+[llm]+model = "your-model-name"+api_key = "your-key"+base_url = "https://your-api-endpoint.com/v1"++[embedder]+model = "your-embedding-model"+```++### LLM 和嵌入器使用不同密钥++```toml+[llm]+model = "gpt-4o-mini"+api_key = "sk-llm-key"++[embedder]+model = "text-embedding-3-small"+api_key = "sk-embedder-key"+```++---++## 故障排除++### 权限被拒绝++```bash+# 修复权限+chmod 600 ~/.he/config.toml+```++### 无效的 TOML 格式++验证配置:++```bash+# 使用 Python+python -c "import tomllib; tomllib.load(open('~/.he/config.toml', 'rb'))"+```++常见问题:+- 字符串缺少引号+- 尾部逗号+- 错误的节标题++---++## 另请参见++- [`he config`](commands/config.md) — 配置命令+- [安装指南](../getting-started/installation.md) — 初始设置diff --git a/docs/zh/cli/index.md b/docs/zh/cli/index.mdindex 7814796..ee7b121 100644--- a/docs/zh/cli/index.md+++ b/docs/zh/cli/index.md@@ -1,50 +1,201 @@-# CLI+# CLI 指南-Hyper-Extract 命令行界面。+Hyper-Extract CLI (`he`) 提供了强大、易用的界面,可直接从终端进行知识提取。++---## 安装```bash-uv pip install hyper-extract+pip install hyper-extract+```++验证安装:++```bash+he --version+```++---++## 快速命令参考++| 命令 | 用途 | 常用参数 |+|---------|---------|--------------|+| `he parse` | 从文档提取知识 | `-t` 模板, `-o` 输出, `-l` 语言 |+| `he show` | 可视化知识图谱 | — |+| `he search` | 知识库语义搜索 | `-n` top-k 结果数 |+| `he talk` | 与知识库对话 | `-i` 交互模式, `-q` 查询 |+| `he feed` | 增量添加文档 | — |+| `he info` | 显示知识库统计信息 | — |+| `he build-index` | 构建/重建搜索索引 | `-f` 强制重建 |+| `he list` | 列出模板和方法 | `template` 或 `method` |+| `he config` | 管理配置 | `init`, `show`, `set` |++---++## 完整工作流程++提取和交互知识的典型工作流程:++```mermaid+graph LR+ A[Document] -->|he parse| B[Knowledge Base]+ B -->|he show| C[Visualize]+ B -->|he search| D[Search]+ B -->|he talk| E[Chat]+ B -->|he feed| B[Update]```+1. **Parse** — 从文档提取知识+2. **Show** — 可视化知识图谱+3. **Search** — 查找特定信息+4. **Talk** — 关于内容进行对话+5. **Feed** — 增量添加更多文档++→ [详细工作流程指南](workflow.md)++---+## 快速开始+### 1. 配置 API 密钥+```bash-# 配置 API Keyhe config init -k YOUR_OPENAI_API_KEY+```-# 提取知识-he parse document.txt -t general/biography_graph -o ./output/+### 2. 提取知识-# 查询知识库-he search ./output/ "有哪些关键事件?"+```bash+he parse document.md -t general/biography_graph -o ./output/ -l en+```++### 3. 可视化-# 可视化知识图谱+```bashhe show ./output/```-## 核心命令+---++## 详细命令++### 知识提取++- **[`he parse`](commands/parse.md)** — 从文档提取知识+- **[`he feed`](commands/feed.md)** — 向现有知识库添加文档++### 探索++- **[`he show`](commands/show.md)** — 可视化知识图谱+- **[`he search`](commands/search.md)** — 语义搜索+- **[`he talk`](commands/talk.md)** — 与知识库对话+- **[`he info`](commands/info.md)** — 查看知识库统计信息-| 命令 | 用途 |-|------|------|-| `he config init` | 配置 API Key |-| `he parse` | 提取知识 |-| `he search` | 查询知识库 |-| `he show` | 可视化图谱 |-| `he feed` | 增量补充 |-| `he list` | 列出模板 |+### 管理++- **[`he build-index`](commands/build-index.md)** — 构建搜索索引+- **[`he list`](commands/list.md)** — 列出可用模板/方法+- **[`he config`](commands/config.md)** — 配置管理++---## 配置-[→ CLI 配置指南](./config.md)+CLI 在 `~/.he/config.toml` 存储配置。++→ [配置参考](configuration.md)++---++## 模板 vs 方法++Hyper-Extract 提供两种提取知识的方式:++### 模板(适用于大多数用户)++特定领域的开箱即用配置:++```bash+he parse doc.md -t general/biography_graph -l en+```++### 方法(高级)++底层提取算法:++```bash+he parse doc.md -m light_rag+```++→ [了解何时使用每种方式](../concepts/architecture.md)++---++## 语言支持++模板支持多种语言:++```bash+# 英文+he parse doc.md -t general/biography_graph -l en++# 中文+he parse doc.md -t general/biography_graph -l zh+```++方法模板始终使用英文提示。++---++## 用例示例++### 研究++```bash+# 从研究论文提取+he parse paper.md -t general/concept_graph -o ./paper_kb/ -l en++# 提问+he talk ./paper_kb/ -q "主要贡献是什么?"+```++### 传记分析++```bash+# 从传记提取+he parse biography.md -t general/biography_graph -o ./bio_kb/ -l en++# 可视化生平事件+he show ./bio_kb/+```++### 法律文档分析++```bash+# 提取合同义务+he parse contract.md -t legal/contract_obligation -o ./contract_kb/ -l en++# 搜索特定条款+he search ./contract_kb/ "termination conditions"+```++---++## 技巧和最佳实践-## 详细指南+1. **为特定领域任务使用模板** — 针对特定用例进行了优化+2. **构建索引** — 搜索和聊天功能需要索引+3. **增量摄入** — 随着时间添加文档,无需重新处理+4. **选择正确的语言** — 改善非英文文档的提取质量-[→ CLI 命令详解](./cli.md)+----## 使用示例+## 获取帮助-- [金融领域模板](../templates/finance.md)-- [法律领域模板](../templates/legal.md)-- [医学领域模板](../templates/medicine.md)+- 查看任何命令的帮助:`he <command> --help`+- 列出所有模板:`he list template`+- 列出所有方法:`he list method`+- [常见问题](../resources/faq.md)+- [故障排除](../resources/troubleshooting.md)diff --git a/docs/zh/cli/workflow.md b/docs/zh/cli/workflow.mdnew file mode 100644index 0000000..69667f9--- /dev/null+++ b/docs/zh/cli/workflow.md@@ -0,0 +1,348 @@+# CLI 工作流程指南++本指南通过实际案例带您完成完整的知识提取工作流程。++---++## 场景:研究论文分析++假设您是一位研究人员,想要从学术论文中提取和交互知识。以下是使用 Hyper-Extract 的方法。++---++## 步骤 1:准备文档++首先,确保您的文档是支持的格式(`.md`、`.txt`):++```bash+# 示例:将 PDF 转换为文本(使用外部工具)+pdftotext paper.pdf paper.md+```++或创建一个示例文档:++```bash+cat > transformer_paper.md << 'EOF'+# Attention Is All You Need++## Abstract+The dominant sequence transduction models are based on complex recurrent or+convolutional neural networks that include an encoder and a decoder. The best+performing models also connect the encoder and decoder through an attention+mechanism. We propose a new simple network architecture, the Transformer,+based solely on attention mechanisms.++## Authors+- Ashish Vaswani+- Noam Shazeer+- Niki Parmar+- Jakob Uszkoreit+- Llion Jones+- Aidan N. Gomez+- Lukasz Kaiser+- Illia Polosukhin++## Key Innovation+The Transformer eschews recurrence and instead relies entirely on an attention+mechanism to draw global dependencies between input and output sequences.++## Performance+On the WMT 2014 English-to-German translation task, our model achieves a+BLEU score of 28.4, outperforming all existing models.+EOF+```++---++## 步骤 2:提取知识++使用 `parse` 命令提取知识:++```bash+he parse transformer_paper.md -t general/concept_graph -o ./transformer_kb/ -l en+```++**发生了什么:**+1. 读取文档并分块(如文档较长)+2. LLM 提取实体和关系+3. 结果保存到 `./transformer_kb/`++**输出结构:**+```+./transformer_kb/+├── data.json # 提取的知识+├── metadata.json # 提取元数据+└── index/ # 向量搜索索引(如已构建)+```++---++## 步骤 3:探索知识++### 查看统计信息++```bash+he info ./transformer_kb/+```++**输出:**+```+Knowledge Abstract Info++Path ./transformer_kb/+Template general/concept_graph+Language en+Created 2024-01-15 10:30:00+Updated 2024-01-15 10:30:00+Nodes 12+Edges 15+Index Built+```++### 可视化++```bash+he show ./transformer_kb/+```++这将在浏览器中打开交互式图谱,显示:+- **节点**:作者、概念、模型、指标+- **边**:它们之间的关系++---++## 步骤 4:搜索信息++### 语义搜索++即使关键词不完全匹配也能查找信息:++```bash+he search ./transformer_kb/ "neural network architecture"+```++**结果:**+```+Found 3 result(s):++Result 1:+{+ "name": "Transformer",+ "type": "model",+ "description": "A neural network architecture based solely on attention mechanisms"+}++Result 2:+{+ "name": "Attention Mechanism",+ "type": "concept",+ "description": "Mechanism to draw global dependencies between sequences"+}+...+```++### 特定查询++```bash+he search ./transformer_kb/ "performance metrics" -n 5+```++---++## 步骤 5:与知识对话++### 单个问题++```bash+he talk ./transformer_kb/ -q "本文的主要贡献是什么?"+```++**响应:**+```+主要贡献是引入了 Transformer 架构,它在机器翻译任务上实现了最先进的成果,+同时比循环或卷积架构更易于并行化,训练时间也显著减少。+```++### 交互模式++```bash+he talk ./transformer_kb/ -i+```++**会话:**+```+Entering interactive mode. Type 'exit' or 'quit' to stop.++> 作者是谁?+本文作者是 Ashish Vaswani、Noam Shazeer、Niki Parmar、Jakob Uszkoreit、+Llion Jones、Aidan N. Gomez、Lukasz Kaiser 和 Illia Polosukhin。++> 他们取得了什么 BLEU 分数?+该模型在 WMT 2014 英德翻译任务上取得了 28.4 的 BLEU 分数。++> exit+Goodbye!+```++---++## 步骤 6:扩展知识库++### 添加另一个文档++```bash+# 下载另一篇论文+curl -o bert_paper.md https://example.com/bert.md++# 添加到现有知识库+he feed ./transformer_kb/ bert_paper.md+```++### 验证更新++```bash+he info ./transformer_kb/+```++注意节点/边数增加和时间戳更新。++---++## 步骤 7:重建索引(如需要)++添加文档后,重建搜索索引:++```bash+he build-index ./transformer_kb/+```++或强制完全重建:++```bash+he build-index ./transformer_kb/ -f+```++---++## 完整脚本示例++以下是一个自动化整个工作流程的 bash 脚本:++```bash+#!/bin/bash++# 配置+INPUT_FILE="paper.md"+OUTPUT_DIR="./paper_kb/"+TEMPLATE="general/concept_graph"+LANGUAGE="en"++echo "=== Hyper-Extract Workflow ==="+echo++# 步骤 1: 解析+echo "Step 1: Extracting knowledge..."+he parse "$INPUT_FILE" -t "$TEMPLATE" -o "$OUTPUT_DIR" -l "$LANGUAGE"+echo++# 步骤 2: 信息+echo "Step 2: Knowledge base info:"+he info "$OUTPUT_DIR"+echo++# 步骤 3: 搜索示例+echo "Step 3: Sample search:"+he search "$OUTPUT_DIR" "main contributions" -n 2+echo++# 步骤 4: 打开可视化+echo "Step 4: Opening visualization..."+he show "$OUTPUT_DIR"++echo "=== Workflow Complete ==="+```++---++## 高级:批量处理++一次处理多个文档:++```bash+# 创建输出目录+mkdir -p ./kb/paper1 ./kb/paper2 ./kb/paper3++# 处理每个+he parse papers/paper1.md -t general/concept_graph -o ./kb/paper1/ -l en+he parse papers/paper2.md -t general/concept_graph -o ./kb/paper2/ -l en+he parse papers/paper3.md -t general/concept_graph -o ./kb/paper3/ -l en++# 或使用循环+for file in papers/*.md; do+ name=$(basename "$file" .md)+ he parse "$file" -t general/concept_graph -o "./kb/$name/" -l en+done+```++---++## 常见模式++### 模式 1:持续构建知识++```bash+# 初始提取+he parse initial_doc.md -t general/biography_graph -o ./kb/ -l en++# 每周更新+he feed ./kb/ week1_update.md+he feed ./kb/ week2_update.md+he feed ./kb/ week3_update.md++# 每月重建+he build-index ./kb/ -f+```++### 模式 2:多领域项目++```bash+# 技术文档+he parse api_docs.md -t general/concept_graph -o ./project_kb/tech/ -l en++# 法律合同+he parse contract.pdf -t legal/contract_obligation -o ./project_kb/legal/ -l en++# 财务报告+he parse q4_report.md -t finance/earnings_summary -o ./project_kb/finance/ -l en+```++### 模式 3:比较文档++```bash+# 提取两个版本+he parse draft_v1.md -t general/concept_graph -o ./kb/v1/ -l en+he parse draft_v2.md -t general/concept_graph -o ./kb/v2/ -l en++# 通过聊天比较+he talk ./kb/v1/ -q "主要主题是什么?"+he talk ./kb/v2/ -q "主要主题是什么?"+```++---++## 故障排除提示++| 问题 | 解决方案 |+|-------|----------|+| 提取速度慢 | 长文档会被分块;使用 `--no-index` 在解析时跳过索引 |+| 搜索无结果 | 确保索引已构建:`he build-index ./kb/` |+| 模板未找到 | 列出可用模板:`he list template` |+| 内存不足 | 减少配置中的块大小或处理较小的文档 |++---++## 下一步++- 了解[所有 CLI 命令](commands/parse.md)+- 探索[模板库](../templates/index.md)+- 阅读[选择正确的自动类型](../concepts/autotypes.md)diff --git a/docs/zh/concepts/architecture.md b/docs/zh/concepts/architecture.mdnew file mode 100644index 0000000..5deb311--- /dev/null+++ b/docs/zh/concepts/architecture.md@@ -0,0 +1,231 @@+# 架构++深入了解 Hyper-Extract 的系统设计和数据流。++---++## 系统概览++```mermaid+graph LR+ A[Input Text] --> B[Text Processor]+ B --> C[LLM Extractor]+ C --> D[Merger]+ D --> E[Auto-Type Instance]+ E --> F[Operations]++ subgraph "Text Processor"+ B1[Chunking] --> B2[Parallel Processing]+ end++ subgraph "LLM Extractor"+ C1[Prompt] --> C2[LLM Call]+ C2 --> C3[Structured Output]+ end++ subgraph "Operations"+ F1[Search]+ F2[Chat]+ F3[Visualize]+ F4[Save/Load]+ end+```++---++## 数据流++### 1. 输入处理++```python+# 输入:原始文本+text = "Nikola Tesla was an inventor..."++# 如果 text > chunk_size,拆分为块+chunks = [+ "Nikola Tesla was an inventor...",+ "He developed the AC system...",+ # ...+]+```++**分块策略:**+- 默认大小:2048 字符+- 重叠:256 字符+- 分隔符:段落、句子、单词++### 2. 提取++```python+# 每个块并行处理+for chunk in chunks:+ # LLM 提取结构化数据+ data = llm.extract(chunk, schema=AutoTypeSchema)+```++**流程:**+1. 使用块格式化提示+2. 使用结构化输出调用 LLM+3. 将响应解析为 Pydantic 模型++### 3. 合并++```python+# 合并所有块的结果+final_data = merge(chunk_results)+```++**合并操作:**+- 实体去重+- 组合关系+- 解决冲突++### 4. 结果++```python+# 带提取数据的 Auto-Type 实例+result = AutoTypeInstance(data=final_data)+```++---++## 组件详情++### 模板引擎++```+Template YAML → Parser → Configuration → Factory → Auto-Type Instance+```++组件:+- **Gallery** — 模板发现和列表+- **Parser** — YAML 解析和验证+- **Factory** — 实例创建++### Auto-Type 基类++```python+class BaseAutoType:+ # 核心功能+ def parse(text) -> AutoType+ def feed_text(text) -> AutoType++ # 查询+ def build_index()+ def search(query) -> List[Item]+ def chat(query) -> Response++ # 持久化+ def dump(path)+ def load(path)++ # 可视化+ def show()+```++### 方法注册表++```python+# 方法在导入时注册+_METHOD_REGISTRY = {+ "light_rag": {+ "class": Light_RAG,+ "type": "graph",+ "description": "..."+ },+ # ...+}+```++---++## 扩展点++### 自定义模板++创建特定领域模板:++```yaml+# my_template.yaml+name: custom_extraction+type: graph+output:+ entities:+ fields:+ - name: name+ type: str+# ...+```++### 自定义自动类型++扩展基类:++```python+from hyperextract.types import AutoGraph++class MyCustomGraph(AutoGraph):+ def _default_prompt(self):+ return "Custom prompt..."+```++### 自定义方法++实现提取算法:++```python+from hyperextract.methods import register_method++class MyMethod:+ def extract(self, text):+ # 自定义提取逻辑+ pass++register_method("my_method", MyMethod, "graph", "Description")+```++---++## 性能考虑++### 分块++| 文档大小 | 块数 | 处理时间 |+|---------------|--------|-----------------|+| < 2KB | 1 | 快 |+| 2-10KB | 2-5 | 中等 |+| 10-50KB | 5-25 | 慢 |+| > 50KB | 25+ | 非常慢 |++### 并行化++```python+# 默认:10 个并发工作线程+# 可通过配置调整+max_workers = 10+```++### 内存使用++```python+# 索引内存估算+# 每个实体/关系约 1KB+```++---++## 设计原则++1. **类型安全** — 全程使用 Pydantic schema+2. **可扩展性** — 方法的插件架构+3. **可用性** — 常见任务的模板+4. **性能** — 分块和并行化+5. **互操作性** — 标准格式(JSON、YAML)++---++## 另请参见++- [自动类型](autotypes.md)+- [方法](methods.md)+- [模板格式](templates-format.md)diff --git a/docs/zh/concepts/auto-types.md b/docs/zh/concepts/auto-types.mddeleted file mode 100644index 0386196..0000000--- a/docs/zh/concepts/auto-types.md+++ /dev/null@@ -1,91 +0,0 @@-# AutoTypes--AutoTypes 定义了知识提取的**输出数据结构**。共有 8 种类型:--## 记录型--### AutoModel--使用 Pydantic 模型提取结构化数据。--```python-from pydantic import BaseModel, Field-from hyperextract import AutoModel--class Summary(BaseModel):- title: str = Field(description="标题")- summary: str = Field(description="摘要")--model = AutoModel(data_schema=Summary, llm_client=llm)-```--### AutoList--提取列表项。--```python-from hyperextract.types import AutoList--lst = AutoList[str](item_schema=str, llm_client=llm)-lst.feed_text(text)-```--### AutoSet--提取唯一项(去重)。--```python-from hyperextract.types import AutoSet--s = AutoSet[str](item_schema=str, llm_client=llm)-s.feed_text(text)-```--## 图谱型--### AutoGraph--实体-关系提取。--```python-from hyperextract.types import AutoGraph--graph = AutoGraph(node_schema=Entity, edge_schema=Relation, llm_client=llm)-graph.feed_text(text)-```--### AutoHypergraph--多元关系(n 元关系)。--```python-from hyperextract.types import AutoHypergraph-```--### AutoTemporalGraph--时间相关的关嗧。--```python-from hyperextract.types import AutoTemporalGraph-```--### AutoSpatialGraph--位置相关的关嗧。--```python-from hyperextract.types import AutoSpatialGraph-```--### AutoSpatioTemporalGraph--时间和位置同时相关的关嗧。--```python-from hyperextract.types import AutoSpatioTemporalGraph-```--## 快速链接--- [领域模板](../templates/index.md) - 开箱即用的模板diff --git a/docs/zh/concepts/autotypes.md b/docs/zh/concepts/autotypes.mdnew file mode 100644index 0000000..c194455--- /dev/null+++ b/docs/zh/concepts/autotypes.md@@ -0,0 +1,362 @@+# 自动类型++8 种知识结构类型的完整指南。++---++## 概述++自动类型是智能数据结构,具有以下功能:+- 定义提取输出格式+- 提供类型安全的 schema+- 包含内置操作(搜索、可视化等)+- 支持序列化++---++## 8 种自动类型++```mermaid+graph TD+ A[Auto-Types] --> B[Scalar Types]+ A --> C[Graph Types]+ A --> D[Temporal Types]++ B --> B1[AutoModel]+ B --> B2[AutoList]+ B --> B3[AutoSet]++ C --> C1[AutoGraph]+ C --> C2[AutoHypergraph]++ D --> D1[AutoTemporalGraph]+ D --> D2[AutoSpatialGraph]+ D --> D3[AutoSpatioTemporalGraph]+```++---++## 标量类型++### AutoModel++**目的**:单个结构化对象++**使用场景**:+- 提取摘要/报告+- 具有已知字段的结构化数据+- 具有许多属性的单个实体++**示例输出**:+```python+{+ "company_name": "Tesla Inc",+ "revenue": 81.46,+ "eps": 4.07,+ "employees": 127855+}+```++**常用模板**:+- `finance/earnings_summary`+- `general/base_model`++---++### AutoList++**目的**:有序集合++**使用场景**:+- 顺序很重要+- 排名项目+- 简单的时间线事件++**示例输出**:+```python+{+ "items": [+ {"name": "AC Motor", "year": 1888},+ {"name": "Tesla Coil", "year": 1891},+ {"name": "Radio", "year": 1898}+ ]+}+```++**常用模板**:+- `general/base_list`+- `legal/compliance_list`++---++### AutoSet++**目的**:去重集合++**使用场景**:+- 仅唯一项目+- 标签/类别+- 成员资格测试++**示例输出**:+```python+{+ "items": [+ "Electrical Engineering",+ "Physics",+ "Invention",+ "Renewable Energy"+ ]+}+```++**常用模板**:+- `general/base_set`+- `finance/risk_factor_set`++---++## 图谱类型++### AutoGraph++**目的**:实体关系网络++**使用场景**:+- 人物、组织、概念+- 二元关系+- 知识图谱++**示例输出**:+```python+{+ "entities": [+ {"name": "Tesla", "type": "person"},+ {"name": "Edison", "type": "person"},+ {"name": "AC Motor", "type": "invention"}+ ],+ "relations": [+ {"source": "Tesla", "target": "AC Motor", "type": "invented"},+ {"source": "Tesla", "target": "Edison", "type": "rivals"}+ ]+}+```++**常用模板**:+- `general/knowledge_graph`+- `general/biography_graph`++---++### AutoHypergraph++**目的**:多实体关系++**使用场景**:+- 关系涉及 3+ 个实体+- 复杂交互+- N 元关联++**示例输出**:+```python+{+ "entities": [...],+ "hyperedges": [+ {+ "entities": ["Tesla", "Westinghouse", "Niagara"],+ "type": "collaboration",+ "description": "Power plant project"+ }+ ]+}+```++**常用模板**:+- `general/base_hypergraph`++---++## 时序类型++### AutoTemporalGraph++**目的**:图谱 + 时间信息++**使用场景**:+- 时间线很重要+- 事件序列+- 历史分析++**示例输出**:+```python+{+ "entities": [...],+ "relations": [+ {+ "source": "Tesla",+ "target": "AC Motor",+ "type": "invented",+ "time": "1888"+ },+ {+ "source": "Tesla",+ "target": "Wardenclyffe Tower",+ "type": "built",+ "time": "1901-1902"+ }+ ]+}+```++**常用模板**:+- `general/base_temporal_graph`+- `finance/event_timeline`++---++### AutoSpatialGraph++**目的**:图谱 + 位置信息++**使用场景**:+- 地理数据+- 基于位置的分析+- 映射++**示例输出**:+```python+{+ "entities": [+ {+ "name": "Colorado Springs",+ "type": "location",+ "coordinates": "38.8339,-104.8214"+ }+ ],+ "relations": [+ {+ "source": "Tesla",+ "target": "Colorado Springs",+ "type": "conducted_experiments",+ "location": "Colorado Springs"+ }+ ]+}+```++**常用模板**:+- `general/base_spatial_graph`++---++### AutoSpatioTemporalGraph++**目的**:图谱 + 时间 + 空间++**使用场景**:+- 需要完整上下文+- 历史地理+- 带时间和地点的事件分析++**示例输出**:+```python+{+ "entities": [...],+ "relations": [+ {+ "source": "Tesla",+ "target": "AC Motor",+ "type": "demonstrated",+ "time": "1888",+ "location": "Pittsburgh",+ "description": "Demonstration at Westinghouse"+ }+ ]+}+```++**常用模板**:+- `general/base_spatio_temporal_graph`++---++## 选择指南++### 决策树++```+需要提取什么?+│+├─ 单个结构化对象 → AutoModel+│+├─ 项目集合+│ ├─ 有序/排名 → AutoList+│ └─ 唯一/标签 → AutoSet+│+└─ 关系+ ├─ 简单(二元)+ │ ├─ 带时间 → AutoTemporalGraph+ │ ├─ 带空间 → AutoSpatialGraph+ │ ├─ 两者都 → AutoSpatioTemporalGraph+ │ └─ 两者都不 → AutoGraph+ │+ └─ 复杂(多实体)+ └─ AutoHypergraph+```++### 按用例++| 用例 | 推荐类型 |+|----------|------------------|+| 公司报告 | AutoModel |+| Top 10 列表 | AutoList |+| 标签/关键词 | AutoSet |+| 人物网络 | AutoGraph |+| 项目团队 | AutoHypergraph |+| 传记时间线 | AutoTemporalGraph |+| 旅行日志 | AutoSpatialGraph |+| 历史事件 | AutoSpatioTemporalGraph |++---++## 常见操作++所有自动类型都支持:++```python+# 提取+result = ka.parse(text)++# 增量更新+result.feed_text(more_text)++# 搜索(需要索引)+result.build_index()+results = result.search("query")++# 聊天(需要索引)+response = result.chat("question")++# 可视化+result.show()++# 持久化+result.dump("./path/")+result.load("./path/")++# 检查空+if result.empty():+ print("No data")++# 清除+result.clear()+result.clear_index()+```++---++## 另请参见++- [使用自动类型](../python/guides/working-with-autotypes.md)+- [模板库](../templates/index.md)+- [方法](methods.md)diff --git a/docs/zh/concepts/index.md b/docs/zh/concepts/index.mdindex bf4b6b7..417fd6a 100644--- a/docs/zh/concepts/index.md+++ b/docs/zh/concepts/index.md@@ -1,18 +1,86 @@-# Concepts+# 核心概念-Hyper-Extract 采用**三层架构**:+了解 Hyper-Extract 的基本概念。-- **Auto-Types** 定义了知识提取的数据结构-- **Methods** 基于 Auto-Types 提供提取算法-- **Templates** 提供领域特定的配置,包含开箱即用的 prompt+----## Auto-Types+## 三层架构-Auto-Types 是基础层 - 它们定义了知识提取的输出数据结构。+Hyper-Extract 构建在三层架构之上:-[→ 了解更多 Auto-Types](./auto-types.md)+```mermaid+graph TD+ subgraph "Layer 3: Interface"+ A[CLI] --> B[Python SDK]+ B --> C[Template API]+ end++ subgraph "Layer 2: Methods"+ C --> D[RAG-Based]+ C --> E[Typical]+ end++ subgraph "Layer 1: Data"+ D --> F[Auto-Types]+ E --> F+ end++ F --> G[Structured Knowledge]+```++| 层级 | 目的 | 组件 |+|-------|---------|------------|+| **自动类型** | 定义数据结构 | 8 种类型类 |+| **方法** | 提取算法 | RAG + 典型方法 |+| **模板** | 特定领域配置 | 80+ 预设模板 |++---++## 核心概念++### [自动类型](autotypes.md)++8 种定义提取输出的数据结构类型:++- **标量类型**:AutoModel、AutoList、AutoSet+- **图谱类型**:AutoGraph、AutoHypergraph+- **时序类型**:AutoTemporalGraph、AutoSpatialGraph、AutoSpatioTemporalGraph++→ [了解自动类型](autotypes.md)++### [方法](methods.md)++提取算法:++- **基于 RAG**:GraphRAG、LightRAG、Hyper-RAG+- **典型方法**:iText2KG、KG-Gen、Atom++→ [了解方法](methods.md)++### [模板格式](templates-format.md)++定义提取模板的 YAML 格式:++- 模式定义+- 提示工程+- 指南和规则++→ [了解模板格式](templates-format.md)++### [架构](architecture.md)++深入了解系统设计:++- 数据流+- 处理管道+- 扩展点++→ [了解架构](architecture.md)++---## 快速链接-- [Auto-Types](./auto-types.md) - 8 种强类型的知识结构-- [Templates](../templates/index.md) - 开箱即用的模板+- [CLI 文档](../cli/index.md) — 终端使用+- [Python SDK](../python/index.md) — 编程使用+- [模板库](../templates/index.md) — 可用模板diff --git a/docs/zh/concepts/methods.md b/docs/zh/concepts/methods.mdnew file mode 100644index 0000000..61b6ec5--- /dev/null+++ b/docs/zh/concepts/methods.md@@ -0,0 +1,300 @@+# 方法++提取算法指南以及何时使用每种方法。++---++## 概述++方法是底层算法,从文本中提取知识。它们决定:+- 如何处理文本+- 如何识别实体和关系+- 提取的质量和风格++---++## 方法类别++### 基于 RAG 的方法++**检索增强生成** 将信息检索与文本生成相结合。++**最适合**:大型文档、复杂查询++```mermaid+graph LR+ A[Document] --> B[Chunk & Index]+ B --> C[Retrieve Relevant]+ C --> D[Generate Answer]+ D --> E[Extract Structure]+```++### 典型方法++**直接提取**,无需检索。++**最适合**:较小的文档、直接提取++```mermaid+graph LR+ A[Document] --> B[LLM Processing]+ B --> C[Extract Structure]+```++---++## 基于 RAG 的方法++### light_rag++**描述**:轻量级基于图谱的 RAG++**特点**:+- 最快的 RAG 方法+- 二元边(source → target)+- 速度/质量的良好平衡+- 适合大多数用例++**最适合**:+- 通用提取+- 中大型文档+- 快速结果++**用法**:+```python+ka = Template.create("method/light_rag")+```++---++### graph_rag++**描述**:带社区检测的 Graph-RAG++**特点**:+- 社区检测用于组织+- 层次化摘要+- 最适合非常大的文档+- 较慢但彻底++**最适合**:+- 非常大的文档(书籍、长论文)+- 复杂主题结构+- 研究论文++**用法**:+```python+ka = Template.create("method/graph_rag")+```++---++### hyper_rag++**描述**:基于超图谱的 RAG++**特点**:+- N 元超边(2+ 个实体)+- 捕获复杂关系+- 更丰富的图谱结构++**最适合**:+- 多方关系+- 项目协作+- 复杂组织结构++**用法**:+```python+ka = Template.create("method/hyper_rag")+```++---++### hypergraph_rag++**描述**:高级超图谱 RAG++**特点**:+- 增强的超图谱能力+- 高级关系建模++**最适合**:+- 复杂超图谱场景+- 高级关系分析++**用法**:+```python+ka = Template.create("method/hypergraph_rag")+```++---++### cog_rag++**描述**:认知 RAG++**特点**:+- 认知检索机制+- 以推理为中心++**最适合**:+- 推理任务+- 问答系统++**用法**:+```python+ka = Template.create("method/cog_rag")+```++---++## 典型方法++### itext2kg++**描述**:高质量基于三元组的提取++**特点**:+- 针对三元组质量优化+- 迭代精炼+- 适合知识库构建++**最适合**:+- 知识图谱构建+- 高质量要求+- 三元组提取++**用法**:+```python+ka = Template.create("method/itext2kg")+```++---++### itext2kg_star++**描述**:增强版 iText2KG++**特点**:+- 提高提取质量+- 更好地处理复杂情况+- 增强的实体链接++**最适合**:+- 质量至关重要时+- 复杂提取场景+- 生产系统++**用法**:+```python+ka = Template.create("method/itext2kg_star")+```++---++### kg_gen++**描述**:知识图谱生成器++**特点**:+- 可配置的生成+- 灵活的 schema+- 快速处理++**最适合**:+- 自定义 schema+- 快速原型+- 灵活需求++**用法**:+```python+ka = Template.create("method/kg_gen")+```++---++### atom++**描述**:带证据的时序知识图谱++**特点**:+- 时序事实提取+- 证据归属+- 置信度评分++**最适合**:+- 时序分析+- 事实验证+- 时间线提取++**用法**:+```python+ka = Template.create("method/atom")+```++---++## 选择指南++### 按文档大小++| 大小 | 推荐 |+|------|-------------|+| 小型(< 1K 词) | itext2kg, kg_gen |+| 中型(1-10K) | light_rag, itext2kg_star |+| 大型(10-50K) | light_rag, graph_rag |+| 非常大(> 50K) | graph_rag |++### 按用例++| 用例 | 推荐 |+|----------|-------------|+| 快速提取 | light_rag |+| 最佳质量 | itext2kg_star |+| 大型文档 | graph_rag |+| 复杂关系 | hyper_rag |+| 时序事实 | atom |+| 知识库 | itext2kg |++### 按优先级++| 优先级 | 推荐 |+|----------|-------------|+| 速度 | light_rag |+| 质量 | itext2kg_star |+| 成本 | light_rag |+| 完整性 | graph_rag |++---++## 比较表++| 方法 | 速度 | 质量 | 内存 | 最佳用途 |+|--------|-------|---------|--------|----------|+| light_rag | ⭐⭐⭐ | ⭐⭐ | ⭐⭐ | 通用 |+| graph_rag | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ | 大型文档 |+| hyper_rag | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | 复杂关系 |+| itext2kg | ⭐⭐⭐ | ⭐⭐⭐ | ⭐ | 质量优先 |+| atom | ⭐⭐ | ⭐⭐⭐ | ⭐⭐ | 时序数据 |+| kg_gen | ⭐⭐⭐ | ⭐⭐ | ⭐ | 灵活性 |++---++## 列出可用方法++```python+from hyperextract.methods import list_methods++methods = list_methods()+for name, info in methods.items():+ print(f"{name}: {info['description']}")+ print(f" Type: {info['type']}")+```++---++## 另请参见++- [选择方法指南](../python/guides/choosing-methods.md)+- [模板](../templates/index.md)+- [自动类型](autotypes.md)diff --git a/docs/zh/concepts/templates-format.md b/docs/zh/concepts/templates-format.mdnew file mode 100644index 0000000..db507d6--- /dev/null+++ b/docs/zh/concepts/templates-format.md@@ -0,0 +1,309 @@+# 模板格式++YAML 模板结构的完整参考。++---++## 概述++模板是定义以下内容的 YAML 文件:+- **输出 schema** — 要提取什么+- **提示** — 如何指导 LLM+- **指南** — 提取规则+- **配置** — 显示和标识符++---++## 模板结构++```yaml+language: [zh, en] # 支持的语言++name: template_name # 模板标识符+type: graph # Auto-Type+tags: [domain, category] # 分类标签++description: # 人类可读的描述+ zh: "中文描述"+ en: "English description"++output: # 输出 schema 定义+ description:+ zh: "输出描述"+ en: "Output description"+ entities: # 实体 schema+ description:+ zh: "实体描述"+ en: "Entity description"+ fields:+ - name: field_name+ type: str+ description:+ zh: "字段描述"+ en: "Field description"+ required: true+ relations: # 关系 schema(用于图谱)+ # 与实体相同的结构++guideline: # LLM 指令+ target:+ zh: "提取目标"+ en: "Extraction target"+ rules_for_entities:+ zh: ["规则1", "规则2"]+ en: ["Rule 1", "Rule 2"]+ rules_for_relations:+ zh: ["规则1"]+ en: ["Rule 1"]++identifiers: # ID 生成规则+ entity_id: name # 用作实体 ID 的字段+ relation_id: "{source}|{type}|{target}" # 关系 ID 格式+ relation_members:+ source: source+ target: target++display: # 可视化设置+ entity_label: "{name} ({type})"+ relation_label: "{type}"+```++---++## 字段类型++| 类型 | 描述 | 示例 |+|------|-------------|---------|+| `str` | 字符串 | `"Hello"` |+| `int` | 整数 | `42` |+| `float` | 浮点数 | `3.14` |+| `bool` | 布尔值 | `true` |+| `list[str]` | 字符串列表 | `["a", "b"]` |+| `datetime` | 日期/时间 | `"2024-01-01"` |++---++## 完整示例++```yaml+language: [zh, en]++name: biography_graph+type: temporal_graph+tags: [general, biography, life, events, timeline]++description:+ zh: '传记图模板 - 从人物传记、回忆录、年谱中提取实体和关系'+ en: 'Biography Graph Template - Extract entities and relationships from biographies'++output:+ description:+ zh: '人物传记中的实体和关系网络'+ en: "Entity and relationship network from biographies"++ entities:+ description:+ zh: '传记中的实体'+ en: 'Entities in the biography'+ fields:+ - name: name+ type: str+ description:+ zh: '实体名称'+ en: 'Entity name'+ required: true++ - name: type+ type: str+ description:+ zh: '实体类型:人物、地点、组织、作品、事件、成就等'+ en: 'Entity type: person, location, organization, work, event, achievement'+ required: true++ - name: description+ type: str+ description:+ zh: '实体描述'+ en: 'Entity description'+ required: false++ relations:+ description:+ zh: '实体之间的关系'+ en: 'Relationships between entities'+ fields:+ - name: source+ type: str+ description:+ zh: '关系起点实体'+ en: 'Source entity'+ required: true++ - name: target+ type: str+ description:+ zh: '关系终点实体'+ en: 'Target entity'+ required: true++ - name: type+ type: str+ description:+ zh: '关系类型'+ en: 'Relation type'+ required: true++ - name: time+ type: str+ description:+ zh: '关系对应的时间'+ en: 'Time associated with the relation'+ required: false++guideline:+ target:+ zh: '你是一位专业的传记分析师...'+ en: 'You are a professional biography analyst...'++ rules_for_entities:+ zh:+ - '提取传记主人公作为核心实体'+ - '同一实体在全文中保持命名一致'+ en:+ - 'Extract the biographical subject as the core entity'+ - 'Maintain consistent naming for the same entity'++ rules_for_relations:+ zh:+ - '仅在文本明确表达两个实体之间存在关系时创建关系'+ en:+ - 'Create relationships only when explicitly expressed'++ rules_for_time:+ zh:+ - '绝对日期:保持原文格式'+ - '相对时间:转换为绝对时间'+ en:+ - 'Absolute dates: Keep as-is'+ - 'Relative time: Convert to absolute'++identifiers:+ entity_id: name+ relation_id: '{source}|{type}|{target}'+ relation_members:+ source: source+ target: target+ time_field: time++display:+ entity_label: '{name}'+ relation_label: '{type}'+```++---++## Auto-Type 特定部分++### 列表/集合++```yaml+output:+ items: # 而不是 entities/relations+ description:+ zh: "列表项"+ en: "List items"+ fields:+ - name: value+ type: str+```++### 模型++```yaml+output:+ fields: # 直接字段(不是 entities)+ - name: company_name+ type: str+ - name: revenue+ type: float+```++### 时序图谱++```yaml+guideline:+ rules_for_time: # 时间提取规则+ zh: ["规则1", "规则2"]+ en: ["Rule 1", "Rule 2"]++identifiers:+ time_field: time # 包含时间的字段+```++### 空间图谱++```yaml+output:+ entities:+ fields:+ - name: location # 位置字段+ type: str+ - name: coordinates # 可选坐标+ type: str++guideline:+ rules_for_location: # 位置提取规则+ zh: ["规则1"]+ en: ["Rule 1"]+```++---++## 模板变量++在指南中使用占位符:++```yaml+guideline:+ target:+ zh: '观察时间基准:{observation_time}'+ en: 'Observation time baseline: {observation_time}'+```++在运行时传递:++```python+ka = Template.create(+ "template_name",+ "en",+ observation_time="2024-01-01"+)+```++---++## 验证++模板验证以下内容:+- 必填字段存在+- 类型定义有效+- 语言代码支持+- Schema 一致性++---++## 最佳实践++1. **清晰的描述** — 帮助用户选择正确的模板+2. **具体的规则** — 指导 LLM 更好地提取+3. **一致的命名** — 使用清晰、一致的字段名+4. **双语支持** — 尽可能提供中英文+5. **必填与可选** — 适当标记字段++---++## 另请参见++- [模板库](../templates/index.md)+- [创建自定义模板](../python/guides/custom-templates.md)+- [自动类型](autotypes.md)diff --git a/docs/zh/getting-started/cli-quickstart.md b/docs/zh/getting-started/cli-quickstart.mdnew file mode 100644index 0000000..da6aa5f--- /dev/null+++ b/docs/zh/getting-started/cli-quickstart.md@@ -0,0 +1,97 @@+# CLI 快速入门++使用命令行从终端提取知识。++---++## 前置要求++- Hyper-Extract 已安装+- API 密钥已配置++→ 如果尚未完成,请查看[安装指南](installation.md)++---++## 快速示例++### 1. 从文档提取知识++```bash+he parse tesla.md -t general/biography_graph -o ./output/ -l en+```++这将从 `tesla.md` 中提取知识图谱并保存到 `./output/` 目录。++### 2. 可视化结果++```bash+he show ./output/+```++### 3. 搜索知识库++```bash+he search ./output/ "AC motor invention"+```++### 4. 与知识库对话++```bash+he talk ./output/ -q "What were Tesla's major inventions?"+```++---++## 完整工作流程++```mermaid+graph LR+ A[Document] -->|he parse| B[Knowledge Base]+ B -->|he show| C[Visualize]+ B -->|he search| D[Search]+ B -->|he talk| E[Chat]+ B -->|he feed| B[Update]+```++1. **Parse** — 从文档提取知识+2. **Show** — 可视化知识图谱+3. **Search** — 查找特定信息+4. **Talk** — 关于内容进行对话+5. **Feed** — 增量添加更多文档++---++## 常用命令++### 解析文档++```bash+# 使用模板+he parse doc.md -t general/biography_graph -o ./out/ -l en++# 使用方法+he parse doc.md -m light_rag -o ./out/+```++### 构建索引++```bash+# 构建搜索索引(搜索和聊天需要)+he build-index ./output/+```++### 查看信息++```bash+# 查看知识库统计+he info ./output/+```++---++## 下一步++- 深入了解 [CLI 命令](../cli/index.md)+- 浏览 [模板库](../templates/index.md)+- 理解 [核心概念](../concepts/index.md)diff --git a/docs/zh/getting-started/index.md b/docs/zh/getting-started/index.mdnew file mode 100644index 0000000..e18d825--- /dev/null+++ b/docs/zh/getting-started/index.md@@ -0,0 +1,53 @@+# 入门指南++欢迎使用 Hyper-Extract!本指南将帮助您快速上手,无论您更喜欢使用命令行还是 Python。++---++## 选择您的路径++<div class="grid cards" markdown>++- :material-console-line:{ .lg .middle } __我想使用终端__++ ---++ CLI 非常适合以下场景:++ - 不想编写代码处理文档+ - 需要批量处理文件+ - 更喜欢终端工作流程++ **从这里开始:** [:octicons-arrow-right-24: 安装](installation.md) → [:octicons-arrow-right-24: CLI 快速入门](cli-quickstart.md)++- :material-language-python:{ .lg .middle } __我想使用 Python__++ ---++ Python SDK 非常适合以下场景:++ - 正在构建应用程序+ - 需要与现有代码集成+ - 想要完全的程序化控制++ **从这里开始:** [:octicons-arrow-right-24: 安装](installation.md) → [:octicons-arrow-right-24: Python 快速入门](python-quickstart.md)++</div>++---++## 接下来做什么?++完成快速入门后:++1. **探索 [CLI 指南](../cli/index.md)** — 了解完整的工作流程+2. **浏览 [模板库](../templates/index.md)** — 找到适合您领域的模板+3. **阅读 [核心概念](../concepts/index.md)** — 了解其工作原理++---++## 需要帮助?++- **有问题?** 查看 [常见问题](../resources/faq.md)+- **遇到问题?** 参见 [故障排除](../resources/troubleshooting.md)+- **发现错误?** [在 GitHub 上报告](https://github.com/yifanfeng97/hyper-extract/issues)diff --git a/docs/zh/getting-started/installation.md b/docs/zh/getting-started/installation.mdnew file mode 100644index 0000000..326fa28--- /dev/null+++ b/docs/zh/getting-started/installation.md@@ -0,0 +1,120 @@+# 安装++Hyper-Extract 需要 **Python 3.11+**。++---++## 从 PyPI 安装++=== "pip"++ ```bash+ pip install hyper-extract+ ```++=== "uv (推荐)"++ ```bash+ uv pip install hyper-extract+ ```++=== "conda"++ ```bash+ pip install hyper-extract+ ```++---++## 验证安装++```bash+he --version+```++您应该看到类似输出:++```+Hyper-Extract CLI version 0.1.0+```++---++## 可选依赖++=== "Anthropic Claude 支持"++ ```bash+ pip install hyper-extract[anthropic]+ ```++=== "Google Gemini 支持"++ ```bash+ pip install hyper-extract[google]+ ```++=== "所有可选依赖"++ ```bash+ pip install hyper-extract[all]+ ```++---++## 配置++在使用 Hyper-Extract 之前,您需要配置 LLM API 凭证。++### 选项 1:使用 CLI(推荐)++```bash+he config init -k YOUR_OPENAI_API_KEY+```++这将在 `~/.he/config.toml` 创建配置文件。++### 选项 2:使用环境变量++```bash+export OPENAI_API_KEY=your-api-key+export OPENAI_BASE_URL=https://api.openai.com/v1 # 可选+```++### 选项 3:使用 .env 文件(仅限 Python)++在项目根目录创建 `.env` 文件:++```+OPENAI_API_KEY=your-api-key+OPENAI_BASE_URL=https://api.openai.com/v1+```++然后在 Python 代码中加载:++```python+from dotenv import load_dotenv+load_dotenv()+```++---++## 开发安装++如果您想贡献或修改源代码:++```bash+# 克隆仓库+git clone https://github.com/yifanfeng97/hyper-extract.git+cd hyper-extract++# 以可编辑模式安装开发依赖+pip install -e ".[dev]"+```++---++## 接下来做什么?++- [:octicons-arrow-right-24: CLI 快速入门](cli-quickstart.md) — 从终端进行首次提取+- [:octicons-arrow-right-24: Python 快速入门](python-quickstart.md) — 使用 Python 进行首次提取diff --git a/docs/zh/getting-started/python-quickstart.md b/docs/zh/getting-started/python-quickstart.mdnew file mode 100644index 0000000..2078cff--- /dev/null+++ b/docs/zh/getting-started/python-quickstart.md@@ -0,0 +1,125 @@+# Python 快速入门++使用 Python API 进行知识提取。++---++## 前置要求++- Hyper-Extract 已安装+- API 密钥已配置++→ 如果尚未完成,请查看[安装指南](installation.md)++---++## 基本示例++### 1. 创建模板++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", language="en")+```++### 2. 提取知识++```python+with open("document.md") as f:+ result = ka.parse(f.read())+```++### 3. 访问数据++```python+print(f"实体数量: {len(result.data.entities)}")+print(f"关系数量: {len(result.data.relations)}")+```++### 4. 可视化++```python+result.show()+```++---++## 完整示例++```python+from hyperextract import Template++# 创建模板+ka = Template.create("general/biography_graph", language="en")++# 提取知识+with open("tesla.md") as f:+ result = ka.parse(f.read())++# 访问数据+print(f"实体: {len(result.data.entities)}")+print(f"关系: {len(result.data.relations)}")++# 构建搜索索引+result.build_index()++# 搜索+results = result.search("AC motor")+print(results)++# 聊天+response = result.chat("What were Tesla's major inventions?")+print(response.content)++# 保存+result.dump("./output/")++# 加载+result.load("./output/")+```++---++## 增量更新++向现有知识库添加更多文档:++```python+# 初始提取+kb = ka.parse(doc1_text)++# 添加更多文档+kb.feed_text(doc2_text)+kb.feed_text(doc3_text)++# 保存+kb.dump("./my_kb/")+```++---++## 使用不同模板++```python+# 财务报告+ka = Template.create("finance/earnings_summary", language="en")+report = ka.parse(earnings_text)+print(report.data.revenue)++# 法律文档+ka = Template.create("legal/contract_obligation", language="en")+contract = ka.parse(contract_text)++# 医疗记录+ka = Template.create("medicine/symptom_list", language="en")+symptoms = ka.parse(medical_text)+```++---++## 下一步++- 阅读 [Python SDK 文档](../python/index.md)+- 探索 [模板库](../templates/index.md)+- 理解 [核心概念](../concepts/index.md)diff --git a/docs/zh/index.md b/docs/zh/index.mdindex c2f71e7..f4194b1 100644--- a/docs/zh/index.md+++ b/docs/zh/index.md@@ -1,167 +1,216 @@-# 🚀 Hyper-Extract-+<div align="center">+ <picture>+ <source media="(prefers-color-scheme: dark)" srcset="../assets/logo/logo-horizontal-dark.svg">+ <source media="(prefers-color-scheme: light)" srcset="../assets/logo/logo-horizontal.svg">+ <img alt="Hyper-Extract Logo" src="../assets/logo/logo-horizontal.svg" width="500">+ </picture>+</div>++<br/>++> **Transform documents into structured knowledge with one command.**+>> *"告别文档焦虑,让信息一目了然"*-> **"Stop reading. Start understanding."**-**智能知识提取 CLI — 一条命令,将文档转化为结构化知识。**+**Hyper-Extract** 是一个智能的、由 LLM(大型语言模型)驱动的知识提取框架。它将非结构化文本转换为持久的、可预测的、强类型的知识结构——从简单的列表到复杂的知识图谱、超图谱和时空图谱。-+----[](https://python.org)-[](LICENSE)-[]()+## ⚡ 5分钟快速入门-## 什么是 Hyper-Extract?+=== "CLI (终端)"-Hyper-Extract 是一个智能的、基于 LLM 的知识提取与演化框架。它极大地简化了将高度非结构化文本转换为持久化、可预测、强类型知识摘要的过程。它能够轻松地将信息提取为多种格式——从简单的**集合**(列表/集合)和 **Pydantic 模型**,到复杂的**知识图谱**、**超图**,甚至**时空图**。+ ```bash+ # 1. 安装+ pip install hyper-extract-## ✨ 核心特性+ # 2. 配置 API 密钥+ he config init -k YOUR_OPENAI_API_KEY-- 🔷 **8 种自动类型:** 从基础的 `AutoModel`/`AutoList` 到高级的 `AutoGraph`、`AutoHypergraph` 和 `AutoSpatioTemporalGraph`。-- 🧠 **10+ 提取引擎:** 开箱即用的前沿检索范式支持,如 `GraphRAG`、`LightRAG`、`Hyper-RAG` 和 `KG-Gen`。-- 📝 **声明式 YAML 模板:** 零代码提取定义。包含 6 个领域 80+ 预设模板。-- 🔄 **增量演化:** 动态输入新文档,持续扩展和更新提取的知识。+ # 3. 从文档中提取知识+ he parse tesla.md -t general/biography_graph -o ./output/ -l en-## ⚡ 快速开始+ # 4. 可视化知识图谱+ he show ./output/+ ```-### 安装+=== "Python"-```bash-uv pip install hyper-extract-```+ ```python+ from hyperextract import Template-### 命令行方式+ # 1. 创建模板+ ka = Template.create("general/biography_graph", "en")-```bash-# 配置 OpenAI API Key-he config init -k YOUR_OPENAI_API_KEY+ # 2. 提取知识+ with open("tesla.md") as f:+ result = ka.parse(f.read())-# 提取知识-he parse examples/en/tesla.md -t general/biography_graph -o ./output/ -l en+ # 3. 可视化+ ka.show(result)+ ```-# 查询知识库-he search ./output/ "Tesla 的主要成就有哪些?"+**→ 想要深入了解?** 查看[入门指南](getting-started/index.md)或直接跳转至 [CLI](cli/index.md) / [Python SDK](python/index.md) 文档。-# 可视化知识图谱-he show ./output/+----# 增量补充知识-he feed ./output/ examples/en/tesla_question.md+## ✨ Hyper-Extract 的独特之处-# 查看更新后的知识图谱-he show ./output/-```+<div class="grid cards" markdown>-### Python API 方式+- :material-shape:{ .lg .middle } **8 种自动类型**-#### 安装+ ----```bash-# 克隆仓库-git clone https://github.com/yifanfeng97/hyper-extract.git-cd hyper-extract+ 从简单的 `AutoList`/`AutoModel` 到高级的 `AutoGraph`、`AutoHypergraph` 和 `AutoSpatioTemporalGraph`。为您的数据选择正确的结构。-# 安装依赖-uv sync-```+- :material-brain:{ .lg .middle } **10+ 种提取引擎**-#### 配置+ ----```bash-# 复制环境变量示例文件-cp .env.example .env+ 内置支持 GraphRAG、LightRAG、Hyper-RAG、KG-Gen、iText2KG 等。选择最适合您用例的方法。-# 编辑 .env 文件配置 API Key 和 Base URL-# OPENAI_API_KEY=your-api-key-# OPENAI_BASE_URL=https://api.openai.com/v1-```+- :material-file-document:{ .lg .middle } **80+ 个领域模板**-#### 使用+ ----```python-import os-from dotenv import load_dotenv+ 开箱即用的金融、法律、医疗、中医和工业模板。无需配置即可使用。-# 从 .env 文件加载环境变量-load_dotenv()+- :material-sync:{ .lg .middle } **增量演进**-from hyperextract import Template+ ----# 创建模板-ka = Template.create("general/biography_graph")+ 向知识库持续添加新文档。无需重新处理所有内容。-# 解析文档-with open("examples/en/tesla.md", "r") as f:- text = f.read()-result = ka.parse(text)+</div>-# 可视化知识图谱-ka.show(result)+----# 增量补充知识-with open("examples/en/tesla_question.md", "r") as f:- new_text = f.read()-ka.feed(result, new_text)+## 🎯 选择您的路径-# 查看更新后的知识图谱-ka.show(result)-```+<div class="grid cards" markdown>++- :material-console:{ .lg .middle } __CLI 用户__++ ---++ 直接从终端处理文档。非常适合:++ - 快速知识提取+ - 批量文档处理+ - 无需编码构建知识库++ [:octicons-arrow-right-24: CLI 指南](cli/index.md)++- :material-language-python:{ .lg .middle } __Python 开发者__-## 🧩 8 种自动类型+ ----我们的框架化繁为简,让您无需编写样板代码。+ 集成到您的 Python 应用程序中。非常适合:++ - 自定义提取管道+ - 与现有工作流程集成+ - 构建 AI 驱动的应用程序++ [:octicons-arrow-right-24: Python SDK](python/index.md)-+- :material-school:{ .lg .middle } __想了解更多?__-| 类型 | 描述 |-|------|------|-| AutoModel | Pydantic 模型提取 |-| AutoList | 列表/集合提取 |-| AutoSet | 唯一集合提取 |-| AutoGraph | 知识图谱提取 |-| AutoHypergraph | 超图提取 |-| AutoTemporalGraph | 时序知识图谱 |-| AutoSpatialGraph | 空间知识图谱 |-| AutoSpatioTemporalGraph | 时空知识图谱 |+ ----### 示例:AutoGraph 可视化+ 了解核心概念和架构:++ - 自动类型如何工作+ - 选择提取方法+ - 创建自定义模板++ [:octicons-arrow-right-24: 核心概念](concepts/index.md)-使用 `AutoGraph` 提取后的知识图谱可视化:+</div>-+----## 🛠️ 系统架构+## 🧩 8 种自动类型一览++| 类型 | 用例 | 示例输出 |+|------|----------|----------------|+| **AutoModel** | 结构化摘要 | 带有特定字段的 Pydantic 模型 |+| **AutoList** | 项目集合 | 实体或事实的列表 |+| **AutoSet** | 去重集合 | 唯一项目的集合 |+| **AutoGraph** | 实体关系网络 | 带节点和边的知识图谱 |+| **AutoHypergraph** | 多实体关系 | 连接多个节点的超级边 |+| **AutoTemporalGraph** | 时间关系 | 带时间信息的图谱 |+| **AutoSpatialGraph** | 位置关系 | 带地理数据的图谱 |+| **AutoSpatioTemporalGraph** | 时间 + 空间组合 | 完整的上下文信息 |++→ [了解如何选择自动类型](concepts/autotypes.md)++---++## 🏗️ 架构概述Hyper-Extract 采用**三层架构**:-- **Auto-Types** 定义了知识提取的数据结构。8 种强类型结构(AutoModel、AutoList、AutoSet、AutoGraph、AutoHypergraph、AutoTemporalGraph、AutoSpatialGraph、AutoSpatioTemporalGraph)作为所有提取的输出格式。+```mermaid+graph TD+ A[Your Document] --> B[Template / Method]+ B --> C[Auto-Type]+ C --> D[Structured Knowledge]++ subgraph "Layer 3: Templates"+ B+ end++ subgraph "Layer 2: Methods"+ B+ end++ subgraph "Layer 1: Auto-Types"+ C+ end+```++1. **自动类型** — 定义输出数据结构(8种类型)+2. **方法** — 提供提取算法(基于 RAG 和典型方法)+3. **模板** — 提供特定领域的开箱即用配置++您可以在任意层级使用 Hyper-Extract:选择模板快速获得结果,选择方法获得更多控制,或直接使用自动类型进行完全定制。++---++## 📊 与其他工具对比++| 功能 | GraphRAG | LightRAG | KG-Gen | **Hyper-Extract** |+|---------|:--------:|:--------:|:------:|:-----------------:|+| 知识图谱 | ✅ | ✅ | ✅ | ✅ |+| 时间图谱 | ✅ | ❌ | ❌ | ✅ |+| 空间图谱 | ❌ | ❌ | ❌ | ✅ |+| 超图谱 | ❌ | ❌ | ❌ | ✅ |+| 领域模板 | ❌ | ❌ | ❌ | ✅ |+| CLI 工具 | ✅ | ❌ | ❌ | ✅ |+| 多语言 | ✅ | ❌ | ❌ | ✅ |-- **Methods** 基于 Auto-Types 提供提取算法。包括典型方法(KG-Gen、iText2KG、iText2KG*)和 RAG 增强方法(GraphRAG、LightRAG、Hyper-RAG、HypergraphRAG、Cog-RAG)。+----- **Templates** 提供领域特定的配置,包含开箱即用的 prompt 和数据结构。覆盖 6 大领域(金融、法律、医疗、中医、工业、通用),提供 80+ 预设模板,用户无需了解底层 Auto-Types 和 Methods 即可直接使用。+## 📚 文档结构-可通过 **CLI**(`he parse`、`he search`、`he show`...)或 **Python API**(`Template.create()`)使用。+- **[入门指南](getting-started/index.md)** — 安装和首次提取+- **[CLI 指南](cli/index.md)** — 完整的终端工作流程文档+- **[Python SDK](python/index.md)** — API 参考和开发者指南+- **[核心概念](concepts/index.md)** — 了解架构+- **[模板库](templates/index.md)** — 特定领域的提取模板+- **[资源](resources/index.md)** — 常见问题、故障排除和贡献指南-+----## 📈 与其他库对比+## 🤝 贡献-| 特性 | GraphRAG | LightRAG | KG-Gen | ATOM | **Hyper-Extract** |-|------|:--------:|:--------:|:------:|:----:|:-----------------:|-| 知识图谱 | ✅ | ✅ | ✅ | ✅ | ✅ |-| 时序图谱 | ✅ | ❌ | ❌ | ✅ | ✅ |-| 空间图谱 | ❌ | ❌ | ❌ | ❌ | ✅ |-| 超图 | ❌ | ❌ | ❌ | ❌ | ✅ |-| 领域模板 | ❌ | ❌ | ❌ | ❌ | ✅ |-| CLI 工具 | ✅ | ❌ | ❌ | ❌ | ✅ |-| 多语言 | ✅ | ❌ | ❌ | ❌ | ✅ |+欢迎贡献!无论是错误报告、功能请求还是文档改进,请随时提交 Issue 或 Pull Request。-## 📚 文档+[:fontawesome-brands-github: GitHub 仓库](https://github.com/yifanfeng97/hyper-extract){ .md-button .md-button--primary }-- [快速开始](cli/index.md) - 快速上手-- [领域模板](templates/index.md) - 开箱即用的模板+----## 🤝 贡献与许可+## 📄 许可证-欢迎贡献!请提交 Issues 和 PRs。-基于 **Apache-2.0** 许可证。+Hyper-Extract 采用 [Apache-2.0 许可证](https://github.com/yifanfeng97/hyper-extract/blob/main/LICENSE)。diff --git a/docs/zh/python/config.md b/docs/zh/python/config.mddeleted file mode 100644index 61a792b..0000000--- a/docs/zh/python/config.md+++ /dev/null@@ -1,74 +0,0 @@-# Python SDK 配置--## 环境变量--在项目根目录创建 `.env` 文件:--```bash-# 复制示例 env 文件-cp .env.example .env-```--编辑 `.env` 文件:--```bash-OPENAI_API_KEY=your-api-key-OPENAI_BASE_URL=https://api.openai.com/v1-```--## 加载环境变量--使用 `python-dotenv` 加载环境变量:--```bash-pip install python-dotenv-```--```python-from dotenv import load_dotenv--# 从 .env 文件加载环境变量-load_dotenv()-```--## 程序化配置--你也可以通过代码设置环境变量:--```python-import os--os.environ["OPENAI_API_KEY"] = "your-api-key"-os.environ["OPENAI_BASE_URL"] = "https://api.openai.com/v1"-```--## 配置选项--### LLM 选项--| 选项 | 环境变量 | 默认值 |-|------|---------|--------|-| API Key | `OPENAI_API_KEY` | 必填 |-| Base URL | `OPENAI_BASE_URL` | OpenAI 默认 |-| Model | `LLM_MODEL` | `gpt-4o-mini` |-| Temperature | `LLM_TEMPERATURE` | `0.0` |--### Embedder 选项--| 选项 | 环境变量 | 默认值 |-|------|---------|--------|-| Model | `EMBEDDER_MODEL` | `text-embedding-3-small` |-| Batch Size | `EMBEDDER_BATCH_SIZE` | `100` |--## 示例 .env 文件--```bash-# 必填-OPENAI_API_KEY=sk-your-api-key--# 可选-OPENAI_BASE_URL=https://api.openai.com/v1-LLM_MODEL=gpt-4o-mini-LLM_TEMPERATURE=0.0-EMBEDDER_MODEL=text-embedding-3-small-```diff --git a/docs/zh/python/core-concepts.md b/docs/zh/python/core-concepts.mdnew file mode 100644index 0000000..f73aad0--- /dev/null+++ b/docs/zh/python/core-concepts.md@@ -0,0 +1,286 @@+# 核心概念++理解 Hyper-Extract 的三大支柱:模板、自动类型和方法。++---++## 架构概览++```mermaid+graph TD+ A[Your Text] --> B{Choose Approach}+ B -->|Quick Start| C[Template]+ B -->|More Control| D[Method]+ C --> E[Auto-Type]+ D --> E+ E --> F[Structured Knowledge]+```++---++## 模板++**模板**是针对特定领域和任务预配置的提取设置。++### 什么是模板?++模板组合了:+- **自动类型** — 输出数据结构+- **提示** — LLM 指令+- **Schema** — 字段定义+- **指南** — 提取规则++### 使用模板++```python+from hyperextract import Template++# 从预设创建+ka = Template.create("general/biography_graph", language="en")++# 从文件创建+ka = Template.create("/path/to/custom.yaml", language="en")++# 列出可用的+templates = Template.list()+for path, cfg in templates.items():+ print(f"{path}: {cfg.description}")+```++### 何时使用模板++✅ **使用模板当:**+- 您想要快速结果+- 处理常见文档类型+- 需要特定领域提取+- 不想配置提示++❌ **不使用模板当:**+- 您需要完全控制提取+- 实现自定义算法+- 研究/实验++---++## 自动类型++**自动类型**定义提取知识的输出数据结构。++### 8 种自动类型++| 类型 | 描述 | 最佳用途 |+|------|-------------|----------|+| `AutoModel` | 单个结构化对象 | 摘要、报告 |+| `AutoList` | 有序集合 | 序列、排名项目 |+| `AutoSet` | 去重集合 | 唯一项目、标签 |+| `AutoGraph` | 实体关系图谱 | 知识图谱 |+| `AutoHypergraph` | 多实体边 | 复杂关系 |+| `AutoTemporalGraph` | 图谱 + 时间 | 时间线、历史 |+| `AutoSpatialGraph` | 图谱 + 空间 | 地图、位置 |+| `AutoSpatioTemporalGraph` | 图谱 + 时间 + 空间 | 完整上下文 |++### 使用自动类型++```python+from hyperextract import Template++# 图谱提取+ka = Template.create("general/knowledge_graph", "en")+result = ka.parse(text)++# 访问图谱数据+for entity in result.data.entities:+ print(f"Entity: {entity.name}")++for relation in result.data.relations:+ print(f"{relation.source} --{relation.type}--> {relation.target}")+```++### 选择自动类型++```mermaid+graph TD+ A[What do you want to extract?] --> B{Structured data?}+ B -->|Yes| C[AutoModel]+ B -->|No| D{Relationships?}+ D -->|No| E{Ordered?}+ E -->|Yes| F[AutoList]+ E -->|No| G[AutoSet]+ D -->|Yes| H{Multi-entity edges?}+ H -->|Yes| I[AutoHypergraph]+ H -->|No| J{Time data?}+ J -->|Yes| K{Space data?}+ K -->|Yes| L[AutoSpatioTemporalGraph]+ K -->|No| M[AutoTemporalGraph]+ J -->|No| N{Space data?}+ N -->|Yes| O[AutoSpatialGraph]+ N -->|No| P[AutoGraph]+```++---++## 方法++**方法**是底层提取算法。++### 方法类型++#### 基于 RAG 的方法++使用检索增强生成处理大型文档:++| 方法 | 描述 |+|--------|-------------|+| `graph_rag` | 基于社区的检索 |+| `light_rag` | 轻量级图谱 RAG |+| `hyper_rag` | 超图谱 RAG |+| `hypergraph_rag` | 高级超图谱 |+| `cog_rag` | 认知 RAG |++#### 典型方法++直接提取方法:++| 方法 | 描述 |+|--------|-------------|+| `itext2kg` | 基于三元组的提取 |+| `itext2kg_star` | 增强版 iText2KG |+| `kg_gen` | 知识图谱生成器 |+| `atom` | 带证据的时序方法 |++### 使用方法++```python+from hyperextract import Template++# 从方法创建+ka = Template.create("method/light_rag")++# 或通过路径+ka = Template.create("method/graph_rag")+```++### 模板 vs 方法++| 方面 | 模板 | 方法 |+|--------|----------|--------|+| **易用性** | ⭐⭐⭐ | ⭐⭐ |+| **灵活性** | ⭐⭐ | ⭐⭐⭐ |+| **领域匹配** | ⭐⭐⭐ | ⭐⭐ |+| **配置** | 最小 | 更多 |+| **语言** | 多语言 | 英文 |++---++## 提取管道++### 1. 文本输入++```python+text = "Your document content here..."+```++### 2. 分块(如需要)++长文档会自动拆分:+- 默认块大小:2048 字符+- 重叠:256 字符+- 多工作线程并行处理++### 3. LLM 提取++每个块由 LLM 处理:+- 使用 Pydantic schema 进行结构化输出+- 并发处理提高速度+- 合并所有块的结果++### 4. 结果++```python+result = ka.parse(text)++# result 包含:+# - result.data: 提取的知识+# - result.metadata: 提取信息+# - 搜索、聊天、保存方法+```++---++## 综合运用++### 模板方法(推荐)++```python+from hyperextract import Template++# 1. 选择模板+ka = Template.create("general/biography_graph", "en")++# 2. 提取+result = ka.parse(text)++# 3. 使用+result.show()+```++### 方法方法(高级)++```python+from hyperextract import Template++# 1. 选择方法+ka = Template.create("method/light_rag")++# 2. 提取+result = ka.parse(text)++# 3. 使用+result.build_index()+results = result.search("query")+```++### 直接使用自动类型(完全控制)++```python+from hyperextract import AutoGraph+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++# 1. 定义 schema+from pydantic import BaseModel++class Entity(BaseModel):+ name: str+ type: str++class Relation(BaseModel):+ source: str+ target: str+ type: str++class MyGraph(BaseModel):+ entities: list[Entity]+ relations: list[Relation]++# 2. 创建自动类型+llm = ChatOpenAI()+emb = OpenAIEmbeddings()++ka = AutoGraph(+ data_schema=MyGraph,+ llm_client=llm,+ embedder=emb+)++# 3. 提取+result = ka.parse(text)+```++---++## 下一步++- [使用模板](guides/using-templates.md)+- [选择方法](guides/choosing-methods.md)+- [使用自动类型](guides/working-with-autotypes.md)+- [自动类型参考](../concepts/autotypes.md)diff --git a/docs/zh/python/guides/choosing-methods.md b/docs/zh/python/guides/choosing-methods.mdnew file mode 100644index 0000000..a69c4de--- /dev/null+++ b/docs/zh/python/guides/choosing-methods.md@@ -0,0 +1,282 @@+# 选择方法++了解提取方法以及何时使用每种方法。++---++## 什么是方法?++方法是驱动知识提取的底层算法。模板提供易于使用、特定领域的配置,而方法让您直接访问提取算法。++---++## 方法类别++### 基于 RAG 的方法++**检索增强生成**方法通过结合检索与生成,在处理大型文档方面表现出色。++| 方法 | 最佳用途 | 关键特性 |+|--------|----------|-------------|+| `light_rag` | 通用用途 | 快速、轻量 |+| `graph_rag` | 大型文档 | 社区检测 |+| `hyper_rag` | 复杂关系 | N 元超边 |+| `hypergraph_rag` | 高级场景 | 增强超图谱 |+| `cog_rag` | 推理任务 | 认知检索 |++### 典型方法++**直接提取**方法,无需检索即可处理文本。++| 方法 | 最佳用途 | 关键特性 |+|--------|----------|-------------|+| `itext2kg` | 质量优先 | 高质量三元组 |+| `itext2kg_star` | 增强质量 | 改进的 iText2KG |+| `kg_gen` | 灵活性 | 可配置生成 |+| `atom` | 时序数据 | 证据归属 |++---++## 使用方法++### 基本用法++```python+from hyperextract import Template++# 从方法创建+ka = Template.create("method/light_rag")++# 提取+result = ka.parse(text)+```++### 使用自定义客户端++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract import Template++llm = ChatOpenAI(model="gpt-4o")+emb = OpenAIEmbeddings(model="text-embedding-3-large")++ka = Template.create(+ "method/graph_rag",+ llm_client=llm,+ embedder=emb+)+```++---++## 方法选择指南++### 决策树++```mermaid+graph TD+ A[Select Method] --> B{Document size?}+ B -->|Large| C[RAG-Based]+ B -->|Small/Medium| D{Need evidence?}+ D -->|Yes| E[atom]+ D -->|No| F{Prioritize quality?}+ F -->|Yes| G[iText2KG]+ F -->|No| H[kg_gen]++ C --> I{Relationship complexity?}+ I -->|Simple| J[light_rag]+ I -->|Standard| K[graph_rag]+ I -->|Complex| L[hyper_rag]+```++### 按用例++#### 快速提取(小文档)++```python+# 快速简单+ka = Template.create("method/kg_gen")+```++#### 高质量结果++```python+# 最佳提取质量+ka = Template.create("method/itext2kg_star")+```++#### 大型文档++```python+# 高效处理+ka = Template.create("method/light_rag")+```++#### 复杂关系++```python+# 多实体关系+ka = Template.create("method/hyper_rag")+```++#### 时序分析++```python+# 带证据的基于时间的方法+ka = Template.create("method/atom")+```++---++## RAG vs 典型方法比较++| 功能 | 基于 RAG | 典型方法 |+|---------|-----------|--------|+| **文档大小** | 大型(10k+ 词) | 小型-中型 |+| **速度** | 较慢(检索步骤) | 较快 |+| **内存** | 较高 | 较低 |+| **质量** | 大型文档良好 | 小型文档更好 |+| **上下文处理** | 优秀 | 良好 |+| **用例** | 书籍、论文、报告 | 文章、摘要 |++---++## 方法详情++### light_rag++最佳用途:通用、快速提取++```python+ka = Template.create("method/light_rag")++# 特点:+# - 最快的 RAG 方法+# - 二元边(source-target)+# - 速度/质量的良好平衡+```++### graph_rag++最佳用途:具有社区结构的大型文档++```python+ka = Template.create("method/graph_rag")++# 特点:+# - 社区检测+# - 层次化摘要+# - 最适合非常大的文档+```++### hyper_rag++最佳用途:复杂多实体关系++```python+ka = Template.create("method/hyper_rag")++# 特点:+# - 超边(连接 2+ 个实体)+# - 捕获复杂关系+# - 更丰富的图谱结构+```++### itext2kg++最佳用途:高质量三元组提取++```python+ka = Template.create("method/itext2kg")++# 特点:+# - 针对三元组质量优化+# - 迭代精炼+# - 适合知识库构建+```++### atom++最佳用途:带证据的时序分析++```python+ka = Template.create("method/atom")++# 特点:+# - 时序事实提取+# - 证据归属+# - 置信度评分+```++---++## 列出方法++```python+from hyperextract import Template+from hyperextract.methods import list_methods++# 列出所有方法+methods = list_methods()+for name, info in methods.items():+ print(f"{name}: {info['description']}")+ print(f" Type: {info['type']}")+```++---++## 方法配置++方法可以通过参数配置:++```python+from hyperextract import Template++# 大多数方法使用默认配置+# 高级用户可以直接创建方法实例+from hyperextract.methods import Light_RAG+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++llm = ChatOpenAI()+emb = OpenAIEmbeddings()++ka = Light_RAG(+ llm_client=llm,+ embedder=emb,+ # 方法特定参数+ chunk_size=1024,+ max_workers=5+)+```++---++## 最佳实践++1. **从 light_rag 开始** — 大多数情况的良好默认值+2. **使用 itext2kg 保证质量** — 当提取质量至关重要时+3. **复杂数据尝试 hyper_rag** — 当关系多方面时+4. **时序数据考虑 atom** — 当时间重要时+5. **在数据上基准测试** — 方法在不同内容上表现不同++---++## 何时使用模板 vs 方法++| 场景 | 建议 |+|----------|----------------|+| 快速开始 | 模板 |+| 特定领域任务 | 模板 |+| 生产系统 | 模板 |+| 研究/实验 | 方法 |+| 自定义需求 | 方法 |+| 性能优化 | 方法 |+| 完全控制 | 方法 |++---++## 另请参见++- [使用模板](using-templates.md)+- [模板库](../../templates/index.md)+- [方法概念文档](../../concepts/methods.md)diff --git a/docs/zh/python/guides/incremental-updates.md b/docs/zh/python/guides/incremental-updates.mdnew file mode 100644index 0000000..6bf96d0--- /dev/null+++ b/docs/zh/python/guides/incremental-updates.md@@ -0,0 +1,278 @@+# 增量更新++在不重新处理的情况下向现有知识库添加新信息。++---++## 概述++`feed_text()` 方法允许您增量地向现有知识库添加文档:++1. **保留现有数据** — 不会覆盖已有内容+2. **智能合并** — 处理重复和冲突+3. **更新元数据** — 跟踪更新发生的时间++---++## 基本用法++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")++# 初始提取+result = ka.parse(initial_text)+print(f"初始: {len(result.data.entities)} 个实体")++# 添加更多内容+result.feed_text(additional_text)+print(f"Feed 后: {len(result.data.entities)} 个实体")+```++---++## 用例++### 随着时间构建知识++```python+ka = Template.create("general/biography_graph", "en")+kb = ka.parse(early_life_text)++# 添加职业信息+kb.feed_text(career_text)++# 添加晚年+kb.feed_text(later_years_text)++# 最终结果组合所有时期+kb.show()+```++### 处理多个文档++```python+ka = Template.create("general/concept_graph", "en")+kb = ka.parse(documents[0])++for doc in documents[1:]:+ kb.feed_text(doc)+ print(f"已添加文档,现在有 {len(kb.data.entities)} 个实体")+```++### 使用新信息更新++```python+# 原始提取+kb = ka.parse(original_paper)++# 添加更正/更新+kb.feed_text(corrections)++# 保存更新版本+kb.dump("./updated_kb/")+```++---++## 合并如何工作++### 实体合并++```python+# 如果相同实体出现在两个文本中+# 结果:合并描述,字段组合+```++### 关系合并++```python+# 如果相同关系存在+# 结果:使用最新信息更新+```++### 重复处理++```python+# 检测并合并完全重复的+# 相似的重复(相似名称)可能创建单独条目+```++---++## 最佳实践++### 1. 相同的模板类型++确保使用兼容的自动类型:++```python+# 好:相同模板类型+kb = ka.parse(text1) # biography_graph+kb.feed_text(text2) # 相同类型++# 注意:混合类型可能导致问题+```++### 2. Feed 后重建索引++```python+kb.feed_text(new_text)+kb.build_index() # 搜索/聊天需要+```++### 3. 保存中间状态++```python+# 重大更新后保存+kb.feed_text(chapter1)+kb.dump("./kb_v1/")++kb.feed_text(chapter2)+kb.dump("./kb_v2/")+```++### 4. 监控增长++```python+initial_count = len(kb.data.entities)+kb.feed_text(new_text)+new_count = len(kb.data.entities)++print(f"添加了 {new_count - initial_count} 个新实体")+```++---++## 完整示例++```python+"""从多个来源增量构建知识库。"""++from hyperextract import Template+from pathlib import Path++def build_knowledge_base(source_dir, output_dir):+ ka = Template.create("general/biography_graph", "en")++ # 获取所有文本文件+ files = sorted(Path(source_dir).glob("*.md"))++ if not files:+ print("未找到文件")+ return++ # 从第一个文件初始提取+ print(f"处理 {files[0].name}...")+ kb = ka.parse(files[0].read_text())++ # Feed 剩余文件+ for file in files[1:]:+ print(f"添加 {file.name}...")+ kb.feed_text(file.read_text())+ print(f" 现在有 {len(kb.data.entities)} 个实体")++ # 为搜索/聊天构建索引+ print("构建搜索索引...")+ kb.build_index()++ # 保存+ print(f"保存到 {output_dir}...")+ kb.dump(output_dir)++ print("完成!")+ return kb++# 用法+kb = build_knowledge_base("./sources/", "./combined_kb/")+```++---++## 比较:Parse vs Feed++| 操作 | 使用时机 | 结果 |+|-----------|----------|--------|+| `parse()` | 从头开始 | 新知识库 |+| `feed_text()` | 添加到现有 | 更新的知识库 |++### 链式操作++```python+# Parse 返回新实例+result1 = ka.parse(text1)+result2 = ka.parse(text2) # 独立于 result1++# Feed 修改现有+result1.feed_text(text2) # result1 被更新+```++---++## 限制++### 1. 内存使用++大型知识库消耗内存:++```python+# 监控大小+import sys+size = sys.getsizeof(kb.data)+print(f"知识库大小: {size} 字节")+```++### 2. 合并质量++合并并不完美:+- 相似但不完全相同的实体可能不会合并+- 非常大的知识库可能会变慢++### 3. 索引过期++Feed 后始终重建:++```python+kb.feed_text(text)+kb.build_index() # 不要忘记!+```++---++## 故障排除++### "内存错误"++分批处理:++```python+for batch in chunks(documents, batch_size=5):+ for doc in batch:+ kb.feed_text(doc)+ kb.dump(f"./kb_checkpoint/") # 定期保存+```++### "重复实体"++规范化实体名称:++```python+# 不要"Nikola Tesla"和"Tesla"混用+# 使用一致的命名+```++### "索引过期"++```python+# 忘记重建了?+kb.build_index()+```++---++## 另请参见++- [保存和加载](saving-loading.md)+- [搜索和聊天](search-and-chat.md)+- [使用自动类型](working-with-autotypes.md)diff --git a/docs/zh/python/guides/saving-loading.md b/docs/zh/python/guides/saving-loading.mdnew file mode 100644index 0000000..f2c611c--- /dev/null+++ b/docs/zh/python/guides/saving-loading.md@@ -0,0 +1,347 @@+# 保存和加载++将知识库持久化到磁盘并在以后恢复。++---++## 概述++Hyper-Extract 提供序列化支持:+- **数据** — 提取的实体和关系+- **元数据** — 提取设置和时间戳+- **索引** — 向量搜索索引(可选)++---++## 保存知识库++### 基本保存++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")+result = ka.parse(text)++# 保存到目录+result.dump("./my_kb/")+```++### 保存结构++```+./my_kb/+├── data.json # 提取的知识+├── metadata.json # 提取信息+└── index/ # 搜索索引(如已构建)+ ├── index.faiss+ └── docstore.json+```++### 保存前++```python+# 首先构建索引(如需要)+result.build_index()++# 然后保存+result.dump("./my_kb/")+```++---++## 加载知识库++### 基本加载++```python+from hyperextract import Template++# 创建模板(必须与原始匹配)+ka = Template.create("general/biography_graph", "en")++# 加载保存的数据+ka.load("./my_kb/")++# 使用+print(f"加载了 {len(ka.data.entities)} 个实体")+```++### 验证加载的数据++```python+ka.load("./my_kb/")++# 检查是否为空+if ka.empty():+ print("警告:未加载数据")+else:+ print(f"实体: {len(ka.data.entities)}")+ print(f"关系: {len(ka.data.relations)}")+```++---++## 用例++### 长期存储++```python+# 提取并保存+ka = Template.create("general/concept_graph", "en")+result = ka.parse(research_paper)+result.build_index()+result.dump("./research_paper_kb/")++# 数周后使用+ka2 = Template.create("general/concept_graph", "en")+ka2.load("./research_paper_kb/")+response = ka2.chat("主要发现是什么?")+```++### 共享知识库++```python+# 保存到共享位置+result.dump("/shared/kb/project_x/")++# 其他人可以加载+ka.load("/shared/kb/project_x/")+```++### 备份和版本控制++```python+from datetime import datetime++# 带时间戳保存+timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")+result.dump(f"./backups/kb_{timestamp}/")++# 或版本化+result.dump("./kb_v1/")+# ... 稍后更新 ...+result.dump("./kb_v2/")+```++---++## 使用文件系统++### 检查现有 KB++```python+from pathlib import Path++kb_path = Path("./my_kb/")++if kb_path.exists() and (kb_path / "data.json").exists():+ print("知识库存在,正在加载...")+ ka.load(kb_path)+else:+ print("创建新知识库...")+ result = ka.parse(text)+ result.dump(kb_path)+```++### 列出保存的 KB++```python+import os++kb_dirs = [d for d in os.listdir("./") if os.path.isdir(d) and "_kb" in d]+print("可用知识库:", kb_dirs)+```++### 移动/复制++```python+import shutil++# 复制知识库+shutil.copytree("./kb_v1/", "./kb_backup/")++# 移动知识库+shutil.move("./old_location/", "./new_location/")+```++---++## 元数据++### 访问元数据++```python+result.dump("./my_kb/")++# 元数据自动保存+# 内容:+# - template: 使用的模板+# - lang: 语言+# - created_at: 创建时间戳+# - updated_at: 最后更新时间戳+```++### 自定义元数据++```python+# 添加自定义元数据+result.metadata["project"] = "研究项目 X"+result.metadata["version"] = "1.0"++result.dump("./my_kb/")+```++---++## 完整示例++```python+"""使用保存/加载管理知识库。"""++from hyperextract import Template+from pathlib import Path+import json++class KnowledgeBaseManager:+ def __init__(self, storage_dir="./knowledge_bases/"):+ self.storage_dir = Path(storage_dir)+ self.storage_dir.mkdir(exist_ok=True)++ def save(self, kb, name):+ """保存知识库。"""+ kb_path = self.storage_dir / name+ kb.dump(kb_path)+ print(f"保存到 {kb_path}")+ return kb_path++ def load(self, name, template="general/biography_graph", lang="en"):+ """加载知识库。"""+ kb_path = self.storage_dir / name++ if not kb_path.exists():+ raise FileNotFoundError(f"未找到知识库: {name}")++ ka = Template.create(template, lang)+ ka.load(kb_path)+ return ka++ def list(self):+ """列出可用的知识库。"""+ return [d.name for d in self.storage_dir.iterdir() if d.is_dir()]++ def info(self, name):+ """获取知识库信息。"""+ kb_path = self.storage_dir / name+ meta_path = kb_path / "metadata.json"++ if meta_path.exists():+ return json.loads(meta_path.read_text())+ return None++# 用法+manager = KnowledgeBaseManager()++# 保存+ka = Template.create("general/biography_graph", "en")+result = ka.parse(text)+manager.save(result, "tesla_biography")++# 列出+print(manager.list()) # ['tesla_biography']++# 加载+kb = manager.load("tesla_biography")+print(kb.chat("特斯拉发明了什么?"))+```++---++## 最佳实践++### 1. 加载时匹配模板++```python+# 使用模板 X 保存+ka = Template.create("general/biography_graph", "en")++# 使用相同模板加载+ka2 = Template.create("general/biography_graph", "en")+ka2.load("./kb/")+```++### 2. 加载后构建索引++```python+ka.load("./kb/")++# 索引可能需要重建+if not ka.has_index(): # 检查索引是否加载+ ka.build_index()+```++### 3. 使用前验证++```python+try:+ ka.load("./kb/")+ if ka.empty():+ print("警告:空知识库")+except FileNotFoundError:+ print("未找到知识库")+```++### 4. 使用描述性名称++```python+# 好+result.dump("./kb/tesla_2024_01_15/")++# 避免+result.dump("./kb/temp/")+```++---++## 故障排除++### "文件未找到"++```python+from pathlib import Path++kb_path = Path("./my_kb/")+if not kb_path.exists():+ print(f"未找到目录: {kb_path}")+ print(f"可用: {list(Path('.').glob('*/'))}")+```++### "数据损坏"++```python+# 检查数据文件+import json+data = json.load(open("./my_kb/data.json"))+print(f"Keys: {data.keys()}")+```++### "索引未加载"++```python+ka.load("./kb/")++# 检查索引是否存在+if (Path("./kb/") / "index").exists():+ print("索引目录存在")+else:+ print("无索引,正在构建...")+ ka.build_index()+```++---++## 另请参见++- [增量更新](incremental-updates.md)+- [搜索和聊天](search-and-chat.md)+- [CLI `he parse` 命令](../../cli/commands/parse.md)diff --git a/docs/zh/python/guides/search-and-chat.md b/docs/zh/python/guides/search-and-chat.mdnew file mode 100644index 0000000..0fa4d52--- /dev/null+++ b/docs/zh/python/guides/search-and-chat.md@@ -0,0 +1,273 @@+# 搜索和聊天++使用语义搜索和对话 AI 查询您的知识库。++---++## 概述++提取知识后,您可以通过两种方式与其交互:++- **搜索** — 查找特定的实体和关系+- **聊天** — 进行自然语言对话++两者都需要首先构建**搜索索引**。++---++## 构建索引++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")+result = ka.parse(text)++# 构建搜索索引+result.build_index()+```++**注意:** 搜索/聊天操作之前需要构建索引。++---++## 语义搜索++### 基本搜索++```python+# 搜索相关项目+results = result.search("inventions", top_k=5)++for item in results:+ print(item)+```++### 搜索参数++```python+results = result.search(+ query="electrical engineering achievements",+ top_k=10 # 结果数量+)+```++### 处理结果++```python+results = result.search("Nobel Prize")++for item in results:+ # 检查项目类型+ if hasattr(item, 'name'): # 实体+ print(f"Entity: {item.name}")+ elif hasattr(item, 'source'): # 关系+ print(f"Relation: {item.source} -> {item.target}")+```++### 搜索用例++```python+# 查找特定人物+people = result.search("scientists who worked with Tesla", top_k=10)++# 查找概念+concepts = result.search("alternating current system", top_k=5)++# 查找事件+events = result.search("important dates in Tesla's life", top_k=10)+```++---++## 聊天接口++### 单个问题++```python+# 提问+response = result.chat("特斯拉的主要成就是什么?")++print(response.content)+```++### 访问检索到的上下文++```python+response = result.chat("特斯拉发明了什么?")++print(response.content)++# 访问用于生成响应的项目+if "retrieved_items" in response.additional_kwargs:+ items = response.additional_kwargs["retrieved_items"]+ print(f"基于 {len(items)} 个项目")+```++### 聊天参数++```python+response = result.chat(+ query="解释电流战争",+ top_k=10 # 复杂问题需要更多上下文+)+```++### 聊天用例++```python+# 摘要+summary = result.chat("用三句话总结特斯拉的职业生涯")++# 解释+explanation = result.chat("特斯拉线圈的意义是什么?")++# 比较+comparison = result.chat("特斯拉的方法与爱迪生有何不同?")++# 时间线+timeline = result.chat("特斯拉在 1880-1890 年间发生了什么?")+```++---++## 搜索 vs 聊天++| 功能 | 搜索 | 聊天 |+|---------|--------|------|+| **返回** | 原始实体/关系 | 自然语言答案 |+| **速度** | 快 | 较慢(LLM 调用) |+| **用于** | 查找特定数据 | 理解/解释 |+| **精度** | 精确匹配 | 解释性 |+| **输出** | 结构化 | 自由文本 |++### 何时使用++**使用搜索当:**+- 您需要特定实体+- 构建报告或摘要+- 导出数据+- 需要快速查找++**使用聊天当:**+- 解释概念+- 回答复杂问题+- 摘要内容+- 交互式探索++---++## 高级模式++### 先搜索后聊天++```python+# 首先,搜索特定项目+items = result.search("wireless technology", top_k=5)++# 然后,询问它们+if items:+ context = ", ".join([item.name for item in items if hasattr(item, 'name')])+ response = result.chat(f"解释 {context} 的意义")+ print(response.content)+```++### 迭代探索++```python+# 从广泛开始+response = result.chat("本文档中的主要主题是什么?")+print(response.content)++# 深入了解+response = result.chat("更多关于特斯拉线圈的信息")+print(response.content)++# 特定问题+response = result.chat("特斯拉线圈是如何工作的?")+print(response.content)+```++### 构建研究助手++```python+class ResearchAssistant:+ def __init__(self, kb_path):+ self.ka = Template.create("general/concept_graph", "en")+ self.ka.load(kb_path)+ self.ka.build_index()++ def ask(self, question):+ response = self.ka.chat(question)+ return response.content++ def find(self, query, n=5):+ return self.ka.search(query, top_k=n)++ def summarize(self):+ return self.ask("总结本文的主要观点")++# 用法+assistant = ResearchAssistant("./paper_kb/")+print(assistant.summarize())+print(assistant.ask("有哪些局限性?"))+```++---++## 最佳实践++### 搜索技巧++1. **使用自然语言** — "无线传输"而非"无线"+2. **要具体** — "特斯拉的专利"而非"特斯拉"+3. **广泛查询时增加 top_k** — `top_k=10` 或更多+4. **按类型筛选结果** — 检查 `hasattr(item, 'name')`++### 聊天技巧++1. **提出清晰的问题** — 具体问题得到更好的答案+2. **使用上下文** — 逐步构建理解+3. **调整 top_k** — 复杂问题需要更多上下文+4. **检查来源** — 审查 `retrieved_items` 以确保准确性++---++## 故障排除++### "索引未构建"++```python+# 错误:需要先构建索引+result.build_index()+```++### "未找到结果"++```python+# 尝试不同的措辞+results = result.search("inventions") # 尝试同义词+results = result.search("discoveries")+results = result.search("contributions")++# 增加 top_k+results = result.search("Tesla", top_k=20)+```++### "无关的聊天响应"++```python+# 增加上下文+response = result.chat(question, top_k=10)++# 或重新表述问题+response = result.chat("请更具体地说明:...")+```++---++## 另请参见++- [使用自动类型](working-with-autotypes.md)+- [增量更新](incremental-updates.md)+- [保存和加载](saving-loading.md)diff --git a/docs/zh/python/guides/using-templates.md b/docs/zh/python/guides/using-templates.mdnew file mode 100644index 0000000..1326281--- /dev/null+++ b/docs/zh/python/guides/using-templates.md@@ -0,0 +1,292 @@+# 使用模板++学习如何使用和自定义模板进行知识提取。++---++## 什么是模板?++模板是预配置的提取设置,组合了:+- **自动类型** — 输出数据结构+- **提示** — LLM 指令+- **Schema** — 字段定义和类型+- **指南** — 提取规则和约束++---++## 内置模板++Hyper-Extract 包含 80+ 个跨 6 个领域的模板:++| 领域 | 模板 | 用例 |+|--------|-----------|-----------|+| `general` | 基础类型、传记、概念图谱 | 常见提取任务 |+| `finance` | 财报、风险、所有权、时间线 | 财务分析 |+| `legal` | 合同、案件、合规 | 法律文档处理 |+| `medicine` | 解剖、药物、治疗 | 医学文本分析 |+| `tcm` | 草药、方剂、证候 | 中医 |+| `industry` | 设备、安全、操作 | 工业文档 |++---++## 创建模板++### 从预设创建++```python+from hyperextract import Template++# 基本用法+ka = Template.create("general/biography_graph", language="en")++# 使用自定义客户端+from langchain_openai import ChatOpenAI, OpenAIEmbeddings++llm = ChatOpenAI(model="gpt-4o")+emb = OpenAIEmbeddings()++ka = Template.create(+ "general/biography_graph",+ language="en",+ llm_client=llm,+ embedder=emb+)+```++### 从文件创建++```python+# 加载自定义 YAML 模板+ka = Template.create(+ "/path/to/my_template.yaml",+ language="en"+)+```++### 从方法创建++```python+# 使用提取方法而非模板+ka = Template.create("method/light_rag")+```++---++## 列出模板++### 所有模板++```python+from hyperextract import Template++templates = Template.list()++for path, cfg in templates.items():+ print(f"{path}: {cfg.description}")+```++### 筛选列表++```python+# 按类型+graphs = Template.list(filter_by_type="graph")++# 按标签+biography = Template.list(filter_by_tag="biography")++# 按语言+zh_templates = Template.list(filter_by_language="zh")++# 组合+results = Template.list(+ filter_by_type="graph",+ filter_by_language="en"+)+```++### 搜索模板++```python+# 在名称和描述中搜索+results = Template.list(filter_by_query="financial")+```++---++## 模板配置++### 获取模板信息++```python+from hyperextract import Template++# 获取模板配置+cfg = Template.get("general/biography_graph")++print(cfg.name) # biography_graph+print(cfg.type) # temporal_graph+print(cfg.description) # 模板描述+```++### 模板属性++```python+cfg = Template.get("general/biography_graph")++# 属性+print(cfg.name) # 模板名称+print(cfg.type) # 自动类型(graph、list 等)+print(cfg.description) # 人类可读描述+print(cfg.tags) # 关联标签+print(cfg.language) # 支持的语言+```++---++## 常见模板模式++### 模式 1:传记分析++```python+from hyperextract import Template++ka = Template.create("general/biography_graph", "en")++with open("biography.md") as f:+ result = ka.parse(f.read())++# 访问时间线数据+for relation in result.data.relations:+ if hasattr(relation, 'time'):+ print(f"{relation.time}: {relation.source} -> {relation.target}")++result.show() # 可视化生平时间线+```++### 模式 2:财务报告++```python+from hyperextract import Template++ka = Template.create("finance/earnings_summary", "en")++report = ka.parse(earnings_text)++# 访问财务数据+print(f"收入: {report.data.revenue}")+print(f"每股收益: {report.data.eps}")+print(f"同比增长: {report.data.yoy_growth}%")+```++### 模式 3:法律合同++```python+from hyperextract import Template++ka = Template.create("legal/contract_obligation", "en")++contract = ka.parse(contract_text)++# 列出义务+for obligation in contract.data.obligations:+ print(f"当事人: {obligation.party}")+ print(f"义务: {obligation.description}")+ print(f"截止日期: {obligation.deadline}")+```++### 模式 4:研究论文++```python+from hyperextract import Template++ka = Template.create("general/concept_graph", "en")++paper = ka.parse(paper_text)++# 构建可搜索知识库+paper.build_index()++# 查询发现+response = paper.chat("主要贡献是什么?")+print(response.content)+```++---++## 多语言支持++模板支持英文和中文:++```python+# 英文文档+ka_en = Template.create("general/biography_graph", "en")+result_en = ka_en.parse(english_text)++# 中文文档+ka_zh = Template.create("general/biography_graph", "zh")+result_zh = ka_zh.parse(chinese_text)+```++**注意:** 使用与文档匹配的语言以获得最佳结果。++---++## 模板缓存++模板被加载和缓存以提高效率:++```python+# 首次调用加载模板+ka1 = Template.create("general/biography_graph", "en")++# 第二次调用使用缓存版本+ka2 = Template.create("general/biography_graph", "en")++# 两者是具有相同配置但独立的实例+```++---++## 错误处理++### 模板未找到++```python+from hyperextract import Template++try:+ ka = Template.create("nonexistent/template")+except ValueError as e:+ print(f"模板未找到: {e}")++ # 列出可用的+ available = Template.list()+ print("可用模板:", list(available.keys()))+```++### 不支持的语言++```python+cfg = Template.get("general/biography_graph")+print(cfg.language) # 检查支持的语言++# 如果语言不支持将抛出错误+ka = Template.create("general/biography_graph", "fr") # 可能失败+```++---++## 最佳实践++1. **选择特定领域的模板** — 比通用模板效果更好+2. **语言匹配文档** — 提高提取质量+3. **缓存模板实例** — 为多个文档重用+4. **检查模板输出 schema** — 了解期望的字段++---++## 另请参见++- [模板库](../../templates/index.md) — 浏览所有模板+- [创建自定义模板](custom-templates.md) — 编写自己的模板+- [选择方法](choosing-methods.md) — 何时使用方法diff --git a/docs/zh/python/guides/working-with-autotypes.md b/docs/zh/python/guides/working-with-autotypes.mdnew file mode 100644index 0000000..b37675d--- /dev/null+++ b/docs/zh/python/guides/working-with-autotypes.md@@ -0,0 +1,319 @@+# 使用自动类型++提取、操作和利用结构化知识数据。++---++## 什么是自动类型?++自动类型是智能数据结构,自动从文本中提取和组织知识。它们提供:++- **类型安全的 schema** — 基于 Pydantic 的验证+- **LLM 驱动的提取** — 自动内容处理+- **内置操作** — 搜索、合并、可视化+- **序列化** — 保存/加载到磁盘++---++## 8 种自动类型++### 标量类型++#### AutoModel++单个结构化对象提取。++```python+from hyperextract import Template++ka = Template.create("finance/earnings_summary", "en")+result = ka.parse(report_text)++# 直接访问字段+print(result.data.company_name)+print(result.data.revenue)+print(result.data.eps)+```++#### AutoList++有序集合提取。++```python+from hyperextract import Template++ka = Template.create("general/base_list", "en")+result = ka.parse(text)++# 遍历项目+for item in result.data.items:+ print(item)++# 按索引访问+first = result.data.items[0]+```++#### AutoSet++去重集合提取。++```python+from hyperextract import Template++ka = Template.create("general/base_set", "en")+result = ka.parse(text)++# 项目是唯一的+for item in result.data.items:+ print(item)+```++### 图谱类型++#### AutoGraph++实体关系图谱提取。++```python+from hyperextract import Template++ka = Template.create("general/knowledge_graph", "en")+result = ka.parse(text)++# 访问实体+for entity in result.data.entities:+ print(f"{entity.name} ({entity.type})")+ print(f" 描述: {entity.description}")++# 访问关系+for relation in result.data.relations:+ print(f"{relation.source} --{relation.type}--> {relation.target}")+```++#### AutoHypergraph++多实体关系提取。++```python+from hyperextract import Template++ka = Template.create("general/base_hypergraph", "en")+result = ka.parse(text)++# 超边连接多个实体+for hyperedge in result.data.hyperedges:+ print(f"类型: {hyperedge.type}")+ print(f"实体: {hyperedge.entities}")+```++#### AutoTemporalGraph++基于时间的图谱提取。++```python+from hyperextract import Template++ka = Template.create("general/base_temporal_graph", "en")+result = ka.parse(text)++# 关系包含时间信息+for relation in result.data.relations:+ print(f"{relation.source} --{relation.type}--> {relation.target}")+ if hasattr(relation, 'time'):+ print(f" 时间: {relation.time}")+```++#### AutoSpatialGraph++基于位置的图谱提取。++```python+from hyperextract import Template++ka = Template.create("general/base_spatial_graph", "en")+result = ka.parse(text)++# 实体/关系包含位置+for entity in result.data.entities:+ if hasattr(entity, 'location'):+ print(f"{entity.name} at {entity.location}")+```++#### AutoSpatioTemporalGraph++时间和空间组合提取。++```python+from hyperextract import Template++ka = Template.create("general/base_spatio_temporal_graph", "en")+result = ka.parse(text)++# 完整上下文:谁、什么、何时、何地+for relation in result.data.relations:+ print(f"事件: {relation.type}")+ if hasattr(relation, 'time'):+ print(f" 何时: {relation.time}")+ if hasattr(relation, 'location'):+ print(f" 何地: {relation.location}")+```++---++## 常见操作++### 检查是否为空++```python+if result.empty():+ print("未提取到数据")+else:+ print(f"提取了 {len(result.data.entities)} 个实体")+```++### 清除数据++```python+# 清除所有数据+result.clear()++# 仅清除索引+result.clear_index()+```++### 合并实例++```python+# 两次独立提取+result1 = ka.parse(text1)+result2 = ka.parse(text2)++# 合并到新实例+combined = result1 + result2+```++---++## 访问数据++### 属性访问++```python+result = ka.parse(text)++# 直接属性访问(Pydantic 模型)+entities = result.data.entities+relations = result.data.relations++# 字典转换+data_dict = result.data.model_dump()+```++### JSON 导出++```python+import json++# 导出为 JSON+json_data = result.data.model_dump_json(indent=2)++# 保存到文件+with open("output.json", "w") as f:+ f.write(json_data)+```++---++## 处理结果++### 迭代模式++```python+# 遍历实体+for entity in result.data.entities:+ print(f"名称: {entity.name}")+ print(f"类型: {entity.type}")+ if hasattr(entity, 'description'):+ print(f"描述: {entity.description}")++# 带筛选的关系迭代+for relation in result.data.relations:+ if relation.type == "worked_with":+ print(f"{relation.source} 与 {relation.target} 合作")+```++### 筛选++```python+# 按类型筛选实体+people = [e for e in result.data.entities if e.type == "person"]+organizations = [e for e in result.data.entities if e.type == "organization"]++# 筛选关系+inventions = [r for r in result.data.relations if "invent" in r.type.lower()]+```++### 统计++```python+# 基本计数+entity_count = len(result.data.entities)+relation_count = len(result.data.relations)++# 类型分布+from collections import Counter+entity_types = Counter(e.type for e in result.data.entities)+relation_types = Counter(r.type for r in result.data.relations)++print(f"实体: {entity_types}")+print(f"关系: {relation_types}")+```++---++## 高级用法++### 自定义 Schema 访问++```python+# 访问 schema+schema = result.data_schema++print(schema.model_fields) # 可用字段+```++### 原始数据访问++```python+# 如需要访问内部数据+internal_data = result._data+```++### 类型检查++```python+from hyperextract import AutoGraph, AutoList++# 检查实例类型+if isinstance(result, AutoGraph):+ print("图谱提取")+elif isinstance(result, AutoList):+ print("列表提取")+```++---++## 最佳实践++1. **访问可选字段前检查 hasattr** — Schema 字段可能不同+2. **处理空结果** — 始终检查 `empty()`+3. **使用 model_dump 进行序列化** — 正确的 JSON 转换+4. **利用类型提示** — IDE 自动完成支持++---++## 另请参见++- [搜索和聊天](search-and-chat.md)+- [保存和加载](saving-loading.md)+- [自动类型参考](../../concepts/autotypes.md)diff --git a/docs/zh/python/index.md b/docs/zh/python/index.mdindex 9bcab12..f81e9f3 100644--- a/docs/zh/python/index.md+++ b/docs/zh/python/index.md@@ -1,6 +1,8 @@# Python SDK-Hyper-Extract Python API。+使用 Hyper-Extract Python API 构建知识提取管道。++---## 安装@@ -8,48 +10,226 @@ Hyper-Extract Python API。pip install hyper-extract```-## 快速开始+开发安装:-```python-from dotenv import load_dotenv+```bash+pip install hyper-extract[dev]+```++----# 从 .env 文件加载环境变量-load_dotenv()+## 快速示例+```pythonfrom hyperextract import Template# 创建模板-ka = Template.create("general/biography_graph", "zh")+ka = Template.create("general/biography_graph", language="en")-# 解析文档-with open("examples/zh/sushi.md", "r", encoding="utf-8") as f:- text = f.read()-result = ka.parse(text)+# 提取知识+with open("document.md") as f:+ result = ka.parse(f.read())++# 访问数据+print(f"实体数: {len(result.data.entities)}")+print(f"关系数: {len(result.data.relations)}")++# 可视化+result.show()+```++---++## 核心类++### Template++模板化提取的主要入口点。++```python+from hyperextract import Template++# 从预设创建+ka = Template.create("general/biography_graph", language="en")++# 从自定义 YAML 创建+ka = Template.create("/path/to/custom_template.yaml", language="en")++# 从方法创建+ka = Template.create("method/light_rag")+```++### 自动类型++8 种用于不同提取需求的数据结构类型:++| 类 | 用例 |+|-------|----------|+| `AutoModel` | 结构化数据模型 |+| `AutoList` | 有序集合 |+| `AutoSet` | 去重集合 |+| `AutoGraph` | 实体关系图谱 |+| `AutoHypergraph` | 多实体关系 |+| `AutoTemporalGraph` | 基于时间的关系 |+| `AutoSpatialGraph` | 基于位置的关系 |+| `AutoSpatioTemporalGraph` | 时间 + 空间组合 |++---++## API 概览++### Template API++```python+# 创建+ka = Template.create(template_path, language="en")++# 提取+result = ka.parse(text) # 新提取+result.feed_text(text) # 添加到现有++# 查询+result.build_index() # 构建搜索索引+results = result.search(query) # 语义搜索+response = result.chat(query) # 与知识聊天++# 持久化+result.dump("./output/") # 保存到磁盘+result.load("./output/") # 从磁盘加载++# 可视化+result.show() # 交互式可视化+```++---++## 文档结构++- **[快速入门](quickstart.md)** — 您的首次 Python 提取+- **[核心概念](core-concepts.md)** — 模板、自动类型、方法详解+- **指南:**+ - [使用模板](guides/using-templates.md)+ - [选择方法](guides/choosing-methods.md)+ - [使用自动类型](guides/working-with-autotypes.md)+ - [搜索和聊天](guides/search-and-chat.md)+ - [增量更新](guides/incremental-updates.md)+ - [保存和加载](guides/saving-loading.md)+- **API 参考:**+ - [模板](api-reference/template.md)+ - [自动类型](api-reference/autotypes.md)+ - [方法](api-reference/methods.md)++---++## 用例示例++### 研究论文分析++```python+from hyperextract import Template++ka = Template.create("general/concept_graph", language="en")++with open("paper.md") as f:+ paper = ka.parse(f.read())++# 构建可搜索知识库+paper.build_index()++# 提问+response = paper.chat("主要贡献是什么?")+print(response.content)+```++### 财务报告提取++```python+from hyperextract import Template++ka = Template.create("finance/earnings_summary", language="en")-# 访问结果-print(result.entities)-print(result.relations)+report = ka.parse(earnings_text)+print(report.data.revenue)+print(report.data.eps)```-## 核心 API+### 构建知识库-| API | 用途 |-|-----|------|-| `Template.create()` | 创建模板实例 |-| `ka.parse()` | 解析文档 |-| `ka.feed()` | 增量补充 |-| `ka.search()` | 查询知识库 |+```python+from hyperextract import Template++ka = Template.create("general/knowledge_graph", language="en")++# 初始提取+kb = ka.parse(doc1_text)++# 添加更多文档+kb.feed_text(doc2_text)+kb.feed_text(doc3_text)++# 保存供以后使用+kb.dump("./my_kb/")+```++---## 配置-[→ Python SDK 配置指南](./config.md)+### 环境变量-## 详细指南+```python+import os++os.environ["OPENAI_API_KEY"] = "your-api-key"+os.environ["OPENAI_BASE_URL"] = "https://api.openai.com/v1"+```++### 使用 .env 文件++```python+from dotenv import load_dotenv+load_dotenv() # 从 .env 文件加载+```++### 自定义客户端++```python+from langchain_openai import ChatOpenAI, OpenAIEmbeddings+from hyperextract import Template++llm = ChatOpenAI(model="gpt-4o")+embedder = OpenAIEmbeddings(model="text-embedding-3-large")++ka = Template.create(+ "general/biography_graph",+ language="en",+ llm_client=llm,+ embedder=embedder+)+```++---++## 类型提示++Hyper-Extract 完全类型化,支持 IDE 自动完成:++```python+from hyperextract import Template, AutoGraph++ka: AutoGraph = Template.create("general/knowledge_graph", "en")+result = ka.parse(text)++# IDE 自动完成有效+entities = result.data.entities+relations = result.data.relations+```-[→ Python API 详解](./python-api.md)+----## 使用示例+## 下一步-- [金融领域模板](../templates/finance.md)-- [法律领域模板](../templates/legal.md)-- [医学领域模板](../templates/medicine.md)+- 遵循[快速入门](quickstart.md)+- 学习[核心概念](core-concepts.md)+- 浏览[指南](guides/using-templates.md)+- 阅读 [API 参考](api-reference/template.md)diff --git a/docs/zh/python/python-api.md b/docs/zh/python/python-api.mddeleted file mode 100644index a020491..0000000--- a/docs/zh/python/python-api.md+++ /dev/null@@ -1,280 +0,0 @@-# Python API--## 安装--```bash-pip install hyper-extract-```--## 配置--使用 dotenv 设置 API key:--```python-from dotenv import load_dotenv-load_dotenv()-```--创建 `.env` 文件:-```bash-OPENAI_API_KEY=your-api-key-```--## 快速开始--```python-from hyperextract import Template--ka = Template.create("general/biography_graph", "zh")-result = ka.parse(document_text)-```-------## AutoTypes--AutoTypes 定义了知识提取的**输出数据结构**。共有 8 种类型:--### AutoModel - 结构化数据--使用 AutoModel 提取结构化摘要:--```python-from pydantic import BaseModel, Field-from hyperextract import AutoModel-from langchain_openai import ChatOpenAI--class BiographySummary(BaseModel):- title: str = Field(description="传记标题")- subject: str = Field(description="主题名称")- summary: str = Field(description="简要摘要")--llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)--model = AutoModel(- data_schema=BiographySummary,- llm_client=llm,-)-model.feed_text(text)-print(model.data.title)-```--### AutoList - 列表数据--提取列表项:--```python-from hyperextract.types import AutoList--llm = ChatOpenAI(model="gpt-4o-mini")-embedder = OpenAIEmbeddings(model="text-embedding-3-small")--lst = AutoList[str](- item_schema=str,- llm_client=llm,- embedder=embedder,-)-lst.feed_text(text)-print(lst.items) # 提取的列表项-```--### AutoSet - 去重集合--提取唯一项(去重):--```python-from hyperextract.types import AutoSet--s = AutoSet[str](- item_schema=str,- llm_client=llm,- embedder=embedder,-)-s.feed_text(text)-print(s.items) # 去重后的列表-```--### AutoGraph - 知识图谱--提取实体和关系:--```python-from pydantic import BaseModel, Field-from hyperextract.types import AutoGraph--class Entity(BaseModel):- name: str = Field(description="实体名称")- type: str = Field(description="实体类型")--class Relation(BaseModel):- source: str = Field(description="源实体")- target: str = Field(description="目标实体")- type: str = Field(description="关系类型")--graph = AutoGraph[Entity, Relation](- node_schema=Entity,- edge_schema=Relation,- node_key_extractor=lambda x: x.name,- edge_key_extractor=lambda x: f"{x.source}-{x.type}-{x.target}",- llm_client=llm,- embedder=embedder,-)-graph.feed_text(text)-print(f"节点: {len(graph.nodes)}, 边: {len(graph.edges)}")-```--### AutoHypergraph - 超图--用于多元关系(n元关系):--```python-from hyperextract.types import AutoHypergraph-# 参考 examples/zh/autotypes/hypergraph_demo.py-```--### AutoTemporalGraph - 时序图--用于时间相关的关嗧:--```python-from hyperextract.types import AutoTemporalGraph-# 参考 examples/zh/autotypes/temporal_graph_demo.py-```--### AutoSpatialGraph - 空间图--用于位置相关的关嗧:--```python-from hyperextract.types import AutoSpatialGraph-# 参考 examples/zh/autotypes/spatial_graph_demo.py-```--### AutoSpatioTemporalGraph - 时空图--用于时间和位置同时相关的关嗧:--```python-from hyperextract.types import AutoSpatioTemporalGraph-# 参考 examples/zh/autotypes/spatio_temporal_graph_demo.py-```-------## Methods vs Templates--Hyper-Extract 提供了两种提取知识的方式:--### Methods(底层方法)--Methods 提供对提取过程的**完全控制**。在以下情况下使用:-- 需要自定义提取逻辑-- 需要特定的算法调优-- 高级用例--```python-from hyperextract.methods.typical import KG_Gen--ka = KG_Gen(llm_client=llm, embedder=embedder)-ka.feed_text(text)-result = ka.chat("主要成就有哪些?")-```--可用的 Methods:-- **典型方法**:KG_Gen、iText2KG、iText2KG*-- **RAG增强方法**:GraphRAG、LightRAG、HyperRAG、HypergraphRAG、CogRAG--### Templates(高层配置)--Templates 是常见用例的**预配置 schema**。在以下情况下使用:-- 快速得到结果-- 领域特定的提取-- 零代码设置--```python-from hyperextract import Template--ka = Template.create("finance/earnings_summary", "zh")-result = ka.parse(text)-```--### 何时使用哪个?--| 场景 | 推荐 |-|------|------|-| 快速原型 | Templates |-| 领域特定提取 | Templates |-| 自定义提取逻辑 | Methods |-| 实时交互 | Methods |-| 批量处理 | 两者皆可 |-------## 高级用法--### 批量处理--```python-from hyperextract import Template--ka = Template.create("general/biography_graph", "zh")-documents = ["doc1.txt", "doc2.txt", "doc3.txt"]--results = []-for doc in documents:- with open(doc, encoding="utf-8") as f:- result = ka.parse(f.read())- results.append(result)-```--### 增量补充(feed)--```python-ka = Template.create("general/biography_graph", "zh")-result = ka.parse(text1)--# 添加更多知识-ka.feed(result, text2)-```--### 知识问答(chat)--```python-ka.build_index()-answer = ka.chat("有哪些关键事件?")-print(answer.content)-```--### 错误处理--```python-from hyperextract import Template--try:- ka = Template.create("template_name", "zh")- result = ka.parse(text)-except Exception as e:- print(f"错误: {e}")- # 处理错误-```--### 结果导出--```python-# 导出为 JSON-json_output = result.to_json()--# 导出为 Dict-dict_output = result.to_dict()--# 导出为三元组-triples = result.to_triples()-```-------## 下一步--- [预设模板](../templates/index.md)-- [领域模板](../templates/index.md)-- [探索示例](https://github.com/yifanfeng97/hyper-extract/tree/main/examples/zh)diff --git a/docs/zh/python/quickstart.md b/docs/zh/python/quickstart.mdnew file mode 100644index 0000000..b9df283--- /dev/null+++ b/docs/zh/python/quickstart.md@@ -0,0 +1,173 @@+# Python 快速入门++在 5 分钟内开始使用 Hyper-Extract Python SDK。++---++## 前置要求++- Python 3.11++- OpenAI API 密钥++## 安装++```bash+pip install hyper-extract+```++## 基本用法++### 1. 配置 API 密钥++```python+import os+os.environ["OPENAI_API_KEY"] = "your-api-key"+```++或使用 `.env` 文件:++```python+from dotenv import load_dotenv+load_dotenv()+```++### 2. 提取知识++```python+from hyperextract import Template++# 创建模板+ka = Template.create("general/biography_graph", language="en")++# 您的文本+text = """+Marie Curie was a Polish-French physicist and chemist who conducted+pioneering research on radioactivity. She was the first woman to win+a Nobel Prize and the only person to win Nobel Prizes in two different+scientific fields.+"""++# 提取+result = ka.parse(text)++# 访问结果+print(f"实体数: {len(result.data.entities)}")+print(f"关系数: {len(result.data.relations)}")++# 打印第一个实体+if result.data.entities:+ entity = result.data.entities[0]+ print(f"\n第一个: {entity.name} ({entity.type})")+ print(f"描述: {entity.description}")+```++**输出:**+```+Entities: 5+Relations: 4++First: Marie Curie (person)+Description: Polish-French physicist and chemist+```++### 3. 可视化++```python+# 打开交互式可视化+result.show()+```++### 4. 搜索++```python+# 构建搜索索引+result.build_index()++# 搜索+results = result.search("Nobel Prize", top_k=3)+for item in results:+ print(item)+```++### 5. 聊天++```python+# 提问+response = result.chat("Marie Curie 发现了什么?")+print(response.content)+```++### 6. 保存和加载++```python+# 保存到磁盘+result.dump("./curie_kb/")++# 之后加载+new_ka = Template.create("general/biography_graph", language="en")+new_ka.load("./curie_kb/")+```++---++## 完整示例++```python+"""完整示例:提取、探索和保存知识。"""++from dotenv import load_dotenv+load_dotenv()++from hyperextract import Template++def main():+ # 创建模板+ print("Creating template...")+ ka = Template.create("general/biography_graph", language="en")++ # 示例文本+ text = """+ Ada Lovelace was an English mathematician and writer, chiefly known+ for her work on Charles Babbage's early mechanical general-purpose+ computer, the Analytical Engine. She is often regarded as the first+ computer programmer.+ """++ # 提取知识+ print("Extracting knowledge...")+ result = ka.parse(text)++ # 显示结果+ print(f"\nExtraction Results:")+ print(f" Entities: {len(result.data.entities)}")+ print(f" Relations: {len(result.data.relations)}")++ # 列出实体+ print("\nEntities found:")+ for entity in result.data.entities:+ print(f" - {entity.name} ({entity.type})")++ # 构建索引和搜索+ print("\nBuilding search index...")+ result.build_index()++ search_results = result.search("computer programming", top_k=2)+ print(f"\nSearch results: {len(search_results)} items")++ # 保存+ print("\nSaving knowledge base...")+ result.dump("./ada_kb/")++ print("\nDone! Try: he show ./ada_kb/")++if __name__ == "__main__":+ main()+```++---++## 下一步++- 学习[核心概念](core-concepts.md)+- 阅读[使用模板指南](guides/using-templates.md)+- 探索 [API 参考](api-reference/template.md)diff --git a/docs/zh/resources/contributing.md b/docs/zh/resources/contributing.mddeleted file mode 100644index dc58d82..0000000--- a/docs/zh/resources/contributing.md+++ /dev/null@@ -1,115 +0,0 @@-# 贡献--感谢您对 Hyper-Extract 贡献的兴趣!--## 贡献方式--### 报告错误--通过提交 GitHub issue 报告错误,包括:-- 错误的清晰描述-- 重现步骤-- 预期与实际行为-- 环境信息(Python 版本、操作系统等)--### 提出功能建议--我们欢迎功能建议!请:-1. 检查现有 issue 以避免重复-2. 清晰描述用例-3. 解释预期行为--### 贡献代码--#### 开发环境设置--1. Fork 仓库-2. 克隆您的 fork:- ```bash- git clone https://github.com/yifanfeng97/hyper-extract.git- cd hyper-extract- ```--3. 创建虚拟环境:- ```bash- python -m venv env- source env/bin/activate # Windows: env\Scripts\activate- ```--4. 安装开发依赖:- ```bash- pip install -e ".[dev]"- ```--5. 安装预提交钩子:- ```bash- pre-commit install- ```--#### 开发工作流程--1. 创建功能分支:- ```bash- git checkout -b feature/your-feature-name- ```--2. 进行更改-3. 运行测试:- ```bash- pytest tests/- ```--4. 运行代码检查:- ```bash- ruff check .- ```--5. 提交您的更改:- ```bash- git commit -m "Add your feature"- ```--6. 推送并创建 Pull Request--#### 代码风格--- 遵循 PEP 8-- 使用类型提示-- 编写文档字符串-- 为新功能添加测试--### 贡献文档--文档改进始终受欢迎:-- 修复拼写错误或不清楚的解释-- 添加示例-- 改进翻译-- 更新过时内容--### 翻译贡献--我们欢迎翻译贡献:-1. 检查现有翻译-2. 通过 PR 提交翻译--## Pull Request 指南--1. 引用相关 issue-2. 为新功能包含测试-3. 根据需要更新文档-4. 遵循代码风格指南-5. 确保所有测试通过--## 开发资源--- [GitHub 仓库](https://github.com/yifanfeng97/hyper-extract)-- [Issue 追踪器](https://github.com/yifanfeng97/hyper-extract/issues)-- [讨论区](https://github.com/yifanfeng97/hyper-extract/discussions)--## 行为准则--请在所有互动中保持尊重和建设性。我们遵循[贡献者公约](https://www.contributor-covenant.org/)。--## 许可证--通过贡献,您同意您的贡献将在 Apache 2.0 许可证下获得许可。diff --git a/docs/zh/resources/faq.md b/docs/zh/resources/faq.mddeleted file mode 100644index 5bec915..0000000--- a/docs/zh/resources/faq.md+++ /dev/null@@ -1,106 +0,0 @@-# 常见问题--关于 Hyper-Extract 的常见问题。--## 概述--### 什么是 Hyper-Extract?--Hyper-Extract 是一个智能的、基于 LLM 的知识提取框架,可将非结构化文档转换为结构化知识表示。--### Hyper-Extract 可以处理哪些类型的文档?--Hyper-Extract 可以处理各种文档类型,包括:-- 纯文本文件-- Markdown 文件-- HTML 文档-- Word 文档--### 支持哪些语言?--Hyper-Extract 支持多种语言,包括英语、中文、日语、韩语等。--## 安装--### 如何安装 Hyper-Extract?--```bash-pip install hyper-extract-```--### 系统要求是什么?--- Python 3.9 或更高版本-- OpenAI API 密钥(或兼容的 LLM API)--## 使用--### 如何从文档中提取知识?--使用 CLI:-```bash-he parse document.txt -o output/-```--使用 Python API:-```python-from hyperextract import Template-ka = Template.create("general/biography")-result = ka.parse(document_text)-```--### 如何选择正确的模板?--根据以下条件选择模板:-1. 文档类型(金融、法律、医学等)-2. 提取目标(实体、关系、列表等)-3. 所需输出格式(图、列表、模型等)--### 如何创建自定义模板?--请参阅[领域模板](../templates/index.md)了解如何创建自定义模板。--## 故障排除--### API 密钥问题--**错误:"未找到 API 密钥"**--解决方案:使用以下方式设置 API 密钥:-```bash-he config init -k <your-api-key>-```--### 提取质量问题--**提取质量低**--改进提示:-1. 使用领域特定模板-2. 提供清晰的提取说明-3. 使用更长的上下文窗口-4. 调整温度设置--### 性能问题--**提取速度慢**--优化提示:-1. 使用 `light_rag` 方法加快提取速度-2. 批量处理多个文档-3. 使用适当的批量大小--## 贡献--### 如何贡献?--请参阅[贡献指南](contributing.md)了解如何贡献。--### 如何报告错误?--请在我们的 GitHub 仓库上提交问题,包括:-- 错误的详细描述-- 重现步骤-- 预期与实际行为-- 环境信息diff --git a/docs/zh/resources/index.md b/docs/zh/resources/index.mddeleted file mode 100644index b09e622..0000000--- a/docs/zh/resources/index.md+++ /dev/null@@ -1,19 +0,0 @@-# 资源--本节提供常见问题解答、故障排除和贡献指南。--## 本节内容--- [常见问题](faq.md) - 常见问题及解答-- [故障排除](troubleshooting.md) - 问题和解决方案-- [贡献指南](contributing.md) - 如何贡献--## 需要帮助?--- 查看 [常见问题](faq.md)-- 阅读 [故障排除](troubleshooting.md)-- 访问 [GitHub Issues](https://github.com/yifanfeng97/hyper-extract/issues)--## 贡献--我们欢迎各种贡献!请参阅[贡献指南](contributing.md)了解如何参与。diff --git a/docs/zh/resources/troubleshooting.md b/docs/zh/resources/troubleshooting.mddeleted file mode 100644index f46551a..0000000--- a/docs/zh/resources/troubleshooting.md+++ /dev/null@@ -1,149 +0,0 @@-# 故障排除--Hyper-Extract 的常见问题和解决方案。--## 安装问题--### 包安装失败--**问题**: `pip install hyper-extract` 失败--**解决方案**:-1. 升级 pip:- ```bash- pip install --upgrade pip- ```--2. 使用虚拟环境:- ```bash- python -m venv env- source env/bin/activate # Windows: env\Scripts\activate- pip install hyper-extract- ```--### 依赖冲突--**问题**: 与其他包的依赖冲突--**解决方案**: 创建干净的虚拟环境:-```bash-python -m venv hyper-extract-env-source hyper-extract-env/bin/activate-pip install hyper-extract-```--## API 问题--### API 密钥无效--**问题**: "API 密钥无效" 错误--**解决方案**:-1. 验证您的 API 密钥是否正确-2. 检查密钥是否已过期-3. 确保有足够的额度/配额--### 速率限制--**问题**: "超出速率限制" 错误--**解决方案**:-1. 在请求之间添加延迟-2. 使用批量处理-3. 在配置中减少 `max_workers`--### 超时错误--**问题**: 请求超时--**解决方案**:-1. 在配置中增加超时时间:- ```yaml- extraction:- timeout: 120- ```-2. 使用更小的文档块-3. 检查网络连接--## 提取问题--### 空结果--**问题**: 提取返回无结果--**解决方案**:-1. 验证文档包含可提取的内容-2. 检查模板是否匹配文档类型-3. 查看提取日志中的错误--### 提取不正确--**问题**: 提取的数据不正确或不完整--**解决方案**:-1. 使用更具体的模板-2. 在模板描述中添加提取提示-3. 尝试不同的提取方法-4. 调整置信度阈值--### 内存问题--**问题**: 内存不足错误--**解决方案**:-1. 以更小的批次处理文档-2. 减少批量大小-3. 定期清除缓存--## 性能问题--### 提取缓慢--**问题**: 提取非常慢--**解决方案**:-1. 使用更快的提取方法:- ```python- result = atom.extract(text) # 最快- ```--2. 启用并行处理:- ```python- results = ka.batch_parse(docs, parallel=True)- ```--3. 使用更轻量的模型加快处理速度--## CLI 问题--### 命令未找到--**问题**: 找不到 `he` 命令--**解决方案**:-1. 重新安装包:- ```bash- pip uninstall hyper-extract- pip install hyper-extract- ```--2. 直接使用 Python 模块:- ```bash- python -m hyperextract parse document.txt- ```--### 未找到配置--**问题**: CLI 找不到配置--**解决方案**: 初始化配置:-```bash-he config init -k <your-api-key>-```--## 仍然有问题?--如果您仍然遇到问题:-1. 查看[常见问题](faq.md)-2. 搜索现有的 [GitHub Issues](https://github.com/yifanfeng97/hyper-extract/issues)-3. 创建包含详细信息的新问题diff --git a/docs/zh/templates/finance.md b/docs/zh/templates/finance.mddeleted file mode 100644index 39cebd8..0000000--- a/docs/zh/templates/finance.md+++ /dev/null@@ -1,230 +0,0 @@-# 金融模板--Hyper-Extract 提供专门的金融文档提取模板。--## 可用模板--### 财报摘要--从财报中提取关键财务指标:--```yaml-language: zh--name: 财报摘要-type: model-tags: [finance]--description: '从财报中提取关键财务指标。'--output:- fields:- - name: company_name- type: str- description: '公司名称'- - name: revenue- type: str- description: '收入金额'- - name: quarter- type: str- description: '季度'--guideline:- target: '提取财报关键信息。'- rules:- - '提取公司名称和财务数据'- - '使用原文中的数字'--display:- label: '{company_name}'-```--### 风险因素--从金融文档中提取风险因素:--```yaml-language: zh--name: 风险因素列表-type: list-tags: [finance]--description: '提取风险因素列表。'--output:- fields:- - name: category- type: str- description: '风险类别'- - name: description- type: str- description: '风险描述'--guideline:- target: '提取风险因素。'- rules:- - '提取所有风险因素'- - '每个风险因素包含类别和描述'--display:- label: '{category}'-```--### 所有权图--提取所有权关系:--```yaml-language: zh--name: 所有权图谱-type: graph-tags: [finance]--description: '提取所有权关系。'--output:- entities:- fields:- - name: name- type: str- description: '实体名称'- - name: type- type: str- description: '实体类型(公司/个人/机构)'- relations:- fields:- - name: source- type: str- description: '所有者'- - name: target- type: str- description: '被拥有的实体'- - name: type- type: str- description: '关系类型'--guideline:- target: '提取所有权关系。'- rules_for_entities:- - '提取公司和个人实体'- - '保持命名一致'- rules_for_relations:- - '仅在文本明确表达时创建关系'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name} ({type})'- relation_label: '{type}'-```--### 事件时序图--提取财务事件及其时间关系:--```yaml-language: zh--name: 事件时序图-type: temporal_graph-tags: [finance]--description: '提取财务事件及其时间关系。'--output:- entities:- fields:- - name: name- type: str- description: '事件名称'- - name: type- type: str- description: '事件类型'- relations:- fields:- - name: source- type: str- description: '源事件'- - name: target- type: str- description: '目标事件'- - name: type- type: str- description: '关系类型'- - name: time- type: str- description: '时间'- required: false--guideline:- target: '提取财务事件和时间关系。'- rules_for_entities:- - '提取关键财务事件'- rules_for_relations:- - '提取事件之间的时间关系'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}|{time}'- relation_members:- source: source- target: target- time_field: time--display:- entity_label: '{name} ({type})'- relation_label: '{type}@{time}'-```--## 使用示例--### CLI--```bash-# 提取财报摘要-he parse earnings_report.pdf -t finance/earnings_summary -o output/--# 提取风险因素-he parse 10k_filing.txt -t finance/risk_factor_set -o output/--# 提取所有权关系-he parse ownership.txt -t finance/ownership_graph -o output/-```--### Python API--```python-from hyperextract import Template--# 加载金融模板-ka = Template.create("finance/earnings_summary", "zh")--# 从文档提取-result = ka.parse(earnings_text)--# 访问结果-print(result.fields)-```--## 支持的文档类型--- 年度报告-- 季度财报-- 10-K/10-Q 文件-- 股权研究报告-- 财经新闻-- IPO 招股说明书-- 供应链分析--## 下一步--- [法律模板](legal.md)-- [医学模板](medicine.md)-diff --git a/docs/zh/templates/index.md b/docs/zh/templates/index.mddeleted file mode 100644index 041c58e..0000000--- a/docs/zh/templates/index.md+++ /dev/null@@ -1,67 +0,0 @@-# Templates--Hyper-Extract 为各行业和用例提供专用模板。--## 可用领域--- [金融](domain/finance.md) - 金融文档提取-- [法律](domain/legal.md) - 法律文档提取-- [医学](domain/medicine.md) - 医疗文档提取-- [中医](domain/tcm.md) - 中医模板--## 快速开始--### 使用领域模板--```bash-# CLI-he parse document.txt -t finance/earnings_summary--# Python API-ka = Template.create("finance/earnings_summary")-result = ka.parse(document_text)-```--## 模板类别--### 金融模板--从各种文档类型中提取财务信息:--- 财报-- 财经新闻-- IPO 招股书-- 券商研报-- 供应链分析--### 法律模板--提取法律信息:--- 判决书-- 合同-- 合规文件-- 法律专著--### 医学模板--提取医学信息:--- 临床指南-- 出院小结-- 药品说明书-- 病理报告-- 医学教材--### 中医模板--中医专用模板:--- 中药属性-- 方剂组成-- 经络图谱-- 证候推理--## 快速链接--- [Concepts](../concepts/index.md) - 了解 AutoTypes 和 Methodsdiff --git a/docs/zh/templates/legal.md b/docs/zh/templates/legal.mddeleted file mode 100644index e5228d7..0000000--- a/docs/zh/templates/legal.md+++ /dev/null@@ -1,230 +0,0 @@-# 法律模板--Hyper-Extract 提供专门的法律文档提取模板。--## 可用模板--### 案件事实--从判决书中提取事实:--```yaml-language: zh--name: 案件事实时序图-type: temporal_graph-tags: [legal]--description: '从判决书中提取案件事实。'--output:- entities:- fields:- - name: name- type: str- description: '实体名称'- - name: type- type: str- description: '实体类型(当事人/事件/事实)'- relations:- fields:- - name: source- type: str- description: '源实体'- - name: target- type: str- description: '目标实体'- - name: type- type: str- description: '关系类型'- - name: time- type: str- description: '时间'- required: false--guideline:- target: '提取案件事实。'- rules_for_entities:- - '提取当事人和事件'- rules_for_relations:- - '提取事实关系'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}|{time}'- relation_members:- source: source- target: target- time_field: time--display:- entity_label: '{name} ({type})'- relation_label: '{type}@{time}'-```--### 合同条款--提取合同义务:--```yaml-language: zh--name: 合同条款列表-type: list-tags: [legal]--description: '提取合同条款。'--output:- fields:- - name: clause_type- type: str- description: '条款类型'- - name: parties- type: str- description: '涉及方'- - name: obligations- type: str- description: '义务内容'--guideline:- target: '提取合同条款。'- rules:- - '提取所有合同条款'- - '每个条款包含类型、涉及方和义务'--display:- label: '{clause_type}'-```--### 合规要求--提取合规要求:--```yaml-language: zh--name: 合规要求列表-type: list-tags: [legal]--description: '提取合规要求。'--output:- fields:- - name: regulation- type: str- description: '法规名称'- - name: requirement- type: str- description: '合规要求'- - name: deadline- type: str- description: '截止日期'- required: false--guideline:- target: '提取合规要求。'- rules:- - '提取所有合规要求'--display:- label: '{regulation}'-```--### 案件引用--提取案件引用关系:--```yaml-language: zh--name: 案件引用图谱-type: graph-tags: [legal]--description: '提取案件引用关系。'--output:- entities:- fields:- - name: name- type: str- description: '案件名称'- - name: type- type: str- description: '案件类型'- relations:- fields:- - name: source- type: str- description: '引用案件'- - name: target- type: str- description: '被引用案件'- - name: type- type: str- description: '引用类型'--guideline:- target: '提取案件引用关系。'- rules_for_entities:- - '提取案件名称'- rules_for_relations:- - '提取案件之间的引用关系'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'-```--## 使用示例--### CLI--```bash-# 提取案件事实-he parse judgment.txt -t legal/case_fact_timeline -o output/--# 提取合同条款-he parse contract.pdf -t legal/contract_obligation -o output/--# 提取合规要求-he parse compliance.txt -t legal/compliance_list -o output/-```--### Python API--```python-from hyperextract import Template--# 加载法律模板-ka = Template.create("legal/case_fact_timeline", "zh")--# 从文档提取-result = ka.parse(judgment_text)--# 访问结果-print(result.entities)-```--## 支持的文档类型--- 法院判决-- 法律合同-- 合规文件-- 法律论文-- 监管文件--## 下一步--- [金融模板](finance.md)-- [医学模板](medicine.md)-diff --git a/docs/zh/templates/medicine.md b/docs/zh/templates/medicine.mddeleted file mode 100644index 269aa05..0000000--- a/docs/zh/templates/medicine.md+++ /dev/null@@ -1,211 +0,0 @@-# 医学模板--Hyper-Extract 提供专门的医疗文档提取模板。--## 可用模板--### 药物相互作用--提取药物相互作用信息:--```yaml-language: zh--name: 药物相互作用图谱-type: graph-tags: [medicine]--description: '提取药物相互作用信息。'--output:- entities:- fields:- - name: name- type: str- description: '药物名称'- - name: type- type: str- description: '药物类型'- relations:- fields:- - name: source- type: str- description: '源药物'- - name: target- type: str- description: '目标药物'- - name: type- type: str- description: '相互作用类型'--guideline:- target: '提取药物相互作用。'- rules_for_entities:- - '提取药物名称'- rules_for_relations:- - '提取药物之间的相互作用'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'-```--### 治疗计划--提取治疗计划详情:--```yaml-language: zh--name: 治疗计划时序图-type: temporal_graph-tags: [medicine]--description: '提取治疗计划详情。'--output:- entities:- fields:- - name: name- type: str- description: '实体名称'- - name: type- type: str- description: '类型(治疗/药物/手术)'- relations:- fields:- - name: source- type: str- description: '源实体'- - name: target- type: str- description: '目标实体'- - name: type- type: str- description: '关系类型'- - name: time- type: str- description: '时间'- required: false--guideline:- target: '提取治疗计划。'- rules_for_entities:- - '提取治疗方案和药物'- rules_for_relations:- - '提取治疗步骤'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}|{time}'- relation_members:- source: source- target: target- time_field: time--display:- entity_label: '{name} ({type})'- relation_label: '{type}@{time}'-```--### 解剖图--提取解剖结构关系:--```yaml-language: zh--name: 解剖结构图谱-type: graph-tags: [medicine]--description: '提取解剖结构关系。'--output:- entities:- fields:- - name: name- type: str- description: '解剖结构名称'- - name: type- type: str- description: '结构类型'- relations:- fields:- - name: source- type: str- description: '源结构'- - name: target- type: str- description: '目标结构'- - name: type- type: str- description: '关系类型'--guideline:- target: '提取解剖结构关系。'- rules_for_entities:- - '提取解剖结构'- rules_for_relations:- - '提取结构之间的关系'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'-```--## 使用示例--### CLI--```bash-# 提取药物相互作用-he parse clinical_note.txt -t medicine/drug_interaction -o output/--# 提取治疗计划-he parse discharge_summary.pdf -t medicine/treatment_map -o output/-```--### Python API--```python-from hyperextract import Template--# 加载医学模板-ka = Template.create("medicine/drug_interaction", "zh")--# 从文档提取-result = ka.parse(clinical_text)--# 访问结果-print(result.entities)-```--## 支持的文档类型--- 临床指南-- 出院小结-- 药品说明书-- 病理报告-- 医学教科书-- 临床笔记--## 下一步--- [中医模板](tcm.md)-- [金融模板](finance.md)-diff --git a/docs/zh/templates/tcm.md b/docs/zh/templates/tcm.mddeleted file mode 100644index b959f4f..0000000--- a/docs/zh/templates/tcm.md+++ /dev/null@@ -1,228 +0,0 @@-# 中医模板--Hyper-Extract 提供专门的中医(Traditional Chinese Medicine)文档提取模板。--## 可用模板--### 中药属性--提取中药属性:--```yaml-language: zh--name: 中药属性-type: model-tags: [tcm]--description: '提取中药属性信息。'--output:- fields:- - name: herb_name- type: str- description: '中药名称'- - name: properties- type: str- description: '药性'- - name: channels- type: str- description: '归经'- - name: indications- type: str- description: '功效主治'--guideline:- target: '提取中药属性。'- rules:- - '提取中药名称和属性'- - '保持原文表述'--display:- label: '{herb_name}'-```--### 方剂组成--提取方剂组成:--```yaml-language: zh--name: 方剂组成列表-type: list-tags: [tcm]--description: '提取方剂组成。'--output:- fields:- - name: herb_name- type: str- description: '药材名称'- - name: dosage- type: str- description: '剂量'- - name: role- type: str- description: '角色(君/臣/佐/使)'- required: false--guideline:- target: '提取方剂组成。'- rules:- - '提取方剂中的所有药材'- - '记录剂量和角色'--display:- label: '{herb_name}'-```--### 经络图--提取经络关系:--```yaml-language: zh--name: 经络关系图谱-type: graph-tags: [tcm]--description: '提取经络关系。'--output:- entities:- fields:- - name: name- type: str- description: '实体名称(中药/经络)'- - name: type- type: str- description: '类型'- relations:- fields:- - name: source- type: str- description: '源实体'- - name: target- type: str- description: '目标实体'- - name: type- type: str- description: '关系类型'--guideline:- target: '提取经络关系。'- rules_for_entities:- - '提取中药和经络'- rules_for_relations:- - '提取中药进入经络的关系'--identifiers:- entity_id: name- relation_id: '{source}|{type}|{target}'- relation_members:- source: source- target: target--display:- entity_label: '{name}'- relation_label: '{type}'-```--### 证候推理--提取证候诊断和推理:--```yaml-language: zh--name: 证候推理超图-type: hypergraph-tags: [tcm]--description: '提取证候诊断和推理。'--output:- entities:- fields:- - name: name- type: str- description: '实体名称'- - name: type- type: str- description: '类型(证候/症状/治法)'- relations:- fields:- - name: syndrome- type: str- description: '证候名称'- - name: symptoms- type: list- description: '相关症状'- - name: treatment- type: str- description: '治疗方法'- required: false--guideline:- target: '提取证候诊断和推理。'- rules_for_entities:- - '提取证候和症状'- rules_for_relations:- - '提取证候与症状、治法的关系'--identifiers:- entity_id: name- relation_id: '{syndrome}'- relation_members: symptoms--display:- entity_label: '{name}'- relation_label: '{syndrome}'-```--## 使用示例--### CLI--```bash-# 提取中药属性-he parse herbal_compendium.txt -t tcm/herb_property -o output/--# 提取方剂组成-he parse prescription.txt -t tcm/formula_composition -o output/--# 提取经络关系-he parse meridian.txt -t tcm/meridian_graph -o output/-```--### Python API--```python-from hyperextract import Template--# 加载中医模板-ka = Template.create("tcm/herb_property", "zh")--# 从文档提取-result = ka.parse(herbal_text)--# 访问结果-print(result.fields)-```--## 支持的文档类型--- 本草纲目-- 处方单-- 医案记录-- 经络论述-- 方剂药典--## 下一步--- [医学模板](medicine.md)-- [金融模板](finance.md)-diff --git a/mkdocs.yml b/mkdocs.ymlindex af0ce30..cf7c3f6 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -17,15 +17,38 @@ theme:- navigation.tabs- navigation.tabs.sticky- navigation.indexes+ - navigation.path+ - navigation.top- search.suggest- search.highlight- content.code.copy+ - content.code.annotate+ - content.tabs.linkpalette:primary: purpleaccent: deep purple+ logo: assets/logo/logo-icon-white.svg+ favicon: assets/logo/logo-icon.svgplugins:- search+ - mkdocstrings:+ handlers:+ python:+ paths: [hyperextract]+ options:+ docstring_style: google+ show_source: true+ show_root_heading: true+ show_object_full_path: false+ show_category_heading: true+ show_if_no_docstring: false+ members_order: source+ group_by_category: true+ filters:+ - "!^_" # Hide private methods+ merge_init_into_class: true+ show_signature_annotations: true- i18n:docs_structure: folderfallback_to_default: true@@ -34,30 +57,120 @@ plugins:name: Englishdefault: truebuild: true+ nav:+ - Home: index.md+ - Getting Started:+ - getting-started/index.md+ - getting-started/installation.md+ - getting-started/cli-quickstart.md+ - getting-started/python-quickstart.md+ - CLI Guide:+ - cli/index.md+ - cli/workflow.md+ - Commands:+ - cli/commands/parse.md+ - cli/commands/show.md+ - cli/commands/search.md+ - cli/commands/talk.md+ - cli/commands/feed.md+ - cli/commands/info.md+ - cli/commands/build-index.md+ - cli/commands/list.md+ - cli/commands/config.md+ - cli/configuration.md+ - Python SDK:+ - python/index.md+ - python/quickstart.md+ - python/core-concepts.md+ - Guides:+ - python/guides/using-templates.md+ - python/guides/choosing-methods.md+ - python/guides/working-with-autotypes.md+ - python/guides/search-and-chat.md+ - python/guides/incremental-updates.md+ - python/guides/saving-loading.md+ - API Reference:+ - python/api-reference/template.md+ - python/api-reference/autotypes.md+ - python/api-reference/methods.md+ - Concepts:+ - concepts/index.md+ - concepts/architecture.md+ - concepts/autotypes.md+ - concepts/methods.md+ - concepts/templates-format.md+ - Templates:+ - templates/index.md+ - templates/how-to-choose.md+ - templates/browse.md+ - General: templates/general/index.md+ - Finance: templates/finance.md+ - Legal: templates/legal.md+ - Medical: templates/medicine.md+ - TCM: templates/tcm.md+ - Industry: templates/industry.md+ - Resources:+ - resources/index.md+ - resources/faq.md+ - resources/troubleshooting.md+ - resources/contributing.md- locale: zhname: 中文build: truenav:- 首页: zh/index.md- - CLI 使用指南:+ - 入门指南:+ - zh/getting-started/index.md+ - zh/getting-started/installation.md+ - zh/getting-started/cli-quickstart.md+ - zh/getting-started/python-quickstart.md+ - CLI 指南:- zh/cli/index.md- - zh/cli/cli.md- - zh/cli/config.md- - Python 开发者指南:+ - zh/cli/workflow.md+ - 命令详解:+ - zh/cli/commands/parse.md+ - zh/cli/commands/show.md+ - zh/cli/commands/search.md+ - zh/cli/commands/talk.md+ - zh/cli/commands/feed.md+ - zh/cli/commands/info.md+ - zh/cli/commands/build-index.md+ - zh/cli/commands/list.md+ - zh/cli/commands/config.md+ - zh/cli/configuration.md+ - Python SDK:- zh/python/index.md- - zh/python/python-api.md- - zh/python/config.md+ - zh/python/quickstart.md+ - zh/python/core-concepts.md+ - 使用指南:+ - zh/python/guides/using-templates.md+ - zh/python/guides/choosing-methods.md+ - zh/python/guides/working-with-autotypes.md+ - zh/python/guides/search-and-chat.md+ - zh/python/guides/incremental-updates.md+ - zh/python/guides/saving-loading.md+ - API 参考:+ - zh/python/api-reference/template.md+ - zh/python/api-reference/autotypes.md+ - zh/python/api-reference/methods.md- 核心概念:- zh/concepts/index.md- - zh/concepts/auto-types.md- - 知识摘要模板:+ - zh/concepts/architecture.md+ - zh/concepts/autotypes.md+ - zh/concepts/methods.md+ - zh/concepts/templates-format.md+ - 模板库:- zh/templates/index.md- - zh/templates/domain/index.md- - zh/templates/domain/finance.md- - zh/templates/domain/legal.md- - zh/templates/domain/medicine.md- - zh/templates/domain/tcm.md- - 帮助指南:+ - zh/templates/how-to-choose.md+ - zh/templates/browse.md+ - 通用: zh/templates/general/index.md+ - 金融: zh/templates/finance.md+ - 法律: zh/templates/legal.md+ - 医学: zh/templates/medicine.md+ - 中医: zh/templates/tcm.md+ - 工业: zh/templates/industry.md+ - 资源:+ - zh/resources/index.md- zh/resources/faq.md- zh/resources/troubleshooting.md- zh/resources/contributing.md@@ -66,39 +179,29 @@ extra:social:- icon: fontawesome/brands/githublink: https://github.com/yifanfeng97/hyper-extract--nav:- - Home: en/index.md- - CLI:- - en/cli/index.md- - en/cli/cli.md- - en/cli/config.md- - Python SDK:- - en/python/index.md- - en/python/python-api.md- - en/python/config.md- - Concepts:- - en/concepts/index.md- - en/concepts/auto-types.md- - Templates:- - en/templates/index.md- - en/templates/domain/index.md- - en/templates/domain/finance.md- - en/templates/domain/legal.md- - en/templates/domain/medicine.md- - en/templates/domain/tcm.md- - Help:- - en/resources/faq.md- - en/resources/troubleshooting.md- - en/resources/contributing.md+ version:+ provider: mikemarkdown_extensions:- pymdownx.highlight:use_pygments: true+ anchor_linenums: true- pymdownx.inlinehilite- - pymdownx.superfences+ - pymdownx.snippets+ - pymdownx.superfences:+ custom_fences:+ - name: mermaid+ class: mermaid+ format: !!python/name:pymdownx.superfences.fence_code_format+ - pymdownx.tabbed:+ alternate_style: true- admonition+ - details- tables- toc:permalink: true- attr_list+ - md_in_html+ - pymdownx.emoji:+ emoji_index: !!python/name:material.extensions.emoji.twemoji+ emoji_generator: !!python/name:material.extensions.emoji.to_svg