Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
docs: rewrite and polish documentation site
KaifAhmad1129edaf05bdc
diff --git a/docs/architecture.md b/docs/architecture.mdindex f230af11..8b73ab40 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -1,180 +1,126 @@# Architecture-Semantica's modular, extensible framework for semantic intelligence and knowledge engineering.+Semantica is built around a three-layer, modular architecture designed for independent use of components, clean separation of concerns, and extensibility at each layer.----## Design Principles--- **Modular**: Independent, reusable components-- **Extensible**: Easy to add new functionality-- **Scalable**: Handle large-scale data processing-- **Maintainable**: Clear separation of concerns-------## System Architecture+## System Overview```mermaidgraph TBA[Data Ingestion Layer] --> B[Semantic Processing Layer]B --> C[Application Layer]-- A1[Files • Web • APIs • Streams] --> A- B1[Parse • Normalize • Extract • Build] --> B- C1[GraphRAG • AI Agents • Analytics] --> C++ A1[Files · Web · APIs · Streams] --> A+ B1[Parse · Normalize · Extract · Build] --> B+ C1[GraphRAG · AI Agents · Analytics] --> C```-### Three-Layer Architecture+----**1. Data Ingestion Layer**-- Multiple file formats (PDF, DOCX, JSON, CSV, etc.)-- Web scraping and APIs-- Real-time streams (Kafka, RabbitMQ)-- Database connectors (SQL, NoSQL)+## Three-Layer Architecture-**2. Semantic Processing Layer**-- Document parsing and normalization-- Entity and relationship extraction-- Embedding generation-- Knowledge graph construction-- Quality assurance and deduplication+### 1. Data Ingestion Layer-**3. Application Layer**-- GraphRAG for enhanced retrieval-- AI agent memory and context-- Multi-agent systems-- Analytics and visualization+Responsible for loading data from any source into the pipeline.----+- **File formats** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives+- **Web** — crawl via `WebIngestor` with configurable depth+- **Databases** — SQL, NoSQL, Snowflake via `DBIngestor` / `SnowflakeIngestor`+- **Streams** — Kafka, real-time feeds-## Core Modules+### 2. Semantic Processing Layer-### Orchestration-- **`semantica.core`** - Main framework class and coordination-- **`semantica.pipeline`** - Pipeline management and execution+The core intelligence engine — transforms raw data into structured knowledge.-### Data Processing-- **`semantica.ingest`** - Universal data ingestion-- **`semantica.parse`** - Document parsing-- **`semantica.normalize`** - Data cleaning and normalization+- Document parsing and normalization+- Entity and relationship extraction (NER, LLM-typed, rule-based)+- Embedding generation+- Knowledge graph construction with entity merging+- Deduplication, conflict detection, and validation-### Semantic Intelligence-- **`semantica.semantic_extract`** - Entity and relationship extraction-- **`semantica.embeddings`** - Vector embedding generation-- **`semantica.ontology`** - Ontology generation and management+### 3. Application Layer-### Knowledge Graphs-- **`semantica.kg`** - Knowledge graph construction-- **`semantica.vector_store`** - Vector storage (Weaviate, FAISS)-- **`semantica.triplet_store`** - RDF triplet storage (Jena, Blazegraph)-- **`semantica.graph_store`** - Property graphs (Neo4j, FalkorDB)+Consumes the knowledge graph for downstream use cases.-### Quality Assurance-- **`semantica.deduplication`** - Entity deduplication-- **`semantica.conflicts`** - Conflict detection and resolution+- GraphRAG — graph-grounded retrieval for LLMs+- AI agent context and decision tracking+- Multi-agent pipelines+- Analytics, visualization, and export---## Data Flow```-1. Ingestion → Raw data from sources-2. Parsing → Structured content extraction-3. Normalization → Cleaned data-4. Semantic Extraction → Entities, relationships, events-5. Graph Construction → Entity resolution, conflict resolution-6. Quality Assurance → Deduplication, validation-7. Storage → Vector, triplet, and graph stores-8. Application → GraphRAG, agents, analytics+Ingest → raw data from sources+Parse → structured text extraction+Normalize → canonical forms, date/name standardization+Extract → entities, relationships, events+Build → entity resolution, graph construction+QA → deduplication, conflict resolution, validation+Store → vector store, graph store, triplet store+Deliver → GraphRAG, agents, export, visualization```---+## Module Map++| Layer | Modules |+|-------|---------|+| **Ingestion** | `ingest`, `parse`, `split`, `normalize` |+| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` |+| **Storage** | `embeddings`, `vector_store`, `graph_store`, `triplet_store` |+| **Quality** | `deduplication`, `conflicts` |+| **Context** | `context`, `provenance`, `change_management` |+| **Output** | `export`, `visualization`, `pipeline` |++For full module documentation, see the [Modules Guide](modules.md).++---+## Extension Points-### Custom Ingestors+### Custom Ingestor```pythonfrom semantica.ingest import BaseIngestorclass CustomIngestor(BaseIngestor):def ingest(self, source):- # Custom ingestion logic- pass+ # Return a list of document dicts+ ...```-### Custom Extractors+### Custom Extractor```pythonfrom semantica.semantic_extract import BaseExtractorclass CustomExtractor(BaseExtractor):def extract(self, text):- # Custom extraction logic- pass+ # Return a list of entity dicts+ ...```-### Custom Validators--Validators can be implemented within domain-specific modules (e.g., graph or ontology) as needed.----## Design Decisions-### Modularity-Independent components that can be used standalone or together. Easy to test, maintain, and extend.--### Plugin System-Extensible architecture allowing custom functionality without modifying core code.--### Configuration Management-Centralized configuration with environment variable support for different deployment environments.+**Modularity** — every component can be used standalone. Import only what you need; the framework never forces a full stack.-### Error Handling-Comprehensive error handling with graceful degradation and recovery mechanisms.+**Pluggability** — extend any layer without modifying core code. Custom ingestors, extractors, validators, and exporters all follow the same base class pattern.------## Performance+**Configuration over convention** — centralized config with environment variable overrides for deployment flexibility.-**Scalability**-- Parallel processing support-- Streaming for large datasets-- Efficient memory usage-- Intelligent caching--**Optimization**-- Lazy loading-- Batch processing-- Connection pooling-- Query optimization+**Provenance by default** — lineage tracking is built into graph construction, not bolted on. Every node traces back to a source document.----## Security--**Data Security**-- Secure credential handling-- Input validation and output sanitization-- Audit logging--**Access Control**-- Authentication and authorization-- API key management-- Role-based access control-------## Future Roadmap--- Distributed processing-- Real-time streaming improvements-- Advanced reasoning capabilities-- Multi-modal expansion-- Enhanced visualization-----+## Performance Characteristics-For detailed module documentation, see [Modules Guide](modules.md)+- **Parallel execution** — `PipelineBuilder` supports configurable worker counts per stage+- **Delta processing** — incremental graph updates without full recompute+- **Streaming ingestion** — process large corpora without loading everything into memory+- **Backend flexibility** — swap in-memory NetworkX for Neo4j/FalkorDB at scale with no API changesdiff --git a/docs/concepts.md b/docs/concepts.mdindex b32ec0a3..e40e868c 100644--- a/docs/concepts.md+++ b/docs/concepts.md@@ -1,372 +1,172 @@-# Core Concepts+# Core Concepts-**Learn the fundamental concepts behind Semantica in simple, practical terms.**+The fundamental ideas behind Semantica — explained plainly.-!!! tip "Quick Start"- New to Semantica? Start with [Getting Started](getting-started.md) for hands-on examples.+!!! tip "New here?"+ Start with [Getting Started](getting-started.md) for hands-on examples, then come back to this page for deeper understanding.---## What is Semantica?-Semantica transforms unstructured data (documents, web pages, reports) into **knowledge graphs** - structured databases that AI systems can understand and reason about.+Semantica transforms unstructured data (documents, web pages, reports, databases) into **knowledge graphs** — structured representations that AI systems can query, reason about, and trace back to sources.-**What it does:**-- **Reads** documents, PDFs, web pages, databases-- **Extracts** entities (people, companies, dates) and relationships-- **Builds** connected knowledge graphs-- **Enables** AI to reason with structured knowledge-------## Core Architecture--Semantica uses a **layered architecture** - use only what you need:--<div class="grid cards" markdown>--- **Input Layer**-- ----- Data ingestion and preparation-- **Modules**: Ingest, Parse, Split, Normalize--- **Semantic Layer**-- ----- Intelligence and understanding-- **Modules**: Semantic Extract, Knowledge Graph, Ontology, Reasoning--- **Storage Layer**-- ----- Persistent data storage-- **Modules**: Embeddings, Vector Store, Graph Store--- **Quality Layer**-- ----- Data quality and consistency-- **Modules**: Deduplication, Conflicts--- **Context & Memory**-- ----- Agent memory and foundation data-- **Modules**: Context, Seed, LLM Providers--- **Output & Orchestration**-- ----- Export, visualization, and workflows-- **Modules**: Export, Visualization, Pipeline--</div>+At its core, Semantica adds a **context and intelligence layer** on top of your existing AI stack: it doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable.---## Knowledge Graphs-The foundation of Semantica - turning data into structured knowledge.+The foundation of everything in Semantica.-### What is a Knowledge Graph?+A knowledge graph stores information as:-A knowledge graph represents real-world information as:-- **Nodes** (entities): People, companies, locations, dates-- **Edges** (relationships): works_for, located_in, founded_by-- **Properties**: Name, date, confidence score, source+- **Nodes (entities)** — people, companies, locations, events, concepts+- **Edges (relationships)** — `works_for`, `located_in`, `founded_by`+- **Properties** — name, date, confidence score, source URL-### Why Knowledge Graphs?--- **Searchable**: Find information instantly-- **Connectable**: Discover hidden relationships-- **Queryable**: Ask complex questions-- **Explainable**: Trace answers back to sources+This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it.---## Entity Extraction (NER)-Finding and classifying entities in text.--### What it does:-- Scans text for people, organizations, locations, dates-- Classifies each entity by type-- Assigns confidence scores-- Tracks source provenance+Scanning text to find and classify real-world entities.-### Example Output:```python-# From: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."+# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."{"entities": [- {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98},- {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99},- {"text": "1976", "type": "DATE", "confidence": 0.95},- {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97}+ {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98},+ {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99},+ {"text": "1976", "type": "DATE", "confidence": 0.95},+ {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97}]}```+Each entity gets a type, confidence score, and a link to its source document.+---## Relationship Extraction-Finding connections between entities.+Finding how entities connect to each other.-### What it does:-- Identifies how entities relate to each other-- Extracts relationship types and directions-- Provides context and confidence-- Links to source documents--### Example Output:```python{"relationships": [- {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92},- {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89}+ {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92},+ {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89}]}```------## Embeddings--Turning text into numerical vectors for AI understanding.--### What are embeddings?-- **Numerical representations** of text, entities, and relationships-- **Similarity calculations** - find related concepts-- **AI-powered search** - semantic understanding-- **Clustering and grouping** - discover patterns--### Use Cases:-- **Semantic Search** - find documents by meaning, not keywords-- **Entity Resolution** - match similar entities across sources-- **Recommendations** - suggest related content-- **AI Input** - provide structured context to LLMs+Relationships can be extracted via rule-based methods, ML models, or LLMs (with `"llm_typed"` metadata).----## Temporal Graphs+## Embeddings-Knowledge graphs that understand time.+Embeddings convert text into numerical vectors so that AI systems can measure semantic similarity — finding related concepts even when the exact words differ.-### What they track:-- **When** events happened-- **How** entities changed over time-- **Temporal relationships** - before, after, during-- **Historical context** - point-in-time snapshots+Semantica uses embeddings for:-### Example Uses:-- **Company History** - track mergers, leadership changes-- **Person Careers** - job changes, relocations-- **Policy Evolution** - law changes over time-- **Research Progress** - scientific discoveries timeline+- **Semantic search** — retrieve by meaning, not just keywords+- **Entity resolution** — match the same entity across different sources+- **Precedent search** — find similar past decisions+- **GraphRAG retrieval** — hybrid vector + graph traversal---## GraphRAG-Enhanced AI retrieval using knowledge graphs.+GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone.++How it works:-### How it works:-1. **Query** user question-2. **Retrieve** relevant graph context-3. **Enhance** with relationships and entities-4. **Generate** AI response with sources+1. User submits a query+2. Semantica retrieves relevant graph context (entities, relationships, reasoning paths)+3. The LLM generates a response grounded in that context+4. Every claim in the response links back to a source node in the graph-### Benefits:-- **More accurate** answers-- **Source attribution** - trace answers back-- **Context awareness** - understand relationships-- **Reduced hallucination** - grounded in facts+This eliminates the hallucination and traceability problems of standard RAG.---## Ontology-Defining the structure and rules of your knowledge.+An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply.-### What it provides:-- **Schema definition** - what types exist-- **Relationship rules** - valid connections-- **Property constraints** - required fields-- **Inheritance hierarchies** - parent-child relationships--### Example:```python-# Define ontology structureontology = {"classes": ["Person", "Organization", "Location"],- "properties": ["name", "date", "confidence"],- "relationships": ["works_for", "located_in", "born_in"],+ "relationships": ["works_for", "located_in", "founded_by"],"rules": {- "Person": ["must_have_name", "can_have_birth_date"],+ "Person": ["must_have_name"],"Organization": ["must_have_name", "can_have_founding_date"]}}```+Semantica can auto-generate ontologies from your knowledge graph, or import existing OWL/RDF/Turtle ontologies.+---## Reasoning & Inference-Making logical deductions from your knowledge.--### What it can do:-- **Infer missing facts** - derive new knowledge-- **Detect inconsistencies** - find contradictions-- **Apply rules** - automate decision making-- **Explain reasoning** - show how conclusions were reached+Semantica includes multiple reasoning engines to derive new knowledge from existing facts.-### Example:```-Known: Steve Jobs founded Apple Inc.-Known: Apple Inc. is headquartered in Cupertino-Inferred: Steve Jobs has connection to Cupertino+Known: Steve Jobs founded Apple Inc.+Known: Apple Inc. is headquartered in Cupertino+Inferred: Steve Jobs has a connection to Cupertino```------## Deduplication & Entity Resolution--Finding and merging duplicate entities.--### What it does:-- **Detects duplicates** - same entity, different names-- **Merges information** - combine attributes-- **Resolves conflicts** - handle contradictory data-- **Maintains provenance** - track original sources--### Example:-```python-# These refer to the same entity:-"Apple Inc." → "Apple" → "Apple Computer Inc."-# Merge into single entity with all attributes-```+Supported engines: forward chaining, Rete network, deductive, abductive, and SPARQL reasoning — all producing **explainable inference paths**, not black-box conclusions.----## Data Normalization--Cleaning and standardizing your data.+## Temporal Graphs-### What it fixes:-- **Format inconsistencies** - dates, names, numbers-- **Canonical forms** - standard representations-- **Data quality** - remove errors and noise-- **Standardization** - consistent naming conventions+Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis.-### Examples:-- **Dates**: "Jan 1, 2020" → "2020-01-01"-- **Names**: "Dr. Smith PhD" → "John Smith"-- **Companies**: "Apple" → "Apple Inc."-- **Locations**: "NYC" → "New York City"+Common uses: tracking company leadership changes, policy evolution, research timelines, financial instrument histories.----## Conflict Detection--Finding and resolving contradictory information.+## Deduplication & Entity Resolution-### What it identifies:-- **Factual conflicts** - different values for same fact-- **Temporal conflicts** - impossible timelines-- **Logical conflicts** - contradictory relationships-- **Source reliability** - trustworthiness assessment+Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.-### Resolution Strategies:-- **Most recent** - prefer newer information-- **Most reliable** - prefer trusted sources-- **Majority vote** - go with consensus-- **Manual review** - flag for human review+Strategies: Jaro-Winkler similarity (v1), `blocking_v2`, `hybrid_v2`, `semantic_v2` (v2 — up to 7x faster).----## Getting Started+## Provenance & Auditability-Ready to build your first knowledge graph?+Every fact in Semantica links back to:-### Quick Start (5 minutes)-```python-from semantica.semantic_extract import NERExtractor-from semantica.kg import GraphBuilder--# Extract entities-ner = NERExtractor()-entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")--# Build graph-kg = GraphBuilder().build({"entities": entities, "relationships": []})-```--### Learn More-- **Getting Started Guide** - [Getting Started](getting-started.md)-- **Cookbook Examples** - [Cookbook](cookbook.md)-- **Module Documentation** - [Reference](reference/)-- **Community Support** - [Community](community.md)+- The source document it came from+- The extraction method used+- The ontology rules applied+- The reasoning steps that produced any inference-### Common Use Cases-- **Document Analysis** - extract knowledge from reports-- **Research Assistant** - find connections in academic papers-- **Business Intelligence** - analyze company relationships-- **Regulatory Compliance** - track policy changes+This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails.----## Best Practices--### Start Small-- Begin with a single document type-- Focus on specific entity types-- Validate results before scaling--### Configure Properly-- Choose appropriate models for your domain-- Set confidence thresholds-- Define clear ontology rules--### Validate Data-- Check extraction quality-- Review relationship accuracy-- Test with known examples--### Handle Errors-- Implement error handling-- Log processing issues-- Provide feedback mechanisms+## Conflict Detection-### Optimize Performance-- Use appropriate storage backends-- Cache frequently accessed data-- Monitor resource usage+When multiple sources disagree on the same fact, Semantica flags and resolves the conflict rather than silently picking one value.-### Document Workflows-- Record processing steps-- Track data sources-- Maintain change logs+Resolution strategies: prefer most recent, prefer most reliable source, majority vote, or flag for manual review.----## Need Help?+## Next Steps-- **Documentation**: [Getting Started](getting-started.md)-- **Examples**: [Cookbook](cookbook.md)-- **Community**: [Discord](community.md)-- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)-- **Support**: [Contact Us](community.md)+- [Quickstart Tutorial](quickstart.md) — build a full pipeline with code+- [Modules Guide](modules.md) — every module explained+- [Use Cases](use-cases.md) — real-world domain examples+- [API Reference](reference/core.md) — complete technical referencediff --git a/docs/contributing.md b/docs/contributing.mdindex b2441e3b..4b6f0e67 100644--- a/docs/contributing.md+++ b/docs/contributing.md@@ -1,130 +1,99 @@-# Contributing+# Contributing-**Help us build Semantica! Every contribution makes the project better.**+Contributions of all kinds are welcome — code, documentation, tests, and community support.----## Getting Started+## Quick Start-### Quick Start-1. **Fork** the repository-2. **Create** a feature branch-3. **Make** your changes-4. **Test** your changes-5. **Submit** a pull request+```bash+# Fork the repo, then:+git clone https://github.com/your-username/semantica.git+cd semantica+pip install -e ".[dev]"+pytest+```-### First Contribution?-Look for issues labeled [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) for beginner-friendly tasks.+First time? Look for [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) labels for beginner-friendly tasks.---## Ways to Contribute-### Code-- **Fix bugs** - Resolve reported issues-- **Add features** - Implement new functionality-- **Improve performance** - Optimize existing code-- **Refactor** - Clean up code structure+**Code**+- Fix bugs and resolve open issues+- Implement new features or integrations+- Optimize performance or refactor existing code-### Documentation-- **Fix typos** - Correct spelling and grammar-- **Improve guides** - Make documentation clearer-- **Add examples** - Provide practical code examples-- **Update API docs** - Keep reference current+**Documentation**+- Fix typos, improve clarity, add examples+- Write tutorials or domain-specific cookbook notebooks+- Keep API reference up to date-### Testing-- **Write tests** - Add test coverage-- **Fix tests** - Resolve test failures-- **Report issues** - Identify bugs through testing+**Testing**+- Add test coverage for untested modules+- Reproduce and confirm reported bugs+- Improve test reliability-### Community-- **Help others** - Answer questions in issues-- **Share knowledge** - Write tutorials and guides-- **Provide feedback** - Review pull requests+**Community**+- Answer questions in issues and discussions+- Review pull requests+- Share Semantica in your blog posts or talks---## Reporting Issues### Bug Reports-When reporting bugs, include:-- **Description** - What happened-- **Steps to reproduce** - How to trigger the issue-- **Expected behavior** - What should happen-- **Environment** - Your setup details++Include: what happened, steps to reproduce, expected behavior, and your environment (Python version, OS, Semantica version).### Feature Requests-When suggesting features, include:-- **Use case** - Why you need this feature-- **Proposed solution** - How it should work-- **Benefits** - How it helps the community++Include: your use case, what you'd like Semantica to do, and how it benefits others.---## Pull Request Guidelines-### Before Submitting-- **Test** your changes thoroughly-- **Document** new features with examples-- **Update** relevant documentation-- **Follow** the existing code style+Before submitting:-### Pull Request Checklist-- [ ] Code follows project style-- [ ] Tests pass locally-- [ ] Documentation is updated-- [ ] Commit messages are clear-- [ ] No merge conflicts+- [ ] Tests pass locally (`pytest`)+- [ ] New features are documented with examples+- [ ] Code follows project style (Black, isort, flake8)+- [ ] Commit messages are clear and descriptive+- [ ] No unresolved merge conflicts---## Development Setup-### Local Development```bash-# Clone your forkgit clone https://github.com/your-username/semantica.gitcd semantica+pip install -e ".[dev]"+```-# Install in development mode-pip install -e .[dev]+Code style tools used: **Black** (formatting), **isort** (imports), **flake8** (linting).-# Run tests+Run the full test suite:++```bashpytest```-### Code Style-We use standard Python formatting:-- **Black** for code formatting-- **isort** for import sorting-- **flake8** for linting-----## Community Guidelines--### Code of Conduct-Please follow our [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md).--### Communication-- **Be respectful** - Treat everyone with kindness-- **Be helpful** - Assist others when you can-- **Be patient** - Allow time for reviews-- **Be constructive** - Provide helpful feedback-----+## Community-## Recognition+Please follow the [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive.-All contributors are recognized in:-- **GitHub contributors** - Automatic recognition-- **Release notes** - Notable contributions-- **Community highlights** - Outstanding work+All contributors are recognized in release notes and the GitHub contributors list.----## Need Help?+## Help-- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Ask questions-- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Community chat-- **[Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md)** - Community standards+- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)+- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)+- [Discord](https://discord.gg/sV34vps5hH)diff --git a/docs/cookbook.md b/docs/cookbook.mdindex f561aeee..95defb06 100644--- a/docs/cookbook.md+++ b/docs/cookbook.md@@ -1,108 +1,58 @@-# 🍳 Semantica Cookbook+# Semantica Cookbook-Welcome to the **Semantica Cookbook**!+Interactive Jupyter notebooks covering everything from your first knowledge graph to production GraphRAG systems.-This collection of Jupyter notebooks is designed to take you from a beginner to an expert in building semantic AI applications. Whether you're looking for quick recipes or deep-dive tutorials, you'll find it here.--!!! tip "How to use this Cookbook"- - **Beginners**: Start with the [Core Tutorials](#core-tutorials) to learn the basics.- - **Developers**: Check out [Advanced Concepts](#advanced-concepts) for deep dives into specific features.- - **Architects**: Explore [Industry Use Cases](#industry-use-cases) for end-to-end solutions.+!!! tip "Where to start"+ - **New to Semantica** — begin with [Core Tutorials](#core-tutorials)+ - **Building an application** — see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases)+ - **Need installation help** — see the [Installation Guide](installation.md)!!! note "Prerequisites"- Before running these notebooks, ensure you have:- - Python 3.8+ installed- - A basic understanding of Python and Jupyter- - An OpenAI API key (for most examples)--!!! success "Installation"- Install Semantica from PyPI (recommended):-- ```bash- pip install semantica- # Or with all optional dependencies:- pip install semantica[all]- ```-- For more installation options, see the [Installation Guide](installation.md).+ Python 3.8+, Jupyter, and an OpenAI API key (for most examples).----## � Featured Recipes--Hand-picked tutorials to show you the power of Semantica.+## Featured Recipes<div class="grid cards" markdown>-- :material-robot: **GraphRAG Complete**+- :material-graph: **Your First Knowledge Graph**---- Build a production-ready Graph Retrieval Augmented Generation system.-- **Topics**: RAG, LLMs, Vector Search, Graph Traversal-- **Difficulty**: Advanced-- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)+ Go from raw text to a queryable knowledge graph in 20 minutes.-- :material-scale-balance: **RAG vs. GraphRAG Comparison**- ---- Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data.-- **Topics**: RAG, GraphRAG, Benchmarking, Visualization-- **Difficulty**: Intermediate-- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)+ **Topics**: Extraction, Graph Construction, Visualization · **Difficulty**: Beginner++ [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)- :material-robot: **GraphRAG Complete**---- Build a production-ready Graph Retrieval Augmented Generation system.-- **New Features**: Graph Validation, Logical Inference, Hybrid Context.-- **Topics**: RAG, LLMs, Vector Search, Graph Traversal-- **Difficulty**: Advanced-+ Build a production-ready Graph Retrieval Augmented Generation system with hybrid retrieval and logical inference.++ **Topics**: RAG, LLMs, Vector Search, Graph Traversal · **Difficulty**: Advanced+[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)- :material-scale-balance: **RAG vs. GraphRAG Comparison**---- Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data.-- **New Features**: Inference-Enhanced GraphRAG, Reasoning Gap Analysis.-- **Topics**: RAG, GraphRAG, Benchmarking, Visualization-- **Difficulty**: Intermediate-- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)+ Side-by-side benchmark of standard RAG vs. GraphRAG on real-world data.-- :material-graph: **Your First Knowledge Graph**- ---- Go from raw text to a queryable knowledge graph in 20 minutes.-- **Topics**: Extraction, Graph Construction, Visualization-- **Difficulty**: Beginner-- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)+ **Topics**: RAG, GraphRAG, Benchmarking, Reasoning Gap · **Difficulty**: Intermediate++ [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)- :material-shield-alert: **Real-Time Anomaly Detection**---- Detect anomalies in streaming data using dynamic graphs.-- **Topics**: Streaming, Security, Dynamic Graphs-- **Difficulty**: Advanced-+ Detect anomalies in streaming data using dynamic knowledge graphs.++ **Topics**: Streaming, Security, Dynamic Graphs · **Difficulty**: Advanced+[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)</div>----## 🏁 Core Tutorials {#core-tutorials}+## Core Tutorials {#core-tutorials}Essential guides to master the Semantica framework.@@ -210,7 +160,7 @@ Essential guides to master the Semantica framework.----## 🧠 Advanced Concepts+## Advanced ConceptsDeep dive into advanced features, customization, and complex workflows.@@ -329,7 +279,7 @@ Deep dive into advanced features, customization, and complex workflows.----## 🏭 Industry Use Cases {#industry-use-cases}+## Industry Use Cases {#industry-use-cases}Real-world examples and end-to-end applications across various industries.@@ -497,7 +447,7 @@ Real-world examples and end-to-end applications across various industries.----## 🛠️ How to Run+## How to RunTo run these notebooks locally:diff --git a/docs/css/custom.css b/docs/css/custom.cssindex 079c651b..9b8b59ae 100644--- a/docs/css/custom.css+++ b/docs/css/custom.css@@ -216,76 +216,92 @@ html {border-left: 2px solid var(--md-accent-fg-color);}-/*+/*==========================================================================Layout Optimization- ==========================================================================+ ==========================================================================*/-/* Reduce spacing between sidebars and content for all pages */-.md-content__inner {- padding-left: 0.75rem;- padding-right: 0.75rem;- margin-left: 0;-}-.md-content {- margin-left: 0;+/* Widen the overall grid */+.md-grid {+ max-width: 1440px;+ margin-left: auto;+ margin-right: auto;padding-left: 0.5rem;- padding-right: 0.5rem;-}--/* Reduce spacing before left sidebar for all pages */-.md-sidebar {- padding-left: 0.25rem;- margin-left: 0;}+/* Narrow left sidebar to give content more room */.md-sidebar--primary {- padding-right: 0.5rem;+ width: 11rem;+ padding-right: 0.25rem;padding-left: 0.25rem;- margin-left: 0;}-/* Reduce spacing after right sidebar (table of contents) and shift it right slightly */+/* Right TOC sidebar */.md-sidebar--secondary {- padding-left: 1.25rem;+ width: 11rem;+ padding-left: 0.5rem;padding-right: 0;- margin-right: 0;- margin-left: 3.5rem;+ margin-left: 0;}-/* Reduce right edge spacing - similar to left */-.md-container {- padding-right: 0;- margin-right: 0;+.md-sidebar--secondary .md-nav {+ width: 11rem;}-.md-main {- margin-right: 0;- padding-right: 0;+/* Tighten TOC list spacing */+.md-sidebar--secondary .md-nav__list {+ padding-bottom: 1.5rem;+ margin: 0;}-/* Reduce margins of the main container */-.md-main__inner {- margin-left: 0;- margin-right: 0;- padding-right: 0;+.md-sidebar--secondary .md-nav__item {+ padding: 0;+ margin: 0;}-/* Reduce right edge spacing on body/html */-body {- margin-right: 0;- padding-right: 0;+.md-sidebar--secondary .md-nav__link {+ white-space: normal;+ word-break: break-word;+ overflow: visible;+ text-overflow: unset;+ padding-top: 0.15rem;+ padding-bottom: 0.15rem;+ line-height: 1.4;+ font-size: 0.7rem;+ margin: 0;}-html {+/* Nested TOC items (h3, h4) */+.md-sidebar--secondary .md-nav__item .md-nav__item .md-nav__link {+ padding-left: 0.6rem;+ font-size: 0.68rem;+}++/* Remove extra gap between TOC title and first item */+.md-sidebar--secondary .md-nav__title {+ margin-bottom: 0.25rem;+ padding-bottom: 0.25rem;+}++/* Give the main content area maximum available width */+.md-content {+ max-width: none;+ padding-left: 1rem;+ padding-right: 1rem;+}++.md-content__inner {+ max-width: none;+ padding-left: 1rem;+ padding-right: 1rem;+ margin-left: 0;margin-right: 0;- padding-right: 0;}-.md-grid {+.md-main__inner {margin-left: 0;- padding-left: 0.5rem;+ margin-right: 0;}/* Ensure text content is left-aligned by default */diff --git a/docs/deep-dive.md b/docs/deep-dive.mdindex 53ada134..3a6ea484 100644--- a/docs/deep-dive.md+++ b/docs/deep-dive.md@@ -1,10 +1,15 @@# Deep Dive-Advanced topics, architecture, and internals of Semantica.+Internals, advanced concepts, and extension points for contributors and power users.-## Architecture Overview+!!! tip "Just getting started?"+ Read [Architecture](architecture.md) for a higher-level overview first.-Semantica follows a modular, extensible architecture:+---++## Pipeline Internals++The full data flow through a Semantica pipeline:```mermaidgraph TB@@ -16,65 +21,79 @@ graph TBF --> G[Knowledge Graph Builder]G --> H[Embedding Generator]H --> I[Export Layer]-+D --> D1[Entity Extractor]D --> D2[Relationship Extractor]D --> D3[Triplet Extractor]-+G --> G1[Graph Validator]G --> G2[Graph Analyzer]-+H --> H1[Text Embeddings]H --> H2[Graph Embeddings]```+### Sequence Diagram++```mermaid+sequenceDiagram+ participant User+ participant Semantica+ participant Ingestor+ participant Parser+ participant Extractor+ participant Resolver+ participant GraphBuilder+ participant Exporter++ User->>Semantica: build_knowledge_base(sources)+ Semantica->>Ingestor: ingest(sources)+ Ingestor->>Parser: parse(documents)+ Parser->>Extractor: extract(text)+ Extractor->>Resolver: resolve_conflicts(entities)+ Resolver->>GraphBuilder: build_graph(resolved_data)+ GraphBuilder->>Exporter: export(graph)+ Exporter->>User: return result+```++---+## System Components-### 1. Ingestion Layer+### Ingestion Layer-Handles data input from various sources:+Handles input from any source:-- **File Ingestor**: PDF, DOCX, HTML, JSON, CSV-- **Web Ingestor**: URLs, web scraping-- **Database Ingestor**: SQL databases-- **Stream Ingestor**: Real-time data streams+- **FileIngestor** — PDF, DOCX, HTML, JSON, CSV, archives+- **WebIngestor** — URL crawling and scraping+- **DBIngestor** / **SnowflakeIngestor** — SQL databases+- **StreamIngestor** — Kafka and real-time feeds-### 2. Parsing Layer+### Parsing Layer-Converts raw data into structured format:+Converts raw data to structured text:-- Document parsing (PDF, Word, etc.)-- Text extraction-- Metadata extraction-- Format normalization+- Text and metadata extraction from documents+- OCR for scanned content+- Layout analysis (via Docling for tables and columns)-### 3. Extraction Layer+### Extraction Layer-Core semantic extraction:+Core semantic processing pipeline:-```python-# Entity extraction pipeline+```text → Tokenization → NER → Entity Linking → Entity Validation```-**Components:**-- Named Entity Recognition (NER)-- Relationship Extraction-- Triplet Extraction-- Coreference Resolution--### 4. Normalization Layer+Components: Named Entity Recognition, Relationship Extraction, Triplet Extraction, Coreference Resolution.-Standardizes extracted data:+### Normalization Layer-- Entity normalization-- Date/time normalization-- Number normalization-- Text cleaning+Standardizes extracted data: entity names, date formats, numbers, encodings, and language normalization.-### 5. Conflict Resolution+### Conflict Resolution-Handles conflicting information:+Handles contradictory facts from multiple sources:```mermaidgraph LR@@ -88,64 +107,28 @@ graph LRE --> HF --> HG --> H-- style A fill:#ffebee- style H fill:#c8e6c9- style C fill:#fff9c4```-### 6. Knowledge Graph Builder--Constructs the knowledge graph:--- Node creation (entities)-- Edge creation (relationships)-- Property assignment-- Graph validation-- Quality checks+### Knowledge Graph Builder-### 7. Embedding Generator+- Entity resolution across sources+- Edge creation (typed relationships)+- Property assignment with confidence scores+- Graph validation and quality checks-Generates vector representations:+### Embedding Generator-- Text embeddings (sentence transformers)-- Graph embeddings (node2vec, GraphSAGE)-- Multimodal embeddings+- Text embeddings (Sentence-Transformers, FastEmbed, OpenAI, BGE)+- Graph embeddings (Node2Vec, GraphSAGE)-## Data Flow--```mermaid-sequenceDiagram- participant User- participant Semantica- participant Ingestor- participant Parser- participant Extractor- participant Resolver- participant GraphBuilder- participant Exporter-- User->>Semantica: build_knowledge_base(sources)- Semantica->>Ingestor: ingest(sources)- Ingestor->>Parser: parse(documents)- Parser->>Extractor: extract(text)- Extractor->>Resolver: resolve_conflicts(entities)- Resolver->>GraphBuilder: build_graph(resolved_data)- GraphBuilder->>Exporter: export(graph)- Exporter->>User: return result-- Note over User,Exporter: Complete pipeline execution-```+---## Advanced Concepts-### Entity Resolution--Matching entities across sources:+### Entity Resolution Algorithm```python-# Entity resolution algorithm-def resolve_entities(entities):+def resolve_entities(entities, threshold=0.85):clusters = []for entity in entities:matched = False@@ -161,123 +144,92 @@ def resolve_entities(entities):### Relationship Inference-Inferring implicit relationships:--- Transitive relationships-- Temporal relationships-- Causal relationships-- Hierarchical relationships+Semantica's reasoning engines can derive implicit relationships:-### Graph Optimization+- **Transitive** — if A→B and B→C, infer A→C+- **Temporal** — before, after, during from timestamped facts+- **Causal** — IF/THEN rules via `Reasoner`+- **Hierarchical** — subclass/instance inference via `OntologyReasoner`-Optimizing knowledge graph structure:--- Node deduplication-- Edge consolidation-- Path compression-- Index optimization--## Performance Considerations--### Scalability--- **Horizontal Scaling**: Process multiple documents in parallel-- **Vertical Scaling**: Use GPU acceleration-- **Caching**: Cache embeddings and parsed documents-- **Lazy Loading**: Load components on demand--### Memory Management+### Batch Processing for Large Datasets```python-# Process large datasets efficientlydef process_large_dataset(sources, batch_size=100):for i in range(0, len(sources), batch_size):- batch = sources[i:i+batch_size]+ batch = sources[i : i + batch_size]result = semantica.build_knowledge_base(batch)- # Save and clear memorysave_result(result)del resultgc.collect()```-## Extension Points+----### Custom Plugins+## Extension Points-Create custom plugins:+### Custom Plugin```pythonfrom semantica.core import Pluginclass CustomPlugin(Plugin):def process(self, data):- # Your custom processing+ # Your custom processing logicreturn processed_data```-### Custom Extractors--Implement custom extractors:+### Custom Extractor```pythonfrom semantica.semantic_extract import BaseExtractorclass DomainSpecificExtractor(BaseExtractor):- def extract_entities(self, text):- # Domain-specific extraction logic+ def extract(self, text):+ # Domain-specific entity extraction logicreturn entities```-## Internal APIs+### Custom Ingestor-### Core APIs--- `Semantica.build_knowledge_base()` - Main entry point-- `KGBuilder.build()` - Graph construction-- `ConflictResolver.resolve()` - Conflict resolution-- `EmbeddingGenerator.generate()` - Embedding generation--### Extension APIs+```python+from semantica.ingest import BaseIngestor-- Plugin registration-- Custom extractor registration-- Custom exporter registration-- Event hooks+class CustomIngestor(BaseIngestor):+ def ingest(self, source):+ # Load and return document dicts+ return documents+```-## Design Decisions+----### Why Modular Architecture?+## Internal APIs-- **Extensibility**: Easy to add new features-- **Testability**: Components can be tested independently-- **Maintainability**: Clear separation of concerns-- **Flexibility**: Swap implementations easily+| API | Purpose |+|-----|---------|+| `Semantica.build_knowledge_base()` | Main orchestration entry point |+| `GraphBuilder.build()` | Graph construction |+| `ConflictResolver.resolve()` | Conflict resolution |+| `EmbeddingGenerator.generate()` | Embedding generation |-### Why Conflict Resolution?+Extension hooks: plugin registration, custom extractor registration, custom exporter registration, event hooks.-- **Data Quality**: Ensures consistent knowledge-- **Multi-Source**: Handles conflicting information-- **Flexibility**: Multiple resolution strategies-- **Transparency**: Track resolution decisions+----## Future Enhancements+## Design Decisions-Planned improvements:+**Why modular architecture?** Each component is independently testable and swappable. You can use `NERExtractor` alone without pulling in graph storage or pipelines.-- Distributed processing-- Real-time streaming-- Advanced reasoning-- Multi-modal support expansion-- Enhanced visualization+**Why built-in conflict resolution?** Multi-source data always has contradictions. Ignoring them produces garbage graphs. Explicit resolution strategies give you control over data quality.-## Contributing to Core+**Why W3C PROV-O for provenance?** It's an industry standard with tooling support. Using a custom format would make lineage data non-portable.-Interested in contributing to Semantica's core? See our [Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md).+**Why multiple reasoning engines?** Different problems need different reasoning: forward chaining for rule application, SPARQL for graph queries, abductive for hypothesis generation. No single engine fits all cases.----For more information:-- **[API Reference](reference/core.md) - Detailed API documentation-- **[Learning More](learning-more.md)** - Additional resources-- **[GitHub Repository](https://github.com/Hawksight-AI/semantica)** - Source code+## Further Reading+- [Architecture](architecture.md) — high-level three-layer overview+- [Modules](modules.md) — every module with code examples+- [API Reference](reference/core.md) — complete technical reference+- [Contributing](contributing.md) — how to extend the frameworkdiff --git a/docs/examples.md b/docs/examples.mdindex 48757e7d..7f2c6bfa 100644--- a/docs/examples.md+++ b/docs/examples.md@@ -1,306 +1,134 @@# Examples-Real-world examples and use cases for Semantica.--!!! tip "Interactive Learning"- For hands-on interactive tutorials, check out our [Cookbook](cookbook.md) with Jupyter notebooks covering everything from basics to advanced use cases.-------## Example Gallery--<div class="grid cards" markdown>--- :material-school: **Getting Started**- ---- Quick examples to get you up and running in 5 minutes.-- [View Examples](#getting-started-5-min-examples)--- :material-cogs: **Core Workflows**- ---- Common workflows for building production-ready graphs.-- [View Examples](#core-workflows-15-min-examples)--- :material-rocket: **Advanced Patterns**- ---- Complex use cases and production deployments.-- [View Examples](#advanced-patterns-30-min-examples)--- :material-factory: **Production Patterns**- ---- Scalable deployment patterns for enterprise use.-- [View Examples](#production-patterns)--</div>+Code examples organized by complexity. For interactive notebooks, see the [Cookbook](cookbook.md).----## Getting Started (5 min examples)--### Example 1: Basic Knowledge Graph--**Difficulty**: Beginner--Build a knowledge graph from a single document using Semantica's modular approach. This example demonstrates the complete workflow from document ingestion to graph construction.--**What it demonstrates:**-- Document ingestion and parsing-- Entity and relationship extraction-- Knowledge graph construction+## Beginner-**For complete step-by-step examples, see:**-- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete walkthrough- - **Topics**: Ingestion, parsing, extraction, graph building- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Learning the complete workflow+### Basic Knowledge Graph-### Example 2: Entity Extraction+```python+from semantica.ingest import FileIngestor+from semantica.parse import DocumentParser+from semantica.semantic_extract import NERExtractor, RelationExtractor+from semantica.kg import GraphBuilder-**Difficulty**: Beginner+ingestor = FileIngestor()+parser = DocumentParser()+ner = NERExtractor()+rel = RelationExtractor()-Extract entities from text using Named Entity Recognition. This example shows how to identify and classify named entities in text.+sources = ingestor.ingest("data/sample.pdf")+parsed = parser.parse(sources[0])-**What it demonstrates:**-- Named Entity Recognition (NER)-- Entity type classification-- Confidence scoring+entities = ner.extract(parsed)+relationships = rel.extract(parsed, entities=entities)-**For complete examples, see:**-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction- - **Topics**: NER methods, entity types, extraction techniques- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Understanding entity extraction- print(f"{entity.text}: {entity.label}")-```--**Expected Output:**-```-Apple Inc.: ORGANIZATION-Steve Jobs: PERSON+kg = GraphBuilder(merge_entities=True).build(+ entities=entities, relationships=relationships+)+print(f"{len(kg.nodes)} nodes, {len(kg.edges)} edges")```-### Example 3: Multi-Source Integration--**Difficulty**: Beginner--Combine data from multiple sources into a unified knowledge graph. This example demonstrates integrating data from diverse sources.--**What it demonstrates:**-- Multi-source data ingestion-- Entity merging and resolution-- Unified graph construction+### Entity Extraction from Text-**For complete examples, see:**-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns- - **Topics**: Multi-source integration, entity resolution, conflict handling- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Building unified knowledge graphs from diverse sources-------## Core Workflows (15 min examples)--### Example 4: Conflict Resolution--**Difficulty**: Intermediate--Resolve conflicts in data from multiple sources. This example shows how to identify and resolve conflicting information.--**What it demonstrates:**-- Conflict detection-- Conflict resolution strategies-- Data quality assurance+```python+from semantica.semantic_extract import NERExtractor-**For complete examples, see:**-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Conflict resolution patterns- - **Topics**: Conflict detection, resolution strategies, data quality- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Data integration, quality assurance-ingestor = FileIngestor()-parser = DocumentParser()-ner = NERExtractor()--all_entities = []-for source in ["source1.pdf", "source2.pdf"]:- doc = ingestor.ingest_file(source)- parsed = parser.parse_document(source)- text = parsed.get("full_text", "")- entities = ner.extract_entities(text)- all_entities.extend(entities)--# Detect and resolve conflicts-detector = ConflictDetector()-conflicts = detector.detect_conflicts(all_entities)+ner = NERExtractor()+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")-resolver = ConflictResolver(default_strategy="voting")-resolved = resolver.resolve_conflicts(conflicts)--print(f"Detected {len(conflicts)} conflicts")-print(f"Resolved {len(resolved)} conflicts")+for entity in entities:+ print(f"{entity['text']}: {entity['type']}")+# Apple Inc.: ORGANIZATION+# Steve Jobs: PERSON+# 1976: DATE```-### Example 5: Custom Entity Extraction Configuration--**Difficulty**: Intermediate--Use custom configuration for entity extraction with specific models and thresholds.+### Custom NER Configuration```pythonfrom semantica.semantic_extract import NERExtractor-from semantica.kg import GraphBuilder-# Use LLM-based extraction with custom configurationner = NERExtractor(method="llm",provider="openai",model="gpt-4",confidence_threshold=0.8,- temperature=0.0-)--text = "Your document text here..."-entities = ner.extract_entities(text)--# Build graph with custom merge settings-builder = GraphBuilder(- merge_entities=True,- merge_threshold=0.9+ temperature=0.0,)-kg = builder.build_graph(entities=entities, relationships=[])-```--### Example 6: Incremental Graph Building--**Difficulty**: Intermediate--Build knowledge graph incrementally from multiple sources.--```python-from semantica.ingest import FileIngestor-from semantica.parse import DocumentParser-from semantica.semantic_extract import NERExtractor, RelationExtractor-from semantica.kg import GraphBuilder, GraphMerger--def build_kg_from_source(source_path):- """Helper function to build a knowledge graph from a single source."""- ingestor = FileIngestor()- parser = DocumentParser()- ner = NERExtractor()- rel_extractor = RelationExtractor()-- doc = ingestor.ingest_file(source_path)- parsed = parser.parse_document(source_path)- text = parsed.get("full_text", "")-- entities = ner.extract_entities(text)- relationships = rel_extractor.extract_relations(text, entities=entities)-- builder = GraphBuilder()- return builder.build_graph(entities=entities, relationships=relationships)--# Build graphs separately-kg1 = build_kg_from_source("source1.pdf")-kg2 = build_kg_from_source("source2.pdf")--# Merge into unified graph-merger = GraphMerger()-merged_kg = merger.merge([kg1, kg2])--print(f"Merged graph: {len(merged_kg.nodes)} nodes, {len(merged_kg.edges)} edges")+entities = ner.extract("Your document text here...")```----## Advanced Patterns (30+ min examples)--### Example 7: Graph Visualization+## Intermediate-**Difficulty**: Beginner--Visualize your knowledge graph to understand entity relationships.+### Multi-Source Integration```pythonfrom semantica.ingest import FileIngestorfrom semantica.parse import DocumentParserfrom semantica.semantic_extract import NERExtractor, RelationExtractorfrom semantica.kg import GraphBuilder-from semantica.visualization import KGVisualizer-# Build a small graphingestor = FileIngestor()-parser = DocumentParser()-ner = NERExtractor()-rel_extractor = RelationExtractor()--doc = ingestor.ingest_file("semantica_intro.pdf")-parsed = parser.parse_document("semantica_intro.pdf")-text = parsed.get("full_text", "")+parser = DocumentParser()+ner = NERExtractor()+rel = RelationExtractor()+builder = GraphBuilder(merge_entities=True)-entities = ner.extract_entities(text)-relationships = rel_extractor.extract_relations(text, entities=entities)+all_entities, all_rels = [], []-builder = GraphBuilder()-kg = builder.build_graph(entities=entities, relationships=relationships)+for path in ["source1.pdf", "source2.pdf", "source3.pdf"]:+ sources = ingestor.ingest(path)+ parsed = parser.parse(sources[0])+ all_entities.extend(ner.extract(parsed))+ all_rels.extend(rel.extract(parsed, entities=all_entities))-# Visualize-viz = KGVisualizer()-viz.visualize_network(kg, output="html", file_path="semantica_knowledge_map.html")-print("Visualization saved to semantica_knowledge_map.html")+kg = builder.build(entities=all_entities, relationships=all_rels)+print(f"Unified graph: {len(kg.nodes)} nodes, {len(kg.edges)} edges")```----+### Conflict Detection and Resolution-## Advanced Patterns (30+ min examples)+```python+from semantica.conflicts import ConflictDetector, ConflictResolver-### Example 8: Persistent Storage (Neo4j)+detector = ConflictDetector()+conflicts = detector.detect_conflicts(all_entities)-**Difficulty**: Intermediate+resolver = ConflictResolver(default_strategy="voting")+resolved = resolver.resolve_conflicts(conflicts)-Store and query knowledge graphs in a persistent graph database.+print(f"Detected {len(conflicts)} conflicts, resolved {len(resolved)}")+```++### Persistent Storage (Neo4j)```pythonfrom semantica.graph_store import GraphStore-# Initialize with Neo4jstore = GraphStore(backend="neo4j",uri="bolt://localhost:7687",user="neo4j",- password="password"+ password="password",)store.connect()-# Create nodes and relationships-apple = store.create_node(- labels=["Company"],- properties={"name": "Apple Inc."}-)-tim = store.create_node(- labels=["Person"],- properties={"name": "Tim Cook"}-)+apple = store.create_node(labels=["Company"], properties={"name": "Apple Inc."})+tim = store.create_node(labels=["Person"], properties={"name": "Tim Cook"})store.create_relationship(start_node_id=tim["id"],end_node_id=apple["id"],- rel_type="CEO_OF"+ rel_type="CEO_OF",)-store.close()```-### Example 9: FalkorDB for Real-Time Applications--**Difficulty**: Intermediate--Ultra-fast graph queries for LLM applications using FalkorDB.+### FalkorDB (High-Speed Queries)```pythonfrom semantica.graph_store import GraphStore@@ -309,182 +137,108 @@ store = GraphStore(backend="falkordb",host="localhost",port=6379,- graph_name="knowledge_graph"+ graph_name="knowledge_graph",)store.connect()--# Fast queries-results = store.execute_query("MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n")+results = store.execute_query(+ "MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n"+)store.close()```-### Example 10: GraphRAG (Knowledge-Powered Retrieval)+----**Difficulty**: Advanced+## Advanced-Build a production-ready GraphRAG system with logical inference and hybrid retrieval.+### GraphRAG with Reasoning```pythonfrom semantica.context import AgentContextfrom semantica.reasoning import Reasoner-# 1. Initialize context with GraphRAG (Hybrid Retrieval)context = AgentContext(- vector_store=vs,+ vector_store=vs,knowledge_graph=kg,graph_expansion=True,- hybrid_alpha=0.7+ hybrid_alpha=0.7,)-# 2. Enrich Knowledge Graph using Logical Reasoningreasoner = Reasoner()--# Add a rule to categorize technology stack itemsreasoner.add_rule("IF Library(?x) AND Language(?y) THEN TechStackItem(?x)")+inferred = reasoner.infer_facts(kg.get_all_triplets())-# Infer new facts from the existing graph-all_facts = kg.get_all_triplets()-inferred = reasoner.infer_facts(all_facts)+for fact in inferred:+ kg.add_fact_from_string(fact)-# Add inferred knowledge back to the graph-for fact_str in inferred:- kg.add_fact_from_string(fact_str)--# 3. Retrieve context for a query (now with enriched knowledge)results = context.retrieve("What technologies are used in this project?")```-[**View Complete GraphRAG Tutorial**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)--### Example 11: RAG vs. GraphRAG Comparison--**Difficulty**: Intermediate--Benchmark standard Vector RAG against Graph-enhanced retrieval.--[**View RAG vs. GraphRAG Comparison**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)+[Full GraphRAG tutorial](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) · [RAG vs. GraphRAG comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)----## Production Patterns--### Example 12: Streaming Data Processing--**Difficulty**: Advanced+## Production-Process data streams in real-time.+### Batch Processing (Large Datasets)```python-from semantica.ingest import StreamIngestor+from semantica.ingest import FileIngestorfrom semantica.parse import DocumentParserfrom semantica.semantic_extract import NERExtractor, RelationExtractorfrom semantica.kg import GraphBuilder-stream_ingestor = StreamIngestor(stream_uri="kafka://localhost:9092/topic")-parser = DocumentParser()-ner = NERExtractor()-rel_extractor = RelationExtractor()-builder = GraphBuilder()+ingestor = FileIngestor()+parser = DocumentParser()+ner = NERExtractor()+rel = RelationExtractor()+builder = GraphBuilder()-for batch in stream_ingestor.stream(batch_size=100):- all_entities = []- all_relationships = []-- for item in batch:- text = str(item) # Convert stream item to text- entities = ner.extract_entities(text)- relationships = rel_extractor.extract_relations(text, entities=entities)- all_entities.extend(entities)- all_relationships.extend(relationships)-- # Build graph from batch- kg = builder.build_graph(entities=all_entities, relationships=all_relationships)- # Process results- print(f"Processed batch: {len(kg.nodes)} nodes")-```+sources = [f"data/doc_{i}.pdf" for i in range(1000)]+batch_size = 50-### Example 13: Batch Processing Large Datasets+for i in range(0, len(sources), batch_size):+ batch = sources[i : i + batch_size]+ all_entities, all_rels = [], []-**Difficulty**: Intermediate+ for path in batch:+ parsed = parser.parse(ingestor.ingest(path)[0])+ all_entities.extend(ner.extract(parsed))+ all_rels.extend(rel.extract(parsed, entities=all_entities))-Process large datasets efficiently with batching.+ kg = builder.build(entities=all_entities, relationships=all_rels)+ print(f"Batch {i // batch_size + 1}: {len(kg.nodes)} nodes")+```++### Real-Time Streaming```python-from semantica.ingest import FileIngestor-from semantica.parse import DocumentParser+from semantica.ingest import StreamIngestorfrom semantica.semantic_extract import NERExtractor, RelationExtractorfrom semantica.kg import GraphBuilder-ingestor = FileIngestor()-parser = DocumentParser()-ner = NERExtractor()-rel_extractor = RelationExtractor()+stream = StreamIngestor(stream_uri="kafka://localhost:9092/topic")+ner = NERExtractor()+rel = RelationExtractor()builder = GraphBuilder()-sources = [f"data/doc_{i}.pdf" for i in range(1000)]-batch_size = 50--for i in range(0, len(sources), batch_size):- batch = sources[i:i+batch_size]-- all_entities = []- all_relationships = []-- for source in batch:- doc = ingestor.ingest_file(source)- parsed = parser.parse_document(source)- text = parsed.get("full_text", "")-- entities = ner.extract_entities(text)- relationships = rel_extractor.extract_relations(text, entities=entities)-- all_entities.extend(entities)- all_relationships.extend(relationships)-- # Build graph from batch- kg = builder.build_graph(entities=all_entities, relationships=all_relationships)-- # Save intermediate results- print(f"Processed batch {i//batch_size + 1}: {len(kg.nodes)} nodes")+for batch in stream.stream(batch_size=100):+ all_entities, all_rels = [], []+ for item in batch:+ text = str(item)+ all_entities.extend(ner.extract(text))+ all_rels.extend(rel.extract(text, entities=all_entities))+ kg = builder.build(entities=all_entities, relationships=all_rels)+ print(f"Processed batch: {len(kg.nodes)} nodes")```---## More Resources-- **[Quick Start Guide](quickstart.md)** - Step-by-step tutorial-- **[API Reference](reference/core.md)** - Complete API documentation-- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks-- **[Use Cases](use-cases.md)** - Real-world applications--### 🍳 Recommended Cookbook Tutorials--- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules- - **Topics**: Framework overview, all modules, architecture, configuration- - **Difficulty**: Beginner- - **Time**: 30-45 minutes- - **Use Cases**: First-time users, understanding the framework--- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Learning the basics, quick start--- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready GraphRAG system- - **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal, LLM integration- - **Difficulty**: Advanced- - **Time**: 1-2 hours- - **Use Cases**: Production RAG applications--- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Benchmark standard RAG vs GraphRAG- - **Topics**: RAG, GraphRAG, benchmarking, visualization, reasoning gap- - **Difficulty**: Intermediate- - **Time**: 45-60 minutes- - **Use Cases**: Understanding GraphRAG advantages, choosing the right approach-------!!! info "Contribute"- Have an example to share? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)+- [Quickstart Tutorial](quickstart.md) — step-by-step first pipeline+- [Cookbook](cookbook.md) — interactive Jupyter notebooks+- [Use Cases](use-cases.md) — domain-specific examples+- [API Reference](reference/core.md) — complete API documentation+!!! info "Have an example to share?"+ [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)diff --git a/docs/faq.md b/docs/faq.mdindex cf96b220..ff5bfc31 100644--- a/docs/faq.md+++ b/docs/faq.md@@ -1,148 +1,149 @@-# Frequently Asked Questions+# Frequently Asked Questions-**Common questions about Semantica and how to use it.**+Common questions about Semantica. Use Ctrl+F to find what you need.---## General### What is Semantica?-Semantica is an open-source framework for building knowledge graphs from unstructured data. It transforms documents, web pages, and databases into structured, queryable knowledge.-### What can I do with Semantica?-- **Build knowledge graphs** from documents and data-- **Extract entities and relationships** automatically-- **Power AI applications** with structured knowledge-- **Create semantic search** and GraphRAG systems-- **Integrate multiple data sources** into unified graphs+Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data — documents, APIs, databases — into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable.++### What can I build with Semantica?++- Knowledge graphs from documents and multi-source data+- GraphRAG systems with graph-grounded retrieval+- AI agents with structured decision history and semantic memory+- Compliance-ready pipelines with W3C PROV-O lineage++### What makes Semantica different from other frameworks?++Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit why an AI reached a conclusion.### Is Semantica free?-Yes! Semantica is open source under the MIT License.-### What makes Semantica different?-- **Modular architecture** - Use only what you need-- **Production-ready** - Built for scale and reliability-- **Extensible** - Add custom models and components-- **Open source** - Transparent and community-driven+Yes — MIT licensed, no vendor lock-in. Some features require third-party API keys (e.g., OpenAI embeddings), but Semantica itself is free.---## Installation### How do I install Semantica?+```bashpip install semantica```+See [Installation](installation.md) for virtual environment setup, optional extras, and troubleshooting.+### What Python version do I need?-Python 3.8 or higher. Python 3.11+ is recommended.++Python 3.8 or higher. Python 3.11+ is recommended for best performance.### What are the system requirements?+- Python 3.8+-- 4GB+ RAM for basic use-- Optional GPU for embeddings and ML models+- 4 GB RAM minimum; 16 GB+ recommended for larger graphs+- Optional GPU for embedding generation and ML inference---## Getting Started-### How do I start using Semantica?-```python-from semantica.semantic_extract import NERExtractor-from semantica.kg import GraphBuilder+### Where do I start?-# Extract entities-ner = NERExtractor()-entities = ner.extract("Apple Inc. was founded by Steve Jobs.")+1. [Installation](installation.md) — get set up+2. [Getting Started](getting-started.md) — core concepts and first example+3. [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline+4. [Cookbook](cookbook.md) — interactive Jupyter notebooks-# Build knowledge graph-kg = GraphBuilder().build({"entities": entities})-```+### What data sources does Semantica support?-### Where can I find examples?-- **[Getting Started Guide](getting-started.md)** - Quick introduction-- **[Cookbook](cookbook.md)** - Practical examples-- **[GitHub Examples](https://github.com/Hawksight-AI/semantica/tree/main/examples)** - Code samples+- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives+- **Web** — crawl with `WebIngestor`, RSS feeds+- **Databases** — PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor`+- **Streams** — Kafka, real-time ingestion+- **Media** — image OCR, audio/video metadata---## Features-### What data sources does Semantica support?-- **Files**: PDF, DOCX, TXT, JSON, CSV-- **Web**: Websites, RSS feeds, APIs-- **Databases**: PostgreSQL, MySQL, Snowflake, MongoDB-- **Streams**: Kafka, RabbitMQ, real-time data--### Can I use custom models?-Yes! Semantica supports custom:-- **Entity extraction models**-- **Embedding models**-- **Language models**-- **Custom processors**+### Can I use my own models?++Yes. Semantica supports custom entity extraction models, embedding models, LLM providers (via LiteLLM — 100+ models), and custom pipeline processors.### Does Semantica support GPUs?-Yes, Semantica automatically uses GPUs when available for:-- **Embedding generation**-- **ML model inference**-- **Vector operations**++Yes. When available, GPUs are used automatically for embedding generation, ML model inference, and vector operations. Install `semantica[gpu]` for CUDA support.++### How does Semantica handle large datasets?++- **Batching** — process documents in configurable chunks+- **Parallel processing** — `PipelineBuilder` supports configurable worker counts+- **Delta processing** — update graphs incrementally without full recompute+- **Graph backends** — swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE at scale---## Technical-### How does Semantica handle large datasets?-- **Batching** - Process data in chunks-- **Streaming** - Handle real-time data-- **Parallel processing** - Use multiple cores-- **Memory management** - Efficient resource usage--### Can I deploy Semantica in production?-Yes! Semantica is production-ready with:-- **Scalable architecture**-- **Error handling**-- **Monitoring support**-- **Container deployment**--### How do I customize Semantica?-- **Custom processors** - Add new extraction logic-- **Custom models** - Use your own ML models-- **Plugins** - Extend functionality-- **Configuration** - Adjust behavior+### What graph databases are supported?++Neo4j, FalkorDB, Apache AGE (PostgreSQL), Amazon Neptune, and in-memory NetworkX for development.++### What export formats are available?++RDF (Turtle, JSON-LD, N-Triples, XML), Apache Parquet, ArangoDB AQL, CSV, YAML, and OWL ontologies.++### Is Semantica production-ready?++Yes. v0.3.0 ships with 886+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, W3C PROV-O provenance, and change management with checksums. See [What's New](index.md#whats-new-in-v030) for details.---## Troubleshooting-### Installation issues-- **Python version**: Ensure Python 3.8+-- **Dependencies**: Install with `pip install -e .[dev]`-- **Permissions**: Use virtual environments+### Import error: `ModuleNotFoundError: No module named 'semantica'`++Ensure you have the correct Python environment active, then:++```bash+pip list | grep semantica+pip install --upgrade semantica+```-### Performance issues-- **Memory**: Increase available RAM-- **GPU**: Install CUDA for GPU acceleration-- **Batching**: Use smaller chunk sizes+### Installation fails with dependency errors-### Common errors-- **Import errors**: Check installation path-- **Model loading**: Verify model availability-- **Memory errors**: Reduce batch sizes+```bash+pip install --upgrade pip wheel+pip install semantica+```++### Memory errors during processing++Reduce batch sizes, enable streaming ingestion, or switch to a persistent graph backend (Neo4j, FalkorDB).++### Slow embedding or inference++Install GPU support (`pip install semantica[gpu]`) and ensure CUDA is available on your system.---## Support### Where can I get help?-- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report problems-- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Ask questions-- **[Documentation](index.md)** - Browse guides and references--### How do I report bugs?-1. **Search** existing issues first-2. **Create** a new issue with details-3. **Include** reproduction steps-4. **Add** environment information--### Can I contribute?-Yes! See the [Contributing Guide](contributing.md) for details on how to help improve Semantica.++- [Discord](https://discord.gg/sV34vps5hH) — community chat and support+- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — bug reports and feature requests+- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) — questions and ideas++### How do I report a bug?++1. Search [existing issues](https://github.com/Hawksight-AI/semantica/issues) first+2. Open a new issue with: description, reproduction steps, expected vs actual behavior, and your environment (Python version, OS, Semantica version)++### How do I contribute?++See the [Contributing Guide](contributing.md).diff --git a/docs/getting-started.md b/docs/getting-started.mdindex fecbc5ff..dcec03a9 100644--- a/docs/getting-started.md+++ b/docs/getting-started.md@@ -1,14 +1,18 @@# Getting Started-## Overview+**Semantica** is the context and intelligence layer for AI — turning raw data into explainable, auditable knowledge graphs for high-stakes domains.-**Semantica** is a semantic intelligence layer that bridges the gap between raw data and trustworthy AI. It transforms unstructured data into explainable, auditable knowledge graphs perfect for high-stakes domains.+!!! tip "Just here for code?"+ Jump straight to the [Quick Start](#quick-start) or explore the [Cookbook](cookbook.md) for interactive notebooks.-### What You Can Build-- **GraphRAG Systems** - Enhanced retrieval with semantic reasoning-- **AI Agents** - Trustworthy agents with explainable memory-- **Knowledge Graphs** - Production-ready semantic databases-- **Compliance-Ready AI** - Auditable systems with full provenance+---++## What You Can Build++- **GraphRAG Systems** — enhanced retrieval with semantic graph reasoning+- **AI Agents** — accountable agents with structured decision history and memory+- **Knowledge Graphs** — production-ready semantic knowledge bases+- **Compliance-Ready AI** — auditable systems with full W3C PROV-O provenance---@@ -18,17 +22,17 @@pip install semantica```-Or with all features:+With all optional dependencies:```bashpip install semantica[all]```-Verify installation:+Verify:```pythonimport semantica-print(f"Semantica {semantica.__version__} installed!")+print(semantica.__version__)```---@@ -36,66 +40,59 @@ print(f"Semantica {semantica.__version__} installed!")## Quick Start```python-from semantica.semantic_extract import NERExtractor-from semantica.kg import GraphBuilder--# Extract entities-ner = NERExtractor(method="ml", model="en_core_web_sm")-entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")--# Build knowledge graph-kg = GraphBuilder().build({"entities": entities, "relationships": []})-print(f"Built KG with {len(kg.get('entities', []))} entities")+from semantica.context import AgentContext, ContextGraph+from semantica.vector_store import VectorStore++context = AgentContext(+ vector_store=VectorStore(backend="faiss", dimension=768),+ knowledge_graph=ContextGraph(advanced_analytics=True),+ decision_tracking=True,+)++# Store a memory+context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")++# Record a decision+decision_id = context.record_decision(+ category="model_selection",+ scenario="Choose LLM for production pipeline",+ reasoning="GPT-4 benchmark advantage justifies cost increase",+ outcome="selected_gpt4",+ confidence=0.91,+)++# Find similar past decisions+precedents = context.find_precedents("model selection", limit=5)```-**What this does:**-- Extracts entities (people, organizations, dates) from text-- Builds a knowledge graph from extracted entities-- Outputs the number of entities found----## Core Architecture-Semantica uses a **modular architecture** - use only what you need:+Semantica uses a modular, layered architecture — import only what you need.-### 1️⃣ Input Layer - Data Ingestion-```python-from semantica.ingest import FileIngestor-documents = FileIngestor().ingest_directory("docs/")-```--### 2️⃣ Semantic Layer - Intelligence Engine-```python-from semantica.semantic_extract import NERExtractor, RelationExtractor-entities = NERExtractor().extract(text)-relationships = RelationExtractor().extract(text, entities)-```--### 3️⃣ Output Layer - Knowledge Assets-```python-from semantica.kg import GraphBuilder-kg = GraphBuilder().build_graph(entities, relationships)-```+| Layer | Modules | Purpose |+|-------|---------|---------|+| **Input** | `ingest`, `parse`, `split`, `normalize` | Load and prepare data |+| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` | Extract meaning |+| **Storage** | `embeddings`, `vector_store`, `graph_store` | Persist knowledge |+| **Quality** | `deduplication`, `conflicts` | Validate and clean |+| **Context** | `context`, `provenance`, `change_management` | Track decisions and lineage |+| **Output** | `export`, `visualization`, `pipeline` | Deliver results |---## Next Steps-### 🍳 Interactive Tutorials-1. **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)** - Complete framework overview-2. **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)** - Hands-on graph building-3. **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)** - Production-ready RAG--### 📚 Learn More-- **[Core Concepts](concepts.md)** - Deep dive into knowledge graphs & ontologies-- **[Cookbook](cookbook.md)** - 14 domain-specific tutorials-- **[API Reference](reference/core.md)** - Complete technical documentation+- [Core Concepts](concepts.md) — knowledge graphs, ontologies, reasoning explained+- [Quickstart Tutorial](quickstart.md) — build a full pipeline step by step+- [Cookbook](cookbook.md) — 14 domain-specific Jupyter notebook tutorials+- [API Reference](reference/core.md) — complete module documentation----## Need Help?+## Help-- **[💬 Discord Community](https://discord.gg/sV34vps5hH)** - Get help from the community-- **[🐛 Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report bugs or request features-- **[📖 Documentation](https://semantica.readthedocs.io/)** - Full documentation site+- [Discord Community](https://discord.gg/sV34vps5hH) — ask questions, share projects+- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — report bugs or request features+- [FAQ](faq.md) — common questions answereddiff --git a/docs/glossary.md b/docs/glossary.mdindex e065699d..4b42ad31 100644--- a/docs/glossary.md+++ b/docs/glossary.md@@ -1,9 +1,9 @@# Glossary-**Comprehensive reference of terms and concepts used in Semantica and semantic intelligence.**+Reference of terms and concepts used throughout Semantica.-!!! tip "Quick Reference"- Looking for a specific term? Use your browser's search function (Ctrl+F) to find terms quickly.+!!! tip "Finding a term"+ Use Ctrl+F to search this page.---@@ -216,17 +216,7 @@ W3C PROV-O compliant tracking of data lineage and source attribution.## See Also-- **[Core Concepts](concepts.md)** - Deep dive into fundamental concepts-- **[Getting Started](getting-started.md)** - Begin your journey with Semantica-- **[Modules Guide](modules.md)** - Complete module overview-- **[API Reference](reference/)** - Technical documentation-------## Need Help?--- **Documentation**: [Getting Started](getting-started.md)-- **Examples**: [Cookbook](cookbook.md)-- **Community**: [Discord](community.md)-- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)-- **Support**: [Contact Us](community.md)+- [Core Concepts](concepts.md) — deeper explanation of key ideas+- [Getting Started](getting-started.md) — first steps+- [Modules Guide](modules.md) — every module explained+- [API Reference](reference/) — technical referencediff --git a/docs/index.md b/docs/index.mdindex b67a3906..63a0ecf7 100644--- a/docs/index.md+++ b/docs/index.md@@ -1,24 +1,23 @@<div align="center">- <img src="assets/img/Semantica Logo.png" alt="Semantica Logo" width="450" height="auto">-+ <img src="assets/img/Semantica Logo.png" alt="Semantica Logo" width="420" height="auto">+<h1>🧠 Semantica</h1>-+<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.8+-blue.svg" alt="Python 3.8+"></a><a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>- <a href="https://badge.fury.io/py/semantica"><img src="https://img.shields.io/badge/pypi-v0.2.3-blue.svg" alt="PyPI version"></a>- <a href="https://pypi.org/project/semantica/"><img src="https://img.shields.io/pypi/dm/semantica" alt="Monthly Downloads"></a>+ <a href="https://pypi.org/project/semantica/"><img src="https://img.shields.io/pypi/v/semantica.svg" alt="PyPI"></a>+ <a href="https://github.com/Hawksight-AI/semantica/releases/tag/v0.3.0"><img src="https://img.shields.io/badge/version-0.3.0-brightgreen.svg" alt="Version"></a><a href="https://pepy.tech/project/semantica"><img src="https://static.pepy.tech/badge/semantica" alt="Total Downloads"></a>- <a href="https://semantica.readthedocs.io/"><img src="https://img.shields.io/badge/docs-latest-brightgreen.svg" alt="Documentation"></a>- <a href="https://discord.gg/sV34vps5hH"><img src="https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white" alt="Discord"></a>-- <p><strong>Open-Source Semantic Layer & Knowledge Engineering Framework</strong></p>-- <p><strong>Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.</strong></p>-- <p><em>The semantic intelligence layer that makes your AI agents auditable, explainable, and trustworthy. Perfect for high-stakes domains where mistakes have real consequences.</em></p>-- <p>🆓 <strong>Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>-+ <a href="https://github.com/Hawksight-AI/semantica/actions"><img src="https://github.com/Hawksight-AI/semantica/workflows/CI/badge.svg" alt="CI"></a>+ <a href="https://discord.gg/sV34vps5hH"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white" alt="Discord"></a>+ <a href="https://x.com/BuildSemantica"><img src="https://img.shields.io/badge/X-Follow-black?logo=x&logoColor=white" alt="X"></a>++ <p><strong>A Framework for Building Context Graphs and Decision Intelligence Layers for AI</strong></p>++ <p>⭐ Give us a Star • 🍴 Fork us • 💬 Join our Discord • 🐦 Follow on X</p>++ <p><em>Transform Chaos into Intelligence. Build AI systems with context graphs, decision tracking, and advanced knowledge engineering that are explainable, traceable, and trustworthy — not black boxes.</em></p>+<p><a href="getting-started/" class="md-button md-button--primary">Get Started</a><a href="https://github.com/Hawksight-AI/semantica" class="md-button">View on GitHub</a>@@ -27,288 +26,266 @@----## 🚀 Why Semantica?+## The Problem++AI agents today are capable but not trustworthy:++- **No memory structure** — agents store embeddings, not meaning. Retrieval is fuzzy; there's no way to ask *why* something was recalled.+- **No decision trail** — agents make decisions continuously but record nothing. When something goes wrong, there's no history to debug or audit.+- **No provenance** — outputs cannot be traced back to source facts. In regulated industries, this is a compliance blocker.+- **No reasoning transparency** — black-box answers with no explanation of how a conclusion was reached.+- **No conflict detection** — contradictory facts silently coexist in vector stores, producing unpredictable answers.++These aren't edge cases. They are the reason AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch.++---++## The Solution++Semantica is the **context and intelligence layer** you add to your AI stack:-**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy.+- **Context Graphs** — structured graph of entities, relationships, and decisions your agent builds as it works. Queryable, traceable, persistent.+- **Decision Intelligence** — every decision is a first-class object: recorded, linked causally, searchable by precedent, and analyzable for downstream impact.+- **Provenance** — every fact links to its source. W3C PROV-O compliant. Full lineage from ingestion to inference.+- **Reasoning engines** — forward chaining, Rete networks, deductive, abductive, and SPARQL reasoning. Explainable inference paths, not black-box answers.+- **Deduplication & QA** — conflict detection, entity resolution, and validation built into the pipeline.-Perfect for **high-stakes domains** where mistakes have real consequences.+Works alongside LangChain, LlamaIndex, AutoGen, CrewAI, and any LLM provider — Semantica is not a replacement, it's the accountability layer on top.----### ⚡ Get Started in 30 Seconds+### ⚡ Quick Installation```bashpip install semantica``````python-from semantica.semantic_extract import NERExtractor-from semantica.kg import GraphBuilder+from semantica.context import AgentContext, ContextGraph+from semantica.vector_store import VectorStore++context = AgentContext(+ vector_store=VectorStore(backend="faiss", dimension=768),+ knowledge_graph=ContextGraph(advanced_analytics=True),+ decision_tracking=True,+)++# Store a memory+context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")++# Record a decision+decision_id = context.record_decision(+ category="model_selection",+ scenario="Choose LLM for production reasoning pipeline",+ reasoning="GPT-4 benchmark advantage justifies 3x cost increase",+ outcome="selected_gpt4",+ confidence=0.91,+)++# Find similar past decisions and analyze downstream impact+precedents = context.find_precedents("model selection reasoning", limit=5)+influence = context.analyze_decision_influence(decision_id)+```-# Extract entities and build knowledge graph-ner = NERExtractor(method="ml", model="en_core_web_sm")-entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")-kg = GraphBuilder().build({"entities": entities, "relationships": []})+**[Full Quick Start](getting-started.md)** • **[Cookbook](cookbook.md)** • **[Join Discord](https://discord.gg/sV34vps5hH)**-print(f"Built KG with {len(kg.get('entities', []))} entities")-```+---++## What's New in v0.3.0++> First stable release (`Production/Stable` on PyPI).-**[📖 Full Quick Start](getting-started.md)** • **[🍳 Cookbook Examples](cookbook.md)** • **[💬 Join Discord](https://discord.gg/sV34vps5hH)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**+| Area | Highlights |+|------|------------|+| **Context Graphs** | Temporal validity windows, weighted BFS, cross-graph navigation with save/load persistence |+| **Decision Intelligence** | Full lifecycle: record → trace → impact → precedent; `PolicyEngine` with versioned rules |+| **KG Algorithms** | PageRank, betweenness, Louvain community detection, Node2Vec, link prediction |+| **Semantic Extraction** | LLM extraction fixed (no silent drops), duplicate relation bug removed, `"llm_typed"` metadata corrected |+| **Deduplication v2** | `blocking_v2`/`hybrid_v2` — 63.6% faster; semantic v2 — 6.98x faster |+| **Delta Processing** | SPARQL-based incremental diff, `delta_mode` pipelines, snapshot versioning |+| **Export** | RDF aliases (`"ttl"`, `"json-ld"`), ArangoDB AQL, Apache Parquet (Spark/BigQuery/Databricks) |+| **Pipeline** | `FailureHandler` with LINEAR/EXPONENTIAL/FIXED backoff; `PipelineValidator` returning `ValidationResult` |+| **Graph Backends** | Apache AGE (SQL injection fixed), AWS Neptune, FalkorDB, PgVector (HNSW/IVFFlat) |+| **Tests** | 886+ passing, 0 failures — 335 context, ~430 KG, 70 semantic extraction, 85 real-world E2E |---## Core Value Proposition| **Trustworthy** | **Explainable** | **Auditable** |-|:------------------:|:------------------:|:-----------------:|+|:---:|:---:|:---:|| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking || Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage || Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification |----## Key Features & Benefits--### Not Just Another Agentic Framework--**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:--| Feature | Benefit |-|:--------|:--------|-| **Auditable** | Complete provenance tracking with W3C PROV-O compliance |-| **Explainable** | Transparent reasoning paths with entity relationships |-| **Provenance-Aware** | End-to-end lineage from documents to responses |-| **Validated** | Built-in conflict detection, deduplication, QA |-| **Governed** | Rule-based validation and semantic consistency |-| **Version Control** | Enterprise-grade change management with integrity verification |--### Perfect For High-Stakes Use Cases--| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |-|:-----------------:|:--------------:|:------------:|-| Clinical decisions | Fraud detection | Evidence-backed research |-| Drug interactions | Regulatory support | Contract analysis |-| Patient safety | Risk assessment | Case law reasoning |--| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |-|:-------------------:|:----------------:|:-------------------:|:-----------------:|-| Threat attribution | Policy decisions | Power grids | Decision logs |-| Incident response | Classified info | Transportation | Safety validation |--### Powers Your AI Stack--- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search-- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory-- **Reasoning Models** — Explainable AI decisions with reasoning paths-- **Enterprise AI** — Governed, auditable platforms that support compliance--### Integrations--- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)-- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication-- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)--> **Built for environments where every answer must be explainable and governed.**-------## 🚨 The Problem: The Semantic Gap--### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.--### Understanding the Semantic Gap--The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).--**Traditional AI approaches:**-- Rely on statistical patterns and text similarity-- Cannot understand relationships between entities-- Cannot reason about domain-specific rules-- Cannot explain why decisions were made-- Cannot trace back to original sources with confidence--**High-stakes AI requires:**-- Semantic understanding of entities and their relationships-- Domain knowledge encoded as formal rules (ontologies)-- Explainable reasoning paths-- Source-level provenance-- Conflict detection and resolution--**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.--### What Organizations Have vs What They Need--| **Current State** | **Required for High-Stakes AI** |-|:---------------------|:-----------------------------------|-| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |-| APIs, databases, streams | Structured and validated entities |-| Conflicting facts and duplicates | Explicit semantic relationships |-| Siloed systems with no lineage | **Explainable reasoning paths** |-| | **Source-level provenance** |-| | **Audit-ready compliance** |--### The Cost of Missing Semantics--- **Decisions cannot be explained** — No transparency in AI reasoning-- **Errors cannot be traced** — No way to debug or improve-- **Conflicts go undetected** — Contradictory information causes failures-- **Compliance becomes impossible** — No audit trails for regulations--**Trustworthy AI requires semantic accountability.**-------## 🆚 Semantica vs Traditional RAG--| Feature | Traditional RAG | Semantica |-|:--------|:----------------|:----------|-| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |-| **Provenance** | ❌ No provenance | ✅ W3C PROV-O compliant lineage tracking |-| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |-| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |-| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |-| **Compliance** | ❌ No audit trails | ✅ Complete audit trails with integrity verification |+## Features++### Context & Decision Intelligence+- **Context Graphs** — structured, persistent graph of entities, relationships, and decisions+- **Decision tracking** — `add_decision()`, `record_decision()` for full lifecycle management+- **Causal chains** — `add_causal_relationship()`, `trace_decision_chain()`+- **Precedent search** — hybrid similarity search over past decisions via `find_similar_decisions()`+- **Influence analysis** — `analyze_decision_impact()`, `analyze_decision_influence()`+- **Policy engine** — `check_decision_rules()` with versioned, automated compliance rules+- **Agent memory** — `AgentMemory` with short/long-term storage and conversation history++### Knowledge Graphs+- **Graph construction** — entities, relationships, properties, typed edges+- **Algorithms** — PageRank, betweenness centrality, clustering coefficient, community detection+- **Node embeddings** — Node2Vec via `NodeEmbedder`; cosine similarity via `SimilarityCalculator`+- **Link prediction** — score potential edges via `LinkPredictor`+- **Temporal graphs** — time-aware nodes and edges with validity windows+- **Delta processing** — incremental updates without full recompute++### Semantic Extraction+- **NER** — named entity recognition, normalization, classification+- **Relation extraction** — triplet generation via LLMs or rule-based methods, with `"llm_typed"` metadata+- **Deduplication v1/v2** — Jaro-Winkler, `blocking_v2`, `hybrid_v2`, `semantic_v2`; `dedup_triplets()` for triples++### Reasoning+- **Forward chaining** — `Reasoner` with IF/THEN string rules and dict facts+- **Rete network** — `ReteEngine` for high-throughput production rule matching+- **Deductive / Abductive** — `DeductiveReasoner`, `AbductiveReasoner`+- **SPARQL** — `SPARQLReasoner` for query-based inference over RDF graphs++### Provenance & Auditability+- **Entity provenance** — `ProvenanceTracker.track_entity()`+- **Algorithm provenance** — `AlgorithmTrackerWithProvenance`+- **W3C PROV-O compliant** — lineage tracking across all modules+- **Change management** — version control with checksums, audit trails, compliance support++### Vector Store+- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory+- **Search modes** — semantic top-k, hybrid (vector + keyword), metadata-filtered++### Data Ingestion+- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives+- **Sources** — web crawl, SQL databases, Snowflake, feeds, email, repositories+- **Docling** — advanced parsing with table and layout extraction+- **Media** — image OCR, audio/video metadata++### Export+- **RDF** — Turtle, JSON-LD, N-Triples, XML via `RDFExporter`+- **Parquet** — `ParquetExporter` for Spark/BigQuery/Databricks pipelines+- **ArangoDB AQL** — ready-to-run INSERT statements+- **OWL ontologies** — Turtle or RDF/XML++### Pipeline & Ontology+- **Pipeline DSL** — `PipelineBuilder` with stage chaining, parallel workers, retry policies+- **Ontology** — auto-generate OWL from KGs, import OWL/RDF/Turtle/JSON-LD, HermiT/Pellet validation----## 🧩 Semantica Architecture--### 1️⃣ Input Layer — Governed Ingestion-- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX-- 🔧 **Docling Support** — Docling parser for table extraction-- 💾 **Data Sources** — Databases, APIs, streams, archives, web content-- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction-- � **Single Pipeline** — Unified ingestion with metadata and source tracking--### 2️⃣ Semantic Layer — Trust & Reasoning Engine-- 🔍 **Entity Extraction** — NER, normalization, classification-- 🔗 **Relationship Discovery** — Triplet generation, semantic links-- 📐 **Ontology Induction** — Automated domain rule generation-- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution-- ✅ **Quality Assurance** — Conflict detection, validation-- 📊 **Provenance Tracking** — W3C PROV-O compliant lineage tracking across all modules-- 🧠 **Reasoning Traces** — Explainable inference paths-- 🔐 **Change Management** — Version control with audit trails, checksums, compliance support--### 3️⃣ Output Layer — Auditable Knowledge Assets-- � **Knowledge Graphs** — Queryable, temporal, explainable-- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support-- 🔢 **Vector Embeddings** — FastEmbed by default-- ☁️ **AWS Neptune** — Amazon Neptune graph database support-- 🔍 **Provenance** — Every AI response links back to:- - 📄 Source documents- - 🏷️ Extracted entities & relations- - 📐 Ontology rules applied- - 🧠 Reasoning steps used+## Modules++| Module | What it provides |+|--------|-----------------|+| `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search, policy engine |+| `semantica.kg` | KG construction, graph algorithms, centrality, community detection, embeddings, link prediction |+| `semantica.semantic_extract` | NER, relation extraction, event extraction, coreference, triplet generation, LLM extraction |+| `semantica.reasoning` | Forward chaining, Rete network, deductive, abductive, SPARQL reasoning, explanation generation |+| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector; hybrid & filtered search |+| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, YAML, OWL, graph formats |+| `semantica.ingest` | Files, web crawl, feeds, databases, Snowflake, MCP, email, repositories |+| `semantica.ontology` | Auto-generation, OWL/RDF export, import, validation, versioning |+| `semantica.pipeline` | Pipeline DSL, parallel workers, validation, retry policies, failure handling |+| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune; Cypher queries |+| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE; similarity calculation |+| `semantica.deduplication` | Entity deduplication, similarity scoring, merging, clustering |+| `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails |+| `semantica.parse` | PDF, DOCX, PPTX, HTML, code, email, structured data, OCR |+| `semantica.split` | Recursive, semantic, entity-aware, relation-aware, graph-based chunking |+| `semantica.normalize` | Text, entities, dates, numbers, quantities, languages, encodings |+| `semantica.conflicts` | Multi-source conflict detection (value, type, temporal, logical) with resolution |+| `semantica.change_management` | Version storage, change tracking, checksums, audit trails |+| `semantica.triplet_store` | Blazegraph, Jena, RDF4J; SPARQL queries and bulk loading |+| `semantica.visualization` | Interactive/static KG, ontology, embedding, and temporal graph visualization |+| `semantica.core` | Framework orchestration, configuration, plugin system |+| `semantica.llms` | Groq, OpenAI, Novita AI, HuggingFace, LiteLLM integrations |----## 🏥 Built for High-Stakes Domains--Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:--- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking-- **💰 Finance & Risk** — Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation-- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning-- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis-- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence-- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response-- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation-----+## Built for High-Stakes Domains-## � Who Uses Semantica?+Where **every decision must be accountable** and **mistakes have real consequences**:-- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents-- **⚙️ Data Engineers** — Creating governed semantic pipelines-- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale-- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure-- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems+- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interactions, patient safety+- **💰 Finance & Risk** — Fraud detection, SOX/GDPR/MiFID II compliance, risk assessment+- **⚖️ Legal & Compliance** — Evidence-backed research, contract analysis, regulatory tracking+- **🔒 Cybersecurity** — Threat attribution, incident response, security audit trails+- **🏛️ Government & Defense** — Policy decisions, classified information handling, defense intelligence+- **🏭 Critical Infrastructure** — Power grids, transportation safety, emergency response+- **🚗 Autonomous Systems** — Self-driving, robotics safety, industrial automation----## 🚀 Choose Your Path+## Choose Your Path<div class="grid cards" markdown>- :material-rocket-launch: **Quick Start**---- Get up and running with Semantica in minutes. Learn the basics of ingestion and extraction.-+ Up and running in minutes.+[:arrow_right: Start Here](getting-started.md)- :material-book-open-page-variant: **Core Concepts**---- Deep dive into Knowledge Graphs, Ontologies, and Semantic Reasoning.-+ Knowledge graphs, ontologies, and semantic reasoning explained.+[:arrow_right: Learn Concepts](concepts.md)- :material-code-braces: **API Reference**---- Detailed technical documentation for all Semantica modules and classes.-+ Full technical documentation for every module and class.+[:arrow_right: View API](reference/core.md)- :material-chef-hat: **Cookbook**---- Interactive tutorials, real-world examples, and **14 domain-specific cookbooks**.-+ 14 domain-specific cookbooks with real-world examples.+[:arrow_right: Explore Cookbook](cookbook.md)</div>----## 📦 Installation+## Installation!!! success "Now Available on PyPI!"- Semantica is officially published on PyPI! Install it with a single command.+ Install with a single command.-=== "From PyPI (Recommended)"-- Install Semantica directly from PyPI:+=== "PyPI (Recommended)"```bash- # Install the core packagepip install semantica- # Or install with all optional dependencies+ # With all optional dependenciespip install semantica[all]```=== "From Source"- Install from the local source for the latest development version:-```bash- # Clone the repositorygit clone https://github.com/Hawksight-AI/semantica.gitcd semantica-- # Install in editable mode with core dependencies- pip install -e .-- # Or install with all optional dependencies- pip install -e ".[all]"+ pip install -e . # core+ pip install -e ".[all]" # all extras```=== "Development"- For contributors who want to modify the framework:-```bash- # Clone the repositorygit clone https://github.com/Hawksight-AI/semantica.gitcd semantica-- # Install in editable mode with dev dependenciespip install -e ".[dev]"```=== "Docker"- Run Semantica in a containerized environment:-```bashdocker pull semantica/semantica:latestdocker run -it semantica/semantica@@ -316,117 +293,41 @@ Designed for domains where **mistakes have real consequences** and **every decis----## 🚦 Quick Example--Semantica uses a modular architecture. You can use individual modules directly for maximum flexibility:--```python-from semantica.ingest import FileIngestor-from semantica.parse import DocumentParser-from semantica.semantic_extract import NERExtractor, RelationExtractor-from semantica.kg import GraphBuilder--# 1. Ingest documents-ingestor = FileIngestor()-documents = ingestor.ingest_directory("documents/", recursive=True)--# 2. Parse documents-parser = DocumentParser()-parsed_docs = [parser.parse_document(doc) for doc in documents]--# 3. Extract entities and relationships-ner = NERExtractor()-rel_extractor = RelationExtractor()--entities = []-relationships = []-for doc in parsed_docs:- text = doc.get("full_text", "")- doc_entities = ner.extract_entities(text)- doc_rels = rel_extractor.extract_relations(text, entities=doc_entities)- entities.extend(doc_entities)- relationships.extend(doc_rels)--# 4. Build knowledge graph-builder = GraphBuilder(merge_entities=True)-kg = builder.build_graph(entities=entities, relationships=relationships)--print(f"Created graph with {len(kg.nodes)} nodes and {len(kg.edges)} edges")-```--!!! tip "Orchestration Option"- For complex workflows, you can also use the `Semantica` class for orchestration. See the [Core Module](reference/core.md) documentation for details.-------## 🎯 Why Semantica?+## Why Semantica?<div class="grid cards" markdown>- **🆓 Open Source**---- MIT licensed. No vendor lock-in. Full transparency.+ MIT licensed. No vendor lock-in.- **🚀 Production Ready**---- Battle-tested with quality assurance, conflict resolution, and validation.+ Battle-tested with QA, conflict resolution, and validation built in.-- **🧩 Modular Architecture**+- **🧩 Modular**---Use only what you need. Swap components easily.- **🌍 Community Driven**---- Built by developers, for developers. Active Discord community.+ Built by developers, for developers. Active Discord.-- **📚 Comprehensive**+- **📚 End-to-End**---- End-to-end solution from ingestion to reasoning. No duct-taping required.+ From ingestion to reasoning — no duct-taping required.- **🔬 Research-Backed**---- Based on latest research in knowledge graphs, ontologies, and semantic web.+ Grounded in knowledge graph, ontology, and semantic web research.</div>----## 🏗️ Built For--- **Data Scientists**: Transform messy data into clean knowledge graphs-- **Data Engineers**: Build scalable data pipelines with semantic enrichment-- **AI Engineers**: Build GraphRAG, AI agents, and multi-agent systems-- **Knowledge Engineers**: Generate and manage formal ontologies-- **Ontologists**: Design and validate domain-specific ontologies and taxonomies-- **Researchers**: Analyze scientific literature and build citation networks-- **ML Engineers**: Create semantic features for machine learning models-- **Enterprises**: Unify data silos into a semantic layer-------## 📚 Learn More--- [Getting Started Guide](getting-started.md) - Your first knowledge graph in 5 minutes-- [Core Concepts](concepts.md) - Deep dive into knowledge graphs and ontologies-- [Cookbook](cookbook.md) - Real-world examples and **14 domain-specific cookbooks**-- [API Reference](reference/core.md) - Complete technical documentation--### 🍳 Recommended Cookbook Tutorials--Get hands-on with interactive Jupyter notebooks:--- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all Semantica modules- - **Topics**: Framework overview, all modules, architecture- - **Difficulty**: Beginner- - **Use Cases**: First-time users, understanding the framework--- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from scratch- - **Topics**: Entity extraction, relationship extraction, graph construction- - **Difficulty**: Beginner- - **Use Cases**: Learning the basics, quick start--- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready Graph Retrieval Augmented Generation- - **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal- - **Difficulty**: Advanced- - **Use Cases**: Building AI applications with knowledge graphs+## Learn More+- [Getting Started](getting-started.md) — your first knowledge graph in 5 minutes+- [Core Concepts](concepts.md) — knowledge graphs, ontologies, and semantic reasoning+- [Cookbook](cookbook.md) — 14 domain-specific cookbooks with Jupyter notebooks+- [API Reference](reference/core.md) — complete technical documentationdiff --git a/docs/installation.md b/docs/installation.mdindex d786c39a..844eaeb0 100644--- a/docs/installation.md+++ b/docs/installation.md@@ -1,293 +1,169 @@# Installation-Get Semantica up and running in minutes.+Get Semantica installed in under a minute.-!!! success "Now Available on PyPI!"- Semantica is officially published on PyPI! Install it with a single command: `pip install semantica`+!!! success "Available on PyPI"+ `pip install semantica` — that's it.-!!! note "System Requirements"- Semantica requires Python 3.8 or higher. For best performance, we recommend Python 3.10+.+!!! note "Requirements"+ Python 3.8 or higher. Python 3.11+ recommended.-## Prerequisites--Before installing Semantica, ensure you have:--- **Python 3.8 or higher** - Check your version:- ```bash- python --version- ```-- **pip** - Python package installer (usually comes with Python)+---## Basic Installation-Install Semantica from PyPI:-```bashpip install semantica```-This installs Semantica with all core dependencies.--### GitHub Workaround--If you encounter issues with the PyPI version, you can install directly from the main branch:+With all optional dependencies:```bash-pip install git+https://github.com/Hawksight-AI/semantica.git@main+pip install semantica[all]```-!!! tip "Virtual Environment"- We recommend installing Semantica in a virtual environment to avoid dependency conflicts. Use `python -m venv venv` to create one, then activate it before installing.--## Verify Installation--Verify that Semantica is installed correctly:+### Verify```bash-python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"-```--!!! info "Windows PyTorch Note"- If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.--Expected output:-```-✓ Semantica ready+python -c "import semantica; print(semantica.__version__)"```-You can also check the installation:+----```bash-pip show semantica-```+## Virtual Environment (Recommended)-## Development Installation+=== "venv"-To install Semantica in development mode (for contributing):+ ```bash+ python -m venv venv+ source venv/bin/activate # Linux / Mac+ venv\Scripts\activate # Windows+ pip install semantica+ ```-```bash-# Clone the repository-git clone https://github.com/Hawksight-AI/semantica.git-cd semantica+=== "conda"-# Install in editable mode-pip install -e .+ ```bash+ conda create -n semantica python=3.11+ conda activate semantica+ pip install semantica+ ```-# Or install with development dependencies-pip install -e ".[dev]"-```+---## Optional Dependencies-Semantica supports optional features that can be installed separately:+Install only what you need:-### GPU Support+=== "GPU"-For GPU-accelerated operations:+ ```bash+ pip install semantica[gpu]+ ```+ Includes PyTorch with CUDA, FAISS GPU, CuPy.-```bash-pip install semantica[gpu]-```+=== "Visualization"-This includes:-- PyTorch with CUDA support-- FAISS GPU-- CuPy+ ```bash+ pip install semantica[viz]+ ```+ Includes PyVis, Graphviz, UMAP.-### Visualization+=== "LLM Providers"-For enhanced visualization capabilities:+ ```bash+ pip install semantica[llm-all] # all providers-```bash-pip install semantica[viz]-```+ pip install semantica[llm-openai] # OpenAI+ pip install semantica[llm-anthropic] # Anthropic+ pip install semantica[llm-gemini] # Google Gemini+ pip install semantica[llm-groq] # Groq+ pip install semantica[llm-ollama] # Ollama (local)+ ```-Includes:-- PyVis for interactive graphs-- Graphviz for static diagrams-- UMAP for dimensionality reduction+=== "Cloud"-### LLM Providers+ ```bash+ pip install semantica[cloud]+ ```+ Includes AWS S3, Azure Blob, Google Cloud Storage.-Install all LLM provider integrations:+----```bash-pip install semantica[llm-all]-```+## Install from Source-Or install specific providers:+For the latest development version or to contribute:```bash-# OpenAI-pip install semantica[llm-openai]--# Anthropic-pip install semantica[llm-anthropic]--# Google Gemini-pip install semantica[llm-gemini]--# Groq-pip install semantica[llm-groq]--# Ollama-pip install semantica[llm-ollama]-```--### Cloud Integrations--For cloud storage and deployment:+git clone https://github.com/Hawksight-AI/semantica.git+cd semantica-```bash-pip install semantica[cloud]+pip install -e . # core only+pip install -e ".[all]" # all extras+pip install -e ".[dev]" # dev tools (pytest, black, etc.)```-Includes:-- AWS S3 (boto3)-- Azure Blob Storage-- Google Cloud Storage-- Kubernetes support--### All Optional Features--Install everything:+If you encounter issues with the PyPI release, install directly from the main branch:```bash-pip install semantica[all]+pip install git+https://github.com/Hawksight-AI/semantica.git@main```-## Virtual Environment (Recommended)--It's recommended to use a virtual environment:--=== "venv"-- ```bash- # Create virtual environment- python -m venv venv-- # Activate (Windows)- venv\Scripts\activate-- # Activate (Linux/Mac)- source venv/bin/activate-- # Install Semantica- pip install semantica- ```--=== "conda"-- ```bash- # Create conda environment- conda create -n semantica python=3.11- conda activate semantica-- # Install Semantica- pip install semantica- ```+---## Troubleshooting-### Common Issues+### ModuleNotFoundError-#### ModuleNotFoundError+Check you have the right environment active:-**Error**: `ModuleNotFoundError: No module named 'semantica'`+```bash+pip list | grep semantica+pip install --upgrade semantica+```-**Solutions**:-- Make sure you've activated the correct Python environment-- Verify installation: `pip list | grep semantica`-- Reinstall: `pip install --upgrade semantica`+### Installation fails with dependency errors-#### Installation Fails+```bash+pip install --upgrade pip+pip install build wheel+pip install semantica --no-deps # install without optional deps first+```-**Error**: Installation fails with dependency errors+### GPU dependencies fail-**Solutions**:-- Upgrade pip: `pip install --upgrade pip`-- Install build tools: `pip install build wheel`-- Try installing without optional dependencies first: `pip install semantica --no-deps`+Install CPU-only first, then add GPU support:-#### GPU Dependencies Fail+```bash+pip install semantica+pip install semantica[gpu]+```-**Error**: GPU dependencies fail to install+### Permission denied-**Solutions**:-- Install CPU-only version first: `pip install semantica`-- Then add GPU support: `pip install semantica[gpu]`-- Check CUDA compatibility for your system+```bash+pip install --user semantica # or use a virtual environment+```-#### Permission Errors+### Windows PyTorch DLL errors-**Error**: Permission denied during installation+Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency, not a Semantica bug.-**Solutions**:-- Use `--user` flag: `pip install --user semantica`-- Use virtual environment (recommended)-- On Linux/Mac, avoid using `sudo` with pip+----### System Requirements+## System Requirements| Component | Minimum | Recommended ||-----------|---------|-------------|| Python | 3.8 | 3.11+ |-| RAM | Moderate | Ample for your dataset |-| Disk Space | Sufficient for data | Generous storage |-| OS | Windows/Linux/Mac | Linux/Mac |--## After Installation+| OS | Windows / Linux / Mac | Linux / Mac |+| RAM | 4 GB | 16 GB+ |+| Storage | 2 GB | 20 GB+ (for models and data) |-Once Semantica is installed, verify your setup and get started:--### Verify Your Installation--Test that everything works correctly:--```bash-python -c "import semantica; print(semantica.__version__)"-```--**For detailed setup verification and first steps, see:**-- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Verify installation and explore all modules- - **Topics**: Framework overview, installation verification, module exploration- - **Difficulty**: Beginner- - **Time**: 30-45 minutes- - **Use Cases**: First-time setup, understanding the framework+---## Next Steps-Now that Semantica is installed:--1. **[Quick Start Guide](quickstart.md)** - Build your first knowledge graph in 5 minutes-2. **[Getting Started Guide](getting-started.md)** - Learn the fundamentals-3. **[Examples](examples.md)** - See real-world use cases-4. **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials--### 🍳 Recommended First Cookbooks--Start with these interactive tutorials:--- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction- - **Topics**: Framework overview, all modules, architecture, configuration- - **Difficulty**: Beginner- - **Time**: 30-45 minutes- - **Use Cases**: First-time users, understanding the framework--- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first graph- - **Topics**: Entity extraction, relationship extraction, graph construction- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Hands-on practice, quick start--## Getting Help--If you encounter issues:--- Check the [troubleshooting section](#troubleshooting) above-- Review [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)-- Ask questions in [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)--**For installation and setup help:**-- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Includes setup verification steps-- **[Installation Troubleshooting Guide](getting-started.md#installation--setup)**: Additional troubleshooting tips+- [Getting Started](getting-started.md) — build your first knowledge graph+- [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline+- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorialsdiff --git a/docs/integrations/docling.md b/docs/integrations/docling.mdindex 1b2f59ed..5a5593cf 100644--- a/docs/integrations/docling.md+++ b/docs/integrations/docling.md@@ -13,7 +13,7 @@ Docling is integrated into Semantica's `parse` module via the `DoclingParser`. T----## 📖 Integration Documentation+## Integration DocumentationThe `DoclingParser` provides a high-level interface for document processing. It supports:@@ -42,7 +42,7 @@ For more details, see the [Parse Reference](../reference/parse.md).----## 🧑🏽🍳 Integration Example+## Integration ExampleWe provide a detailed cookbook and clear code examples to help you get started quickly.@@ -81,7 +81,7 @@ See more in our [Code Examples](../CodeExamples.md).----## 💻 GitHub Source+## GitHub SourceThe integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.@@ -89,7 +89,7 @@ The integration is open-source and available on GitHub. You can explore the impl----## 📦 PyPI & Installation+## PyPI & InstallationDocling is an optional but highly recommended dependency for Semantica. You can install it along with Semantica or as a separate requirement.diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.mdindex 8e624262..d0b1d092 100644--- a/docs/integrations/snowflake.md+++ b/docs/integrations/snowflake.md@@ -13,7 +13,7 @@ Snowflake is integrated into Semantica's `ingest` module via the `SnowflakeInges----## 📖 Integration Documentation+## Integration DocumentationThe `SnowflakeIngestor` provides a high-level interface for Snowflake data ingestion. It supports:@@ -42,7 +42,7 @@ For more details, see the [Ingest Reference](../reference/ingest.md).----## 🧑🏽🍳 Integration Example+## Integration ExampleWe provide a detailed cookbook and clear code examples to help you get started quickly.@@ -97,7 +97,7 @@ See more in our [Code Examples](../CodeExamples.md).----## 💻 GitHub Source+## GitHub SourceThe integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.@@ -105,7 +105,7 @@ The integration is open-source and available on GitHub. You can explore the impl----## 📦 PyPI & Installation+## PyPI & InstallationSnowflake connector is an optional dependency for Semantica. You can install it along with Semantica or as a separate requirement.@@ -128,7 +128,7 @@ For full installation details, see the [Installation Guide](../installation.md).----## 🔐 Authentication Methods+## Authentication MethodsSnowflake integration supports multiple authentication methods for different security requirements:@@ -175,7 +175,7 @@ ingestor = SnowflakeIngestor(----## 🚀 Advanced Features+## Advanced Features### Schema Introspection```python@@ -209,7 +209,7 @@ data = ingestor.ingest_query(----## 📊 Best Practices+## Best Practices### Use Environment Variables```python@@ -244,7 +244,7 @@ for page in range(total_pages):----## 🔍 Troubleshooting+## Troubleshooting### Connection Issues```python@@ -272,7 +272,7 @@ ingestor = SnowflakeIngestor(----## 📚 See Also+## See Also- **[Ingest Module Reference](../reference/ingest.md)** - Complete ingestion documentation- **[Getting Started Guide](../getting-started.md)** - Quick start with Semanticadiff --git a/docs/learning-more.md b/docs/learning-more.mdindex def221ea..49299455 100644--- a/docs/learning-more.md+++ b/docs/learning-more.md@@ -1,213 +1,104 @@# Learning More-Additional resources, tutorials, and advanced learning materials for Semantica.--!!! info "About This Guide"- This guide provides structured learning paths, quick references, troubleshooting guides, and advanced topics to help you master Semantica.+Structured learning paths, quick references, and performance guidance for going deeper with Semantica.----## Structured Learning Paths+## Learning Paths<div class="grid cards" markdown>-- :material-school: **Beginner Path**+- :material-school: **Beginner** (1–2 hours)---- Perfect for those new to Semantica and knowledge graphs.-+ New to Semantica and knowledge graphs.-- [Start Path](#beginner-path-1-2-hours)+ [Start here](#beginner-path)-- :material-compass: **Intermediate Path**+- :material-compass: **Intermediate** (4–6 hours)---- For users comfortable with basics who want to build production applications.-+ Comfortable with basics, building production applications.-- [Start Path](#intermediate-path-4-6-hours)+ [Start here](#intermediate-path)-- :material-rocket: **Advanced Path**+- :material-rocket: **Advanced** (8+ hours)---- For experienced users building enterprise applications.-+ Enterprise applications and customization.-- [Start Path](#advanced-path-8-hours)+ [Start here](#advanced-path)</div>----### Beginner Path (1-2 hours)--1. **Installation & Setup** (15 min)- - [Installation Guide](installation.md)- - **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction- - **Topics**: Framework overview, all modules, architecture, configuration- - **Difficulty**: Beginner- - **Time**: 30-45 minutes- - **Use Cases**: First-time users, understanding the framework--2. **Core Concepts** (30 min)- - [Core Concepts](concepts.md)- - [Getting Started Guide](getting-started.md)- - **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources- - **Topics**: File, web, feed, stream, database ingestion- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Loading data from various sources--3. **First Knowledge Graph** (30 min)- - [Quickstart Tutorial](quickstart.md)- - **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Learning the basics, quick start--4. **Basic Operations** (30 min)- - [Examples](examples.md)- - **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction- - **Topics**: Named entity recognition, entity types, extraction methods- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Understanding entity extraction-----+### Beginner Path-### Intermediate Path (4-6 hours)--1. **Advanced Concepts** (1 hour)- - [Modules Guide](modules.md)- - **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Advanced graph construction- - **Topics**: Graph building, entity merging, conflict resolution, temporal graphs- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Production graph construction- - **[Embeddings Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)**: Learn embeddings- - **Topics**: Embedding generation, similarity search, vector operations- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Understanding embeddings, semantic search--2. **Use Cases** (1 hour)- - [Use Cases Guide](use-cases.md)- - **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Build production GraphRAG- - **Topics**: GraphRAG, hybrid retrieval, graph traversal, LLM integration- - **Difficulty**: Advanced- - **Time**: 1-2 hours- - **Use Cases**: Production GraphRAG systems--3. **Advanced Examples** (1 hour)- - [Examples](examples.md)- - **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns- - **Topics**: Custom entity types, domain-specific extraction, hybrid methods- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Domain-specific extraction--4. **Quality & Optimization** (1 hour)- - [Quality Assurance](concepts.md#8-quality-assurance)- - [Performance Optimization](#performance-optimization)- - **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Integrate multiple sources- - **Topics**: Multi-source integration, entity resolution, conflict handling- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Building unified knowledge graphs+1. **Installation & Setup** — [Installation Guide](installation.md)+2. **Core Concepts** — [Core Concepts](concepts.md) + [Getting Started](getting-started.md)+3. **First Knowledge Graph** — [Quickstart Tutorial](quickstart.md)+4. **Interactive Introduction** — [Welcome to Semantica notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)+5. **Hands-On Practice** — [Your First Knowledge Graph notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)----### Advanced Path (8+ hours)--1. **Advanced Architecture** (2 hours)- - [Architecture Guide](architecture.md)- - **[Temporal Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)**: Build temporal graphs- - **Topics**: Time-stamped entities, temporal relationships, historical queries- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Time-aware knowledge graphs- - **[Ontology Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)**: Generate ontologies- - **Topics**: Ontology generation, OWL, schema design- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Formal knowledge representation--2. **Production Deployment** (2 hours)- - [Security Best Practices](#security-best-practices)- - **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG- - **Topics**: Production deployment, scalability, optimization- - **Difficulty**: Advanced- - **Time**: 1-2 hours- - **Use Cases**: Production systems--3. **Customization** (2 hours)- - **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization- - **Topics**: Custom layouts, filtering, styling, multiple graph types- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Production visualizations- - **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns- - **Topics**: Batch export, custom formats, format conversion- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Production exports+### Intermediate Path----+1. **All Modules** — [Modules Guide](modules.md)+2. **Advanced Graph Construction** — [Building Knowledge Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)+3. **Embeddings & Search** — [Embeddings notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)+4. **GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)+5. **Multi-Source Integration** — [Multi-Source Data Integration notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)+6. **Use Case Examples** — [Use Cases](use-cases.md)-## Quick Reference--### Common Operations+----The typical workflow involves these steps:+### Advanced Path-1. **Ingest** documents using `` `FileIngestor` ``-2. **Parse** documents using `` `DocumentParser` ``-3. **Extract** entities and relationships using `` `NERExtractor` `` and `` `RelationExtractor` ``-4. **Build** knowledge graph using `` `GraphBuilder` ``-5. **Generate** embeddings using `` `TextEmbedder` ``+1. **Architecture Deep Dive** — [Architecture Guide](architecture.md)+2. **Temporal Graphs** — [Temporal Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)+3. **Ontologies** — [Ontology notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)+4. **Visualization** — [Complete Visualization Suite notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)+5. **Export Pipelines** — [Multi-Format Export notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)+6. **Production GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)-**For complete examples, see:**-- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete workflow example-- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: All modules overview+----### Configuration Reference+## Configuration Reference-| Setting | Environment Variable | Config File | Default |-| :--- | :--- | :--- | :--- |-| OpenAI API Key | `OPENAI_API_KEY` | `api_keys.openai` | `None` |-| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `embedding.provider` | `"openai"` |-| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `knowledge_graph.backend` | `"networkx"` |+| Setting | Environment Variable | Default |+|---------|---------------------|---------|+| OpenAI API Key | `OPENAI_API_KEY` | `None` |+| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `"openai"` |+| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `"networkx"` |----## Troubleshooting Guide+## Troubleshooting<div class="grid cards" markdown>- :material-alert: **Import Errors**---`ModuleNotFoundError`-- **Solution**: Verify installation (`pip list`) and Python version (3.8+).++ Verify installation: `pip list | grep semantica`. Ensure Python 3.8+.- :material-key: **API Key Errors**---`AuthenticationError`-- **Solution**: Set `OPENAI_API_KEY` environment variable.++ Set `OPENAI_API_KEY` (or the relevant provider key) as an environment variable.- :material-memory: **Memory Errors**---- `MemoryError`-- **Solution**: Use batch processing and graph stores (Neo4j).+ `MemoryError` or OOM crashes++ Reduce batch sizes or switch to a persistent graph backend (Neo4j, FalkorDB).- :material-speedometer: **Slow Processing**---- Long processing times-- **Solution**: Enable parallel processing and GPU acceleration.+ Long runtimes on large datasets++ Enable parallel processing (`PipelineBuilder` workers) and GPU acceleration.</div>@@ -215,80 +106,46 @@ The typical workflow involves these steps:## Performance Optimization-### 1. Batch Processing+### Batch Processing-Process multiple documents together for better throughput. Use batch processing when working with large document collections.+Process documents in batches rather than one at a time. Configure chunk sizes based on available RAM.-**For examples, see:**-- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Batch ingestion patterns-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration+### Parallel Execution-### 2. Parallel Execution+`PipelineBuilder` supports configurable worker counts per stage for independent operations.-Use parallel processing for independent operations to improve performance on multi-core systems.+### Backend Selection-### 3. Backend Selection+| Operation | NetworkX | Neo4j / FalkorDB |+|-----------|----------|------------------|+| Graph construction | Fast | Moderate |+| Query performance | Moderate | Fast |+| Scalability | Low (in-memory) | High (persistent) |-| Operation | NetworkX | Neo4j |-| :--- | :--- | :--- |-| **Graph Construction** | ⚡⚡⚡ | ⚡⚡ |-| **Query Performance** | ⚡⚡ | ⚡⚡⚡ |-| **Scalability** | Low | High |+Use NetworkX for development and smaller graphs; switch to a persistent backend for production at scale.---## Security Best Practices-### API Key Management--- **DO**: Use environment variables, rotate keys regularly.-- **DON'T**: Hardcode keys, commit to version control.--### Data Privacy+**API keys**+- Store in environment variables or a secrets manager+- Never hardcode keys or commit them to version control+- Rotate keys regularly-- **DO**: Encrypt sensitive data, use local models.-- **DON'T**: Send PII to external APIs without protection.-------## FAQ--**Q: What is Semantica?**-A: A framework for building knowledge graphs and semantic applications.--**Q: Is Semantica free?**-A: Yes, it is open source. Some features (e.g., OpenAI) require paid APIs.--**Q: Can I use Semantica in production?**-A: Yes, it is designed for production with proper configuration.+**Data privacy**+- Use local embedding models for sensitive data+- Avoid sending PII to external APIs without appropriate data handling agreements+- Encrypt sensitive graph exports at rest---## Next Steps-Continue your learning journey:--- **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials-- **[API Reference](reference/core.md)** - Complete API documentation-- **[Use Cases](use-cases.md)** - Real-world applications-- **[Examples](examples.md)** - Code examples and patterns--### 🍳 Recommended Next Cookbooks--- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG system- - **Topics**: GraphRAG, hybrid retrieval, LLM integration- - **Difficulty**: Advanced- - **Time**: 1-2 hours- - **Use Cases**: Production RAG applications--- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Understand the differences- - **Topics**: RAG comparison, reasoning gap, inference engines- - **Difficulty**: Intermediate- - **Time**: 45-60 minutes- - **Use Cases**: Choosing the right approach-------!!! info "Contribute"- Have questions? [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions)!+- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials+- [API Reference](reference/core.md) — complete technical documentation+- [Use Cases](use-cases.md) — real-world domain examples+- [FAQ](faq.md) — common questions+!!! info "Questions or feedback?"+ [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions).diff --git a/docs/modules.md b/docs/modules.mdindex 23c36801..d03bc0a3 100644--- a/docs/modules.md+++ b/docs/modules.md@@ -1,9 +1,9 @@-# Modules & Architecture+# Modules-**Complete guide to Semantica's modular architecture and how to use each component.**+Every Semantica module works independently — use only what you need.-!!! tip "Modular Design"- Each Semantica module works independently. Use only what you need for your specific use case.+!!! tip "Just need a quick reference?"+ Jump to the [Module Index](#module-index) at the bottom of this page.---@@ -82,13 +82,11 @@ web_ingestor = WebIngestor()pages = web_ingestor.ingest_urls(["https://example.com"])```-**What it does:**- **File formats** - PDF, DOCX, TXT, JSON, CSV- **Web scraping** - Extract content from websites- **Database** - Connect to SQL and NoSQL databases- **Batch processing** - Handle large datasets efficiently-**Use Cases:**- Document processing pipelines- Web data extraction- Database integration@@ -106,13 +104,11 @@ text = parsed["full_text"]metadata = parsed["metadata"]```-**What it does:**- **Text extraction** - Extract clean text from documents- **Metadata parsing** - Extract titles, authors, dates- **Structure analysis** - Identify sections, headings- **OCR support** - Handle scanned documents-**Use Cases:**- PDF processing- Document analysis- Content extraction@@ -130,13 +126,11 @@ splitter = TextSplitter(method="semantic")chunks = splitter.split(text, chunk_size=1000, overlap=200)```-**What it does:**- **Intelligent chunking** - Split text while preserving context- **Semantic splitting** - Break at natural boundaries- **Size control** - Manage chunk sizes for processing- **Overlap handling** - Maintain context between chunks-**Use Cases:**- Document preprocessing- Embedding preparation- RAG systems@@ -155,13 +149,11 @@ clean_text = normalizer.normalize_text(text)standardized_date = normalizer.normalize_date("Jan 1st, 2020")```-**What it does:**- **Text cleaning** - Remove noise and artifacts- **Date standardization** - Convert to ISO format- **Name normalization** - Standardize person names- **Entity normalization** - Clean up company names-**Use Cases:**- Data preprocessing- Quality improvement- Standardization@@ -186,13 +178,11 @@ rel_extractor = RelationExtractor()relationships = rel_extractor.extract(text, entities)```-**What it does:**- **Named Entity Recognition** - Find people, orgs, locations- **Relationship extraction** - Find connections between entities- **Custom entities** - Define your own entity types- **Confidence scoring** - Quality assessment for extractions-**Use Cases:**- Knowledge graph construction- Document analysis- Information extraction@@ -215,13 +205,11 @@ analyzer = GraphAnalyzer()stats = analyzer.analyze(kg)```-**What it does:**- **Graph construction** - Build knowledge graphs from data- **Graph analysis** - Calculate metrics and statistics- **Graph querying** - Search and retrieve information- **Graph manipulation** - Merge, split, transform graphs-**Use Cases:**- Knowledge base creation- Graph analytics- Information retrieval@@ -244,13 +232,11 @@ ontology.add_relationship("works_for", "Person", "Organization")is_valid = ontology.validate_graph(kg)```-**What it does:**- **Schema definition** - Define data structure- **Data validation** - Ensure data conforms to schema- **Inheritance** - Create hierarchical relationships- **Constraints** - Enforce data quality rules-**Use Cases:**- Data modeling- Quality assurance- Schema management@@ -268,13 +254,11 @@ engine = ReasoningEngine()inferences = engine.infer(kg, rules=["transitivity", "symmetry"])```-**What it does:**- **Logical inference** - Derive new facts from existing ones- **Pattern matching** - Find complex patterns in data- **Consistency checking** - Detect contradictions- **Decision support** - Automated reasoning-**Use Cases:**- Knowledge discovery- Decision making- Consistency checking@@ -295,13 +279,11 @@ embeddings = generator.generate(["text1", "text2"])similarity = generator.similarity(embeddings[0], embeddings[1])```-**What it does:**- **Text embeddings** - Convert text to vectors- **Similarity search** - Find similar content- **Clustering** - Group related items- **AI integration** - Provide context to LLMs-**Use Cases:**- Semantic search- Recommendation systems- Clustering@@ -320,13 +302,11 @@ store.add_vectors(embeddings, ids)results = store.search(query_vector, top_k=10)```-**What it does:**- **Vector storage** - Efficient vector database- **Fast search** - Approximate nearest neighbor search- **Indexing** - Optimize for performance- **Batch operations** - Handle large datasets-**Use Cases:**- Semantic search- RAG systems- Recommendation engines@@ -346,13 +326,11 @@ store.add_edges(relationships)results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m")```-**What it does:**- **Graph persistence** - Store graphs in databases- **Graph queries** - Cypher and Gremlin support- **Graph algorithms** - Path finding, centrality- **Transactions** - ACID compliance-**Use Cases:**- Knowledge graph storage- Graph analytics- Network analysis@@ -371,13 +349,11 @@ store.add_triplets(subject, predicate, object)triplets = store.get_triplets(entity="Apple Inc.")```-**What it does:**- **Triple storage** - Store (subject, predicate, object) triples- **Pattern matching** - Find specific patterns- **RDF support** - Semantic web standards- **Bulk operations** - Efficient batch processing-**Use Cases:**- Semantic web- Knowledge representation- Linked data@@ -397,13 +373,11 @@ resolver = EntityResolver()merged_entities = resolver.resolve(entities, strategy="semantic")```-**What it does:**- **Duplicate detection** - Find similar entities- **Entity resolution** - Merge duplicate records- **Similarity scoring** - Quality assessment- **Record linkage** - Connect related records-**Use Cases:**- Data cleaning- Master data management- Record linkage@@ -422,13 +396,11 @@ conflicts = detector.detect_conflicts(kg)resolved = detector.resolve(conflicts, strategy="most_recent")```-**What it does:**- **Conflict detection** - Find contradictory information- **Resolution strategies** - Automated conflict resolution- **Source reliability** - Trustworthiness assessment- **Temporal analysis** - Time-based conflict handling-**Use Cases:**- Data quality- Consistency checking- Trust management@@ -448,13 +420,11 @@ manager = ContextManager()context = manager.get_context(query, history)```-**What it does:**- **Context tracking** - Maintain conversation context- **Memory management** - Store and retrieve context- **Relevance scoring** - Find relevant context- **Session management** - Handle multiple conversations-**Use Cases:**- AI agents- Chatbots- Conversational AI@@ -472,13 +442,11 @@ seed = SeedData()knowledge = seed.get_knowledge("technology", "companies")```-**What it does:**- **Seed knowledge** - Foundation data for domains- **Knowledge bases** - Pre-built domain knowledge- **Quick start** - Bootstrap applications- **Domain models** - Industry-specific data-**Use Cases:**- Domain bootstrapping- Quick start data- Industry knowledge@@ -496,13 +464,11 @@ provider = LLMProvider(model="gpt-4")response = provider.generate(prompt, context=kg)```-**What it does:**- **LLM integration** - Connect to various LLM providers- **Prompt engineering** - Optimize prompts for results- **Context injection** - Provide knowledge graph context- **Response parsing** - Extract structured outputs-**Use Cases:**- AI generation- Question answering- Text completion@@ -522,13 +488,11 @@ exporter = GraphExporter()exporter.export(kg, format="json", filename="output.json")```-**What it does:**- **Multiple formats** - JSON, CSV, RDF, GraphML- **Database export** - Export to various databases- **Streaming** - Handle large datasets- **Filtering** - Export specific data subsets-**Use Cases:**- Data sharing- System integration- Backup and restore@@ -546,13 +510,11 @@ visualizer = GraphVisualizer()visualizer.plot(kg, layout="force_directed")```-**What it does:**- **Graph visualization** - Interactive graph plots- **Custom styling** - Tailored visual appearance- **Analytics charts** - Statistics and metrics- **Exploration tools** - Interactive data exploration-**Use Cases:**- Data exploration- Presentation- Analysis@@ -573,13 +535,11 @@ pipeline.add_step("build", GraphBuilder())result = pipeline.run("data/")```-**What it does:**- **Workflow orchestration** - Coordinate multiple steps- **Parallel processing** - Run steps concurrently- **Progress tracking** - Monitor pipeline execution- **Error handling** - Robust error management-**Use Cases:**- Data processing- Workflow automation- Batch processing@@ -587,7 +547,7 @@ result = pipeline.run("data/")----## New Features & Modules+## Additional Modules### Change Management Module**Version control and audit trails**@@ -599,13 +559,11 @@ manager = TemporalVersionManager(storage_path="versions.db")snapshot = manager.create_snapshot(kg, "v1.0", "[email protected]", "Initial version")```-**What it does:**- **Version control** - Track changes over time- **Audit trails** - Complete change history- **Data integrity** - SHA-256 checksums- **Change comparison** - Detailed diff analysis-**Use Cases:**- Knowledge graph versioning- Compliance tracking- Data governance@@ -623,13 +581,11 @@ manager = ProvenanceManager()manager.track_entity("entity_1", "document.pdf", "person")```-**What it does:**- **W3C PROV-O compliant** - Industry standard tracking- **Complete lineage** - End-to-end traceability- **Source attribution** - Track data origins- **Integrity verification** - Tamper detection-**Use Cases:**- Regulatory compliance- Data provenance- Audit trails@@ -648,13 +604,11 @@ semantica = Semantica(config=Config())result = semantica.process("data/")```-**What it does:**- **Framework orchestration** - Central coordination- **Configuration management** - Settings and preferences- **Lifecycle management** - Start/stop/restart- **Plugin system** - Extensible architecture-**Use Cases:**- Framework initialization- Configuration management- Plugin development@@ -662,46 +616,18 @@ result = semantica.process("data/")----## Getting Started+## Common Module Chains-### Quick Start Example--```python-# Complete pipeline example-from semantica.ingest import FileIngestor-from semantica.semantic_extract import NERExtractor, RelationExtractor-from semantica.kg import GraphBuilder-from semantica.pipeline import Pipeline--# Create pipeline-pipeline = Pipeline()-pipeline.add_step("ingest", FileIngestor())-pipeline.add_step("ner", NERExtractor())-pipeline.add_step("relations", RelationExtractor())-pipeline.add_step("build", GraphBuilder())--# Run pipeline-kg = pipeline.run("documents/")-print(f"Built graph with {len(kg['entities'])} entities")-```--### Choose Your Modules--**For Document Processing:**-- Ingest → Parse → Split → Semantic Extract → Knowledge Graph--**For Web Scraping:**-- Ingest (Web) → Normalize → Semantic Extract → Graph Store--**For AI Agents:**-- Context → LLM Providers → Reasoning → Export--**For Analytics:**-- Knowledge Graph → Graph Store → Visualization → Export+| Goal | Modules |+|------|---------|+| Document processing | Ingest → Parse → Split → Semantic Extract → KG |+| Web scraping | Ingest (Web) → Normalize → Semantic Extract → Graph Store |+| AI agents | Context → LLM Providers → Reasoning → Export |+| Analytics | KG → Graph Store → Visualization → Export |----## Module Reference+## Module Index| Module | Purpose | Key Classes | Use Cases ||--------|---------|-------------|-----------|@@ -731,10 +657,9 @@ print(f"Built graph with {len(kg['entities'])} entities")----## Need Help?+## More-- **Documentation**: [Getting Started](getting-started.md)-- **Examples**: [Cookbook](cookbook.md)-- **Community**: [Discord](community.md)-- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)-- **Support**: [Contact Us](community.md)+- [Getting Started](getting-started.md)+- [Examples](examples.md)+- [Cookbook](cookbook.md)+- [API Reference](reference/core.md)diff --git a/docs/quickstart.md b/docs/quickstart.mdindex f3d9b00e..9c6d5392 100644--- a/docs/quickstart.md+++ b/docs/quickstart.md@@ -1,246 +1,150 @@# Quickstart-Get started with Semantica in 5 minutes. This guide will walk you through building your first knowledge graph.+Build your first knowledge graph in 5 minutes.-!!! tip "Before You Start"- Make sure you have Semantica installed. If not, follow the [Installation Guide](installation.md) first. This quickstart assumes basic Python knowledge.+!!! tip "Prerequisites"+ Semantica installed (`pip install semantica`). If not, see the [Installation Guide](installation.md).-## Overview+---++## Pipeline Overview```mermaidflowchart LR- A[Install] --> B[Initialize]- B --> C[Load Data]- C --> D[Extract]- D --> E[Build Graph]- E --> F[Visualize]-- style A fill:#e3f2fd- style F fill:#c8e6c9+ A[Ingest] --> B[Parse]+ B --> C[Extract]+ C --> D[Build Graph]+ D --> E[Visualize / Export]```-## Step 1: Installation+---++## Step 1 — Ingest++Load documents from files, directories, or the web.-If you haven't installed Semantica yet:+```python+from semantica.ingest import FileIngestor-```bash-pip install semantica+ingestor = FileIngestor()+sources = ingestor.ingest("data/sample.pdf")```-See the [Installation Guide](installation.md) for detailed instructions.+Supported formats: PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives. For web content, use `WebIngestor`.++---++## Step 2 — Parse++Extract structured text from raw documents.-!!! note "Installation Options"- For production use, consider installing with optional dependencies for better performance: `pip install semantica[all]`. See the [Installation Guide](installation.md) for all options.+```python+from semantica.parse import DocumentParser-## Step 2: Your First Knowledge Graph+parser = DocumentParser()+parsed = parser.parse(sources[0])+```-Building a knowledge graph involves these key steps:+For complex layouts (tables, columns): use `DoclingParser` instead — it handles PDF tables and structured DOCX/PPTX better.-1. **Ingest** your documents using `FileIngestor`-2. **Parse** documents to extract text using `DocumentParser` or `DoclingParser` (for enhanced layout support)-3. **Extract** entities and relationships using `NERExtractor` and `RelationExtractor`-4. **Build** the graph using `GraphBuilder`-5. **Generate** embeddings (optional) using `TextEmbedder`+---++## Step 3 — Extract Entities and Relationships-**Quick Example:**```python-from semantica.ingest import FileIngestor-from semantica.parse import DocumentParser, DoclingParserfrom semantica.semantic_extract import NERExtractor, RelationExtractor-from semantica.kg import GraphBuilder-# 1. Ingest document-ingestor = FileIngestor()-sources = ingestor.ingest("data/sample.pdf")+ner = NERExtractor()+entities = ner.extract(parsed)-# 2. Parse (choose your parser)-# Option A: Standard parser-parser = DocumentParser()-parsed_content = parser.parse(sources[0])+rel = RelationExtractor()+relationships = rel.extract(parsed, entities=entities)+```-# Option B: Enhanced Docling parser (recommended for complex tables)-# docling_parser = DoclingParser()-# parsed_content = docling_parser.parse(sources[0])+Each entity gets a type, confidence score, and source reference. Relationships are extracted as typed triplets: `(subject, predicate, object)`.-# 3. Extract entities and relations-ner = NERExtractor()-entities = ner.extract(parsed_content)+---++## Step 4 — Build the Knowledge Graph-relations = RelationExtractor()-relationships = relations.extract(parsed_content, entities=entities)+```python+from semantica.kg import GraphBuilder-# 4. Build graph-builder = GraphBuilder()+builder = GraphBuilder(merge_entities=True)graph = builder.build(entities=entities, relationships=relationships)-print(f"Built knowledge graph with {len(graph.nodes)} nodes and {len(graph.edges)} edges")+print(f"{len(graph.nodes)} nodes, {len(graph.edges)} edges")```-**For complete step-by-step examples with detailed explanations, see:**-- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Full tutorial with detailed explanations and expected outputs- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Learning the basics, quick start--## Step 3: Extract Entities and Relationships--The semantic extraction step identifies named entities (people, organizations, locations) and relationships between them from your text.--**What gets extracted:**-- **Entities**: People, organizations, locations, dates, and other named entities-- **Relationships**: Connections between entities (e.g., `founded_by`, `located_in`, `has_ceo`)--**For detailed examples and different extraction methods, see:**-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn different NER methods and configurations- - **Topics**: Named entity recognition, entity types, confidence scores- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Understanding entity extraction options--- **[Relation Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)**: Learn to extract relationships between entities- - **Topics**: Relationship extraction, dependency parsing, semantic role labeling- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Building rich knowledge graphs with relationships--## Step 4: Build Knowledge Graph from Multiple Sources--You can combine data from multiple sources (files, web, databases) to build a unified knowledge graph. The process involves:--1. **Ingest** from multiple sources using different ingestors-2. **Parse** all documents to extract text-3. **Extract** entities and relationships from each source-4. **Build** a unified graph with entity merging enabled--**For complete examples with multiple sources, see:**-- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from files, web, feeds, streams, and databases- - **Topics**: File, web, feed, stream, database ingestion- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Loading data from various sources--- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced patterns for integrating multiple data sources- - **Topics**: Multi-source integration, entity resolution, conflict handling- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Building knowledge graphs from diverse data sources--## Step 5: Visualize Your Knowledge Graph--Visualization helps you understand and explore your knowledge graph structure. Semantica supports multiple visualization formats including interactive HTML, static images, and export formats.--**For detailed visualization examples, see:**-- **[Visualization Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)**: Learn to create interactive and static visualizations- - **Topics**: Network graphs, interactive HTML, static images, export formats- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Exploring graph structure, presentations, analysis--- **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization techniques- - **Topics**: Custom layouts, filtering, styling, multiple graph types- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Production visualizations, custom dashboards--## Step 6: Export Your Knowledge Graph--Export your knowledge graph to various formats for integration with other systems or tools. Semantica supports RDF, JSON, CSV, OWL, GraphML, and more.--**Supported export formats:**-- **RDF**: Turtle, RDF/XML, JSON-LD, N-Triples-- **JSON**: Standard JSON, JSON-LD, Cytoscape.js format-- **CSV**: Node and edge lists for spreadsheet tools-- **OWL**: OWL/XML and Turtle for ontologies-- **Graph Formats**: GraphML, GEXF, DOT for visualization tools--**For detailed export examples, see:**-- **[Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)**: Learn to export to all supported formats- - **Topics**: RDF, JSON, CSV, OWL, GraphML export- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Data integration, sharing knowledge graphs--- **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns- - **Topics**: Batch export, custom formats, format conversion- - **Difficulty**: Intermediate- - **Time**: 30-45 minutes- - **Use Cases**: Production exports, format migration+`merge_entities=True` resolves duplicates across sources automatically.-## Common Patterns+----### Pattern 1: Process Text Directly+## Step 5 — Visualize-You can process text directly without file ingestion. This is useful when you already have text content in memory.+```python+from semantica.visualization import GraphVisualizer-**For examples, see:**-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Processing text directly-- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction from text+viz = GraphVisualizer()+viz.visualize(graph, output="graph.html") # interactive HTML+```-### Pattern 2: Custom Entity Extraction+----Configure entity extraction with different methods (ML models, LLMs) and parameters for your specific needs.+## Step 6 — Export-**For examples, see:**-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Different extraction methods and configurations-- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns+```python+from semantica.export import RDFExporter-### Pattern 3: Incremental Building+exporter = RDFExporter()+rdf = exporter.export_to_rdf(graph, format="turtle")+```-Build knowledge graphs incrementally from multiple sources and merge them together.+Other formats: `"json-ld"`, `"nt"`, `"xml"`, Parquet, ArangoDB AQL. See [Export Reference](reference/export.md).-**For examples, see:**-- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction and merging-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns+----## Next Steps+## Common Patterns-Now that you've built your first knowledge graph:+### Process text directly (no file)-1. **[Explore Examples](examples.md)** - See more advanced use cases-2. **[API Reference](reference/core.md)** - Learn about all available methods-3. **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks-4. **[Full Documentation](https://github.com/Hawksight-AI/semantica/blob/main/README.md)** - Comprehensive guide+```python+from semantica.semantic_extract import NERExtractor-### 🍳 Recommended Cookbook Tutorials+ner = NERExtractor()+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")+```-Continue learning with these interactive tutorials:+### Incremental build from multiple sources-- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules- - **Topics**: Framework overview, all modules, architecture, configuration- - **Difficulty**: Beginner- - **Time**: 30-45 minutes- - **Use Cases**: Understanding the complete framework+```python+from semantica.kg import GraphBuilder-- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization- - **Difficulty**: Beginner- - **Time**: 20-30 minutes- - **Use Cases**: Hands-on practice with complete workflow+all_entities, all_rels = [], []+for doc in parsed_docs:+ all_entities.extend(ner.extract(doc))+ all_rels.extend(rel.extract(doc, entities=all_entities))-- **[Data Ingestion](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources- - **Topics**: File, web, feed, stream, database ingestion- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Loading data from various sources+graph = GraphBuilder(merge_entities=True).build(+ entities=all_entities, relationships=all_rels+)+```-- **[Document Parsing](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)**: Parse various document formats- - **Topics**: PDF, DOCX, HTML, JSON parsing- - **Difficulty**: Beginner- - **Time**: 15-20 minutes- - **Use Cases**: Extracting text from different file formats+---## Troubleshooting-### Common Issues+| Problem | Fix |+|---------|-----|+| No entities extracted | Check the document has machine-readable text (not just scanned images) |+| Slow processing | Process in chunks; use GPU acceleration (`pip install semantica[gpu]`) |+| Memory errors | Reduce batch size or switch to a persistent graph backend |-**Issue**: No entities extracted-- **Solution**: Check that your document contains text content. PDFs with images only won't work without OCR.+----**Issue**: Slow processing-- **Solution**: For large documents, consider processing in chunks or using GPU acceleration.--**Issue**: Memory errors-- **Solution**: Process documents one at a time or reduce batch sizes.+## Next Steps-Need help? Check the [Installation Troubleshooting](installation.md#troubleshooting) or [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues).+- [Core Concepts](concepts.md) — understand how knowledge graphs and reasoning work+- [Modules Guide](modules.md) — every module explained+- [Use Cases](use-cases.md) — domain-specific examples+- [Cookbook](cookbook.md) — interactive Jupyter notebooks for each stepdiff --git a/docs/reference/change_management.md b/docs/reference/change_management.mdindex 7fb9a1a4..47d40499 100644--- a/docs/reference/change_management.md+++ b/docs/reference/change_management.md@@ -58,16 +58,14 @@ The Semantica change management module provides enterprise-grade version control</div>-### Key Features--- ✅ **Enterprise Version Control** — Complete snapshot management with SHA-256 integrity verification-- ✅ **Dual Storage Backends** — InMemory (development) and SQLite (production) with ACID guarantees-- ✅ **Knowledge Graph Versioning** — Entity and relationship-level change tracking with detailed diffs-- ✅ **Ontology Versioning** — Structural change tracking for classes, properties, and axioms-- ✅ **Audit Trail Compliance** — Complete change logs with author attribution and timestamps-- ✅ **Data Integrity** — SHA-256 checksums for tamper detection and verification-- ✅ **Change Comparison** — Detailed diff algorithms for entities, relationships, and ontology structures-- ✅ **Backward Compatibility** — Legacy support for existing ontology version management+- **Enterprise Version Control** — Complete snapshot management with SHA-256 integrity verification+- **Dual Storage Backends** — InMemory (development) and SQLite (production) with ACID guarantees+- **Knowledge Graph Versioning** — Entity and relationship-level change tracking with detailed diffs+- **Ontology Versioning** — Structural change tracking for classes, properties, and axioms+- **Audit Trail Compliance** — Complete change logs with author attribution and timestamps+- **Data Integrity** — SHA-256 checksums for tamper detection and verification+- **Change Comparison** — Detailed diff algorithms for entities, relationships, and ontology structures+- **Backward Compatibility** — Legacy support for existing ontology version management---@@ -121,24 +119,21 @@ class ChangeLogEntry:### Storage Backends-#### InMemoryVersionStorage-Fast, volatile storage for development and testing.+**InMemoryVersionStorage** — Fast, volatile storage for development and testing.```pythonfrom semantica.change_management import InMemoryVersionStoragestorage = InMemoryVersionStorage()```-#### SQLiteVersionStorage-Persistent storage with ACID guarantees for production.+**SQLiteVersionStorage** — Persistent storage with ACID guarantees for production.```pythonfrom semantica.change_management import SQLiteVersionStoragestorage = SQLiteVersionStorage("versions.db")```-#### VersionStorage (Abstract)-Base interface for custom storage implementations.+**VersionStorage (Abstract)** — Base interface for custom storage implementations.**Core Methods:**- `save(snapshot)` - Store version snapshot@@ -251,7 +246,7 @@ For large-scale knowledge graphs, reprocessing the entire dataset on every updatSemantica supports **Delta-Aware Pipelines**, allowing you to compute the exact differences (added and removed triples)between the two graph snapshots and run validation, enrichment, or export jobs *only* on the changes.-### Delta Pipeline Example+**Delta Pipeline Example**```pythonfrom semantica.change_management import TemporalVersionManager@@ -286,6 +281,7 @@ result = engine.execute_pipeline(triplet_store=triplet_store)```+---## Data Integrity@@ -399,7 +395,7 @@ for version in prod_manager.list_versions():Semantica allows you to treat ontology schema changes with the same rigor as database migrations. By comparing two versions, you can generate a machine-readable diff and a structured impact report to catch breaking changes before they reach production.-### Comparing Versions+**Comparing Versions**The `OntologyEngine` provides a high-level API to orchestrate the comparison of two schema versions.@@ -410,21 +406,18 @@ engine = OntologyEngine()# Generate a migration impact report between v1.0 and v2.0report = engine.compare_versions(- base_id="v1.0",+ base_id="v1.0",target_id="v2.0")print(f"Total changes detected: {report['summary']['total_changes']}")```------### Understanding the Report Format--The `compare_versions` method returns a comprehensive dictionary containing both a machine-readable diff and a human-readable impact analysis.+---+**Report Format**-Here is the exact structure of the returned report:+The `compare_versions` method returns a dictionary with a machine-readable diff and a human-readable impact analysis:```json{@@ -467,3 +460,4 @@ Here is the exact structure of the returned report:"warnings": []}}+```diff --git a/docs/use-cases.md b/docs/use-cases.mdindex 9ad0013c..2d9c4ba2 100644--- a/docs/use-cases.md+++ b/docs/use-cases.md@@ -1,210 +1,125 @@# Use Cases-Semantica is designed to solve complex data challenges across various domains. This guide explores common use cases and how to implement them.--!!! info "About This Guide"- This guide provides detailed implementation guides for real-world use cases, complete with code examples, prerequisites, and step-by-step instructions.+Real-world applications of Semantica across domains, with linked cookbook notebooks for each.----## Use Case Comparison--| Use Case | Difficulty | Time | Domain | Key Features | Cookbook |-| :-------------------------------- | :------------ | :---------- | :---------- | :---------------------------------------------- | :------------------------------------------ |-| **Biomedical Knowledge Graphs** | Intermediate | 1-2 hours | Healthcare | Gene-protein-disease relationships | Drug Discovery, Genomic Variant Analysis |-| **Financial Data Integration** | Intermediate | 1-2 hours | Finance | MCP integration, real-time data | Financial Data Integration MCP |-| **Fraud Detection** | Advanced | 2-3 hours | Finance | Temporal graphs, pattern detection | Fraud Detection |-| **Blockchain Analytics** | Intermediate | 1-2 hours | Finance | Transaction tracing, DeFi intelligence | DeFi Protocol Intelligence, Transaction Network |-| **Cybersecurity Threat Intelligence**| Advanced | 2-3 hours | Security | Threat mapping, anomaly detection | Real-Time Anomaly Detection, Threat Intelligence |-| **Intelligence Analysis** | Intermediate | 1-2 hours | Security | Criminal networks, OSINT analysis | Criminal Network Analysis, Intelligence Orchestrator |-| **Supply Chain Optimization** | Intermediate | 1-2 hours | Industry | Data integration, route optimization | Supply Chain Data Integration |-| **Renewable Energy Management** | Intermediate | 1-2 hours | Energy | Energy market analysis, optimization | Energy Market Analysis |-| **GraphRAG** | Advanced | 1-2 hours | AI | Enhanced RAG with knowledge graphs | GraphRAG Complete, RAG vs GraphRAG |--**Difficulty Levels**:-- **Beginner**: Basic Semantica knowledge required-- **Intermediate**: Some domain knowledge helpful-- **Advanced**: Requires domain expertise and advanced Semantica features+## Overview++| Use Case | Domain | Difficulty | Estimated Time |+|----------|--------|------------|----------------|+| Biomedical Knowledge Graphs | Healthcare | Intermediate | 1–2 hours |+| Financial Data Integration | Finance | Intermediate | 1–2 hours |+| Fraud Detection | Finance | Advanced | 2–3 hours |+| Blockchain Analytics | Finance | Intermediate | 1–2 hours |+| Cybersecurity Threat Intelligence | Security | Advanced | 2–3 hours |+| Criminal Network Analysis | Security / Intelligence | Intermediate | 1–2 hours |+| Intelligence Analysis Orchestrator | Intelligence | Intermediate | 1–2 hours |+| Supply Chain Optimization | Operations | Intermediate | 1–2 hours |+| Renewable Energy Management | Energy | Intermediate | 1–2 hours |+| GraphRAG | AI | Advanced | 1–2 hours |++**Difficulty levels**:++- **Beginner** — basic Semantica knowledge only+- **Intermediate** — some domain knowledge helpful+- **Advanced** — domain expertise + advanced Semantica features---## Research & Science-<div class="grid cards" markdown>+### Biomedical Knowledge Graphs++Connect genes, proteins, drugs, and diseases from scientific literature and databases to accelerate drug discovery and understand disease pathways.-- :material-dna: **Biomedical Knowledge Graphs**- ---- Accelerate drug discovery and understand disease pathways by connecting genes, proteins, drugs, and diseases.-- **Goal**: Connect genes, proteins, drugs, and diseases from scientific literature and databases.-- **Difficulty**: Intermediate-- [:material-arrow-right: Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)-- [:material-arrow-right: Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)+**Cookbooks**:+- [Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search+- [Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — bioRxiv RSS, temporal KGs, deduplication, pathway analysis-</div>+----### Biomedical Knowledge Graphs Implementation+## Finance & Trading-**Prerequisites**:-- Domain knowledge of biomedical concepts-- Access to biomedical literature/databases+### Financial Data Integration-**Implementation Guides:**+Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph.-- **[Drug Discovery Pipeline Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)**: Build knowledge graphs from PubMed RSS feeds- - **Topics**: PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search- - **Difficulty**: Intermediate- - **Time**: 1-2 hours- - **Use Cases**: Drug discovery, biomedical research+**Cookbook**: [Financial Data Integration (MCP)](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Alpha Vantage API, MCP servers, seed data, real-time ingestion-- **[Genomic Variant Analysis Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)**: Analyze genomic variants using temporal knowledge graphs- - **Topics**: bioRxiv RSS, temporal KGs, deduplication, pathway analysis- - **Difficulty**: Intermediate- - **Time**: 1-2 hours- - **Use Cases**: Genomic research, variant analysis+### Fraud Detection----+Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data.-## Finance & Trading+**Cookbook**: [Fraud Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — temporal KGs, cycle detection, fraud pattern analysis-<div class="grid cards" markdown>--- :material-finance: **Financial Data Integration**- ---- Integrate financial data from multiple sources using MCP servers and real-time ingestion.-- **Goal**: Connect Alpha Vantage API, MCP servers, seed data, and real-time ingestion for comprehensive financial analysis.-- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb)--- :material-shield-alert: **Fraud Detection**- ---- Detect complex fraud rings using temporal knowledge graphs and pattern detection.-- **Goal**: Build a graph of Users, Devices, IP Addresses, and Transactions to find cycles and detect fraud patterns.-- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb)--- :material-bitcoin: **Blockchain Analytics**- ---- Analyze DeFi protocols and transaction networks for intelligence and fraud detection.-- **Goal**: Map transaction flows between wallets and exchanges, analyze DeFi protocols, and detect illicit activity.-- [:material-arrow-right: DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)-- [:material-arrow-right: Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)--</div>+### Blockchain Analytics----+Map transaction flows, analyze DeFi protocols, and detect illicit activity across wallet and exchange networks.+**Cookbooks**:+- [DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)+- [Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)---## Security & Intelligence-<div class="grid cards" markdown>--- :material-shield-lock: **Cybersecurity Threat Intelligence**- ---- Proactively identify and mitigate cyber threats using real-time anomaly detection and threat intelligence.-- **Goal**: Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs.-- [:material-arrow-right: Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)-- [:material-arrow-right: Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)--- :material-account-network: **Criminal Network Analysis**- ---- Analyze criminal networks to identify key players, communities, and suspicious patterns using OSINT RSS feeds, deduplication, and network centrality analysis.-- **Goal**: Build knowledge graphs from police reports, court records, and surveillance data.-- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)--- :material-file-search: **Intelligence Analysis Orchestrator Worker**- ---- Comprehensive intelligence analysis using pipeline orchestrator with multiple RSS feeds, conflict detection, and multi-source integration.-- **Goal**: Process multiple intelligence sources in parallel using orchestrator-worker pattern.-- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb)--</div>+### Cybersecurity Threat Intelligence----+Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs for proactive defense.-## Industry & Operations+**Cookbooks**:+- [Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)+- [Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)-<div class="grid cards" markdown>+### Criminal Network Analysis-- :material-truck-delivery: **Supply Chain Optimization**- ---- Visualize and optimize complex global supply chains.-- **Goal**: Map suppliers, logistics routes, and inventory levels to identify bottlenecks.-- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)+Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns using network centrality analysis.-- :material-wind-turbine: **Renewable Energy Management**- ---- Optimize grid operations and asset maintenance.-- **Goal**: Connect sensor data, weather forecasts, and maintenance logs to predict failures.-- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)+**Cookbook**: [Criminal Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)-</div>+### Intelligence Analysis Orchestrator++Process multiple intelligence sources in parallel using an orchestrator-worker pipeline pattern with multi-source conflict detection and integration.++**Cookbook**: [Intelligence Analysis Orchestrator-Worker](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb)----## Advanced AI Patterns+## Industry & Operations-<div class="grid cards" markdown>+### Supply Chain Optimization-- :material-robot: **Graph-Augmented Generation (GraphRAG)**- ---- Enhance LLM responses with structured ground truth using knowledge graphs.-- **Goal**: Use the knowledge graph to retrieve precise context for RAG applications with hybrid retrieval and logical inference.-- [:material-arrow-right: GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)-- [:material-scale-balance: RAG vs GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)+Map suppliers, logistics routes, and inventory levels to identify bottlenecks and optimize global supply chains.-</div>+**Cookbook**: [Supply Chain Data Integration](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)----+### Renewable Energy Management++Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations.+**Cookbook**: [Energy Market Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)----## Summary+## Advanced AI Patterns++### GraphRAG (Graph-Augmented Generation)-This guide covered use cases across multiple domains with corresponding cookbooks:+Use knowledge graphs to retrieve precise, structured context for LLM responses — with hybrid retrieval, logical inference, and source attribution.-- **Research & Science**: Biomedical knowledge graphs (Drug Discovery, Genomic Variant Analysis)-- **Finance & Trading**: Financial data integration, fraud detection, blockchain analytics-- **Security & Intelligence**: Cybersecurity threat intelligence, criminal network analysis, intelligence orchestration-- **Industry**: Supply chain optimization, renewable energy management-- **AI Applications**: GraphRAG (Complete implementation and comparison)+**Cookbooks**:+- [GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — production-ready implementation+- [RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — side-by-side comparison---## Next Steps-- **[Examples](examples.md)** - More detailed code examples-- **[Modules Guide](modules.md)** - Learn about available modules-- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks-- **[API Reference](reference/core.md)** - Complete API documentation-------!!! info "Contribute"- Have a use case to add? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)+- [Cookbook](cookbook.md) — full notebook catalog organized by topic and difficulty+- [Modules Guide](modules.md) — every module with examples+- [API Reference](reference/core.md) — complete technical documentation+!!! info "Have a use case to add?"+ [Open a PR](https://github.com/Hawksight-AI/semantica) or start a discussion on GitHub.diff --git a/mkdocs_local.yml b/mkdocs_local.ymlnew file mode 100644index 00000000..26261fc3--- /dev/null+++ b/mkdocs_local.yml@@ -0,0 +1,14 @@+INHERIT: mkdocs.yml+plugins:+ - search:+ lang: en+ - minify:+ minify_html: true+ - mkdocstrings:+ handlers:+ python:+ options:+ docstring_style: google+ show_source: true+ show_root_heading: true+ show_category_heading: true