Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
Utlis Update Readme (#369)
Mohd Kaifa4caafbb6d65
diff --git a/README.md b/README.mdindex d3e446d2..be25236f 100644--- a/README.md+++ b/README.md@@ -1,15 +1,18 @@<div align="center">-<img src="Semantica Logo.png" alt="Semantica Logo" width="460"/>+<img src="Semantica Logo.png" alt="Semantica Logo" width="420"/># π§ Semantica-### Open-Source Semantic Layer & Context Graph Framework++**A Framework for Building Context Graphs and Decision Intelligence Layers for AI**[](https://www.python.org/)[](https://opensource.org/licenses/MIT)[](https://pypi.org/project/semantica/)[](https://pepy.tech/project/semantica)[](https://github.com/Hawksight-AI/semantica/actions)+[](https://discord.gg/sV34vps5hH)+[](https://x.com/BuildSemantica)[](https://discord.gg/sV34vps5hH)[](https://x.com/BuildSemantica)@@ -21,71 +24,170 @@----## π Why Semantica?+## The Problem-**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy.+AI agents today are capable but not trustworthy:-Perfect for **high-stakes domains** where mistakes have real consequences.+- **No memory structure** β agents store embeddings, not meaning. Retrieval is fuzzy; there's no way to ask *why* something was recalled.+- **No decision trail** β agents make decisions continuously but record nothing. When something goes wrong, there's no history to debug or audit.+- **No provenance** β outputs cannot be traced back to source facts. In regulated industries, this is a compliance blocker.+- **No reasoning transparency** β black-box answers with no explanation of how a conclusion was reached.+- **No conflict detection** β contradictory facts silently coexist in vector stores, producing unpredictable answers.----+These aren't edge cases. They are the reason AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch.++## The Solution-### β‘ Get Started in 30 Seconds+Semantica is the **context and intelligence layer** you add to your AI stack:++- **Context Graphs** β structured graph of entities, relationships, and decisions your agent builds as it works. Queryable, traceable, persistent.+- **Decision Intelligence** β every decision is a first-class object: recorded, linked causally, searchable by precedent, and analyzable for downstream impact.+- **Provenance** β every fact links to its source. W3C PROV-O compliant. Full lineage from ingestion to inference.+- **Reasoning engines** β forward chaining, Rete networks, deductive, abductive, and SPARQL reasoning. Explainable inference paths, not black-box answers.+- **Deduplication & QA** β conflict detection, entity resolution, and validation built into the pipeline.++Works alongside LangChain, LlamaIndex, AutoGen, CrewAI, and any LLM provider β Semantica is not a replacement, it's the accountability layer on top.```bashpip install semantica```+---++## Features++### Context & Decision Intelligence+- **Context Graphs** β structured graph of entities, relationships, and decisions; queryable, causal, persistent+- **Decision tracking** β record, link, and analyze every agent decision with `add_decision()`, `record_decision()`+- **Causal chains** β link decisions with `add_causal_relationship()`, trace lineage with `trace_decision_chain()`+- **Precedent search** β hybrid similarity search over past decisions with `find_similar_decisions()`+- **Influence analysis** β `analyze_decision_impact()`, `analyze_decision_influence()` β understand downstream effects+- **Policy engine** β enforce business rules with `check_decision_rules()`; automated compliance validation+- **Agent memory** β `AgentMemory` with short/long-term storage, conversation history, and statistics+- **Cross-system context capture** β `capture_cross_system_inputs()` for multi-agent pipelines++### Knowledge Graphs+- **Knowledge graph construction** β entities, relationships, properties, typed edges+- **Graph algorithms** β PageRank, betweenness centrality, clustering coefficient, community detection+- **Node embeddings** β Node2Vec embeddings via `NodeEmbedder`+- **Similarity** β cosine similarity via `SimilarityCalculator`+- **Link prediction** β score potential new edges via `LinkPredictor`+- **Temporal graphs** β time-aware nodes and edges+- **Incremental / delta processing** β update graphs without full recompute++### Semantic Extraction+- **Entity extraction** β named entity recognition, normalization, classification+- **Relation extraction** β triplet generation from raw text using LLMs or rule-based methods+- **LLM-typed extraction** β extraction with typed relation metadata+- **Deduplication v1** β Jaro-Winkler similarity, basic blocking+- **Deduplication v2** β `blocking_v2`, `hybrid_v2`, `semantic_v2` strategies with `max_candidates_per_entity`+- **Triplet deduplication** β `dedup_triplets()` for removing duplicate (subject, predicate, object) triples++### Reasoning Engines+- **Forward chaining** β `Reasoner` with IF/THEN string rules and dict facts+- **Rete network** β `ReteEngine` for high-throughput production rule matching+- **Deductive reasoning** β `DeductiveReasoner` for classical inference+- **Abductive reasoning** β `AbductiveReasoner` for hypothesis generation from observations+- **SPARQL reasoning** β `SPARQLReasoner` for query-based inference over RDF graphs++### Provenance & Auditability+- **Entity provenance** β `ProvenanceTracker.track_entity(id, source_url, metadata)`+- **Algorithm provenance** β `AlgorithmTrackerWithProvenance` tracks computation lineage+- **Graph builder provenance** β `GraphBuilderWithProvenance` records entity source lineage from URLs+- **W3C PROV-O compliant** β lineage tracking across all modules+- **Change management** β version control with checksums, audit trails, compliance support++### Vector Store+- **Backends** β FAISS, in-memory+- **Semantic search** β top-k retrieval by embedding similarity+- **Hybrid search** β vector + keyword with configurable weights+- **Filtered search** β metadata-based filtering on any field+- **Custom similarity weights** β tune retrieval per use case++### π Graph Database Support+- **AWS Neptune** β Amazon Neptune graph database with IAM authentication+- **Apache AGE** β PostgreSQL graph extension with openCypher via SQL+- **FalkorDB** β native support; `DecisionQuery` and `CausalChainAnalyzer` work directly with FalkorDB row/header shapes++### Data Ingestion+- **File formats** β PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives+- **Web crawl** β `WebIngestor` with configurable depth+- **Databases** β `DBIngestor` with SQL query support+- **Docling** β advanced document parsing with table and layout extraction (PDF, DOCX, PPTX, XLSX)+- **Media** β image OCR, audio/video metadata extraction++### Export Formats+- **RDF** β Turtle (`.ttl`), JSON-LD, N-Triples (`.nt`), XML via `RDFExporter`+- **Parquet** β `ParquetExporter` for entities, relationships, and full KG export+- **ArangoDB AQL** β ready-to-run INSERT statements via `ArangoAQLExporter`+- **OWL ontologies** β export generated ontologies in Turtle or RDF/XML++### Pipeline & Production+- **Pipeline builder** β `PipelineBuilder` with stage chaining and parallel workers+- **Validation** β `PipelineValidator` returns `ValidationResult(valid, errors, warnings)` before execution+- **Failure handling** β `FailureHandler` with `RetryPolicy` and `RetryStrategy` (exponential backoff, fixed, etc.)+- **Parallel processing** β configurable worker count per pipeline stage+- **LLM providers** β 100+ models via LiteLLM (OpenAI, Anthropic, Cohere, Mistral, Ollama, and more)++### Ontology+- **Auto-generation** β derive OWL ontologies from knowledge graphs via `OntologyGenerator`+- **Import** β load existing OWL, RDF, Turtle, JSON-LD ontologies via `OntologyImporter`+- **Validation** β HermiT/Pellet compatible consistency checking++---++## Modules++| Module | What it provides |+|---|---|+| `semantica.context` | Context graphs, agent memory, decision tracking, policy engine, causal chains |+| `semantica.kg` | Knowledge graphs, graph algorithms, embeddings, link prediction, provenance |+| `semantica.semantic_extract` | Entity & relation extraction, deduplication, triplet generation |+| `semantica.reasoning` | Forward chaining, Rete, deductive, abductive, SPARQL reasoning |+| `semantica.vector_store` | FAISS, in-memory, hybrid search with custom similarity weights |+| `semantica.export` | RDF (Turtle/JSON-LD/N-Triples/XML), Parquet, ArangoDB AQL |+| `semantica.ingest` | Files (PDF, DOCX, CSV, HTML), web crawl, databases, Docling |+| `semantica.ontology` | OWL generation, import (OWL/RDF/Turtle/JSON-LD), validation |+| `semantica.pipeline` | Parallel workers, validation, failure handling, retry policies |++---++## β‘ Quick Start+```python+import semanticafrom semantica.context import AgentContext, ContextGraphfrom semantica.vector_store import VectorStore-# Initialize with enhanced context features-vs = VectorStore(backend="faiss", dimension=768)-kg = ContextGraph(advanced_analytics=True)+# Build an agent with structured contextcontext = AgentContext(- vector_store=vs,- knowledge_graph=kg,+ vector_store=VectorStore(backend="faiss", dimension=768),+ knowledge_graph=ContextGraph(advanced_analytics=True),decision_tracking=True,- advanced_analytics=True,kg_algorithms=True,- vector_store_features=True,- graph_expansion=True)-# Store memory with automatic context graph building+# Store memorymemory_id = context.store(- "User is working on a React project with FastAPI",- conversation_id="session_1"-)--# Easy decision recording with convenience methods-decision_id = context.graph_builder.add_decision(- category="technology_choice",- scenario="Framework selection for web API",- reasoning="React ecosystem with FastAPI provides best performance",- outcome="selected_fastapi",- confidence=0.92+ "GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%",+ conversation_id="research_session_1",)-# Find similar decisions with advanced analytics-similar_decisions = context.graph_builder.find_similar_decisions(- scenario="Framework selection",- max_results=5+# Record a decision with full context+decision_id = context.record_decision(+ category="model_selection",+ scenario="Choose LLM for production reasoning pipeline",+ reasoning="GPT-4 benchmark advantage justifies 3x cost increase",+ outcome="selected_gpt4",+ confidence=0.91,+ entities=["gpt4", "gpt35", "reasoning_pipeline"],)-# Analyze decision impact and influence-impact = context.graph_builder.analyze_decision_impact(decision_id)+# Find similar decisions from history+precedents = context.find_precedents("model selection reasoning", limit=5)-# Check compliance with business rules-compliance = context.graph_builder.check_decision_rules({- "category": "technology_choice",- "confidence": 0.92-})--print(f"Memory stored: {memory_id}")-print(f"Decision recorded: {decision_id}")-print(f"Found {len(similar_decisions)} similar decisions")-print(f"Compliance check: {compliance.get('compliant', False)}")+# Analyze downstream influence of this decision+influence = context.analyze_decision_influence(decision_id)```**[π Full Quick Start](#-quick-start)** β’ **[π³ Cookbook Examples](#-semantica-cookbook)** β’ **[π¬ Join Discord](https://discord.gg/sV34vps5hH)** β’ **[β Star Us](https://github.com/Hawksight-AI/semantica)**@@ -154,1127 +256,375 @@ print(f"Compliance check: {compliance.get('compliant', False)}")----## π§ Context Module: Advanced Context Engineering & Decision Intelligence--The **Context Module** is Semantica's flagship component, providing sophisticated context management with **context graphs**, **advanced decision tracking**, **knowledge graph analytics**, and **easy-to-use interfaces**.--### π― Core Capabilities--| **Feature** | **Description** | **Use Case** |-|------------|-------------|------------|-| **Context Graphs** | Structured knowledge representation with entity relationships | Knowledge management, decision support |-| **Advanced Decision Tracking** | Complete decision lifecycle with precedent search, causal analysis, and policy enforcement | Banking approvals, healthcare decisions |-| **Easy-to-Use Methods** | 10 convenience methods for common operations without complexity | Rapid development, user-friendly API |-| **KG Algorithms** | Advanced graph analytics (centrality, community detection, Node2Vec) | Influence analysis, similarity search |-| **Policy Engine** | Automated compliance checking with business rules and exception handling | Regulatory compliance, business rules |-| **Vector Store Integration** | Hybrid search with custom similarity weights | Advanced retrieval and filtering |-| **Memory Management** | Hierarchical memory with short-term and long-term storage | Agent conversation history |+## Context Graphs & Decision Tracking-### π Enhanced Features--- **Easy Decision Recording**: `add_decision()` with automatic entity linking-- **Smart Precedent Search**: `find_similar_decisions()` with hybrid similarity-- **Impact Analysis**: `analyze_decision_impact()` with influence scoring-- **Policy Compliance**: `check_decision_rules()` with automated validation-- **Causal Chains**: `trace_decision_chain()` for decision lineage-- **Graph Analytics**: `get_node_importance()`, `analyze_connections()` for insights-- **Hybrid Retrieval**: Combines vector search, graph traversal, and keyword matching-- **Multi-Hop Reasoning**: Trace relationships across multiple graph hops-- **Production Ready**: Comprehensive error handling and scalability--### π§ Easy-to-Use API+Semantica's flagship module. Tracks every decision your agent makes as a structured graph node β with causal links, precedent search, impact analysis, and policy enforcement.```python-# Simple usage with convenience methodsfrom semantica.context import ContextGraphgraph = ContextGraph(advanced_analytics=True)-# Add decision with ease-decision_id = graph.add_decision(+# Record a loan approval decision+loan_id = graph.add_decision(category="loan_approval",- scenario="Mortgage application",- reasoning="Good credit score",+ scenario="Mortgage application β 780 credit score, 28% DTI",+ reasoning="Strong credit history, stable income for 8 years, low DTI",outcome="approved",- confidence=0.95+ confidence=0.95,)-# Find similar decisions-similar = graph.find_similar_decisions("mortgage", max_results=5)+# Record a downstream dependent decision+rate_id = graph.add_decision(+ category="interest_rate",+ scenario="Set rate for approved mortgage",+ reasoning="Prime applicant qualifies for lowest tier rate",+ outcome="rate_set_6.2pct",+ confidence=0.98,+)-# Analyze impact-impact = graph.analyze_decision_impact(decision_id)+# Link the decisions causally+graph.add_causal_relationship(loan_id, rate_id, relationship_type="enables")-# Check compliance-compliance = graph.check_decision_rules({- "category": "loan_approval",- "confidence": 0.95-})+# Find similar past decisions using hybrid similarity+similar = graph.find_similar_decisions("mortgage approval", max_results=5)+chain = graph.trace_decision_chain(loan_id)+impact = graph.analyze_decision_impact(loan_id)+compliance = graph.check_decision_rules({"category": "loan_approval", "confidence": 0.95})+insights = graph.get_decision_insights()```-### π’ Enterprise Integration-```python-# Full enterprise setup with AgentContext-from semantica.context import AgentContext+from semantica.context import AgentContext, AgentMemoryfrom semantica.vector_store import VectorStorecontext = AgentContext(- vector_store=VectorStore(backend="faiss"),+ vector_store=VectorStore(backend="faiss", dimension=768),knowledge_graph=ContextGraph(advanced_analytics=True),decision_tracking=True,+ graph_expansion=True,kg_algorithms=True,- vector_store_features=True)-# Record decision with full context-decision_id = context.record_decision(- category="fraud_detection",- scenario="Suspicious transaction pattern",- reasoning="Multiple high-value transactions in short timeframe",- outcome="flagged_for_review",- confidence=0.87,- entities=["transaction_123", "customer_456"]-)--# Advanced precedent search with KG features-precedents = context.find_precedents(- "suspicious transaction",- category="fraud_detection",- use_kg_features=True-)+context.store("Regulation EU 2024/1689 requires explainability for high-risk AI", conversation_id="compliance_review")+context.store("Our fraud model flags 0.3% of transactions", conversation_id="compliance_review")-# Comprehensive influence analysis-influence = context.analyze_decision_influence(decision_id)+results = context.retrieve("AI regulation explainability requirements", limit=3)+history = context.get_conversation_history("compliance_review")+stats = context.get_statistics()```----## AgentContext - Your Agent's Brain--The main interface that makes your agent intelligent. It handles memory, decisions, and knowledge organization automatically.+## Knowledge Graphs-### Quick Start```python-from semantica.context import AgentContext-from semantica.vector_store import VectorStore+from semantica.kg import KnowledgeGraph, Entity, Relationship+from semantica.kg import CentralityAnalyzer, NodeEmbedder, LinkPredictor-# Create your intelligent agent-agent = AgentContext(vector_store=VectorStore(backend="inmemory", dimension=384))+kg = KnowledgeGraph()-# Your agent can now remember things-memory_id = agent.store("User asked about Python programming")-print(f"Agent remembered: {memory_id}")+kg.add_entity(Entity(id="transformer", label="Transformer", type="Architecture",+ properties={"year": 2017, "paper": "Attention Is All You Need"}))+kg.add_entity(Entity(id="bert", label="BERT", type="Model",+ properties={"year": 2018, "parameters": "340M"}))+kg.add_entity(Entity(id="gpt4", label="GPT-4", type="Model", properties={"year": 2023}))-# And find information when needed-results = agent.retrieve("Python tutorials")-print(f"Agent found {len(results)} relevant memories")-```+kg.add_relationship(Relationship(source="bert", target="transformer", type="based_on"))+kg.add_relationship(Relationship(source="gpt4", target="transformer", type="based_on"))-### Easy Decision Learning-```python-# Your agent learns from its decisions-decision_id = agent.record_decision(- category="content_recommendation",- scenario="User wants Python tutorial",- reasoning="User mentioned being a beginner",- outcome="recommended_basics",- confidence=0.85-)+# Graph algorithms+analyzer = CentralityAnalyzer(kg)+centrality = analyzer.compute_pagerank()+betweenness = analyzer.compute_betweenness()-# Your agent can now find similar past decisions-similar_decisions = agent.find_precedents("Python tutorial", limit=3)-print(f"Agent found {len(similar_decisions)} similar past decisions")-```+# Node embeddings (Node2Vec)+embedder = NodeEmbedder()+embeddings = embedder.compute_embeddings(kg, node_labels=["Model"], relationship_types=["based_on"])-### Getting Smarter Over Time-```python-# Enable all learning features-smart_agent = AgentContext(- vector_store=vector_store,- decision_tracking=True, # Learn from decisions- graph_expansion=True, # Find related information- advanced_analytics=True, # Understand patterns- kg_algorithms=True, # Advanced analysis- vector_store_features=True-)+# Link prediction+predictor = LinkPredictor()+score = predictor.score_link(kg, "gpt4", "bert", method="common_neighbors")-# Get insights about your agent's learning-insights = smart_agent.get_context_insights()-print(f"Total decisions learned: {insights.get('total_decisions', 0)}")-print(f"Decision categories: {list(insights.get('categories', {}).keys())}")+models = kg.find_nodes(type="Model")+descendants = kg.get_neighbors("transformer", direction="incoming")```----## The Problem: The Semantic Gap--### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.--### Understanding the Semantic Gap--The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).--**Traditional AI approaches:**-- Rely on statistical patterns and text similarity-- Cannot understand relationships between entities-- Cannot reason about domain-specific rules-- Cannot explain why decisions were made-- Cannot trace back to original sources with confidence--**High-stakes AI requires:**-- Semantic understanding of entities and their relationships-- Domain knowledge encoded as formal rules (ontologies)-- Explainable reasoning paths-- Source-level provenance-- Conflict detection and resolution--**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.--### What Organizations Have vs What They Need--| **Current State** | **Required for High-Stakes AI** |-|:---------------------|:-----------------------------------|-| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |-| APIs, databases, streams | Structured and validated entities |-| Conflicting facts and duplicates | Explicit semantic relationships |-| Siloed systems with no lineage | **Explainable reasoning paths** |-| | **Source-level provenance** |-| | **Audit-ready compliance** |--### The Cost of Missing Semantics--- **Decisions cannot be explained** β No transparency in AI reasoning-- **Errors cannot be traced** β No way to debug or improve-- **Conflicts go undetected** β Contradictory information causes failures-- **Compliance becomes impossible** β No audit trails for regulations--**Trustworthy AI requires semantic accountability.**-------## Semantica vs Traditional RAG--| Feature | Traditional RAG | Semantica |-|:--------|:----------------|:----------|-| **Reasoning** | Black-box answers | Explainable reasoning paths |-| **Provenance** | No provenance | W3C PROV-O compliant lineage tracking |-| **Search** | Vector similarity only | Semantic + graph reasoning |-| **Quality** | No conflict handling | Explicit contradiction detection |-| **Safety** | Unsafe for high-stakes | Designed for governed environments |-| **Compliance** | No audit trails | Complete audit trails with integrity verification |-------## Semantica Architecture--### Input Layer β Governed Ingestion-- **Multiple Formats** β PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX-- **Docling Support** β Docling parser for table extraction-- **Data Sources** β Databases, APIs, streams, archives, web content-- **Media Support** β Image parsing with OCR, audio/video metadata extraction-- **Single Pipeline** β Unified ingestion with metadata and source tracking--### Semantic Layer β Trust & Reasoning Engine-- **Entity Extraction** β NER, normalization, classification-- **Relationship Discovery** β Triplet generation, semantic links-- **Ontology Induction** β Automated domain rule generation-- **Deduplication** β Jaro-Winkler similarity, conflict resolution-- **Quality Assurance** β Conflict detection, validation-- **Provenance Tracking** β W3C PROV-O compliant lineage tracking across all modules-- **Reasoning Traces** β Explainable inference paths-- **Change Management** β Version control with audit trails, checksums, compliance support--### Output Layer β Auditable Knowledge Assets-- **Knowledge Graphs** β Queryable, temporal, explainable-- **OWL Ontologies** β HermiT/Pellet validated, custom ontology import support-- **Vector Embeddings** β FastEmbed by default-- **AWS Neptune** β Amazon Neptune graph database support-- **Apache AGE** β PostgreSQL graph extension with openCypher support-- **Provenance** β Every AI response links back to:- - π Source documents- - π·οΈ Extracted entities & relations- - π Ontology rules applied- - π§ Reasoning steps used-------## Built for High-Stakes Domains--Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:--- **Healthcare & Life Sciences** β Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking-- **Finance & Risk** β Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation-- **Legal & Compliance** β Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning-- **Cybersecurity & Intelligence** β Threat attribution, incident response, security audit trails, intelligence analysis-- **Government & Defense** β Governed AI systems, policy decisions, classified information handling, defense intelligence-- **Critical Infrastructure** β Power grid management, transportation safety, water treatment, emergency response-- **Autonomous Systems** β Self-driving vehicles, drone navigation, robotics safety, industrial automation-------## π₯ Who Uses Semantica?--- **AI / ML Engineers** β Building explainable GraphRAG & agents-- **Data Engineers** β Creating governed semantic pipelines-- **Knowledge Engineers** β Managing ontologies & KGs at scale-- **Enterprise Teams** β Requiring trustworthy AI infrastructure-- **Risk & Compliance Teams** β Needing audit-ready systems-------## π¦ Installation--### Install from PyPI (Recommended)--```bash-pip install semantica-# or-pip install semantica[all]-```--### Install from Source (Development)--```bash-# Clone and install in editable mode-git clone https://github.com/Hawksight-AI/semantica.git-cd semantica-pip install -e .--# Or with all optional dependencies-pip install -e ".[all]"--# Development setup-pip install -e ".[dev]"-```--## π Resources--> **New to Semantica?** Check out the [**Cookbook**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook) for hands-on examples!--- [**Cookbook**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook) - Interactive notebooks- - [Introduction](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction) - Getting started tutorials- - [Advanced](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced) - Advanced techniques- - [Use Cases](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases) - Real-world applications--## β¨ Core Capabilities--| **Data Ingestion** | **Semantic Extract** | **Knowledge Graphs** | **Ontology** |-|:--------------------:|:----------------------:|:----------------------:|:--------------:|-| [Multiple Formats](#universal-data-ingestion) | [Entity & Relations](#semantic-intelligence-engine) | [Graph Analytics](#knowledge-graph-construction) | [Auto Generation](#ontology-generation--management) |-| **Context** | **GraphRAG** | **LLM Providers** | **Pipeline** |-| [Agent Memory, Context Graph, Context Retriever](#context-engineering--memory-systems) | [Hybrid RAG](#knowledge-graph-powered-rag-graphrag) | [100+ LLMs](#llm-providers-module) | [Parallel Workers](#pipeline-orchestration--parallel-processing) |-| **QA** | **Reasoning** | | |-| [Conflict Resolution](#production-ready-quality-assurance) | [Rule-based Inference](#reasoning--inference-engine) | | |-------### Universal Data Ingestion--> **Multiple file formats** β’ PDF, DOCX, HTML, JSON, CSV, databases, feeds, archives--```python-from semantica.ingest import FileIngestor, WebIngestor, DBIngestor--file_ingestor = FileIngestor(recursive=True)-web_ingestor = WebIngestor(max_depth=3)-db_ingestor = DBIngestor(connection_string="postgresql://...")--sources = []-sources.extend(file_ingestor.ingest("documents/"))-sources.extend(web_ingestor.ingest("https://example.com"))-sources.extend(db_ingestor.ingest(query="SELECT * FROM articles"))--print(f" Ingested {len(sources)} sources")-```--[**Cookbook: Data Ingestion**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/02_Data_Ingestion.ipynb)--### Document Parsing & Processing--> **Multi-format parsing** β’ **Docling Support** β’ **Text normalization** β’ **Intelligent chunking**--```python-from semantica.parse import DocumentParser, DoclingParser-from semantica.normalize import TextNormalizer-from semantica.split import TextSplitter--# Standard parsing-parser = DocumentParser()-parsed = parser.parse("document.pdf", format="auto")--# Parsing with Docling (for complex layouts/tables)-# Requires: pip install docling-docling_parser = DoclingParser(enable_ocr=True)-result = docling_parser.parse("complex_table.pdf")--print(f"Text (Markdown): {result['full_text'][:100]}...")-print(f"Extracted {len(result['tables'])} tables")-for i, table in enumerate(result['tables']):- print(f"Table {i+1} headers: {table.get('headers', [])}")--# Normalize text-normalizer = TextNormalizer()-normalized = normalizer.normalize(parsed, clean_html=True, normalize_entities=True)--# Split into chunks-splitter = TextSplitter(method="token", chunk_size=1000, chunk_overlap=200)-chunks = splitter.split(normalized)-```--[**Cookbook: Document Parsing**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/03_Document_Parsing.ipynb) β’ [**Data Normalization**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/04_Data_Normalization.ipynb) β’ [**Chunking & Splitting**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/11_Chunking_and_Splitting.ipynb)--### Semantic Intelligence Engine--> **Entity & Relation Extraction** β’ NER, Relationships, Events, Triplets with LLM Enhancement+## Semantic Extraction```python-from semantica.semantic_extract import NERExtractor, RelationExtractor+from semantica.semantic_extract import extract_entities, extract_relations, extract_triplets-text = "Apple Inc., founded by Steve Jobs in 1976, acquired Beats Electronics for $3 billion."+text = """+OpenAI released GPT-4 in March 2023. Microsoft integrated GPT-4 into Azure OpenAI Service.+Anthropic, founded by former OpenAI researchers, released Claude as a competing model.+"""-# Extract entities-ner_extractor = NERExtractor(method="ml", model="en_core_web_sm")-entities = ner_extractor.extract(text)+entities = extract_entities(text)+# β [Entity(label="OpenAI", type="Organization"), Entity(label="GPT-4", type="Model"), ...]-# Extract relationships-relation_extractor = RelationExtractor(method="dependency", model="en_core_web_sm")-relationships = relation_extractor.extract(text, entities=entities)+relations = extract_relations(text)+# β [Relation(source="OpenAI", type="released", target="GPT-4"), ...]-print(f"Entities: {len(entities)}, Relationships: {len(relationships)}")+triplets = extract_triplets(text)```-[**Cookbook: Entity Extraction**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/05_Entity_Extraction.ipynb) β’ [**Relation Extraction**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/06_Relation_Extraction.ipynb) β’ [**Advanced Extraction**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/01_Advanced_Extraction.ipynb)--### Knowledge Graph Construction--> **Production-Ready KGs** β’ **30+ Graph Algorithms** β’ **Entity Resolution** β’ **Temporal Support** β’ **Provenance Tracking**-```python-from semantica.semantic_extract import NERExtractor, RelationExtractor-from semantica.kg import GraphBuilder, NodeEmbedder, SimilarityCalculator, CentralityCalculator--# Extract entities and relationships-ner_extractor = NERExtractor(method="ml", model="en_core_web_sm")-relation_extractor = RelationExtractor(method="dependency", model="en_core_web_sm")--entities = ner_extractor.extract(text)-relationships = relation_extractor.extract(text, entities=entities)--# Build knowledge graph with provenance-builder = GraphBuilder()-kg = builder.build({"entities": entities, "relationships": relationships})--# Advanced graph analytics-embedder = NodeEmbedder(method="node2vec", embedding_dimension=128)-embeddings = embedder.compute_embeddings(kg, ["Entity"], ["RELATED_TO"])+from semantica.deduplication import DuplicateDetector-# Find similar nodes-calc = SimilarityCalculator()-similar_nodes = calc.find_most_similar(embeddings, embeddings["target_node"], top_k=5)+entities = [+ {"id": "e1", "name": "OpenAI Inc.", "type": "Organization"},+ {"id": "e2", "name": "Open AI", "type": "Organization"},+ {"id": "e3", "name": "Anthropic", "type": "Organization"},+]-# Analyze importance-centrality = CentralityCalculator()-importance_scores = centrality.calculate_all_centrality(kg)+detector = DuplicateDetector()+duplicates = detector.detect_duplicates(entities, threshold=0.85)+# β [("e1", "e2")]-print(f"Nodes: {len(kg.get('entities', []))}, Edges: {len(kg.get('relationships', []))}")-print(f"Similar nodes: {len(similar_nodes)}, Centrality measures: {len(importance_scores)}")+duplicates_v2 = detector.detect_duplicates(entities, threshold=0.85, strategy="semantic_v2")```-**New Enhanced Algorithms:**-- **Node Embeddings**: Node2Vec, DeepWalk, Word2Vec for structural similarity-- **Similarity Analysis**: Cosine, Euclidean, Manhattan, Correlation metrics-- **Path Finding**: Dijkstra, A*, BFS, K-shortest paths for route analysis-- **Link Prediction**: Preferential attachment, Jaccard, Adamic-Adar for network completion-- **Centrality Analysis**: Degree, Betweenness, Closeness, PageRank for importance ranking-- **Community Detection**: Louvain, Leiden, Label propagation for clustering-- **Connectivity Analysis**: Components, bridges, density for network robustness-- **Provenance Tracking**: Complete audit trail for all graph operations--[**Cookbook: Building Knowledge Graphs**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) β’ [**Graph Analytics**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/10_Graph_Analytics.ipynb) β’ [**Advanced Graph Analytics**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb)--### Embeddings & Vector Store+----> **FastEmbed by default** β’ **Multiple backends** (FAISS, PostgreSQL/pgvector, Weaviate, Qdrant, Milvus, Pinecone) β’ **Semantic search**+## Reasoning Engines```python-from semantica.embeddings import EmbeddingGenerator-from semantica.vector_store import VectorStore--# Generate embeddings-embedding_gen = EmbeddingGenerator(model_name="sentence-transformers/all-MiniLM-L6-v2", dimension=384)-embeddings = embedding_gen.generate_embeddings(chunks, data_type="text")--# Store in vector database-vector_store = VectorStore(backend="faiss", dimension=384)-vector_store.store_vectors(vectors=embeddings, metadata=[{"text": chunk} for chunk in chunks])--# Search-results = vector_store.search(query="supply chain", top_k=5)-```--[**Cookbook: Embedding Generation**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/12_Embedding_Generation.ipynb) β’ [**Vector Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/13_Vector_Store.ipynb)--### Graph Store & Triplet Store--> **Neo4j, FalkorDB, Amazon Neptune, Apache AGE** β’ **SPARQL queries** β’ **RDF triplets**+from semantica.reasoning import Reasoner-```python-from semantica.graph_store import GraphStore-from semantica.triplet_store import TripletStore--# Graph Store (Neo4j, FalkorDB, Apache AGE)-graph_store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", password="password")-graph_store.add_nodes([{"id": "n1", "labels": ["Person"], "properties": {"name": "Alice"}}])--# Amazon Neptune Graph Store (OpenCypher via HTTP with IAM Auth)-neptune_store = GraphStore(- backend="neptune",- endpoint="your-cluster.us-east-1.neptune.amazonaws.com",- port=8182,- region="us-east-1",- iam_auth=True, # Uses AWS credential chain (boto3, env vars, or IAM role)-)+reasoner = Reasoner()+reasoner.add_rule("IF Person(?x) THEN Mortal(?x)")+reasoner.add_rule("IF Employee(?x) AND WorksAt(?x, ?y) THEN HasEmployer(?x, ?y)")-# Node Operations-neptune_store.add_nodes([- {"labels": ["Person"], "properties": {"id": "alice", "name": "Alice", "age": 30}},- {"labels": ["Person"], "properties": {"id": "bob", "name": "Bob", "age": 25}},+results = reasoner.infer_facts([+ "Person(Socrates)",+ "Employee(Alice)",+ {"source_name": "Alice", "target_name": "OpenAI", "type": "WorksAt"},])--# Query Operations-result = neptune_store.execute_query("MATCH (p:Person) RETURN p.name, p.age")--# Apache AGE Graph Store (PostgreSQL + openCypher)-age_store = GraphStore(- backend="age",- connection_string="host=localhost dbname=agedb user=postgres password=secret",- graph_name="semantica",-)-age_store.connect()-age_store.create_node(labels=["Person"], properties={"name": "Alice", "age": 30})--# Triplet Store (Blazegraph, Jena, RDF4J)-triplet_store = TripletStore(backend="blazegraph", endpoint="http://localhost:9999/blazegraph")-triplet_store.add_triplet({"subject": "Alice", "predicate": "knows", "object": "Bob"})-results = triplet_store.execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10")+# β ["Mortal(Socrates)", "HasEmployer(Alice, OpenAI)"]```-[**Cookbook: Graph Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/09_Graph_Store.ipynb) β’ [**Triplet Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/20_Triplet_Store.ipynb)--### Ontology Generation & Management--> **6-Stage LLM Pipeline** β’ Automatic OWL Generation β’ HermiT/Pellet Validation β’ **Custom Ontology Import** (OWL, RDF, Turtle, JSON-LD)-```python-from semantica.ontology import OntologyGenerator-from semantica.ingest import ingest_ontology--# Generate ontology automatically-generator = OntologyGenerator(llm_provider="openai", model="gpt-4")-ontology = generator.generate_from_documents(sources=["domain_docs/"])--# Or import your existing ontology-custom_ontology = ingest_ontology("my_ontology.ttl") # Supports OWL, RDF, Turtle, JSON-LD-print(f"Classes: {len(custom_ontology.classes)}")+from semantica.reasoning import ReteEngine++rete = ReteEngine()+rete.add_rule({+ "name": "flag_high_risk_transaction",+ "conditions": [+ {"field": "amount", "operator": ">", "value": 10000},+ {"field": "country", "operator": "in", "value": ["IR", "KP", "SY"]},+ ],+ "action": "flag_for_compliance_review",+})+matches = rete.match({"amount": 15000, "country": "IR", "id": "txn_9921"})```-[**Cookbook: Ontology**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/14_Ontology.ipynb)--### Change Management & Version Control--> **Version Control for Knowledge Graphs & Ontologies** β’ **SQLite & In-Memory Storage** β’ **SHA-256 Integrity Verification**-```python-from semantica.change_management import TemporalVersionManager, OntologyVersionManager--# Knowledge Graph versioning with audit trails-kg_manager = TemporalVersionManager(storage_path="kg_versions.db")--# Create versioned snapshot-snapshot = kg_manager.create_snapshot(- knowledge_graph,- version_label="v1.0",- author="[email protected]",- description="Initial patient record"-)+from semantica.reasoning import DeductiveReasoner, AbductiveReasoner-# Compare versions with detailed diffs-diff = kg_manager.compare_versions("v1.0", "v2.0")-print(f"Entities added: {diff['summary']['entities_added']}")-print(f"Entities modified: {diff['summary']['entities_modified']}")+deductive = DeductiveReasoner()+deductive.add_axiom("All transformers use attention mechanisms")+deductive.add_fact("BERT is a transformer")+conclusion = deductive.reason("Does BERT use attention?")-# Verify data integrity-is_valid = kg_manager.verify_checksum(snapshot)+abductive = AbductiveReasoner()+abductive.add_observation("The model accuracy dropped 12% after deployment")+hypotheses = abductive.generate_hypotheses()+# β ["Distribution shift in production data", "Preprocessing pipeline mismatch", ...]```-**What We Provide:**-- **Persistent Storage** β SQLite and in-memory backends implemented-- **Detailed Diffs** β Entity-level and relationship-level change tracking-- **Data Integrity** β SHA-256 checksums with tamper detection-- **Standardized Metadata** β ChangeLogEntry with author, timestamp, description-- **Performance Tested** β Tested with large-scale entity datasets-- **Test Coverage** β Comprehensive test coverage covering core functionality--**Compliance Note:** Provides technical infrastructure (audit trails, checksums, temporal tracking) that supports compliance efforts for HIPAA, SOX, FDA 21 CFR Part 11. Organizations must implement additional policies and procedures for full regulatory compliance.--[**Documentation: Change Management**](docs/reference/change_management.md) β’ [**Usage Guide**](semantica/change_management/change_management_usage.md)--### Provenance Tracking β W3C PROV-O Compliant Lineage+----> **W3C PROV-O Implementation** β’ **17 Module Integrations** β’ **Opt-In Design** β’ **Zero Breaking Changes**+## Provenance Tracking-**β οΈ Compliance Note:** Provides technical infrastructure for provenance tracking that supports compliance efforts. Organizations must implement additional policies, procedures, and controls for full regulatory compliance.+W3C PROV-O compliant lineage tracking. Every fact traces back to its origin.```python-from semantica.semantic_extract.semantic_extract_provenance import NERExtractorWithProvenance-from semantica.llms.llms_provenance import GroqLLMWithProvenance-from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance--# Enable provenance tracking - just add provenance=True-ner = NERExtractorWithProvenance(provenance=True)-entities = ner.extract(- text="Apple Inc. was founded by Steve Jobs.",- source="biography.pdf"-)+from semantica.kg import ProvenanceTracker, AlgorithmTrackerWithProvenance-# Track LLM calls with costs and latency-llm = GroqLLMWithProvenance(provenance=True, model="llama-3.1-70b")-response = llm.generate("Summarize the document")+tracker = ProvenanceTracker()+tracker.track_entity("gpt4_benchmark",+ source_url="https://openai.com/research/gpt-4",+ metadata={"metric": "MMLU", "score": 86.4})-# Store in graph with complete lineage-graph = GraphStoreWithProvenance(provenance=True)-graph.add_node(entity, source="biography.pdf")--# Retrieve complete provenance-lineage = ner._prov_manager.get_lineage("entity_id")-print(f"Source: {lineage['source']}")-print(f"Lineage chain: {lineage['lineage_chain']}")-```--**What We Provide:**-- β **W3C PROV-O Implementation** β Data schemas implementing prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom-- β **17 Module Integrations** β Provenance-enabled versions of semantic extract, LLMs, pipeline, context, ingest, embeddings, reasoning, conflicts, deduplication, export, parse, normalize, ontology, visualization, graph/vector/triplet stores-- β **Opt-In Design** β Zero breaking changes, `provenance=False` by default-- β **Lineage Tracking** β Document β Chunk β Entity β Relationship β Graph lineage chains-- β **LLM Tracking** β Token counts, costs, and latency tracking for LLM calls-- β **Source Tracking Fields** β Document identifiers, page numbers, sections, and quote fields in schemas-- β **Storage Backends** β InMemoryStorage (fast) and SQLiteStorage (persistent) implemented-- β **Bridge Axioms** β BridgeAxiom and TranslationChain classes for domain transformations (L1 β L2 β L3)-- β **Integrity Verification** β SHA-256 checksum computation and verification functions-- β **No New Dependencies** β Uses Python stdlib only (sqlite3, json, dataclasses)--**Supported Modules:**-```python-# Semantic Extract-from semantica.semantic_extract.semantic_extract_provenance import (- NERExtractorWithProvenance, RelationExtractorWithProvenance, EventDetectorWithProvenance-)--# LLM Providers-from semantica.llms.llms_provenance import (- GroqLLMWithProvenance, OpenAILLMWithProvenance, HuggingFaceLLMWithProvenance+algo_tracker = AlgorithmTrackerWithProvenance(provenance=True)+algo_tracker.track_graph_construction(+ algorithm="node2vec",+ input_data={"nodes": 1500, "edges": 4200},+ parameters={"dimensions": 128, "walk_length": 80},)-# Storage & Processing-from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance-from semantica.vector_store.vector_store_provenance import VectorStoreWithProvenance-from semantica.pipeline.pipeline_provenance import PipelineWithProvenance--# ... and 12 more modules+sources = tracker.get_all_sources("gpt4_benchmark")+all_entities = tracker.get_all_entities()```-**High-Stakes Use Cases:**-- π₯ **Healthcare** β Clinical decision audit trails with source tracking-- π° **Finance** β Fraud detection provenance with complete lineage-- βοΈ **Legal** β Evidence chain of custody with temporal tracking-- π **Cybersecurity** β Threat attribution with relationship tracking-- ποΈ **Government** β Policy decision audit trails with integrity verification--**Note:** Provenance tracking provides the *technical infrastructure* for compliance. Organizations must implement additional policies and procedures to meet specific regulatory requirements (HIPAA, SOX, FDA 21 CFR Part 11, etc.).--[**Documentation: Provenance Tracking**](semantica/provenance/provenance_usage.md)--### Context Engineering & Memory Systems+----> **Persistent Memory** β’ **Context Graph** β’ **Context Retriever** β’ **Hybrid Retrieval (Vector + Graph)** β’ **Production Graph Store (Neo4j)** β’ **Entity Linking** β’ **Multi-Hop Reasoning**+## Vector Store & Hybrid Search```python-from semantica.context import AgentContext, ContextGraph, ContextRetrieverfrom semantica.vector_store import VectorStore-from semantica.graph_store import GraphStore-from semantica.llms import Groq-# Initialize Context with Hybrid Retrieval (Graph + Vector)-context = AgentContext(- vector_store=VectorStore(backend="faiss"),- knowledge_graph=GraphStore(backend="neo4j"), # Optional: Use persistent graph- hybrid_alpha=0.75 # Balanced weight between Knowledge Graph and Vector-)+vs = VectorStore(backend="faiss", dimension=768)-# Build Context Graph from entities and relationships-graph_stats = context.build_graph(- entities=kg.get('entities', []),- relationships=kg.get('relationships', []),- link_entities=True-)+vs.store("The Transformer architecture revolutionized NLP",+ metadata={"source": "arxiv", "year": 2017}, id="doc_001")+vs.store("BERT introduced bidirectional pre-training for language understanding",+ metadata={"source": "arxiv", "year": 2018}, id="doc_002")-# Store memory with automatic entity linking-context.store(- "User is building a RAG system with Semantica",- metadata={"priority": "high", "topic": "rag"}-)+results = vs.search("attention mechanisms in language models", top_k=5)-# Use Context Retriever for hybrid retrieval-retriever = context.retriever # Access underlying ContextRetriever-results = retriever.retrieve(- query="What is the user building?",- max_results=10,- graph_expansion=True+results = vs.hybrid_search(+ query="transformer pre-training",+ top_k=10,+ vector_weight=0.6,+ keyword_weight=0.4,)-# Retrieve with context expansion-results = context.retrieve("What is the user building?", graph_expansion=True)--# Query with reasoning and LLM-generated responses-llm_provider = Groq(model="llama-3.1-8b-instant", api_key=os.getenv("GROQ_API_KEY"))-reasoned_result = context.query_with_reasoning(- query="What is the user building?",- llm_provider=llm_provider,- max_hops=2-)+results = vs.search("pre-training", top_k=5, filter={"year": 2018})```-**Core Components:**-- **ContextGraph**: Builds and manages context graphs from entities and relationships for enhanced retrieval-- **ContextRetriever**: Performs hybrid retrieval combining vector search, graph traversal, and memory for optimal context relevance-- **AgentContext**: High-level interface integrating Context Graph and Context Retriever for GraphRAG applications+----#### Context Graphs: Advanced Decision Tracking & Analytics+## Data Ingestion```python-from semantica.context import AgentContext, ContextGraph-from semantica.vector_store import VectorStore+from semantica.ingest import FileIngestor, WebIngestor, DBIngestor-# Initialize with advanced decision tracking-context = AgentContext(- vector_store=VectorStore(backend="inmemory", dimension=128),- knowledge_graph=ContextGraph(advanced_analytics=True),- decision_tracking=True,- kg_algorithms=True, # Enable advanced graph analytics-)+file_ingestor = FileIngestor(recursive=True)+docs = file_ingestor.ingest("./research_papers/")-# Easy decision recording with convenience methods-decision_id = context.graph_builder.add_decision(- category="credit_approval",- scenario="High-risk credit limit increase",- reasoning="Recent velocity-check failure and prior fraud flag",- outcome="rejected",- confidence=0.78,- entities=["customer:jessica_norris"],-)+web_ingestor = WebIngestor(max_depth=2)+web_docs = web_ingestor.ingest("https://arxiv.org/abs/1706.03762")-# Find similar decisions with advanced analytics-similar_decisions = context.graph_builder.find_similar_decisions(- scenario="credit increase",- category="credit_approval",- max_results=5,-)+db_ingestor = DBIngestor(connection_string="postgresql://user:pass@localhost/kg_db")+db_docs = db_ingestor.ingest(query="SELECT title, abstract FROM papers WHERE year >= 2020")-# Analyze decision impact and influence-impact_analysis = context.graph_builder.analyze_decision_impact(decision_id)-node_importance = context.graph_builder.get_node_importance("customer:jessica_norris")--# Check compliance with business rules-compliance = context.graph_builder.check_decision_rules({- "category": "credit_approval",- "scenario": "Credit limit increase",- "reasoning": "Risk assessment completed",- "outcome": "rejected",- "confidence": 0.78-})+all_sources = docs + web_docs + db_docs```-**Enhanced Features:**-- **Easy-to-Use Methods**: 10 convenience methods for common operations-- **Decision Analytics**: Influence analysis, centrality measures, community detection-- **Policy Engine**: Automated compliance checking with business rules-- **Causal Analysis**: Trace decision causality and impact chains-- **Graph Analytics**: Advanced KG algorithms (Node2Vec, centrality, community detection)-- **Hybrid Search**: Semantic + structural + category similarity-- **Production Ready**: Scalable architecture with comprehensive error handling--## Configuration Options--### Simple Setup (Most Common)```python-# Just memory and basic learning-agent = AgentContext(vector_store=vector_store)-```+from semantica.parse import DoclingParser-### Smart Setup (Recommended)-```python-# Memory + decision learning-agent = AgentContext(- vector_store=vector_store,- decision_tracking=True,- graph_expansion=True-)+# Advanced table and layout extraction+docling = DoclingParser()+parsed = docling.parse("financial_report.pdf")```-### Complete Setup (Maximum Power)-```python-# Everything enabled-agent = AgentContext(- vector_store=vector_store,- knowledge_graph=ContextGraph(advanced_analytics=True),- decision_tracking=True,- graph_expansion=True,- advanced_analytics=True,- kg_algorithms=True,- vector_store_features=True-)-```--### ContextGraph Options-```python-# Basic knowledge graph-graph = ContextGraph()--# Advanced knowledge graph-graph = ContextGraph(- advanced_analytics=True, # Enable smart algorithms- centrality_analysis=True, # Find important concepts- community_detection=True, # Find groups of related concepts- node_embeddings=True # Understand concept similarity-)-```--**Core Notebooks:**-- [**Context Module Introduction**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/19_Context_Module.ipynb) - Basic memory and storage.-- [**Advanced Context Engineering**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb) - Hybrid retrieval, graph builders, and custom memory policies.-- [**Fraud Detection**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) - Demonstrates Context Graph and Context Retriever for fraud detection with GraphRAG.--**Related Components:**-[**Vector Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/13_Vector_Store.ipynb) β’ [**Embedding Generation**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/12_Embedding_Generation.ipynb) β’ [**Advanced Vector Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/Advanced_Vector_Store_and_Search.ipynb)--### Knowledge Graph-Powered RAG (GraphRAG)+----> **Vector + Graph Hybrid Search** β’ **Multi-Hop Reasoning** β’ **LLM-Generated Responses** β’ **Semantic Re-ranking**+## Export```python-from semantica.context import AgentContext-from semantica.llms import Groq, OpenAI, LiteLLM-from semantica.vector_store import VectorStore-import os+from semantica.export import RDFExporter, ParquetExporter, ArangoAQLExporter-# Initialize GraphRAG with hybrid retrieval-context = AgentContext(- vector_store=VectorStore(backend="faiss"),- knowledge_graph=kg-)--# Configure LLM provider (supports Groq, OpenAI, HuggingFace, LiteLLM)-llm_provider = Groq(- model="llama-3.1-8b-instant",- api_key=os.getenv("GROQ_API_KEY")-)+rdf_exporter = RDFExporter()+turtle = rdf_exporter.export_to_rdf(kg, format="turtle")+jsonld = rdf_exporter.export_to_rdf(kg, format="json-ld")+ntriples = rdf_exporter.export_to_rdf(kg, format="nt")-# Query with multi-hop reasoning and LLM-generated responses-result = context.query_with_reasoning(- query="What IPs are associated with security alerts?",- llm_provider=llm_provider,- max_results=10,- max_hops=2-)+parquet_exporter = ParquetExporter()+parquet_exporter.export_entities(kg, path="output/entities.parquet")+parquet_exporter.export_relationships(kg, path="output/relationships.parquet")+parquet_exporter.export_knowledge_graph(kg, path="output/")-print(f"Response: {result['response']}")-print(f"Reasoning Path: {result['reasoning_path']}")-print(f"Confidence: {result['confidence']:.3f}")+aql_exporter = ArangoAQLExporter()+aql_exporter.export(kg, path="output/insert.aql")```-**Key Features:**-- **Multi-Hop Reasoning**: Traverses knowledge graph up to N hops to find related entities-- **LLM-Generated Responses**: Natural language answers grounded in graph context-- **Reasoning Trace**: Shows entity relationship paths used in reasoning-- **Multiple LLM Providers**: Supports Groq, OpenAI, HuggingFace, and LiteLLM (100+ LLMs)--[**Cookbook: GraphRAG**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) β’ [**Real-Time Anomaly Detection**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)--### LLM Providers Module+----> **Unified LLM Interface** β’ **100+ LLM Support via LiteLLM** β’ **Clean Imports** β’ **Multiple Providers**+## Pipeline Orchestration```python-from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM-import os--# Groq-groq = Groq(- model="llama-3.1-8b-instant",- api_key=os.getenv("GROQ_API_KEY")-)-response = groq.generate("What is AI?")+from semantica.pipeline import PipelineBuilder, PipelineValidator, FailureHandler+from semantica.pipeline import RetryPolicy, RetryStrategy-# OpenAI-openai = OpenAI(- model="gpt-4",- api_key=os.getenv("OPENAI_API_KEY")+builder = (+ PipelineBuilder()+ .add_stage("ingest", FileIngestor(recursive=True))+ .add_stage("extract", extract_triplets)+ .add_stage("deduplicate", DuplicateDetector())+ .add_stage("build_kg", KnowledgeGraph())+ .add_stage("export", RDFExporter())+ .with_parallel_workers(4))-response = openai.generate("What is AI?")--# HuggingFace - Local models-hf = HuggingFaceLLM(model_name="gpt2")-response = hf.generate("What is AI?")--# LiteLLM - Unified interface to 100+ LLMs-litellm = LiteLLM(- model="openai/gpt-4o", # or "anthropic/claude-sonnet-4-20250514", "groq/llama-3.1-8b-instant", etc.- api_key=os.getenv("OPENAI_API_KEY")-)-response = litellm.generate("What is AI?")--# Structured output-structured = groq.generate_structured("Extract entities from: Apple Inc. was founded by Steve Jobs.")-```--**Supported Providers:**-- **Groq**: Inference with Llama models-- **OpenAI**: GPT-3.5, GPT-4, and other OpenAI models-- **HuggingFace**: Local LLM inference with Transformers-- **LiteLLM**: Unified interface to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, and more)--### Reasoning & Inference Engine--> **Rule-based Inference** β’ **Forward/Backward Chaining** β’ **Rete Algorithm** β’ **Explanation Generation**--```python-from semantica.reasoning import Reasoner--# Initialize Reasoner-reasoner = Reasoner()--# Define rules and facts-rules = ["IF Parent(?a, ?b) AND Parent(?b, ?c) THEN Grandparent(?a, ?c)"]-facts = ["Parent(Alice, Bob)", "Parent(Bob, Charlie)"]-# Infer new facts (Forward Chaining)-inferred = reasoner.infer_facts(facts, rules)-print(f"Inferred: {inferred}") # ['Grandparent(Alice, Charlie)']+validator = PipelineValidator()+result = validator.validate(builder)+if result.valid:+ pipeline = builder.build()+ pipeline.run(input_path="./documents/")-# Explain reasoning-from semantica.reasoning import ExplanationGenerator-explainer = ExplanationGenerator()-# ... generate explanation for inferred facts+retry_policy = RetryPolicy(strategy=RetryStrategy.EXPONENTIAL_BACKOFF, max_retries=3)+handler = FailureHandler()+handler.handle_failure(error=last_error, policy=retry_policy, retry_count=1)```-[**Cookbook: Reasoning**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb) β’ [**Rete Engine**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/09_Rete_Engine.ipynb)--### Pipeline Orchestration & Parallel Processing--> **Orchestrator-Worker Pattern** β’ Parallel Execution β’ Scalable Processing--```python-from semantica.pipeline import PipelineBuilder, ExecutionEngine--pipeline = PipelineBuilder() \- .add_step("ingest", "custom", func=ingest_data) \- .add_step("extract", "custom", func=extract_entities) \- .add_step("build", "custom", func=build_graph) \- .build()--result = ExecutionEngine().execute_pipeline(pipeline, parallel=True)-```----### Production-Ready Quality Assurance+----> **Enterprise-Grade QA** β’ Conflict Detection β’ Deduplication+## Ontology```python-from semantica.deduplication import DuplicateDetector-from semantica.conflicts import ConflictDetector--entities = kg.get("entities", [])-conflicts = ConflictDetector().detect_conflicts(entities)-duplicates = DuplicateDetector(similarity_threshold=0.85).detect_duplicates(entities)--print(f"Conflicts: {len(conflicts)} | Duplicates: {len(duplicates)}")-```+from semantica.ontology import OntologyGenerator, OntologyImporter-[**Cookbook: Conflict Detection & Resolution**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/17_Conflict_Detection_and_Resolution.ipynb) β’ [**Deduplication**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/18_Deduplication.ipynb)+generator = OntologyGenerator()+ontology = generator.generate(kg)+generator.export(ontology, path="domain_ontology.owl", format="turtle")-### Visualization & Export--> **Interactive graphs** β’ **Multi-format export** β’ **Graph analytics**--```python-from semantica.visualization import KGVisualizer-from semantica.export import GraphExporter--# Visualize knowledge graph-viz = KGVisualizer(layout="force")-fig = viz.visualize_network(kg, output="interactive")-fig.show()--# Export to multiple formats-exporter = GraphExporter()-exporter.export(kg, format="json", output_path="graph.json")-exporter.export(kg, format="graphml", output_path="graph.graphml")+importer = OntologyImporter()+ontology = importer.load("existing_ontology.owl")+ontology = importer.load("schema.ttl", format="turtle")+ontology = importer.load("context.jsonld")```-[**Cookbook: Visualization**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/16_Visualization.ipynb) β’ [**Export**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/15_Export.ipynb)--### Seed Data Integration+----> **Foundation data** β’ **Entity resolution** β’ **Domain knowledge**+## Integrations-```python-from semantica.seed import SeedDataManager+**Graph Databases**+- AWS Neptune β Amazon Neptune with IAM authentication+- Apache AGE β PostgreSQL + openCypher via SQL+- FalkorDB β native support for decision queries and causal analysis-seed_manager = SeedDataManager()-seed_manager.seed_data.entities = [- {"id": "s1", "text": "Supplier A", "type": "Supplier", "source": "foundation", "verified": True}-]+**Vector Databases**+- FAISS β high-performance dense vector search+- In-memory β lightweight, zero-dependency store for development and testing-# Use seed data for entity resolution-resolved = seed_manager.resolve_entities(extracted_entities)-```+**Document Parsing**+- Docling β PDF, DOCX, PPTX, XLSX with table and layout extraction-[**Cookbook: Seed Data**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/07_Seed_Data_Integration.ipynb)+**LLM Providers**+- 100+ models via LiteLLM β OpenAI, Anthropic, Cohere, Mistral, Ollama, Azure, AWS Bedrock, and more-## π Quick Start+**AI Frameworks**+- Complements LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK-> **For comprehensive examples, see the [**Cookbook**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook) with interactive notebooks!**+**Export**+- RDF: Turtle, JSON-LD, N-Triples, XML Β· Parquet Β· ArangoDB AQL-```python-from semantica.semantic_extract import NERExtractor, RelationExtractor-from semantica.kg import GraphBuilder-from semantica.context import AgentContext, ContextGraph-from semantica.vector_store import VectorStore+----# Extract entities and relationships-ner_extractor = NERExtractor(method="ml", model="en_core_web_sm")-relation_extractor = RelationExtractor(method="dependency", model="en_core_web_sm")+## Installation-text = "Apple Inc. was founded by Steve Jobs in 1976."-entities = ner_extractor.extract(text)-relationships = relation_extractor.extract(text, entities=entities)+```bash+# Core+pip install semantica-# Build knowledge graph-builder = GraphBuilder()-kg = builder.build({"entities": entities, "relationships": relationships})+# With all optional dependencies+pip install semantica[all]-# Query using GraphRAG-vector_store = VectorStore(backend="faiss", dimension=384)-context_graph = ContextGraph()-context_graph.build_from_entities_and_relationships(- entities=kg.get('entities', []),- relationships=kg.get('relationships', [])-)-context = AgentContext(vector_store=vector_store, knowledge_graph=context_graph)+# From source+git clone https://github.com/Hawksight-AI/semantica.git+cd semantica+pip install -e ".[dev]"-results = context.retrieve("Who founded Apple?", max_results=5)-print(f"Found {len(results)} results")+# Run tests+pytest tests/```-[**Cookbook: Your First Knowledge Graph**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)--## π― Use Cases--**Enterprise Knowledge Engineering** β Unify data sources into knowledge graphs, breaking down silos.--**AI Agents & Autonomous Systems** β Build agents with persistent memory and semantic understanding.--**Multi-Format Document Processing** β Process multiple formats through a unified pipeline.--**Data Pipeline Processing** β Build scalable pipelines with parallel execution.--**Intelligence & Security** β Analyze networks, threat intelligence, forensic analysis.--**Finance & Trading** β Fraud detection, market intelligence, risk assessment.--**Biomedical** β Drug discovery, medical literature analysis.----## π³ Semantica Cookbook--> **Interactive Jupyter Notebooks** designed to take you from beginner to expert.--[**View Full Cookbook**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook)--### Featured Recipes--| **Recipe** | **Description** | **Link** |-|:-----------|:----------------|:---------|-| **GraphRAG Complete** | Build a production-ready **Graph Retrieval Augmented Generation** system. Features **Graph Validation**, **Hybrid Retrieval**, and **Logical Inference**. | [Open Notebook](cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) |-| **RAG vs. GraphRAG** | Side-by-side comparison. Demonstrates the **Reasoning Gap** and how GraphRAG solves it with **Inference Engines**. | [Open Notebook](cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) |-| **First Knowledge Graph** | Go from raw text to a queryable knowledge graph in 20 minutes. | [Open Notebook](cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb) |-| **Real-Time Anomalies** | Detect anomalies in streaming data using temporal knowledge graphs and pattern detection. | [Open Notebook](cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) |--### Core Tutorials--- [**Welcome to Semantica**](cookbook/introduction/01_Welcome_to_Semantica.ipynb) - Framework Overview-- [**Data Ingestion**](cookbook/introduction/02_Data_Ingestion.ipynb) - Universal Ingestion-- [**Entity Extraction**](cookbook/introduction/05_Entity_Extraction.ipynb) - NER & Relationships-- [**Building Knowledge Graphs**](cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) - Graph Construction--### Industry Use Cases (14 Cookbooks)--**Domain-Specific Cookbooks** showcasing real-world applications with real data sources, advanced chunking strategies, temporal KGs, GraphRAG, and comprehensive Semantica module integration:--#### Biomedical-- [**Drug Discovery Pipeline**](cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) - PubMed RSS, entity-aware chunking, GraphRAG, vector similarity search-- [**Genomic Variant Analysis**](cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) - bioRxiv RSS, temporal KGs, deduplication, pathway analysis--#### Finance-- [**Financial Data Integration MCP**](cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) - Alpha Vantage API, MCP servers, seed data, real-time ingestion-- [**Fraud Detection**](cookbook/use_cases/finance/02_Fraud_Detection.ipynb) - Transaction streams, temporal KGs, pattern detection, conflict resolution, **Context Graph**, **Context Retriever**, GraphRAG with Groq LLM--#### Blockchain-- [**DeFi Protocol Intelligence**](cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) - CoinDesk RSS, ontology-aware chunking, conflict detection, ontology generation-- [**Transaction Network Analysis**](cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) - Blockchain APIs, deduplication, network analytics--#### Cybersecurity-- [**Real-Time Anomaly Detection**](cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) - CVE RSS, Kafka streams, temporal KGs, sentence chunking-- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, GraphRAG, deduplication--#### Intelligence & Law Enforcement-- [**Criminal Network Analysis**](cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) - OSINT RSS, deduplication, network centrality, graph analytics-- [**Intelligence Analysis Orchestrator Worker**](cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) - Pipeline orchestrator, multi-source integration, conflict detection--#### Renewable Energy-- [**Energy Market Analysis**](cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) - Energy RSS, EIA API, temporal KGs, TemporalPatternDetector, trend prediction--#### Supply Chain-- [**Supply Chain Data Integration**](cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) - Logistics RSS, deduplication, relationship mapping---[**Explore Use Case Examples**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/use_cases) β See real-world implementations in finance, biomedical, cybersecurity, and more. **14 comprehensive domain-specific cookbooks** with real data sources, advanced chunking strategies, temporal KGs, GraphRAG, and full Semantica module integration.--## π¬ Advanced Features--**Docling Integration** β Document parsing with table extraction for PDFs, DOCX, PPTX, and XLSX files. Supports OCR and multiple export formats.--**AWS Neptune Support** β Amazon Neptune graph database integration with IAM authentication and OpenCypher queries.--**Custom Ontology Import** β Import existing ontologies (OWL, RDF, Turtle, JSON-LD, N3) and extend Schema.org, FOAF, Dublin Core, or custom ontologies.--**Multi-Language Support** β Process multiple languages with automatic detection.--**Advanced Reasoning** β Forward/backward chaining, Rete-based pattern matching, and automated explanation generation.--**Graph Analytics** β Centrality, community detection, path finding, temporal analysis.--**Custom Pipelines** β Build custom pipelines with parallel execution.--**API Integration** β Integrate external APIs for entity enrichment.--[**See Advanced Examples**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced) β Advanced extraction, graph analytics, reasoning, and more.-----## π€ Community & Support@@ -1306,44 +656,39 @@ Enterprise support, professional services, and commercial licensing will be avaiStay tuned for updates!-## π€ Contributing+- **AI / ML engineers** β GraphRAG, explainable agents, decision tracing+- **Data engineers** β governed semantic pipelines with full provenance+- **Knowledge engineers** β ontology management and KG construction at scale+- **High-stakes domains** β healthcare, finance, legal, cybersecurity, government-### How to Contribute--```bash-# Fork and clone-git clone https://github.com/your-username/semantica.git-cd semantica--# Create branch-git checkout -b feature/your-feature+----# Install dev dependencies-pip install -e ".[dev,test]"+## Resources-# Make changes and test-pytest tests/-black semantica/-flake8 semantica/+- [Documentation](https://github.com/Hawksight-AI/semantica/tree/main/docs)+- [Cookbook & Notebooks](https://github.com/Hawksight-AI/semantica/tree/main/cookbook)+- [Contributing Guide](CONTRIBUTING.md)+- [Changelog](https://github.com/Hawksight-AI/semantica/releases)+- [π¬ Discord Community](https://discord.gg/sV34vps5hH)+- [Follow on X](https://x.com/BuildSemantica)-# Commit and push-git commit -m "Add feature"-git push origin feature/your-feature-```+----### Contribution Types+## Contributing-1. **Code** - New features, bug fixes-2. **Documentation** - Improvements, tutorials-3. **Bug Reports** - [Create issue](https://github.com/Hawksight-AI/semantica/issues/new)-4. **Feature Requests** - [Request feature](https://github.com/Hawksight-AI/semantica/issues/new)+All contributions welcome β bug fixes, new features, tests, and docs.+1. Fork the repo and create a branch+2. `pip install -e ".[dev]"`+3. Write tests alongside your changes+4. Open a PR and tag `@KaifAhmad1` for review-## π License+See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines.-Semantica is licensed under the **MIT License** - see the [LICENSE](https://github.com/Hawksight-AI/semantica/blob/main/LICENSE) file for details.+----**Built by the Semantica Community**+<div align="center">+MIT License Β· Built by [Hawksight AI](https://github.com/Hawksight-AI) Β· [β Star on GitHub](https://github.com/Hawksight-AI/semantica)[GitHub](https://github.com/Hawksight-AI/semantica) β’ [Discord](https://discord.gg/sV34vps5hH)