Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
Enhance README with new features and integrations
Mohd Kaif420ccfe45abe
diff --git a/README.md b/README.mdindex 8e540df3..a15400ba 100644--- a/README.md+++ b/README.md@@ -100,7 +100,7 @@ pip install semantica- **Change management** — version control with checksums, audit trails, compliance support### Vector Store-- **Backends** — FAISS, in-memory+- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory- **Semantic search** — top-k retrieval by embedding similarity- **Hybrid search** — vector + keyword with configurable weights- **Filtered search** — metadata-based filtering on any field@@ -115,6 +115,7 @@ pip install semantica- **File formats** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives- **Web crawl** — `WebIngestor` with configurable depth- **Databases** — `DBIngestor` with SQL query support+- **Snowflake** — `SnowflakeIngestor` with table/query ingestion, pagination, and key-pair/OAuth auth- **Docling** — advanced document parsing with table and layout extraction (PDF, DOCX, PPTX, XLSX)- **Media** — image OCR, audio/video metadata extraction@@ -142,15 +143,30 @@ pip install semantica| Module | What it provides ||---|---|-| `semantica.context` | Context graphs, agent memory, decision tracking, policy engine, causal chains |-| `semantica.kg` | Knowledge graphs, graph algorithms, embeddings, link prediction, provenance |-| `semantica.semantic_extract` | Entity & relation extraction, deduplication, triplet generation |-| `semantica.reasoning` | Forward chaining, Rete, deductive, abductive, SPARQL reasoning |-| `semantica.vector_store` | FAISS, in-memory, hybrid search with custom similarity weights |-| `semantica.export` | RDF (Turtle/JSON-LD/N-Triples/XML), Parquet, ArangoDB AQL |-| `semantica.ingest` | Files (PDF, DOCX, CSV, HTML), web crawl, databases, Docling |-| `semantica.ontology` | OWL generation, import (OWL/RDF/Turtle/JSON-LD), validation |-| `semantica.pipeline` | Parallel workers, validation, failure handling, retry policies |+| `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search, policy engine |+| `semantica.kg` | Knowledge graph construction, graph algorithms, centrality, community detection, embeddings, link prediction, provenance |+| `semantica.semantic_extract` | NER, relation extraction, event extraction, coreference, triplet generation, LLM-enhanced extraction |+| `semantica.reasoning` | Forward chaining, Rete network, deductive, abductive, SPARQL reasoning, explanation generation |+| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory; hybrid & filtered search |+| `semantica.export` | RDF (Turtle/JSON-LD/N-Triples/XML), Parquet, ArangoDB AQL, CSV, YAML, OWL, graph formats |+| `semantica.ingest` | Files (PDF, DOCX, CSV, HTML), web crawl, feeds, databases, Snowflake, MCP, email, repositories |+| `semantica.ontology` | Auto-generation (6-stage pipeline), OWL/RDF export, import (OWL/RDF/Turtle/JSON-LD), validation, versioning |+| `semantica.pipeline` | Pipeline DSL, parallel workers, validation, retry policies, failure handling, resource scheduling |+| `semantica.graph_store` | Graph database backends — Neo4j, FalkorDB, Apache AGE, Amazon Neptune; Cypher queries |+| `semantica.embeddings` | Text embedding generation — Sentence-Transformers, FastEmbed, OpenAI, BGE; similarity calculation |+| `semantica.deduplication` | Entity deduplication, similarity scoring, merging, clustering; blocking and semantic strategies |+| `semantica.provenance` | W3C PROV-O compliant end-to-end lineage tracking, source attribution, audit trails |+| `semantica.parse` | Document parsing — PDF, DOCX, PPTX, HTML, code, email, structured data, media with OCR |+| `semantica.split` | Document chunking — recursive, semantic, entity-aware, relation-aware, graph-based, ontology-aware |+| `semantica.normalize` | Data normalization for text, entities, dates, numbers, quantities, languages, encodings |+| `semantica.conflicts` | Multi-source conflict detection (value, type, relationship, temporal, logical) with resolution strategies |+| `semantica.change_management` | Version storage, change tracking, checksums, audit trails, compliance support for KGs and ontologies |+| `semantica.triplet_store` | RDF triplet store integration — Blazegraph, Jena, RDF4J; SPARQL queries and bulk loading |+| `semantica.visualization` | Interactive and static visualization of KGs, ontologies, embeddings, analytics, and temporal graphs |+| `semantica.seed` | Seed data management for initial KG construction from CSV, JSON, databases, and APIs |+| `semantica.core` | Framework orchestration, configuration management, knowledge base construction, plugin system |+| `semantica.llms` | LLM provider integrations — Groq, OpenAI, HuggingFace, LiteLLM |+| `semantica.utils` | Shared utilities — logging, validation, exception handling, constants, types, progress tracking |---@@ -252,6 +268,7 @@ influence = context.analyze_decision_influence(decision_id)- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication- **Apache AGE** — PostgreSQL graph extension backend (openCypher via SQL)+- **Snowflake** — Native ingestion with `SnowflakeIngestor`; table/query ingestion, pagination, key-pair & OAuth auth- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)> **Built for environments where every answer must be explainable and governed.**@@ -515,6 +532,39 @@ docling = DoclingParser()parsed = docling.parse("financial_report.pdf")```+```python+from semantica.ingest import SnowflakeIngestor++# Connect to Snowflake and ingest a table+ingestor = SnowflakeIngestor(+ account="myorg-myaccount",+ user="analyst",+ password="...",+ warehouse="COMPUTE_WH",+ database="ANALYTICS",+ schema="PUBLIC",+)++# Ingest a table with optional filtering and pagination+data = ingestor.ingest_table(+ table_name="customer_events",+ where="event_date >= '2024-01-01'",+ limit=10000,+)++# Or run a custom SQL query+data = ingestor.ingest_query(+ query="SELECT id, content, tags FROM knowledge_base WHERE active = TRUE",+ batch_size=500,+)++# Convert to Semantica documents for downstream pipeline use+docs = ingestor.export_as_documents(data, id_field="id", text_fields=["content"])++# Key-pair and OAuth auth are also supported via env vars:+# SNOWFLAKE_PRIVATE_KEY_PATH, SNOWFLAKE_TOKEN, SNOWFLAKE_AUTHENTICATOR+```+---## Export@@ -593,8 +643,16 @@ ontology = importer.load("context.jsonld")**Vector Databases**- FAISS — high-performance dense vector search+- Pinecone — serverless and pod-based managed vector database (`pip install semantica[vectorstore-pinecone]`)+- Weaviate — GraphQL-based vector store with rich schema management (`pip install semantica[vectorstore-weaviate]`)+- Qdrant — collection-based store with payload filtering (`pip install semantica[vectorstore-qdrant]`)+- Milvus — scalable store with partition support and multiple index types (`pip install semantica[vectorstore-milvus]`)+- PgVector — PostgreSQL pgvector extension with JSONB metadata (`pip install semantica[vectorstore-pgvector]`)- In-memory — lightweight, zero-dependency store for development and testing+**Data Sources**+- Snowflake — `SnowflakeIngestor` for table/query ingestion, schema introspection, pagination, and multiple auth methods (password, key-pair, OAuth, SSO) (`pip install semantica[db-snowflake]`)+**Document Parsing**- Docling — PDF, DOCX, PPTX, XLSX with table and layout extraction@@ -618,6 +676,16 @@ pip install semantica# With all optional dependenciespip install semantica[all]+# Vector store backends (install only what you need)+pip install semantica[vectorstore-pinecone]+pip install semantica[vectorstore-weaviate]+pip install semantica[vectorstore-qdrant]+pip install semantica[vectorstore-milvus]+pip install semantica[vectorstore-pgvector]++# Snowflake ingestion+pip install semantica[db-snowflake]+# From sourcegit clone https://github.com/Hawksight-AI/semantica.gitcd semantica