Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

Enhance README with new features and integrations

Mohd Kaif420ccfe45abe
diff --git a/README.md b/README.md
index 8e540df3..a15400ba 100644
--- a/README.md
+++ b/README.md
@@ -100,7 +100,7 @@ pip install semantica
- **Change management** — version control with checksums, audit trails, compliance support
### Vector Store
-- **Backends** — FAISS, in-memory
+- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory
- **Semantic search** — top-k retrieval by embedding similarity
- **Hybrid search** — vector + keyword with configurable weights
- **Filtered search** — metadata-based filtering on any field
@@ -115,6 +115,7 @@ pip install semantica
- **File formats** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives
- **Web crawl** — `WebIngestor` with configurable depth
- **Databases** — `DBIngestor` with SQL query support
+- **Snowflake** — `SnowflakeIngestor` with table/query ingestion, pagination, and key-pair/OAuth auth
- **Docling** — advanced document parsing with table and layout extraction (PDF, DOCX, PPTX, XLSX)
- **Media** — image OCR, audio/video metadata extraction
@@ -142,15 +143,30 @@ pip install semantica
| Module | What it provides |
|---|---|
-| `semantica.context` | Context graphs, agent memory, decision tracking, policy engine, causal chains |
-| `semantica.kg` | Knowledge graphs, graph algorithms, embeddings, link prediction, provenance |
-| `semantica.semantic_extract` | Entity & relation extraction, deduplication, triplet generation |
-| `semantica.reasoning` | Forward chaining, Rete, deductive, abductive, SPARQL reasoning |
-| `semantica.vector_store` | FAISS, in-memory, hybrid search with custom similarity weights |
-| `semantica.export` | RDF (Turtle/JSON-LD/N-Triples/XML), Parquet, ArangoDB AQL |
-| `semantica.ingest` | Files (PDF, DOCX, CSV, HTML), web crawl, databases, Docling |
-| `semantica.ontology` | OWL generation, import (OWL/RDF/Turtle/JSON-LD), validation |
-| `semantica.pipeline` | Parallel workers, validation, failure handling, retry policies |
+| `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search, policy engine |
+| `semantica.kg` | Knowledge graph construction, graph algorithms, centrality, community detection, embeddings, link prediction, provenance |
+| `semantica.semantic_extract` | NER, relation extraction, event extraction, coreference, triplet generation, LLM-enhanced extraction |
+| `semantica.reasoning` | Forward chaining, Rete network, deductive, abductive, SPARQL reasoning, explanation generation |
+| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory; hybrid & filtered search |
+| `semantica.export` | RDF (Turtle/JSON-LD/N-Triples/XML), Parquet, ArangoDB AQL, CSV, YAML, OWL, graph formats |
+| `semantica.ingest` | Files (PDF, DOCX, CSV, HTML), web crawl, feeds, databases, Snowflake, MCP, email, repositories |
+| `semantica.ontology` | Auto-generation (6-stage pipeline), OWL/RDF export, import (OWL/RDF/Turtle/JSON-LD), validation, versioning |
+| `semantica.pipeline` | Pipeline DSL, parallel workers, validation, retry policies, failure handling, resource scheduling |
+| `semantica.graph_store` | Graph database backends — Neo4j, FalkorDB, Apache AGE, Amazon Neptune; Cypher queries |
+| `semantica.embeddings` | Text embedding generation — Sentence-Transformers, FastEmbed, OpenAI, BGE; similarity calculation |
+| `semantica.deduplication` | Entity deduplication, similarity scoring, merging, clustering; blocking and semantic strategies |
+| `semantica.provenance` | W3C PROV-O compliant end-to-end lineage tracking, source attribution, audit trails |
+| `semantica.parse` | Document parsing — PDF, DOCX, PPTX, HTML, code, email, structured data, media with OCR |
+| `semantica.split` | Document chunking — recursive, semantic, entity-aware, relation-aware, graph-based, ontology-aware |
+| `semantica.normalize` | Data normalization for text, entities, dates, numbers, quantities, languages, encodings |
+| `semantica.conflicts` | Multi-source conflict detection (value, type, relationship, temporal, logical) with resolution strategies |
+| `semantica.change_management` | Version storage, change tracking, checksums, audit trails, compliance support for KGs and ontologies |
+| `semantica.triplet_store` | RDF triplet store integration — Blazegraph, Jena, RDF4J; SPARQL queries and bulk loading |
+| `semantica.visualization` | Interactive and static visualization of KGs, ontologies, embeddings, analytics, and temporal graphs |
+| `semantica.seed` | Seed data management for initial KG construction from CSV, JSON, databases, and APIs |
+| `semantica.core` | Framework orchestration, configuration management, knowledge base construction, plugin system |
+| `semantica.llms` | LLM provider integrations — Groq, OpenAI, HuggingFace, LiteLLM |
+| `semantica.utils` | Shared utilities — logging, validation, exception handling, constants, types, progress tracking |
---
@@ -252,6 +268,7 @@ influence = context.analyze_decision_influence(decision_id)
- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
- **Apache AGE** — PostgreSQL graph extension backend (openCypher via SQL)
+- **Snowflake** — Native ingestion with `SnowflakeIngestor`; table/query ingestion, pagination, key-pair & OAuth auth
- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
> **Built for environments where every answer must be explainable and governed.**
@@ -515,6 +532,39 @@ docling = DoclingParser()
parsed = docling.parse("financial_report.pdf")
```
+```python
+from semantica.ingest import SnowflakeIngestor
+
+# Connect to Snowflake and ingest a table
+ingestor = SnowflakeIngestor(
+ account="myorg-myaccount",
+ user="analyst",
+ password="...",
+ warehouse="COMPUTE_WH",
+ database="ANALYTICS",
+ schema="PUBLIC",
+)
+
+# Ingest a table with optional filtering and pagination
+data = ingestor.ingest_table(
+ table_name="customer_events",
+ where="event_date >= '2024-01-01'",
+ limit=10000,
+)
+
+# Or run a custom SQL query
+data = ingestor.ingest_query(
+ query="SELECT id, content, tags FROM knowledge_base WHERE active = TRUE",
+ batch_size=500,
+)
+
+# Convert to Semantica documents for downstream pipeline use
+docs = ingestor.export_as_documents(data, id_field="id", text_fields=["content"])
+
+# Key-pair and OAuth auth are also supported via env vars:
+# SNOWFLAKE_PRIVATE_KEY_PATH, SNOWFLAKE_TOKEN, SNOWFLAKE_AUTHENTICATOR
+```
+
---
## Export
@@ -593,8 +643,16 @@ ontology = importer.load("context.jsonld")
**Vector Databases**
- FAISS — high-performance dense vector search
+- Pinecone — serverless and pod-based managed vector database (`pip install semantica[vectorstore-pinecone]`)
+- Weaviate — GraphQL-based vector store with rich schema management (`pip install semantica[vectorstore-weaviate]`)
+- Qdrant — collection-based store with payload filtering (`pip install semantica[vectorstore-qdrant]`)
+- Milvus — scalable store with partition support and multiple index types (`pip install semantica[vectorstore-milvus]`)
+- PgVector — PostgreSQL pgvector extension with JSONB metadata (`pip install semantica[vectorstore-pgvector]`)
- In-memory — lightweight, zero-dependency store for development and testing
+**Data Sources**
+- Snowflake — `SnowflakeIngestor` for table/query ingestion, schema introspection, pagination, and multiple auth methods (password, key-pair, OAuth, SSO) (`pip install semantica[db-snowflake]`)
+
**Document Parsing**
- Docling — PDF, DOCX, PPTX, XLSX with table and layout extraction
@@ -618,6 +676,16 @@ pip install semantica
# With all optional dependencies
pip install semantica[all]
+# Vector store backends (install only what you need)
+pip install semantica[vectorstore-pinecone]
+pip install semantica[vectorstore-weaviate]
+pip install semantica[vectorstore-qdrant]
+pip install semantica[vectorstore-milvus]
+pip install semantica[vectorstore-pgvector]
+
+# Snowflake ingestion
+pip install semantica[db-snowflake]
+
# From source
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica