A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.
feat: add GraphRAG Knowledge System using Microsoft GraphRAG methodology
Tiioluwanied4e98ba0e35
diff --git a/rag_apps/graphrag_knowledge_system/.env.example b/rag_apps/graphrag_knowledge_system/.env.examplenew file mode 100644index 0000000..192619e--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/.env.example@@ -0,0 +1,4 @@+MISTRAL_API_KEY=your_mistral_api_key_here++# Optional — defaults to http://localhost:11434+# OLLAMA_BASE_URL=http://localhost:11434diff --git a/rag_apps/graphrag_knowledge_system/README.md b/rag_apps/graphrag_knowledge_system/README.mdnew file mode 100644index 0000000..ad6e763--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/README.md@@ -0,0 +1,169 @@+# GraphRAG Knowledge System++> Upload documents, build a local knowledge graph, and ask complex questions answered by Mistral AI with entity and relationship context.++## Overview++GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully local Streamlit application. Documents are chunked and processed through an entity-extraction pipeline powered by Mistral Small 4. Extracted entities and relationships are stored in a NetworkX graph persisted to disk, while text embeddings are stored in ChromaDB using Ollama's `nomic-embed-text` model. At query time the app supports two retrieval modes — **Local Search** for pinpointing specific entities and **Global Search** for synthesising broad themes across the entire graph — before generating a grounded answer with supporting knowledge graph context.++## Demo++++## Features++- **Document ingestion** — upload PDF or TXT files; documents are chunked, entity-extracted, and indexed automatically+- **GraphRAG entity extraction** — extracts typed entities (Person, Organization, Location, Technology, Concept, Event) and weighted relationships from every chunk using Mistral+- **Local Knowledge Graph** — NetworkX `DiGraph` persisted as JSON; no external database required+- **Local embeddings** — `nomic-embed-text` via Ollama runs entirely on your machine; no embedding API key needed+- **Local vector store** — ChromaDB stores chunk embeddings with persistent local storage+- **Dual search modes** — Local Search for entity-centric retrieval, Global Search for community-level thematic synthesis+- **Document caching** — SHA-256 file hashing prevents re-indexing the same file on every run+- **Community detection** — automatic Louvain community detection groups related entities and generates community summaries for Global Search++## Tech Stack++| Component | Technology |+|---|---|+| Graph Indexing & Retrieval | Microsoft GraphRAG methodology + NetworkX |+| Document Ingestion | LangChain |+| LLM | Mistral Small 4 (`mistral-small-latest`) via Mistral AI API |+| Embeddings | `nomic-embed-text` via Ollama (local) |+| Graph Store | NetworkX DiGraph — persisted to `graph_store.json` |+| Vector Database | ChromaDB (local persistent storage in `chroma_db/`) |+| UI | Streamlit |++## Prerequisites++- Python 3.10 or later+- [Ollama](https://ollama.com/download) installed and running+- A [Mistral AI API key](https://platform.mistral.ai)++## Installation++**1. Clone the repository**++```bash+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git+cd Hands-On-AI-Engineering/rag_apps/graphrag_knowledge_system+```++**2. Create and activate a virtual environment**++*Windows*+```bash+python -m venv .venv+.venv\Scripts\activate+```++*macOS / Linux*+```bash+python -m venv .venv+source .venv/bin/activate+```++**3. Install dependencies**++```bash+pip install -r requirements.txt+```++**4. Pull the embedding model**++```bash+ollama pull nomic-embed-text+```++**5. Configure environment variables**++```bash+cp .env.example .env+```++Open `.env` and add your Mistral API key (see [Environment Variables](#environment-variables)).++## Usage++**Start Ollama** (if it is not already running)++```bash+ollama serve+```++**Launch the app**++```bash+streamlit run app.py+```++Open [http://localhost:8501](http://localhost:8501) in your browser.++**Workflow**++1. Upload one or more PDF or TXT files using the sidebar file uploader+2. Click **⚙️ Process Documents** — the app chunks each file, extracts entities and relationships, builds the knowledge graph, and indexes embeddings+3. Type a question in the main panel and choose a search mode:+ - **🔍 Local Search** — best for specific questions about named entities, events, or facts+ - **🌐 Global Search** — best for broad thematic questions that span the whole document set+4. Click **🚀 Ask** — the answer appears as a paragraph with supporting entities and relationships listed below++**Example**++*Question:* `What role does transformer architecture play in modern language models?`++*Local Search output*+```text+Transformer architecture is the foundational design behind modern language models such as GPT+and BERT. It introduces a self-attention mechanism that allows the model to weigh the relevance+of every token relative to every other token in a sequence, enabling parallelised training on+large corpora. Positional encodings preserve sequence order without recurrence, while multi-head+attention captures multiple relationship types simultaneously.++Supporting entities: Transformer, Self-Attention, GPT, BERT, Positional Encoding+Key relationships: Transformer → GPT (weight 0.92), Transformer → BERT (weight 0.89)+```++*Global Search output*+```text+Across the document set, transformer-based architectures emerge as the central theme linking+advances in natural language understanding, code generation, and multimodal reasoning.+Community analysis reveals three clusters: foundational architecture research, fine-tuning and+alignment techniques, and real-world deployment optimisations. The transformer's attention+mechanism is the most-referenced concept across all communities, forming the backbone of every+major model discussed.+```++## Environment Variables++Copy `.env.example` to `.env` and fill in the value below.++| Variable | Required | Description |+|---|---|---|+| `MISTRAL_API_KEY` | Yes | Your Mistral AI API key — get one at [platform.mistral.ai](https://platform.mistral.ai) |++```env+MISTRAL_API_KEY=your_mistral_api_key_here+```++> `OLLAMA_BASE_URL` defaults to `http://localhost:11434` and does not need to be set unless you run Ollama on a different host or port.++## Project Structure++```text+graphrag-knowledge-system/+├── app.py # Streamlit UI, ingestion pipeline, community detection+├── src/+│ ├── cache_manager.py # SHA-256 file-hash cache (cache.json)+│ ├── document_processor.py # LangChain PDF/TXT loader and text splitter+│ ├── entity_extractor.py # GraphRAG-style entity and relationship extraction via Mistral+│ ├── graph_store.py # NetworkX DiGraph store persisted to graph_store.json+│ ├── llm_client.py # Mistral AI chat client with retry logic+│ ├── search_engine.py # Local search and Global search implementations+│ └── vector_store.py # ChromaDB collection with Ollama embedding function+├── chroma_db/ # Auto-created — ChromaDB persistent storage+├── graph_store.json # Auto-created — serialised NetworkX graph+├── cache.json # Auto-created — processed file hashes+├── requirements.txt+├── .env.example+└── README.md+```diff --git a/rag_apps/graphrag_knowledge_system/app.py b/rag_apps/graphrag_knowledge_system/app.pynew file mode 100644index 0000000..bf67b21--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/app.py@@ -0,0 +1,547 @@+import uuid+import json+import os++import networkx as nx+import streamlit as st+from dotenv import load_dotenv++load_dotenv(override=True)++from src.cache_manager import CacheManager+from src.document_processor import DocumentProcessor+from src.entity_extractor import EntityExtractor+from src.graph_store import GraphStore+from src.llm_client import LLMClient+from src.search_engine import SearchEngine+from src.vector_store import VectorStore++# ── Page config ───────────────────────────────────────────────────────────────++st.set_page_config(+ page_title="GraphRAG Knowledge System",+ page_icon="🕸️",+ layout="wide",+ initial_sidebar_state="expanded",+)++st.markdown(+ """+ <style>+ .main-header {font-size: 2rem; font-weight: 700; color: #1f77b4;}+ .sub-header {font-size: 1.1rem; color: #555; margin-bottom: 1.5rem;}+ .entity-card {+ background: #f0f7ff;+ border-left: 4px solid #1f77b4;+ padding: 0.6rem 1rem;+ margin: 0.3rem 0;+ border-radius: 4px;+ }+ .rel-card {+ background: #fff8f0;+ border-left: 4px solid #ff7f0e;+ padding: 0.6rem 1rem;+ margin: 0.3rem 0;+ border-radius: 4px;+ }+ .answer-box {+ background: #f8fff8;+ border: 1px solid #2ca02c;+ border-radius: 8px;+ padding: 1.2rem;+ margin-top: 1rem;+ }+ .badge {+ display: inline-block;+ padding: 2px 8px;+ border-radius: 12px;+ font-size: 0.75rem;+ font-weight: 600;+ margin-left: 6px;+ }+ </style>+ """,+ unsafe_allow_html=True,+)++# ── Cached component initialisation ──────────────────────────────────────────++[email protected]_resource(show_spinner=False)+def get_components():+ graph_store = GraphStore()+ vector_store = VectorStore()+ entity_extractor = EntityExtractor()+ search_engine = SearchEngine(graph_store, vector_store)+ cache_manager = CacheManager()+ doc_processor = DocumentProcessor()+ llm_client = LLMClient()+ return (+ graph_store,+ vector_store,+ entity_extractor,+ search_engine,+ cache_manager,+ doc_processor,+ llm_client,+ )+++# ── Community detection + summarisation ──────────────────────────────────────++COMMUNITY_SUMMARY_PROMPT = """You are summarizing a group of related entities from a knowledge graph.++Entity group ({count} entities):+{entities}++Write a 2–4 sentence summary describing:+1. What this group of entities represents as a theme or domain+2. Key characteristics and relationships within this group+3. Why these entities are connected++Summary:"""+++def run_community_detection(+ graph_store: GraphStore, llm_client: LLMClient, status_fn+) -> int:+ full_graph = graph_store.get_full_graph()+ entities = full_graph["entities"]+ relationships = full_graph["relationships"]++ if len(entities) < 2:+ return 0++ G = nx.Graph()+ G.add_nodes_from(e["name"] for e in entities)+ for r in relationships:+ G.add_edge(r["source"], r["target"], weight=r.get("weight", 0.5))++ # Remove isolated nodes for cleaner communities+ G_connected = G.copy()+ isolated = [n for n in G_connected.nodes if G_connected.degree(n) == 0]+ G_connected.remove_nodes_from(isolated)++ if G_connected.number_of_nodes() < 2:+ return 0++ communities_raw = list(+ nx.algorithms.community.greedy_modularity_communities(G_connected)+ )++ community_count = 0+ for i, comm in enumerate(communities_raw):+ comm_list = list(comm)+ if len(comm_list) < 2:+ continue++ # Assign community id to entities+ cid = f"community_{i}"+ for name in comm_list:+ graph_store.set_entity_community(name, cid)++ # Build summary prompt using entity descriptions+ descs = graph_store.get_entity_descriptions(comm_list[:15])+ entity_lines = "\n".join(+ f"- {d['name']} ({d.get('type','OTHER')}): {d.get('description','')}"+ for d in descs+ )++ status_fn(f"Summarising community {i + 1}/{len(communities_raw)}…")++ try:+ summary = llm_client.complete(+ COMMUNITY_SUMMARY_PROMPT.format(+ count=len(comm_list), entities=entity_lines+ )+ )+ except Exception:+ summary = f"Community of {len(comm_list)} entities: " + ", ".join(+ comm_list[:5]+ )++ graph_store.create_community(cid, comm_list, summary, level=0)+ community_count += 1++ return community_count+++# ── Document ingestion pipeline ───────────────────────────────────────────────+++def ingest_document(+ file_content: bytes,+ filename: str,+ graph_store: GraphStore,+ vector_store: VectorStore,+ entity_extractor: EntityExtractor,+ doc_processor: DocumentProcessor,+ cache_manager: CacheManager,+ llm_client: LLMClient,+) -> dict:+ file_hash = cache_manager.file_hash(file_content)++ if cache_manager.is_processed(file_hash):+ return {+ "status": "cached",+ "message": f"'{filename}' was already processed — skipping.",+ }++ # ── Step 1: Load and chunk ────────────────────────────────────────────────+ progress_bar = st.progress(0, text="Loading and chunking document…")++ try:+ chunks = doc_processor.process(file_content, filename)+ except Exception as e:+ progress_bar.empty()+ return {"status": "error", "message": f"Document loading failed: {e}"}++ if not chunks:+ progress_bar.empty()+ return {"status": "error", "message": "No text could be extracted from the document."}++ doc_id = str(uuid.uuid4())+ total_chunks = len(chunks)++ # ── Step 2: Create document node ─────────────────────────────────────────+ graph_store.create_document(doc_id, filename, file_hash)+ progress_bar.progress(0.05, text=f"Indexed document node. Processing {total_chunks} chunks…")++ # ── Step 3: Extract entities per chunk ───────────────────────────────────+ all_entity_names_per_chunk: dict[str, list] = {}++ for idx, chunk in enumerate(chunks):+ chunk_id = f"{doc_id}_chunk_{idx}"+ chunk_text = chunk.page_content++ # Store chunk in Neo4j+ graph_store.create_chunk(chunk_id, chunk_text, doc_id)++ # Extract entities and relationships+ extraction = entity_extractor.extract(chunk_text, chunk_id)+ chunk_entity_names = []++ for entity in extraction["entities"]:+ graph_store.upsert_entity(+ entity["name"],+ entity["type"],+ entity["description"],+ chunk_id,+ )+ chunk_entity_names.append(entity["name"])++ for rel in extraction["relationships"]:+ graph_store.upsert_relationship(+ rel["source"], rel["target"], rel["description"], rel["weight"]+ )++ all_entity_names_per_chunk[chunk_id] = chunk_entity_names++ # Store chunk in ChromaDB with entity metadata+ vector_store.add_chunk(+ chunk_id,+ chunk_text,+ {+ "document_id": doc_id,+ "filename": filename,+ "chunk_index": str(idx),+ "entity_names": "|".join(chunk_entity_names),+ },+ )++ pct = 0.05 + 0.75 * ((idx + 1) / total_chunks)+ progress_bar.progress(+ pct,+ text=f"Extracting entities: chunk {idx + 1}/{total_chunks} "+ f"({len(chunk_entity_names)} entities found)…",+ )++ # ── Step 4: Community detection ───────────────────────────────────────────+ progress_bar.progress(0.82, text="Running community detection on knowledge graph…")++ n_communities = run_community_detection(+ graph_store,+ llm_client,+ lambda msg: progress_bar.progress(0.85, text=msg),+ )++ # ── Step 5: Finalise ──────────────────────────────────────────────────────+ progress_bar.progress(1.0, text="Done!")+ cache_manager.mark_processed(file_hash, filename, doc_id, total_chunks)+ progress_bar.empty()++ return {+ "status": "success",+ "message": (+ f"Processed **{filename}**: {total_chunks} chunks, "+ f"{n_communities} communities detected."+ ),+ "doc_id": doc_id,+ "chunks": total_chunks,+ "communities": n_communities,+ }+++# ── UI helpers ────────────────────────────────────────────────────────────────+++TYPE_COLORS = {+ "PERSON": "#4c72b0",+ "ORGANIZATION": "#dd8452",+ "LOCATION": "#55a868",+ "TECHNOLOGY": "#c44e52",+ "CONCEPT": "#8172b2",+ "EVENT": "#937860",+}+++def _type_badge(entity_type: str) -> str:+ color = TYPE_COLORS.get(entity_type, "#888")+ return (+ f'<span style="background:{color};color:#fff;padding:2px 10px;'+ f'border-radius:10px;font-size:0.72rem;font-weight:600;">'+ f"{entity_type}</span>"+ )+++def render_entity_list(entities: list):+ if not entities:+ st.caption("No entities retrieved.")+ return+ for e in entities[:20]:+ name = e.get("name", "")+ etype = e.get("type", "OTHER")+ desc = (e.get("description") or "")[:200]+ col_name, col_tag = st.columns([4, 1])+ with col_name:+ st.markdown(f"**{name}**")+ if desc:+ st.caption(desc)+ with col_tag:+ st.markdown(_type_badge(etype), unsafe_allow_html=True)+ st.divider()+++def render_relationship_list(relationships: list):+ if not relationships:+ st.caption("No relationships retrieved.")+ return+ for r in relationships[:15]:+ source = r.get("source", "")+ target = r.get("target", "")+ desc = (r.get("description") or "")[:200]+ weight = r.get("weight", 0.5)+ st.markdown(f"**{source}** → **{target}** `weight {weight:.2f}`", unsafe_allow_html=True)+ if desc:+ st.caption(desc)+ st.divider()+++# ── Main layout ───────────────────────────────────────────────────────────────+++def main():+ # ── Header ────────────────────────────────────────────────────────────────+ st.markdown(+ '<div class="main-header">🕸️ GraphRAG Knowledge System</div>', unsafe_allow_html=True+ )+ st.markdown(+ '<div class="sub-header">Upload documents → Build a knowledge graph → Ask complex questions</div>',+ unsafe_allow_html=True,+ )++ # ── Component init ────────────────────────────────────────────────────────+ try:+ (+ graph_store,+ vector_store,+ entity_extractor,+ search_engine,+ cache_manager,+ doc_processor,+ llm_client,+ ) = get_components()+ except Exception as e:+ st.error(f"Failed to initialise components: {e}")+ st.info(+ "Check your `.env` file — make sure NEO4J_URI, NEO4J_PASSWORD, "+ "MISTRAL_API_KEY are set and Ollama is running on http://localhost:11434"+ )+ get_components.clear()+ return++ # Auto-clear stale cache when the GraphStore class has been updated since+ # the resource was last cached (e.g. after a hot-reload added new methods).+ if not hasattr(graph_store, "ping"):+ get_components.clear()+ st.rerun()++ # ── Graph store health check ──────────────────────────────────────────────+ db_ok, db_msg = graph_store.ping()+ if not db_ok:+ st.error(f"Graph store unavailable: {db_msg}")+ get_components.clear()+ return++ # ── Sidebar ───────────────────────────────────────────────────────────────+ with st.sidebar:+ st.header("📄 Document Ingestion")++ uploaded_files = st.file_uploader(+ "Upload PDF or TXT files",+ type=["pdf", "txt"],+ accept_multiple_files=True,+ help="Documents are chunked, entity-extracted, and stored in Neo4j + ChromaDB.",+ )++ if uploaded_files and st.button("⚙️ Process Documents", type="primary", use_container_width=True):+ for uf in uploaded_files:+ st.subheader(f"Processing: {uf.name}")+ content = uf.read()+ result = ingest_document(+ content,+ uf.name,+ graph_store,+ vector_store,+ entity_extractor,+ doc_processor,+ cache_manager,+ llm_client,+ )+ if result["status"] == "success":+ st.success(result["message"])+ elif result["status"] == "cached":+ st.info(result["message"])+ else:+ st.error(result["message"])++ st.divider()+ st.subheader("📋 Processed Documents")+ processed = cache_manager.get_all()+ if processed:+ for doc in processed:+ with st.expander(doc["filename"]):+ st.write(f"**Chunks:** {doc['chunk_count']}")+ st.write(f"**Processed:** {doc['processed_at'][:19]}")+ else:+ st.caption("No documents processed yet.")++ st.divider()+ st.subheader("📊 Graph Stats")+ col1, col2 = st.columns(2)+ with col1:+ try:+ st.metric("Chunks", vector_store.count())+ except Exception:+ st.metric("Chunks", "—")+ with col2:+ try:+ communities = graph_store.get_all_communities()+ st.metric("Communities", len(communities))+ except Exception:+ st.metric("Communities", "—")++ if st.button("🔄 Refresh Stats", use_container_width=True):+ st.rerun()++ if st.button("♻️ Reset App Cache", use_container_width=True, help="Force-reinitialise all components (use after code changes or connection errors)"):+ get_components.clear()+ st.rerun()++ # ── Main panel ────────────────────────────────────────────────────────────+ col_query, col_mode = st.columns([4, 1])++ with col_mode:+ search_mode = st.radio(+ "Search mode",+ ["🔍 Local", "🌐 Global"],+ help=(+ "**Local**: pinpoint entities in matching chunks. "+ "**Global**: synthesise across whole-graph communities."+ ),+ )++ with col_query:+ question = st.text_area(+ "Ask a question about your documents",+ placeholder="e.g. 'What are the key relationships between X and Y?' or 'What themes emerge across the documents?'",+ height=100,+ )++ ask_col, _ = st.columns([1, 4])+ with ask_col:+ ask = st.button("🚀 Ask", type="primary", use_container_width=True)++ if ask and question.strip():+ with st.spinner("Retrieving graph context and generating answer…"):+ try:+ if "Local" in search_mode:+ result = search_engine.local_search(question.strip())+ else:+ result = search_engine.global_search(question.strip())+ except Exception as e:+ st.error(f"Search failed: {e}")+ return++ # ── Answer ────────────────────────────────────────────────────────────+ mode_label = "Local Search" if "Local" in search_mode else "Global Search"+ st.markdown(f"### Answer — {mode_label}")+ st.markdown(result["answer"])++ # ── Supporting context ────────────────────────────────────────────────+ entities = result.get("entities", [])+ relationships = result.get("relationships", [])+ source_chunks = result.get("source_chunks", [])++ if entities or relationships:+ st.markdown("---")+ st.markdown("#### Supporting Knowledge Graph Context")+ col_ent, col_rel = st.columns(2)+ with col_ent:+ st.markdown(f"**Entities ({len(entities)})**")+ render_entity_list(entities)+ with col_rel:+ st.markdown(f"**Relationships ({len(relationships)})**")+ render_relationship_list(relationships)++ if source_chunks:+ with st.expander(f"📄 Source Excerpts ({len(source_chunks)})"):+ for i, chunk in enumerate(source_chunks):+ st.markdown(f"**Excerpt {i + 1}**")+ st.write(chunk)+ if i < len(source_chunks) - 1:+ st.divider()+ elif "Global" in search_mode and result.get("communities_used", 0):+ st.caption(+ f"Global search synthesised across {result['communities_used']} communities."+ )++ elif ask and not question.strip():+ st.warning("Please enter a question before clicking Ask.")++ # ── Welcome state ─────────────────────────────────────────────────────────+ if not ask:+ st.markdown("---")+ info_col1, info_col2, info_col3 = st.columns(3)+ with info_col1:+ st.info(+ "**1. Upload Documents**\n\n"+ "Use the sidebar to upload PDF or TXT files. "+ "Each document is chunked and entities are extracted."+ )+ with info_col2:+ st.info(+ "**2. Knowledge Graph Built**\n\n"+ "Entities and relationships are stored in Neo4j. "+ "Text embeddings go into ChromaDB via Ollama."+ )+ with info_col3:+ st.info(+ "**3. Ask Questions**\n\n"+ "Use **Local** search for specific entity lookups, "+ "or **Global** search for broad thematic questions."+ )+++if __name__ == "__main__":+ main()diff --git a/rag_apps/graphrag_knowledge_system/assets/demo.png b/rag_apps/graphrag_knowledge_system/assets/demo.pngnew file mode 100644index 0000000..c25113eBinary files /dev/null and b/rag_apps/graphrag_knowledge_system/assets/demo.png differdiff --git a/rag_apps/graphrag_knowledge_system/requirements.txt b/rag_apps/graphrag_knowledge_system/requirements.txtnew file mode 100644index 0000000..9556e8b--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/requirements.txt@@ -0,0 +1,14 @@+streamlit>=1.32.0+python-dotenv>=1.0.0+langchain>=0.2.0+langchain-community>=0.2.0+pypdf>=4.2.0+chromadb>=0.5.3+langchain-mistralai>=0.2.0+requests>=2.31.0+numpy>=1.26.0+networkx>=3.2.0+pandas>=2.2.0+tiktoken>=0.7.0+langchain-text-splitters>=0.3.0+langchain-ollama>=0.2.0diff --git a/rag_apps/graphrag_knowledge_system/src/__init__.py b/rag_apps/graphrag_knowledge_system/src/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/rag_apps/graphrag_knowledge_system/src/cache_manager.py b/rag_apps/graphrag_knowledge_system/src/cache_manager.pynew file mode 100644index 0000000..247fcfe--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/cache_manager.py@@ -0,0 +1,54 @@+import hashlib+import json+from datetime import datetime+from pathlib import Path+++class CacheManager:+ def __init__(self, cache_path: str = "./cache.json"):+ self.cache_path = Path(cache_path)+ self._load()++ def _load(self):+ if self.cache_path.exists():+ with open(self.cache_path, "r") as f:+ self.cache = json.load(f)+ else:+ self.cache = {}++ def _save(self):+ with open(self.cache_path, "w") as f:+ json.dump(self.cache, f, indent=2)++ def file_hash(self, content: bytes) -> str:+ return hashlib.sha256(content).hexdigest()++ def is_processed(self, file_hash: str) -> bool:+ return file_hash in self.cache++ def mark_processed(+ self, file_hash: str, filename: str, doc_id: str, chunk_count: int+ ):+ self.cache[file_hash] = {+ "filename": filename,+ "document_id": doc_id,+ "chunk_count": chunk_count,+ "processed_at": datetime.now().isoformat(),+ }+ self._save()++ def get_all(self) -> list:+ return [+ {+ "filename": v["filename"],+ "document_id": v["document_id"],+ "chunk_count": v["chunk_count"],+ "processed_at": v["processed_at"],+ }+ for v in self.cache.values()+ ]++ def remove(self, file_hash: str):+ if file_hash in self.cache:+ del self.cache[file_hash]+ self._save()diff --git a/rag_apps/graphrag_knowledge_system/src/document_processor.py b/rag_apps/graphrag_knowledge_system/src/document_processor.pynew file mode 100644index 0000000..85267ae--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/document_processor.py@@ -0,0 +1,36 @@+import os+import tempfile+from pathlib import Path++from langchain_text_splitters import RecursiveCharacterTextSplitter+from langchain_community.document_loaders import PyPDFLoader, TextLoader+++class DocumentProcessor:+ def __init__(self, chunk_size: int = 800, chunk_overlap: int = 150):+ self.splitter = RecursiveCharacterTextSplitter(+ chunk_size=chunk_size,+ chunk_overlap=chunk_overlap,+ separators=["\n\n", "\n", ". ", " ", ""],+ )++ def process(self, file_content: bytes, filename: str) -> list:+ suffix = Path(filename).suffix.lower()+ if suffix not in {".pdf", ".txt"}:+ raise ValueError(f"Unsupported file type: {suffix}. Use PDF or TXT.")++ with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:+ tmp.write(file_content)+ tmp_path = tmp.name++ try:+ if suffix == ".pdf":+ loader = PyPDFLoader(tmp_path)+ else:+ loader = TextLoader(tmp_path, encoding="utf-8")++ docs = loader.load()+ chunks = self.splitter.split_documents(docs)+ return chunks+ finally:+ os.unlink(tmp_path)diff --git a/rag_apps/graphrag_knowledge_system/src/entity_extractor.py b/rag_apps/graphrag_knowledge_system/src/entity_extractor.pynew file mode 100644index 0000000..8045e72--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/entity_extractor.py@@ -0,0 +1,83 @@+import json+from src.llm_client import LLMClient++EXTRACTION_PROMPT = """You are an expert information extractor implementing the GraphRAG methodology.++Analyze the text and extract ALL significant entities and relationships.++Return ONLY a valid JSON object with this exact structure:+{{+ "entities": [+ {{+ "name": "Exact entity name, properly capitalized",+ "type": "PERSON|ORGANIZATION|LOCATION|CONCEPT|EVENT|TECHNOLOGY|PRODUCT|OTHER",+ "description": "Detailed description based on what the text says about this entity"+ }}+ ],+ "relationships": [+ {{+ "source": "Source entity name (must match an extracted entity)",+ "target": "Target entity name (must match an extracted entity)",+ "description": "Precise description of the relationship between source and target",+ "weight": 0.8+ }}+ ]+}}++Extraction rules:+- Extract 3–12 meaningful, specific entities; skip generic words+- Every relationship source and target MUST be an entity you extracted+- weight is 0.0–1.0 reflecting relationship strength or importance+- Return only valid JSON — no markdown fences, no extra text++Text to analyze:+{text}"""+++class EntityExtractor:+ def __init__(self):+ self.llm = LLMClient()++ def extract(self, text: str, chunk_id: str) -> dict:+ prompt = EXTRACTION_PROMPT.format(text=text[:3000])++ try:+ raw = self.llm.complete(prompt, json_mode=True)+ data = json.loads(raw)+ except Exception:+ return {"entities": [], "relationships": []}++ entities = []+ for e in data.get("entities", []):+ name = (e.get("name") or "").strip()+ etype = (e.get("type") or "OTHER").strip()+ if name:+ entities.append(+ {+ "name": name,+ "type": etype,+ "description": (e.get("description") or "").strip(),+ "chunk_id": chunk_id,+ }+ )++ entity_names = {e["name"] for e in entities}+ relationships = []+ for r in data.get("relationships", []):+ src = (r.get("source") or "").strip()+ tgt = (r.get("target") or "").strip()+ if src and tgt and src in entity_names and tgt in entity_names:+ try:+ weight = float(r.get("weight", 0.5))+ except (TypeError, ValueError):+ weight = 0.5+ relationships.append(+ {+ "source": src,+ "target": tgt,+ "description": (r.get("description") or "").strip(),+ "weight": max(0.0, min(1.0, weight)),+ }+ )++ return {"entities": entities, "relationships": relationships}diff --git a/rag_apps/graphrag_knowledge_system/src/graph_store.py b/rag_apps/graphrag_knowledge_system/src/graph_store.pynew file mode 100644index 0000000..bb53cda--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/graph_store.py@@ -0,0 +1,238 @@+import json+from pathlib import Path+from typing import Dict, List, Tuple++import networkx as nx+++class GraphStore:+ """+ Local knowledge graph backed by a NetworkX DiGraph.+ Persisted to disk as JSON via node_link_data so the graph survives restarts.+ """++ def __init__(self, persist_path: str = "./graph_store.json"):+ self.persist_path = Path(persist_path)+ self.G: nx.DiGraph = self._load()++ # ── Persistence ───────────────────────────────────────────────────────────++ def _load(self) -> nx.DiGraph:+ if self.persist_path.exists():+ try:+ with open(self.persist_path, "r", encoding="utf-8") as f:+ data = json.load(f)+ return nx.node_link_graph(data, directed=True, multigraph=False)+ except Exception:+ pass+ return nx.DiGraph()++ def _save(self):+ data = nx.node_link_data(self.G)+ with open(self.persist_path, "w", encoding="utf-8") as f:+ json.dump(data, f, ensure_ascii=False)++ # ── Document ─────────────────────────────────────────────────────────────++ def create_document(self, doc_id: str, filename: str, file_hash: str):+ self.G.add_node(+ doc_id, node_type="document", filename=filename, file_hash=file_hash+ )+ self._save()++ def document_exists(self, file_hash: str) -> bool:+ return any(+ d.get("node_type") == "document" and d.get("file_hash") == file_hash+ for _, d in self.G.nodes(data=True)+ )++ # ── Chunks ────────────────────────────────────────────────────────────────++ def create_chunk(self, chunk_id: str, content: str, doc_id: str):+ self.G.add_node(chunk_id, node_type="chunk", content=content, doc_id=doc_id)+ if self.G.has_node(doc_id):+ self.G.add_edge(chunk_id, doc_id, rel_type="PART_OF")+ self._save()++ # ── Entities ──────────────────────────────────────────────────────────────++ def upsert_entity(+ self, name: str, entity_type: str, description: str, chunk_id: str+ ):+ if self.G.has_node(name) and self.G.nodes[name].get("node_type") == "entity":+ existing = self.G.nodes[name]+ if len(description) > len(existing.get("description", "")):+ existing["description"] = description+ else:+ self.G.add_node(+ name,+ node_type="entity",+ name=name,+ type=entity_type,+ description=description,+ community_id=None,+ )+ if self.G.has_node(chunk_id):+ self.G.add_edge(name, chunk_id, rel_type="APPEARS_IN")+ self._save()++ # ── Relationships ─────────────────────────────────────────────────────────++ def upsert_relationship(+ self, source: str, target: str, description: str, weight: float+ ):+ if not self.G.has_node(source) or not self.G.has_node(target):+ return+ if self.G.has_edge(source, target):+ ed = self.G[source][target]+ if ed.get("rel_type") == "RELATED_TO":+ ed["weight"] = (ed.get("weight", weight) + weight) / 2.0+ if len(description) > len(ed.get("description", "")):+ ed["description"] = description+ self._save()+ return+ self.G.add_edge(+ source, target,+ rel_type="RELATED_TO",+ description=description,+ weight=weight,+ )+ self._save()++ # ── Community ─────────────────────────────────────────────────────────────++ def set_entity_community(self, entity_name: str, community_id: str):+ # No _save() here — called in a tight loop; caller saves via create_community()+ if self.G.has_node(entity_name):+ self.G.nodes[entity_name]["community_id"] = community_id++ def create_community(+ self, community_id: str, entity_names: List[str], summary: str, level: int = 0+ ):+ self.G.add_node(+ community_id,+ node_type="community",+ summary=summary,+ level=level,+ entity_count=len(entity_names),+ )+ for name in entity_names:+ if self.G.has_node(name):+ self.G.add_edge(name, community_id, rel_type="IN_COMMUNITY")+ self._save()++ def get_all_communities(self) -> List[Dict]:+ result = []+ for node_id, data in self.G.nodes(data=True):+ if data.get("node_type") != "community":+ continue+ entity_names = [+ u+ for u, v, d in self.G.edges(data=True)+ if v == node_id and d.get("rel_type") == "IN_COMMUNITY"+ ]+ result.append(+ {+ "id": node_id,+ "summary": data.get("summary", ""),+ "level": data.get("level", 0),+ "entity_count": data.get("entity_count", 0),+ "entities": entity_names,+ }+ )+ return result++ def communities_exist(self) -> bool:+ return any(+ d.get("node_type") == "community" for _, d in self.G.nodes(data=True)+ )++ # ── Graph export for community detection ─────────────────────────────────++ def get_full_graph(self) -> Dict:+ entities = [+ {"name": n, "type": d.get("type", "")}+ for n, d in self.G.nodes(data=True)+ if d.get("node_type") == "entity"+ ]+ relationships = [+ {"source": u, "target": v, "weight": d.get("weight", 0.5)}+ for u, v, d in self.G.edges(data=True)+ if d.get("rel_type") == "RELATED_TO"+ ]+ return {"entities": entities, "relationships": relationships}++ def get_entity_descriptions(self, names: List[str]) -> List[Dict]:+ result = []+ for name in names:+ if not self.G.has_node(name):+ continue+ d = self.G.nodes[name]+ if d.get("node_type") == "entity":+ result.append(+ {+ "name": name,+ "type": d.get("type", "OTHER"),+ "description": d.get("description", ""),+ }+ )+ return result++ # ── Search context ────────────────────────────────────────────────────────++ def get_entity_neighborhood(self, entity_names: List[str]) -> Dict:+ seen_entities: set = set()+ seen_rels: set = set()+ entities: List[Dict] = []+ relationships: List[Dict] = []+ target_set = set(entity_names)++ def _add_entity(n: str):+ if n in seen_entities or not self.G.has_node(n):+ return+ d = self.G.nodes[n]+ if d.get("node_type") != "entity":+ return+ seen_entities.add(n)+ entities.append(+ {+ "name": n,+ "type": d.get("type", "OTHER"),+ "description": d.get("description", ""),+ "community": d.get("community_id"),+ }+ )++ # Seed with the requested entities+ for name in entity_names:+ _add_entity(name)++ # Collect all RELATED_TO edges that touch any requested entity+ for u, v, d in self.G.edges(data=True):+ if d.get("rel_type") != "RELATED_TO":+ continue+ if u not in target_set and v not in target_set:+ continue+ rkey = tuple(sorted([u, v]))+ if rkey not in seen_rels:+ seen_rels.add(rkey)+ relationships.append(+ {+ "source": u,+ "target": v,+ "description": d.get("description", ""),+ "weight": d.get("weight", 0.5),+ }+ )+ _add_entity(u)+ _add_entity(v)++ return {"entities": entities, "relationships": relationships}++ # ── Helpers ───────────────────────────────────────────────────────────────++ def ping(self) -> Tuple[bool, str]:+ return True, "Connected (local NetworkX graph)"++ def close(self):+ self._save()diff --git a/rag_apps/graphrag_knowledge_system/src/llm_client.py b/rag_apps/graphrag_knowledge_system/src/llm_client.pynew file mode 100644index 0000000..3a3c7b6--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/llm_client.py@@ -0,0 +1,43 @@+import os+import time+from langchain_mistralai import ChatMistralAI+from langchain_core.messages import HumanMessage, SystemMessage+++class LLMClient:+ def __init__(self):+ api_key = os.getenv("MISTRAL_API_KEY")+ if not api_key:+ raise ValueError("MISTRAL_API_KEY not set in environment")+ self.llm = ChatMistralAI(+ model="mistral-small-latest",+ mistral_api_key=api_key,+ max_tokens=2048,+ )+ self.llm_json = ChatMistralAI(+ model="mistral-small-latest",+ mistral_api_key=api_key,+ max_tokens=2048,+ response_format={"type": "json_object"},+ )++ def complete(self, prompt: str, json_mode: bool = False, max_retries: int = 3) -> str:+ client = self.llm_json if json_mode else self.llm+ messages = [HumanMessage(content=prompt)]+ for attempt in range(max_retries):+ try:+ response = client.invoke(messages)+ return response.content+ except Exception as e:+ if attempt < max_retries - 1:+ time.sleep(2 ** attempt)+ else:+ raise RuntimeError(f"Mistral API error after {max_retries} attempts: {e}")++ def complete_with_system(+ self, system: str, user: str, json_mode: bool = False+ ) -> str:+ client = self.llm_json if json_mode else self.llm+ messages = [SystemMessage(content=system), HumanMessage(content=user)]+ response = client.invoke(messages)+ return response.contentdiff --git a/rag_apps/graphrag_knowledge_system/src/search_engine.py b/rag_apps/graphrag_knowledge_system/src/search_engine.pynew file mode 100644index 0000000..fcb02c3--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/search_engine.py@@ -0,0 +1,205 @@+import json+from typing import Dict, List++from src.graph_store import GraphStore+from src.llm_client import LLMClient+from src.vector_store import VectorStore++LOCAL_SEARCH_PROMPT = """You are an expert analyst answering questions using a knowledge graph.++## Knowledge Graph Context++### Entities+{entities}++### Relationships+{relationships}++### Source Text Excerpts+{chunks}++## Question+{question}++## Instructions+Answer the question using only the knowledge graph context above.+- Reference specific entities and relationships by name+- Be factual and precise+- If the context is insufficient, state what is known and what remains unclear++Answer:"""++COMMUNITY_RELEVANCE_PROMPT = """Evaluate whether this knowledge graph community summary helps answer the question.++Question: {question}++Community summary ({count} entities):+{summary}++Return JSON exactly:+{{"relevant": true_or_false, "relevance_score": 0.0_to_1.0, "key_points": ["point1", "point2"]}}"""++GLOBAL_ANSWER_PROMPT = """You are an expert analyst synthesizing a knowledge graph to answer a complex question.++## Community Knowledge (most relevant communities)+{community_contexts}++## Question+{question}++## Instructions+Provide a comprehensive answer that:+1. Synthesizes information across the relevant communities+2. Identifies overarching themes and patterns+3. Names specific entities where helpful+4. Acknowledges gaps if context is insufficient++Answer:"""+++class SearchEngine:+ def __init__(self, graph_store: GraphStore, vector_store: VectorStore):+ self.graph = graph_store+ self.vector = vector_store+ self.llm = LLMClient()++ # ── Local Search ──────────────────────────────────────────────────────────++ def local_search(self, query: str, n_chunks: int = 5) -> Dict:+ if self.vector.count() == 0:+ return self._no_docs_result()++ chunk_hits = self.vector.search(query, n_results=n_chunks)+ if not chunk_hits:+ return self._no_docs_result()++ # Collect entity names from chunk metadata+ entity_names: List[str] = []+ for hit in chunk_hits:+ raw = hit.get("metadata", {}).get("entity_names", "")+ if raw:+ entity_names.extend(raw.split("|"))+ entity_names = list(set(filter(None, entity_names)))[:20]++ graph_ctx = {"entities": [], "relationships": []}+ if entity_names:+ graph_ctx = self.graph.get_entity_neighborhood(entity_names)++ entities_text = (+ "\n".join(+ f"- **{e['name']}** ({e['type']}): {e['description']}"+ for e in graph_ctx["entities"]+ )+ or "No entity context available"+ )+ rels_text = (+ "\n".join(+ f"- {r['source']} → {r['target']}: {r['description']}"+ for r in graph_ctx["relationships"]+ )+ or "No relationship context available"+ )+ chunks_text = "\n\n".join(+ f"[Excerpt {i + 1}]:\n{hit['content'][:600]}"+ for i, hit in enumerate(chunk_hits)+ )++ answer = self.llm.complete(+ LOCAL_SEARCH_PROMPT.format(+ entities=entities_text,+ relationships=rels_text,+ chunks=chunks_text,+ question=query,+ )+ )++ return {+ "answer": answer,+ "entities": graph_ctx["entities"],+ "relationships": graph_ctx["relationships"],+ "source_chunks": [h["content"][:300] for h in chunk_hits],+ "mode": "local",+ }++ # ── Global Search ─────────────────────────────────────────────────────────++ def global_search(self, query: str) -> Dict:+ communities = self.graph.get_all_communities()+ if not communities:+ return self.local_search(query)++ # Rate each community's relevance+ scored = []+ for comm in communities[:25]:+ summary = (comm.get("summary") or "").strip()+ if not summary:+ continue+ try:+ raw = self.llm.complete(+ COMMUNITY_RELEVANCE_PROMPT.format(+ question=query,+ summary=summary,+ count=comm.get("entity_count", "?"),+ ),+ json_mode=True,+ )+ rating = json.loads(raw)+ if rating.get("relevant") and float(rating.get("relevance_score", 0)) > 0.25:+ scored.append(+ {+ **comm,+ "relevance_score": float(rating.get("relevance_score", 0.5)),+ "key_points": rating.get("key_points", []),+ }+ )+ except Exception:+ continue++ scored.sort(key=lambda x: x.get("relevance_score", 0), reverse=True)+ top = scored[:6]++ if not top:+ return self.local_search(query)++ community_contexts = "\n\n".join(+ f"### Community {i + 1} (relevance {c['relevance_score']:.2f})\n"+ f"Entities ({c['entity_count']}): {', '.join(c.get('entities', [])[:12])}\n"+ f"Summary: {c['summary']}\n"+ f"Key points: {', '.join(c.get('key_points', []))}"+ for i, c in enumerate(top)+ )++ answer = self.llm.complete(+ GLOBAL_ANSWER_PROMPT.format(+ community_contexts=community_contexts,+ question=query,+ )+ )++ all_entity_names = list(+ {name for c in top for name in c.get("entities", [])}+ )[:20]+ graph_ctx = {"entities": [], "relationships": []}+ if all_entity_names:+ graph_ctx = self.graph.get_entity_neighborhood(all_entity_names)++ return {+ "answer": answer,+ "entities": graph_ctx["entities"],+ "relationships": graph_ctx["relationships"],+ "communities_used": len(top),+ "source_chunks": [],+ "mode": "global",+ }++ # ── Helpers ───────────────────────────────────────────────────────────────++ @staticmethod+ def _no_docs_result() -> Dict:+ return {+ "answer": "No documents have been indexed yet. Please upload and process documents first.",+ "entities": [],+ "relationships": [],+ "source_chunks": [],+ "mode": "none",+ }diff --git a/rag_apps/graphrag_knowledge_system/src/vector_store.py b/rag_apps/graphrag_knowledge_system/src/vector_store.pynew file mode 100644index 0000000..89e3930--- /dev/null+++ b/rag_apps/graphrag_knowledge_system/src/vector_store.py@@ -0,0 +1,84 @@+import os+from typing import Any, Dict, List++import chromadb+from chromadb.api.types import Documents, EmbeddingFunction, Embeddings+from langchain_ollama import OllamaEmbeddings+++class OllamaEmbeddingFunction(EmbeddingFunction[Documents]):+ """ChromaDB EmbeddingFunction subclass backed by LangChain's OllamaEmbeddings."""++ def __init__(self):+ base_url = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")+ self._embedder = OllamaEmbeddings(+ model="nomic-embed-text",+ base_url=base_url,+ )+ self._base_url = base_url++ def __call__(self, input: Documents) -> Embeddings:+ return self._embedder.embed_documents(list(input))++ @staticmethod+ def name() -> str:+ return "ollama-nomic-embed-text"++ @staticmethod+ def build_from_config(config: Dict[str, Any]) -> "OllamaEmbeddingFunction":+ return OllamaEmbeddingFunction()++ def get_config(self) -> Dict[str, Any]:+ return {"base_url": self._base_url, "model": "nomic-embed-text"}+++class VectorStore:+ def __init__(self, persist_path: str = "./chroma_db"):+ self._embedding_fn = OllamaEmbeddingFunction()+ self.client = chromadb.PersistentClient(path=persist_path)+ self.collection = self.client.get_or_create_collection(+ name="text_chunks",+ embedding_function=self._embedding_fn,+ metadata={"hnsw:space": "cosine"},+ )++ def add_chunk(self, chunk_id: str, text: str, metadata: Dict):+ clean_meta = {k: str(v) for k, v in metadata.items()}+ try:+ self.collection.add(+ ids=[chunk_id],+ documents=[text],+ metadatas=[clean_meta],+ )+ except Exception as e:+ if "already exists" not in str(e).lower():+ raise++ def search(self, query: str, n_results: int = 5) -> List[Dict]:+ total = self.collection.count()+ if total == 0:+ return []+ k = min(n_results, total)+ results = self.collection.query(query_texts=[query], n_results=k)+ hits = []+ if results["ids"] and results["ids"][0]:+ for i, doc_id in enumerate(results["ids"][0]):+ hits.append(+ {+ "id": doc_id,+ "content": results["documents"][0][i],+ "metadata": results["metadatas"][0][i],+ "distance": (+ results["distances"][0][i]+ if results.get("distances")+ else None+ ),+ }+ )+ return hits++ def count(self) -> int:+ return self.collection.count()++ def embed_query(self, text: str) -> List[float]:+ return self._embedding_fn([text])[0]