A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.
audit: fix graphrag_knowledge_system - add env comment, rewrite em dashes, remove Neo4j refs, add 41 docstrings, improve README
Tiioluwani01e35ece4ece
diff --git a/rag_apps/graphrag_knowledge_system/.env.example b/rag_apps/graphrag_knowledge_system/.env.exampleindex 192619e..54aa524 100644--- a/rag_apps/graphrag_knowledge_system/.env.example+++ b/rag_apps/graphrag_knowledge_system/.env.example@@ -1,4 +1,6 @@+# Mistral API key used for entity extraction and answer generation.+# Get one free at https://platform.mistral.aiMISTRAL_API_KEY=your_mistral_api_key_here-# Optional — defaults to http://localhost:11434+# Optional, defaults to http://localhost:11434# OLLAMA_BASE_URL=http://localhost:11434diff --git a/rag_apps/graphrag_knowledge_system/README.md b/rag_apps/graphrag_knowledge_system/README.mdindex ad6e763..5c1f24f 100644--- a/rag_apps/graphrag_knowledge_system/README.md+++ b/rag_apps/graphrag_knowledge_system/README.md@@ -4,7 +4,7 @@## Overview-GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully local Streamlit application. Documents are chunked and processed through an entity-extraction pipeline powered by Mistral Small 4. Extracted entities and relationships are stored in a NetworkX graph persisted to disk, while text embeddings are stored in ChromaDB using Ollama's `nomic-embed-text` model. At query time the app supports two retrieval modes — **Local Search** for pinpointing specific entities and **Global Search** for synthesising broad themes across the entire graph — before generating a grounded answer with supporting knowledge graph context.+GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully local Streamlit application. Documents are chunked and processed through an entity-extraction pipeline powered by Mistral Small 4. Extracted entities and relationships are stored in a NetworkX graph persisted to disk, while text embeddings are stored in ChromaDB using Ollama's `nomic-embed-text` model. At query time the app supports two retrieval modes: **Local Search** for pinpointing specific entities and **Global Search** for synthesising broad themes across the entire graph, before generating a grounded answer with supporting knowledge graph context.## Demo@@ -12,14 +12,14 @@ GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully## Features-- **Document ingestion** — upload PDF or TXT files; documents are chunked, entity-extracted, and indexed automatically-- **GraphRAG entity extraction** — extracts typed entities (Person, Organization, Location, Technology, Concept, Event) and weighted relationships from every chunk using Mistral-- **Local Knowledge Graph** — NetworkX `DiGraph` persisted as JSON; no external database required-- **Local embeddings** — `nomic-embed-text` via Ollama runs entirely on your machine; no embedding API key needed-- **Local vector store** — ChromaDB stores chunk embeddings with persistent local storage-- **Dual search modes** — Local Search for entity-centric retrieval, Global Search for community-level thematic synthesis-- **Document caching** — SHA-256 file hashing prevents re-indexing the same file on every run-- **Community detection** — automatic Louvain community detection groups related entities and generates community summaries for Global Search+- **Document ingestion**: upload PDF or TXT files; documents are chunked, entity-extracted, and indexed automatically+- **GraphRAG entity extraction**: extracts typed entities (Person, Organization, Location, Technology, Concept, Event) and weighted relationships from every chunk using Mistral+- **Local Knowledge Graph**: NetworkX `DiGraph` persisted as JSON; no external database required+- **Local embeddings**: `nomic-embed-text` via Ollama runs entirely on your machine; no embedding API key needed+- **Local vector store**: ChromaDB stores chunk embeddings with persistent local storage+- **Dual search modes**: Local Search for entity-centric retrieval, Global Search for community-level thematic synthesis+- **Document caching**: SHA-256 file hashing prevents re-indexing the same file on every run+- **Community detection**: automatic Louvain community detection groups related entities and generates community summaries for Global Search## Tech Stack@@ -29,7 +29,7 @@ GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully| Document Ingestion | LangChain || LLM | Mistral Small 4 (`mistral-small-latest`) via Mistral AI API || Embeddings | `nomic-embed-text` via Ollama (local) |-| Graph Store | NetworkX DiGraph — persisted to `graph_store.json` |+| Graph Store | NetworkX DiGraph, persisted to `graph_store.json` || Vector Database | ChromaDB (local persistent storage in `chroma_db/`) || UI | Streamlit |@@ -101,11 +101,11 @@ Open [http://localhost:8501](http://localhost:8501) in your browser.**Workflow**1. Upload one or more PDF or TXT files using the sidebar file uploader-2. Click **⚙️ Process Documents** — the app chunks each file, extracts entities and relationships, builds the knowledge graph, and indexes embeddings+2. Click **⚙️ Process Documents**: the app chunks each file, extracts entities and relationships, builds the knowledge graph, and indexes embeddings3. Type a question in the main panel and choose a search mode:- - **🔍 Local Search** — best for specific questions about named entities, events, or facts- - **🌐 Global Search** — best for broad thematic questions that span the whole document set-4. Click **🚀 Ask** — the answer appears as a paragraph with supporting entities and relationships listed below+ - **🔍 Local Search**: best for specific questions about named entities, events, or facts+ - **🌐 Global Search**: best for broad thematic questions that span the whole document set+4. Click **🚀 Ask**: the answer appears as a paragraph with supporting entities and relationships listed below**Example**@@ -139,7 +139,7 @@ Copy `.env.example` to `.env` and fill in the value below.| Variable | Required | Description ||---|---|---|-| `MISTRAL_API_KEY` | Yes | Your Mistral AI API key — get one at [platform.mistral.ai](https://platform.mistral.ai) |+| `MISTRAL_API_KEY` | Yes | Your Mistral AI API key, get one at [platform.mistral.ai](https://platform.mistral.ai) |```envMISTRAL_API_KEY=your_mistral_api_key_here@@ -150,7 +150,7 @@ MISTRAL_API_KEY=your_mistral_api_key_here## Project Structure```text-graphrag-knowledge-system/+graphrag_knowledge_system/├── app.py # Streamlit UI, ingestion pipeline, community detection├── src/│ ├── cache_manager.py # SHA-256 file-hash cache (cache.json)@@ -160,10 +160,12 @@ graphrag-knowledge-system/│ ├── llm_client.py # Mistral AI chat client with retry logic│ ├── search_engine.py # Local search and Global search implementations│ └── vector_store.py # ChromaDB collection with Ollama embedding function-├── chroma_db/ # Auto-created — ChromaDB persistent storage-├── graph_store.json # Auto-created — serialised NetworkX graph-├── cache.json # Auto-created — processed file hashes+├── chroma_db/ # Auto-created, ChromaDB persistent storage+├── graph_store.json # Auto-created, serialised NetworkX graph+├── cache.json # Auto-created, processed file hashes├── requirements.txt├── .env.example└── README.md```++[Back to top](#graphrag-knowledge-system)diff --git a/rag_apps/graphrag_knowledge_system/app.py b/rag_apps/graphrag_knowledge_system/app.pyindex bf67b21..12c8aba 100644--- a/rag_apps/graphrag_knowledge_system/app.py+++ b/rag_apps/graphrag_knowledge_system/app.py@@ -1,3 +1,8 @@+"""+GraphRAG Knowledge System Streamlit app: document ingestion, knowledge graph+construction, community detection, and Local/Global search over the graph.+"""+import uuidimport jsonimport os@@ -69,6 +74,7 @@ st.markdown(@st.cache_resource(show_spinner=False)def get_components():+ """Construct and cache the pipeline components shared across reruns."""graph_store = GraphStore()vector_store = VectorStore()entity_extractor = EntityExtractor()@@ -105,6 +111,7 @@ Summary:"""def run_community_detection(graph_store: GraphStore, llm_client: LLMClient, status_fn) -> int:+ """Detect entity communities in the graph and summarise each one with the LLM."""full_graph = graph_store.get_full_graph()entities = full_graph["entities"]relationships = full_graph["relationships"]@@ -179,12 +186,13 @@ def ingest_document(cache_manager: CacheManager,llm_client: LLMClient,) -> dict:+ """Chunk, extract entities from, and index a single uploaded document."""file_hash = cache_manager.file_hash(file_content)if cache_manager.is_processed(file_hash):return {"status": "cached",- "message": f"'{filename}' was already processed — skipping.",+ "message": f"'{filename}' was already processed, skipping.",}# ── Step 1: Load and chunk ────────────────────────────────────────────────@@ -214,7 +222,7 @@ def ingest_document(chunk_id = f"{doc_id}_chunk_{idx}"chunk_text = chunk.page_content- # Store chunk in Neo4j+ # Store chunk in the knowledge graphgraph_store.create_chunk(chunk_id, chunk_text, doc_id)# Extract entities and relationships@@ -296,6 +304,7 @@ TYPE_COLORS = {def _type_badge(entity_type: str) -> str:+ """Render an entity type as a small colored HTML badge."""color = TYPE_COLORS.get(entity_type, "#888")return (f'<span style="background:{color};color:#fff;padding:2px 10px;'@@ -305,6 +314,7 @@ def _type_badge(entity_type: str) -> str:def render_entity_list(entities: list):+ """Render up to 20 entities as name, description, and type badge rows."""if not entities:st.caption("No entities retrieved.")return@@ -323,6 +333,7 @@ def render_entity_list(entities: list):def render_relationship_list(relationships: list):+ """Render up to 15 relationships as source-target-weight rows."""if not relationships:st.caption("No relationships retrieved.")return@@ -341,6 +352,7 @@ def render_relationship_list(relationships: list):def main():+ """Render the Streamlit page: ingestion sidebar, search controls, and results."""# ── Header ────────────────────────────────────────────────────────────────st.markdown('<div class="main-header">🕸️ GraphRAG Knowledge System</div>', unsafe_allow_html=True@@ -364,8 +376,8 @@ def main():except Exception as e:st.error(f"Failed to initialise components: {e}")st.info(- "Check your `.env` file — make sure NEO4J_URI, NEO4J_PASSWORD, "- "MISTRAL_API_KEY are set and Ollama is running on http://localhost:11434"+ "Check your `.env` file, make sure MISTRAL_API_KEY is set "+ "and Ollama is running on http://localhost:11434")get_components.clear()return@@ -391,7 +403,7 @@ def main():"Upload PDF or TXT files",type=["pdf", "txt"],accept_multiple_files=True,- help="Documents are chunked, entity-extracted, and stored in Neo4j + ChromaDB.",+ help="Documents are chunked, entity-extracted, and stored in the local knowledge graph and ChromaDB.",)if uploaded_files and st.button("⚙️ Process Documents", type="primary", use_container_width=True):@@ -433,13 +445,13 @@ def main():try:st.metric("Chunks", vector_store.count())except Exception:- st.metric("Chunks", "—")+ st.metric("Chunks", "N/A")with col2:try:communities = graph_store.get_all_communities()st.metric("Communities", len(communities))except Exception:- st.metric("Communities", "—")+ st.metric("Communities", "N/A")if st.button("🔄 Refresh Stats", use_container_width=True):st.rerun()@@ -485,7 +497,7 @@ def main():# ── Answer ────────────────────────────────────────────────────────────mode_label = "Local Search" if "Local" in search_mode else "Global Search"- st.markdown(f"### Answer — {mode_label}")+ st.markdown(f"### Answer: {mode_label}")st.markdown(result["answer"])# ── Supporting context ────────────────────────────────────────────────@@ -532,7 +544,7 @@ def main():with info_col2:st.info("**2. Knowledge Graph Built**\n\n"- "Entities and relationships are stored in Neo4j. "+ "Entities and relationships are stored in the local knowledge graph. ""Text embeddings go into ChromaDB via Ollama.")with info_col3:diff --git a/rag_apps/graphrag_knowledge_system/src/cache_manager.py b/rag_apps/graphrag_knowledge_system/src/cache_manager.pyindex 247fcfe..289b4c0 100644--- a/rag_apps/graphrag_knowledge_system/src/cache_manager.py+++ b/rag_apps/graphrag_knowledge_system/src/cache_manager.py@@ -6,10 +6,12 @@ from pathlib import Pathclass CacheManager:def __init__(self, cache_path: str = "./cache.json"):+ """Load the processed-file cache from disk."""self.cache_path = Path(cache_path)self._load()def _load(self):+ """Read the cache JSON file from disk, or start empty if none exists."""if self.cache_path.exists():with open(self.cache_path, "r") as f:self.cache = json.load(f)@@ -17,18 +19,22 @@ class CacheManager:self.cache = {}def _save(self):+ """Write the current cache to disk as JSON."""with open(self.cache_path, "w") as f:json.dump(self.cache, f, indent=2)def file_hash(self, content: bytes) -> str:+ """Compute a SHA-256 hash of the given file content."""return hashlib.sha256(content).hexdigest()def is_processed(self, file_hash: str) -> bool:+ """Check whether a file with this hash has already been indexed."""return file_hash in self.cachedef mark_processed(self, file_hash: str, filename: str, doc_id: str, chunk_count: int):+ """Record a file as processed and persist the cache."""self.cache[file_hash] = {"filename": filename,"document_id": doc_id,@@ -38,6 +44,7 @@ class CacheManager:self._save()def get_all(self) -> list:+ """Return summary info for every processed file in the cache."""return [{"filename": v["filename"],@@ -49,6 +56,7 @@ class CacheManager:]def remove(self, file_hash: str):+ """Remove a file's cache entry, if present, and persist the change."""if file_hash in self.cache:del self.cache[file_hash]self._save()diff --git a/rag_apps/graphrag_knowledge_system/src/document_processor.py b/rag_apps/graphrag_knowledge_system/src/document_processor.pyindex 85267ae..c34b169 100644--- a/rag_apps/graphrag_knowledge_system/src/document_processor.py+++ b/rag_apps/graphrag_knowledge_system/src/document_processor.py@@ -8,6 +8,7 @@ from langchain_community.document_loaders import PyPDFLoader, TextLoaderclass DocumentProcessor:def __init__(self, chunk_size: int = 800, chunk_overlap: int = 150):+ """Configure the text splitter used to chunk loaded documents."""self.splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size,chunk_overlap=chunk_overlap,@@ -15,6 +16,7 @@ class DocumentProcessor:)def process(self, file_content: bytes, filename: str) -> list:+ """Load a PDF or TXT file from bytes and split it into text chunks."""suffix = Path(filename).suffix.lower()if suffix not in {".pdf", ".txt"}:raise ValueError(f"Unsupported file type: {suffix}. Use PDF or TXT.")diff --git a/rag_apps/graphrag_knowledge_system/src/entity_extractor.py b/rag_apps/graphrag_knowledge_system/src/entity_extractor.pyindex 8045e72..615876c 100644--- a/rag_apps/graphrag_knowledge_system/src/entity_extractor.py+++ b/rag_apps/graphrag_knowledge_system/src/entity_extractor.py@@ -28,7 +28,7 @@ Extraction rules:- Extract 3–12 meaningful, specific entities; skip generic words- Every relationship source and target MUST be an entity you extracted- weight is 0.0–1.0 reflecting relationship strength or importance-- Return only valid JSON — no markdown fences, no extra text+- Return only valid JSON, no markdown fences, no extra textText to analyze:{text}"""@@ -36,9 +36,11 @@ Text to analyze:class EntityExtractor:def __init__(self):+ """Create the LLM client used for entity and relationship extraction."""self.llm = LLMClient()def extract(self, text: str, chunk_id: str) -> dict:+ """Extract typed entities and weighted relationships from a text chunk via Mistral."""prompt = EXTRACTION_PROMPT.format(text=text[:3000])try:diff --git a/rag_apps/graphrag_knowledge_system/src/graph_store.py b/rag_apps/graphrag_knowledge_system/src/graph_store.pyindex bb53cda..3012fc7 100644--- a/rag_apps/graphrag_knowledge_system/src/graph_store.py+++ b/rag_apps/graphrag_knowledge_system/src/graph_store.py@@ -12,12 +12,14 @@ class GraphStore:"""def __init__(self, persist_path: str = "./graph_store.json"):+ """Load the persisted graph from disk, or start a fresh one if none exists."""self.persist_path = Path(persist_path)self.G: nx.DiGraph = self._load()# ── Persistence ───────────────────────────────────────────────────────────def _load(self) -> nx.DiGraph:+ """Read the graph JSON file from disk and rebuild it as a DiGraph."""if self.persist_path.exists():try:with open(self.persist_path, "r", encoding="utf-8") as f:@@ -28,6 +30,7 @@ class GraphStore:return nx.DiGraph()def _save(self):+ """Serialise the current graph to disk as JSON."""data = nx.node_link_data(self.G)with open(self.persist_path, "w", encoding="utf-8") as f:json.dump(data, f, ensure_ascii=False)@@ -35,12 +38,14 @@ class GraphStore:# ── Document ─────────────────────────────────────────────────────────────def create_document(self, doc_id: str, filename: str, file_hash: str):+ """Add a document node to the graph and persist the change."""self.G.add_node(doc_id, node_type="document", filename=filename, file_hash=file_hash)self._save()def document_exists(self, file_hash: str) -> bool:+ """Check whether a document with the given file hash is already in the graph."""return any(d.get("node_type") == "document" and d.get("file_hash") == file_hashfor _, d in self.G.nodes(data=True)@@ -49,6 +54,7 @@ class GraphStore:# ── Chunks ────────────────────────────────────────────────────────────────def create_chunk(self, chunk_id: str, content: str, doc_id: str):+ """Add a chunk node linked to its parent document and persist the change."""self.G.add_node(chunk_id, node_type="chunk", content=content, doc_id=doc_id)if self.G.has_node(doc_id):self.G.add_edge(chunk_id, doc_id, rel_type="PART_OF")@@ -59,6 +65,7 @@ class GraphStore:def upsert_entity(self, name: str, entity_type: str, description: str, chunk_id: str):+ """Create or update an entity node and link it to the chunk it appeared in."""if self.G.has_node(name) and self.G.nodes[name].get("node_type") == "entity":existing = self.G.nodes[name]if len(description) > len(existing.get("description", "")):@@ -81,6 +88,7 @@ class GraphStore:def upsert_relationship(self, source: str, target: str, description: str, weight: float):+ """Create a relationship edge between two entities, or average the weight if it exists."""if not self.G.has_node(source) or not self.G.has_node(target):returnif self.G.has_edge(source, target):@@ -102,13 +110,15 @@ class GraphStore:# ── Community ─────────────────────────────────────────────────────────────def set_entity_community(self, entity_name: str, community_id: str):- # No _save() here — called in a tight loop; caller saves via create_community()+ """Tag an entity node with the community it belongs to."""+ # No _save() here since this runs in a tight loop; caller saves via create_community()if self.G.has_node(entity_name):self.G.nodes[entity_name]["community_id"] = community_iddef create_community(self, community_id: str, entity_names: List[str], summary: str, level: int = 0):+ """Add a community node, link its member entities, and persist the change."""self.G.add_node(community_id,node_type="community",@@ -122,6 +132,7 @@ class GraphStore:self._save()def get_all_communities(self) -> List[Dict]:+ """Return every community node along with its member entity names."""result = []for node_id, data in self.G.nodes(data=True):if data.get("node_type") != "community":@@ -143,6 +154,7 @@ class GraphStore:return resultdef communities_exist(self) -> bool:+ """Check whether any community nodes have been created yet."""return any(d.get("node_type") == "community" for _, d in self.G.nodes(data=True))@@ -150,6 +162,7 @@ class GraphStore:# ── Graph export for community detection ─────────────────────────────────def get_full_graph(self) -> Dict:+ """Return every entity and weighted relationship in the graph for community detection."""entities = [{"name": n, "type": d.get("type", "")}for n, d in self.G.nodes(data=True)@@ -163,6 +176,7 @@ class GraphStore:return {"entities": entities, "relationships": relationships}def get_entity_descriptions(self, names: List[str]) -> List[Dict]:+ """Look up name, type, and description for each entity in the given list."""result = []for name in names:if not self.G.has_node(name):@@ -181,6 +195,7 @@ class GraphStore:# ── Search context ────────────────────────────────────────────────────────def get_entity_neighborhood(self, entity_names: List[str]) -> Dict:+ """Collect the given entities plus everything directly related to them."""seen_entities: set = set()seen_rels: set = set()entities: List[Dict] = []@@ -232,7 +247,9 @@ class GraphStore:# ── Helpers ───────────────────────────────────────────────────────────────def ping(self) -> Tuple[bool, str]:+ """Report that the local graph store is always available."""return True, "Connected (local NetworkX graph)"def close(self):+ """Persist the graph to disk before shutting down."""self._save()diff --git a/rag_apps/graphrag_knowledge_system/src/llm_client.py b/rag_apps/graphrag_knowledge_system/src/llm_client.pyindex 3a3c7b6..3c570ae 100644--- a/rag_apps/graphrag_knowledge_system/src/llm_client.py+++ b/rag_apps/graphrag_knowledge_system/src/llm_client.py@@ -6,6 +6,7 @@ from langchain_core.messages import HumanMessage, SystemMessageclass LLMClient:def __init__(self):+ """Create the plain-text and JSON-mode Mistral chat clients."""api_key = os.getenv("MISTRAL_API_KEY")if not api_key:raise ValueError("MISTRAL_API_KEY not set in environment")@@ -22,6 +23,7 @@ class LLMClient:)def complete(self, prompt: str, json_mode: bool = False, max_retries: int = 3) -> str:+ """Send a single prompt to Mistral, retrying with backoff on failure."""client = self.llm_json if json_mode else self.llmmessages = [HumanMessage(content=prompt)]for attempt in range(max_retries):@@ -37,6 +39,7 @@ class LLMClient:def complete_with_system(self, system: str, user: str, json_mode: bool = False) -> str:+ """Send a system and user message pair to Mistral and return the response text."""client = self.llm_json if json_mode else self.llmmessages = [SystemMessage(content=system), HumanMessage(content=user)]response = client.invoke(messages)diff --git a/rag_apps/graphrag_knowledge_system/src/search_engine.py b/rag_apps/graphrag_knowledge_system/src/search_engine.pyindex fcb02c3..10cf18d 100644--- a/rag_apps/graphrag_knowledge_system/src/search_engine.py+++ b/rag_apps/graphrag_knowledge_system/src/search_engine.py@@ -59,6 +59,7 @@ Answer:"""class SearchEngine:def __init__(self, graph_store: GraphStore, vector_store: VectorStore):+ """Wire up the graph store, vector store, and LLM client used for search."""self.graph = graph_storeself.vector = vector_storeself.llm = LLMClient()@@ -66,6 +67,7 @@ class SearchEngine:# ── Local Search ──────────────────────────────────────────────────────────def local_search(self, query: str, n_chunks: int = 5) -> Dict:+ """Retrieve matching chunks and their graph context, then answer the question."""if self.vector.count() == 0:return self._no_docs_result()@@ -124,6 +126,7 @@ class SearchEngine:# ── Global Search ─────────────────────────────────────────────────────────def global_search(self, query: str) -> Dict:+ """Rate each community's relevance to the question and synthesise an answer from the top ones."""communities = self.graph.get_all_communities()if not communities:return self.local_search(query)@@ -196,6 +199,7 @@ class SearchEngine:@staticmethoddef _no_docs_result() -> Dict:+ """Build the placeholder result returned when no documents have been indexed."""return {"answer": "No documents have been indexed yet. Please upload and process documents first.","entities": [],