Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

audit: fix graphrag_knowledge_system - add env comment, rewrite em dashes, remove Neo4j refs, add 41 docstrings, improve README

Tiioluwani01e35ece4ece
diff --git a/rag_apps/graphrag_knowledge_system/.env.example b/rag_apps/graphrag_knowledge_system/.env.example
index 192619e..54aa524 100644
--- a/rag_apps/graphrag_knowledge_system/.env.example
+++ b/rag_apps/graphrag_knowledge_system/.env.example
@@ -1,4 +1,6 @@
+# Mistral API key used for entity extraction and answer generation.
+# Get one free at https://platform.mistral.ai
MISTRAL_API_KEY=your_mistral_api_key_here
-# Optional — defaults to http://localhost:11434
+# Optional, defaults to http://localhost:11434
# OLLAMA_BASE_URL=http://localhost:11434
diff --git a/rag_apps/graphrag_knowledge_system/README.md b/rag_apps/graphrag_knowledge_system/README.md
index ad6e763..5c1f24f 100644
--- a/rag_apps/graphrag_knowledge_system/README.md
+++ b/rag_apps/graphrag_knowledge_system/README.md
@@ -4,7 +4,7 @@
## Overview
-GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully local Streamlit application. Documents are chunked and processed through an entity-extraction pipeline powered by Mistral Small 4. Extracted entities and relationships are stored in a NetworkX graph persisted to disk, while text embeddings are stored in ChromaDB using Ollama's `nomic-embed-text` model. At query time the app supports two retrieval modes — **Local Search** for pinpointing specific entities and **Global Search** for synthesising broad themes across the entire graph — before generating a grounded answer with supporting knowledge graph context.
+GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully local Streamlit application. Documents are chunked and processed through an entity-extraction pipeline powered by Mistral Small 4. Extracted entities and relationships are stored in a NetworkX graph persisted to disk, while text embeddings are stored in ChromaDB using Ollama's `nomic-embed-text` model. At query time the app supports two retrieval modes: **Local Search** for pinpointing specific entities and **Global Search** for synthesising broad themes across the entire graph, before generating a grounded answer with supporting knowledge graph context.
## Demo
@@ -12,14 +12,14 @@ GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully
## Features
-- **Document ingestion** — upload PDF or TXT files; documents are chunked, entity-extracted, and indexed automatically
-- **GraphRAG entity extraction** — extracts typed entities (Person, Organization, Location, Technology, Concept, Event) and weighted relationships from every chunk using Mistral
-- **Local Knowledge Graph** — NetworkX `DiGraph` persisted as JSON; no external database required
-- **Local embeddings** — `nomic-embed-text` via Ollama runs entirely on your machine; no embedding API key needed
-- **Local vector store** — ChromaDB stores chunk embeddings with persistent local storage
-- **Dual search modes** — Local Search for entity-centric retrieval, Global Search for community-level thematic synthesis
-- **Document caching** — SHA-256 file hashing prevents re-indexing the same file on every run
-- **Community detection** — automatic Louvain community detection groups related entities and generates community summaries for Global Search
+- **Document ingestion**: upload PDF or TXT files; documents are chunked, entity-extracted, and indexed automatically
+- **GraphRAG entity extraction**: extracts typed entities (Person, Organization, Location, Technology, Concept, Event) and weighted relationships from every chunk using Mistral
+- **Local Knowledge Graph**: NetworkX `DiGraph` persisted as JSON; no external database required
+- **Local embeddings**: `nomic-embed-text` via Ollama runs entirely on your machine; no embedding API key needed
+- **Local vector store**: ChromaDB stores chunk embeddings with persistent local storage
+- **Dual search modes**: Local Search for entity-centric retrieval, Global Search for community-level thematic synthesis
+- **Document caching**: SHA-256 file hashing prevents re-indexing the same file on every run
+- **Community detection**: automatic Louvain community detection groups related entities and generates community summaries for Global Search
## Tech Stack
@@ -29,7 +29,7 @@ GraphRAG Knowledge System implements Microsoft's GraphRAG methodology as a fully
| Document Ingestion | LangChain |
| LLM | Mistral Small 4 (`mistral-small-latest`) via Mistral AI API |
| Embeddings | `nomic-embed-text` via Ollama (local) |
-| Graph Store | NetworkX DiGraph — persisted to `graph_store.json` |
+| Graph Store | NetworkX DiGraph, persisted to `graph_store.json` |
| Vector Database | ChromaDB (local persistent storage in `chroma_db/`) |
| UI | Streamlit |
@@ -101,11 +101,11 @@ Open [http://localhost:8501](http://localhost:8501) in your browser.
**Workflow**
1. Upload one or more PDF or TXT files using the sidebar file uploader
-2. Click **⚙️ Process Documents** — the app chunks each file, extracts entities and relationships, builds the knowledge graph, and indexes embeddings
+2. Click **⚙️ Process Documents**: the app chunks each file, extracts entities and relationships, builds the knowledge graph, and indexes embeddings
3. Type a question in the main panel and choose a search mode:
- - **🔍 Local Search** — best for specific questions about named entities, events, or facts
- - **🌐 Global Search** — best for broad thematic questions that span the whole document set
-4. Click **🚀 Ask** — the answer appears as a paragraph with supporting entities and relationships listed below
+ - **🔍 Local Search**: best for specific questions about named entities, events, or facts
+ - **🌐 Global Search**: best for broad thematic questions that span the whole document set
+4. Click **🚀 Ask**: the answer appears as a paragraph with supporting entities and relationships listed below
**Example**
@@ -139,7 +139,7 @@ Copy `.env.example` to `.env` and fill in the value below.
| Variable | Required | Description |
|---|---|---|
-| `MISTRAL_API_KEY` | Yes | Your Mistral AI API key — get one at [platform.mistral.ai](https://platform.mistral.ai) |
+| `MISTRAL_API_KEY` | Yes | Your Mistral AI API key, get one at [platform.mistral.ai](https://platform.mistral.ai) |
```env
MISTRAL_API_KEY=your_mistral_api_key_here
@@ -150,7 +150,7 @@ MISTRAL_API_KEY=your_mistral_api_key_here
## Project Structure
```text
-graphrag-knowledge-system/
+graphrag_knowledge_system/
├── app.py # Streamlit UI, ingestion pipeline, community detection
├── src/
│ ├── cache_manager.py # SHA-256 file-hash cache (cache.json)
@@ -160,10 +160,12 @@ graphrag-knowledge-system/
│ ├── llm_client.py # Mistral AI chat client with retry logic
│ ├── search_engine.py # Local search and Global search implementations
│ └── vector_store.py # ChromaDB collection with Ollama embedding function
-├── chroma_db/ # Auto-created — ChromaDB persistent storage
-├── graph_store.json # Auto-created — serialised NetworkX graph
-├── cache.json # Auto-created — processed file hashes
+├── chroma_db/ # Auto-created, ChromaDB persistent storage
+├── graph_store.json # Auto-created, serialised NetworkX graph
+├── cache.json # Auto-created, processed file hashes
├── requirements.txt
├── .env.example
└── README.md
```
+
+[Back to top](#graphrag-knowledge-system)
diff --git a/rag_apps/graphrag_knowledge_system/app.py b/rag_apps/graphrag_knowledge_system/app.py
index bf67b21..12c8aba 100644
--- a/rag_apps/graphrag_knowledge_system/app.py
+++ b/rag_apps/graphrag_knowledge_system/app.py
@@ -1,3 +1,8 @@
+"""
+GraphRAG Knowledge System Streamlit app: document ingestion, knowledge graph
+construction, community detection, and Local/Global search over the graph.
+"""
+
import uuid
import json
import os
@@ -69,6 +74,7 @@ st.markdown(
@st.cache_resource(show_spinner=False)
def get_components():
+ """Construct and cache the pipeline components shared across reruns."""
graph_store = GraphStore()
vector_store = VectorStore()
entity_extractor = EntityExtractor()
@@ -105,6 +111,7 @@ Summary:"""
def run_community_detection(
graph_store: GraphStore, llm_client: LLMClient, status_fn
) -> int:
+ """Detect entity communities in the graph and summarise each one with the LLM."""
full_graph = graph_store.get_full_graph()
entities = full_graph["entities"]
relationships = full_graph["relationships"]
@@ -179,12 +186,13 @@ def ingest_document(
cache_manager: CacheManager,
llm_client: LLMClient,
) -> dict:
+ """Chunk, extract entities from, and index a single uploaded document."""
file_hash = cache_manager.file_hash(file_content)
if cache_manager.is_processed(file_hash):
return {
"status": "cached",
- "message": f"'{filename}' was already processed — skipping.",
+ "message": f"'{filename}' was already processed, skipping.",
}
# ── Step 1: Load and chunk ────────────────────────────────────────────────
@@ -214,7 +222,7 @@ def ingest_document(
chunk_id = f"{doc_id}_chunk_{idx}"
chunk_text = chunk.page_content
- # Store chunk in Neo4j
+ # Store chunk in the knowledge graph
graph_store.create_chunk(chunk_id, chunk_text, doc_id)
# Extract entities and relationships
@@ -296,6 +304,7 @@ TYPE_COLORS = {
def _type_badge(entity_type: str) -> str:
+ """Render an entity type as a small colored HTML badge."""
color = TYPE_COLORS.get(entity_type, "#888")
return (
f'<span style="background:{color};color:#fff;padding:2px 10px;'
@@ -305,6 +314,7 @@ def _type_badge(entity_type: str) -> str:
def render_entity_list(entities: list):
+ """Render up to 20 entities as name, description, and type badge rows."""
if not entities:
st.caption("No entities retrieved.")
return
@@ -323,6 +333,7 @@ def render_entity_list(entities: list):
def render_relationship_list(relationships: list):
+ """Render up to 15 relationships as source-target-weight rows."""
if not relationships:
st.caption("No relationships retrieved.")
return
@@ -341,6 +352,7 @@ def render_relationship_list(relationships: list):
def main():
+ """Render the Streamlit page: ingestion sidebar, search controls, and results."""
# ── Header ────────────────────────────────────────────────────────────────
st.markdown(
'<div class="main-header">🕸️ GraphRAG Knowledge System</div>', unsafe_allow_html=True
@@ -364,8 +376,8 @@ def main():
except Exception as e:
st.error(f"Failed to initialise components: {e}")
st.info(
- "Check your `.env` file — make sure NEO4J_URI, NEO4J_PASSWORD, "
- "MISTRAL_API_KEY are set and Ollama is running on http://localhost:11434"
+ "Check your `.env` file, make sure MISTRAL_API_KEY is set "
+ "and Ollama is running on http://localhost:11434"
)
get_components.clear()
return
@@ -391,7 +403,7 @@ def main():
"Upload PDF or TXT files",
type=["pdf", "txt"],
accept_multiple_files=True,
- help="Documents are chunked, entity-extracted, and stored in Neo4j + ChromaDB.",
+ help="Documents are chunked, entity-extracted, and stored in the local knowledge graph and ChromaDB.",
)
if uploaded_files and st.button("⚙️ Process Documents", type="primary", use_container_width=True):
@@ -433,13 +445,13 @@ def main():
try:
st.metric("Chunks", vector_store.count())
except Exception:
- st.metric("Chunks", "—")
+ st.metric("Chunks", "N/A")
with col2:
try:
communities = graph_store.get_all_communities()
st.metric("Communities", len(communities))
except Exception:
- st.metric("Communities", "—")
+ st.metric("Communities", "N/A")
if st.button("🔄 Refresh Stats", use_container_width=True):
st.rerun()
@@ -485,7 +497,7 @@ def main():
# ── Answer ────────────────────────────────────────────────────────────
mode_label = "Local Search" if "Local" in search_mode else "Global Search"
- st.markdown(f"### Answer — {mode_label}")
+ st.markdown(f"### Answer: {mode_label}")
st.markdown(result["answer"])
# ── Supporting context ────────────────────────────────────────────────
@@ -532,7 +544,7 @@ def main():
with info_col2:
st.info(
"**2. Knowledge Graph Built**\n\n"
- "Entities and relationships are stored in Neo4j. "
+ "Entities and relationships are stored in the local knowledge graph. "
"Text embeddings go into ChromaDB via Ollama."
)
with info_col3:
diff --git a/rag_apps/graphrag_knowledge_system/src/cache_manager.py b/rag_apps/graphrag_knowledge_system/src/cache_manager.py
index 247fcfe..289b4c0 100644
--- a/rag_apps/graphrag_knowledge_system/src/cache_manager.py
+++ b/rag_apps/graphrag_knowledge_system/src/cache_manager.py
@@ -6,10 +6,12 @@ from pathlib import Path
class CacheManager:
def __init__(self, cache_path: str = "./cache.json"):
+ """Load the processed-file cache from disk."""
self.cache_path = Path(cache_path)
self._load()
def _load(self):
+ """Read the cache JSON file from disk, or start empty if none exists."""
if self.cache_path.exists():
with open(self.cache_path, "r") as f:
self.cache = json.load(f)
@@ -17,18 +19,22 @@ class CacheManager:
self.cache = {}
def _save(self):
+ """Write the current cache to disk as JSON."""
with open(self.cache_path, "w") as f:
json.dump(self.cache, f, indent=2)
def file_hash(self, content: bytes) -> str:
+ """Compute a SHA-256 hash of the given file content."""
return hashlib.sha256(content).hexdigest()
def is_processed(self, file_hash: str) -> bool:
+ """Check whether a file with this hash has already been indexed."""
return file_hash in self.cache
def mark_processed(
self, file_hash: str, filename: str, doc_id: str, chunk_count: int
):
+ """Record a file as processed and persist the cache."""
self.cache[file_hash] = {
"filename": filename,
"document_id": doc_id,
@@ -38,6 +44,7 @@ class CacheManager:
self._save()
def get_all(self) -> list:
+ """Return summary info for every processed file in the cache."""
return [
{
"filename": v["filename"],
@@ -49,6 +56,7 @@ class CacheManager:
]
def remove(self, file_hash: str):
+ """Remove a file's cache entry, if present, and persist the change."""
if file_hash in self.cache:
del self.cache[file_hash]
self._save()
diff --git a/rag_apps/graphrag_knowledge_system/src/document_processor.py b/rag_apps/graphrag_knowledge_system/src/document_processor.py
index 85267ae..c34b169 100644
--- a/rag_apps/graphrag_knowledge_system/src/document_processor.py
+++ b/rag_apps/graphrag_knowledge_system/src/document_processor.py
@@ -8,6 +8,7 @@ from langchain_community.document_loaders import PyPDFLoader, TextLoader
class DocumentProcessor:
def __init__(self, chunk_size: int = 800, chunk_overlap: int = 150):
+ """Configure the text splitter used to chunk loaded documents."""
self.splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
@@ -15,6 +16,7 @@ class DocumentProcessor:
)
def process(self, file_content: bytes, filename: str) -> list:
+ """Load a PDF or TXT file from bytes and split it into text chunks."""
suffix = Path(filename).suffix.lower()
if suffix not in {".pdf", ".txt"}:
raise ValueError(f"Unsupported file type: {suffix}. Use PDF or TXT.")
diff --git a/rag_apps/graphrag_knowledge_system/src/entity_extractor.py b/rag_apps/graphrag_knowledge_system/src/entity_extractor.py
index 8045e72..615876c 100644
--- a/rag_apps/graphrag_knowledge_system/src/entity_extractor.py
+++ b/rag_apps/graphrag_knowledge_system/src/entity_extractor.py
@@ -28,7 +28,7 @@ Extraction rules:
- Extract 3–12 meaningful, specific entities; skip generic words
- Every relationship source and target MUST be an entity you extracted
- weight is 0.0–1.0 reflecting relationship strength or importance
-- Return only valid JSON — no markdown fences, no extra text
+- Return only valid JSON, no markdown fences, no extra text
Text to analyze:
{text}"""
@@ -36,9 +36,11 @@ Text to analyze:
class EntityExtractor:
def __init__(self):
+ """Create the LLM client used for entity and relationship extraction."""
self.llm = LLMClient()
def extract(self, text: str, chunk_id: str) -> dict:
+ """Extract typed entities and weighted relationships from a text chunk via Mistral."""
prompt = EXTRACTION_PROMPT.format(text=text[:3000])
try:
diff --git a/rag_apps/graphrag_knowledge_system/src/graph_store.py b/rag_apps/graphrag_knowledge_system/src/graph_store.py
index bb53cda..3012fc7 100644
--- a/rag_apps/graphrag_knowledge_system/src/graph_store.py
+++ b/rag_apps/graphrag_knowledge_system/src/graph_store.py
@@ -12,12 +12,14 @@ class GraphStore:
"""
def __init__(self, persist_path: str = "./graph_store.json"):
+ """Load the persisted graph from disk, or start a fresh one if none exists."""
self.persist_path = Path(persist_path)
self.G: nx.DiGraph = self._load()
# ── Persistence ───────────────────────────────────────────────────────────
def _load(self) -> nx.DiGraph:
+ """Read the graph JSON file from disk and rebuild it as a DiGraph."""
if self.persist_path.exists():
try:
with open(self.persist_path, "r", encoding="utf-8") as f:
@@ -28,6 +30,7 @@ class GraphStore:
return nx.DiGraph()
def _save(self):
+ """Serialise the current graph to disk as JSON."""
data = nx.node_link_data(self.G)
with open(self.persist_path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False)
@@ -35,12 +38,14 @@ class GraphStore:
# ── Document ─────────────────────────────────────────────────────────────
def create_document(self, doc_id: str, filename: str, file_hash: str):
+ """Add a document node to the graph and persist the change."""
self.G.add_node(
doc_id, node_type="document", filename=filename, file_hash=file_hash
)
self._save()
def document_exists(self, file_hash: str) -> bool:
+ """Check whether a document with the given file hash is already in the graph."""
return any(
d.get("node_type") == "document" and d.get("file_hash") == file_hash
for _, d in self.G.nodes(data=True)
@@ -49,6 +54,7 @@ class GraphStore:
# ── Chunks ────────────────────────────────────────────────────────────────
def create_chunk(self, chunk_id: str, content: str, doc_id: str):
+ """Add a chunk node linked to its parent document and persist the change."""
self.G.add_node(chunk_id, node_type="chunk", content=content, doc_id=doc_id)
if self.G.has_node(doc_id):
self.G.add_edge(chunk_id, doc_id, rel_type="PART_OF")
@@ -59,6 +65,7 @@ class GraphStore:
def upsert_entity(
self, name: str, entity_type: str, description: str, chunk_id: str
):
+ """Create or update an entity node and link it to the chunk it appeared in."""
if self.G.has_node(name) and self.G.nodes[name].get("node_type") == "entity":
existing = self.G.nodes[name]
if len(description) > len(existing.get("description", "")):
@@ -81,6 +88,7 @@ class GraphStore:
def upsert_relationship(
self, source: str, target: str, description: str, weight: float
):
+ """Create a relationship edge between two entities, or average the weight if it exists."""
if not self.G.has_node(source) or not self.G.has_node(target):
return
if self.G.has_edge(source, target):
@@ -102,13 +110,15 @@ class GraphStore:
# ── Community ─────────────────────────────────────────────────────────────
def set_entity_community(self, entity_name: str, community_id: str):
- # No _save() here — called in a tight loop; caller saves via create_community()
+ """Tag an entity node with the community it belongs to."""
+ # No _save() here since this runs in a tight loop; caller saves via create_community()
if self.G.has_node(entity_name):
self.G.nodes[entity_name]["community_id"] = community_id
def create_community(
self, community_id: str, entity_names: List[str], summary: str, level: int = 0
):
+ """Add a community node, link its member entities, and persist the change."""
self.G.add_node(
community_id,
node_type="community",
@@ -122,6 +132,7 @@ class GraphStore:
self._save()
def get_all_communities(self) -> List[Dict]:
+ """Return every community node along with its member entity names."""
result = []
for node_id, data in self.G.nodes(data=True):
if data.get("node_type") != "community":
@@ -143,6 +154,7 @@ class GraphStore:
return result
def communities_exist(self) -> bool:
+ """Check whether any community nodes have been created yet."""
return any(
d.get("node_type") == "community" for _, d in self.G.nodes(data=True)
)
@@ -150,6 +162,7 @@ class GraphStore:
# ── Graph export for community detection ─────────────────────────────────
def get_full_graph(self) -> Dict:
+ """Return every entity and weighted relationship in the graph for community detection."""
entities = [
{"name": n, "type": d.get("type", "")}
for n, d in self.G.nodes(data=True)
@@ -163,6 +176,7 @@ class GraphStore:
return {"entities": entities, "relationships": relationships}
def get_entity_descriptions(self, names: List[str]) -> List[Dict]:
+ """Look up name, type, and description for each entity in the given list."""
result = []
for name in names:
if not self.G.has_node(name):
@@ -181,6 +195,7 @@ class GraphStore:
# ── Search context ────────────────────────────────────────────────────────
def get_entity_neighborhood(self, entity_names: List[str]) -> Dict:
+ """Collect the given entities plus everything directly related to them."""
seen_entities: set = set()
seen_rels: set = set()
entities: List[Dict] = []
@@ -232,7 +247,9 @@ class GraphStore:
# ── Helpers ───────────────────────────────────────────────────────────────
def ping(self) -> Tuple[bool, str]:
+ """Report that the local graph store is always available."""
return True, "Connected (local NetworkX graph)"
def close(self):
+ """Persist the graph to disk before shutting down."""
self._save()
diff --git a/rag_apps/graphrag_knowledge_system/src/llm_client.py b/rag_apps/graphrag_knowledge_system/src/llm_client.py
index 3a3c7b6..3c570ae 100644
--- a/rag_apps/graphrag_knowledge_system/src/llm_client.py
+++ b/rag_apps/graphrag_knowledge_system/src/llm_client.py
@@ -6,6 +6,7 @@ from langchain_core.messages import HumanMessage, SystemMessage
class LLMClient:
def __init__(self):
+ """Create the plain-text and JSON-mode Mistral chat clients."""
api_key = os.getenv("MISTRAL_API_KEY")
if not api_key:
raise ValueError("MISTRAL_API_KEY not set in environment")
@@ -22,6 +23,7 @@ class LLMClient:
)
def complete(self, prompt: str, json_mode: bool = False, max_retries: int = 3) -> str:
+ """Send a single prompt to Mistral, retrying with backoff on failure."""
client = self.llm_json if json_mode else self.llm
messages = [HumanMessage(content=prompt)]
for attempt in range(max_retries):
@@ -37,6 +39,7 @@ class LLMClient:
def complete_with_system(
self, system: str, user: str, json_mode: bool = False
) -> str:
+ """Send a system and user message pair to Mistral and return the response text."""
client = self.llm_json if json_mode else self.llm
messages = [SystemMessage(content=system), HumanMessage(content=user)]
response = client.invoke(messages)
diff --git a/rag_apps/graphrag_knowledge_system/src/search_engine.py b/rag_apps/graphrag_knowledge_system/src/search_engine.py
index fcb02c3..10cf18d 100644
--- a/rag_apps/graphrag_knowledge_system/src/search_engine.py
+++ b/rag_apps/graphrag_knowledge_system/src/search_engine.py
@@ -59,6 +59,7 @@ Answer:"""
class SearchEngine:
def __init__(self, graph_store: GraphStore, vector_store: VectorStore):
+ """Wire up the graph store, vector store, and LLM client used for search."""
self.graph = graph_store
self.vector = vector_store
self.llm = LLMClient()
@@ -66,6 +67,7 @@ class SearchEngine:
# ── Local Search ──────────────────────────────────────────────────────────
def local_search(self, query: str, n_chunks: int = 5) -> Dict:
+ """Retrieve matching chunks and their graph context, then answer the question."""
if self.vector.count() == 0:
return self._no_docs_result()
@@ -124,6 +126,7 @@ class SearchEngine:
# ── Global Search ─────────────────────────────────────────────────────────
def global_search(self, query: str) -> Dict:
+ """Rate each community's relevance to the question and synthesise an answer from the top ones."""
communities = self.graph.get_all_communities()
if not communities:
return self.local_search(query)
@@ -196,6 +199,7 @@ class SearchEngine:
@staticmethod
def _no_docs_result() -> Dict:
+ """Build the placeholder result returned when no documents have been indexed."""
return {
"answer": "No documents have been indexed yet. Please upload and process documents first.",
"entities": [],