Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

feat: add Hybrid RAG System combining knowledge graph and vector retrieval

Tiioluwani4280070a8698
diff --git a/rag_apps/hybrid_rag_system/.env.example b/rag_apps/hybrid_rag_system/.env.example
new file mode 100644
index 0000000..b263636
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/.env.example
@@ -0,0 +1 @@
+MISTRAL_API_KEY=your_mistral_api_key_here
diff --git a/rag_apps/hybrid_rag_system/README.md b/rag_apps/hybrid_rag_system/README.md
new file mode 100644
index 0000000..04fb207
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/README.md
@@ -0,0 +1,138 @@
+# Hybrid RAG System
+
+> Combines knowledge graph retrieval and vector retrieval into a unified pipeline for answering complex questions over your own documents.
+
+## Overview
+
+Hybrid RAG System dual-indexes every uploaded document: a knowledge graph captures named entities and their relationships, while a vector store captures semantic meaning at the chunk level. When a question is asked, both retrieval paths run in parallel and their results are fused into a single context before being passed to Mistral Small 4. The answer is displayed alongside clearly labeled graph sources (entities and relationships) and vector sources (ranked text chunks), so you can see exactly what evidence the model used.
+
+## Demo
+
+![Demo](assets/demo.png)
+
+## Features
+
+- **Dual indexing** — every document is indexed into both a knowledge graph and a vector store in a single ingestion pass
+- **Parallel retrieval** — graph and vector queries run concurrently via `ThreadPoolExecutor`, minimising latency
+- **Context fusion** — graph entities/relationships and ranked vector chunks are merged into one unified prompt
+- **Adjustable retrieval** — sliders to tune top-K vector chunks (1–20) and max graph entities (5–50) at query time
+- **Document cache** — SHA-256 hash check skips re-indexing files that are already processed
+- **Progress tracking** — two-phase progress bar shows entity extraction and embedding status during ingestion
+- **Source transparency** — answer panel shows graph sources and vector sources in separate labelled sections with similarity scores
+- **Performance metrics** — per-query timing for graph retrieval, vector retrieval, and generation
+
+## Tech Stack
+
+| Layer | Technology |
+|---|---|
+| Graph indexing & retrieval | GraphRAG + NetworkX |
+| Vector retrieval | LangChain + ChromaDB |
+| LLM | Mistral Small 4 (`mistral-small-latest`) via Mistral AI API |
+| Embeddings | `nomic-embed-text` via Ollama (local, no API key needed) |
+| Graph store | NetworkX — persisted as JSON |
+| Vector database | ChromaDB — local persistent storage |
+| UI | Streamlit |
+| Configuration | python-dotenv |
+
+## Prerequisites
+
+- Python 3.10 or higher
+- [Ollama](https://ollama.com/download) installed and running
+- A [Mistral AI](https://platform.mistral.ai) API key
+
+## Installation
+
+**1. Clone the repository**
+
+```bash
+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
+cd Hands-On-AI-Engineering/rag_apps/hybrid_rag_system
+```
+
+**2. Create a virtual environment**
+
+*Windows*
+```bash
+python -m venv .venv
+.venv\Scripts\activate
+```
+
+*macOS / Linux*
+```bash
+python -m venv .venv
+source .venv/bin/activate
+```
+
+**3. Install dependencies**
+
+```bash
+pip install -r requirements.txt
+```
+
+**4. Configure environment variables**
+
+```bash
+cp .env.example .env
+```
+
+Open `.env` and add your Mistral API key (see [Environment Variables](#environment-variables)).
+
+**5. Pull the embedding model**
+
+```bash
+ollama pull nomic-embed-text
+```
+
+## Usage
+
+Ensure Ollama is running, then start the app:
+
+```bash
+streamlit run app.py
+```
+
+The app opens at `http://localhost:8501`.
+
+1. Upload a PDF or TXT file using the sidebar uploader
+2. Click **Process Document** — the progress bar tracks entity extraction (Phase 1) then chunk embedding (Phase 2)
+3. Type a question in the main panel and click **Ask**
+4. Review the answer and expand **Graph Sources** and **Vector Sources** to inspect the retrieved evidence
+
+## Environment Variables
+
+Copy `.env.example` to `.env` and fill in the value below.
+
+| Variable | Required | Description |
+|---|---|---|
+| `MISTRAL_API_KEY` | Yes | API key from [platform.mistral.ai](https://platform.mistral.ai) |
+
+```env
+MISTRAL_API_KEY=your_mistral_api_key_here
+```
+
+Ollama runs locally and requires no API key.
+
+## Project Structure
+
+```text
+hybrid-rag-system/
+├── app.py # Streamlit application — UI, session state, parallel retrieval
+├── config.py # Constants and automatic data-directory creation
+├── src/
+│ ├── document_processor.py # PDF/TXT loading, text chunking, hash-based document cache
+│ ├── graph_indexer.py # Entity/relationship extraction via Mistral → NetworkX graph → JSON
+│ ├── vector_indexer.py # Ollama embedding function, ChromaDB client, chunk indexing
+│ ├── graph_retriever.py # Query-term matching and BFS subgraph expansion
+│ ├── vector_retriever.py # ChromaDB semantic search with similarity scores
+│ ├── context_fusion.py # Formats and merges graph and vector contexts for the prompt
+│ └── mistral_client.py # Mistral Small 4 answer generation
+├── data/
+│ ├── graphs/ # Per-document graphs ({hash}.json) and combined_graph.json
+│ ├── chroma/ # ChromaDB persistent storage
+│ └── cache/
+│ └── processed_docs.json # Registry of indexed documents
+├── assets/
+│ └── demo.png # Demo screenshot
+├── .env.example # Environment variable template
+└── requirements.txt # Python dependencies
+```
diff --git a/rag_apps/hybrid_rag_system/app.py b/rag_apps/hybrid_rag_system/app.py
new file mode 100644
index 0000000..d48b13d
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/app.py
@@ -0,0 +1,446 @@
+import os
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+import streamlit as st
+from dotenv import load_dotenv
+
+sys.path.insert(0, str(Path(__file__).parent))
+load_dotenv()
+
+from config import DEFAULT_TOP_K, DEFAULT_MAX_ENTITIES
+from src.document_processor import (
+ load_document, chunk_text, compute_hash,
+ is_cached, add_to_cache, list_cached_docs,
+)
+from src.graph_indexer import (
+ build_doc_graph, merge_into_combined,
+ save_doc_graph, load_combined_graph,
+)
+from src.vector_indexer import (
+ check_ollama, get_chroma_collection,
+ index_chunks, collection_has_doc,
+)
+from src.graph_retriever import retrieve_from_graph
+from src.vector_retriever import retrieve_from_vector
+from src.context_fusion import fuse_context, format_graph_context, format_vector_context
+from src.mistral_client import generate_answer
+
+# ─── Page config ────────────────────────────────────────────────────────────
+
+st.set_page_config(
+ page_title="Hybrid RAG System",
+ page_icon="🔮",
+ layout="wide",
+ initial_sidebar_state="expanded",
+)
+
+# ─── Custom CSS ─────────────────────────────────────────────────────────────
+
+st.markdown("""
+<style>
+.graph-box {
+ background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);
+ border: 1px solid #4a9eff;
+ border-radius: 10px;
+ padding: 16px;
+ margin: 8px 0;
+}
+.vector-box {
+ background: linear-gradient(135deg, #1a2e1a 0%, #162e16 100%);
+ border: 1px solid #4aff6e;
+ border-radius: 10px;
+ padding: 16px;
+ margin: 8px 0;
+}
+.answer-box {
+ background: linear-gradient(135deg, #2e1a1a 0%, #2e1616 100%);
+ border: 1px solid #ff6e4a;
+ border-radius: 10px;
+ padding: 20px;
+ margin: 12px 0;
+}
+.metric-card {
+ background: #1e1e2e;
+ border-radius: 8px;
+ padding: 10px;
+ text-align: center;
+}
+.status-ok { color: #4aff6e; font-weight: bold; }
+.status-err { color: #ff4a4a; font-weight: bold; }
+</style>
+""", unsafe_allow_html=True)
+
+# ─── Cached resources ────────────────────────────────────────────────────────
+
[email protected]_resource(show_spinner=False)
+def _get_mistral_client():
+ from mistralai.client import Mistral
+ api_key = os.environ.get("MISTRAL_API_KEY", "")
+ if not api_key:
+ return None
+ return Mistral(api_key=api_key)
+
+
[email protected]_resource(show_spinner=False)
+def _get_collection():
+ try:
+ return get_chroma_collection()
+ except Exception as e:
+ return None
+
+
+# ─── Session state init ──────────────────────────────────────────────────────
+
+if "graph" not in st.session_state:
+ st.session_state.graph = load_combined_graph()
+
+if "last_answer" not in st.session_state:
+ st.session_state.last_answer = None
+
+if "last_graph_results" not in st.session_state:
+ st.session_state.last_graph_results = None
+
+if "last_vector_results" not in st.session_state:
+ st.session_state.last_vector_results = None
+
+if "last_question" not in st.session_state:
+ st.session_state.last_question = None
+
+# ─── Sidebar ─────────────────────────────────────────────────────────────────
+
+with st.sidebar:
+ st.title("🔮 Hybrid RAG")
+ st.caption("Knowledge Graph + Vector Retrieval")
+
+ # Status checks
+ st.subheader("System Status")
+ col1, col2 = st.columns(2)
+
+ api_key = os.environ.get("MISTRAL_API_KEY", "")
+ mistral_ok = bool(api_key)
+ with col1:
+ if mistral_ok:
+ st.markdown('<span class="status-ok">✓ Mistral</span>', unsafe_allow_html=True)
+ else:
+ st.markdown('<span class="status-err">✗ Mistral</span>', unsafe_allow_html=True)
+
+ ollama_ok, ollama_msg = check_ollama()
+ with col2:
+ if ollama_ok:
+ st.markdown('<span class="status-ok">✓ Ollama</span>', unsafe_allow_html=True)
+ else:
+ st.markdown('<span class="status-err">✗ Ollama</span>', unsafe_allow_html=True)
+
+ if not mistral_ok:
+ st.warning("Set MISTRAL_API_KEY in your .env file.")
+ if not ollama_ok:
+ st.warning(ollama_msg)
+
+ st.divider()
+
+ # Retrieval settings
+ st.subheader("Retrieval Settings")
+ top_k = st.slider(
+ "Top-K vector chunks",
+ min_value=1, max_value=20,
+ value=DEFAULT_TOP_K,
+ help="Number of document chunks retrieved via semantic search",
+ )
+ max_entities = st.slider(
+ "Max graph entities",
+ min_value=5, max_value=50,
+ value=DEFAULT_MAX_ENTITIES,
+ help="Maximum number of entities retrieved from the knowledge graph",
+ )
+
+ st.divider()
+
+ # Document upload
+ st.subheader("Document Ingestion")
+ uploaded_file = st.file_uploader(
+ "Upload PDF or TXT",
+ type=["pdf", "txt"],
+ help="Documents are dual-indexed: entities → knowledge graph, chunks → ChromaDB",
+ )
+
+ if uploaded_file is not None:
+ file_bytes = uploaded_file.read()
+ try:
+ text = load_document(file_bytes, uploaded_file.name)
+ doc_hash = compute_hash(text)
+ cached = is_cached(doc_hash)
+
+ if cached:
+ st.success(f"Already indexed: {uploaded_file.name}")
+ st.caption(f"Chunks: {cached['num_chunks']} | Entities: {cached['num_entities']}")
+ else:
+ st.info(f"Ready to index: {uploaded_file.name}")
+ st.caption(f"Size: {len(text):,} characters")
+
+ if st.button("Process Document", type="primary", use_container_width=True):
+ if not mistral_ok:
+ st.error("Mistral API key required for entity extraction.")
+ elif not ollama_ok:
+ st.error("Ollama required for embeddings.")
+ else:
+ chunks = chunk_text(text)
+ collection = _get_collection()
+ mistral_client = _get_mistral_client()
+
+ st.markdown("**Ingestion Progress**")
+ phase_label = st.empty()
+ progress_bar = st.progress(0)
+ stat_label = st.empty()
+
+ # --- Phase 1: Graph indexing ---
+ phase_label.markdown("Phase 1/2: Building knowledge graph...")
+ graph_progress = [0]
+
+ def graph_cb(done, total):
+ graph_progress[0] = done
+ pct = int(done / total * 50)
+ progress_bar.progress(pct)
+ stat_label.caption(f"Extracting entities: chunk {done}/{total}")
+
+ G_doc = build_doc_graph(chunks, mistral_client, progress_cb=graph_cb)
+ save_doc_graph(G_doc, doc_hash)
+ G_combined = merge_into_combined(G_doc)
+ st.session_state.graph = G_combined
+ num_entities = G_doc.number_of_nodes()
+
+ # --- Phase 2: Vector indexing ---
+ phase_label.markdown("Phase 2/2: Embedding chunks into ChromaDB...")
+ vec_progress = [0]
+
+ def vec_cb(done, total):
+ vec_progress[0] = done
+ pct = 50 + int(done / total * 50)
+ progress_bar.progress(pct)
+ stat_label.caption(f"Embedding chunks: {done}/{total}")
+
+ index_chunks(chunks, collection, doc_hash, progress_cb=vec_cb)
+
+ progress_bar.progress(100)
+ phase_label.empty()
+ stat_label.empty()
+
+ add_to_cache(doc_hash, uploaded_file.name, len(chunks), num_entities)
+ st.success(f"Indexed {len(chunks)} chunks, {num_entities} entities")
+ st.rerun()
+
+ except Exception as e:
+ st.error(f"Error loading file: {e}")
+
+ st.divider()
+
+ # Indexed documents list
+ st.subheader("Indexed Documents")
+ cached_docs = list_cached_docs()
+ G = st.session_state.graph
+ st.caption(f"Graph: {G.number_of_nodes()} entities, {G.number_of_edges()} relationships")
+
+ if cached_docs:
+ for doc in cached_docs:
+ with st.expander(f"📄 {doc['filename']}", expanded=False):
+ st.caption(f"Chunks: {doc['num_chunks']}")
+ st.caption(f"Entities: {doc['num_entities']}")
+ st.caption(f"Hash: {doc['hash'][:12]}...")
+ else:
+ st.caption("No documents indexed yet.")
+
+# ─── Main area ────────────────────────────────────────────────────────────────
+
+st.title("🔮 Hybrid RAG System")
+st.markdown(
+ "Combines **Knowledge Graph** retrieval and **Vector** retrieval "
+ "into a unified pipeline for answering complex questions."
+)
+
+# Architecture diagram (text-based)
+with st.expander("How it works", expanded=False):
+ st.markdown("""
+ ```
+ Document Upload
+ │
+ ├──► GraphRAG (Mistral) ──► NetworkX Graph ──► JSON
+ │ Entity + Relationship Extraction
+ │
+ └──► Text Chunker ──► nomic-embed-text ──► ChromaDB
+ Semantic Embeddings
+
+ User Question
+ │
+ ├──► Graph Retrieval (parallel) ──► Entities & Relationships
+ │
+ └──► Vector Retrieval (parallel) ──► Top-K Chunks
+ │
+ ▼
+ Context Fusion
+ │
+ ▼
+ Mistral Small 4
+ │
+ ▼
+ Answer + Sources Display
+ ```
+ """)
+
+st.divider()
+
+# Q&A section
+docs_ready = bool(list_cached_docs())
+G = st.session_state.graph
+
+if not docs_ready:
+ st.info("Upload and process at least one document in the sidebar to begin.")
+else:
+ st.subheader("Ask a Question")
+
+ with st.form("qa_form", clear_on_submit=False):
+ question = st.text_area(
+ "Your question",
+ placeholder="What are the main concepts discussed? Who are the key people involved? How do X and Y relate?",
+ height=80,
+ )
+ submitted = st.form_submit_button("Ask", type="primary", use_container_width=True)
+
+ if submitted and question.strip():
+ if not mistral_ok:
+ st.error("Mistral API key required. Set MISTRAL_API_KEY in .env")
+ elif not ollama_ok:
+ st.error(f"Ollama required for embeddings: {ollama_msg}")
+ else:
+ collection = _get_collection()
+ mistral_client = _get_mistral_client()
+ G = st.session_state.graph
+
+ with st.spinner("Running parallel retrieval..."):
+ retrieval_times = {}
+
+ def run_graph():
+ t0 = time.time()
+ result = retrieve_from_graph(G, question, max_entities)
+ retrieval_times["graph"] = round(time.time() - t0, 2)
+ return result
+
+ def run_vector():
+ t0 = time.time()
+ result = retrieve_from_vector(collection, question, top_k)
+ retrieval_times["vector"] = round(time.time() - t0, 2)
+ return result
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ graph_future = executor.submit(run_graph)
+ vector_future = executor.submit(run_vector)
+ graph_results = graph_future.result()
+ vector_results = vector_future.result()
+
+ with st.spinner("Generating answer with Mistral Small 4..."):
+ t0 = time.time()
+ fused = fuse_context(graph_results, vector_results)
+ answer = generate_answer(mistral_client, question, fused)
+ gen_time = round(time.time() - t0, 2)
+
+ st.session_state.last_answer = answer
+ st.session_state.last_graph_results = graph_results
+ st.session_state.last_vector_results = vector_results
+ st.session_state.last_question = question
+ st.session_state.retrieval_times = retrieval_times
+ st.session_state.gen_time = gen_time
+
+ # Display results
+ if st.session_state.last_answer:
+ st.subheader("Answer")
+
+ # Timing metrics
+ rt = getattr(st.session_state, "retrieval_times", {})
+ gt = getattr(st.session_state, "gen_time", 0)
+ m1, m2, m3, m4 = st.columns(4)
+ m1.metric("Graph retrieval", f"{rt.get('graph', 0):.2f}s")
+ m2.metric("Vector retrieval", f"{rt.get('vector', 0):.2f}s")
+ m3.metric("Generation", f"{gt:.2f}s")
+ m4.metric(
+ "Sources",
+ f"{len(st.session_state.last_graph_results.get('entities', []))}E + "
+ f"{len(st.session_state.last_vector_results)}C"
+ )
+
+ st.markdown(
+ f'<div class="answer-box">{st.session_state.last_answer}</div>',
+ unsafe_allow_html=True,
+ )
+
+ st.subheader("Sources")
+ col_graph, col_vector = st.columns(2)
+
+ with col_graph:
+ gr = st.session_state.last_graph_results
+ n_ent = len(gr.get("entities", []))
+ n_rel = len(gr.get("relationships", []))
+ st.markdown(
+ f'<div style="color:#4a9eff;font-weight:bold;font-size:1.05rem;">'
+ f'Graph Sources ({n_ent} entities, {n_rel} relationships)</div>',
+ unsafe_allow_html=True,
+ )
+
+ if gr.get("entities"):
+ with st.expander("Entities", expanded=True):
+ for e in gr["entities"]:
+ badge_color = {
+ "PERSON": "#ff9999", "ORGANIZATION": "#99ccff",
+ "LOCATION": "#99ff99", "TECHNOLOGY": "#ffcc99",
+ "CONCEPT": "#cc99ff", "EVENT": "#ffff99",
+ }.get(e["type"], "#cccccc")
+ st.markdown(
+ f'<span style="background:{badge_color};color:#000;'
+ f'border-radius:4px;padding:1px 6px;font-size:0.75rem;">'
+ f'{e["type"]}</span> **{e["name"]}**',
+ unsafe_allow_html=True,
+ )
+ if e["description"]:
+ st.caption(e["description"])
+
+ if gr.get("relationships"):
+ with st.expander("Relationships", expanded=True):
+ for r in gr["relationships"]:
+ rels = " | ".join(r["relations"])
+ st.markdown(f"`{r['source']}` → **{rels}** → `{r['target']}`")
+ if r["description"]:
+ st.caption(r["description"])
+
+ if not gr.get("entities") and not gr.get("relationships"):
+ st.caption("No graph context found for this query.")
+
+ with col_vector:
+ vr = st.session_state.last_vector_results
+ st.markdown(
+ f'<div style="color:#4aff6e;font-weight:bold;font-size:1.05rem;">'
+ f'Vector Sources ({len(vr)} chunks)</div>',
+ unsafe_allow_html=True,
+ )
+
+ if vr:
+ for i, chunk in enumerate(vr, 1):
+ sim = chunk.get("similarity", 0.0)
+ sim_pct = int(sim * 100)
+ with st.expander(f"Chunk {i} — similarity {sim_pct}%", expanded=(i == 1)):
+ st.markdown(chunk["text"])
+ meta = chunk.get("metadata", {})
+ if meta.get("doc_hash"):
+ st.caption(f"Doc: {meta['doc_hash'][:12]}... | Chunk #{meta.get('chunk_index', '?')}")
+ else:
+ st.caption("No vector context found for this query.")
+
+ elif submitted and question.strip():
+ pass # errors already shown above
+
+# ─── Footer ──────────────────────────────────────────────────────────────────
+
+st.divider()
+st.caption(
+ "Hybrid RAG System — GraphRAG (NetworkX) + Vector (ChromaDB) | "
+ "Embeddings: nomic-embed-text via Ollama | LLM: Mistral Small 4"
+)
diff --git a/rag_apps/hybrid_rag_system/assets/demo.png b/rag_apps/hybrid_rag_system/assets/demo.png
new file mode 100644
index 0000000..5d05179
Binary files /dev/null and b/rag_apps/hybrid_rag_system/assets/demo.png differ
diff --git a/rag_apps/hybrid_rag_system/config.py b/rag_apps/hybrid_rag_system/config.py
new file mode 100644
index 0000000..ed7557b
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/config.py
@@ -0,0 +1,23 @@
+from pathlib import Path
+
+BASE_DIR = Path(__file__).parent
+DATA_DIR = BASE_DIR / "data"
+GRAPHS_DIR = DATA_DIR / "graphs"
+CHROMA_DIR = DATA_DIR / "chroma"
+CACHE_DIR = DATA_DIR / "cache"
+CACHE_FILE = CACHE_DIR / "processed_docs.json"
+COMBINED_GRAPH_FILE = GRAPHS_DIR / "combined_graph.json"
+
+for _d in [DATA_DIR, GRAPHS_DIR, CHROMA_DIR, CACHE_DIR]:
+ _d.mkdir(parents=True, exist_ok=True)
+
+OLLAMA_URL = "http://localhost:11434"
+EMBED_MODEL = "nomic-embed-text"
+LLM_MODEL = "mistral-small-latest"
+CHROMA_COLLECTION = "hybrid-rag"
+
+CHUNK_SIZE = 800
+CHUNK_OVERLAP = 150
+
+DEFAULT_TOP_K = 5
+DEFAULT_MAX_ENTITIES = 15
diff --git a/rag_apps/hybrid_rag_system/requirements.txt b/rag_apps/hybrid_rag_system/requirements.txt
new file mode 100644
index 0000000..713bb0e
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/requirements.txt
@@ -0,0 +1,9 @@
+streamlit>=1.32.0
+mistralai>=1.5.0
+chromadb>=0.5.0
+langchain-text-splitters>=0.2.0
+networkx>=3.2.0
+pypdf>=4.0.0
+python-dotenv>=1.0.0
+requests>=2.31.0
+numpy>=1.26.0
diff --git a/rag_apps/hybrid_rag_system/src/__init__.py b/rag_apps/hybrid_rag_system/src/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/rag_apps/hybrid_rag_system/src/context_fusion.py b/rag_apps/hybrid_rag_system/src/context_fusion.py
new file mode 100644
index 0000000..e17f28c
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/context_fusion.py
@@ -0,0 +1,52 @@
+from typing import Any
+
+
+def format_graph_context(graph_results: dict[str, Any]) -> str:
+ entities = graph_results.get("entities", [])
+ relationships = graph_results.get("relationships", [])
+
+ if not entities and not relationships:
+ return "No graph context available."
+
+ lines = []
+ if entities:
+ lines.append("ENTITIES:")
+ for e in entities:
+ desc = e["description"] if e["description"] else "no description"
+ lines.append(f" • {e['name']} [{e['type']}]: {desc}")
+
+ if relationships:
+ lines.append("\nRELATIONSHIPS:")
+ for r in relationships:
+ rels = " | ".join(r["relations"])
+ lines.append(f" • {r['source']} --[{rels}]--> {r['target']}")
+ if r["description"]:
+ lines.append(f" ({r['description']})")
+
+ return "\n".join(lines)
+
+
+def format_vector_context(vector_results: list[dict[str, Any]]) -> str:
+ if not vector_results:
+ return "No vector context available."
+ lines = []
+ for i, chunk in enumerate(vector_results, 1):
+ sim = chunk.get("similarity", 0.0)
+ lines.append(f"[Chunk {i} | similarity: {sim:.3f}]")
+ lines.append(chunk["text"].strip())
+ lines.append("")
+ return "\n".join(lines)
+
+
+def fuse_context(
+ graph_results: dict[str, Any],
+ vector_results: list[dict[str, Any]],
+) -> str:
+ graph_ctx = format_graph_context(graph_results)
+ vector_ctx = format_vector_context(vector_results)
+ return (
+ "=== KNOWLEDGE GRAPH CONTEXT ===\n"
+ f"{graph_ctx}\n\n"
+ "=== DOCUMENT CHUNKS (Vector Retrieval) ===\n"
+ f"{vector_ctx}"
+ )
diff --git a/rag_apps/hybrid_rag_system/src/document_processor.py b/rag_apps/hybrid_rag_system/src/document_processor.py
new file mode 100644
index 0000000..40318be
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/document_processor.py
@@ -0,0 +1,70 @@
+import hashlib
+import json
+from pathlib import Path
+from typing import Optional
+import pypdf
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+from config import CACHE_FILE, CHUNK_SIZE, CHUNK_OVERLAP
+
+
+def load_document(file_bytes: bytes, filename: str) -> str:
+ suffix = Path(filename).suffix.lower()
+ if suffix == ".pdf":
+ import io
+ reader = pypdf.PdfReader(io.BytesIO(file_bytes))
+ pages = []
+ for page in reader.pages:
+ text = page.extract_text()
+ if text:
+ pages.append(text)
+ return "\n\n".join(pages)
+ elif suffix == ".txt":
+ return file_bytes.decode("utf-8", errors="replace")
+ else:
+ raise ValueError(f"Unsupported file type: {suffix}. Use PDF or TXT.")
+
+
+def chunk_text(text: str) -> list[str]:
+ splitter = RecursiveCharacterTextSplitter(
+ chunk_size=CHUNK_SIZE,
+ chunk_overlap=CHUNK_OVERLAP,
+ separators=["\n\n", "\n", ". ", "! ", "? ", " ", ""],
+ )
+ return [c for c in splitter.split_text(text) if c.strip()]
+
+
+def compute_hash(content: str) -> str:
+ return hashlib.md5(content.encode()).hexdigest()
+
+
+def load_cache() -> dict:
+ if CACHE_FILE.exists():
+ try:
+ return json.loads(CACHE_FILE.read_text(encoding="utf-8"))
+ except Exception:
+ return {}
+ return {}
+
+
+def save_cache(cache: dict) -> None:
+ CACHE_FILE.write_text(json.dumps(cache, indent=2), encoding="utf-8")
+
+
+def is_cached(content_hash: str) -> Optional[dict]:
+ return load_cache().get(content_hash)
+
+
+def add_to_cache(content_hash: str, filename: str, num_chunks: int, num_entities: int) -> None:
+ cache = load_cache()
+ cache[content_hash] = {
+ "filename": filename,
+ "num_chunks": num_chunks,
+ "num_entities": num_entities,
+ "hash": content_hash,
+ }
+ save_cache(cache)
+
+
+def list_cached_docs() -> list[dict]:
+ return list(load_cache().values())
diff --git a/rag_apps/hybrid_rag_system/src/graph_indexer.py b/rag_apps/hybrid_rag_system/src/graph_indexer.py
new file mode 100644
index 0000000..affd22b
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/graph_indexer.py
@@ -0,0 +1,147 @@
+import json
+import re
+import time
+from typing import Callable, Optional
+import networkx as nx
+from mistralai.client import Mistral
+
+from config import GRAPHS_DIR, COMBINED_GRAPH_FILE
+
+_EXTRACT_PROMPT = """Extract entities and relationships from the text below.
+
+Return ONLY a valid JSON object with this exact structure (no extra text):
+{{
+ "entities": [
+ {{"name": "EntityName", "type": "EntityType", "description": "one sentence"}}
+ ],
+ "relationships": [
+ {{"source": "Entity1", "target": "Entity2", "relation": "RELATION_TYPE", "description": "one sentence"}}
+ ]
+}}
+
+Entity types: PERSON, ORGANIZATION, LOCATION, CONCEPT, EVENT, TECHNOLOGY, PRODUCT, OTHER
+Relation types: WORKS_FOR, LOCATED_IN, PART_OF, CREATED_BY, CAUSES, RELATED_TO, LEADS, USES, OWNS, MEMBER_OF
+
+Extract only the most significant entities and relationships. Keep names concise.
+
+TEXT:
+{text}
+
+JSON:"""
+
+
+def _parse_extraction(raw: str) -> dict:
+ match = re.search(r'\{.*\}', raw, re.DOTALL)
+ if match:
+ try:
+ return json.loads(match.group())
+ except json.JSONDecodeError:
+ pass
+ return {"entities": [], "relationships": []}
+
+
+def _extract_from_chunk(client: Mistral, text: str) -> dict:
+ for attempt in range(3):
+ try:
+ resp = client.chat.complete(
+ model="mistral-small-latest",
+ messages=[{"role": "user", "content": _EXTRACT_PROMPT.format(text=text[:2500])}],
+ temperature=0.1,
+ max_tokens=800,
+ )
+ return _parse_extraction(resp.choices[0].message.content.strip())
+ except Exception:
+ if attempt < 2:
+ time.sleep(1.5 * (attempt + 1))
+ return {"entities": [], "relationships": []}
+
+
+def build_doc_graph(
+ chunks: list[str],
+ client: Mistral,
+ progress_cb: Optional[Callable[[int, int], None]] = None,
+) -> nx.Graph:
+ G = nx.Graph()
+ for i, chunk in enumerate(chunks):
+ if progress_cb:
+ progress_cb(i + 1, len(chunks))
+
+ extracted = _extract_from_chunk(client, chunk)
+
+ for ent in extracted.get("entities", []):
+ name = ent.get("name", "").strip()
+ if not name or len(name) > 100:
+ continue
+ if G.has_node(name):
+ G.nodes[name]["mentions"] = G.nodes[name].get("mentions", 1) + 1
+ else:
+ G.add_node(
+ name,
+ type=ent.get("type", "OTHER"),
+ description=ent.get("description", ""),
+ mentions=1,
+ )
+
+ for rel in extracted.get("relationships", []):
+ src = rel.get("source", "").strip()
+ tgt = rel.get("target", "").strip()
+ if not src or not tgt or len(src) > 100 or len(tgt) > 100:
+ continue
+ for node in (src, tgt):
+ if not G.has_node(node):
+ G.add_node(node, type="OTHER", description="", mentions=1)
+ if G.has_edge(src, tgt):
+ existing = G[src][tgt].get("relations", [])
+ new_rel = rel.get("relation", "RELATED_TO")
+ if new_rel not in existing:
+ existing.append(new_rel)
+ G[src][tgt]["relations"] = existing
+ else:
+ G.add_edge(
+ src, tgt,
+ relations=[rel.get("relation", "RELATED_TO")],
+ description=rel.get("description", ""),
+ )
+ return G
+
+
+def merge_into_combined(G_new: nx.Graph) -> nx.Graph:
+ G = load_combined_graph()
+ for node, data in G_new.nodes(data=True):
+ if G.has_node(node):
+ G.nodes[node]["mentions"] = G.nodes[node].get("mentions", 1) + data.get("mentions", 1)
+ if not G.nodes[node].get("description") and data.get("description"):
+ G.nodes[node]["description"] = data["description"]
+ else:
+ G.add_node(node, **data)
+ for src, tgt, data in G_new.edges(data=True):
+ if G.has_edge(src, tgt):
+ merged_rels = list(set(
+ G[src][tgt].get("relations", []) + data.get("relations", [])
+ ))
+ G[src][tgt]["relations"] = merged_rels
+ else:
+ G.add_edge(src, tgt, **data)
+ save_combined_graph(G)
+ return G
+
+
+def save_doc_graph(G: nx.Graph, doc_hash: str) -> None:
+ path = GRAPHS_DIR / f"{doc_hash}.json"
+ data = nx.node_link_data(G)
+ path.write_text(json.dumps(data, indent=2), encoding="utf-8")
+
+
+def load_combined_graph() -> nx.Graph:
+ if COMBINED_GRAPH_FILE.exists():
+ try:
+ data = json.loads(COMBINED_GRAPH_FILE.read_text(encoding="utf-8"))
+ return nx.node_link_graph(data)
+ except Exception:
+ pass
+ return nx.Graph()
+
+
+def save_combined_graph(G: nx.Graph) -> None:
+ data = nx.node_link_data(G)
+ COMBINED_GRAPH_FILE.write_text(json.dumps(data, indent=2), encoding="utf-8")
diff --git a/rag_apps/hybrid_rag_system/src/graph_retriever.py b/rag_apps/hybrid_rag_system/src/graph_retriever.py
new file mode 100644
index 0000000..139963c
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/graph_retriever.py
@@ -0,0 +1,91 @@
+import re
+from typing import Any
+import networkx as nx
+
+_STOP_WORDS = {
+ "the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
+ "have", "has", "had", "do", "does", "did", "will", "would", "could",
+ "should", "may", "might", "what", "how", "when", "where", "who",
+ "which", "that", "this", "these", "those", "and", "or", "but", "in",
+ "on", "at", "to", "for", "of", "with", "by", "from", "about", "tell",
+ "me", "explain", "describe", "give", "list", "show", "find", "get",
+ "can", "please", "provide", "summarize", "discuss", "compare",
+}
+
+
+def _query_terms(query: str) -> list[str]:
+ words = re.findall(r'\b\w+\b', query.lower())
+ return [w for w in words if w not in _STOP_WORDS and len(w) > 2]
+
+
+def _score_node(node_name: str, terms: list[str]) -> float:
+ name_lower = node_name.lower()
+ score = 0.0
+ for term in terms:
+ if term == name_lower:
+ score += 2.0
+ elif term in name_lower or name_lower in term:
+ score += 1.0
+ return score
+
+
+def retrieve_from_graph(G: nx.Graph, query: str, max_entities: int) -> dict[str, Any]:
+ if G.number_of_nodes() == 0:
+ return {"entities": [], "relationships": [], "seed_nodes": []}
+
+ terms = _query_terms(query)
+
+ scored = [
+ (node, _score_node(node, terms))
+ for node in G.nodes()
+ ]
+ scored.sort(key=lambda x: (x[1], G.nodes[x[0]].get("mentions", 1)), reverse=True)
+
+ seed_nodes = [n for n, s in scored if s > 0][:5]
+ if not seed_nodes:
+ seed_nodes = [n for n, _ in scored[:3]]
+
+ visited_nodes: set[str] = set()
+ visited_edges: set[tuple] = set()
+ queue = [(n, 0) for n in seed_nodes]
+
+ while queue and len(visited_nodes) < max_entities:
+ node, depth = queue.pop(0)
+ if node in visited_nodes:
+ continue
+ visited_nodes.add(node)
+ if depth >= 2:
+ continue
+ for neighbor in G.neighbors(node):
+ edge_key = tuple(sorted([node, neighbor]))
+ visited_edges.add(edge_key)
+ if neighbor not in visited_nodes:
+ queue.append((neighbor, depth + 1))
+
+ entities = []
+ for node in visited_nodes:
+ d = G.nodes[node]
+ entities.append({
+ "name": node,
+ "type": d.get("type", "OTHER"),
+ "description": d.get("description", ""),
+ "mentions": d.get("mentions", 1),
+ })
+ entities.sort(key=lambda x: x["mentions"], reverse=True)
+
+ relationships = []
+ for src, tgt in visited_edges:
+ if G.has_edge(src, tgt):
+ ed = G[src][tgt]
+ relationships.append({
+ "source": src,
+ "target": tgt,
+ "relations": ed.get("relations", ["RELATED_TO"]),
+ "description": ed.get("description", ""),
+ })
+
+ return {
+ "entities": entities,
+ "relationships": relationships,
+ "seed_nodes": seed_nodes,
+ }
diff --git a/rag_apps/hybrid_rag_system/src/mistral_client.py b/rag_apps/hybrid_rag_system/src/mistral_client.py
new file mode 100644
index 0000000..17a77a3
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/mistral_client.py
@@ -0,0 +1,33 @@
+from mistralai.client import Mistral
+
+_SYSTEM = (
+ "You are an expert research assistant. Answer questions using the provided context "
+ "which comes from two sources: a knowledge graph (entities and relationships) and "
+ "relevant document chunks. Synthesize both sources to give a comprehensive, accurate answer. "
+ "If the context is insufficient, say so clearly and state what information is missing."
+)
+
+_USER_TEMPLATE = """\
+CONTEXT:
+{context}
+
+QUESTION:
+{question}
+
+Provide a clear, well-structured answer based on the context above."""
+
+
+def generate_answer(client: Mistral, question: str, fused_context: str) -> str:
+ resp = client.chat.complete(
+ model="mistral-small-latest",
+ messages=[
+ {"role": "system", "content": _SYSTEM},
+ {"role": "user", "content": _USER_TEMPLATE.format(
+ context=fused_context,
+ question=question,
+ )},
+ ],
+ temperature=0.3,
+ max_tokens=2048,
+ )
+ return resp.choices[0].message.content
diff --git a/rag_apps/hybrid_rag_system/src/vector_indexer.py b/rag_apps/hybrid_rag_system/src/vector_indexer.py
new file mode 100644
index 0000000..7f0c053
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/vector_indexer.py
@@ -0,0 +1,68 @@
+from typing import Callable, Optional
+import requests
+import chromadb
+from chromadb import EmbeddingFunction, Documents, Embeddings
+
+from config import CHROMA_DIR, CHROMA_COLLECTION, OLLAMA_URL, EMBED_MODEL
+
+
+class OllamaEmbeddingFunction(EmbeddingFunction):
+ def __call__(self, input: Documents) -> Embeddings:
+ embeddings = []
+ for text in input:
+ resp = requests.post(
+ f"{OLLAMA_URL}/api/embeddings",
+ json={"model": EMBED_MODEL, "prompt": text},
+ timeout=60,
+ )
+ resp.raise_for_status()
+ embeddings.append(resp.json()["embedding"])
+ return embeddings
+
+
+def check_ollama() -> tuple[bool, str]:
+ try:
+ resp = requests.get(f"{OLLAMA_URL}/api/tags", timeout=5)
+ if resp.status_code == 200:
+ models = [m["name"] for m in resp.json().get("models", [])]
+ has_embed = any(EMBED_MODEL in m for m in models)
+ if not has_embed:
+ return False, f"Model '{EMBED_MODEL}' not found. Run: ollama pull {EMBED_MODEL}"
+ return True, "Ollama is running"
+ return False, "Ollama returned unexpected status"
+ except Exception as e:
+ return False, f"Ollama not reachable at {OLLAMA_URL}: {e}"
+
+
+def get_chroma_collection() -> chromadb.Collection:
+ client = chromadb.PersistentClient(path=str(CHROMA_DIR))
+ return client.get_or_create_collection(
+ name=CHROMA_COLLECTION,
+ embedding_function=OllamaEmbeddingFunction(),
+ metadata={"hnsw:space": "cosine"},
+ )
+
+
+def index_chunks(
+ chunks: list[str],
+ collection: chromadb.Collection,
+ doc_hash: str,
+ progress_cb: Optional[Callable[[int, int], None]] = None,
+) -> None:
+ batch_size = 5
+ total = len(chunks)
+ for i in range(0, total, batch_size):
+ batch = chunks[i : i + batch_size]
+ ids = [f"{doc_hash}_{i + j}" for j in range(len(batch))]
+ metadatas = [{"doc_hash": doc_hash, "chunk_index": i + j} for j in range(len(batch))]
+ collection.add(documents=batch, ids=ids, metadatas=metadatas)
+ if progress_cb:
+ progress_cb(min(i + batch_size, total), total)
+
+
+def collection_has_doc(collection: chromadb.Collection, doc_hash: str) -> bool:
+ try:
+ results = collection.get(where={"doc_hash": doc_hash}, limit=1)
+ return len(results["ids"]) > 0
+ except Exception:
+ return False
diff --git a/rag_apps/hybrid_rag_system/src/vector_retriever.py b/rag_apps/hybrid_rag_system/src/vector_retriever.py
new file mode 100644
index 0000000..b1cebf7
--- /dev/null
+++ b/rag_apps/hybrid_rag_system/src/vector_retriever.py
@@ -0,0 +1,33 @@
+from typing import Any
+import chromadb
+
+
+def retrieve_from_vector(
+ collection: chromadb.Collection,
+ query: str,
+ top_k: int,
+) -> list[dict[str, Any]]:
+ count = collection.count()
+ if count == 0:
+ return []
+
+ n = min(top_k, count)
+ results = collection.query(
+ query_texts=[query],
+ n_results=n,
+ include=["documents", "metadatas", "distances"],
+ )
+
+ chunks = []
+ docs = results.get("documents", [[]])[0]
+ metas = results.get("metadatas", [[]])[0]
+ dists = results.get("distances", [[]])[0]
+
+ for doc, meta, dist in zip(docs, metas, dists):
+ chunks.append({
+ "text": doc,
+ "metadata": meta,
+ "similarity": round(1.0 - dist, 4),
+ })
+
+ return chunks