Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

feat: add YouTube Transcript RAG using Whisper, ChromaDB and Mistral

Tiioluwanibd862d9f2d6f
diff --git a/rag_apps/youtube_transcript_rag/.env.example b/rag_apps/youtube_transcript_rag/.env.example
new file mode 100644
index 0000000..7296f5c
--- /dev/null
+++ b/rag_apps/youtube_transcript_rag/.env.example
@@ -0,0 +1,2 @@
+# Get your key at https://platform.mistral.ai
+MISTRAL_API_KEY=your_mistral_api_key_here
diff --git a/rag_apps/youtube_transcript_rag/README.md b/rag_apps/youtube_transcript_rag/README.md
new file mode 100644
index 0000000..2440584
--- /dev/null
+++ b/rag_apps/youtube_transcript_rag/README.md
@@ -0,0 +1,126 @@
+# YouTube Transcript RAG
+
+> Chat with any YouTube video using local transcription and semantic search powered by Mistral AI.
+
+## Overview
+
+Paste a YouTube URL, and the app downloads the audio, transcribes it locally with Whisper, and stores the transcript as embeddings in ChromaDB. You can then ask questions in a chat interface and get answers grounded in the video content — complete with clickable timestamp links pointing to the exact moment in the video.
+
+## Demo
+
+![YouTube Transcript RAG Demo](assets/demo.png)
+
+## Features
+
+- Local transcription with OpenAI Whisper — no third-party transcription API needed
+- Semantic retrieval of the top 5 most relevant transcript chunks per question
+- Grounded answers from Mistral Small 4 with inline timestamp references
+- Persistent ChromaDB storage — the same URL is never re-transcribed
+- Source chunks shown in a collapsible expander with clickable timestamps
+- Multi-turn chat with the last 10 turns kept in context
+
+## Tech Stack
+
+| Layer | Library / Tool |
+|---|---|
+| UI | Streamlit |
+| Audio download | yt-dlp + ffmpeg |
+| Transcription | OpenAI Whisper (local) |
+| Embeddings | sentence-transformers `all-MiniLM-L6-v2` (local) |
+| Vector database | ChromaDB (local persistent storage) |
+| LLM | Mistral Small 4 (`mistral-small-latest`) |
+| LLM framework | LangChain (`langchain-mistralai`) |
+
+## Prerequisites
+
+- Python 3.10+
+- ffmpeg
+
+**Install ffmpeg:**
+
+```bash
+# Windows
+winget install ffmpeg
+
+# macOS
+brew install ffmpeg
+```
+
+A [Mistral AI API key](https://platform.mistral.ai) is required for answer generation. Transcription and embeddings run entirely locally.
+
+## Installation
+
+**1. Clone the repository**
+
+```bash
+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
+cd Hands-On-AI-Engineering/rag_apps/youtube_transcript_rag
+```
+
+**2. Create and activate a virtual environment**
+
+```bash
+# Windows
+python -m venv .venv
+.venv\Scripts\activate
+
+# macOS / Linux
+python -m venv .venv
+source .venv/bin/activate
+```
+
+**3. Install dependencies**
+
+```bash
+pip install -r requirements.txt
+```
+
+**4. Configure your API key**
+
+```bash
+cp .env.example .env
+```
+
+Open `.env` and set your `MISTRAL_API_KEY`.
+
+## Usage
+
+```bash
+streamlit run app.py
+```
+
+The app opens in your browser at `http://localhost:8501`.
+
+1. Paste a YouTube URL in the sidebar — for example:
+ ```text
+ https://www.youtube.com/watch?v=dQw4w9WgXcQ
+ ```
+2. Click **Process Video**. Whisper transcribes the audio locally (a few minutes for longer videos).
+3. Ask a question in the chat input — for example:
+ ```text
+ What is the main topic discussed in the video?
+ ```
+4. The app retrieves the most relevant transcript segments and returns a grounded answer:
+ ```text
+ The video focuses on [topic], as explained at [0:42](https://youtu.be/dQw4w9WgXcQ?t=42)
+ and revisited at [2:15](https://youtu.be/dQw4w9WgXcQ?t=135).
+ ```
+5. Expand **Source transcript chunks** below the answer to read the raw segments and jump to any timestamp.
+
+Pasting the same URL again loads the transcript from ChromaDB instantly — no re-transcription.
+
+## Environment Variables
+
+| Variable | Description |
+|---|---|
+| `MISTRAL_API_KEY` | API key from [platform.mistral.ai](https://platform.mistral.ai) |
+
+## Project Structure
+
+```text
+youtube-transcript-rag/
+├── app.py # Streamlit app — pipeline, retrieval, chat UI
+├── requirements.txt # Python dependencies
+├── .env.example # Environment variable template
+└── chroma_db/ # Persistent ChromaDB storage (auto-created)
+```
diff --git a/rag_apps/youtube_transcript_rag/app.py b/rag_apps/youtube_transcript_rag/app.py
new file mode 100644
index 0000000..8c1e08b
--- /dev/null
+++ b/rag_apps/youtube_transcript_rag/app.py
@@ -0,0 +1,362 @@
+import streamlit as st
+import os
+import re
+import tempfile
+from dotenv import load_dotenv
+
+load_dotenv()
+
+# ── Constants ─────────────────────────────────────────────────────────────────
+CHROMA_DB_PATH = "./chroma_db"
+CHUNK_SIZE = 500 # words per chunk
+CHUNK_OVERLAP = 50 # word overlap between chunks
+TOP_K = 5 # chunks retrieved per question
+WHISPER_MODEL_SIZE = "base"
+EMBEDDING_MODEL = "all-MiniLM-L6-v2"
+MISTRAL_MODEL = "mistral-small-latest"
+
+
+# ── Helpers ───────────────────────────────────────────────────────────────────
+
+def extract_video_id(url: str) -> str | None:
+ patterns = [
+ r"[?&]v=([a-zA-Z0-9_-]{11})",
+ r"youtu\.be/([a-zA-Z0-9_-]{11})",
+ r"youtube\.com/embed/([a-zA-Z0-9_-]{11})",
+ r"youtube\.com/v/([a-zA-Z0-9_-]{11})",
+ r"youtube\.com/shorts/([a-zA-Z0-9_-]{11})",
+ ]
+ for pattern in patterns:
+ m = re.search(pattern, url)
+ if m:
+ return m.group(1)
+ return None
+
+
+def collection_name_for(video_id: str) -> str:
+ # ChromaDB: 3-63 chars, [a-zA-Z0-9_-], no leading/trailing - or _
+ # Wrapping with alpha chars guarantees valid boundaries
+ name = f"yt{video_id}yt"
+ return name[:63]
+
+
+def fmt_ts(seconds: float) -> str:
+ h = int(seconds // 3600)
+ m = int((seconds % 3600) // 60)
+ s = int(seconds % 60)
+ return f"{h}:{m:02d}:{s:02d}" if h else f"{m}:{s:02d}"
+
+
+def yt_link(video_id: str, seconds: float) -> str:
+ return f"https://youtu.be/{video_id}?t={int(seconds)}"
+
+
+# ── Cached resources ──────────────────────────────────────────────────────────
+
[email protected]_resource(show_spinner=False)
+def load_whisper_model():
+ import whisper
+ return whisper.load_model(WHISPER_MODEL_SIZE)
+
+
[email protected]_resource(show_spinner=False)
+def get_embedding_fn():
+ from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
+ return SentenceTransformerEmbeddingFunction(model_name=EMBEDDING_MODEL)
+
+
[email protected]_resource(show_spinner=False)
+def get_chroma_client():
+ import chromadb
+ return chromadb.PersistentClient(path=CHROMA_DB_PATH)
+
+
[email protected]_resource(show_spinner=False)
+def get_mistral_llm():
+ from langchain_mistralai import ChatMistralAI
+ api_key = os.getenv("MISTRAL_API_KEY")
+ if not api_key:
+ return None
+ return ChatMistralAI(model=MISTRAL_MODEL, api_key=api_key)
+
+
+# ── Core pipeline ─────────────────────────────────────────────────────────────
+
+def collection_exists(client, name: str) -> bool:
+ try:
+ client.get_collection(name)
+ return True
+ except Exception:
+ return False
+
+
+def download_audio(url: str, out_dir: str) -> str:
+ import yt_dlp
+ out_path = os.path.join(out_dir, "audio")
+ opts = {
+ "format": "bestaudio/best",
+ "outtmpl": out_path,
+ "postprocessors": [{"key": "FFmpegExtractAudio", "preferredcodec": "mp3"}],
+ "quiet": True,
+ "no_warnings": True,
+ }
+ with yt_dlp.YoutubeDL(opts) as ydl:
+ ydl.download([url])
+ return out_path + ".mp3"
+
+
+def transcribe_audio(audio_path: str) -> list[dict]:
+ model = load_whisper_model()
+ result = model.transcribe(audio_path, verbose=False)
+ return result["segments"]
+
+
+def chunk_segments(segments: list[dict]) -> list[dict]:
+ """Flatten whisper segments into overlapping ~500-word chunks with timestamps."""
+ word_entries = []
+ for seg in segments:
+ for word in seg["text"].strip().split():
+ word_entries.append({"word": word, "start": seg["start"]})
+
+ chunks = []
+ i = 0
+ while i < len(word_entries):
+ batch = word_entries[i : i + CHUNK_SIZE]
+ chunks.append({
+ "text": " ".join(e["word"] for e in batch),
+ "start": batch[0]["start"],
+ "index": len(chunks),
+ })
+ i += CHUNK_SIZE - CHUNK_OVERLAP
+
+ return chunks
+
+
+def ingest_video(url: str, video_id: str, status_widget) -> object:
+ """Full pipeline: download → transcribe → chunk → embed → store."""
+ chroma = get_chroma_client()
+ emb_fn = get_embedding_fn()
+ cname = collection_name_for(video_id)
+
+ with tempfile.TemporaryDirectory() as tmp:
+ status_widget.write("⬇️ Downloading audio…")
+ audio_path = download_audio(url, tmp)
+
+ status_widget.write("🎙️ Transcribing with Whisper (this can take a few minutes for long videos)…")
+ segments = transcribe_audio(audio_path)
+
+ status_widget.write(f"✂️ Splitting into chunks…")
+ chunks = chunk_segments(segments)
+
+ status_widget.write(f"🗄️ Generating embeddings and storing in ChromaDB…")
+ col = chroma.get_or_create_collection(cname, embedding_function=emb_fn)
+ col.add(
+ documents=[c["text"] for c in chunks],
+ metadatas=[{"start": c["start"], "video_id": video_id} for c in chunks],
+ ids=[f"{cname}_{c['index']}" for c in chunks],
+ )
+ return col
+
+
+def load_collection(video_id: str) -> object:
+ chroma = get_chroma_client()
+ emb_fn = get_embedding_fn()
+ return chroma.get_collection(collection_name_for(video_id), embedding_function=emb_fn)
+
+
+def retrieve_chunks(collection, question: str) -> list[dict]:
+ res = collection.query(query_texts=[question], n_results=TOP_K)
+ return [
+ {"text": doc, "start": meta["start"], "video_id": meta["video_id"]}
+ for doc, meta in zip(res["documents"][0], res["metadatas"][0])
+ ]
+
+
+def build_context_block(hits: list[dict]) -> str:
+ parts = []
+ for h in hits:
+ ts = fmt_ts(h["start"])
+ link = yt_link(h["video_id"], h["start"])
+ parts.append(f"[Timestamp {ts}]({link})\n{h['text']}")
+ return "\n\n---\n\n".join(parts)
+
+
+def generate_answer(question: str, hits: list[dict], history: list[dict]) -> str:
+ from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
+
+ llm = get_mistral_llm()
+ if llm is None:
+ return "⚠️ `MISTRAL_API_KEY` is not set. Please add it to your `.env` file."
+
+ context = build_context_block(hits)
+ system = (
+ "You are a knowledgeable assistant that answers questions strictly based on provided YouTube video transcripts.\n"
+ "You are given the most relevant transcript chunks, each preceded by a clickable timestamp.\n"
+ "When your answer references specific content, include the timestamp as a markdown link, e.g. "
+ "[1:23](https://youtu.be/VIDEO_ID?t=83).\n"
+ "If the transcript chunks do not contain enough information to answer, say so honestly.\n"
+ "Do not fabricate information.\n\n"
+ f"Relevant transcript segments:\n\n{context}"
+ )
+
+ messages = [SystemMessage(content=system)]
+ for msg in history[-10:]:
+ if msg["role"] == "user":
+ messages.append(HumanMessage(content=msg["content"]))
+ else:
+ messages.append(AIMessage(content=msg["content"]))
+ messages.append(HumanMessage(content=question))
+
+ resp = llm.invoke(messages)
+ return resp.content
+
+
+# ── Streamlit UI ──────────────────────────────────────────────────────────────
+
+st.set_page_config(
+ page_title="YouTube Transcript RAG",
+ page_icon="🎬",
+ layout="wide",
+ initial_sidebar_state="expanded",
+)
+
+# Session state init
+for key, default in {
+ "messages": [],
+ "history": [],
+ "video_id": None,
+ "collection": None,
+}.items():
+ if key not in st.session_state:
+ st.session_state[key] = default
+
+# ── Sidebar ───────────────────────────────────────────────────────────────────
+with st.sidebar:
+ st.title("🎬 YouTube RAG")
+ st.caption("Ask questions about any YouTube video.")
+ st.divider()
+
+ url_input = st.text_input(
+ "YouTube URL",
+ placeholder="https://www.youtube.com/watch?v=...",
+ label_visibility="collapsed",
+ )
+ process_btn = st.button(
+ "▶ Process Video",
+ use_container_width=True,
+ type="primary",
+ disabled=not url_input.strip(),
+ )
+
+ if st.session_state.video_id:
+ vid_id = st.session_state.video_id
+ st.success(f"✓ Loaded: `{vid_id}`")
+ st.markdown(f"[Open on YouTube ↗](https://youtu.be/{vid_id})")
+ if st.button("🔄 Reset", use_container_width=True):
+ st.session_state.messages = []
+ st.session_state.history = []
+ st.session_state.video_id = None
+ st.session_state.collection = None
+ st.rerun()
+
+ st.divider()
+ with st.expander("ℹ️ How it works"):
+ st.markdown(
+ "1. Paste a YouTube URL and click **Process Video**\n"
+ "2. Audio is downloaded with **yt-dlp**\n"
+ "3. Transcribed locally with **OpenAI Whisper**\n"
+ "4. Chunks stored as embeddings in **ChromaDB**\n"
+ "5. Your question retrieves the top 5 relevant chunks\n"
+ "6. **Mistral Small 4** answers with timestamp references\n\n"
+ "Re-pasting the same URL skips transcription — loaded from cache instantly."
+ )
+
+ if not os.getenv("MISTRAL_API_KEY"):
+ st.warning("⚠️ `MISTRAL_API_KEY` not found in `.env`")
+
+# ── Video processing ──────────────────────────────────────────────────────────
+if process_btn and url_input.strip():
+ vid = extract_video_id(url_input.strip())
+ if not vid:
+ st.sidebar.error("Could not find a video ID in that URL.")
+ elif vid == st.session_state.video_id:
+ st.sidebar.info("This video is already loaded and ready.")
+ else:
+ chroma = get_chroma_client()
+ cname = collection_name_for(vid)
+
+ if collection_exists(chroma, cname):
+ with st.spinner("Loading existing transcript from ChromaDB…"):
+ st.session_state.collection = load_collection(vid)
+ st.session_state.video_id = vid
+ st.session_state.messages = []
+ st.session_state.history = []
+ st.toast("Loaded from cache!", icon="✅")
+ st.rerun()
+ else:
+ with st.status("Processing video…", expanded=True) as status:
+ try:
+ col = ingest_video(url_input.strip(), vid, status)
+ st.session_state.collection = col
+ st.session_state.video_id = vid
+ st.session_state.messages = []
+ st.session_state.history = []
+ status.update(label="✅ Video ready! Start asking questions.", state="complete")
+ except Exception as exc:
+ status.update(label=f"Failed: {exc}", state="error")
+ st.error(
+ f"**Processing error:** {exc}\n\n"
+ "Make sure `ffmpeg` is installed and the video is publicly accessible."
+ )
+
+# ── Main chat area ────────────────────────────────────────────────────────────
+st.title("YouTube Transcript RAG")
+st.markdown(
+ "Semantic search over video transcripts — powered by **Whisper**, **ChromaDB** & **Mistral Small 4**."
+)
+
+if st.session_state.collection is None:
+ st.info(
+ "👈 Paste a YouTube URL in the sidebar and click **Process Video** to get started.\n\n"
+ "Transcription happens locally — no OpenAI API key needed."
+ )
+else:
+ vid_id = st.session_state.video_id
+ st.markdown(
+ f"**Chatting about:** [youtube.com/watch?v={vid_id}](https://www.youtube.com/watch?v={vid_id})"
+ )
+ st.divider()
+
+ # Render existing messages
+ for msg in st.session_state.messages:
+ with st.chat_message(msg["role"]):
+ st.markdown(msg["content"])
+
+ # Accept new question
+ if question := st.chat_input("Ask something about the video…"):
+ # Show user message immediately
+ st.session_state.messages.append({"role": "user", "content": question})
+ with st.chat_message("user"):
+ st.markdown(question)
+
+ # Retrieve & generate
+ with st.chat_message("assistant"):
+ with st.spinner("Searching transcript and generating answer…"):
+ hits = retrieve_chunks(st.session_state.collection, question)
+ answer = generate_answer(question, hits, st.session_state.history)
+ st.markdown(answer)
+
+ # Show source chunks in expander
+ with st.expander("📄 Source transcript chunks", expanded=False):
+ for i, h in enumerate(hits, 1):
+ ts = fmt_ts(h["start"])
+ link = yt_link(h["video_id"], h["start"])
+ st.markdown(f"**Chunk {i}** — [{ts}]({link})")
+ st.caption(h["text"])
+ if i < len(hits):
+ st.divider()
+
+ # Persist to session state
+ st.session_state.messages.append({"role": "assistant", "content": answer})
+ st.session_state.history.append({"role": "user", "content": question})
+ st.session_state.history.append({"role": "assistant", "content": answer})
diff --git a/rag_apps/youtube_transcript_rag/assets/demo.png b/rag_apps/youtube_transcript_rag/assets/demo.png
new file mode 100644
index 0000000..631abc2
Binary files /dev/null and b/rag_apps/youtube_transcript_rag/assets/demo.png differ
diff --git a/rag_apps/youtube_transcript_rag/requirements.txt b/rag_apps/youtube_transcript_rag/requirements.txt
new file mode 100644
index 0000000..a00ddfd
--- /dev/null
+++ b/rag_apps/youtube_transcript_rag/requirements.txt
@@ -0,0 +1,9 @@
+streamlit>=1.32.0
+yt-dlp>=2024.1.1
+openai-whisper>=20231117
+chromadb>=0.5.0
+sentence-transformers>=3.0.0
+langchain-mistralai>=0.2.0
+langchain-core>=0.3.0
+python-dotenv>=1.0.0
+torch>=2.1.0