Skip to content

armgit / camel

Public Imported from GitHub
0

๐Ÿซ CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org

usecase: youtube video comprehension (#2503)

Bipul70701498fc822a084
diff --git a/examples/usecases/chat_with_youtube/README.md b/examples/usecases/chat_with_youtube/README.md
new file mode 100644
index 000000000..3a4188de4
--- /dev/null
+++ b/examples/usecases/chat_with_youtube/README.md
@@ -0,0 +1,146 @@
+# ๐ŸŽฅ Video Content Q&A with CAMEL-AI
+
+An intelligent Streamlit-based application that uses the [CAMEL-AI](https://www.camel-ai.org/) framework to allow users to extract, understand, and query the contents of YouTube videos. It leverages CAMEL toolkits and OpenAI models for transcription, summarization, and natural language Q&A.
+
+---
+
+## ๐Ÿš€ Features
+
+- **YouTube Video Downloader**: Fetches video content directly from YouTube using CAMEL's `VideoDownloaderToolkit`.
+- **Audio Transcription**: Extracts and transcribes audio with `AudioAnalysisToolkit`.
+- **Summarization Agent**: Uses `ChatAgent` with GPT-4o-mini to generate concise summaries of the transcript.
+- **Question Answering**: Ask any natural language questions based on the transcript โ€” answers are grounded in the video's content.
+
+---
+
+## ๐Ÿงฐ Toolkits & Architecture
+
+This app utilizes the following CAMEL-AI components:
+
+- `VideoDownloaderToolkit`: For downloading YouTube videos.
+- `AudioAnalysisToolkit`: For audio-to-text transcription.
+- `ChatAgent`: To summarize transcripts and answer questions using OpenAI models.
+
+---
+
+## ๐Ÿ› ๏ธ Installation
+
+1. **Clone the Repository**:
+
+ ```bash
+ git clone https://github.com/camel-ai/camel.git
+ cd examples/usecases/chat_with_youtube
+ ```
+
+2. **Set Up Virtual Environment**:
+
+ ```bash
+ python -m venv venv
+ source venv/bin/activate # On Windows: venv\Scripts\activate
+ ```
+
+3. **Install Dependencies**:
+
+ ```bash
+ pip install -r requirements.txt
+ ```
+
+4. **Configure Environment Variables**:
+
+ Create a `.env` file and add your OpenAI API key:
+
+ ```env
+ OPENAI_API_KEY=your_openai_api_key
+ ```
+
+5. **Install ffmpeg** (for audio extraction):
+
+ - macOS: `brew install ffmpeg`
+ - Ubuntu: `sudo apt install ffmpeg`
+ - Windows: [Download from ffmpeg.org](https://ffmpeg.org/download.html)
+
+---
+## ๐Ÿช Using Authentication Cookies with yt-dlp
+
+YouTube may require authentication for certain videos. To allow yt-dlp to access such content, you need to provide your own `cookies.txt` file.
+
+### Exporting Cookies from Your Browser
+
+1. **Using a Browser Extension**:
+
+ - **Chrome**: Install [Get cookies.txt LOCALLY](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)
+ - **Firefox**: Install [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/)
+
+ After installing the extension:
+
+ - Navigate to [YouTube](https://www.youtube.com) and log in.
+ - Click the extension icon and export the cookies.
+ - Save the file as `cookies.txt` in the `examples/usecases/chat_with_youtube` directory.
+
+
+### Updating the Application to Use Your Cookies
+
+In your code, specify the path to your `cookies.txt`:
+
+```python
+video_toolkit = VideoAnalysisToolkit(cookies_path="cookies.txt")
+```
+
+Ensure that this path points to your exported `cookies.txt` file.
+
+---
+
+## โ–ถ๏ธ Running the Application
+
+Start the Streamlit server:
+
+```bash
+streamlit run app.py
+```
+
+Then, paste a YouTube link, and the app will download the video, transcribe it, summarize it, and let you chat with it.
+
+---
+
+## ๐Ÿ“‚ Project Structure
+
+```
+video-qa-camel/
+โ”œโ”€โ”€ app.py # Main Streamlit application
+โ”œโ”€โ”€ downloads/ # Temporary video/audio files
+โ”œโ”€โ”€ .env # Environment variables (not committed)
+โ”œโ”€โ”€ requirements.txt # Python dependencies
+โ””โ”€โ”€ README.md # Project documentation
+```
+
+---
+
+## ๐Ÿ’ก Example Usage
+
+1. Launch the app using Streamlit.
+2. Paste a YouTube video URL.
+3. Wait for the download, audio extraction, and transcription to complete.
+4. View the summary.
+5. Ask any question based on the video's content and get an AI-generated answer.
+
+---
+
+## ๐Ÿ“Œ Notes
+
+- This app does not yet support chunked retrieval-based Q&A for long transcripts. For scalable Q&A over long-form content, consider integrating an external vector database (e.g., Milvus, Pinecone) with chunking and embedding.
+- All answers are context-aware, based on the entire transcript, but may be truncated for long videos due to model token limits.
+
+
+---
+
+## ๐Ÿค Contributing
+
+Feel free to open pull requests, issues, or discussions for improvements, new features, or bug fixes.
+
+---
+
+## ๐Ÿง  Powered by
+
+- [CAMEL-AI](https://github.com/camel-ai/camel)
+- [OpenAI GPT Models](https://openai.com/)
+- [Streamlit](https://streamlit.io/)
diff --git a/examples/usecases/chat_with_youtube/app.py b/examples/usecases/chat_with_youtube/app.py
new file mode 100644
index 000000000..0332f974d
--- /dev/null
+++ b/examples/usecases/chat_with_youtube/app.py
@@ -0,0 +1,76 @@
+# ========= Copyright 2023-2024 @ CAMEL-AI.org. All Rights Reserved. =========
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ========= Copyright 2023-2024 @ CAMEL-AI.org. All Rights Reserved. =========
+
+import streamlit as st
+import ffmpeg
+from camel.toolkits.video_download_toolkit import VideoDownloaderToolkit
+from camel.toolkits.audio_analysis_toolkit import AudioAnalysisToolkit
+from camel.agents import ChatAgent
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+# Initialize toolkits
+video_downloader = VideoDownloaderToolkit(download_directory="downloads/")
+audio_toolkit = AudioAnalysisToolkit(cache_dir="downloads/")
+summarizer_agent = ChatAgent(system_message="You are a helpful assistant that summarizes transcripts.", model="gpt-4o-mini")
+qa_agent = ChatAgent(system_message="Answer the question based on the context provided.", model="gpt-4o-mini")
+
+st.title("๐ŸŽฅ Video Content Q&A with CAMEL")
+
+# Input URL
+url = st.text_input("Enter a YouTube video URL:",value="https://www.youtube.com/watch?v=hT_nvWreIhg")
+
+if st.button("Process Video"):
+ if url:
+ with st.spinner("Downloading and processing video..."):
+ # Download video
+ video_path = video_downloader.download_video(url)
+ st.video(video_path)
+
+ # Extract audio
+ audio_path = os.path.splitext(video_path)[0] + ".wav"
+ ffmpeg.input(video_path).output(audio_path, ac=1, ar=16000).run(overwrite_output=True)
+
+ # Transcribe
+ transcript = audio_toolkit.audio2text(audio_path)
+
+ # Summarize
+ summary = summarizer_agent.step(transcript)
+ st.subheader("๐Ÿ“ Summary")
+ st.write(summary)
+
+ # Store transcript for Q&A
+ st.session_state.transcript = transcript
+ st.session_state.qa_agent = qa_agent
+
+if "transcript" in st.session_state:
+ st.subheader("๐Ÿ’ฌ Ask Questions about the Video")
+
+ if "chat_history" not in st.session_state:
+ st.session_state.chat_history = []
+
+ user_q = st.text_input("Your question:")
+
+ if user_q:
+ transcript = st.session_state.transcript
+ prompt = f"Context: {transcript}\n\nQuestion: {user_q}"
+ answer = st.session_state.qa_agent.step(prompt).msgs[0].content
+
+ st.session_state.chat_history.append((user_q, answer))
+
+ for q, a in st.session_state.chat_history:
+ st.markdown(f"**User:** {q}")
+ st.markdown(f"**AI:** {a}")
diff --git a/examples/usecases/chat_with_youtube/requirements.txt b/examples/usecases/chat_with_youtube/requirements.txt
new file mode 100644
index 000000000..eac962344
--- /dev/null
+++ b/examples/usecases/chat_with_youtube/requirements.txt
@@ -0,0 +1,3 @@
+streamlit==1.45.1
+camel-ai[all]==0.2.61
+docx2markdown==0.1.1
\ No newline at end of file