Skip to content

armgit / camel

Public Imported from GitHub
0

๐Ÿซ CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org

usecase: Youtube ocr usecase (#2523)

Bipul7070129ea1f178d8a
diff --git a/examples/usecases/youtube_ocr/README.md b/examples/usecases/youtube_ocr/README.md
new file mode 100644
index 000000000..c917acec6
--- /dev/null
+++ b/examples/usecases/youtube_ocr/README.md
@@ -0,0 +1,107 @@
+# ๐ŸŽฅ YouTube Video Q&A with CAMEL (Audio + Visual OCR)
+
+This project enables users to input a YouTube URL and ask questions about the video's **spoken content** and **on-screen text (e.g., PPT slides)**. It integrates:
+
+- โœ… Audio transcription via Whisper
+- ๐Ÿ–ผ๏ธ Visual OCR from extracted video frames
+- ๐Ÿง  CAMEL AI multi-agent reasoning
+- ๐ŸŒ Streamlit interface for interaction
+
+---
+
+## ๐Ÿš€ Features
+
+- Downloads YouTube videos via CAMEL toolkit
+- Transcribes audio using OpenAI Whisper
+- Extracts key video frames at regular intervals
+- Performs OCR on frames to extract visible text
+- Combines transcript and OCR data as knowledge
+- Allows users to ask questions answered by CAMEL agent
+
+---
+
+## ๐Ÿ“ฆ Requirements
+
+Install Python packages with specific versions:
+
+```bash
+pip install -r requirements.txt
+```
+
+### ๐Ÿงฐ Also install system dependencies:
+
+**Tesseract OCR:**
+
+- **Windows:** [Download here](https://github.com/UB-Mannheim/tesseract/wiki)
+- **macOS:**
+ ```bash
+ brew install tesseract
+ ```
+- **Ubuntu:**
+ ```bash
+ sudo apt-get install tesseract-ocr
+ ```
+
+**FFmpeg** (required by Whisper and for frame extraction):
+
+- **Windows:** [Download here](https://ffmpeg.org/download.html) (add `/bin` to PATH)
+- **macOS:**
+ ```bash
+ brew install ffmpeg
+ ```
+- **Ubuntu:**
+ ```bash
+ sudo apt-get install ffmpeg
+ ```
+
+---
+
+## ๐Ÿ› ๏ธ Setup for Windows
+
+If Tesseract is not found, add this in your Python code:
+
+```python
+import pytesseract
+pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"
+```
+
+Ensure the path is correct based on your installation.
+
+---
+
+## ๐Ÿงช Run the App
+
+```bash
+streamlit run app.py
+```
+
+Then, open the Streamlit interface in your browser.
+
+---
+
+## ๐Ÿ“ File Structure
+
+```
+app.py # Main Streamlit app
+README.md # This file
+requirements.txt # Python dependencies
+```
+
+---
+
+## ๐Ÿง  Powered By
+
+- [CAMEL-AI](https://github.com/camel-ai/camel)
+- [Whisper by OpenAI](https://github.com/openai/whisper)
+- [Tesseract OCR](https://github.com/tesseract-ocr/tesseract)
+- [Streamlit](https://streamlit.io/)
+
+---
+
+## ๐Ÿ’ฌ Example Usage
+
+1. Paste a YouTube link of a tech tutorial with slides.
+2. Ask: _"What is the title of the slide shown at 5 minutes?"_
+3. The app returns relevant text from OCR and transcript.
+
+---
diff --git a/examples/usecases/youtube_ocr/app.py b/examples/usecases/youtube_ocr/app.py
new file mode 100644
index 000000000..8a486203d
--- /dev/null
+++ b/examples/usecases/youtube_ocr/app.py
@@ -0,0 +1,72 @@
+# ========= Copyright 2023-2024 @ CAMEL-AI.org. All Rights Reserved. =========
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ========= Copyright 2023-2024 @ CAMEL-AI.org. All Rights Reserved. =========
+
+import ffmpeg, cv2, whisper, pytesseract, streamlit as st
+from camel.toolkits.video_download_toolkit import VideoDownloaderToolkit
+from camel.agents import ChatAgent
+from camel.messages import BaseMessage
+from PIL import Image
+from dotenv import load_dotenv
+import os
+
+load_dotenv()
+pytesseract.pytesseract.tesseract_cmd = os.getenv("TESSERACT_CMD", "tesseract")
+
+
+# Initialize CAMEL tools/agent
+video_tool = VideoDownloaderToolkit()
+agent = ChatAgent()
+
+def process_video(url, question):
+ video_path = video_tool.download_video(url)
+ # Extract audio
+ audio_path = "audio.wav"
+ ffmpeg.input(video_path).output(audio_path, acodec='pcm_s16le', ac=1, ar='16k').run()
+ # Transcribe
+ model = whisper.load_model("base")
+ transcript = model.transcribe(audio_path)["text"]
+ # Extract frames every 4s
+ cap = cv2.VideoCapture(video_path)
+ fps = cap.get(cv2.CAP_PROP_FPS); interval = 4
+ frames = []
+ frame_count = 0
+ while True:
+ ret, frame = cap.read()
+ if not ret: break
+ if frame_count % int(fps*interval) == 0:
+ fname = f"frame_{frame_count}.png"
+ cv2.imwrite(fname, frame)
+ frames.append(fname)
+ frame_count += 1
+ cap.release()
+ # OCR on frames
+ ocr_texts = []
+ for img_path in frames:
+ text = pytesseract.image_to_string(Image.open(img_path))
+ ocr_texts.append(text)
+ ocr_content = "\n".join(ocr_texts)
+ # Prepare context and query agent
+ knowledge = f"Transcript:\n{transcript}\n\nOn-screen Text:\n{ocr_content}"
+ user_msg = BaseMessage.make_user_message(role_name="User",
+ content=f"{knowledge}\n\nQuestion: {question}")
+ response = agent.step(user_msg)
+ return response.msgs[0].content
+
+# Streamlit UI
+st.title("YouTube Video Q&A with CAMEL")
+youtube_url = st.text_input("YouTube URL:")
+user_q = st.text_input("Your Question:")
+if st.button("Ask"):
+ answer = process_video(youtube_url, user_q)
+ st.write(answer)
diff --git a/examples/usecases/youtube_ocr/requirements.txt b/examples/usecases/youtube_ocr/requirements.txt
new file mode 100644
index 000000000..e1b759b9d
--- /dev/null
+++ b/examples/usecases/youtube_ocr/requirements.txt
@@ -0,0 +1,9 @@
+ffmpeg-python==0.2.0
+opencv-python==4.9.0.80
+streamlit==1.45.0
+pytesseract==0.3.13
+streamlit==1.45.1
+camel-ai==0.2.61
+python-dotenv==1.1.0
+Pillow==11.2.1
+git+https://github.com/openai/whisper.git