๐ซ CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org
Python agentai-societiesartificial-intelligencecommunicative-aicooperative-aideep-learninglarge-language-modelsmulti-agent-systemsnatural-language-processing
usecase: Youtube ocr usecase (#2523)
Bipul7070129ea1f178d8a
diff --git a/examples/usecases/youtube_ocr/README.md b/examples/usecases/youtube_ocr/README.mdnew file mode 100644index 000000000..c917acec6--- /dev/null+++ b/examples/usecases/youtube_ocr/README.md@@ -0,0 +1,107 @@+# ๐ฅ YouTube Video Q&A with CAMEL (Audio + Visual OCR)++This project enables users to input a YouTube URL and ask questions about the video's **spoken content** and **on-screen text (e.g., PPT slides)**. It integrates:++- โ Audio transcription via Whisper+- ๐ผ๏ธ Visual OCR from extracted video frames+- ๐ง CAMEL AI multi-agent reasoning+- ๐ Streamlit interface for interaction++---++## ๐ Features++- Downloads YouTube videos via CAMEL toolkit+- Transcribes audio using OpenAI Whisper+- Extracts key video frames at regular intervals+- Performs OCR on frames to extract visible text+- Combines transcript and OCR data as knowledge+- Allows users to ask questions answered by CAMEL agent++---++## ๐ฆ Requirements++Install Python packages with specific versions:++```bash+pip install -r requirements.txt+```++### ๐งฐ Also install system dependencies:++**Tesseract OCR:**++- **Windows:** [Download here](https://github.com/UB-Mannheim/tesseract/wiki)+- **macOS:**+ ```bash+ brew install tesseract+ ```+- **Ubuntu:**+ ```bash+ sudo apt-get install tesseract-ocr+ ```++**FFmpeg** (required by Whisper and for frame extraction):++- **Windows:** [Download here](https://ffmpeg.org/download.html) (add `/bin` to PATH)+- **macOS:**+ ```bash+ brew install ffmpeg+ ```+- **Ubuntu:**+ ```bash+ sudo apt-get install ffmpeg+ ```++---++## ๐ ๏ธ Setup for Windows++If Tesseract is not found, add this in your Python code:++```python+import pytesseract+pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"+```++Ensure the path is correct based on your installation.++---++## ๐งช Run the App++```bash+streamlit run app.py+```++Then, open the Streamlit interface in your browser.++---++## ๐ File Structure++```+app.py # Main Streamlit app+README.md # This file+requirements.txt # Python dependencies+```++---++## ๐ง Powered By++- [CAMEL-AI](https://github.com/camel-ai/camel)+- [Whisper by OpenAI](https://github.com/openai/whisper)+- [Tesseract OCR](https://github.com/tesseract-ocr/tesseract)+- [Streamlit](https://streamlit.io/)++---++## ๐ฌ Example Usage++1. Paste a YouTube link of a tech tutorial with slides.+2. Ask: _"What is the title of the slide shown at 5 minutes?"_+3. The app returns relevant text from OCR and transcript.++---diff --git a/examples/usecases/youtube_ocr/app.py b/examples/usecases/youtube_ocr/app.pynew file mode 100644index 000000000..8a486203d--- /dev/null+++ b/examples/usecases/youtube_ocr/app.py@@ -0,0 +1,72 @@+# ========= Copyright 2023-2024 @ CAMEL-AI.org. All Rights Reserved. =========+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# ========= Copyright 2023-2024 @ CAMEL-AI.org. All Rights Reserved. =========++import ffmpeg, cv2, whisper, pytesseract, streamlit as st+from camel.toolkits.video_download_toolkit import VideoDownloaderToolkit+from camel.agents import ChatAgent+from camel.messages import BaseMessage+from PIL import Image+from dotenv import load_dotenv+import os++load_dotenv()+pytesseract.pytesseract.tesseract_cmd = os.getenv("TESSERACT_CMD", "tesseract")+++# Initialize CAMEL tools/agent+video_tool = VideoDownloaderToolkit()+agent = ChatAgent()++def process_video(url, question):+ video_path = video_tool.download_video(url)+ # Extract audio+ audio_path = "audio.wav"+ ffmpeg.input(video_path).output(audio_path, acodec='pcm_s16le', ac=1, ar='16k').run()+ # Transcribe+ model = whisper.load_model("base")+ transcript = model.transcribe(audio_path)["text"]+ # Extract frames every 4s+ cap = cv2.VideoCapture(video_path)+ fps = cap.get(cv2.CAP_PROP_FPS); interval = 4+ frames = []+ frame_count = 0+ while True:+ ret, frame = cap.read()+ if not ret: break+ if frame_count % int(fps*interval) == 0:+ fname = f"frame_{frame_count}.png"+ cv2.imwrite(fname, frame)+ frames.append(fname)+ frame_count += 1+ cap.release()+ # OCR on frames+ ocr_texts = []+ for img_path in frames:+ text = pytesseract.image_to_string(Image.open(img_path))+ ocr_texts.append(text)+ ocr_content = "\n".join(ocr_texts)+ # Prepare context and query agent+ knowledge = f"Transcript:\n{transcript}\n\nOn-screen Text:\n{ocr_content}"+ user_msg = BaseMessage.make_user_message(role_name="User",+ content=f"{knowledge}\n\nQuestion: {question}")+ response = agent.step(user_msg)+ return response.msgs[0].content++# Streamlit UI+st.title("YouTube Video Q&A with CAMEL")+youtube_url = st.text_input("YouTube URL:")+user_q = st.text_input("Your Question:")+if st.button("Ask"):+ answer = process_video(youtube_url, user_q)+ st.write(answer)diff --git a/examples/usecases/youtube_ocr/requirements.txt b/examples/usecases/youtube_ocr/requirements.txtnew file mode 100644index 000000000..e1b759b9d--- /dev/null+++ b/examples/usecases/youtube_ocr/requirements.txt@@ -0,0 +1,9 @@+ffmpeg-python==0.2.0+opencv-python==4.9.0.80+streamlit==1.45.0+pytesseract==0.3.13+streamlit==1.45.1+camel-ai==0.2.61+python-dotenv==1.1.0+Pillow==11.2.1+git+https://github.com/openai/whisper.git