Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

feat: add Multilingual Audio Translator using faster-whisper, Orq.ai and Kokoro TTS

Tiioluwani4cf606f641d6
diff --git a/audio/multilingual_audio_translator/.env.example b/audio/multilingual_audio_translator/.env.example
new file mode 100644
index 0000000..1ccb63e
--- /dev/null
+++ b/audio/multilingual_audio_translator/.env.example
@@ -0,0 +1,5 @@
+ORQ_API_KEY=your-orq-api-key-here
+
+# Optional overrides
+# TRANSLATION_MODEL=gemini-3-flash-preview
+# WHISPER_MODEL=large-v3
diff --git a/audio/multilingual_audio_translator/README.md b/audio/multilingual_audio_translator/README.md
new file mode 100644
index 0000000..780d563
--- /dev/null
+++ b/audio/multilingual_audio_translator/README.md
@@ -0,0 +1,174 @@
+# Multilingual Audio Translator
+
+> Transcribe speech in any language, translate it with an LLM, and hear the result spoken back in your chosen target language.
+
+## Overview
+
+Multilingual Audio Translator is a Gradio web application that converts spoken audio into translated speech. Users upload an MP3 or WAV file or record directly in the browser. The app transcribes the audio with faster-whisper, detects the source language automatically, translates the transcript through Orq.ai using Gemini, and synthesizes the translated text locally with Kokoro TTS. The interface returns the original transcript, translated text, detected language metadata, and a playable audio file.
+
+## Demo
+
+![Demo](assets/demo.png)
+
+## Features
+
+- Upload MP3 or WAV files, or record audio in the browser
+- Automatic source language detection with confidence scores via faster-whisper (`large-v3`)
+- LLM translation to 20 target languages through the Orq.ai router
+- Local text-to-speech synthesis with Kokoro-82M (Apache-2.0, no cloud TTS required)
+- Side-by-side output of original transcript, translated text, and synthesized audio
+- Single API key required (`ORQ_API_KEY`) for the translation step only
+- GPU acceleration for Whisper when CUDA is available; CPU fallback supported
+
+## Tech Stack
+
+Frameworks & Libraries:
+
+- [faster-whisper](https://github.com/SYSTRAN/faster-whisper) for speech-to-text and language detection
+- [OpenAI Python SDK](https://github.com/openai/openai-python) for Orq.ai-compatible API calls
+- [Kokoro TTS](https://github.com/hexgrad/kokoro) for local speech synthesis
+- [Gradio](https://www.gradio.app/) for the web interface
+- [python-dotenv](https://github.com/theskumar/python-dotenv) for environment configuration
+- [soundfile](https://github.com/bastibe/python-soundfile) and [NumPy](https://numpy.org/) for audio file handling
+
+Additional Tools:
+
+- STT: faster-whisper (whisper-large-v3)
+- LLM: Orq.ai AI Router (`gemini-3-flash-preview`)
+- TTS: Kokoro TTS
+- Web Framework: Gradio
+
+## Prerequisites
+
+- Python 3.10, 3.11, or 3.12 (required by Kokoro TTS; Python 3.13+ is not yet supported)
+- [espeak-ng](https://github.com/espeak-ng/espeak-ng/releases) installed on your system (required by Kokoro for phoneme generation)
+- API keys for:
+ - Orq.ai (`ORQ_API_KEY`) from the [Orq.ai dashboard](https://orq.ai/)
+- Orq.ai provider setup: connect Google AI (or your Gemini provider) in the AI Router so `gemini-3-flash-preview` can run
+
+## Installation
+
+### 1. Clone the Repository
+
+```bash
+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
+cd Hands-On-AI-Engineering/audio/multilingual_audio_translator
+```
+
+### 2. Create Virtual Environment (Recommended)
+
+```bash
+python -m venv venv
+```
+
+**Windows:**
+
+```bash
+venv\Scripts\activate
+```
+
+**macOS/Linux:**
+
+```bash
+source venv/bin/activate
+```
+
+### 3. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+For Japanese or Mandarin Chinese TTS output, install the optional G2P extras:
+
+```bash
+pip install "misaki[ja]"
+pip install "misaki[zh]"
+```
+
+### 4. Set Up Environment Variables
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env` and set your Orq.ai API key:
+
+```env
+ORQ_API_KEY=your-orq-api-key-here
+```
+
+On Windows Command Prompt, use `copy .env.example .env` instead of `cp`.
+
+Optional overrides:
+
+| Variable | Description |
+|----------|-------------|
+| `TRANSLATION_MODEL` | Orq.ai model ID (default: `gemini-3-flash-preview`) |
+| `WHISPER_MODEL` | Whisper model size (default: `large-v3`) |
+
+## Usage
+
+### Running the Application
+
+```bash
+gradio app.py
+```
+
+You can also run:
+
+```bash
+python app.py
+```
+
+Open the local URL shown in the terminal (typically `http://127.0.0.1:7860`). Upload or record audio, select a target language from the dropdown, and click **Translate**.
+
+### Example Usage
+
+| Input audio | Target language | What the translator returns |
+|-------------|-----------------|----------------------------|
+| Spanish voice memo (MP3) | English | Detected `es` with confidence score, Spanish transcript, English translation, English WAV audio |
+| English podcast clip | French | Detected `en`, English transcript, French translation, French WAV audio |
+| Hindi recording | English | Detected `hi`, Hindi transcript, English translation, English WAV audio |
+| Browser microphone recording | Japanese | Live capture transcribed, translated to Japanese text, Japanese speech via Kokoro TTS |
+| German lecture excerpt | Portuguese | Detected `de`, German transcript, Portuguese translation, Portuguese WAV audio |
+| Mandarin interview clip | English | Detected `zh`, Chinese transcript, English translation, English WAV audio |
+
+## Project Structure
+
+```
+multilingual_audio_translator/
+├── app.py # Gradio UI and pipeline orchestration
+├── transcriber.py # faster-whisper transcription and language detection
+├── translator.py # Orq.ai LLM translation
+├── synthesizer.py # Kokoro TTS speech synthesis
+├── requirements.txt # Python dependencies
+├── .env.example # Environment variable template
+├── .gitignore
+├── README.md
+└── assets/
+ └── demo.png # Demo screenshot for the README
+```
+
+## How It Works
+
+1. **Audio upload**
+ The user uploads an MP3 or WAV file or records audio in the browser. Gradio passes the file path to `process_audio()` in `app.py`.
+
+2. **Transcription**
+ `transcribe()` in `transcriber.py` loads a cached `WhisperModel` (`large-v3`) and runs `model.transcribe()` with voice activity detection enabled. It returns the full transcript, the detected language code, and a confidence score from the transcription metadata.
+
+3. **Translation**
+ `translate()` in `translator.py` sends the transcript to Orq.ai through the OpenAI SDK with `base_url` set to `https://api.orq.ai/v3/router`. The default model is `gemini-3-flash-preview`. A system prompt instructs the model to return only the translated text in the user's chosen target language.
+
+4. **Speech synthesis**
+ `synthesize()` in `synthesizer.py` maps the target language to a Kokoro `lang_code` and voice, runs `KPipeline` locally, concatenates the generated audio chunks, and writes a temporary WAV file at 24 kHz sample rate.
+
+5. **Audio output**
+ Gradio displays the detected source language, the original transcript, the translated text, and an audio player for the synthesized speech. The user can read the translation and play the generated audio immediately.
+
+**Pipeline flow:**
+
+```
+Audio file → faster-whisper (STT + language detection) → Orq.ai LLM (translate) → Kokoro TTS (synthesize) → Translated text + WAV
+```
diff --git a/audio/multilingual_audio_translator/app.py b/audio/multilingual_audio_translator/app.py
new file mode 100644
index 0000000..ca24881
--- /dev/null
+++ b/audio/multilingual_audio_translator/app.py
@@ -0,0 +1,117 @@
+"""Gradio UI for the Multilingual Audio Translator."""
+
+from __future__ import annotations
+
+import gradio as gr
+
+from synthesizer import synthesize
+from transcriber import transcribe
+from translator import LANGUAGE_NAMES, translate
+
+TARGET_LANGUAGES = list(LANGUAGE_NAMES.items())
+
+
+def process_audio(audio_path: str | None, target_language: str):
+ if not audio_path:
+ raise gr.Error("Please upload or record an audio file.")
+
+ if not target_language:
+ raise gr.Error("Please select a target language.")
+
+ transcription = transcribe(audio_path)
+ translated_text = translate(
+ transcription.text,
+ transcription.language,
+ target_language,
+ )
+ output_audio = synthesize(translated_text, target_language)
+
+ source_label = LANGUAGE_NAMES.get(
+ transcription.language,
+ transcription.language,
+ )
+ target_label = LANGUAGE_NAMES.get(target_language, target_language)
+
+ source_info = (
+ f"**Detected language:** {source_label} ({transcription.language}) \n"
+ f"**Confidence:** {transcription.language_probability:.1%}"
+ )
+ status = (
+ f"Translated from {source_label} to {target_label}. "
+ "Play the synthesized audio below."
+ )
+
+ return (
+ source_info,
+ transcription.text,
+ translated_text,
+ output_audio,
+ status,
+ )
+
+
+def build_app() -> gr.Blocks:
+ with gr.Blocks(
+ title="Multilingual Audio Translator",
+ theme=gr.themes.Soft(),
+ ) as demo:
+ gr.Markdown(
+ """
+ # Multilingual Audio Translator
+
+ Upload or record speech in any language. The app transcribes it with
+ **faster-whisper**, translates the text with an LLM via **Orq.ai**, and
+ synthesizes the translation with **Kokoro TTS**.
+ """
+ )
+
+ with gr.Row():
+ with gr.Column():
+ audio_input = gr.Audio(
+ label="Input audio (upload MP3/WAV or record)",
+ sources=["upload", "microphone"],
+ type="filepath",
+ )
+ target_language = gr.Dropdown(
+ label="Target language",
+ choices=TARGET_LANGUAGES,
+ value="en",
+ )
+ translate_btn = gr.Button("Translate", variant="primary")
+
+ with gr.Column():
+ status = gr.Markdown("")
+ source_info = gr.Markdown(label="Source language")
+ original_text = gr.Textbox(
+ label="Original transcript",
+ lines=4,
+ interactive=False,
+ )
+ translated_text = gr.Textbox(
+ label="Translated text",
+ lines=4,
+ interactive=False,
+ )
+ output_audio = gr.Audio(
+ label="Translated speech",
+ type="filepath",
+ interactive=False,
+ )
+
+ translate_btn.click(
+ fn=process_audio,
+ inputs=[audio_input, target_language],
+ outputs=[
+ source_info,
+ original_text,
+ translated_text,
+ output_audio,
+ status,
+ ],
+ )
+
+ return demo
+
+
+if __name__ == "__main__":
+ build_app().launch()
diff --git a/audio/multilingual_audio_translator/assets/demo.png b/audio/multilingual_audio_translator/assets/demo.png
new file mode 100644
index 0000000..d6593f5
Binary files /dev/null and b/audio/multilingual_audio_translator/assets/demo.png differ
diff --git a/audio/multilingual_audio_translator/requirements.txt b/audio/multilingual_audio_translator/requirements.txt
new file mode 100644
index 0000000..f667469
--- /dev/null
+++ b/audio/multilingual_audio_translator/requirements.txt
@@ -0,0 +1,9 @@
+faster-whisper>=1.1.0
+openai>=1.0.0
+gradio>=5.0.0
+python-dotenv>=1.0.0
+kokoro>=0.9.4
+soundfile>=0.12.1
+numpy>=1.24.0
+torch>=2.0.0
+misaki[en]>=0.9.0
diff --git a/audio/multilingual_audio_translator/synthesizer.py b/audio/multilingual_audio_translator/synthesizer.py
new file mode 100644
index 0000000..2d37c72
--- /dev/null
+++ b/audio/multilingual_audio_translator/synthesizer.py
@@ -0,0 +1,63 @@
+"""Text-to-speech synthesis using Kokoro TTS."""
+
+from __future__ import annotations
+
+import tempfile
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+from kokoro import KPipeline
+
+SAMPLE_RATE = 24_000
+
+# Kokoro lang_code and default voice per target language.
+LANG_CONFIG: dict[str, tuple[str, str]] = {
+ "en": ("a", "af_heart"),
+ "es": ("e", "af_heart"),
+ "fr": ("f", "af_heart"),
+ "de": ("a", "af_heart"),
+ "it": ("i", "af_heart"),
+ "pt": ("p", "af_heart"),
+ "hi": ("h", "af_heart"),
+ "ja": ("j", "af_heart"),
+ "zh": ("z", "af_heart"),
+ "ko": ("a", "af_heart"),
+ "ar": ("a", "af_heart"),
+ "ru": ("a", "af_heart"),
+ "nl": ("a", "af_heart"),
+ "pl": ("a", "af_heart"),
+ "tr": ("a", "af_heart"),
+ "vi": ("a", "af_heart"),
+ "th": ("a", "af_heart"),
+ "id": ("a", "af_heart"),
+ "sv": ("a", "af_heart"),
+ "uk": ("a", "af_heart"),
+}
+
+_pipelines: dict[str, KPipeline] = {}
+
+
+def _get_pipeline(lang_code: str) -> KPipeline:
+ if lang_code not in _pipelines:
+ _pipelines[lang_code] = KPipeline(lang_code=lang_code)
+ return _pipelines[lang_code]
+
+
+def synthesize(text: str, target_language: str) -> str:
+ """Synthesize speech from text and return the path to a WAV file."""
+ if not text.strip():
+ raise ValueError("Nothing to synthesize.")
+
+ lang_code, voice = LANG_CONFIG.get(target_language, ("a", "af_heart"))
+ pipeline = _get_pipeline(lang_code)
+
+ chunks = [audio for _, _, audio in pipeline(text, voice=voice)]
+ if not chunks:
+ raise ValueError("Speech synthesis produced no audio.")
+
+ audio = np.concatenate(chunks)
+
+ output = Path(tempfile.mkstemp(suffix=".wav")[1])
+ sf.write(str(output), audio, SAMPLE_RATE)
+ return str(output)
diff --git a/audio/multilingual_audio_translator/transcriber.py b/audio/multilingual_audio_translator/transcriber.py
new file mode 100644
index 0000000..3fccd6f
--- /dev/null
+++ b/audio/multilingual_audio_translator/transcriber.py
@@ -0,0 +1,57 @@
+"""Speech-to-text transcription and language detection using faster-whisper."""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+import torch
+from faster_whisper import WhisperModel
+
+MODEL_SIZE = os.getenv("WHISPER_MODEL", "large-v3")
+
+_model: WhisperModel | None = None
+
+
+def _get_device_config() -> tuple[str, str]:
+ if torch.cuda.is_available():
+ return "cuda", "float16"
+ return "cpu", "int8"
+
+
+def get_model() -> WhisperModel:
+ global _model
+ if _model is None:
+ device, compute_type = _get_device_config()
+ _model = WhisperModel(MODEL_SIZE, device=device, compute_type=compute_type)
+ return _model
+
+
+@dataclass
+class TranscriptionResult:
+ text: str
+ language: str
+ language_probability: float
+
+
+def transcribe(audio_path: str) -> TranscriptionResult:
+ """Transcribe audio and detect the spoken language."""
+ if not audio_path:
+ raise ValueError("No audio file provided.")
+
+ model = get_model()
+ segments, info = model.transcribe(
+ audio_path,
+ beam_size=5,
+ vad_filter=True,
+ )
+
+ text = " ".join(segment.text.strip() for segment in segments).strip()
+ if not text:
+ raise ValueError("No speech detected in the audio.")
+
+ return TranscriptionResult(
+ text=text,
+ language=info.language,
+ language_probability=info.language_probability,
+ )
diff --git a/audio/multilingual_audio_translator/translator.py b/audio/multilingual_audio_translator/translator.py
new file mode 100644
index 0000000..c6763a9
--- /dev/null
+++ b/audio/multilingual_audio_translator/translator.py
@@ -0,0 +1,77 @@
+"""LLM-based translation via Orq.ai."""
+
+from __future__ import annotations
+
+import os
+
+from dotenv import load_dotenv
+from openai import OpenAI
+
+load_dotenv()
+
+ORQ_BASE_URL = "https://api.orq.ai/v3/router"
+DEFAULT_MODEL = os.getenv("TRANSLATION_MODEL", "gemini-3-flash-preview")
+
+LANGUAGE_NAMES = {
+ "en": "English",
+ "es": "Spanish",
+ "fr": "French",
+ "de": "German",
+ "it": "Italian",
+ "pt": "Portuguese",
+ "hi": "Hindi",
+ "ja": "Japanese",
+ "zh": "Chinese",
+ "ko": "Korean",
+ "ar": "Arabic",
+ "ru": "Russian",
+ "nl": "Dutch",
+ "pl": "Polish",
+ "tr": "Turkish",
+ "vi": "Vietnamese",
+ "th": "Thai",
+ "id": "Indonesian",
+ "sv": "Swedish",
+ "uk": "Ukrainian",
+}
+
+
+def _get_client() -> OpenAI:
+ api_key = os.getenv("ORQ_API_KEY")
+ if not api_key:
+ raise ValueError(
+ "ORQ_API_KEY is not set. Copy .env.example to .env and add your key."
+ )
+ return OpenAI(base_url=ORQ_BASE_URL, api_key=api_key)
+
+
+def translate(text: str, source_language: str, target_language: str) -> str:
+ """Translate transcript text to the target language."""
+ if not text.strip():
+ raise ValueError("Nothing to translate.")
+
+ source_name = LANGUAGE_NAMES.get(source_language, source_language)
+ target_name = LANGUAGE_NAMES.get(target_language, target_language)
+
+ client = _get_client()
+ response = client.chat.completions.create(
+ model=DEFAULT_MODEL,
+ messages=[
+ {
+ "role": "system",
+ "content": (
+ "You are a professional translator. Translate the user's text "
+ f"from {source_name} to {target_name}. Preserve meaning and tone. "
+ "Return only the translated text with no commentary."
+ ),
+ },
+ {"role": "user", "content": text},
+ ],
+ temperature=0.3,
+ )
+
+ translated = response.choices[0].message.content
+ if not translated or not translated.strip():
+ raise ValueError("Translation model returned an empty response.")
+
+ return translated.strip()