Skip to content

World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.

feat(stt): add Azure AI Speech as an optional cloud speech-to-text provider

amartya-deva2a0d8c8af0c
diff --git a/.agents/skills/azure-speech-to-text/SKILL.md b/.agents/skills/azure-speech-to-text/SKILL.md
new file mode 100644
index 00000000..60b1b8a4
--- /dev/null
+++ b/.agents/skills/azure-speech-to-text/SKILL.md
@@ -0,0 +1,115 @@
+---
+name: azure-speech-to-text
+description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.
+license: MIT
+compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
+metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
+---
+
+# Azure AI Speech — Speech-to-Text
+
+Transcribe audio to text with **Azure Fast Transcription** — synchronous,
+word-level timestamps, speaker diarization, and multi-language identification.
+In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,
+`provider=azure`). It is an **optional cloud STT provider** — when
+`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local
+`transcriber` tool (faster-whisper) remains the **default offline path** and the
+fallback when Azure is unavailable.
+
+> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)
+
+## Why Fast Transcription (not Batch)
+
+Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because
+the pipeline transcribes **local audio files**:
+
+| Surface | Input | Latency | Needs |
+|---------|-------|---------|-------|
+| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |
+| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |
+| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |
+
+Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just
+`requests` and the two env vars.
+
+## Setup
+
+Create a **Speech** resource in the [Azure portal](https://portal.azure.com);
+copy the key and region from its **Keys and Endpoint** page.
+
+```bash
+export AZURE_SPEECH_KEY=your_speech_resource_key
+export AZURE_SPEECH_REGION=eastus # your resource's region
+# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region
+```
+
+`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
+`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.
+
+## Using it in a pipeline
+
+Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output
+matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`
+and any stage that consumes a transcript.
+
+```python
+from tools.tool_registry import registry
+registry.discover()
+stt = registry._tools["azure_stt"]
+
+result = stt.execute({
+ "input_path": "projects/my-video/assets/audio/narration.mp3",
+ # "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID
+ # "diarize": True, # speaker labels, no HuggingFace token needed
+ # "max_speakers": 4,
+ "output_dir": "projects/my-video/artifacts",
+})
+if result.success:
+ segs = result.data["segments"] # [{id,start,end,text,words:[...]}]
+ words = result.data["word_timestamps"] # flat [{word,start,end,probability}]
+```
+
+If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`
+(local whisper) — its `execute` signature and output are identical.
+
+## Parameters that matter
+
+- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it
+ when you know the language; it is faster and more accurate than auto-ID.
+- **`candidate_locales`** — when `language` is omitted, Azure runs language
+ identification across this shortlist. Narrow it to the languages you actually
+ expect; a huge list slows detection and invites misclassification.
+- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,
+ podcasts). Set `max_speakers` to the real upper bound.
+- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.
+
+## Response shape (mapped to the transcriber schema)
+
+The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is
+converted to seconds and the OpenMontage transcript schema:
+
+```json
+{
+ "segments": [
+ {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",
+ "speaker": 1,
+ "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}
+ ],
+ "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],
+ "language": "en-US",
+ "duration_seconds": 2.4,
+ "provider": "azure"
+}
+```
+
+Note: Fast Transcription has no *per-word* confidence, so each word carries the
+**phrase** confidence in `probability`.
+
+## Limits & tips
+
+- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk
+ jobs, use Azure Batch Transcription instead.
+- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if
+ you only need speech — smaller upload, same result.
+- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check
+ the first and last cues against the source audio.
diff --git a/.claude/skills/azure-speech-to-text/SKILL.md b/.claude/skills/azure-speech-to-text/SKILL.md
new file mode 100644
index 00000000..60b1b8a4
--- /dev/null
+++ b/.claude/skills/azure-speech-to-text/SKILL.md
@@ -0,0 +1,115 @@
+---
+name: azure-speech-to-text
+description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.
+license: MIT
+compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).
+metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}
+---
+
+# Azure AI Speech — Speech-to-Text
+
+Transcribe audio to text with **Azure Fast Transcription** — synchronous,
+word-level timestamps, speaker diarization, and multi-language identification.
+In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,
+`provider=azure`). It is an **optional cloud STT provider** — when
+`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local
+`transcriber` tool (faster-whisper) remains the **default offline path** and the
+fallback when Azure is unavailable.
+
+> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)
+
+## Why Fast Transcription (not Batch)
+
+Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because
+the pipeline transcribes **local audio files**:
+
+| Surface | Input | Latency | Needs |
+|---------|-------|---------|-------|
+| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |
+| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |
+| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |
+
+Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just
+`requests` and the two env vars.
+
+## Setup
+
+Create a **Speech** resource in the [Azure portal](https://portal.azure.com);
+copy the key and region from its **Keys and Endpoint** page.
+
+```bash
+export AZURE_SPEECH_KEY=your_speech_resource_key
+export AZURE_SPEECH_REGION=eastus # your resource's region
+# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region
+```
+
+`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either
+`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.
+
+## Using it in a pipeline
+
+Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output
+matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`
+and any stage that consumes a transcript.
+
+```python
+from tools.tool_registry import registry
+registry.discover()
+stt = registry._tools["azure_stt"]
+
+result = stt.execute({
+ "input_path": "projects/my-video/assets/audio/narration.mp3",
+ # "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID
+ # "diarize": True, # speaker labels, no HuggingFace token needed
+ # "max_speakers": 4,
+ "output_dir": "projects/my-video/artifacts",
+})
+if result.success:
+ segs = result.data["segments"] # [{id,start,end,text,words:[...]}]
+ words = result.data["word_timestamps"] # flat [{word,start,end,probability}]
+```
+
+If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`
+(local whisper) — its `execute` signature and output are identical.
+
+## Parameters that matter
+
+- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it
+ when you know the language; it is faster and more accurate than auto-ID.
+- **`candidate_locales`** — when `language` is omitted, Azure runs language
+ identification across this shortlist. Narrow it to the languages you actually
+ expect; a huge list slows detection and invites misclassification.
+- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,
+ podcasts). Set `max_speakers` to the real upper bound.
+- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.
+
+## Response shape (mapped to the transcriber schema)
+
+The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is
+converted to seconds and the OpenMontage transcript schema:
+
+```json
+{
+ "segments": [
+ {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",
+ "speaker": 1,
+ "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}
+ ],
+ "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],
+ "language": "en-US",
+ "duration_seconds": 2.4,
+ "provider": "azure"
+}
+```
+
+Note: Fast Transcription has no *per-word* confidence, so each word carries the
+**phrase** confidence in `probability`.
+
+## Limits & tips
+
+- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk
+ jobs, use Azure Batch Transcription instead.
+- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if
+ you only need speech — smaller upload, same result.
+- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check
+ the first and last cues against the source audio.
diff --git a/.env.example b/.env.example
index ac6f0708..14f2a351 100644
--- a/.env.example
+++ b/.env.example
@@ -56,6 +56,12 @@ UNSPLASH_ACCESS_KEY= # Unsplash stock images (free developer key)
# --- Analysis ---
HF_TOKEN= # HuggingFace token — enables speaker diarization in transcriber
+# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the
+# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber
+# remains the default offline path.
+AZURE_SPEECH_KEY= # Azure AI Speech resource key ('Keys and Endpoint' page)
+AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
+# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region)
# --- Avatar (local installs) ---
# WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync)
diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md
index feb67699..2a0b6bd5 100644
--- a/AGENT_GUIDE.md
+++ b/AGENT_GUIDE.md
@@ -678,7 +678,8 @@ The `.agents/skills/` directory is large. When you're not coming in through a to
| **Image generation** | `bfl-api`, `flux-best-practices` |
| **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` |
| **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |
-| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `speech-to-text`, `agents` |
+| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |
+| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` |
| **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) |
| **Visualization** | `beautiful-mermaid`, `d3-viz`, `manim-composer`, `manimce-best-practices`, `manimgl-best-practices` |
| **Media editing** | `video-edit`, `video-download`, `video-understand`, `video-toolkit`, `visual-style` |
diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md
index 8cee5bb2..be77ba3f 100644
--- a/docs/PROVIDERS.md
+++ b/docs/PROVIDERS.md
@@ -43,6 +43,10 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin nar
DOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice type
DASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)
+# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)
+AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)
+AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus
+
# MULTI-MODEL GATEWAY (one key, 6+ tools)
FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video
@@ -245,6 +249,54 @@ Doubao Speech 2.0 is billed by character package or usage in Volcengine. OpenMon
---
+### Azure AI Speech — Speech-to-Text
+
+> **Cloud transcription.** Azure AI Speech Fast Transcription turns local audio into text with word-level timestamps, speaker diarization, and multi-language identification — no GPU required. Optional: the local faster-whisper `transcriber` remains the default offline STT path. When `AZURE_SPEECH_KEY` is set, the agent prefers `azure_stt` for cloud transcription.
+
+**Tools unlocked:** `azure_stt`
+**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_SPEECH_ENDPOINT`)
+
+#### Setup
+
+1. In the [Azure portal](https://portal.azure.com), create a **Speech** resource (Azure AI services → Speech service).
+2. Open the resource's **Keys and Endpoint** page.
+3. Copy **KEY 1** and the **Location/Region** (e.g. `eastus`).
+4. Add to `.env`:
+ ```bash
+ AZURE_SPEECH_KEY=your-speech-resource-key
+ AZURE_SPEECH_REGION=eastus
+ # AZURE_SPEECH_ENDPOINT=https://<custom>... # optional, overrides region
+ ```
+
+#### API Notes
+
+OpenMontage uses the **Fast Transcription** REST endpoint, which accepts a local
+audio file directly (multipart upload) and returns a synchronous result — no
+Azure Blob storage, SAS URLs, or async job polling:
+
+```text
+POST https://{region}.api.cognitive.microsoft.com/speechtotext/transcriptions:transcribe?api-version=2024-11-15
+Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}
+```
+
+For files longer than ~2 hours or bulk jobs, use Azure Batch Transcription instead (not wired into OpenMontage).
+
+#### What It Is Best For
+
+- Cloud transcription with word-level timestamps and no local GPU
+- Multi-language auto-detection across a candidate locale set
+- Speaker diarization without a HuggingFace token
+- Subtitle timing metadata that flows straight into `subtitle_gen`
+
+#### Pricing
+
+Azure AI Speech Standard (S0) bills speech-to-text by audio-hour (roughly
+$1.00/audio-hour at time of writing; a free F0 tier includes a limited monthly
+allowance). OpenMontage estimates cost from the transcribed audio duration. See
+[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.
+
+---
+
### Google — TTS + Imagen + Music + Video (Shared Key)
> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.
diff --git a/skills/INDEX.md b/skills/INDEX.md
index c8a05090..dc81f0e6 100644
--- a/skills/INDEX.md
+++ b/skills/INDEX.md
@@ -81,7 +81,8 @@ Key capability families to look for in the output:
| FFmpeg | `core/ffmpeg.md` | Video encoding, filtering, composition | `ffmpeg`, `video-toolkit` |
| Remotion | `core/remotion.md` | React-based composition, Phase 3+ | `remotion-best-practices`, `remotion` |
| HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |
-| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps | `speech-to-text` |
+| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |
+| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` |
| Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` |
| Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |
@@ -306,7 +307,7 @@ Claude Code accesses them via symlinks in `.claude/skills/`.
|----------|-----------------|--------|
| **Video Composition** | `remotion-best-practices`, `remotion`, `hyperframes` (router), `hyperframes-core`, `hyperframes-creative`, `hyperframes-media`, `hyperframes-animation`, `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video`, `remotion-to-hyperframes`, `website-to-video` | `remotion-dev/skills`, `digitalsamba/claude-code-video-toolkit`, `heygen-com/hyperframes` (vendored v0.7.17, see `.agents/skills/hyperframes/PROVENANCE.md`) |
| **Video Processing** | `ffmpeg`, `video-toolkit` | `digitalsamba/claude-code-video-toolkit` |
-| **TTS & Audio** | `text-to-speech`, `speech-to-text`, `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |
+| **TTS & Audio** | `text-to-speech`, `speech-to-text` (whisper, default STT), `azure-speech-to-text` (optional cloud STT), `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |
| **Image Generation** | `flux-best-practices`, `bfl-api`, `grok-media` | `black-forest-labs/skills`, local OpenMontage skill |
| **Math Animation** | `manimce-best-practices`, `manimgl-best-practices`, `manim-composer` | `adithya-s-k/manim_skill` |
| **3D Graphics** | `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | `cloudai-x/threejs-skills` |
diff --git a/tests/tools/test_azure_stt.py b/tests/tools/test_azure_stt.py
new file mode 100644
index 00000000..49c0cb64
--- /dev/null
+++ b/tests/tools/test_azure_stt.py
@@ -0,0 +1,242 @@
+"""Focused tests for the Azure AI Speech (Fast Transcription) STT tool.
+
+No live API calls: the network layer is monkeypatched. Covers the tool
+contract, registry discovery, status behavior, the response→transcript
+mapping, execute() guardrails, and downstream subtitle compatibility.
+"""
+
+import sys
+from pathlib import Path
+
+import pytest
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime
+from tools.tool_registry import ToolRegistry
+from tools.analysis.azure_stt import AzureSpeechToText
+from tools.subtitle.subtitle_gen import SubtitleGen
+
+
+# A representative Fast Transcription response payload.
+SAMPLE_PAYLOAD = {
+ "durationMilliseconds": 2400,
+ "combinedPhrases": [{"text": "Hello world this is a test"}],
+ "phrases": [
+ {
+ "offsetMilliseconds": 0,
+ "durationMilliseconds": 1200,
+ "text": "Hello world",
+ "locale": "en-US",
+ "confidence": 0.97,
+ "speaker": 1,
+ "words": [
+ {"text": "Hello", "offsetMilliseconds": 0, "durationMilliseconds": 500},
+ {"text": "world", "offsetMilliseconds": 500, "durationMilliseconds": 700},
+ ],
+ },
+ {
+ "offsetMilliseconds": 1200,
+ "durationMilliseconds": 1200,
+ "text": "this is a test",
+ "locale": "en-US",
+ "confidence": 0.95,
+ "words": [
+ {"text": "this", "offsetMilliseconds": 1200, "durationMilliseconds": 300},
+ {"text": "is", "offsetMilliseconds": 1500, "durationMilliseconds": 200},
+ {"text": "a", "offsetMilliseconds": 1700, "durationMilliseconds": 200},
+ {"text": "test", "offsetMilliseconds": 1900, "durationMilliseconds": 500},
+ ],
+ },
+ ],
+}
+
+
+class _FakeResponse:
+ def __init__(self, payload, status_code=200, text="ok"):
+ self._payload = payload
+ self.status_code = status_code
+ self.text = text
+
+ def json(self):
+ return self._payload
+
+
+def azure_env(monkeypatch):
+ monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
+ monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus")
+
+
+# ---- Contract ----
+
+class TestContract:
+ def test_inherits_base_tool(self):
+ assert issubclass(AzureSpeechToText, BaseTool)
+
+ def test_identity(self):
+ t = AzureSpeechToText()
+ assert t.name == "azure_stt"
+ assert t.capability == "analysis"
+ assert t.provider == "azure"
+ assert t.runtime == ToolRuntime.API
+ assert t.tier == ToolTier.CORE
+ assert t.fallback == "transcriber"
+ assert "azure-speech-to-text" in t.agent_skills
+ assert len(t.capabilities) > 0
+
+ def test_get_info_valid(self):
+ info = AzureSpeechToText().get_info()
+ assert info["name"] == "azure_stt"
+ assert info["capability"] == "analysis"
+ assert "input_path" in info["input_schema"]["properties"]
+
+ def test_estimate_cost_by_duration(self):
+ t = AzureSpeechToText()
+ assert t.estimate_cost({"duration_seconds": 3600}) == pytest.approx(1.0)
+ assert t.estimate_cost({}) == 0.0
+
+
+# ---- Registry discovery ----
+
+class TestDiscovery:
+ def test_discoverable(self):
+ reg = ToolRegistry()
+ reg.discover("tools")
+ assert reg.get("azure_stt") is not None
+
+ def test_capability_routing(self):
+ reg = ToolRegistry()
+ reg.discover("tools")
+ names = [t.name for t in reg.get_by_capability("analysis")]
+ assert "azure_stt" in names
+
+
+# ---- Status behavior ----
+
+class TestStatus:
+ def test_unavailable_without_env(self, monkeypatch):
+ monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
+ monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False)
+ assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE
+
+ def test_available_with_key_and_region(self, azure_env):
+ assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE
+
+ def test_available_with_key_and_endpoint(self, monkeypatch):
+ monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
+ monkeypatch.setenv("AZURE_SPEECH_ENDPOINT", "https://custom.example.com")
+ assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE
+
+ def test_key_alone_is_not_enough(self, monkeypatch):
+ monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")
+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
+ monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False)
+ assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE
+
+
+# ---- Response → transcript mapping (the risky logic) ----
+
+class TestParsePayload:
+ def test_maps_segments_words_and_timestamps(self):
+ data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"])
+ assert data["language"] == "en-US"
+ assert data["duration_seconds"] == 2.4
+ assert len(data["segments"]) == 2
+
+ seg0 = data["segments"][0]
+ assert seg0["text"] == "Hello world"
+ assert seg0["start"] == 0.0 and seg0["end"] == 1.2
+ assert seg0["speaker"] == 1
+ # ms → seconds conversion on words
+ assert seg0["words"][0] == {
+ "word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.97,
+ }
+ # flat word stream aggregates every phrase
+ assert len(data["word_timestamps"]) == 6
+
+ def test_locale_resolution(self):
+ r = AzureSpeechToText._resolve_locales
+ assert r({"language": "en"}) == ["en-US"] # ISO → default locale
+ assert r({"language": "en-GB"}) == ["en-GB"] # explicit locale kept
+ assert r({"candidate_locales": ["fr-FR"]}) == ["fr-FR"]
+ assert len(r({})) > 1 # falls back to a shortlist
+
+ def test_duration_falls_back_to_last_segment(self):
+ payload = {k: v for k, v in SAMPLE_PAYLOAD.items() if k != "durationMilliseconds"}
+ data = AzureSpeechToText._parse_payload(payload, ["en-US"])
+ assert data["duration_seconds"] == pytest.approx(2.4)
+
+ def test_output_feeds_subtitle_gen(self, tmp_path):
+ """Parsed segments must be a drop-in for the subtitle generator."""
+ data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"])
+ out = tmp_path / "captions.srt"
+ res = SubtitleGen().execute(
+ {"segments": data["segments"], "format": "srt", "output_path": str(out)}
+ )
+ assert res.success
+ assert out.exists()
+ assert "Hello world" in out.read_text()
+
+
+# ---- execute() guardrails + mocked success ----
+
+class TestExecute:
+ def test_missing_file(self, azure_env, tmp_path):
+ res = AzureSpeechToText().execute({"input_path": str(tmp_path / "nope.wav")})
+ assert not res.success
+ assert "not found" in res.error.lower()
+
+ def test_missing_credentials(self, monkeypatch, tmp_path):
+ monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)
+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)
+ audio = tmp_path / "a.wav"
+ audio.write_bytes(b"RIFF....")
+ res = AzureSpeechToText().execute({"input_path": str(audio)})
+ assert not res.success
+ assert "not configured" in res.error.lower()
+
+ def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch):
+ import requests
+
+ captured = {}
+
+ def fake_post(url, headers=None, files=None, timeout=None):
+ captured["url"] = url
+ captured["headers"] = headers
+ return _FakeResponse(SAMPLE_PAYLOAD)
+
+ monkeypatch.setattr(requests, "post", fake_post)
+
+ audio = tmp_path / "a.wav"
+ audio.write_bytes(b"RIFF....")
+ res = AzureSpeechToText().execute(
+ {"input_path": str(audio), "language": "en", "output_dir": str(tmp_path)}
+ )
+
+ assert res.success
+ assert res.model == "azure-fast-transcription"
+ assert res.cost_usd == pytest.approx(0.0007, abs=1e-4) # 2.4s at $1/hr
+ assert res.data["provider"] == "azure"
+ assert len(res.data["segments"]) == 2
+ # transcript JSON artifact was written
+ assert res.artifacts and Path(res.artifacts[0]).exists()
+ # correct endpoint + auth header used
+ assert "transcriptions:transcribe" in captured["url"]
+ assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key"
+
+ def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch):
+ import requests
+
+ monkeypatch.setattr(
+ requests, "post",
+ lambda *a, **k: _FakeResponse(None, status_code=401, text="Unauthorized"),
+ )
+ audio = tmp_path / "a.wav"
+ audio.write_bytes(b"RIFF....")
+ res = AzureSpeechToText().execute({"input_path": str(audio), "output_dir": str(tmp_path)})
+ assert not res.success
+ assert "401" in res.error
diff --git a/tools/analysis/azure_stt.py b/tools/analysis/azure_stt.py
new file mode 100644
index 00000000..38fd9a93
--- /dev/null
+++ b/tools/analysis/azure_stt.py
@@ -0,0 +1,365 @@
+"""Azure AI Speech transcription tool (Fast Transcription REST API).
+
+Speech-to-text with word-level timestamps and optional speaker diarization,
+served by Azure AI Speech. This is an optional cloud STT provider; when
+`AZURE_SPEECH_KEY` is configured the agent may prefer it for cloud
+transcription, while the local faster-whisper `transcriber` tool remains the
+default offline path.
+
+Uses the **Fast Transcription** REST endpoint, which accepts a local audio
+file directly (multipart upload) and returns a synchronous JSON result — no
+Azure Blob storage, SAS URLs, or async job polling required. Output is shaped
+to match `transcriber` exactly (segments + word_timestamps) so it is a
+drop-in for `subtitle_gen` and every pipeline that consumes a transcript.
+
+Docs: https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+from pathlib import Path
+from typing import Any
+
+from tools.base_tool import (
+ BaseTool,
+ Determinism,
+ ExecutionMode,
+ ResourceProfile,
+ ResumeSupport,
+ RetryPolicy,
+ ToolResult,
+ ToolRuntime,
+ ToolStability,
+ ToolStatus,
+ ToolTier,
+)
+
+# Fast Transcription API version (GA).
+_API_VERSION = "2024-11-15"
+
+# Candidate locales used for automatic language identification when the caller
+# does not pin a language. Azure Fast Transcription accepts several locales and
+# picks the best match per phrase.
+_DEFAULT_CANDIDATE_LOCALES = [
+ "en-US",
+ "es-ES",
+ "fr-FR",
+ "de-DE",
+ "it-IT",
+ "pt-BR",
+ "hi-IN",
+ "ja-JP",
+ "zh-CN",
+]
+
+# Minimal ISO 639-1 -> default BCP-47 locale map so callers can keep passing the
+# short codes the whisper transcriber accepts (e.g. "en", "es").
+_ISO_TO_LOCALE = {
+ "en": "en-US",
+ "es": "es-ES",
+ "fr": "fr-FR",
+ "de": "de-DE",
+ "it": "it-IT",
+ "pt": "pt-BR",
+ "hi": "hi-IN",
+ "ja": "ja-JP",
+ "ko": "ko-KR",
+ "zh": "zh-CN",
+ "ar": "ar-EG",
+ "ru": "ru-RU",
+ "nl": "nl-NL",
+ "tr": "tr-TR",
+ "pl": "pl-PL",
+ "id": "id-ID",
+}
+
+
+class AzureSpeechToText(BaseTool):
+ name = "azure_stt"
+ version = "0.1.0"
+ tier = ToolTier.CORE
+ capability = "analysis"
+ provider = "azure"
+ stability = ToolStability.BETA
+ execution_mode = ExecutionMode.SYNC
+ determinism = Determinism.DETERMINISTIC
+ runtime = ToolRuntime.API
+
+ # Availability is decided by get_status() (env var check), mirroring the
+ # ElevenLabs provider tool — dependencies stays empty.
+ dependencies = []
+ install_instructions = (
+ "Set your Azure AI Speech credentials:\n"
+ " export AZURE_SPEECH_KEY=your_speech_resource_key\n"
+ " export AZURE_SPEECH_REGION=eastus # your Speech resource region\n"
+ "Create a Speech resource in the Azure portal "
+ "(https://portal.azure.com) — the key and region are on its "
+ "'Keys and Endpoint' page. Optionally set AZURE_SPEECH_ENDPOINT to a "
+ "full custom endpoint URL instead of the region."
+ )
+ agent_skills = ["azure-speech-to-text"]
+
+ capabilities = [
+ "transcribe",
+ "word_timestamps",
+ "diarization",
+ "language_detection",
+ ]
+
+ best_for = [
+ "Cloud transcription with word-level timestamps and no GPU",
+ "Multi-language auto-detection across a candidate locale set",
+ "Speaker diarization without a HuggingFace token",
+ ]
+ not_good_for = [
+ "Fully offline runs (use the whisper `transcriber` fallback)",
+ "Single files longer than ~2 hours (use Azure Batch transcription)",
+ ]
+
+ input_schema = {
+ "type": "object",
+ "required": ["input_path"],
+ "properties": {
+ "input_path": {"type": "string", "description": "Path to audio or video file"},
+ "language": {
+ "type": "string",
+ "description": (
+ "ISO 639-1 code (e.g. 'en') or BCP-47 locale (e.g. 'en-US'). "
+ "Omit for automatic language identification across common locales."
+ ),
+ },
+ "candidate_locales": {
+ "type": "array",
+ "items": {"type": "string"},
+ "description": (
+ "BCP-47 locales to consider for language ID when `language` is "
+ "not set. Defaults to a common-language shortlist."
+ ),
+ },
+ "diarize": {"type": "boolean", "default": False},
+ "max_speakers": {"type": "integer", "default": 4},
+ "profanity_filter": {
+ "type": "string",
+ "enum": ["None", "Masked", "Removed", "Tags"],
+ "default": "Masked",
+ },
+ "output_dir": {"type": "string", "description": "Directory for output files"},
+ },
+ }
+
+ output_schema = {
+ "type": "object",
+ "properties": {
+ "segments": {"type": "array"},
+ "word_timestamps": {"type": "array"},
+ "language": {"type": "string"},
+ "duration_seconds": {"type": "number"},
+ },
+ }
+
+ resource_profile = ResourceProfile(
+ cpu_cores=1,
+ ram_mb=256,
+ vram_mb=0,
+ disk_mb=50,
+ network_required=True,
+ )
+
+ retry_policy = RetryPolicy(
+ max_retries=2,
+ retryable_errors=["ConnectionError", "Timeout", "429", "503"],
+ )
+ resume_support = ResumeSupport.FROM_START
+ idempotency_key_fields = ["input_path", "language", "diarize"]
+ side_effects = ["writes transcript JSON to output_dir", "sends audio to Azure AI Speech"]
+ fallback = "transcriber"
+ fallback_tools = ["transcriber"]
+ user_visible_verification = [
+ "Check transcript text against source audio",
+ "Verify word timestamps align with speech",
+ ]
+
+ def get_status(self) -> ToolStatus:
+ if os.environ.get("AZURE_SPEECH_KEY") and (
+ os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT")
+ ):
+ return ToolStatus.AVAILABLE
+ return ToolStatus.UNAVAILABLE
+
+ # Azure AI Speech Fast Transcription bills per audio-hour (Standard tier is
+ # roughly $1.00/audio-hour at time of writing).
+ COST_PER_AUDIO_HOUR = 1.0
+
+ def estimate_cost(self, inputs: dict[str, Any]) -> float:
+ # Duration is unknown until the audio is transcribed; return the actual
+ # cost by passing duration_seconds, else 0.0 as a pre-call estimate.
+ seconds = inputs.get("duration_seconds", 0) or 0
+ return round((seconds / 3600.0) * self.COST_PER_AUDIO_HOUR, 4)
+
+ def estimate_runtime(self, inputs: dict[str, Any]) -> float:
+ # Fast Transcription is well under real-time; conservative default.
+ return 20.0
+
+ def _endpoint(self) -> str:
+ endpoint = os.environ.get("AZURE_SPEECH_ENDPOINT")
+ if endpoint:
+ return endpoint.rstrip("/")
+ region = os.environ.get("AZURE_SPEECH_REGION", "").strip()
+ return f"https://{region}.api.cognitive.microsoft.com"
+
+ @staticmethod
+ def _resolve_locales(inputs: dict[str, Any]) -> list[str]:
+ language = inputs.get("language")
+ if language:
+ if "-" in language: # already a BCP-47 locale
+ return [language]
+ return [_ISO_TO_LOCALE.get(language.lower(), f"{language.lower()}-US")]
+ candidates = inputs.get("candidate_locales")
+ return list(candidates) if candidates else list(_DEFAULT_CANDIDATE_LOCALES)
+
+ def execute(self, inputs: dict[str, Any]) -> ToolResult:
+ input_path = Path(inputs["input_path"])
+ if not input_path.exists():
+ return ToolResult(success=False, error=f"Input file not found: {input_path}")
+
+ api_key = os.environ.get("AZURE_SPEECH_KEY")
+ if not api_key or not (
+ os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT")
+ ):
+ return ToolResult(
+ success=False,
+ error="Azure Speech is not configured. " + self.install_instructions,
+ )
+
+ start = time.time()
+ try:
+ result = self._transcribe(inputs, api_key, input_path)
+ except Exception as exc:
+ return ToolResult(success=False, error=f"Transcription failed: {exc}")
+
+ result.duration_seconds = round(time.time() - start, 2)
+ result.model = "azure-fast-transcription"
+ result.cost_usd = self.estimate_cost(
+ {"duration_seconds": result.data.get("duration_seconds", 0)}
+ )
+ return result
+
+ def _transcribe(
+ self, inputs: dict[str, Any], api_key: str, input_path: Path
+ ) -> ToolResult:
+ import requests
+
+ output_dir = Path(inputs.get("output_dir", input_path.parent))
+ diarize = inputs.get("diarize", False)
+ output_dir.mkdir(parents=True, exist_ok=True)
+
+ locales = self._resolve_locales(inputs)
+ definition: dict[str, Any] = {
+ "locales": locales,
+ "profanityFilterMode": inputs.get("profanity_filter", "Masked"),
+ }
+ if diarize:
+ definition["diarization"] = {
+ "maxSpeakers": inputs.get("max_speakers", 4),
+ "enabled": True,
+ }
+
+ url = f"{self._endpoint()}/speechtotext/transcriptions:transcribe?api-version={_API_VERSION}"
+ headers = {"Ocp-Apim-Subscription-Key": api_key}
+
+ try:
+ with open(input_path, "rb") as audio_file:
+ response = requests.post(
+ url,
+ headers=headers,
+ files={
+ "audio": (input_path.name, audio_file, "application/octet-stream"),
+ "definition": (None, json.dumps(definition), "application/json"),
+ },
+ timeout=600,
+ )
+ except requests.RequestException as exc: # network/timeout
+ return ToolResult(success=False, error=f"Azure Speech request failed: {exc}")
+
+ if response.status_code != 200:
+ detail = response.text[:500] if response.text else ""
+ return ToolResult(
+ success=False,
+ error=f"Azure Speech returned HTTP {response.status_code}: {detail}",
+ )
+
+ try:
+ payload = response.json()
+ except ValueError:
+ return ToolResult(
+ success=False,
+ error="Azure Speech returned a non-JSON response.",
+ )
+
+ result_data = self._parse_payload(payload, locales)
+ result_data["provider"] = "azure"
+ output_path = output_dir / f"{input_path.stem}_transcript.json"
+ output_path.write_text(json.dumps(result_data, indent=2), encoding="utf-8")
+
+ return ToolResult(
+ success=True,
+ data=result_data,
+ artifacts=[str(output_path)],
+ )
+
+ @staticmethod
+ def _parse_payload(payload: dict[str, Any], requested_locales: list[str]) -> dict[str, Any]:
+ """Map the Fast Transcription response onto the transcriber output schema."""
+ segments: list[dict[str, Any]] = []
+ word_timestamps: list[dict[str, Any]] = []
+ detected_locale: str | None = None
+
+ for idx, phrase in enumerate(payload.get("phrases", [])):
+ offset_ms = phrase.get("offsetMilliseconds", 0)
+ dur_ms = phrase.get("durationMilliseconds", 0)
+ confidence = phrase.get("confidence")
+ if detected_locale is None and phrase.get("locale"):
+ detected_locale = phrase["locale"]
+
+ seg: dict[str, Any] = {
+ "id": idx,
+ "start": round(offset_ms / 1000.0, 3),
+ "end": round((offset_ms + dur_ms) / 1000.0, 3),
+ "text": (phrase.get("text") or "").strip(),
+ }
+ if "speaker" in phrase:
+ seg["speaker"] = phrase["speaker"]
+
+ words = phrase.get("words") or []
+ if words:
+ seg_words = []
+ for w in words:
+ w_offset = w.get("offsetMilliseconds", 0)
+ w_dur = w.get("durationMilliseconds", 0)
+ word_entry = {
+ "word": w.get("text", ""),
+ "start": round(w_offset / 1000.0, 3),
+ "end": round((w_offset + w_dur) / 1000.0, 3),
+ # Fast Transcription has no per-word confidence; carry the
+ # phrase confidence so downstream schemas stay populated.
+ "probability": round(confidence, 3) if confidence is not None else None,
+ }
+ seg_words.append(word_entry)
+ word_timestamps.append(word_entry)
+ seg["words"] = seg_words
+
+ segments.append(seg)
+
+ duration_ms = payload.get("durationMilliseconds")
+ if duration_ms is None and segments:
+ duration_ms = max(int(s["end"] * 1000) for s in segments)
+
+ return {
+ "segments": segments,
+ "word_timestamps": word_timestamps,
+ "language": detected_locale or (requested_locales[0] if requested_locales else None),
+ "duration_seconds": round((duration_ms or 0) / 1000.0, 3),
+ }