World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
Python agentagentic-aiaiclaudecopilotcursorelevenlabsffmpegfluximage-generationopen-sourceopenaipythonremotionstable-diffusiontext-to-speechtext-to-videovideo-generationvideo-production
feat(stt): add Azure AI Speech as an optional cloud speech-to-text provider
amartya-deva2a0d8c8af0c
diff --git a/.agents/skills/azure-speech-to-text/SKILL.md b/.agents/skills/azure-speech-to-text/SKILL.mdnew file mode 100644index 00000000..60b1b8a4--- /dev/null+++ b/.agents/skills/azure-speech-to-text/SKILL.md@@ -0,0 +1,115 @@+---+name: azure-speech-to-text+description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.+license: MIT+compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).+metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}+---++# Azure AI Speech — Speech-to-Text++Transcribe audio to text with **Azure Fast Transcription** — synchronous,+word-level timestamps, speaker diarization, and multi-language identification.+In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,+`provider=azure`). It is an **optional cloud STT provider** — when+`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local+`transcriber` tool (faster-whisper) remains the **default offline path** and the+fallback when Azure is unavailable.++> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)++## Why Fast Transcription (not Batch)++Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because+the pipeline transcribes **local audio files**:++| Surface | Input | Latency | Needs |+|---------|-------|---------|-------|+| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |+| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |+| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |++Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just+`requests` and the two env vars.++## Setup++Create a **Speech** resource in the [Azure portal](https://portal.azure.com);+copy the key and region from its **Keys and Endpoint** page.++```bash+export AZURE_SPEECH_KEY=your_speech_resource_key+export AZURE_SPEECH_REGION=eastus # your resource's region+# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region+```++`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either+`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.++## Using it in a pipeline++Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output+matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`+and any stage that consumes a transcript.++```python+from tools.tool_registry import registry+registry.discover()+stt = registry._tools["azure_stt"]++result = stt.execute({+ "input_path": "projects/my-video/assets/audio/narration.mp3",+ # "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID+ # "diarize": True, # speaker labels, no HuggingFace token needed+ # "max_speakers": 4,+ "output_dir": "projects/my-video/artifacts",+})+if result.success:+ segs = result.data["segments"] # [{id,start,end,text,words:[...]}]+ words = result.data["word_timestamps"] # flat [{word,start,end,probability}]+```++If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`+(local whisper) — its `execute` signature and output are identical.++## Parameters that matter++- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it+ when you know the language; it is faster and more accurate than auto-ID.+- **`candidate_locales`** — when `language` is omitted, Azure runs language+ identification across this shortlist. Narrow it to the languages you actually+ expect; a huge list slows detection and invites misclassification.+- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,+ podcasts). Set `max_speakers` to the real upper bound.+- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.++## Response shape (mapped to the transcriber schema)++The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is+converted to seconds and the OpenMontage transcript schema:++```json+{+ "segments": [+ {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",+ "speaker": 1,+ "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}+ ],+ "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],+ "language": "en-US",+ "duration_seconds": 2.4,+ "provider": "azure"+}+```++Note: Fast Transcription has no *per-word* confidence, so each word carries the+**phrase** confidence in `probability`.++## Limits & tips++- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk+ jobs, use Azure Batch Transcription instead.+- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if+ you only need speech — smaller upload, same result.+- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check+ the first and last cues against the source audio.diff --git a/.claude/skills/azure-speech-to-text/SKILL.md b/.claude/skills/azure-speech-to-text/SKILL.mdnew file mode 100644index 00000000..60b1b8a4--- /dev/null+++ b/.claude/skills/azure-speech-to-text/SKILL.md@@ -0,0 +1,115 @@+---+name: azure-speech-to-text+description: Transcribe audio to text using Azure AI Speech (Fast Transcription REST API). Use when converting audio/video to text, generating subtitles, or processing spoken content in OpenMontage. Optional cloud STT provider — preferred when AZURE_SPEECH_KEY is configured; the local faster-whisper `transcriber` is the default offline path.+license: MIT+compatibility: Requires internet access and an Azure AI Speech resource (AZURE_SPEECH_KEY + AZURE_SPEECH_REGION).+metadata: {"openclaw": {"requires": {"env": ["AZURE_SPEECH_KEY", "AZURE_SPEECH_REGION"]}, "primaryEnv": "AZURE_SPEECH_KEY"}}+---++# Azure AI Speech — Speech-to-Text++Transcribe audio to text with **Azure Fast Transcription** — synchronous,+word-level timestamps, speaker diarization, and multi-language identification.+In OpenMontage this is exposed through the `azure_stt` tool (`capability=analysis`,+`provider=azure`). It is an **optional cloud STT provider** — when+`AZURE_SPEECH_KEY` is configured, prefer it for cloud transcription. The local+`transcriber` tool (faster-whisper) remains the **default offline path** and the+fallback when Azure is unavailable.++> Docs: [Fast Transcription](https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create) · [Speech service overview](https://learn.microsoft.com/azure/ai-services/speech-service/spx-overview)++## Why Fast Transcription (not Batch)++Azure exposes three STT surfaces. OpenMontage uses **Fast Transcription** because+the pipeline transcribes **local audio files**:++| Surface | Input | Latency | Needs |+|---------|-------|---------|-------|+| **Fast Transcription** (used here) | local file, multipart POST | synchronous, sub-real-time | key + region |+| Batch Transcription | audio at a URL (Blob + SAS) | async job + polling | Blob storage plumbing |+| Speech SDK (`spx`) | mic / stream / file | streaming | native `azure-cognitiveservices-speech` package |++Fast Transcription needs no Blob storage, no SAS URLs, and no native SDK — just+`requests` and the two env vars.++## Setup++Create a **Speech** resource in the [Azure portal](https://portal.azure.com);+copy the key and region from its **Keys and Endpoint** page.++```bash+export AZURE_SPEECH_KEY=your_speech_resource_key+export AZURE_SPEECH_REGION=eastus # your resource's region+# export AZURE_SPEECH_ENDPOINT=https://... # optional: overrides region+```++`azure_stt` reports `AVAILABLE` once `AZURE_SPEECH_KEY` plus either+`AZURE_SPEECH_REGION` or `AZURE_SPEECH_ENDPOINT` are set.++## Using it in a pipeline++Prefer `azure_stt` over `transcriber` unless the run must be offline. Its output+matches the `transcriber` schema exactly, so it is a drop-in for `subtitle_gen`+and any stage that consumes a transcript.++```python+from tools.tool_registry import registry+registry.discover()+stt = registry._tools["azure_stt"]++result = stt.execute({+ "input_path": "projects/my-video/assets/audio/narration.mp3",+ # "language": "en", # ISO 639-1 or BCP-47 ("en-US"); omit for auto-ID+ # "diarize": True, # speaker labels, no HuggingFace token needed+ # "max_speakers": 4,+ "output_dir": "projects/my-video/artifacts",+})+if result.success:+ segs = result.data["segments"] # [{id,start,end,text,words:[...]}]+ words = result.data["word_timestamps"] # flat [{word,start,end,probability}]+```++If `azure_stt` is unavailable (no key) or errors, fall back to `transcriber`+(local whisper) — its `execute` signature and output are identical.++## Parameters that matter++- **`language`** — pass an ISO code (`"en"`) or a full locale (`"en-US"`). Pin it+ when you know the language; it is faster and more accurate than auto-ID.+- **`candidate_locales`** — when `language` is omitted, Azure runs language+ identification across this shortlist. Narrow it to the languages you actually+ expect; a huge list slows detection and invites misclassification.+- **`diarize` / `max_speakers`** — enable for multi-speaker audio (interviews,+ podcasts). Set `max_speakers` to the real upper bound.+- **`profanity_filter`** — `None` | `Masked` (default) | `Removed` | `Tags`.++## Response shape (mapped to the transcriber schema)++The raw Azure response (`phrases[]` with `offsetMilliseconds` / `words[]`) is+converted to seconds and the OpenMontage transcript schema:++```json+{+ "segments": [+ {"id": 0, "start": 0.0, "end": 2.4, "text": "Hello world",+ "speaker": 1,+ "words": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}]}+ ],+ "word_timestamps": [{"word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.98}],+ "language": "en-US",+ "duration_seconds": 2.4,+ "provider": "azure"+}+```++Note: Fast Transcription has no *per-word* confidence, so each word carries the+**phrase** confidence in `probability`.++## Limits & tips++- Single file up to ~2 hours / a few hundred MB per request. For longer or bulk+ jobs, use Azure Batch Transcription instead.+- Send clean audio (16 kHz+ mono is plenty). Transcode video to audio first if+ you only need speech — smaller upload, same result.+- Verify timing: word timestamps drive subtitle cues in `subtitle_gen`. Spot-check+ the first and last cues against the source audio.diff --git a/.env.example b/.env.exampleindex ac6f0708..14f2a351 100644--- a/.env.example+++ b/.env.example@@ -56,6 +56,12 @@ UNSPLASH_ACCESS_KEY= # Unsplash stock images (free developer key)# --- Analysis ---HF_TOKEN= # HuggingFace token — enables speaker diarization in transcriber+# Speech-to-text: optional Azure AI Speech (Fast Transcription). When set, the+# agent prefers azure_stt for cloud STT; the local faster-whisper transcriber+# remains the default offline path.+AZURE_SPEECH_KEY= # Azure AI Speech resource key ('Keys and Endpoint' page)+AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus+# AZURE_SPEECH_ENDPOINT= # Optional: full custom endpoint URL (overrides region)# --- Avatar (local installs) ---# WAV2LIP_PATH= # Path to cloned Wav2Lip repo (for lip sync)diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.mdindex feb67699..2a0b6bd5 100644--- a/AGENT_GUIDE.md+++ b/AGENT_GUIDE.md@@ -678,7 +678,8 @@ The `.agents/skills/` directory is large. When you're not coming in through a to| **Image generation** | `bfl-api`, `flux-best-practices` || **Video generation** | `seedance-2-0` (preferred premium default — cinematic, trailer, multi-shot, synced audio, lip-sync), `gemini-omni` (conversational video editing, reference tags, timecoded beats), `ai-video-gen`, `ltx2` || **Audio** | `elevenlabs`, `music`, `sound-effects`, `acestep`, `text-to-speech`, `setup-api-key` |-| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `speech-to-text`, `agents` |+| **Speech-to-text** | `speech-to-text` (whisper `transcriber` — default, offline), `azure-speech-to-text` (optional cloud STT — tool `azure_stt`, preferred when `AZURE_SPEECH_KEY` is set) |+| **Avatar / lip-sync** | `avatar-video`, `heygen`, `create-video`, `faceswap`, `video-translate`, `agents` || **Capture** | `playwright-recording` (browser flows), `ffmpeg` (post) || **Visualization** | `beautiful-mermaid`, `d3-viz`, `manim-composer`, `manimce-best-practices`, `manimgl-best-practices` || **Media editing** | `video-edit`, `video-download`, `video-understand`, `video-toolkit`, `visual-style` |diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.mdindex 8cee5bb2..be77ba3f 100644--- a/docs/PROVIDERS.md+++ b/docs/PROVIDERS.md@@ -43,6 +43,10 @@ DOUBAO_SPEECH_API_KEY= # Volcengine Doubao Speech TTS (strong Mandarin narDOUBAO_SPEECH_VOICE_TYPE= # Default Doubao speaker/voice typeDASHSCOPE_API_KEY= # Alibaba DashScope (Qwen image gen, TTS, ASR with word timestamps)+# SPEECH-TO-TEXT (optional cloud transcription; local whisper is the default)+AZURE_SPEECH_KEY= # Azure AI Speech — Fast Transcription (word-level timestamps)+AZURE_SPEECH_REGION= # Speech resource region, e.g. eastus+# MULTI-MODEL GATEWAY (one key, 6+ tools)FAL_KEY= # FLUX, Recraft, Kling, Veo, MiniMax video@@ -245,6 +249,54 @@ Doubao Speech 2.0 is billed by character package or usage in Volcengine. OpenMon---+### Azure AI Speech — Speech-to-Text++> **Cloud transcription.** Azure AI Speech Fast Transcription turns local audio into text with word-level timestamps, speaker diarization, and multi-language identification — no GPU required. Optional: the local faster-whisper `transcriber` remains the default offline STT path. When `AZURE_SPEECH_KEY` is set, the agent prefers `azure_stt` for cloud transcription.++**Tools unlocked:** `azure_stt`+**Env vars:** `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` (or `AZURE_SPEECH_ENDPOINT`)++#### Setup++1. In the [Azure portal](https://portal.azure.com), create a **Speech** resource (Azure AI services → Speech service).+2. Open the resource's **Keys and Endpoint** page.+3. Copy **KEY 1** and the **Location/Region** (e.g. `eastus`).+4. Add to `.env`:+ ```bash+ AZURE_SPEECH_KEY=your-speech-resource-key+ AZURE_SPEECH_REGION=eastus+ # AZURE_SPEECH_ENDPOINT=https://<custom>... # optional, overrides region+ ```++#### API Notes++OpenMontage uses the **Fast Transcription** REST endpoint, which accepts a local+audio file directly (multipart upload) and returns a synchronous result — no+Azure Blob storage, SAS URLs, or async job polling:++```text+POST https://{region}.api.cognitive.microsoft.com/speechtotext/transcriptions:transcribe?api-version=2024-11-15+Ocp-Apim-Subscription-Key: ${AZURE_SPEECH_KEY}+```++For files longer than ~2 hours or bulk jobs, use Azure Batch Transcription instead (not wired into OpenMontage).++#### What It Is Best For++- Cloud transcription with word-level timestamps and no local GPU+- Multi-language auto-detection across a candidate locale set+- Speaker diarization without a HuggingFace token+- Subtitle timing metadata that flows straight into `subtitle_gen`++#### Pricing++Azure AI Speech Standard (S0) bills speech-to-text by audio-hour (roughly+$1.00/audio-hour at time of writing; a free F0 tier includes a limited monthly+allowance). OpenMontage estimates cost from the transcribed audio duration. See+[Azure AI Speech pricing](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) for current rates.++---+### Google — TTS + Imagen + Music + Video (Shared Key)> **One key, five tools.** Google Cloud TTS has 700+ voices in 50+ languages — the strongest localization option. Imagen 4 generates high-quality images. Google Lyria generates high-quality background music. Gemini Omni Flash supports conversational video editing, and direct Veo generation covers premium short video clips.diff --git a/skills/INDEX.md b/skills/INDEX.mdindex c8a05090..dc81f0e6 100644--- a/skills/INDEX.md+++ b/skills/INDEX.md@@ -81,7 +81,8 @@ Key capability families to look for in the output:| FFmpeg | `core/ffmpeg.md` | Video encoding, filtering, composition | `ffmpeg`, `video-toolkit` || Remotion | `core/remotion.md` | React-based composition, Phase 3+ | `remotion-best-practices`, `remotion` || HyperFrames | `core/hyperframes.md` | HTML/CSS/GSAP composition runtime — kinetic typography, music-to-video, product promos, website capture. Vendored at v0.7.17 (2026-06-27). | `hyperframes` (router) → `hyperframes-core` (contract), `hyperframes-creative` (palette/type/narration), `hyperframes-media` (TTS/BGM/SFX/captions), `hyperframes-animation` (all motion), `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video` (beats-driven), `website-to-video`, `remotion-to-hyperframes` (migration), `gsap-core`, `gsap-timeline` |-| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps | `speech-to-text` |+| WhisperX | `core/whisperx.md` | Transcription with word-level timestamps — default STT (offline, free) | `speech-to-text` |+| Azure STT | (tool: `azure_stt`) | Optional cloud speech-to-text, word-level timestamps — preferred when `AZURE_SPEECH_KEY` is set | `azure-speech-to-text` || Subtitle Sync | `core/subtitle-sync.md` | Subtitle timing and alignment | `remotion-best-practices` || Color Grading | `core/color-grading.md` | FFmpeg color profiles, LUT workflow, accessibility | `ffmpeg` |@@ -306,7 +307,7 @@ Claude Code accesses them via symlinks in `.claude/skills/`.|----------|-----------------|--------|| **Video Composition** | `remotion-best-practices`, `remotion`, `hyperframes` (router), `hyperframes-core`, `hyperframes-creative`, `hyperframes-media`, `hyperframes-animation`, `hyperframes-cli`, `hyperframes-registry`, `media-use`, `motion-graphics`, `music-to-video`, `remotion-to-hyperframes`, `website-to-video` | `remotion-dev/skills`, `digitalsamba/claude-code-video-toolkit`, `heygen-com/hyperframes` (vendored v0.7.17, see `.agents/skills/hyperframes/PROVENANCE.md`) || **Video Processing** | `ffmpeg`, `video-toolkit` | `digitalsamba/claude-code-video-toolkit` |-| **TTS & Audio** | `text-to-speech`, `speech-to-text`, `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` |+| **TTS & Audio** | `text-to-speech`, `speech-to-text` (whisper, default STT), `azure-speech-to-text` (optional cloud STT), `music`, `sound-effects`, `elevenlabs`, `agents`, `setup-api-key` | `elevenlabs/skills`, `digitalsamba/claude-code-video-toolkit` || **Image Generation** | `flux-best-practices`, `bfl-api`, `grok-media` | `black-forest-labs/skills`, local OpenMontage skill || **Math Animation** | `manimce-best-practices`, `manimgl-best-practices`, `manim-composer` | `adithya-s-k/manim_skill` || **3D Graphics** | `threejs-animation`, `threejs-fundamentals`, `threejs-geometry`, `threejs-interaction`, `threejs-lighting`, `threejs-loaders`, `threejs-materials`, `threejs-postprocessing`, `threejs-shaders`, `threejs-textures` | `cloudai-x/threejs-skills` |diff --git a/tests/tools/test_azure_stt.py b/tests/tools/test_azure_stt.pynew file mode 100644index 00000000..49c0cb64--- /dev/null+++ b/tests/tools/test_azure_stt.py@@ -0,0 +1,242 @@+"""Focused tests for the Azure AI Speech (Fast Transcription) STT tool.++No live API calls: the network layer is monkeypatched. Covers the tool+contract, registry discovery, status behavior, the response→transcript+mapping, execute() guardrails, and downstream subtitle compatibility.+"""++import sys+from pathlib import Path++import pytest++PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent+sys.path.insert(0, str(PROJECT_ROOT))++from tools.base_tool import BaseTool, ToolStatus, ToolTier, ToolRuntime+from tools.tool_registry import ToolRegistry+from tools.analysis.azure_stt import AzureSpeechToText+from tools.subtitle.subtitle_gen import SubtitleGen+++# A representative Fast Transcription response payload.+SAMPLE_PAYLOAD = {+ "durationMilliseconds": 2400,+ "combinedPhrases": [{"text": "Hello world this is a test"}],+ "phrases": [+ {+ "offsetMilliseconds": 0,+ "durationMilliseconds": 1200,+ "text": "Hello world",+ "locale": "en-US",+ "confidence": 0.97,+ "speaker": 1,+ "words": [+ {"text": "Hello", "offsetMilliseconds": 0, "durationMilliseconds": 500},+ {"text": "world", "offsetMilliseconds": 500, "durationMilliseconds": 700},+ ],+ },+ {+ "offsetMilliseconds": 1200,+ "durationMilliseconds": 1200,+ "text": "this is a test",+ "locale": "en-US",+ "confidence": 0.95,+ "words": [+ {"text": "this", "offsetMilliseconds": 1200, "durationMilliseconds": 300},+ {"text": "is", "offsetMilliseconds": 1500, "durationMilliseconds": 200},+ {"text": "a", "offsetMilliseconds": 1700, "durationMilliseconds": 200},+ {"text": "test", "offsetMilliseconds": 1900, "durationMilliseconds": 500},+ ],+ },+ ],+}+++class _FakeResponse:+ def __init__(self, payload, status_code=200, text="ok"):+ self._payload = payload+ self.status_code = status_code+ self.text = text++ def json(self):+ return self._payload+++def azure_env(monkeypatch):+ monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")+ monkeypatch.setenv("AZURE_SPEECH_REGION", "eastus")+++# ---- Contract ----++class TestContract:+ def test_inherits_base_tool(self):+ assert issubclass(AzureSpeechToText, BaseTool)++ def test_identity(self):+ t = AzureSpeechToText()+ assert t.name == "azure_stt"+ assert t.capability == "analysis"+ assert t.provider == "azure"+ assert t.runtime == ToolRuntime.API+ assert t.tier == ToolTier.CORE+ assert t.fallback == "transcriber"+ assert "azure-speech-to-text" in t.agent_skills+ assert len(t.capabilities) > 0++ def test_get_info_valid(self):+ info = AzureSpeechToText().get_info()+ assert info["name"] == "azure_stt"+ assert info["capability"] == "analysis"+ assert "input_path" in info["input_schema"]["properties"]++ def test_estimate_cost_by_duration(self):+ t = AzureSpeechToText()+ assert t.estimate_cost({"duration_seconds": 3600}) == pytest.approx(1.0)+ assert t.estimate_cost({}) == 0.0+++# ---- Registry discovery ----++class TestDiscovery:+ def test_discoverable(self):+ reg = ToolRegistry()+ reg.discover("tools")+ assert reg.get("azure_stt") is not None++ def test_capability_routing(self):+ reg = ToolRegistry()+ reg.discover("tools")+ names = [t.name for t in reg.get_by_capability("analysis")]+ assert "azure_stt" in names+++# ---- Status behavior ----++class TestStatus:+ def test_unavailable_without_env(self, monkeypatch):+ monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)+ monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False)+ assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE++ def test_available_with_key_and_region(self, azure_env):+ assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE++ def test_available_with_key_and_endpoint(self, monkeypatch):+ monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)+ monkeypatch.setenv("AZURE_SPEECH_ENDPOINT", "https://custom.example.com")+ assert AzureSpeechToText().get_status() == ToolStatus.AVAILABLE++ def test_key_alone_is_not_enough(self, monkeypatch):+ monkeypatch.setenv("AZURE_SPEECH_KEY", "fake-key")+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)+ monkeypatch.delenv("AZURE_SPEECH_ENDPOINT", raising=False)+ assert AzureSpeechToText().get_status() == ToolStatus.UNAVAILABLE+++# ---- Response → transcript mapping (the risky logic) ----++class TestParsePayload:+ def test_maps_segments_words_and_timestamps(self):+ data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"])+ assert data["language"] == "en-US"+ assert data["duration_seconds"] == 2.4+ assert len(data["segments"]) == 2++ seg0 = data["segments"][0]+ assert seg0["text"] == "Hello world"+ assert seg0["start"] == 0.0 and seg0["end"] == 1.2+ assert seg0["speaker"] == 1+ # ms → seconds conversion on words+ assert seg0["words"][0] == {+ "word": "Hello", "start": 0.0, "end": 0.5, "probability": 0.97,+ }+ # flat word stream aggregates every phrase+ assert len(data["word_timestamps"]) == 6++ def test_locale_resolution(self):+ r = AzureSpeechToText._resolve_locales+ assert r({"language": "en"}) == ["en-US"] # ISO → default locale+ assert r({"language": "en-GB"}) == ["en-GB"] # explicit locale kept+ assert r({"candidate_locales": ["fr-FR"]}) == ["fr-FR"]+ assert len(r({})) > 1 # falls back to a shortlist++ def test_duration_falls_back_to_last_segment(self):+ payload = {k: v for k, v in SAMPLE_PAYLOAD.items() if k != "durationMilliseconds"}+ data = AzureSpeechToText._parse_payload(payload, ["en-US"])+ assert data["duration_seconds"] == pytest.approx(2.4)++ def test_output_feeds_subtitle_gen(self, tmp_path):+ """Parsed segments must be a drop-in for the subtitle generator."""+ data = AzureSpeechToText._parse_payload(SAMPLE_PAYLOAD, ["en-US"])+ out = tmp_path / "captions.srt"+ res = SubtitleGen().execute(+ {"segments": data["segments"], "format": "srt", "output_path": str(out)}+ )+ assert res.success+ assert out.exists()+ assert "Hello world" in out.read_text()+++# ---- execute() guardrails + mocked success ----++class TestExecute:+ def test_missing_file(self, azure_env, tmp_path):+ res = AzureSpeechToText().execute({"input_path": str(tmp_path / "nope.wav")})+ assert not res.success+ assert "not found" in res.error.lower()++ def test_missing_credentials(self, monkeypatch, tmp_path):+ monkeypatch.delenv("AZURE_SPEECH_KEY", raising=False)+ monkeypatch.delenv("AZURE_SPEECH_REGION", raising=False)+ audio = tmp_path / "a.wav"+ audio.write_bytes(b"RIFF....")+ res = AzureSpeechToText().execute({"input_path": str(audio)})+ assert not res.success+ assert "not configured" in res.error.lower()++ def test_success_path_mocked(self, azure_env, tmp_path, monkeypatch):+ import requests++ captured = {}++ def fake_post(url, headers=None, files=None, timeout=None):+ captured["url"] = url+ captured["headers"] = headers+ return _FakeResponse(SAMPLE_PAYLOAD)++ monkeypatch.setattr(requests, "post", fake_post)++ audio = tmp_path / "a.wav"+ audio.write_bytes(b"RIFF....")+ res = AzureSpeechToText().execute(+ {"input_path": str(audio), "language": "en", "output_dir": str(tmp_path)}+ )++ assert res.success+ assert res.model == "azure-fast-transcription"+ assert res.cost_usd == pytest.approx(0.0007, abs=1e-4) # 2.4s at $1/hr+ assert res.data["provider"] == "azure"+ assert len(res.data["segments"]) == 2+ # transcript JSON artifact was written+ assert res.artifacts and Path(res.artifacts[0]).exists()+ # correct endpoint + auth header used+ assert "transcriptions:transcribe" in captured["url"]+ assert captured["headers"]["Ocp-Apim-Subscription-Key"] == "fake-key"++ def test_http_error_surfaced(self, azure_env, tmp_path, monkeypatch):+ import requests++ monkeypatch.setattr(+ requests, "post",+ lambda *a, **k: _FakeResponse(None, status_code=401, text="Unauthorized"),+ )+ audio = tmp_path / "a.wav"+ audio.write_bytes(b"RIFF....")+ res = AzureSpeechToText().execute({"input_path": str(audio), "output_dir": str(tmp_path)})+ assert not res.success+ assert "401" in res.errordiff --git a/tools/analysis/azure_stt.py b/tools/analysis/azure_stt.pynew file mode 100644index 00000000..38fd9a93--- /dev/null+++ b/tools/analysis/azure_stt.py@@ -0,0 +1,365 @@+"""Azure AI Speech transcription tool (Fast Transcription REST API).++Speech-to-text with word-level timestamps and optional speaker diarization,+served by Azure AI Speech. This is an optional cloud STT provider; when+`AZURE_SPEECH_KEY` is configured the agent may prefer it for cloud+transcription, while the local faster-whisper `transcriber` tool remains the+default offline path.++Uses the **Fast Transcription** REST endpoint, which accepts a local audio+file directly (multipart upload) and returns a synchronous JSON result — no+Azure Blob storage, SAS URLs, or async job polling required. Output is shaped+to match `transcriber` exactly (segments + word_timestamps) so it is a+drop-in for `subtitle_gen` and every pipeline that consumes a transcript.++Docs: https://learn.microsoft.com/azure/ai-services/speech-service/fast-transcription-create+"""++from __future__ import annotations++import json+import os+import time+from pathlib import Path+from typing import Any++from tools.base_tool import (+ BaseTool,+ Determinism,+ ExecutionMode,+ ResourceProfile,+ ResumeSupport,+ RetryPolicy,+ ToolResult,+ ToolRuntime,+ ToolStability,+ ToolStatus,+ ToolTier,+)++# Fast Transcription API version (GA).+_API_VERSION = "2024-11-15"++# Candidate locales used for automatic language identification when the caller+# does not pin a language. Azure Fast Transcription accepts several locales and+# picks the best match per phrase.+_DEFAULT_CANDIDATE_LOCALES = [+ "en-US",+ "es-ES",+ "fr-FR",+ "de-DE",+ "it-IT",+ "pt-BR",+ "hi-IN",+ "ja-JP",+ "zh-CN",+]++# Minimal ISO 639-1 -> default BCP-47 locale map so callers can keep passing the+# short codes the whisper transcriber accepts (e.g. "en", "es").+_ISO_TO_LOCALE = {+ "en": "en-US",+ "es": "es-ES",+ "fr": "fr-FR",+ "de": "de-DE",+ "it": "it-IT",+ "pt": "pt-BR",+ "hi": "hi-IN",+ "ja": "ja-JP",+ "ko": "ko-KR",+ "zh": "zh-CN",+ "ar": "ar-EG",+ "ru": "ru-RU",+ "nl": "nl-NL",+ "tr": "tr-TR",+ "pl": "pl-PL",+ "id": "id-ID",+}+++class AzureSpeechToText(BaseTool):+ name = "azure_stt"+ version = "0.1.0"+ tier = ToolTier.CORE+ capability = "analysis"+ provider = "azure"+ stability = ToolStability.BETA+ execution_mode = ExecutionMode.SYNC+ determinism = Determinism.DETERMINISTIC+ runtime = ToolRuntime.API++ # Availability is decided by get_status() (env var check), mirroring the+ # ElevenLabs provider tool — dependencies stays empty.+ dependencies = []+ install_instructions = (+ "Set your Azure AI Speech credentials:\n"+ " export AZURE_SPEECH_KEY=your_speech_resource_key\n"+ " export AZURE_SPEECH_REGION=eastus # your Speech resource region\n"+ "Create a Speech resource in the Azure portal "+ "(https://portal.azure.com) — the key and region are on its "+ "'Keys and Endpoint' page. Optionally set AZURE_SPEECH_ENDPOINT to a "+ "full custom endpoint URL instead of the region."+ )+ agent_skills = ["azure-speech-to-text"]++ capabilities = [+ "transcribe",+ "word_timestamps",+ "diarization",+ "language_detection",+ ]++ best_for = [+ "Cloud transcription with word-level timestamps and no GPU",+ "Multi-language auto-detection across a candidate locale set",+ "Speaker diarization without a HuggingFace token",+ ]+ not_good_for = [+ "Fully offline runs (use the whisper `transcriber` fallback)",+ "Single files longer than ~2 hours (use Azure Batch transcription)",+ ]++ input_schema = {+ "type": "object",+ "required": ["input_path"],+ "properties": {+ "input_path": {"type": "string", "description": "Path to audio or video file"},+ "language": {+ "type": "string",+ "description": (+ "ISO 639-1 code (e.g. 'en') or BCP-47 locale (e.g. 'en-US'). "+ "Omit for automatic language identification across common locales."+ ),+ },+ "candidate_locales": {+ "type": "array",+ "items": {"type": "string"},+ "description": (+ "BCP-47 locales to consider for language ID when `language` is "+ "not set. Defaults to a common-language shortlist."+ ),+ },+ "diarize": {"type": "boolean", "default": False},+ "max_speakers": {"type": "integer", "default": 4},+ "profanity_filter": {+ "type": "string",+ "enum": ["None", "Masked", "Removed", "Tags"],+ "default": "Masked",+ },+ "output_dir": {"type": "string", "description": "Directory for output files"},+ },+ }++ output_schema = {+ "type": "object",+ "properties": {+ "segments": {"type": "array"},+ "word_timestamps": {"type": "array"},+ "language": {"type": "string"},+ "duration_seconds": {"type": "number"},+ },+ }++ resource_profile = ResourceProfile(+ cpu_cores=1,+ ram_mb=256,+ vram_mb=0,+ disk_mb=50,+ network_required=True,+ )++ retry_policy = RetryPolicy(+ max_retries=2,+ retryable_errors=["ConnectionError", "Timeout", "429", "503"],+ )+ resume_support = ResumeSupport.FROM_START+ idempotency_key_fields = ["input_path", "language", "diarize"]+ side_effects = ["writes transcript JSON to output_dir", "sends audio to Azure AI Speech"]+ fallback = "transcriber"+ fallback_tools = ["transcriber"]+ user_visible_verification = [+ "Check transcript text against source audio",+ "Verify word timestamps align with speech",+ ]++ def get_status(self) -> ToolStatus:+ if os.environ.get("AZURE_SPEECH_KEY") and (+ os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT")+ ):+ return ToolStatus.AVAILABLE+ return ToolStatus.UNAVAILABLE++ # Azure AI Speech Fast Transcription bills per audio-hour (Standard tier is+ # roughly $1.00/audio-hour at time of writing).+ COST_PER_AUDIO_HOUR = 1.0++ def estimate_cost(self, inputs: dict[str, Any]) -> float:+ # Duration is unknown until the audio is transcribed; return the actual+ # cost by passing duration_seconds, else 0.0 as a pre-call estimate.+ seconds = inputs.get("duration_seconds", 0) or 0+ return round((seconds / 3600.0) * self.COST_PER_AUDIO_HOUR, 4)++ def estimate_runtime(self, inputs: dict[str, Any]) -> float:+ # Fast Transcription is well under real-time; conservative default.+ return 20.0++ def _endpoint(self) -> str:+ endpoint = os.environ.get("AZURE_SPEECH_ENDPOINT")+ if endpoint:+ return endpoint.rstrip("/")+ region = os.environ.get("AZURE_SPEECH_REGION", "").strip()+ return f"https://{region}.api.cognitive.microsoft.com"++ @staticmethod+ def _resolve_locales(inputs: dict[str, Any]) -> list[str]:+ language = inputs.get("language")+ if language:+ if "-" in language: # already a BCP-47 locale+ return [language]+ return [_ISO_TO_LOCALE.get(language.lower(), f"{language.lower()}-US")]+ candidates = inputs.get("candidate_locales")+ return list(candidates) if candidates else list(_DEFAULT_CANDIDATE_LOCALES)++ def execute(self, inputs: dict[str, Any]) -> ToolResult:+ input_path = Path(inputs["input_path"])+ if not input_path.exists():+ return ToolResult(success=False, error=f"Input file not found: {input_path}")++ api_key = os.environ.get("AZURE_SPEECH_KEY")+ if not api_key or not (+ os.environ.get("AZURE_SPEECH_REGION") or os.environ.get("AZURE_SPEECH_ENDPOINT")+ ):+ return ToolResult(+ success=False,+ error="Azure Speech is not configured. " + self.install_instructions,+ )++ start = time.time()+ try:+ result = self._transcribe(inputs, api_key, input_path)+ except Exception as exc:+ return ToolResult(success=False, error=f"Transcription failed: {exc}")++ result.duration_seconds = round(time.time() - start, 2)+ result.model = "azure-fast-transcription"+ result.cost_usd = self.estimate_cost(+ {"duration_seconds": result.data.get("duration_seconds", 0)}+ )+ return result++ def _transcribe(+ self, inputs: dict[str, Any], api_key: str, input_path: Path+ ) -> ToolResult:+ import requests++ output_dir = Path(inputs.get("output_dir", input_path.parent))+ diarize = inputs.get("diarize", False)+ output_dir.mkdir(parents=True, exist_ok=True)++ locales = self._resolve_locales(inputs)+ definition: dict[str, Any] = {+ "locales": locales,+ "profanityFilterMode": inputs.get("profanity_filter", "Masked"),+ }+ if diarize:+ definition["diarization"] = {+ "maxSpeakers": inputs.get("max_speakers", 4),+ "enabled": True,+ }++ url = f"{self._endpoint()}/speechtotext/transcriptions:transcribe?api-version={_API_VERSION}"+ headers = {"Ocp-Apim-Subscription-Key": api_key}++ try:+ with open(input_path, "rb") as audio_file:+ response = requests.post(+ url,+ headers=headers,+ files={+ "audio": (input_path.name, audio_file, "application/octet-stream"),+ "definition": (None, json.dumps(definition), "application/json"),+ },+ timeout=600,+ )+ except requests.RequestException as exc: # network/timeout+ return ToolResult(success=False, error=f"Azure Speech request failed: {exc}")++ if response.status_code != 200:+ detail = response.text[:500] if response.text else ""+ return ToolResult(+ success=False,+ error=f"Azure Speech returned HTTP {response.status_code}: {detail}",+ )++ try:+ payload = response.json()+ except ValueError:+ return ToolResult(+ success=False,+ error="Azure Speech returned a non-JSON response.",+ )++ result_data = self._parse_payload(payload, locales)+ result_data["provider"] = "azure"+ output_path = output_dir / f"{input_path.stem}_transcript.json"+ output_path.write_text(json.dumps(result_data, indent=2), encoding="utf-8")++ return ToolResult(+ success=True,+ data=result_data,+ artifacts=[str(output_path)],+ )++ @staticmethod+ def _parse_payload(payload: dict[str, Any], requested_locales: list[str]) -> dict[str, Any]:+ """Map the Fast Transcription response onto the transcriber output schema."""+ segments: list[dict[str, Any]] = []+ word_timestamps: list[dict[str, Any]] = []+ detected_locale: str | None = None++ for idx, phrase in enumerate(payload.get("phrases", [])):+ offset_ms = phrase.get("offsetMilliseconds", 0)+ dur_ms = phrase.get("durationMilliseconds", 0)+ confidence = phrase.get("confidence")+ if detected_locale is None and phrase.get("locale"):+ detected_locale = phrase["locale"]++ seg: dict[str, Any] = {+ "id": idx,+ "start": round(offset_ms / 1000.0, 3),+ "end": round((offset_ms + dur_ms) / 1000.0, 3),+ "text": (phrase.get("text") or "").strip(),+ }+ if "speaker" in phrase:+ seg["speaker"] = phrase["speaker"]++ words = phrase.get("words") or []+ if words:+ seg_words = []+ for w in words:+ w_offset = w.get("offsetMilliseconds", 0)+ w_dur = w.get("durationMilliseconds", 0)+ word_entry = {+ "word": w.get("text", ""),+ "start": round(w_offset / 1000.0, 3),+ "end": round((w_offset + w_dur) / 1000.0, 3),+ # Fast Transcription has no per-word confidence; carry the+ # phrase confidence so downstream schemas stay populated.+ "probability": round(confidence, 3) if confidence is not None else None,+ }+ seg_words.append(word_entry)+ word_timestamps.append(word_entry)+ seg["words"] = seg_words++ segments.append(seg)++ duration_ms = payload.get("durationMilliseconds")+ if duration_ms is None and segments:+ duration_ms = max(int(s["end"] * 1000) for s in segments)++ return {+ "segments": segments,+ "word_timestamps": word_timestamps,+ "language": detected_locale or (requested_locales[0] if requested_locales else None),+ "duration_seconds": round((duration_ms or 0) / 1000.0, 3),+ }