Skip to content

Self-improving AI Employees for everyone

feat(speech): provider-abstracted text-to-speech and speech-to-text nodes

trohitga29cdb7e45ec
diff --git a/CLAUDE.md b/CLAUDE.md
index 952164c1..f37bcbf9 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -32,7 +32,8 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[GitHub Service](./docs-internal/github_service.md)** | GitHub CLI integration — `githubAction` (repo_clone / pr_create / pr_list / pr_merge / issue_create / issue_list + `custom` passthrough; AI tool `github`). **Strict Stripe pattern: the gh CLI owns its own auth** — no token stored/injected by OpenCompany, no auth pre-flight (gh's own error surfaces), marker OAuth row for the modal badge, fieldless catalogue entry. Login = spawned `gh auth login --hostname github.com --git-protocol https --web` — source-verified: headless takes the `isInteractive=false` branch (no Press-Enter), prints one-time code + `github.com/login/device` URL on stderr; the modal shows the code via the generic `verificationCode` plumbing (`useCredentialPanel` → `OAuthConnect`). `login_env()` strips `GH_TOKEN`/`GITHUB_TOKEN` (login aborts otherwise). Success gate = `gh auth status` exit 0, then best-effort **`gh auth setup-git`** (official bridge — the future git node needs zero auth code). gh binary: **project-local, pooch-driven** (temporal `_install.py` idiom) — pinned release under `package_dir("gh")`, system gh never consulted (auth state still shared: gh config paths are user-level). Palette group `vcs`. |
| **[Cloudflare Service](./docs-internal/cloudflare_service.md)** | Cloudflare CLI integration — `cloudflareAction` (whoami / zones_list / dns_records CRUD / `graphql_query` / `custom` passthrough; AI tool `cloudflare`) wrapping the official `cf` CLI (npm `[email protected]`, technical preview — "the next version of Wrangler"). **Two cf-specific variants of the CLI-managed-auth pattern**: (1) the CLI opens the browser ITSELF (loopback PKCE callback on fixed port 8877) so the login handler proxies nothing — single-flight guard (concurrent logins collide on the fixed port) + never-kill-the-shim invariant (killing the npm `.cmd` wrapper orphans the node child holding 8877); success gate = `cf auth whoami` JSON (`authenticated: true` — cf exits 0 in BOTH auth states). (2) **Fixed 86-scope OAuth grant with no `--scopes` flag** — only `dns_analytics:read` for analytics; no RUM/zone-analytics OAuth scope exists anywhere, so the optional `cloudflare_api_token` field (injected as `CLOUDFLARE_API_TOKEN`, cf's documented first-priority credential) is the only path to Web Analytics/RUM + the GraphQL Analytics API. `graphql_query` op POSTs the official `client/v4/graphql` endpoint directly (outside the OpenAPI schema cf/SDKs are generated from) — the replacement for the sunsetted Zone Analytics REST API; needs `Account > Account Analytics > Read`. Pinned npm install into the shared packages tree, system cf never consulted (preview argv surface drifts across versions). Palette group `deployment`. |
| **[Google Cloud Service](./docs-internal/gcloud_service.md)** | Google Cloud CLI integration — `gcloudAction` (auth/config/projects + set_project / Compute Engine instances list-start-stop-describe / Cloud Run deploy-list-describe / Cloud Storage ls-cp-rm / `custom` passthrough; AI tool `gcloud`) wrapping the official gcloud CLI (pinned versioned archive via pooch into `package_dir("gcloud")` — NOT npm; the Windows bundled-python zip keeps the legacy `google-cloud-sdk-` filename prefix, the `-cli-` spelling 404s, locked by test). Cloudflare login variant (CLI opens the browser itself, random loopback port — no fixed-port hazard) + the gh marker-token pattern; success gate = `gcloud auth list --filter=status:ACTIVE --format=json` non-empty (exit codes never trusted). **Config isolation divergence from gh**: every invocation pins `CLOUDSDK_CONFIG=<DATA_DIR>/gcloud/`, so terminal logins against the operator's global gcloud are deliberately NOT visible to the node. ADC note: node login mints gcloud USER creds only (CLI commands never read ADC); client-library code gets ADC via the `custom` op `auth application-default login` (lands isolated in the pinned dir). Cold install ~100 MB → login answers `pending: True` within the 22 s WS budget. Provider id `gcloud` is orthogonal to the Workspace `google` OAuth2 provider. Palette group `deployment`. |
-| **[Sarvam AI Service](./docs-internal/sarvam_service.md)** | Sarvam AI (Indic-first) — `sarvamChatModel` registered as an **OpenAI-compat provider** (one `_COMPAT_PROVIDERS` entry + a JSON block; no provider subclass) plus five REST nodes under `nodes/sarvam/` (`sarvamTranslate` / `sarvamTransliterate` / `sarvamDetectLanguage` / `sarvamSpeechToText` / `sarvamTextToSpeech`, palette group `language`). **One stored key spans two auth styles** — the chat route takes `Authorization: Bearer` (what the openai SDK sends) while every other endpoint takes only `api-subscription-key`, so `SarvamCredential` declares the native header and both surfaces share it. Introduces the generic **`supports_model_listing: false`** flag: Sarvam ships no `/v1/models` route, and unhandled that 404 becomes a `NodeUserError` that `AIService.fetch_models` re-raises *before* its curated fallback, breaking key validation and the model dropdown — so `OpenAIProvider.fetch_models` serves the curated list and probes with a one-token completion instead (defaults `true`; the other 12 providers are untouched, locked by test). Reasoning needs zero Python (`thinking_type: "effort"` + `reasoning_content` are already handled); `thinking_default_on` and `reasoning_models` are deliberately omitted. Also added generic **`Connection.request(files=)`** multipart support for the STT node (threaded through the auth-retry rebuild). TTS writes audio to `<workspace>/audio/` by default — inline base64 is opt-in and size-capped because a full request is ~12 MB. |
+| **[Sarvam AI Service](./docs-internal/sarvam_service.md)** | Sarvam AI (Indic-first) — `sarvamChatModel` registered as an **OpenAI-compat provider** (one `_COMPAT_PROVIDERS` entry + a JSON block; no provider subclass) plus three text REST nodes under `nodes/sarvam/` (`sarvamTranslate` / `sarvamTransliterate` / `sarvamDetectLanguage`, palette group `language`). Sarvam's speech endpoints are NOT nodes here — they are one provider inside the abstracted `textToSpeech` / `speechToText` nodes (see **[Speech Provider RFC](./docs-internal/speech_provider_rfc.md)**). **One stored key spans two auth styles** — the chat route takes `Authorization: Bearer` (what the openai SDK sends) while every other endpoint takes only `api-subscription-key`, so `SarvamCredential` declares the native header and every surface shares it. Introduces the generic **`supports_model_listing: false`** flag: Sarvam ships no `/v1/models` route, and unhandled that 404 becomes a `NodeUserError` that `AIService.fetch_models` re-raises *before* its curated fallback, breaking key validation and the model dropdown — so `OpenAIProvider.fetch_models` serves the curated list and probes with a one-token completion instead (defaults `true`; the other 12 providers are untouched, locked by test). Reasoning needs zero Python (`thinking_type: "effort"` + `reasoning_content` are already handled); `thinking_default_on` and `reasoning_models` are deliberately omitted. Also added generic **`Connection.request(files=)`** multipart support (threaded through the auth-retry rebuild), which the speech plugin still relies on. |
+| **[Speech Provider RFC](./docs-internal/speech_provider_rfc.md)** | Provider-abstracted speech — two nodes (`textToSpeech`, `speechToText`) with a `provider` dropdown, backed by **two registries, one per direction** (registry membership IS the capability: a synthesis-only vendor simply never appears in the transcription enum). Everything vendor-specific lives in the plugin folder [`server/nodes/speech/`](./server/nodes/speech/) — protocol, registries, dispatch, per-vendor modules; only the generic `services/provider_registry.py` and the vendor-neutral `services/media/` are shared. v1 providers: OpenAI (both directions), ElevenLabs (TTS), Deepgram + Groq (STT), Sarvam (both). Capabilities are JSON (`server/config/speech_defaults.json`) with per-model overrides, so no shared code branches on a provider name. **Audio never travels as bytes** — nodes return an `AudioRef` (a ~400 B reference) because a 12 MB base64 result blows Temporal's 2 MiB blob limit and retries 3× re-billing the provider each time. Backing routes: `GET/POST /api/workspace/{workflow_id}/…` in [`routers/workspace.py`](./server/routers/workspace.py). |
| **[CI/CD Pipeline](./docs-internal/ci_cd.md)** | GitHub Actions workflows, predeploy validation, release publishing, and composite setup action |
| **[Performance](./docs-internal/performance.md)** | Cold-start measurements (Application startup complete 2.90 s warm, 21.5 s cold post-`company clean` — was 71 s before the 2026-07-14 boot-delay fixes; first WS connect 8.29 s) + warm and cold per-phase timelines + optimisation history (lazy LangChain imports, lazy LLM-SDK exception refs, esbuild sidecar bundle, bytecode-at-build, PartySocket reconnect, TanStack Query auth bootstrap, `manualChunks`, Vite dep-cache preservation, off-loop Temporal build-id hash) + bottleneck inventory + reproduction commands + anti-patterns to never reintroduce (incl. eager SDK import at LLM provider registration, unconditional `.vite` wipe, synchronous `Worker()` on the event loop). Pairs with the build-time pipeline doc below. |
| **[Release Build Pipeline](./docs-internal/release_build_pipeline.md)** | npm-distribution build pipeline: `tsgo` for type-check (5× faster than tsc), Vite `manualChunks` + `target: 'es2022'`, esbuild Node.js sidecar bundle (drops `tsx` interpreter cost), bytecode pre-compile in two halves — `[tool.uv] compile-bytecode = true` (uv sync compiles `.venv/` site-packages) + `python -m compileall` for project source (plain `.pyc`, **no `-O`** — per PEP 488 the non-`-O` runtimes never load `.opt-1.pyc`; excludes `.venv/`, `tests/`). Single source of truth for the bytecode-compile path list: `cli.commands.build.COMPILEALL_SOURCE_DIRS`. |
@@ -618,7 +619,7 @@ The project uses WebSocket as the primary communication method between frontend
> The authoritative, per-node reference is the backend plugin registry plus the per-node "logic-flow" cards under [docs-internal/node-logic-flows/](./docs-internal/node-logic-flows/) (one card per node, grouped by category, with handles / params / outputs / side-effects / edge-cases). Live node list = the plugin registry (one folder per node under `server/nodes/<group>/`, 131 node types today via `len(services.node_registry.NODE_METADATA)`; a bare `__init__.py` glob overcounts by also matching the group packages). Do NOT maintain a per-node catalogue here — it drifts on every plugin add.
-Node groups (palette categories): agent, model, skill, tool, trigger, workflow, search, google, android, whatsapp, telegram, twitter, social, email, proxy, chat, scheduler, text, code, document, location, utility, browser, scraper, filesystem, stripe, vercel, github (palette group `vcs`), cloudflare and gcloud (palette group `deployment`), sarvam (palette group `language` — translate / transliterate / detect-language / speech-to-text / text-to-speech, sharing one API key with `sarvamChatModel`). See [docs-internal/node-logic-flows/](./docs-internal/node-logic-flows/) for the card index.
+Node groups (palette categories): agent, model, skill, tool, trigger, workflow, search, google, android, whatsapp, telegram, twitter, social, email, proxy, chat, scheduler, text, code, document, location, utility, browser, scraper, filesystem, stripe, vercel, github (palette group `vcs`), cloudflare and gcloud (palette group `deployment`), sarvam (palette group `language` — translate / transliterate / detect-language, sharing one API key with `sarvamChatModel`), speech (palette group `language` — provider-abstracted `textToSpeech` / `speechToText`). See [docs-internal/node-logic-flows/](./docs-internal/node-logic-flows/) for the card index.
## Backend Services
diff --git a/client/src/components/ParameterRenderer.tsx b/client/src/components/ParameterRenderer.tsx
index 9fa15144..86faa1f0 100644
--- a/client/src/components/ParameterRenderer.tsx
+++ b/client/src/components/ParameterRenderer.tsx
@@ -33,6 +33,7 @@ import { shouldShowParameter } from '../utils/parameterVisibility';
import { AI_MODEL_PROVIDER_MAP } from '../lib/aiModelProviders';
import { resolveNodeDescription } from '../lib/nodeSpec';
+import { uploadToWorkspace } from '../lib/workspaceUpload';
/**
* Radix reserves the empty string to clear a Select, so raw node option
@@ -1627,16 +1628,43 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({
);
case 'file':
+ // Hooks inside a switch case are a pre-existing pattern in this
+ // renderer (`fileInputRef` predates this change). Kept consistent
+ // rather than restructuring the whole component here.
const fileInputRef = React.useRef<HTMLInputElement>(null);
-
- const handleFileUpload = (e: React.ChangeEvent<HTMLInputElement>) => {
+ const [isUploading, setIsUploading] = useState(false);
+ const [uploadError, setUploadError] = useState<string | null>(null);
+
+ // Uploads go to the workspace and the parameter stores a ~400 byte
+ // reference. The legacy path — base64 straight into the parameter —
+ // is kept only as a fallback for a workflow that has never been
+ // saved, because it has no workspace to upload into yet. That path
+ // is a known hazard at media sizes: the blob rides the WebSocket, is
+ // persisted, is re-broadcast to every client, and then exceeds
+ // Temporal's payload limit. `coerce_file_param` still accepts it
+ // server-side so existing saved rows keep working.
+ const handleFileUpload = async (e: React.ChangeEvent<HTMLInputElement>) => {
const file = e.target.files?.[0];
if (!file) return;
+ const workflowId = useAppStore.getState().currentWorkflow?.id;
+ if (workflowId) {
+ try {
+ setUploadError(null);
+ setIsUploading(true);
+ onChange(await uploadToWorkspace(file, workflowId));
+ return;
+ } catch (err) {
+ setUploadError(err instanceof Error ? err.message : 'Upload failed.');
+ return;
+ } finally {
+ setIsUploading(false);
+ }
+ }
+
const reader = new FileReader();
reader.onload = () => {
const base64 = (reader.result as string).split(',')[1]; // Remove data:mime;base64, prefix
- // Store as object with base64 data, filename, and mime type
onChange({
type: 'upload',
data: base64,
@@ -1648,6 +1676,7 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({
};
const isUploadedFile = currentValue && typeof currentValue === 'object' && currentValue.type === 'upload';
+ const isFileRef = currentValue && typeof currentValue === 'object' && currentValue.kind === 'audio';
// Determine file accept type based on context (e.g., message_type for WhatsApp)
const getFileAcceptType = () => {
@@ -1677,14 +1706,20 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({
<div className="flex items-center gap-2">
<ShadcnInput
type="text"
- value={isUploadedFile ? `[Uploaded] ${currentValue.filename}` : (currentValue || '')}
+ value={
+ isFileRef
+ ? currentValue.filename
+ : isUploadedFile
+ ? `[Uploaded] ${currentValue.filename}`
+ : (currentValue || '')
+ }
onChange={(e) => onChange(e.target.value)}
placeholder={parameter.placeholder || 'Enter file path or upload'}
onDragOver={handleDragOver}
onDragLeave={handleDragLeave}
onDrop={handleDrop}
- readOnly={isUploadedFile}
- className={`flex-1 font-mono ${isDragOver ? 'border-ring ring-3 ring-ring/50 bg-accent/10' : ''} ${isUploadedFile ? 'bg-muted text-success' : (isFileTemplate ? 'text-info' : '')}`}
+ readOnly={isUploadedFile || isFileRef}
+ className={`flex-1 font-mono ${isDragOver ? 'border-ring ring-3 ring-ring/50 bg-accent/10' : ''} ${isUploadedFile || isFileRef ? 'bg-muted text-success' : (isFileTemplate ? 'text-info' : '')}`}
/>
<input
key={`file-input-${allParameters?.message_type || 'default'}`}
@@ -1697,15 +1732,17 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({
<ActionButton
intent="config"
onClick={() => fileInputRef.current?.click()}
+ disabled={isUploading}
title="Upload file"
>
- Upload
+ {isUploading ? 'Uploading…' : 'Upload'}
</ActionButton>
- {isUploadedFile && (
+ {(isUploadedFile || isFileRef) && (
<ActionButton
intent="stop"
onClick={() => {
onChange('');
+ setUploadError(null);
if (fileInputRef.current) fileInputRef.current.value = '';
}}
title="Clear uploaded file"
@@ -1714,10 +1751,16 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({
</ActionButton>
)}
</div>
- <div className="mt-1 text-[11px] italic text-muted-foreground">
- {isUploadedFile
- ? `Size: ${(currentValue.data.length * 0.75 / 1024).toFixed(1)} KB | Type: ${currentValue.mimeType}`
- : 'Enter server path or click Upload to select a file'}
+ <div
+ className={`mt-1 text-[11px] italic ${uploadError ? 'text-destructive' : 'text-muted-foreground'}`}
+ >
+ {uploadError
+ ? uploadError
+ : isFileRef
+ ? `Saved to the workspace: ${currentValue.path}`
+ : isUploadedFile
+ ? `Size: ${(currentValue.data.length * 0.75 / 1024).toFixed(1)} KB | Type: ${currentValue.mimeType}`
+ : 'Enter server path or click Upload to select a file'}
</div>
</div>
);
diff --git a/client/src/components/__tests__/OutputPanel.test.tsx b/client/src/components/__tests__/OutputPanel.test.tsx
index 4064f75b..fb17b8ca 100644
--- a/client/src/components/__tests__/OutputPanel.test.tsx
+++ b/client/src/components/__tests__/OutputPanel.test.tsx
@@ -94,3 +94,83 @@ describe('OutputPanel response routing', () => {
expect(views.some(v => v.textContent?.includes('42'))).toBe(true);
});
});
+
+/**
+ * Audio results.
+ *
+ * An AudioRef is an object, so without a dedicated branch it falls into
+ * the JSON tree and the user gets a wall of metadata instead of something
+ * they can play. Detection is structural (`kind === 'audio'`) rather than
+ * hint-driven, so a ref reaching this panel from anywhere still renders.
+ */
+describe('OutputPanel — audio', () => {
+ const SPEECH_NODE = { id: 'tts-1', type: 'textToSpeech' } as any;
+
+ const audioRef = (overrides: Record<string, any> = {}) => ({
+ kind: 'audio',
+ path: 'audio/greeting-abc123.wav',
+ workflow_id: 'wf-1',
+ filename: 'greeting-abc123.wav',
+ mime_type: 'audio/wav',
+ format: 'wav',
+ size_bytes: 24000,
+ duration_seconds: 1.5,
+ url: '/api/workspace/wf-1/files/audio/greeting-abc123.wav',
+ ...overrides,
+ });
+
+ const makeSpeechResult = (outputs: Record<string, any>) =>
+ [{ nodeId: 'tts-1', success: true, executionTime: 900, outputs }] as any;
+
+ it('renders a player instead of dumping the reference as JSON', () => {
+ specState.spec = { uiHints: { outputMode: 'audio' } };
+ const { container } = renderWithProviders(
+ <OutputPanel
+ results={makeSpeechResult({ audio: audioRef(), files: [audioRef()], chunk_count: 1 })}
+ selectedNode={SPEECH_NODE}
+ />,
+ );
+
+ const player = container.querySelector('audio');
+ expect(player).toBeTruthy();
+ expect(player?.getAttribute('src')).toContain(
+ '/api/workspace/wf-1/files/audio/greeting-abc123.wav',
+ );
+ expect(screen.getByText('greeting-abc123.wav')).toBeTruthy();
+ });
+
+ it('renders one player per clip when a provider split the input', () => {
+ specState.spec = { uiHints: { outputMode: 'audio' } };
+ const { container } = renderWithProviders(
+ <OutputPanel
+ results={makeSpeechResult({
+ audio: audioRef({ path: 'audio/a.wav', filename: 'a.wav', sha256: 'aa' }),
+ files: [
+ audioRef({ path: 'audio/a.wav', filename: 'a.wav', sha256: 'aa' }),
+ audioRef({ path: 'audio/b.wav', filename: 'b.wav', sha256: 'bb' }),
+ ],
+ chunk_count: 2,
+ })}
+ selectedNode={SPEECH_NODE}
+ />,
+ );
+
+ expect(container.querySelectorAll('audio')).toHaveLength(2);
+ // The clips are separate files, not parts of one stream — the UI has to
+ // say so, because concatenating them produces unplayable audio.
+ expect(screen.getByText(/separate files/i)).toBeTruthy();
+ });
+
+ it('does not hijack a non-audio result', () => {
+ specState.spec = null;
+ const { container } = renderWithProviders(
+ <OutputPanel
+ results={makeSpeechResult({ response: 'plain text' })}
+ selectedNode={SPEECH_NODE}
+ />,
+ );
+
+ expect(container.querySelector('audio')).toBeNull();
+ expect(screen.getByTestId('markdown')).toBeTruthy();
+ });
+});
diff --git a/client/src/components/output/AudioPreview.tsx b/client/src/components/output/AudioPreview.tsx
new file mode 100644
index 00000000..61cc1499
--- /dev/null
+++ b/client/src/components/output/AudioPreview.tsx
@@ -0,0 +1,108 @@
+import { useState } from 'react';
+import { AlertCircle } from 'lucide-react';
+import { buildApiUrl } from '../../config/api';
+
+/**
+ * An AudioRef as the backend serializes it (`services/media/refs.py`).
+ * Deliberately has no bytes/base64 field — audio travels as a reference,
+ * never as data. See `services/media/limits.py` for the measured reason.
+ */
+export interface AudioRef {
+ kind: 'audio';
+ path: string;
+ workflow_id?: string | null;
+ filename: string;
+ mime_type?: string;
+ format?: string;
+ size_bytes?: number;
+ duration_seconds?: number | null;
+ sample_rate?: number | null;
+ channels?: number | null;
+ sha256?: string | null;
+ url?: string | null;
+}
+
+/** Structural check — cheaper and more honest than trusting a uiHint alone. */
+export const isAudioRef = (value: unknown): value is AudioRef =>
+ !!value &&
+ typeof value === 'object' &&
+ (value as AudioRef).kind === 'audio' &&
+ typeof (value as AudioRef).path === 'string';
+
+const formatDuration = (seconds?: number | null): string | null => {
+ if (seconds == null || !Number.isFinite(seconds)) return null;
+ const total = Math.round(seconds);
+ return `${Math.floor(total / 60)}:${String(total % 60).padStart(2, '0')}`;
+};
+
+const formatSize = (bytes?: number): string | null => {
+ if (!bytes) return null;
+ return bytes < 1024 * 1024
+ ? `${(bytes / 1024).toFixed(0)} KB`
+ : `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
+};
+
+function AudioClip({ audio }: { audio: AudioRef }) {
+ const [failed, setFailed] = useState(false);
+
+ // `url` is path-only so a remote backend can be prefixed; fall back to
+ // composing it when an older ref predates the field.
+ const src = buildApiUrl(
+ audio.url || `/api/workspace/${audio.workflow_id ?? ''}/files/${audio.path}`
+ );
+
+ const meta = [formatDuration(audio.duration_seconds), formatSize(audio.size_bytes), audio.format]
+ .filter(Boolean)
+ .join(' · ');
+
+ return (
+ <div className="rounded-md border border-border bg-card p-3">
+ <div className="mb-2 flex items-baseline justify-between gap-2">
+ <span className="truncate font-mono text-xs text-foreground">{audio.filename}</span>
+ {meta && <span className="shrink-0 text-xs text-muted-foreground">{meta}</span>}
+ </div>
+
+ {failed ? (
+ // A 401 surfaces to <audio> as a silent `error` event, so an
+ // unauthenticated session would otherwise show a dead player with
+ // no explanation.
+ <div className="flex items-center gap-2 text-xs text-destructive">
+ <AlertCircle className="h-3.5 w-3.5 shrink-0" />
+ <span>Could not load this clip. It may have been cleaned up, or the session expired.</span>
+ </div>
+ ) : (
+ <audio
+ controls
+ preload="metadata"
+ src={src}
+ onError={() => setFailed(true)}
+ className="w-full"
+ />
+ )}
+ </div>
+ );
+}
+
+/**
+ * Renders one or more generated clips.
+ *
+ * Several clips are not one stream: providers that split long input return
+ * standalone files, each with its own container header, so they are shown
+ * as separate players rather than concatenated.
+ */
+export default function AudioPreview({ clips }: { clips: AudioRef[] }) {
+ if (!clips.length) return null;
+
+ return (
+ <div className="flex flex-col gap-2">
+ {clips.length > 1 && (
+ <span className="text-xs text-muted-foreground">
+ {clips.length} clips — separate files, not parts of one stream.
+ </span>
+ )}
+ {clips.map((clip, index) => (
+ <AudioClip key={clip.sha256 || clip.path || index} audio={clip} />
+ ))}
+ </div>
+ );
+}
diff --git a/client/src/components/output/OutputPanel.tsx b/client/src/components/output/OutputPanel.tsx
index 1194db8d..2eaa00a9 100644
--- a/client/src/components/output/OutputPanel.tsx
+++ b/client/src/components/output/OutputPanel.tsx
@@ -23,6 +23,7 @@ import { ExecutionResult } from '@/services/executionService';
import { useNodeSpec } from '@/lib/nodeSpec';
import { copyToClipboard, tryParseJson } from '@/utils/formatters';
import { cn } from '@/lib/utils';
+import AudioPreview, { AudioRef, isAudioRef } from './AudioPreview';
/** Extract output data from ExecutionResult. */
const getData = (r: ExecutionResult) =>
@@ -136,6 +137,14 @@ export default function OutputPanel({ results, onClear, selectedNode }: Props) {
// Terminal nodes sometimes emit JSON on stdout anyway — a tree view
// beats a wall of braces (stdlib JSON.parse via shared formatter).
const stdoutJson = isTerminal && typeof response === 'string' ? tryParseJson(response) : null;
+ // Audio results are AudioRef objects, which would otherwise fall into the
+ // JSON tree above. Detected structurally rather than by uiHint alone, so a
+ // ref reaching this panel from anywhere still renders as a player.
+ const audioClips: AudioRef[] = (
+ Array.isArray(data?.files) && data.files.some(isAudioRef)
+ ? data.files
+ : [data?.audio ?? response]
+ ).filter(isAudioRef);
const thinking = data?.thinking;
const metaTags = ['model', 'provider', 'agent_type'].filter(k => data?.[k]);
@@ -173,7 +182,13 @@ export default function OutputPanel({ results, onClear, selectedNode }: Props) {
)}
<div className="space-y-0">
- {response && (
+ {audioClips.length > 0 && (
+ <Section label="Audio" defaultOpen>
+ <AudioPreview clips={audioClips} />
+ </Section>
+ )}
+
+ {response && audioClips.length === 0 && (
<Section label="Response" defaultOpen>
{typeof response !== 'string' ? (
// Object/array response — themed JSON viewer (same --code-*
diff --git a/client/src/lib/workspaceUpload.ts b/client/src/lib/workspaceUpload.ts
new file mode 100644
index 00000000..01eda1f7
--- /dev/null
+++ b/client/src/lib/workspaceUpload.ts
@@ -0,0 +1,59 @@
+import { buildApiUrl } from '../config/api';
+import type { AudioRef } from '../components/output/AudioPreview';
+
+/**
+ * Mirrors `MEDIA_MAX_UPLOAD_BYTES` in `server/services/media/limits.py`.
+ * The server is the authority and returns 413 regardless; checking here
+ * only buys a better message before 25 MB goes over the wire.
+ */
+export const MEDIA_MAX_UPLOAD_BYTES = 25 * 1024 * 1024;
+
+export class UploadError extends Error {}
+
+/**
+ * POST a file into the workflow's workspace and return the stored reference.
+ *
+ * This exists because the alternative — base64 inside the node's parameters —
+ * is actively harmful at audio sizes. That path JSON-stringifies the blob onto
+ * the WebSocket, persists it in the `node_parameters` row, re-broadcasts it to
+ * every connected client, and then dies at Temporal's 2 MiB payload limit,
+ * burning three retries on the way. A ~1.5 MB clip is already ~2 MB of base64.
+ *
+ * The reference that comes back is ~400 bytes.
+ */
+export async function uploadToWorkspace(
+ file: File,
+ workflowId: string
+): Promise<AudioRef> {
+ if (!workflowId) {
+ throw new UploadError('Save the workflow before uploading a file to it.');
+ }
+ if (file.size > MEDIA_MAX_UPLOAD_BYTES) {
+ throw new UploadError(
+ `${file.name} is ${(file.size / (1024 * 1024)).toFixed(1)} MB; the limit is ` +
+ `${MEDIA_MAX_UPLOAD_BYTES / (1024 * 1024)} MB.`
+ );
+ }
+
+ const body = new FormData();
+ body.append('file', file, file.name);
+
+ // No explicit Content-Type: the browser must set it so the multipart
+ // boundary is generated. Setting it by hand produces a body the server
+ // cannot parse.
+ const response = await fetch(buildApiUrl(`/api/workspace/${workflowId}/uploads`), {
+ method: 'POST',
+ body,
+ credentials: 'include',
+ });
+
+ if (!response.ok) {
+ const detail = await response
+ .json()
+ .then((payload) => payload?.detail)
+ .catch(() => null);
+ throw new UploadError(detail || `Upload failed (${response.status}).`);
+ }
+
+ return response.json();
+}
diff --git a/client/src/types/INodeProperties.ts b/client/src/types/INodeProperties.ts
index 568deb63..ead88fde 100644
--- a/client/src/types/INodeProperties.ts
+++ b/client/src/types/INodeProperties.ts
@@ -224,8 +224,13 @@ export interface INodeUIHints {
/** OutputPanel: how the node's textual output renders. `"terminal"`
* = preformatted CLI text (never markdown — `#` would become
* headings and indentation would collapse). Declared by CLI-wrapper
- * plugins (githubAction, vercelAction, shell). */
- outputMode?: 'terminal';
+ * plugins (githubAction, vercelAction, shell). `"audio"` = render an
+ * `<audio>` player for a result carrying an AudioRef (`kind:
+ * "audio"`), declared by textToSpeech; without it the ref falls
+ * through to the JSON tree. Widening this union needs no change to
+ * the backend's `test_ui_hints_only_carry_known_flags` — that
+ * invariant checks flag *names*, not values. */
+ outputMode?: 'terminal' | 'audio';
}
export interface INodeTypeDescription {
diff --git a/docs-internal/node-logic-flows/README.md b/docs-internal/node-logic-flows/README.md
index 9331d123..28a0dc62 100644
--- a/docs-internal/node-logic-flows/README.md
+++ b/docs-internal/node-logic-flows/README.md
@@ -137,8 +137,8 @@ Sarvam AI's non-chat REST APIs. The chat models live under `ai_chat_models`
and share the same credential.
- [Sarvam Detect Language (`sarvamDetectLanguage`)](./language/sarvamDetectLanguage.md)
-- [Sarvam Speech to Text (`sarvamSpeechToText`)](./language/sarvamSpeechToText.md)
-- [Sarvam Text to Speech (`sarvamTextToSpeech`)](./language/sarvamTextToSpeech.md)
+- [Speech to Text (`speechToText`)](./language/speechToText.md)
+- [Text to Speech (`textToSpeech`)](./language/textToSpeech.md)
- [Sarvam Translate (`sarvamTranslate`)](./language/sarvamTranslate.md)
- [Sarvam Transliterate (`sarvamTransliterate`)](./language/sarvamTransliterate.md)
diff --git a/docs-internal/node-logic-flows/language/sarvamSpeechToText.md b/docs-internal/node-logic-flows/language/sarvamSpeechToText.md
deleted file mode 100644
index bf1add32..00000000
--- a/docs-internal/node-logic-flows/language/sarvamSpeechToText.md
+++ /dev/null
@@ -1,119 +0,0 @@
-# Sarvam Speech to Text (`sarvamSpeechToText`)
-
-| Field | Value |
-|------|-------|
-| **Category** | language |
-| **Backend handler** | [`server/nodes/sarvam/sarvam_speech_to_text.py`](../../../server/nodes/sarvam/sarvam_speech_to_text.py) |
-| **Shared helpers** | [`server/nodes/sarvam/_base.py`](../../../server/nodes/sarvam/_base.py) (`post_multipart`, `track_sarvam_usage`) |
-| **Tests** | [`server/tests/nodes/test_sarvam.py`](../../../server/tests/nodes/test_sarvam.py), [`server/tests/services/test_connection_multipart.py`](../../../server/tests/services/test_connection_multipart.py) |
-| **Skill (if any)** | [`server/skills/language_agent/sarvam-speech-skill/SKILL.md`](../../../server/skills/language_agent/sarvam-speech-skill/SKILL.md) |
-| **Dual-purpose tool** | yes — tool name `sarvam_speech_to_text` |
-
-## Purpose
-
-Transcribe short audio clips across 23 Indian languages using Sarvam's Saaras
-models, with optional direct translation to English. This is the repo's first
-`multipart/form-data` node.
-
-## Inputs (handles)
-
-| Handle | Connection type | Required | Purpose |
-|--------|-----------------|----------|---------|
-| `input-main` | main | no | Upstream data injected into params (commonly a file path) |
-
-## Parameters
-
-| Name | Type | Default | Required | displayOptions.show | Description |
-|------|------|---------|----------|---------------------|-------------|
-| `audio_file` | string \| object | `""` | yes | - | Path, **or** the file-widget upload envelope. `widget: file`, `accept: audio/*` |
-| `model` | enum | `saaras:v3` | no | - | `saaras:v3` (23 langs) / `saarika:v2.5` (legacy) |
-| `language_code` | string | `unknown` | no | - | BCP-47, or `unknown` to auto-detect |
-| `mode` | enum | `transcribe` | no | - | `transcribe` / `translate` / `verbatim` / `translit` / `codemix` |
-| `input_audio_codec` | enum\|null | `null` | no | - | `pcm_s16le` / `pcm_l16` / `pcm_raw` — headerless PCM only |
-
-## Outputs (handles)
-
-| Handle | Shape | Description |
-|--------|-------|-------------|
-| `output-main` | object | Standard envelope payload |
-
-### Output payload
-
-```ts
-{
- transcript: string;
- language_code: string | null;
- language_probability: number | null; // null when language_code was supplied
- timestamps: object | null; // conditional
- diarized_transcript: object | null; // Batch API only — always null here
- request_id: string | null;
-}
-```
-
-## Logic Flow
-
-```mermaid
-flowchart TD
- A[Receive params] --> B{mode advanced && model != saaras:v3?}
- B -- yes --> X[NodeUserError]
- B -- no --> C[_read_audio: dict upload or path]
- C -- dict, bad base64 --> X
- C -- path missing --> X
- C -- empty or oversize --> X
- C -- ok --> D[guess mime from filename]
- D --> E[ctx.connection sarvam -> POST /speech-to-text<br/>files= + data=, no Content-Type]
- E -- status >= 400 --> Y[NodeUserError w/ Sarvam message]
- E -- ok --> F[track_sarvam_usage speech_to_text, 30s]
- F --> G[Return SarvamSpeechToTextOutput]
-```
-
-## Decision Logic
-
-- **Mode/model compatibility**: `verbatim` / `translit` / `codemix` require
- `saaras:v3`; asking for them on `saarika:v2.5` fails before the call.
-- **Dual-shaped `audio_file`** (`_read_audio`): the frontend file widget sends
- `{"type": "upload", "data": "<base64>", "filename", "mimeType"}` when the
- user picks a file, and a bare string when they type a path or drag an
- upstream value in. The param is typed `Union[str, Dict]` for exactly this
- reason — `whatsapp_send` declares the same field as `str` and only survives
- because its handler reads the raw dict.
-- **Path resolution**: relative paths resolve against `ctx.workspace_dir`, so
- they line up with `fileDownloader` and `sarvamTextToSpeech` output.
-- **Size guard**: empty files and payloads over 24 MB fail with a message that
- names the 30-second limit and points at the Batch API.
-- **Billing**: `resource_count` is the documented 30-second ceiling; the node
- does not decode the clip to measure real duration.
-
-## Side Effects
-
-- **Database writes**: one `api_usage_metrics` row per call (seconds).
-- **Broadcasts**: standard node status only.
-- **External API calls**: `POST https://api.sarvam.ai/speech-to-text` as `multipart/form-data`, `api-subscription-key` header.
-- **File I/O**: reads the audio file when given a path. Writes nothing.
-- **Subprocess**: none.
-
-## External Dependencies
-
-- **Credentials**: `SarvamCredential` via `ctx.connection("sarvam")`.
-- **Framework**: `Connection.request(files=)` — added generically for this
- node and threaded through the auth-retry rebuild, so a 401 replays the file
- parts rather than an empty body. Pass **bytes**, never an open handle.
-- **Python packages**: `httpx` (through `Connection`), stdlib `base64` / `mimetypes`.
-
-## Edge cases & known limits
-
-- **30-second hard limit.** This is the synchronous endpoint; longer audio
- needs Sarvam's Batch API, which is a separate job-based surface and is not
- wired. Tell the user rather than retrying.
-- **Diarization is Batch-only.** `diarized_transcript` is declared on the
- output model for shape stability but will always be `null` here.
-- Accuracy is best at 16 kHz; PCM inputs are limited to 16 kHz.
-- `language_probability` is `null` whenever `language_code` was supplied
- explicitly (nothing was inferred).
-- No streaming — the WebSocket transcription surface is not wired.
-
-## Related
-
-- **Skills using this as a tool**: `sarvam-speech-skill`.
-- **Peer nodes**: [`sarvamTextToSpeech`](./sarvamTextToSpeech.md) (the other half of a voice loop).
-- **Architecture docs**: [Sarvam AI Service](../../sarvam_service.md).
diff --git a/docs-internal/node-logic-flows/language/sarvamTextToSpeech.md b/docs-internal/node-logic-flows/language/sarvamTextToSpeech.md
deleted file mode 100644
index 88291663..00000000
--- a/docs-internal/node-logic-flows/language/sarvamTextToSpeech.md
+++ /dev/null
@@ -1,141 +0,0 @@
-# Sarvam Text to Speech (`sarvamTextToSpeech`)
-
-| Field | Value |
-|------|-------|
-| **Category** | language |
-| **Backend handler** | [`server/nodes/sarvam/sarvam_text_to_speech.py`](../../../server/nodes/sarvam/sarvam_text_to_speech.py) |
-| **Shared helpers** | [`server/nodes/sarvam/_base.py`](../../../server/nodes/sarvam/_base.py) (`post_json`, `track_sarvam_usage`) |
-| **Tests** | [`server/tests/nodes/test_sarvam.py`](../../../server/tests/nodes/test_sarvam.py) |
-| **Skill (if any)** | [`server/skills/language_agent/sarvam-speech-skill/SKILL.md`](../../../server/skills/language_agent/sarvam-speech-skill/SKILL.md) |
-| **Dual-purpose tool** | yes — tool name `sarvam_text_to_speech` |
-
-## Purpose
-
-Synthesize natural speech in 11 Indian languages using Sarvam's Bulbul voices
-(37 on v3, 7 on v2). Writes audio into the workflow workspace and returns a
-path; inline base64 is opt-in and size-capped.
-
-## Inputs (handles)
-
-| Handle | Connection type | Required | Purpose |
-|--------|-----------------|----------|---------|
-| `input-main` | main | no | Upstream data injected into params |
-
-## Parameters
-
-| Name | Type | Default | Required | displayOptions.show | Description |
-|------|------|---------|----------|---------------------|-------------|
-| `text` | string | - | yes | - | Text to speak (2500 chars v3, 1500 v2) |
-| `target_language_code` | enum | `hi-IN` | no | - | 11 core codes |
-| `model` | enum | `bulbul:v3` | no | - | `bulbul:v3` / `bulbul:v2` |
-| `speaker` | string | `shubh` | no | - | Voice name, lowercase. Per-model sets |
-| `pace` | number | `1.0` | no | - | v3 0.5-2.0, v2 0.3-3.0 |
-| `speech_sample_rate` | enum | `24000` | no | - | 8000 / 16000 / 22050 / 24000 / 32000 / 44100 / 48000 |
-| `output_audio_codec` | enum | `wav` | no | - | wav / mp3 / linear16 / mulaw / alaw / opus / flac / aac |
-| `temperature` | number\|null | `null` | no | `model: [bulbul:v3]` | 0.01-2.0, v3 only |
-| `pitch` | number\|null | `null` | no | `model: [bulbul:v2]` | -0.75 to 0.75, v2 only |
-| `loudness` | number\|null | `null` | no | `model: [bulbul:v2]` | 0.3-3.0, v2 only |
-| `enable_preprocessing` | boolean\|null | `null` | no | `model: [bulbul:v2]` | v2 only (v3 always preprocesses) |
-| `return_audio` | enum | `file` | no | - | `file` / `base64` |
-
-## Outputs (handles)
-
-| Handle | Shape | Description |
-|--------|-------|-------------|
-| `output-main` | object | Standard envelope payload |
-
-### Output payload
-
-```ts
-{
- file_path: string | null; // chunk 1; null in base64 mode
- files: string[]; // one entry per chunk
- chunk_count: number;
- audio_format: string; // echoes output_audio_codec
- audio_base64: string | null; // opt-in, size-capped
- note: string | null; // multi-chunk or suppression explanation
- request_id: string | null;
-}
-```
-
-## Logic Flow
-
-```mermaid
-flowchart TD
- A[Receive params] --> B{len text > model cap?}
- B -- yes --> X[NodeUserError]
- B -- no --> C{speaker in model's voice set?}
- C -- no --> X
- C -- yes --> D{v3 && pace outside 0.5-2.0?}
- D -- yes --> X
- D -- no --> E[Build body: v3 -> temperature; v2 -> pitch/loudness/preprocessing]
- E --> F[ctx.connection sarvam -> POST /text-to-speech]
- F -- status >= 400 --> Y[NodeUserError w/ Sarvam message]
- F -- ok --> G{audios empty?}
- G -- yes --> X
- G -- no --> H{return_audio}
- H -- file --> I[_write_chunks -> workspace/audio, one file per chunk]
- H -- base64 --> J{chunk 1 over _MAX_INLINE_B64?}
- J -- yes --> K[audio_base64 null + note]
- J -- no --> L[audio_base64 = chunk 1]
- I --> M[track usage v3 or v2 rate]
- K --> M
- L --> M
- M --> N[Return SarvamTextToSpeechOutput]
-```
-
-## Decision Logic
-
-- **Per-model validation**: character cap, voice-set membership and (for v3)
- pace range all fail before the HTTP call. A v2 voice on v3 is an error, not
- a silent fallback.
-- **Version-gated body**: v3 sends `temperature`; v2 sends `pitch` /
- `loudness` / `enable_preprocessing`. They are mutually exclusive because v3
- rejects the v2 fields outright — `displayOptions` hides them in the UI and
- the body builder filters them regardless of what the caller sent.
-- **Default is file, not base64.** A 2500-character v3 request is roughly
- 12 MB of base64, which would land in the `node_outputs` JSON column, ride a
- status-WebSocket broadcast, and — because this node is tool-exposed — be
- serialized into an LLM message.
-- **Inline cap**: `_MAX_INLINE_B64 = 1_000_000`. Over it, `audio_base64` stays
- `None` and `note` says why and names `return_audio='file'`. Never a silent
- truncation.
-- **Multi-chunk**: Sarvam may split long text across `audios[]`. Each chunk is
- a standalone container with its own header, so byte-wise concatenation
- produces an unplayable file. One file per chunk; `file_path` is chunk 1;
- `note` explains.
-- **Billing**: `bulbul:v2` is half the v3 character rate, so the tracked
- action differs (`text_to_speech` vs `text_to_speech_v2`).
-
-## Side Effects
-
-- **Database writes**: one `api_usage_metrics` row per call (characters).
-- **Broadcasts**: standard node status only.
-- **External API calls**: `POST https://api.sarvam.ai/text-to-speech`, `api-subscription-key` header.
-- **File I/O**: writes `<workspace_dir>/audio/<slug>-<node8>-<rand6>[-N].<ext>` in `file` mode; creates the directory if absent. Falls back to a relative `audio/` dir when no workspace is set.
-- **Subprocess**: none.
-
-## External Dependencies
-
-- **Credentials**: `SarvamCredential` via `ctx.connection("sarvam")`.
-- **Python packages**: `httpx` (through `Connection`), stdlib `base64` / `uuid` / `re`.
-
-## Edge cases & known limits
-
-- **Audio is a file, not bytes.** Downstream nodes should consume `file_path`.
- Only reach for `return_audio: "base64"` for very short clips that must be
- inlined (e.g. chaining into `whatsappSend` with `media_source: "base64"`).
-- Voice sets do not overlap between v2 and v3.
-- `pitch` / `loudness` are silently unavailable on v3 — that is Sarvam's
- limitation, surfaced through `displayOptions` rather than an error.
-- Sample rates above 24000 are v3-REST-only.
-- No streaming — the HTTP-stream and WebSocket TTS surfaces are not wired.
-- Filenames embed a random suffix, so repeated runs never collide or
- silently overwrite prior audio.
-
-## Related
-
-- **Skills using this as a tool**: `sarvam-speech-skill`.
-- **Upstream nodes**: [`sarvamTransliterate`](./sarvamTransliterate.md) with `spoken_form: true` produces speech-ready text.
-- **Peer nodes**: [`sarvamSpeechToText`](./sarvamSpeechToText.md).
-- **Architecture docs**: [Sarvam AI Service](../../sarvam_service.md).
diff --git a/docs-internal/node-logic-flows/language/sarvamTransliterate.md b/docs-internal/node-logic-flows/language/sarvamTransliterate.md
index 69000258..6f4cc7d3 100644
--- a/docs-internal/node-logic-flows/language/sarvamTransliterate.md
+++ b/docs-internal/node-logic-flows/language/sarvamTransliterate.md
@@ -15,7 +15,7 @@ Convert text between scripts while preserving pronunciation — romanised Hindi
("namaste") to Devanagari ("नमस्ते") and back. **This is not translation**:
the words and meaning are unchanged, only the writing system. Optionally
expands numerals and abbreviations into spoken form, which is useful when the
-output feeds `sarvamTextToSpeech`.
+output feeds `textToSpeech`.
## Inputs (handles)
@@ -98,5 +98,5 @@ flowchart TD
## Related
- **Skills using this as a tool**: `sarvam-translate-skill`.
-- **Peer nodes**: [`sarvamTranslate`](./sarvamTranslate.md), [`sarvamTextToSpeech`](./sarvamTextToSpeech.md) (a common downstream consumer of spoken-form output).
+- **Peer nodes**: [`sarvamTranslate`](./sarvamTranslate.md), [`textToSpeech`](./textToSpeech.md) (a common downstream consumer of spoken-form output).
- **Architecture docs**: [Sarvam AI Service](../../sarvam_service.md).
diff --git a/docs-internal/node-logic-flows/language/speechToText.md b/docs-internal/node-logic-flows/language/speechToText.md
new file mode 100644
index 00000000..31dcc47e
--- /dev/null
+++ b/docs-internal/node-logic-flows/language/speechToText.md
@@ -0,0 +1,84 @@
+# Speech to Text (`speechToText`)
+
+| Field | Value |
+|------|-------|
+| **Category** | language |
+| **Backend handler** | [`server/nodes/speech/speech_to_text.py`](../../../server/nodes/speech/speech_to_text.py) |
+| **Shared helpers** | [`server/nodes/speech/_base.py`](../../../server/nodes/speech/_base.py), [`_unifier.py`](../../../server/nodes/speech/_unifier.py), [`_providers/`](../../../server/nodes/speech/_providers/) |
+| **Capabilities config** | [`server/config/speech_defaults.json`](../../../server/config/speech_defaults.json) |
+| **Tests** | [`server/tests/nodes/test_speech.py`](../../../server/tests/nodes/test_speech.py) |
+| **Skill (if any)** | [`server/skills/language_agent/speech-skill/SKILL.md`](../../../server/skills/language_agent/speech-skill/SKILL.md) |
+| **Dual-purpose tool** | yes — tool name `speech_to_text` |
+
+## Purpose
+
+Transcribe audio through any configured provider. Replaces the vendor-locked
+`sarvamSpeechToText`, and fixes two of its defects along the way (see
+**Edge cases**).
+
+## Inputs (handles)
+
+| Handle | Connection type | Required | Purpose |
+|--------|-----------------|----------|---------|
+| `input-main` | main | no | Upstream data injected into params |
+
+## Parameters
+
+| Name | Type | Default | Required | Description |
+|------|------|---------|----------|-------------|
+| `provider` | enum | `deepgram` | no | Registry-driven: the enum is `stt_providers()` |
+| `audio_file` | string \| AudioRef \| upload | `""` | yes | Widget `file`. Three shapes accepted — see below |
+| `stt_model` | string | `""` | no | Blank uses the provider default. **Not** named `model` |
+| `language` | string | `""` | no | Blank auto-detects |
+| `translate` | boolean | `false` | no | Translate to English rather than transcribe |
+| `diarize` | boolean | `false` | no | Speaker labels, where supported |
+| `timestamps` | boolean | `false` | no | Per-word timing, where supported |
+| `provider_options` | object | `{}` | no | Vendor-specific keys, passed through untouched |
+
+`audio_file` accepts an `AudioRef` (from an upstream node or the upload
+route), a bare workspace path, or the legacy `{type: "upload", data: base64}`
+envelope the file widget used to emit. All three route through
+`services.media`, which resolves under containment.
+
+## Outputs
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `transcript` | string | The text |
+| `language` / `language_confidence` | string / float | Detected or echoed |
+| `duration_seconds` | float | Provider-reported, else probed from the file |
+| `words` | object[] | `word` / `start` / `end` / `speaker` / `confidence` |
+| `segments` | object[] | Utterances or segments, provider-shaped |
+| `provider` / `stt_model` / `request_id` | string | What actually ran |
+
+## Side effects
+
+- Records an `APIUsageMetric` keyed on the provider id, in that provider's
+ own billing unit — seconds for OpenAI / Groq / Sarvam, **minutes** for
+ Deepgram (converted by the provider module, not by shared code)
+
+## Edge cases
+
+- **Path traversal — fixed.** The predecessor joined a user-supplied path
+ onto the workspace root unchecked, so `audio_file="../../credentials.db"`
+ read the encrypted credential store and uploaded it to the provider.
+ Every input shape now resolves through `services.media`, which rejects
+ `..` / `~` / drive prefixes and re-checks containment after resolution.
+- **Billing — fixed.** The predecessor charged every transcription as a flat
+ 30 seconds because it never measured the clip. Duration is now taken from
+ the provider when reported, otherwise probed with `inspect_audio`; when
+ neither is possible no metric is written, because an under-count is honest
+ and a fabricated one is not.
+- **Capability downgrades**: `verbose_json` is a 400 on `gpt-4o-transcribe`,
+ so a timestamp request on a gated model is downgraded with a WARN rather
+ than failing.
+- **Groq**: bills a 10-second floor per request; the provider applies it so
+ cost reporting matches the invoice. `whisper-large-v3-turbo` cannot
+ translate and falls back with a WARN.
+- **Sarvam**: timestamps and diarization are batch-API-only and come back
+ empty on the synchronous endpoint.
+
+## Related
+
+- **Peer nodes**: [`textToSpeech`](./textToSpeech.md) — the other half of a voice loop.
+- **RFC**: [`speech_provider_rfc.md`](../../speech_provider_rfc.md)
diff --git a/docs-internal/node-logic-flows/language/textToSpeech.md b/docs-internal/node-logic-flows/language/textToSpeech.md
new file mode 100644
index 00000000..82c5b6ff
--- /dev/null
+++ b/docs-internal/node-logic-flows/language/textToSpeech.md
@@ -0,0 +1,84 @@
+# Text to Speech (`textToSpeech`)
+
+| Field | Value |
+|------|-------|
+| **Category** | language |
+| **Backend handler** | [`server/nodes/speech/text_to_speech.py`](../../../server/nodes/speech/text_to_speech.py) |
+| **Shared helpers** | [`server/nodes/speech/_base.py`](../../../server/nodes/speech/_base.py), [`_unifier.py`](../../../server/nodes/speech/_unifier.py), [`_providers/`](../../../server/nodes/speech/_providers/) |
+| **Capabilities config** | [`server/config/speech_defaults.json`](../../../server/config/speech_defaults.json) |
+| **Tests** | [`server/tests/nodes/test_speech.py`](../../../server/tests/nodes/test_speech.py) |
+| **Skill (if any)** | [`server/skills/language_agent/speech-skill/SKILL.md`](../../../server/skills/language_agent/speech-skill/SKILL.md) |
+| **Dual-purpose tool** | yes — tool name `text_to_speech` |
+
+## Purpose
+
+Synthesize speech through any configured provider. Replaces the vendor-locked
+`sarvamTextToSpeech`: the provider is a parameter, so switching vendors never
+means swapping the node out of a workflow.
+
+## Inputs (handles)
+
+| Handle | Connection type | Required | Purpose |
+|--------|-----------------|----------|---------|
+| `input-main` | main | no | Upstream data injected into params |
+
+Handles are declared visible explicitly (`hide_input_handle = False` /
+`hide_output_handle = False`) because `usable_as_tool = True` otherwise
+auto-hides both, which would break chaining into `speechToText`.
+
+## Parameters
+
+| Name | Type | Default | Required | Description |
+|------|------|---------|----------|-------------|
+| `provider` | enum | `elevenlabs` | no | Registry-driven: the enum is `tts_providers()` |
+| `text` | string | - | yes | Text to speak; capped per provider/model |
+| `tts_model` | string | `""` | no | Blank uses the provider default. **Not** named `model` — see below |
+| `voice` | string | `""` | no | Loader `speechVoices`; live catalogue on ElevenLabs, config elsewhere |
+| `language` | string | `""` | no | Required by Sarvam (e.g. `hi-IN`), auto-detected elsewhere |
+| `speed` | number | `null` | no | Clamped to the provider's documented range |
+| `output_format` | string | `""` | no | Blank uses the provider default |
+| `sample_rate` | number | `null` | no | Where the provider supports it |
+| `provider_options` | object | `{}` | no | Vendor-specific keys, passed through untouched |
+
+**`tts_model`, not `model`.** [`ParameterRenderer.tsx:866`](../../../client/src/components/ParameterRenderer.tsx#L866)
+overwrites any field literally named `model` or `api_key` with chat-model data
+whenever a sibling `provider` field exists — and it never checks that the
+provider is an LLM provider. A field named `model` here would be cleared the
+moment a user picked ElevenLabs.
+
+## Outputs
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `audio` | AudioRef | The generated clip; the first when several |
+| `files` | AudioRef[] | Every clip |
+| `chunk_count` | int | >1 when the provider split long input |
+| `provider` / `tts_model` / `voice` | string | What actually ran |
+| `request_id` | string | Provider request id, where returned |
+| `note` | string | Set when several clips came back |
+
+`AudioRef` carries a path and metadata, never bytes — see
+[`services/media/limits.py`](../../../server/services/media/limits.py) for the
+measured reason.
+
+## Side effects
+
+- Writes one file per clip into `<workspace>/audio/` via `write_audio`
+ (atomic, random-suffixed filename, no collisions across runs)
+- Records an `APIUsageMetric` keyed on the provider id, in that provider's
+ own billing unit (characters for all three current TTS providers)
+
+## Edge cases
+
+- **Several clips**: Sarvam splits long input into standalone files. Each has
+ its own container header, so byte-concatenation produces audio that plays
+ only the first chunk. `note` says so explicitly.
+- **ElevenLabs without a voice**: refused up front — there is no
+ account-wide default voice.
+- **Over-cap text**: refused before the paid call, naming the cap.
+- **Unsupported provider**: `NodeUserError` listing the registered ones.
+
+## Related
+
+- **Peer nodes**: [`speechToText`](./speechToText.md) — the other half of a voice loop.
+- **RFC**: [`speech_provider_rfc.md`](../../speech_provider_rfc.md)
diff --git a/docs-internal/sarvam_service.md b/docs-internal/sarvam_service.md
index 7f08f2ce..37fc52c7 100644
--- a/docs-internal/sarvam_service.md
+++ b/docs-internal/sarvam_service.md
@@ -144,8 +144,13 @@ nothing here owns long-lived state, so the folder is helpers plus node files.
| `sarvamTranslate` | `sarvam_translate` | `POST /translate` | characters |
| `sarvamTransliterate` | `sarvam_transliterate` | `POST /transliterate` | characters |
| `sarvamDetectLanguage` | `sarvam_detect_language` | `POST /text-lid` | characters |
-| `sarvamSpeechToText` | `sarvam_speech_to_text` | `POST /speech-to-text` (multipart) | seconds |
-| `sarvamTextToSpeech` | `sarvam_text_to_speech` | `POST /text-to-speech` | characters |
+
+Sarvam's two speech endpoints are no longer nodes in this folder. They are
+reached through the provider-abstracted `textToSpeech` / `speechToText` nodes
+in [`server/nodes/speech/`](../server/nodes/speech/), where Sarvam is one
+provider among several; the wire logic was ported verbatim into
+[`_providers/sarvam.py`](../server/nodes/speech/_providers/sarvam.py). See
+[Speech Provider RFC](./speech_provider_rfc.md).
All authenticate through `ctx.connection("sarvam")`, which injects
`api-subscription-key` via `ApiKeyCredential.inject` — no per-node auth code.
@@ -161,84 +166,30 @@ Per-model input caps are enforced client-side (`mayura:v1` 1000 chars,
`sarvam-translate:v1` 2000, `/text-lid` 1000) so the LLM gets an actionable
message instead of an opaque 422.
-### Speech-to-text: multipart and the 30-second wall
+### Speech: moved to the provider-abstracted nodes
-This is the repo's first multipart upload. Rather than bypass the authed
-facade, `Connection.request` gained a generic `files=` parameter
-([`connection.py`](../server/services/plugin/connection.py)) — threaded into
-**both** the initial request and the auth-retry rebuild, so a 401 replays the
+The multipart plumbing this folder introduced is still load-bearing and still
+generic: `Connection.request` gained a `files=` parameter
+([`connection.py`](../server/services/plugin/connection.py)), threaded into
+**both** the initial request and the auth-retry rebuild so a 401 replays the
file parts instead of an empty body. Pass file *bytes*, never an open handle,
for exactly that reason.
-`audio_file` accepts **either** shape the frontend can produce: a path string
-(absolute, or relative to the workflow workspace) or the file widget's
-`{"type": "upload", "data": "<base64>", "filename", "mimeType"}` envelope.
-Typing it as `Union[str, Dict]` is deliberate — `whatsapp_send` declares the
-same field as `str` and only survives because its handler reads the raw dict.
-
-**The synchronous endpoint caps at 30 seconds of audio.** Longer recordings
-need Sarvam's Batch API, a separate job-based surface that is out of scope.
-Diarization is Batch-only too, so `diarized_transcript` is declared on the
-output model but will be `None` here.
-
-### Text-to-speech: files, not base64
-
-Sarvam returns audio as base64 inside the JSON body. A 2500-character v3
-request is roughly **12 MB of base64** — which would land in the
-`node_outputs` JSON column, ride a status-WebSocket broadcast, and (because the
-node is tool-exposed) be serialized into an LLM message.
-
-So the default is `return_audio: "file"`: audio is written under
-`<workspace>/audio/` and the node returns `file_path`. `return_audio: "base64"`
-is opt-in and capped at `_MAX_INLINE_B64 = 1_000_000`; over the cap
-`audio_base64` stays `None` and `note` explains why and points at the file
-mode. It never silently truncates.
-
-Sarvam may split long text into several chunks in `audios[]`. Each chunk is a
-standalone container with its own header, so **concatenating them byte-wise
-produces an unplayable file**. The node writes one file per chunk, points
-`file_path` at the first, lists all of them in `files`, and sets `note`.
-
-Voices are per model and do not overlap — 37 for `bulbul:v3`, 7 for
-`bulbul:v2`. Asking for a v2 voice on v3 is a `NodeUserError`, not a silent
-fallback. v2-only params (`pitch`, `loudness`, `enable_preprocessing`) and
-v3-only params (`temperature`, `dict_id`) are gated by `displayOptions` in the
-UI *and* filtered from the request body, since v3 rejects the v2 fields.
-
----
-
-## Cost tracking
-
-`@Operation(cost=...)` is declarative metadata that nothing reads at runtime,
-so attribution is an explicit `track_sarvam_usage()` call per operation —
-the same shape as [`nodes/twitter/_base.py`](../server/nodes/twitter/_base.py).
-It never raises: a metrics failure must not fail a successful API call.
-
-Sarvam publishes INR list prices. [`pricing.json`](../server/config/pricing.json)
-stores USD converted at **1 INR = 0.0113 USD (~88.5 INR/USD, 2026-07)**, with
-the rate and the source figures recorded in a `_note` key. Both an
-`api.sarvam` block and an `operation_map.sarvam` block are required —
-`calculate_api_cost` returns 0 without the latter.
-
-Token pricing for the chat models lives separately under `llm.sarvam`.
-
----
-
-## Icons
-
-`@lobehub/icons` has no Sarvam brand (278 entries, none matching), so the
-usual `lobehub:<brand>` route in `visuals.json` is unavailable. Instead the
-mark ships as SVG and is served by the backend:
-
-- `server/nodes/model/sarvam_chat_model/icon.svg` and `server/nodes/sarvam/icon.svg` → `GET /api/schemas/nodes/<type>/icon`
-- `server/credentials/icons/sarvam.svg` → `GET /api/schemas/credentials/sarvam/icon`
-
-The frontend's `/api/` branch in
-[`assets/icons/index.ts`](../client/src/assets/icons/index.ts) already resolves
-that wire format, so no icon-registry edit was needed — only an `<img>` wrapper
-in `AIProviderIcons.tsx` for the direct-FC consumers. Skills inherit the same
-icons automatically: `_parse_skill_metadata` tries `get_plugin_icon_path`
-before `visuals.json`.
+Two behaviours that were discovered here and are preserved in the speech
+plugin:
+
+- **Sarvam returns TTS audio as base64 inside JSON, and `audios` is an
+ array.** Long input comes back as several standalone clips, each with its
+ own container header, so concatenating them byte-wise produces audio that
+ plays only the first chunk. The node writes one file per clip and says so.
+- **Timestamps and diarization are Batch-API-only.** The synchronous endpoint
+ returns them as `null` regardless of what is requested.
+
+One claim from the original implementation did **not** survive verification:
+it asserted the synchronous endpoint "caps at 30 seconds of audio", and billed
+every transcription at that ceiling. Sarvam's only 30-second reference is
+about *response latency*, not audio duration. The speech node measures real
+duration instead.
### The mark itself
diff --git a/docs-internal/speech_provider_rfc.md b/docs-internal/speech_provider_rfc.md
index 46f0629a..4f32aa0d 100644
--- a/docs-internal/speech_provider_rfc.md
+++ b/docs-internal/speech_provider_rfc.md
@@ -117,8 +117,14 @@ hardcodes `self.credentials[0]` ([`base.py:566`](../server/services/plugin/base.
provider-agnostic. Copying it into `services/speech/` would fork the boot-time-import-avoidance
logic that exists specifically to keep ~7 s (warm) / ~45 s (cold) of SDK imports off startup.
-`services/provider_registry.py` + `services/provider_clients.py`; `services/llm/registry.py`
-becomes a shim. **Success criterion: `server/tests/llm/` passes untouched.**
+`services/provider_registry.py`; `services/llm/registry.py` becomes a shim.
+**Success criterion: `server/tests/llm/` passes untouched.** Met — 168 tests, no edits. `_REGISTRY`
+survives as an alias bound to the *same dict object* the registry mutates, because several tests
+swap provider factories in place through it.
+
+A companion `services/provider_clients.py` (the lease-counted client cache) was scoped here and
+deliberately **not** built. See §4: speech makes one HTTP call per node execution, so the cache has
+no second consumer and would have earned only a shutdown hook in `main.py`.
### D7 — `tinytag` for duration and format
@@ -159,42 +165,86 @@ retrying again.
## 4. Architecture
+Speech is a **plugin**, not a service. Everything vendor-specific lives in the plugin folder;
+only genuinely shared machinery sits under `services/`.
+
```
-services/provider_registry.py generic ProviderSpec + registry (extracted from llm/)
-services/provider_clients.py generic lease-counted client cache (extracted from llm/)
+services/provider_registry.py generic ProviderSpec + registry (shared with services/llm)
-services/media/ audio transport — vendor-neutral, reusable for image/video
+services/media/ audio transport -- vendor-neutral, reusable for image/video
refs.py AudioRef
workspace.py write_audio / resolve_media / read_media_bytes / coerce_file_param
inspect.py tinytag -> wave -> PCM arithmetic; never raises
limits.py every size constant, one place
-services/speech/ provider abstraction — mirrors services/llm/
- protocol.py TtsProvider / SttProvider, requests, results, SpeechError
- registry.py two registries over the generic one
- config.py reads server/config/speech_defaults.json
- unifier.py SpeechUnifier — dispatch + typed-error -> NodeUserError
- providers/ _http.py, _openai_compat.py (openai+groq STT), elevenlabs.py,
- sarvam.py, deepgram.py, gemini.py
-
-nodes/speech/ the two nodes
+nodes/speech/ the whole speech surface
+ text_to_speech.py the two nodes
+ speech_to_text.py
+ _protocol.py TtsProvider / SttProvider, requests, results, SpeechError
+ _registry.py two registries over the generic one
+ _config.py reads server/config/speech_defaults.json
+ _unifier.py dispatch + typed-error -> NodeUserError
+ _providers/ _http.py, _openai_compat.py (openai+groq), elevenlabs.py,
+ deepgram.py, sarvam.py
+ _credentials.py ElevenLabs + Deepgram (the rest are shared with nodes/model)
+ _base.py, _option_loaders.py
+
+config/speech_defaults.json operator-editable capabilities
routers/workspace.py GET file (Range-capable) + POST upload
```
-**Layering rule.** `services/speech` takes credential **id strings**, never `Credential` classes.
-The classes live under `nodes/`, and a `services → nodes` import inverts the layering and breaks
-`test_plugin_self_containment.py`. A test locks the two sides agree.
+**Why not `services/speech/`.** An earlier draft put the provider layer under `services/`, mirroring
+`services/llm/`. That mirror is misleading: `services/llm` earns its place because agents, chat
+models and a dozen nodes all consume it, whereas speech has exactly two consumers and both live in
+`nodes/speech/`. Per-vendor code under `services/` is the pattern Wave 11.I explicitly retired --
+`services.whatsapp_service`, `services.maps` and `services.nodejs_client` all moved into plugin
+folders, and `test_plugin_self_containment.py` names them in a forbidden-import list. Speech would
+have been the next entry on it.
+
+**Layering rule.** The speech layer takes credential **id strings**, never `Credential` classes.
+`nodes/speech/_config.credential_id(provider)` reads the id from JSON, and a test asserts every
+registered provider's id resolves in `CREDENTIAL_REGISTRY`. Cross-plugin *credential* imports are
+fine and idiomatic (`from ..model._credentials import OpenAICredential`, exactly what `nodes/sarvam/`
+already does); what stays out is any `services/` module knowing a vendor name.
+
+**No client cache.** `ChatUnifier` keeps a lease-counted LRU because an agent loop makes many model
+calls inside one node execution. Speech is the opposite shape -- one HTTP request per execution,
+where client setup is invisible next to a multi-second synthesis. Caching would buy nothing and cost
+a process-wide singleton plus a shutdown hook wired into `main.py`. This also settles the question
+left open when the registry was extracted: no second consumer materialised, so no shared client
+cache was built.
---
-## 5. Two bugs this work closes
-
-1. **Path traversal in `sarvam_speech_to_text`.** `_read_audio` does `Path(workspace_dir) / raw`
- with no containment check, so `audio_file="../../credentials.db"` reads the encrypted credential
- store and uploads it to the provider. `coerce_file_param` closes it by construction.
-2. **Workflow rename orphans every workspace file.** `rename_workflow` is a single-row `UPDATE`; the
+## 5. Bugs this work closes
+
+1. **Path traversal in `sarvam_speech_to_text`.** `_read_audio` did `Path(workspace_dir) / raw`
+ with no containment check, so `audio_file="../../credentials.db"` read the encrypted credential
+ store and uploaded it to the provider. `coerce_file_param` closes it by construction, and the
+ node that carried the bug no longer exists.
+2. **Flat 30-second billing.** The same node charged every transcription as 30 seconds, with the
+ comment "we do not decode the clip to measure it". It also justified the figure as a documented
+ endpoint limit, which the docs do not actually say (§7). Duration is now measured.
+3. **`workspace_root()` resolved by id where the directory is named by slug.** Introduced by this
+ work's own first wave: the ctx-less fallback composed `workspaces/<workflow_id>/`, a path that
+ never exists, because `WorkflowService._get_workspace_dir` names those directories after
+ `Workflow.slug`. It stayed invisible because every caller happened to pass a `NodeContext` — the
+ workspace HTTP route is the first that cannot. The fallback now raises rather than guessing, and
+ the route does the id → slug lookup itself, since that needs a database read and `services.media`
+ is synchronous by contract. `core.paths.workspace_dir`'s parameter was renamed to
+ `workflow_slug` so the next reader is not misled the same way.
+4. **Workflow rename orphans every workspace file.** `rename_workflow` is a single-row `UPDATE`; the
directory is never moved, and `_get_workspace_dir` keys on the mutable slug. Pre-existing and not
audio-specific, but `AudioRef` makes it fixable — a best-effort `os.replace` in the rename path.
+ Still outstanding.
+
+### On migrating existing workflows
+
+None was written. The two Sarvam speech nodes shipped only days before this work and no saved
+workflow references them, so a `workflow_migrations` entry would have been dead code guarding
+against a case that does not exist. Should one appear, the graph rewrite is the easy half; the
+parameter half is best-effort, because only 2 of the 5 migration call sites pass `node_parameters`
+and neither of those is the load path.
---
@@ -212,33 +262,66 @@ The classes live under `nodes/`, and a `services → nodes` import inverts the l
## 7. Verified provider surface
-Confirmed against live documentation on 2026-07-25. Anything not listed here was **not** verified
-and must be checked before it is coded.
+Confirmed against live vendor documentation on 2026-07-25/26, for the **v1 provider set only**
+(OpenAI, Groq, ElevenLabs, Deepgram, Sarvam). Gemini, Azure, AssemblyAI and Cartesia are out of
+scope and deliberately unverified.
-**OpenAI** — TTS `POST /v1/audio/speech`, models `gpt-4o-mini-tts` / `tts-1` / `tts-1-hd`, voices
-`alloy ash ballad coral echo fable nova onyx sage shimmer verse marin cedar` (subset for tts-1),
-`response_format` ∈ `mp3 opus aac flac wav pcm`, plus `instructions` and `speed`. Raw binary
-response. STT `POST /v1/audio/transcriptions`, multipart.
+The set diverges on every axis, which is what makes it a real test of the abstraction rather than
+four variations of one shape:
-**ElevenLabs** — TTS `POST /v1/text-to-speech/{voice_id}`, header `xi-api-key`, body `text`,
-`model_id` (default `eleven_multilingual_v2`), `voice_settings{stability, similarity_boost, style,
-use_speaker_boost, speed}`, `language_code`, `seed`, `apply_text_normalization` ∈ `auto|on|off`;
-**query** `output_format` (default `mp3_44100_128`, 27 values). Binary response. Models via
-`GET /v1/models`, voices via `GET /v1/voices`. STT `POST /v1/speech-to-text`, multipart,
-`model_id` ∈ `scribe_v2|scribe_v1`, `language_code`, `tag_audio_events` (default true),
-`num_speakers` (≤32), `timestamps_granularity` ∈ `none|word|character`, `diarize` (default false).
-Response `{language_code, language_probability, text, words[]}`.
-
-**Groq** — STT `POST https://api.groq.com/openai/v1/audio/transcriptions`, OpenAI-compatible,
-`whisper-large-v3` / `whisper-large-v3-turbo`. This is why `_openai_compat.py` covers both vendors
-off one factory.
-
-**Deepgram** — TTS `POST /v1/speak`, header `Authorization: Token <key>`, model names of the form
-`aura-2-<voice>-<lang>`. STT `POST /v1/listen`. Query-parameter driven; exact parameter set to be
-confirmed at implementation time.
-
-**Sarvam** — already implemented; see [`sarvam_service.md`](./sarvam_service.md). TTS returns
-base64 inside JSON (`{audios: […]}`), the one provider that does.
-
-**Gemini** — TTS via `generateContent` with an audio response modality. Shape **not yet verified**;
-must be confirmed before coding.
+| | auth header | request transport | response transport |
+|---|---|---|---|
+| OpenAI TTS | `Authorization: Bearer` | JSON body | **raw audio bytes** |
+| OpenAI / Groq STT | `Authorization: Bearer` | multipart | JSON |
+| ElevenLabs TTS | **`xi-api-key`** (no scheme) | JSON body + **query params** | **raw audio bytes** |
+| Deepgram STT | **`Authorization: Token`** | **raw body bytes** + **query params** | JSON |
+| Sarvam TTS | `api-subscription-key` | JSON body | **base64 in a JSON array** |
+| Sarvam STT | `api-subscription-key` | multipart | JSON |
+
+**OpenAI** -- TTS `POST /v1/audio/speech`; models `gpt-4o-mini-tts` (default), `tts-1`, `tts-1-hd`;
+13 voices, of which `tts-1` / `tts-1-hd` support only 9 (`ballad`, `verse`, `marin`, `cedar` are
+excluded); `response_format` in `mp3 opus aac flac wav pcm`; `speed` 0.25-4.0; input <=4096 chars;
+`instructions` works only on `gpt-4o-mini-tts`. STT `POST /v1/audio/transcriptions` (multipart), and
+**`response_format` is model-gated**: `whisper-1` allows `json text srt verbose_json vtt`, the
+`gpt-4o-*-transcribe` models allow only `json text`. Since `timestamp_granularities` requires
+`verbose_json`, word timestamps are effectively whisper-1 only, and requesting them elsewhere is a
+400 -- so the node downgrades rather than sends.
+
+**Groq** -- STT `POST https://api.groq.com/openai/v1/audio/transcriptions`, OpenAI-compatible;
+`whisper-large-v3-turbo` (default) and `whisper-large-v3`. **Turbo omits translation entirely.**
+`prompt` is capped at 224 tokens, and Groq bills a **10-second minimum per request** regardless of
+clip length.
+
+**ElevenLabs** -- TTS `POST /v1/text-to-speech/{voice_id}`; header `xi-api-key`, bare, no scheme
+keyword; body `text`, `model_id` (default `eleven_multilingual_v2`), `language_code`,
+`voice_settings`, `seed`, `apply_text_normalization`; **`output_format` is a query parameter**
+(default `mp3_44100_128`) and is silently ignored if placed in the body. Voices via
+`GET /v2/voices`, cursor-paginated (`has_more` + `next_page_token`; the docs say not to rely on
+`total_count`). `speed` has **two conflicting official ranges** -- schema 0.5-2.0, best-practices
+prose 0.7-1.2, never reconciled; the schema bound is what the API accepts and is what is enforced.
+Whether `voice_settings` are honoured on `eleven_v3` is **undocumented**.
+
+**Deepgram** -- STT `POST /v1/listen`; `Authorization: Token <key>`, **not** Bearer; **all options
+are query parameters** (`punctuate`, `diarize`, `smart_format`, `detect_language`, `paragraphs`,
+`utterances`, ...), with multi-value options as repeated keys rather than comma lists; audio is the
+**raw request body** with an `audio/*` content type, not multipart. Transcript at
+`results.channels[N].alternatives[N].transcript`. The documented default model is `base-general`, so
+an explicit default is configured instead. **No cloud file-size or duration cap is published** -- the
+25 MB figure in circulation belongs to their self-hosted SageMaker docs and is not asserted here.
+
+**Sarvam** -- TTS `POST /text-to-speech` returns **base64 inside JSON**, and `audios` is an **array**
+of standalone clips. Explicit nulls are rejected where an absent key is accepted. `bulbul:v3` takes
+`temperature` and rejects `pitch` / `loudness` / `enable_preprocessing`; `bulbul:v2` is the mirror
+image. STT `POST /speech-to-text` (multipart); timestamps and diarization are **batch-API-only** and
+come back null here.
+
+### Corrections to earlier drafts
+
+- An earlier version of this section listed an ElevenLabs STT endpoint and a Deepgram TTS endpoint.
+ Both products exist, but neither is in the v1 set and neither was re-verified, so neither is
+ implemented.
+- The Sarvam node asserted that its synchronous STT endpoint "caps at 30 seconds of audio", and
+ billed every transcription at that ceiling. The docs' only 30-second reference is about **response
+ latency**, not audio duration. The claim was wrong and the billing with it.
+- Sarvam's own docs contradict themselves on `speech_sample_rate` (schema `22050`, prose `24000`).
+ The configured default is `24000`, matching the prose and the previous implementation.
diff --git a/server/config/pricing.json b/server/config/pricing.json
index 6fb9b517..7b5741da 100644
--- a/server/config/pricing.json
+++ b/server/config/pricing.json
@@ -264,6 +264,35 @@
"text_to_speech_v2": 0.00001695,
"speech_to_text": 0.00009417,
"_default": 0.0000226
+ },
+ "elevenlabs": {
+ "_description": "Text-to-speech, billed per CHARACTER of input. ElevenLabs prices by subscription tier rather than by a flat published per-character rate, so this is the Creator-tier effective rate (~$0.30 per 1,000 characters) and is an ESTIMATE. Operators on another tier should edit this value — the node reports the character count truthfully either way.",
+ "_source": "https://elevenlabs.io/pricing",
+ "_updated": "2026-07",
+ "text_to_speech": 0.0003,
+ "_default": 0.0003
+ },
+ "deepgram": {
+ "_description": "Speech-to-text, billed per MINUTE of audio. Pay-as-you-go rate for the nova-3 family. The node converts measured seconds to minutes before recording, so resource_count here is minutes, not seconds.",
+ "_source": "https://deepgram.com/pricing",
+ "_updated": "2026-07",
+ "speech_to_text": 0.0043,
+ "_default": 0.0043
+ },
+ "openai": {
+ "_description": "OpenAI's /audio/* routes only. These are NOT token-priced, so they live here rather than under `llm.openai` — the two blocks are separate namespaces and do not collide. text_to_speech bills per CHARACTER of input ($0.60 per 1M characters for gpt-4o-mini-tts, the default model; tts-1 is ~25x that, and this table has no per-model axis, so a tts-1-heavy install should raise this value). speech_to_text bills per SECOND ($0.006/minute for whisper-1).",
+ "_source": "https://openai.com/api/pricing/",
+ "_updated": "2026-07",
+ "text_to_speech": 0.0000006,
+ "speech_to_text": 0.0001,
+ "_default": 0.0001
+ },
+ "groq": {
+ "_description": "Groq transcription only (token pricing lives under `llm.groq`). Billed per SECOND of audio: $0.04/hour for whisper-large-v3-turbo, the default model. Groq bills a 10-second floor per request, which the provider module applies before reporting the count.",
+ "_source": "https://groq.com/pricing/",
+ "_updated": "2026-07",
+ "speech_to_text": 0.0000111,
+ "_default": 0.0000111
}
},
@@ -368,6 +397,20 @@
"text_to_speech": "text_to_speech",
"text_to_speech_v2": "text_to_speech_v2",
"speech_to_text": "speech_to_text"
+ },
+ "_comment_speech": "Every speech provider needs an entry here as well as in api_pricing: calculate_api_cost() returns a zero cost SILENTLY when an action has no operation_map entry, so a missing line reads as 'free' rather than as an error.",
+ "elevenlabs": {
+ "text_to_speech": "text_to_speech"
+ },
+ "deepgram": {
+ "speech_to_text": "speech_to_text"
+ },
+ "openai": {
+ "text_to_speech": "text_to_speech",
+ "speech_to_text": "speech_to_text"
+ },
+ "groq": {
+ "speech_to_text": "speech_to_text"
}
},
diff --git a/server/config/speech_defaults.json b/server/config/speech_defaults.json
new file mode 100644
index 00000000..a70ac8d9
--- /dev/null
+++ b/server/config/speech_defaults.json
@@ -0,0 +1,206 @@
+{
+ "_comment": "Per-provider speech capabilities. Sibling of llm_defaults.json and read the same way: services/speech/config.py loads this once at import and every capability lookup goes through it, so shared code never branches on a provider name. Values here were verified against live official docs; see docs-internal/speech_provider_rfc.md section 7 for the sources and for the handful of items the docs do not state. Per-model overrides use the same {'model-id': value, '_default': value} shape as llm_defaults.json's max_output_tokens.",
+ "version": "1.0",
+ "providers": {
+ "openai": {
+ "base_url": "https://api.openai.com/v1",
+ "credential_id": "openai",
+ "tts": {
+ "endpoint": "/audio/speech",
+ "default_model": "gpt-4o-mini-tts",
+ "models": ["gpt-4o-mini-tts", "tts-1", "tts-1-hd"],
+ "_voices_comment": "13 voices exist, but tts-1 and tts-1-hd support only the first 9. Per-model override rather than a flat list so the node can validate before the provider 400s.",
+ "voices": {
+ "tts-1": ["alloy", "ash", "coral", "echo", "sage", "shimmer", "fable", "onyx", "nova"],
+ "tts-1-hd": ["alloy", "ash", "coral", "echo", "sage", "shimmer", "fable", "onyx", "nova"],
+ "_default": ["alloy", "ash", "ballad", "coral", "echo", "sage", "shimmer", "verse", "marin", "cedar", "fable", "onyx", "nova"]
+ },
+ "default_voice": "alloy",
+ "output_formats": ["mp3", "opus", "aac", "flac", "wav", "pcm"],
+ "default_output_format": "mp3",
+ "speed_range": [0.25, 4.0],
+ "max_input_chars": 4096,
+ "billed_unit": "characters",
+ "_instructions_comment": "Steerability via the `instructions` body field is gpt-4o-mini-tts only; it is silently ignored on tts-1 / tts-1-hd.",
+ "supports_instructions": {
+ "tts-1": false,
+ "tts-1-hd": false,
+ "_default": true
+ }
+ },
+ "stt": {
+ "endpoint": "/audio/transcriptions",
+ "translate_endpoint": "/audio/translations",
+ "default_model": "whisper-1",
+ "models": ["whisper-1", "gpt-4o-transcribe", "gpt-4o-mini-transcribe", "gpt-4o-transcribe-diarize"],
+ "_response_formats_comment": "Model-gated, and getting this wrong is a 400 rather than a degraded result. Because timestamp_granularities requires verbose_json, word timestamps are effectively whisper-1 only.",
+ "response_formats": {
+ "whisper-1": ["json", "text", "srt", "verbose_json", "vtt"],
+ "gpt-4o-transcribe-diarize": ["json", "text", "diarized_json"],
+ "_default": ["json", "text"]
+ },
+ "default_response_format": "json",
+ "supports_timestamps": {
+ "whisper-1": true,
+ "_default": false
+ },
+ "supports_diarization": {
+ "gpt-4o-transcribe-diarize": true,
+ "_default": false
+ },
+ "supports_prompt": {
+ "gpt-4o-transcribe-diarize": false,
+ "_default": true
+ },
+ "max_upload_bytes": 26214400,
+ "input_formats": ["flac", "mp3", "mp4", "mpeg", "mpga", "m4a", "ogg", "wav", "webm"],
+ "billed_unit": "seconds"
+ }
+ },
+ "groq": {
+ "_comment": "Serves OpenAI's exact transcription wire format at its own base URL, so it rides the same provider class. Transcription-only: no TTS product.",
+ "base_url": "https://api.groq.com/openai/v1",
+ "credential_id": "groq",
+ "stt": {
+ "endpoint": "/audio/transcriptions",
+ "translate_endpoint": "/audio/translations",
+ "default_model": "whisper-large-v3-turbo",
+ "models": ["whisper-large-v3-turbo", "whisper-large-v3"],
+ "_translate_comment": "The turbo model omits translation entirely; only whisper-large-v3 can translate.",
+ "supports_translate": {
+ "whisper-large-v3-turbo": false,
+ "_default": true
+ },
+ "response_formats": {
+ "_default": ["json", "text", "verbose_json"]
+ },
+ "default_response_format": "json",
+ "supports_timestamps": {
+ "_default": true
+ },
+ "supports_diarization": {
+ "_default": false
+ },
+ "max_prompt_tokens": 224,
+ "max_upload_bytes": 26214400,
+ "billed_unit": "seconds",
+ "_min_billed_comment": "Groq bills a 10-second floor regardless of clip length; the provider reports max(duration, 10).",
+ "min_billed_seconds": 10
+ }
+ },
+ "elevenlabs": {
+ "base_url": "https://api.elevenlabs.io",
+ "credential_id": "elevenlabs",
+ "tts": {
+ "_endpoint_comment": "voice_id is a PATH segment, and output_format is a QUERY parameter rather than a body field. Sending output_format in the JSON body yields the default silently instead of an error.",
+ "endpoint": "/v1/text-to-speech/{voice_id}",
+ "voices_endpoint": "/v2/voices",
+ "default_model": "eleven_multilingual_v2",
+ "models": ["eleven_multilingual_v2", "eleven_v3", "eleven_flash_v2_5", "eleven_flash_v2", "eleven_ttv_v3"],
+ "default_voice": "",
+ "output_formats": [
+ "mp3_22050_32", "mp3_24000_48", "mp3_44100_32", "mp3_44100_64",
+ "mp3_44100_96", "mp3_44100_128", "mp3_44100_192",
+ "opus_48000_32", "opus_48000_64", "opus_48000_96", "opus_48000_128", "opus_48000_192",
+ "pcm_8000", "pcm_16000", "pcm_22050", "pcm_24000", "pcm_32000", "pcm_44100", "pcm_48000",
+ "wav_8000", "wav_16000", "wav_22050", "wav_24000", "wav_32000", "wav_44100", "wav_48000",
+ "alaw_8000", "ulaw_8000"
+ ],
+ "default_output_format": "mp3_44100_128",
+ "_tier_gated_comment": "mp3 at 192 kbps needs Creator+; PCM and WAV at 44.1 kHz need Pro+. A tier miss surfaces as a permission error, which is why the panel does not pre-filter.",
+ "_speed_comment": "The OpenAPI schema allows 0.5-2.0 while the best-practices guide recommends 0.7-1.2. Both are official and were never reconciled, so the API bound is enforced and the narrower range is guidance only.",
+ "speed_range": [0.5, 2.0],
+ "voice_settings_defaults": {
+ "stability": 0.5,
+ "similarity_boost": 0.75,
+ "style": 0.0,
+ "use_speaker_boost": true
+ },
+ "billed_unit": "characters"
+ }
+ },
+ "deepgram": {
+ "_comment": "Options are QUERY parameters, audio is sent as a RAW body with an audio/* Content-Type (not multipart), and auth is 'Authorization: Token <key>' rather than Bearer. All three differ from every other provider here.",
+ "base_url": "https://api.deepgram.com/v1",
+ "credential_id": "deepgram",
+ "stt": {
+ "endpoint": "/listen",
+ "_default_model_comment": "Deepgram's own documented default is base-general, which is not what anyone wants in 2026, so an explicit default is set here rather than inherited.",
+ "default_model": "nova-3",
+ "models": [
+ "nova-3", "nova-3-general", "nova-3-medical",
+ "nova-2", "nova-2-meeting", "nova-2-phonecall", "nova-2-voicemail",
+ "nova-2-finance", "nova-2-conversationalai", "nova-2-video", "nova-2-medical",
+ "nova", "nova-general", "nova-phonecall",
+ "whisper-large", "whisper-medium", "whisper-small", "whisper-base", "whisper-tiny"
+ ],
+ "supports_timestamps": {
+ "_default": true
+ },
+ "supports_diarization": {
+ "_default": true
+ },
+ "_smart_format_comment": "smart_format supersedes punctuate and additionally applies paragraphing plus number/date/currency formatting.",
+ "default_smart_format": true,
+ "_max_upload_comment": "Deepgram publishes no cloud file-size or duration cap. The 25 MB figure in circulation belongs to their self-hosted SageMaker docs and does not apply here, so no limit is asserted; services.media's read cap is the only ceiling.",
+ "max_upload_bytes": null,
+ "billed_unit": "minutes"
+ }
+ },
+ "sarvam": {
+ "_comment": "Indic-first. TTS returns base64 inside a JSON array rather than raw bytes, and long input comes back as several standalone clips that must not be byte-concatenated.",
+ "base_url": "https://api.sarvam.ai",
+ "credential_id": "sarvam",
+ "tts": {
+ "endpoint": "/text-to-speech",
+ "default_model": "bulbul:v3",
+ "models": ["bulbul:v3", "bulbul:v2"],
+ "voices": {
+ "bulbul:v2": ["anushka", "abhilash", "manisha", "vidya", "arya", "karun", "hitesh"],
+ "_default": [
+ "shubh", "aditya", "ritu", "priya", "neha", "rahul", "pooja", "rohan",
+ "simran", "kavya", "amit", "dev", "ishita", "shreya", "ratan", "varun",
+ "manan", "sumit", "roopa", "kabir", "aayan", "ashutosh", "advait", "anand",
+ "tanya", "tarun", "sunny", "mani", "gokul", "vijay", "shruti", "suhani",
+ "mohit", "kavitha", "rehan", "soham", "rupali"
+ ]
+ },
+ "default_voice": "shubh",
+ "languages": ["bn-IN", "en-IN", "gu-IN", "hi-IN", "kn-IN", "ml-IN", "mr-IN", "od-IN", "pa-IN", "ta-IN", "te-IN"],
+ "default_language": "hi-IN",
+ "output_formats": ["wav", "mp3", "linear16", "mulaw", "alaw", "opus", "flac", "aac"],
+ "default_output_format": "wav",
+ "max_input_chars": {
+ "bulbul:v2": 1500,
+ "_default": 2500
+ },
+ "speed_range": {
+ "bulbul:v2": [0.3, 3.0],
+ "_default": [0.5, 2.0]
+ },
+ "_sample_rate_comment": "Sarvam's own docs disagree with themselves here: the OpenAPI schema says 22050 while the field description says 24000. 24000 matches what the previous node shipped and what the description states.",
+ "sample_rates": [8000, 16000, 22050, 24000, 32000, 44100, 48000],
+ "default_sample_rate": 24000,
+ "billed_unit": "characters"
+ },
+ "stt": {
+ "endpoint": "/speech-to-text",
+ "translate_endpoint": "/speech-to-text-translate",
+ "default_model": "saaras:v3",
+ "models": ["saaras:v3", "saarika:v2.5"],
+ "_modes_comment": "Modes beyond transcribe/translate require saaras:v3.",
+ "modes": ["transcribe", "translate", "verbatim", "translit", "codemix"],
+ "_timestamps_comment": "Timestamps and diarization exist only on the batch job API, never on this synchronous endpoint. The response fields are always null here.",
+ "supports_timestamps": {
+ "_default": false
+ },
+ "supports_diarization": {
+ "_default": false
+ },
+ "default_language": "unknown",
+ "max_upload_bytes": 25165824,
+ "billed_unit": "seconds"
+ }
+ }
+ }
+}
diff --git a/server/core/paths.py b/server/core/paths.py
index fe57ba57..8186effd 100644
--- a/server/core/paths.py
+++ b/server/core/paths.py
@@ -262,8 +262,15 @@ def workspaces_dir() -> Path:
return data_path(Settings().workspace_base_dir)
-def workspace_dir(workflow_id: str) -> Path:
- """Per-workflow workspace at ``<opencompany_root>/workspaces/<workflow_id>/``.
+def workspace_dir(workflow_slug: str) -> Path:
+ """Per-workflow workspace at ``<opencompany_root>/workspaces/<slug>/``.
+
+ Keyed by ``Workflow.slug``, NOT by ``Workflow.id`` — the parameter was
+ named ``workflow_id`` until it was noticed that every real caller
+ passes a slug (see ``WorkflowService._get_workspace_dir``, which is
+ what actually creates these directories, and the rename path in
+ ``workflow_storage/handlers.py`` which moves them when the slug
+ changes). Passing an id here yields a directory that does not exist.
The workflow executor injects this into the execution context as
``ctx.raw["workspace_dir"]`` and the cli_agent service splices it
@@ -272,7 +279,7 @@ def workspace_dir(workflow_id: str) -> Path:
executors) + materialise its connected skills under
``<workspace_dir>/.claude/skills/``.
"""
- return workspaces_dir() / workflow_id
+ return workspaces_dir() / workflow_slug
def example_workflows_dir() -> Path:
diff --git a/server/main.py b/server/main.py
index 8fb8f4f0..d104ff4e 100644
--- a/server/main.py
+++ b/server/main.py
@@ -59,7 +59,7 @@ _startup_log("Importing DI container + all services...")
from core.container import container
_startup_log("Importing routers...")
-from routers import workflow, database, websocket, webhook, auth, credentials, schemas
+from routers import workflow, database, websocket, webhook, auth, credentials, schemas, workspace
_startup_log("All imports complete")
@@ -612,6 +612,7 @@ app.include_router(database.router)
app.include_router(websocket.router)
app.include_router(credentials.router) # Credentials panel - lazy per-tile icon endpoint (n8n pattern)
app.include_router(schemas.router) # Per-node output schema endpoint (GET /api/schemas/nodes/{type}.json)
+app.include_router(workspace.router) # Per-workflow workspace file serving + uploads
# Routers awaiting migration into their plugin folders. As each plugin
# moves to the self-contained pattern (nodes/<plugin>/_router.py +
diff --git a/server/nodes/sarvam/__init__.py b/server/nodes/sarvam/__init__.py
index bc7c7543..005afb15 100644
--- a/server/nodes/sarvam/__init__.py
+++ b/server/nodes/sarvam/__init__.py
@@ -1,15 +1,22 @@
"""Sarvam AI service plugins.
-Five stateless REST nodes over ``api.sarvam.ai`` — translate,
-transliterate, language ID, speech-to-text and text-to-speech — sharing
-one ``api-subscription-key`` credential. Node classes register themselves
-on import via ``BaseNode.__init_subclass__``; the only explicit wiring
-here is the output-schema registry, which the parameter panel reads to
-render "Input Data & Variables" for downstream nodes.
+Three stateless REST nodes over ``api.sarvam.ai`` — translate,
+transliterate and language ID — sharing one ``api-subscription-key``
+credential. Node classes register themselves on import via
+``BaseNode.__init_subclass__``; the only explicit wiring here is the
+output-schema registry, which the parameter panel reads to render
+"Input Data & Variables" for downstream nodes.
+
+Speech used to live here too, as ``sarvamTextToSpeech`` and
+``sarvamSpeechToText``. Both were retired in favour of the
+provider-abstracted ``textToSpeech`` / ``speechToText`` nodes in
+``nodes/speech/``, where Sarvam is one provider among several and the
+same canvas node can switch vendors without being replaced. The Sarvam
+wire logic was ported verbatim into ``nodes/speech/_providers/sarvam.py``.
Sarvam's OpenAI-compatible chat endpoint is a separate plugin
(``nodes/model/sarvam_chat_model``) registered through the standard
-LLM-provider path, and both surfaces share ``SarvamCredential``.
+LLM-provider path, and every surface shares ``SarvamCredential``.
"""
from services.node_output_schemas import register_output_schema
@@ -18,14 +25,6 @@ from .sarvam_detect_language import (
SarvamDetectLanguageNode,
SarvamDetectLanguageOutput,
)
-from .sarvam_speech_to_text import (
- SarvamSpeechToTextNode,
- SarvamSpeechToTextOutput,
-)
-from .sarvam_text_to_speech import (
- SarvamTextToSpeechNode,
- SarvamTextToSpeechOutput,
-)
from .sarvam_translate import SarvamTranslateNode, SarvamTranslateOutput
from .sarvam_transliterate import (
SarvamTransliterateNode,
@@ -35,13 +34,9 @@ from .sarvam_transliterate import (
register_output_schema(SarvamTranslateNode.type, SarvamTranslateOutput)
register_output_schema(SarvamTransliterateNode.type, SarvamTransliterateOutput)
register_output_schema(SarvamDetectLanguageNode.type, SarvamDetectLanguageOutput)
-register_output_schema(SarvamSpeechToTextNode.type, SarvamSpeechToTextOutput)
-register_output_schema(SarvamTextToSpeechNode.type, SarvamTextToSpeechOutput)
__all__ = [
"SarvamDetectLanguageNode",
- "SarvamSpeechToTextNode",
- "SarvamTextToSpeechNode",
"SarvamTranslateNode",
"SarvamTransliterateNode",
]
diff --git a/server/nodes/sarvam/sarvam_speech_to_text.py b/server/nodes/sarvam/sarvam_speech_to_text.py
deleted file mode 100644
index a80370d5..00000000
--- a/server/nodes/sarvam/sarvam_speech_to_text.py
+++ /dev/null
@@ -1,201 +0,0 @@
-"""Sarvam Speech to Text — POST /speech-to-text (multipart transcription)."""
-
-from __future__ import annotations
-
-import base64
-import binascii
-import mimetypes
-from pathlib import Path
-from typing import Any, Dict, List, Literal, Optional, Union
-
-from pydantic import BaseModel, Field
-
-from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue
-
-from ._base import post_multipart, track_sarvam_usage
-from ..model._credentials import SarvamCredential
-
-# The synchronous REST endpoint is documented for clips under 30 seconds;
-# anything longer needs Sarvam's Batch API, which is a different job-based
-# surface and out of scope for this node.
-_MAX_SECONDS = 30
-# Guard against a caller pasting a multi-hundred-MB blob into the param.
-_MAX_BYTES = 24 * 1024 * 1024
-
-
-class SarvamSpeechToTextParams(BaseModel):
- tool_name: str = Field(
- default="sarvam_speech_to_text",
- description="Override name shown to the LLM when used as a tool.",
- )
- tool_description: str = Field(
- default=(
- "Transcribe a short audio clip (under 30 seconds) in any of 23 Indian "
- "languages. Can also translate speech directly into English."
- ),
- description="Override description shown to the LLM when used as a tool.",
- json_schema_extra={"rows": 3},
- )
- # Accepts either a workspace/absolute path OR the upload envelope the
- # file widget emits ({type, data, filename, mimeType}) — the frontend
- # sends whichever the user chose, so the field must tolerate both.
- audio_file: Union[str, Dict[str, Any]] = Field(
- default="",
- description="Path to an audio file, or an uploaded file.",
- json_schema_extra={
- "widget": "file",
- "accept": "audio/*,.wav,.mp3,.m4a,.ogg,.opus,.flac,.aac,.webm,.amr",
- },
- )
- model: Literal["saaras:v3", "saarika:v2.5"] = Field(
- default="saaras:v3",
- description="saaras:v3 covers 23 languages; saarika:v2.5 is the legacy 11-language model.",
- )
- language_code: str = Field(
- default="unknown",
- description="BCP-47 code of the spoken language, or 'unknown' to auto-detect.",
- )
- mode: Literal["transcribe", "translate", "verbatim", "translit", "codemix"] = Field(
- default="transcribe",
- description=(
- "transcribe = native script; translate = English output; verbatim keeps "
- "disfluencies; translit = Roman script; codemix preserves mixed-language input. "
- "Everything except transcribe/translate requires saaras:v3."
- ),
- )
- input_audio_codec: Optional[Literal["pcm_s16le", "pcm_l16", "pcm_raw"]] = Field(
- default=None,
- description="Only needed for headerless PCM; every other format is auto-detected.",
- )
-
- model_config = {"extra": "ignore"}
-
-
-class SarvamSpeechToTextOutput(BaseModel):
- transcript: str = ""
- language_code: Optional[str] = None
- language_probability: Optional[float] = None
- # Present only when Sarvam returns them; the sync endpoint omits
- # diarization entirely (Batch API only).
- timestamps: Optional[Dict[str, Any]] = None
- diarized_transcript: Optional[Dict[str, Any]] = None
- request_id: Optional[str] = None
-
- model_config = {"extra": "allow"}
-
-
-class SarvamSpeechToTextNode(ActionNode):
- type = "sarvamSpeechToText"
- display_name = "Sarvam Speech to Text"
- subtitle = "Transcribe Audio"
- group = ("language", "tool")
- description = "Transcribe or translate short audio clips across 23 Indian languages"
- component_kind = "square"
- tool_name = "sarvam_speech_to_text"
- tool_description = (
- "Transcribe a short audio clip (under 30 seconds) in any of 23 Indian languages. "
- "Pass a file path in `audio_file`. Set mode='translate' to get English text from "
- "non-English speech. Leave language_code='unknown' to auto-detect. Returns the "
- "transcript and the detected language."
- )
- handles = (
- {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},
- {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},
- )
- credentials = (SarvamCredential,)
- annotations = {"destructive": False, "readonly": True, "open_world": True}
- task_queue = TaskQueue.REST_API
- usable_as_tool = True
-
- Params = SarvamSpeechToTextParams
- Output = SarvamSpeechToTextOutput
-
- @Operation("transcribe", cost={"service": "sarvam", "action": "speech_to_text", "count": 1})
- async def transcribe(
- self, ctx: NodeContext, params: SarvamSpeechToTextParams
- ) -> SarvamSpeechToTextOutput:
- if params.mode not in ("transcribe", "translate") and params.model != "saaras:v3":
- raise NodeUserError(
- f"mode='{params.mode}' requires model='saaras:v3'; "
- f"'{params.model}' supports only transcribe and translate."
- )
-
- filename, blob = _read_audio(params.audio_file, ctx.workspace_dir)
- mime = mimetypes.guess_type(filename)[0] or "application/octet-stream"
-
- data = await post_multipart(
- ctx,
- "/speech-to-text",
- files={"file": (filename, blob, mime)},
- data={
- "model": params.model,
- "language_code": params.language_code,
- "mode": params.mode,
- "input_audio_codec": params.input_audio_codec,
- },
- )
-
- # Billed per second of audio; we do not decode the clip to measure
- # it, so charge the documented sync-endpoint ceiling.
- await track_sarvam_usage(ctx, "speech_to_text", _MAX_SECONDS)
- return SarvamSpeechToTextOutput(
- transcript=data.get("transcript", ""),
- language_code=data.get("language_code"),
- language_probability=data.get("language_probability"),
- timestamps=data.get("timestamps"),
- diarized_transcript=data.get("diarized_transcript"),
- request_id=data.get("request_id"),
- )
-
-
-def _read_audio(
- audio_file: Union[str, Dict[str, Any]], workspace_dir: Optional[str]
-) -> tuple[str, bytes]:
- """Resolve the dual-shaped ``audio_file`` param to ``(filename, bytes)``.
-
- The file widget emits ``{"type": "upload", "data": "<base64>", ...}``
- when the user picks a local file, and a bare path string when they type
- one or drag an upstream value in. Relative paths resolve against the
- per-workflow workspace so they line up with ``fileDownloader`` output.
- """
- if isinstance(audio_file, dict):
- if audio_file.get("type") != "upload" or not audio_file.get("data"):
- raise NodeUserError(
- "audio_file is an object but not a file upload. Provide a file "
- "path or re-select the file."
- )
- try:
- blob = base64.b64decode(audio_file["data"], validate=True)
- except (binascii.Error, ValueError) as e:
- raise NodeUserError(f"Uploaded audio is not valid base64: {e}") from e
- name = str(audio_file.get("filename") or "audio.wav")
- else:
- raw = (audio_file or "").strip()
- if not raw:
- raise NodeUserError(
- "audio_file is required — provide a path to an audio file or upload one."
- )
- path = Path(raw)
- if not path.is_absolute() and workspace_dir:
- path = Path(workspace_dir) / path
- if not path.is_file():
- raise NodeUserError(f"Audio file not found: {path}")
- blob = path.read_bytes()
- name = path.name
-
- if not blob:
- raise NodeUserError("Audio file is empty.")
- if len(blob) > _MAX_BYTES:
- raise NodeUserError(
- f"Audio is {len(blob) // (1024 * 1024)} MB. The synchronous endpoint takes "
- f"clips under {_MAX_SECONDS} seconds — trim it, or use Sarvam's Batch API "
- "for long recordings."
- )
- return name, blob
-
-
-__all__: List[str] = [
- "SarvamSpeechToTextNode",
- "SarvamSpeechToTextOutput",
- "SarvamSpeechToTextParams",
-]
diff --git a/server/nodes/sarvam/sarvam_text_to_speech.py b/server/nodes/sarvam/sarvam_text_to_speech.py
deleted file mode 100644
index a26cf82d..00000000
--- a/server/nodes/sarvam/sarvam_text_to_speech.py
+++ /dev/null
@@ -1,284 +0,0 @@
-"""Sarvam Text to Speech — POST /text-to-speech (Bulbul v2/v3)."""
-
-from __future__ import annotations
-
-import base64
-import binascii
-import re
-import uuid
-from pathlib import Path
-from typing import List, Literal, Optional
-
-from pydantic import BaseModel, Field
-
-from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue
-
-from ._base import post_json, track_sarvam_usage
-from ..model._credentials import SarvamCredential
-
-_MAX_TEXT = {"bulbul:v3": 2500, "bulbul:v2": 1500}
-
-# Sarvam returns audio as base64 inside the JSON body. A 2500-character v3
-# request is roughly 12 MB of base64, and node outputs are persisted to the
-# DB, broadcast over the status WebSocket, and — for a tool-exposed node —
-# serialized into an LLM message. Writing to the workspace is therefore the
-# default; inline base64 is opt-in and capped.
-_MAX_INLINE_B64 = 1_000_000
-
-_V3_SPEAKERS = (
- "shubh", "aditya", "ritu", "priya", "neha", "rahul", "pooja", "rohan",
- "simran", "kavya", "amit", "dev", "ishita", "shreya", "ratan", "varun",
- "manan", "sumit", "roopa", "kabir", "aayan", "ashutosh", "advait",
- "anand", "tanya", "tarun", "sunny", "mani", "gokul", "vijay", "shruti",
- "suhani", "mohit", "kavitha", "rehan", "soham", "rupali",
-)
-_V2_SPEAKERS = (
- "anushka", "manisha", "vidya", "arya", "abhilash", "karun", "hitesh",
-)
-
-_CODEC_EXT = {
- "wav": "wav", "mp3": "mp3", "flac": "flac", "aac": "aac",
- "opus": "opus", "linear16": "pcm", "mulaw": "ulaw", "alaw": "alaw",
-}
-
-
-class SarvamTextToSpeechParams(BaseModel):
- tool_name: str = Field(
- default="sarvam_text_to_speech",
- description="Override name shown to the LLM when used as a tool.",
- )
- tool_description: str = Field(
- default=(
- "Synthesize natural speech from text in 11 Indian languages using Sarvam's "
- "Bulbul voices. Writes an audio file and returns its path."
- ),
- description="Override description shown to the LLM when used as a tool.",
- json_schema_extra={"rows": 3},
- )
- text: str = Field(
- ...,
- min_length=1,
- description="Text to speak.",
- json_schema_extra={"rows": 4},
- )
- target_language_code: Literal[
- "bn-IN", "en-IN", "gu-IN", "hi-IN", "kn-IN", "ml-IN",
- "mr-IN", "od-IN", "pa-IN", "ta-IN", "te-IN",
- ] = Field(default="hi-IN", description="Language of the spoken output.")
- model: Literal["bulbul:v3", "bulbul:v2"] = Field(
- default="bulbul:v3",
- description="bulbul:v3 is the current model (37 voices, 2500 chars).",
- )
- speaker: str = Field(
- default="shubh",
- description="Voice name (lowercase). v3 and v2 have separate voice sets.",
- )
- pace: float = Field(
- default=1.0, ge=0.3, le=3.0, description="Speaking rate. v3 allows 0.5-2.0; v2 allows 0.3-3.0."
- )
- speech_sample_rate: Literal[8000, 16000, 22050, 24000, 32000, 44100, 48000] = Field(
- default=24000, description="Output sample rate in Hz."
- )
- output_audio_codec: Literal[
- "wav", "mp3", "linear16", "mulaw", "alaw", "opus", "flac", "aac"
- ] = Field(default="wav", description="Audio container/codec.")
- temperature: Optional[float] = Field(
- default=None,
- ge=0.01,
- le=2.0,
- description="Expressiveness. bulbul:v3 only.",
- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v3"]}}},
- )
- pitch: Optional[float] = Field(
- default=None,
- ge=-0.75,
- le=0.75,
- description="Pitch shift. bulbul:v2 only.",
- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v2"]}}},
- )
- loudness: Optional[float] = Field(
- default=None,
- ge=0.3,
- le=3.0,
- description="Output gain. bulbul:v2 only.",
- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v2"]}}},
- )
- enable_preprocessing: Optional[bool] = Field(
- default=None,
- description="Normalise numbers and abbreviations. bulbul:v2 only (v3 always preprocesses).",
- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v2"]}}},
- )
- return_audio: Literal["file", "base64"] = Field(
- default="file",
- description=(
- "'file' writes the audio into the workflow workspace and returns its path "
- "(recommended). 'base64' returns the bytes inline — only for short clips."
- ),
- )
-
- model_config = {"extra": "ignore"}
-
-
-class SarvamTextToSpeechOutput(BaseModel):
- file_path: Optional[str] = None
- files: List[str] = Field(default_factory=list)
- chunk_count: int = 0
- audio_format: str = "wav"
- audio_base64: Optional[str] = None
- note: Optional[str] = None
- request_id: Optional[str] = None
-
- model_config = {"extra": "allow"}
-
-
-class SarvamTextToSpeechNode(ActionNode):
- type = "sarvamTextToSpeech"
- display_name = "Sarvam Text to Speech"
- subtitle = "Synthesize Speech"
- group = ("language", "tool")
- description = "Generate natural speech in 11 Indian languages with Sarvam Bulbul voices"
- component_kind = "square"
- tool_name = "sarvam_text_to_speech"
- tool_description = (
- "Synthesize natural speech from text in 11 Indian languages using Sarvam's Bulbul "
- "voices. Writes an audio file into the workflow workspace and returns its path in "
- "`file_path` — pass that path to other nodes rather than expecting audio bytes. "
- "Keep `text` under 2500 characters."
- )
- handles = (
- {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},
- {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},
- )
- credentials = (SarvamCredential,)
- annotations = {"destructive": False, "readonly": False, "open_world": True}
- task_queue = TaskQueue.REST_API
- usable_as_tool = True
-
- Params = SarvamTextToSpeechParams
- Output = SarvamTextToSpeechOutput
-
- @Operation("synthesize", cost={"service": "sarvam", "action": "text_to_speech", "count": 1})
- async def synthesize(
- self, ctx: NodeContext, params: SarvamTextToSpeechParams
- ) -> SarvamTextToSpeechOutput:
- is_v3 = params.model == "bulbul:v3"
- cap = _MAX_TEXT[params.model]
- if len(params.text) > cap:
- raise NodeUserError(
- f"text is {len(params.text)} characters; {params.model} accepts at most "
- f"{cap}. Split the text across multiple calls."
- )
-
- speakers = _V3_SPEAKERS if is_v3 else _V2_SPEAKERS
- speaker = params.speaker.strip().lower()
- if speaker not in speakers:
- raise NodeUserError(
- f"'{params.speaker}' is not a {params.model} voice. Available: "
- + ", ".join(speakers)
- )
- if is_v3 and not (0.5 <= params.pace <= 2.0):
- raise NodeUserError(
- f"pace {params.pace} is out of range for bulbul:v3 (0.5-2.0)."
- )
-
- body = {
- "text": params.text,
- "target_language_code": params.target_language_code,
- "model": params.model,
- "speaker": speaker,
- "pace": params.pace,
- "speech_sample_rate": params.speech_sample_rate,
- "output_audio_codec": params.output_audio_codec,
- }
- if is_v3:
- body["temperature"] = params.temperature
- else:
- # v3 rejects these outright, so they are only sent for v2.
- body["pitch"] = params.pitch
- body["loudness"] = params.loudness
- body["enable_preprocessing"] = params.enable_preprocessing
-
- data = await post_json(ctx, "/text-to-speech", body)
-
- chunks = [c for c in (data.get("audios") or []) if c]
- if not chunks:
- raise NodeUserError("Sarvam returned no audio for this request.")
-
- out = SarvamTextToSpeechOutput(
- chunk_count=len(chunks),
- audio_format=params.output_audio_codec,
- request_id=data.get("request_id"),
- )
-
- if params.return_audio == "file":
- out.files = _write_chunks(
- chunks,
- workspace_dir=ctx.workspace_dir,
- node_id=ctx.node_id,
- codec=params.output_audio_codec,
- text=params.text,
- )
- out.file_path = out.files[0]
- if len(chunks) > 1:
- out.note = (
- f"Sarvam split this request into {len(chunks)} audio chunks. "
- "file_path is the first; see `files` for all of them. Each chunk "
- "is a standalone file — concatenating them byte-wise is invalid."
- )
- else:
- first = chunks[0]
- if len(first) > _MAX_INLINE_B64:
- out.note = (
- f"Inline audio suppressed: {len(first)} base64 characters exceeds the "
- f"{_MAX_INLINE_B64} limit. Re-run with return_audio='file'."
- )
- else:
- out.audio_base64 = first
- if len(chunks) > 1:
- out.note = (
- f"audio_base64 is chunk 1 of {len(chunks)}. Use "
- "return_audio='file' to capture every chunk."
- )
-
- # bulbul:v2 is billed at half the v3 character rate.
- await track_sarvam_usage(
- ctx,
- "text_to_speech" if is_v3 else "text_to_speech_v2",
- len(params.text),
- )
- return out
-
-
-def _write_chunks(
- chunks: List[str],
- *,
- workspace_dir: Optional[str],
- node_id: str,
- codec: str,
- text: str,
-) -> List[str]:
- """Decode each base64 chunk to its own file under ``<workspace>/audio``.
-
- One file per chunk: every chunk carries its own container header, so
- concatenating them produces an unplayable file.
- """
- base = Path(workspace_dir) / "audio" if workspace_dir else Path("audio")
- base.mkdir(parents=True, exist_ok=True)
-
- slug = re.sub(r"[^a-z0-9]+", "-", text[:32].lower()).strip("-") or "speech"
- stem = f"{slug}-{node_id[:8]}-{uuid.uuid4().hex[:6]}"
- ext = _CODEC_EXT.get(codec, "wav")
-
- paths: List[str] = []
- for index, chunk in enumerate(chunks, start=1):
- try:
- blob = base64.b64decode(chunk, validate=True)
- except (binascii.Error, ValueError) as e:
- raise NodeUserError(
- f"Sarvam returned audio chunk {index} that is not valid base64: {e}"
- ) from e
- suffix = "" if len(chunks) == 1 else f"-{index}"
- target = base / f"{stem}{suffix}.{ext}"
- target.write_bytes(blob)
- paths.append(str(target))
- return paths
diff --git a/server/nodes/speech/__init__.py b/server/nodes/speech/__init__.py
new file mode 100644
index 00000000..49d75353
--- /dev/null
+++ b/server/nodes/speech/__init__.py
@@ -0,0 +1,57 @@
+"""Speech plugin — provider-abstracted text-to-speech and speech-to-text.
+
+Two nodes, one provider dropdown each. Everything a vendor needs lives in
+this folder: the wire protocol, the two registries, the JSON-backed
+capability lookups, dispatch, and the provider implementations. Nothing
+under ``services/`` knows a speech provider's name.
+
+What this folder deliberately does *not* own, because it is genuinely
+shared:
+
+``services/provider_registry.py``
+ The registry mechanism, shared with ``services/llm``.
+``services/media/``
+ Audio transport and workspace containment — vendor-neutral, and usable
+ by any node that produces or consumes media.
+``server/config/speech_defaults.json``
+ Operator-editable capability data, alongside ``llm_defaults.json`` and
+ ``email_providers.json``.
+
+Layout::
+
+ text_to_speech.py / speech_to_text.py the two nodes
+ _protocol.py requests, results, errors, protocols
+ _registry.py one registry per direction
+ _config.py speech_defaults.json lookups
+ _unifier.py dispatch + error translation
+ _providers/ one module per vendor
+ _credentials.py speech-only credentials
+ _base.py shared node helpers
+ _option_loaders.py dropdown loaders
+"""
+
+from __future__ import annotations
+
+from services.node_output_schemas import register_output_schema
+from services.ws_handler_registry import register_option_loader
+
+from ._option_loaders import load_speech_models, load_speech_voices
+from .speech_to_text import SpeechToTextNode, SpeechToTextOutput
+from .text_to_speech import TextToSpeechNode, TextToSpeechOutput
+
+# Runtime output shapes, so the parameter panel can offer these fields as
+# drag sources for downstream nodes.
+register_output_schema(TextToSpeechNode.type, TextToSpeechOutput)
+register_output_schema(SpeechToTextNode.type, SpeechToTextOutput)
+
+# Dropdown loaders. The names must match the `loadOptionsMethod` strings on
+# the Params fields or `test_node_spec.py` fails.
+register_option_loader("speechModels", load_speech_models)
+register_option_loader("speechVoices", load_speech_voices)
+
+__all__ = [
+ "SpeechToTextNode",
+ "SpeechToTextOutput",
+ "TextToSpeechNode",
+ "TextToSpeechOutput",
+]
diff --git a/server/nodes/speech/_base.py b/server/nodes/speech/_base.py
new file mode 100644
index 00000000..bd919800
--- /dev/null
+++ b/server/nodes/speech/_base.py
@@ -0,0 +1,159 @@
+"""Shared helpers for the two speech nodes.
+
+Credential resolution, audio input handling, and cost attribution -- the
+parts both nodes need and neither should own.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any, Dict, Optional, Sequence, Tuple
+
+from core.logging import get_logger
+from services.media import coerce_file_param, read_media_bytes, resolve_media
+from services.media.inspect import inspect_audio
+from services.plugin import NodeContext, NodeUserError
+
+from . import _config as speech_config
+
+logger = get_logger(__name__)
+
+
+async def provider_api_key(ctx: NodeContext, provider: str) -> str:
+ """Resolve the stored key for ``provider`` through its credential.
+
+ Goes through ``ctx.connection(...)`` rather than reading the auth
+ service directly so a missing key raises the framework's annotated
+ ``PermissionError`` -- which ``BaseNode.execute`` turns into a
+ credential envelope plus a CloudEvents broadcast, so the Credentials
+ modal lights up the right provider.
+ """
+ credential_id = speech_config.credential_id(provider)
+ async with ctx.connection(credential_id) as conn:
+ secrets = await conn.credentials()
+ api_key = str(secrets.get("api_key") or "")
+ if not api_key:
+ raise NodeUserError(
+ f"No API key stored for '{credential_id}'. Add one in the "
+ "Credentials modal."
+ )
+ return api_key
+
+
+def require_provider(provider: str, available: Sequence[str], direction: str) -> str:
+ """Validate a provider selection against what is actually registered."""
+ if provider in available:
+ return provider
+ raise NodeUserError(
+ f"'{provider}' is not a {direction} provider. Available: "
+ f"{', '.join(available)}."
+ )
+
+
+def read_audio_input(
+ value: Any, ctx: NodeContext, *, max_bytes: Optional[int] = None
+) -> Tuple[str, bytes, Optional[Path]]:
+ """Resolve an audio parameter to ``(filename, bytes, path_or_None)``.
+
+ The path comes back when one exists because it is what makes real
+ duration billing possible -- ``inspect_audio`` needs a file. A legacy
+ base64 upload has no path, and rather than inventing a duration for it
+ the node bills nothing. That is the fix for the old Sarvam node, which
+ charged every clip as 30 seconds because it never measured.
+
+ All three input shapes route through ``services.media``, so the
+ traversal that let ``audio_file="../../credentials.db"`` read the
+ credential store is closed here by construction.
+ """
+ kwargs: Dict[str, Any] = {"ctx": ctx}
+ if max_bytes:
+ kwargs["max_bytes"] = max_bytes
+
+ path: Optional[Path] = None
+ if isinstance(value, str) and value.strip():
+ # A path or workspace-relative string: resolve it so we keep the
+ # location for probing, then read through the contained reader.
+ path = resolve_media(value, ctx=ctx)
+ filename, blob = read_media_bytes(value, **kwargs)
+ elif isinstance(value, dict) and value.get("kind") == "audio":
+ from services.media import AudioRef
+
+ ref = AudioRef.model_validate(value)
+ path = resolve_media(ref, ctx=ctx)
+ filename, blob = read_media_bytes(ref, **kwargs)
+ else:
+ filename, blob = coerce_file_param(value, **kwargs)
+
+ return filename, blob, path
+
+
+def measure_seconds(path: Optional[Path], declared_format: str = "") -> Optional[float]:
+ """Real duration for billing, or ``None`` when it cannot be measured.
+
+ Never raises and never guesses: ``inspect_audio`` degrades to an empty
+ probe on an unknown container, and a missing duration means the caller
+ skips per-second attribution rather than inventing a figure.
+ """
+ if path is None:
+ return None
+ return inspect_audio(path, declared_format=declared_format).duration_seconds
+
+
+async def track_usage(
+ ctx: NodeContext,
+ *,
+ provider: str,
+ operation: str,
+ units: Optional[float],
+ unit: str,
+) -> None:
+ """Record an API usage metric. Never raises.
+
+ ``service`` is the provider id rather than a generic ``"speech"`` so
+ per-provider dashboards group correctly, and the unit is whatever that
+ provider actually bills in -- characters, seconds or minutes. Both come
+ from the provider module, because only it knows.
+
+ A provider with no ``operation_map`` entry in ``pricing.json`` yields a
+ zero cost silently, which is why every provider added here needs both
+ an ``api_pricing`` and an ``operation_map`` block.
+ """
+ if units is None or units <= 0:
+ return
+ try:
+ from services.plugin.deps import get_database
+ from services.pricing import get_pricing_service
+
+ cost_data = get_pricing_service().calculate_api_cost(
+ provider, operation, units
+ )
+ await get_database().save_api_usage_metric(
+ {
+ "session_id": ctx.raw.get("session_id", "default"),
+ "node_id": ctx.node_id,
+ "workflow_id": ctx.workflow_id,
+ "service": provider,
+ "operation": cost_data.get("operation", operation),
+ "endpoint": operation,
+ "resource_count": units,
+ "cost": cost_data.get("total_cost", 0.0),
+ }
+ )
+ except Exception as exc:
+ # Cost attribution must never fail a workflow that already did the
+ # paid work.
+ logger.warning(
+ "failed to record speech usage",
+ provider=provider,
+ operation=operation,
+ error=str(exc),
+ )
+
+
+__all__ = [
+ "measure_seconds",
+ "provider_api_key",
+ "read_audio_input",
+ "require_provider",
+ "track_usage",
+]
diff --git a/server/nodes/speech/_config.py b/server/nodes/speech/_config.py
new file mode 100644
index 00000000..7d0a4c84
--- /dev/null
+++ b/server/nodes/speech/_config.py
@@ -0,0 +1,244 @@
+"""Capability lookups backed by ``server/config/speech_defaults.json``.
+
+Same contract as :mod:`services.llm.config`: the JSON is loaded once at
+import into a module-level dict, failure is soft, and every capability
+question is answered from here so no shared code ever branches on a provider
+name.
+
+The one idea worth understanding is :func:`capability`. A capability value in
+the JSON is either a plain scalar (applies to every model) or a mapping of
+model id to value with a ``_default`` fallback. Callers never care which was
+written -- they ask for ``(provider, direction, key, model)`` and get a value.
+That is what lets OpenAI declare "``response_formats`` is ``json|text`` except
+on whisper-1 where it is five formats" without a line of Python.
+
+Lookup order inside a mapping is exact match, then longest prefix match, then
+``_default``. Prefix matching earns its keep on dated snapshots -- a key of
+``gpt-4o-mini-transcribe`` also answers for
+``gpt-4o-mini-transcribe-2025-12-15`` without the JSON tracking every release.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from core.logging import get_logger
+
+logger = get_logger(__name__)
+
+TTS = "tts"
+STT = "stt"
+
+_CONFIG_PATH = Path(__file__).parent.parent.parent / "config" / "speech_defaults.json"
+
+
+def _load_speech_defaults() -> Dict[str, Any]:
+ try:
+ with open(_CONFIG_PATH, encoding="utf-8") as handle:
+ return json.load(handle)
+ except Exception as exc:
+ # Soft-fail, matching services/llm/config.py: a malformed config
+ # degrades capability lookups to their permissive defaults rather
+ # than preventing the process from booting.
+ logger.warning("Could not load speech_defaults.json", error=str(exc))
+ return {"providers": {}}
+
+
+SPEECH_DEFAULTS: Dict[str, Any] = _load_speech_defaults()
+
+
+def reload_defaults() -> None:
+ """Re-read the JSON from disk. Mutates in place.
+
+ In-place rather than rebinding because callers may hold a reference to
+ the dict (the unifier does), and rebinding would leave them on the stale
+ object -- a trap ``services/llm/config.reload_defaults`` still has.
+ """
+ fresh = _load_speech_defaults()
+ SPEECH_DEFAULTS.clear()
+ SPEECH_DEFAULTS.update(fresh)
+
+
+# ---------------------------------------------------------------------------
+# Raw block access
+# ---------------------------------------------------------------------------
+
+
+def provider_block(provider: str) -> Dict[str, Any]:
+ """The whole ``providers.<name>`` block, or ``{}``."""
+ block = SPEECH_DEFAULTS.get("providers", {}).get(provider)
+ return block if isinstance(block, dict) else {}
+
+
+def direction_block(provider: str, direction: str) -> Dict[str, Any]:
+ """The ``providers.<name>.<tts|stt>`` block, or ``{}``."""
+ block = provider_block(provider).get(direction)
+ return block if isinstance(block, dict) else {}
+
+
+def supports_direction(provider: str, direction: str) -> bool:
+ """Whether the JSON declares this provider for this direction.
+
+ Advisory only -- registry membership is the authority (a provider that
+ never calls ``register_tts_provider`` cannot be selected regardless of
+ what the JSON says). Useful for config-consistency tests.
+ """
+ return bool(direction_block(provider, direction))
+
+
+def base_url(provider: str) -> str:
+ return str(provider_block(provider).get("base_url") or "")
+
+
+def credential_id(provider: str) -> str:
+ """Credential id string for this provider.
+
+ A *string*, never a ``Credential`` class: the classes live under
+ ``nodes/`` and importing one here would invert the layering and break
+ ``test_plugin_self_containment.py``. Falls back to the provider name,
+ which is the convention every current provider follows.
+ """
+ return str(provider_block(provider).get("credential_id") or provider)
+
+
+# ---------------------------------------------------------------------------
+# Capability resolution
+# ---------------------------------------------------------------------------
+
+
+def capability(
+ provider: str,
+ direction: str,
+ key: str,
+ *,
+ model: Optional[str] = None,
+ default: Any = None,
+) -> Any:
+ """Resolve one capability, honouring per-model overrides.
+
+ Returns ``default`` when the key is absent entirely. A declared ``null``
+ is returned as ``None`` and is meaningful -- Deepgram's
+ ``max_upload_bytes`` is explicitly null because no cap is documented,
+ which is different from "not configured".
+ """
+ block = direction_block(provider, direction)
+ if key not in block:
+ return default
+
+ value = block[key]
+ if not isinstance(value, dict):
+ return value
+
+ # A mapping means per-model overrides. Without a model we can still
+ # answer from _default, which is what the panel does before the user
+ # has picked one.
+ if model:
+ if model in value:
+ return value[model]
+ prefix_matches = [k for k in value if k != "_default" and model.startswith(k)]
+ if prefix_matches:
+ return value[max(prefix_matches, key=len)]
+ if "_default" in value:
+ return value["_default"]
+ return default
+
+
+def default_model(provider: str, direction: str) -> str:
+ return str(capability(provider, direction, "default_model", default="") or "")
+
+
+def models(provider: str, direction: str) -> List[str]:
+ value = capability(provider, direction, "models", default=[])
+ return list(value or [])
+
+
+def voices(provider: str, *, model: Optional[str] = None) -> List[str]:
+ """Static voice ids from config.
+
+ Only a fallback. Providers that expose a live voices endpoint
+ (ElevenLabs) answer the dropdown from the API instead, because their
+ catalogue is per-account and changes without a release.
+ """
+ value = capability(provider, TTS, "voices", model=model, default=[])
+ return list(value or [])
+
+
+def default_voice(provider: str) -> str:
+ return str(capability(provider, TTS, "default_voice", default="") or "")
+
+
+def output_formats(provider: str, direction: str = TTS) -> List[str]:
+ return list(capability(provider, direction, "output_formats", default=[]) or [])
+
+
+def billed_unit(provider: str, direction: str) -> str:
+ """``characters`` / ``seconds`` / ``minutes`` -- for cost attribution."""
+ return str(capability(provider, direction, "billed_unit", default="") or "")
+
+
+def endpoint(provider: str, direction: str, key: str = "endpoint") -> str:
+ return str(capability(provider, direction, key, default="") or "")
+
+
+def max_input_chars(provider: str, model: Optional[str] = None) -> Optional[int]:
+ value = capability(provider, TTS, "max_input_chars", model=model)
+ return int(value) if isinstance(value, (int, float)) else None
+
+
+def speed_range(
+ provider: str, model: Optional[str] = None
+) -> Optional[tuple[float, float]]:
+ value = capability(provider, TTS, "speed_range", model=model)
+ if isinstance(value, (list, tuple)) and len(value) == 2:
+ return float(value[0]), float(value[1])
+ return None
+
+
+def supports(
+ provider: str, direction: str, key: str, *, model: Optional[str] = None
+) -> bool:
+ """Boolean capability probe.
+
+ Defaults to ``True`` -- permissive, matching ``supports_model_listing``
+ in the LLM layer. A provider opts *out* explicitly; forgetting to
+ declare a flag never silently disables a working feature.
+ """
+ return bool(capability(provider, direction, key, model=model, default=True))
+
+
+def response_formats(provider: str, model: Optional[str] = None) -> List[str]:
+ """Allowed ``response_format`` values for a transcription model.
+
+ Model-gated on OpenAI and a 400 when violated, which is why this is
+ config rather than something the node discovers at runtime.
+ """
+ return list(
+ capability(provider, STT, "response_formats", model=model, default=[]) or []
+ )
+
+
+__all__ = [
+ "SPEECH_DEFAULTS",
+ "STT",
+ "TTS",
+ "base_url",
+ "billed_unit",
+ "capability",
+ "credential_id",
+ "default_model",
+ "default_voice",
+ "direction_block",
+ "endpoint",
+ "max_input_chars",
+ "models",
+ "output_formats",
+ "provider_block",
+ "reload_defaults",
+ "response_formats",
+ "speed_range",
+ "supports",
+ "supports_direction",
+ "voices",
+]
diff --git a/server/nodes/speech/_credentials.py b/server/nodes/speech/_credentials.py
new file mode 100644
index 00000000..984b8e1f
--- /dev/null
+++ b/server/nodes/speech/_credentials.py
@@ -0,0 +1,63 @@
+"""Credentials for speech-only providers.
+
+Only two live here. OpenAI, Groq and Sarvam already have credentials under
+``nodes/model/_credentials.py`` because the same stored key authenticates
+their chat surfaces, and a second class for the same ``id`` would collide in
+``CREDENTIAL_REGISTRY``. The speech nodes import those rather than
+redeclaring them -- the same cross-plugin import ``nodes/sarvam/`` already
+does.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from services.plugin.credential import ApiKeyCredential
+
+
+class ElevenLabsCredential(ApiKeyCredential):
+ """ElevenLabs API key.
+
+ Sent as a bare ``xi-api-key`` header with no scheme keyword;
+ ``Authorization: Bearer`` is rejected. The probe lists one voice, which
+ is the cheapest authenticated call the API offers.
+ """
+
+ id = "elevenlabs"
+ display_name = "ElevenLabs"
+ category = "AI"
+ key_name = "xi-api-key"
+ key_location = "header"
+ docs_url = "https://elevenlabs.io/app/settings/api-keys"
+ probe_url = "https://api.elevenlabs.io/v2/voices"
+ probe_params = {"page_size": 1}
+
+
+class DeepgramCredential(ApiKeyCredential):
+ """Deepgram API key.
+
+ Deepgram's scheme keyword is ``Token``, not ``Bearer``, which none of
+ the three built-in ``key_location`` modes can express: ``header`` would
+ send the raw key with no keyword and ``bearer`` would send the wrong
+ one. Hence the :meth:`inject` override -- and it is load-bearing for the
+ declarative probe too, which builds its request through the same method.
+ """
+
+ id = "deepgram"
+ display_name = "Deepgram"
+ category = "AI"
+ key_name = "Authorization"
+ key_location = "header"
+ docs_url = "https://console.deepgram.com"
+ probe_url = "https://api.deepgram.com/v1/projects"
+
+ @classmethod
+ def inject(
+ cls, secrets: Dict[str, Any], request: Dict[str, Any]
+ ) -> Dict[str, Any]:
+ headers = dict(request.get("headers") or {})
+ headers["Authorization"] = f"Token {secrets.get('api_key', '')}"
+ return {**request, "headers": headers}
+
+
+__all__ = ["DeepgramCredential", "ElevenLabsCredential"]
diff --git a/server/nodes/speech/_option_loaders.py b/server/nodes/speech/_option_loaders.py
new file mode 100644
index 00000000..4d922024
--- /dev/null
+++ b/server/nodes/speech/_option_loaders.py
@@ -0,0 +1,101 @@
+"""Dynamic dropdown loaders for the speech nodes.
+
+Both loaders read ``provider`` out of the parameter dict they are handed
+rather than declaring ``loadOptionsDependsOn: ["provider"]``. That attribute
+is lifted by the client adapter but has no usages anywhere in the codebase,
+so it is untested machinery; the parameter dict, by contrast, is how every
+existing loader in the repo works and it re-fires when parameters change.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List
+
+from core.logging import get_logger
+
+from . import _config as speech_config
+from ._registry import stt_providers, tts_providers
+
+logger = get_logger(__name__)
+
+
+def _direction_for(params: Dict[str, Any]) -> str:
+ """Infer direction from which node asked.
+
+ The two nodes have distinct model field names, which doubles as the
+ signal for which direction's catalogue to serve.
+ """
+ if "stt_model" in params or params.get("node_type") == "speechToText":
+ return speech_config.STT
+ return speech_config.TTS
+
+
+def _selected_provider(params: Dict[str, Any], direction: str) -> str:
+ provider = str(params.get("provider") or "")
+ available = tts_providers() if direction == speech_config.TTS else stt_providers()
+ if provider in available:
+ return provider
+ return available[0] if available else ""
+
+
+async def load_speech_models(params: Dict[str, Any]) -> List[Dict[str, Any]]:
+ """Model ids for the selected provider and direction."""
+ direction = _direction_for(params)
+ provider = _selected_provider(params, direction)
+ if not provider:
+ return []
+
+ default = speech_config.default_model(provider, direction)
+ options: List[Dict[str, Any]] = []
+ for model in speech_config.models(provider, direction):
+ option: Dict[str, Any] = {"value": model, "label": model}
+ if model == default:
+ option["description"] = "Provider default"
+ options.append(option)
+ return options
+
+
+async def load_speech_voices(params: Dict[str, Any]) -> List[Dict[str, Any]]:
+ """Voices for the selected synthesis provider.
+
+ Providers with a live catalogue (ElevenLabs) are queried, because their
+ voice list is per-account and changes without a release. Everyone else
+ answers from config. A live lookup that fails degrades to the configured
+ list rather than leaving the user with an empty dropdown and no way to
+ proceed.
+ """
+ provider = _selected_provider(params, speech_config.TTS)
+ if not provider:
+ return []
+
+ model = str(params.get("tts_model") or "")
+ static = [
+ {"value": voice, "label": voice.replace("_", " ").title()}
+ for voice in speech_config.voices(provider, model=model or None)
+ ]
+
+ if not speech_config.capability(provider, speech_config.TTS, "voices_endpoint"):
+ return static
+
+ try:
+ from services.plugin.deps import get_auth_service
+
+ from . import _unifier
+
+ api_key = await get_auth_service().get_api_key(
+ speech_config.credential_id(provider)
+ )
+ if not api_key:
+ return static
+ voices = await _unifier.list_voices(provider=provider, api_key=api_key)
+ return [voice.as_option() for voice in voices] or static
+ except Exception as exc:
+ logger.warning(
+ "live voice lookup failed; falling back to the configured list",
+ provider=provider,
+ error=str(exc),
+ )
+ return static
+
+
+__all__ = ["load_speech_models", "load_speech_voices"]
diff --git a/server/nodes/speech/_protocol.py b/server/nodes/speech/_protocol.py
new file mode 100644
index 00000000..bd0a0aa9
--- /dev/null
+++ b/server/nodes/speech/_protocol.py
@@ -0,0 +1,444 @@
+"""Provider-neutral speech types — the contract every provider implements.
+
+Mirrors :mod:`services.llm.protocol`: stdlib dataclasses only, no Pydantic,
+no SDK objects. These values cross the Temporal boundary and land in node
+results, so everything here must stay JSON-safe.
+
+The one place this deliberately diverges from the LLM layer is the error
+taxonomy. ``LLMError`` carries ``CONTEXT_LENGTH``, which is meaningless for
+speech, and speech has failure modes the LLM layer has never seen
+(``AUDIO_TOO_LONG``, ``UNSUPPORTED_FORMAT``). Sharing one enum would mean
+half the members raising "can't happen here" questions on both sides, so
+each layer owns its own. The HTTP status ladder underneath is similar by
+coincidence of HTTP, not by shared domain.
+
+Audio bytes appear in exactly two places here — :class:`AudioPayload` on the
+way out of a TTS provider, and :attr:`SttRequest.audio` on the way in. Both
+are consumed inside the node before it returns; what leaves the node is an
+``AudioRef`` (see :mod:`services.media.refs`). Nothing in this module is
+allowed into a node ``Output`` model.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any, Dict, List, Mapping, Optional, Protocol, runtime_checkable
+
+
+# ---------------------------------------------------------------------------
+# Requests
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class TtsRequest:
+ """One synthesis call, in provider-neutral terms.
+
+ ``provider_options`` is the escape hatch for vendor-specific keys that
+ do not generalise (ElevenLabs' ``stability``, Sarvam's ``pitch``). It is
+ passed through untouched, which keeps the common surface small without
+ making any provider's full capability unreachable.
+ """
+
+ text: str
+ model: str = ""
+ voice: str = ""
+ language: str = ""
+ speed: Optional[float] = None
+ output_format: str = ""
+ sample_rate: Optional[int] = None
+ provider_options: Dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class SttRequest:
+ """One transcription call.
+
+ ``audio`` is raw bytes rather than a path or a file handle: providers
+ disagree on transport (multipart vs. raw body), and a handle cannot be
+ replayed by ``Connection``'s auth-retry. The caller has already applied
+ containment via ``services.media.coerce_file_param``.
+ """
+
+ audio: bytes
+ filename: str = "audio.wav"
+ mime_type: str = "application/octet-stream"
+ model: str = ""
+ language: str = ""
+ prompt: str = ""
+ translate: bool = False
+ diarize: bool = False
+ timestamps: bool = False
+ provider_options: Dict[str, Any] = field(default_factory=dict)
+
+
+# ---------------------------------------------------------------------------
+# Results
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class AudioPayload:
+ """One synthesized audio blob.
+
+ Providers return a *list* of these. Most emit exactly one; Sarvam splits
+ long input into several standalone clips, and those are not
+ byte-concatenatable, so the plural is the honest shape rather than a
+ convenience.
+ """
+
+ data: bytes
+ format: str = ""
+ mime_type: str = ""
+ sample_rate: Optional[int] = None
+ channels: Optional[int] = None
+
+
+@dataclass
+class TtsResult:
+ audio: List[AudioPayload] = field(default_factory=list)
+ model: str = ""
+ voice: str = ""
+ request_id: Optional[str] = None
+ # Billing units differ per provider (characters / seconds / minutes), so
+ # the provider that made the call reports both the count and its unit.
+ # Central code never guesses.
+ billed_units: Optional[float] = None
+ billed_unit: str = ""
+
+
+@dataclass
+class TranscriptWord:
+ word: str
+ start: Optional[float] = None
+ end: Optional[float] = None
+ speaker: Optional[str] = None
+ confidence: Optional[float] = None
+
+
+@dataclass
+class SttResult:
+ text: str = ""
+ language: Optional[str] = None
+ language_confidence: Optional[float] = None
+ duration_seconds: Optional[float] = None
+ words: List[TranscriptWord] = field(default_factory=list)
+ segments: List[Dict[str, Any]] = field(default_factory=list)
+ request_id: Optional[str] = None
+ billed_units: Optional[float] = None
+ billed_unit: str = ""
+
+
+@dataclass
+class Voice:
+ """One selectable voice, for the node's dropdown loader."""
+
+ id: str
+ name: str = ""
+ description: str = ""
+ language: str = ""
+ preview_url: str = ""
+
+ def as_option(self) -> Dict[str, str]:
+ """Shape the ``loadOptionsMethod`` contract expects."""
+ option = {"value": self.id, "label": self.name or self.id}
+ if self.description:
+ option["description"] = self.description
+ return option
+
+
+# ---------------------------------------------------------------------------
+# Errors
+# ---------------------------------------------------------------------------
+
+
+class SpeechErrorCategory(str, Enum):
+ """Stable categories used by retry and user-error policies."""
+
+ AUTHENTICATION = "authentication"
+ PERMISSION = "permission"
+ RATE_LIMIT = "rate_limit"
+ INVALID_REQUEST = "invalid_request"
+ NOT_FOUND = "not_found"
+ AUDIO_TOO_LONG = "audio_too_long"
+ UNSUPPORTED_FORMAT = "unsupported_format"
+ TIMEOUT = "timeout"
+ CONNECTION = "connection"
+ SERVER = "server"
+ UNKNOWN = "unknown"
+
+
+_RETRYABLE = {
+ SpeechErrorCategory.RATE_LIMIT,
+ SpeechErrorCategory.TIMEOUT,
+ SpeechErrorCategory.CONNECTION,
+ SpeechErrorCategory.SERVER,
+}
+
+_PROVIDER_NAMES = {
+ "openai": "OpenAI",
+ "groq": "Groq",
+ "elevenlabs": "ElevenLabs",
+ "deepgram": "Deepgram",
+ "sarvam": "Sarvam AI",
+}
+
+
+@dataclass
+class SpeechError(Exception):
+ """Provider-independent structured speech failure."""
+
+ message: str
+ provider: str
+ category: SpeechErrorCategory = SpeechErrorCategory.UNKNOWN
+ retryable: bool = False
+ status_code: Optional[int] = None
+ provider_code: Optional[str] = None
+ request_id: Optional[str] = None
+ retry_after: Optional[float] = None
+ retry_after_raw: Optional[str] = None
+
+ def __post_init__(self) -> None:
+ Exception.__init__(self, self.message)
+
+ @property
+ def user_message(self) -> str:
+ """A category-based message that is safe for public surfaces.
+
+ ``message`` retains the original provider text for operator
+ diagnostics and exception chaining. Provider bodies routinely echo
+ request fragments and internal endpoint URLs, so execution
+ boundaries must expose this property instead.
+ """
+ known = str(self.provider or "").strip().lower() in _PROVIDER_NAMES
+ provider = _PROVIDER_NAMES.get(
+ str(self.provider or "").strip().lower(), "The speech provider"
+ )
+ provider_object = provider if known else "the speech provider"
+ category = (
+ self.category.value
+ if isinstance(self.category, SpeechErrorCategory)
+ else str(self.category or SpeechErrorCategory.UNKNOWN.value)
+ )
+ messages = {
+ SpeechErrorCategory.AUTHENTICATION.value: (
+ f"{provider} authentication failed. Check the configured API key."
+ ),
+ SpeechErrorCategory.PERMISSION.value: (
+ f"{provider} denied this request. Check account, plan tier and "
+ "model access."
+ ),
+ SpeechErrorCategory.RATE_LIMIT.value: (
+ f"{provider} is rate-limiting requests. Retry after a short delay."
+ ),
+ SpeechErrorCategory.INVALID_REQUEST.value: (
+ f"{provider} rejected the request configuration. Check the "
+ "selected model, voice and output format."
+ ),
+ SpeechErrorCategory.NOT_FOUND.value: (
+ f"The configured {provider_object} model, voice or endpoint was "
+ "not found."
+ ),
+ SpeechErrorCategory.AUDIO_TOO_LONG.value: (
+ f"The audio is longer or larger than {provider_object} accepts "
+ "on this endpoint. Trim or split the clip."
+ ),
+ SpeechErrorCategory.UNSUPPORTED_FORMAT.value: (
+ f"{provider} does not accept this audio format. Convert the "
+ "file and try again."
+ ),
+ SpeechErrorCategory.TIMEOUT.value: (
+ f"The request to {provider_object} timed out."
+ ),
+ SpeechErrorCategory.CONNECTION.value: (
+ f"Could not connect to {provider_object}."
+ ),
+ SpeechErrorCategory.SERVER.value: (
+ f"{provider} is temporarily unavailable."
+ ),
+ SpeechErrorCategory.UNKNOWN.value: f"{provider} request failed.",
+ }
+ return messages.get(category, messages[SpeechErrorCategory.UNKNOWN.value])
+
+ @classmethod
+ def from_exception(cls, provider: str, exc: BaseException) -> "SpeechError":
+ """Normalize an httpx / SDK exception into a structured error.
+
+ Duck-typed rather than isinstance-based so a provider may raise its
+ own SDK error without this module importing that SDK.
+ """
+ response = getattr(exc, "response", None)
+ status = _optional_int(
+ getattr(exc, "status_code", None)
+ or getattr(response, "status_code", None)
+ )
+ body_text = _response_text(response)
+ code = _provider_code(response, body_text)
+ request_id = (
+ getattr(exc, "request_id", None)
+ or _header(response, "x-request-id")
+ or _header(response, "request-id")
+ or _header(response, "dg-request-id")
+ )
+ retry_after_raw = _header(response, "retry-after")
+ category = classify(exc, status, body_text)
+ return cls(
+ message=_detail(exc, body_text),
+ provider=provider,
+ category=category,
+ retryable=category in _RETRYABLE,
+ status_code=status,
+ provider_code=str(code) if code is not None else None,
+ request_id=str(request_id) if request_id is not None else None,
+ retry_after=_optional_float(retry_after_raw),
+ retry_after_raw=(
+ str(retry_after_raw) if retry_after_raw is not None else None
+ ),
+ )
+
+
+def classify(
+ exc: BaseException, status: Optional[int], body_text: str = ""
+) -> SpeechErrorCategory:
+ """Map an exception + HTTP status onto a category.
+
+ Status wins where it is unambiguous; name and body substrings resolve
+ the rest. 413 and 415 are the two speech-specific rungs — every provider
+ in the v1 set uses them for oversize and unsupported-codec respectively.
+ """
+ name = type(exc).__name__.lower()
+ haystack = f"{exc} {body_text}".lower()
+
+ if status == 401 or "authentication" in name or "api key" in haystack:
+ return SpeechErrorCategory.AUTHENTICATION
+ if status == 403 or "permission" in name:
+ return SpeechErrorCategory.PERMISSION
+ if status == 429 or "ratelimit" in name or "rate limit" in haystack:
+ return SpeechErrorCategory.RATE_LIMIT
+ if status == 404 or "notfound" in name:
+ return SpeechErrorCategory.NOT_FOUND
+ if status == 413 or "too large" in haystack or "too long" in haystack:
+ return SpeechErrorCategory.AUDIO_TOO_LONG
+ if status == 415 or "unsupported" in haystack or "invalid audio" in haystack:
+ return SpeechErrorCategory.UNSUPPORTED_FORMAT
+ if status in {400, 409, 422} or "badrequest" in name:
+ return SpeechErrorCategory.INVALID_REQUEST
+ if status == 408 or "timeout" in name:
+ return SpeechErrorCategory.TIMEOUT
+ if "connect" in name:
+ return SpeechErrorCategory.CONNECTION
+ if status is not None and status >= 500:
+ return SpeechErrorCategory.SERVER
+ return SpeechErrorCategory.UNKNOWN
+
+
+# ---------------------------------------------------------------------------
+# Small helpers (duck-typed; never import a provider SDK)
+# ---------------------------------------------------------------------------
+
+
+def _optional_int(value: Any) -> Optional[int]:
+ try:
+ return int(value) if value is not None else None
+ except (TypeError, ValueError, OverflowError):
+ return None
+
+
+def _optional_float(value: Any) -> Optional[float]:
+ try:
+ return float(value) if value is not None else None
+ except (TypeError, ValueError, OverflowError):
+ return None
+
+
+def _header(response: Any, name: str) -> Optional[str]:
+ headers = getattr(response, "headers", None)
+ if headers is None:
+ return None
+ try:
+ return headers.get(name)
+ except (AttributeError, TypeError):
+ return None
+
+
+def _response_text(response: Any) -> str:
+ """Best-effort body text, capped. Never raises."""
+ if response is None:
+ return ""
+ try:
+ return str(getattr(response, "text", "") or "")[:600]
+ except Exception:
+ return ""
+
+
+def _provider_code(response: Any, body_text: str) -> Optional[str]:
+ """Pull a vendor error code out of a JSON body if one is there."""
+ if response is None:
+ return None
+ try:
+ payload = response.json()
+ except Exception:
+ return None
+ if not isinstance(payload, Mapping):
+ return None
+ error = payload.get("error")
+ if isinstance(error, Mapping):
+ return error.get("code") or error.get("type")
+ return payload.get("code") or payload.get("err_code")
+
+
+def _detail(exc: BaseException, body_text: str) -> str:
+ """Operator-facing message: exception text plus body when it adds signal."""
+ base = str(exc)
+ if body_text and body_text not in base:
+ return f"{base} :: {body_text}"
+ return base
+
+
+# ---------------------------------------------------------------------------
+# Provider protocols (structural typing)
+# ---------------------------------------------------------------------------
+#
+# Two protocols, not one. AssemblyAI is transcription-only and Cartesia is
+# synthesis-only; a single protocol would force dead methods that raise at
+# runtime *after* the provider has already appeared in a dropdown. With one
+# registry per direction, capability is registry membership and there is no
+# ``supports_tts`` flag to keep in sync with reality.
+#
+# ``aclose`` is deliberately absent from both. The unifier probes for it with
+# ``getattr`` — every provider here happens to hold an httpx client and
+# implement it, but keeping it out of the protocol means a stateless provider
+# never has to declare an empty one.
+
+
+@runtime_checkable
+class TtsProvider(Protocol):
+ provider_name: str
+
+ async def synthesize(self, req: TtsRequest) -> TtsResult: ...
+
+ async def list_voices(self) -> List[Voice]: ...
+
+
+@runtime_checkable
+class SttProvider(Protocol):
+ provider_name: str
+
+ async def transcribe(self, req: SttRequest) -> SttResult: ...
+
+
+__all__ = [
+ "AudioPayload",
+ "SpeechError",
+ "SpeechErrorCategory",
+ "SttProvider",
+ "SttRequest",
+ "SttResult",
+ "TranscriptWord",
+ "TtsProvider",
+ "TtsRequest",
+ "TtsResult",
+ "Voice",
+ "classify",
+]
diff --git a/server/nodes/speech/_providers/__init__.py b/server/nodes/speech/_providers/__init__.py
new file mode 100644
index 00000000..429e7b18
--- /dev/null
+++ b/server/nodes/speech/_providers/__init__.py
@@ -0,0 +1,35 @@
+"""Speech provider plugins — importing this package registers them all.
+
+Adding a provider:
+
+1. Add a ``providers.<name>`` block to ``server/config/speech_defaults.json``
+ with at least ``base_url`` and a ``tts`` and/or ``stt`` sub-block. Which
+ sub-blocks exist declares which directions it serves.
+2. Write ``<name>.py`` here implementing the matching protocol(s) from
+ :mod:`nodes.speech._protocol`, subclassing
+ :class:`nodes.speech._providers._http.HttpSpeechProvider` if it is a
+ plain HTTP API.
+3. Call ``register_tts_provider`` / ``register_stt_provider`` at the bottom
+ of that module.
+4. Add the side-effect import below.
+5. Add the credential id to ``server/config/credential_providers.json`` and a
+ ``Credential`` subclass under ``nodes/speech/_credentials.py``.
+
+If the provider speaks OpenAI's ``/audio/*`` wire format, steps 2-4 collapse
+into adding its name to ``_COMPAT_STT_PROVIDERS`` in ``_openai_compat.py``.
+
+**Never import a provider SDK at module level.** ``sdk_exception_refs`` are
+lazy ``"module:ClassName"`` strings precisely so registration stays free;
+``tests/speech/test_lazy_sdk_imports.py`` enforces it from a clean
+interpreter.
+"""
+
+from __future__ import annotations
+
+# Side-effect imports — each module registers into one or both registries at
+# the bottom of the file. Order is documentation only; Python's module cache
+# makes duplicate registration impossible.
+from . import _openai_compat # noqa: F401
+from . import elevenlabs # noqa: F401
+from . import deepgram # noqa: F401
+from . import sarvam # noqa: F401
diff --git a/server/nodes/speech/_providers/_http.py b/server/nodes/speech/_providers/_http.py
new file mode 100644
index 00000000..3ff69322
--- /dev/null
+++ b/server/nodes/speech/_providers/_http.py
@@ -0,0 +1,168 @@
+"""Shared httpx plumbing for speech providers.
+
+None of the v1 speech providers ship a Python SDK worth taking a dependency
+on, so every one of them is a plain HTTP client. This base holds the parts
+that are genuinely identical -- lazy client construction, timeouts, status
+handling -- and leaves the parts that are not to subclasses.
+
+Authentication is the interesting one. All four providers use a different
+scheme, and none of them is plain Bearer across the board:
+
+=========== ==========================================
+openai/groq ``Authorization: Bearer <key>``
+elevenlabs ``xi-api-key: <key>`` (no scheme)
+deepgram ``Authorization: Token <key>`` (not Bearer)
+sarvam ``api-subscription-key: <key>``
+=========== ==========================================
+
+So :meth:`HttpSpeechProvider.auth_headers` is the single override point, and
+nothing above this layer ever has to know.
+
+Timeouts are generous by speech standards: synthesis of a few thousand
+characters routinely takes tens of seconds, and transcription of a long clip
+longer still. The default here is well under the Temporal activity
+``start_to_close`` so a hung provider surfaces as a provider error rather
+than an activity timeout.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, Optional
+
+import httpx
+
+from core.logging import get_logger
+from .. import _config as speech_config
+
+logger = get_logger(__name__)
+
+DEFAULT_TIMEOUT_SECONDS = 180.0
+
+
+class HttpSpeechProvider:
+ """Base for HTTP-backed speech providers.
+
+ Subclasses set :attr:`provider_name` and implement whichever of
+ ``synthesize`` / ``transcribe`` / ``list_voices`` their direction
+ requires. Structural typing means there is no ABC to satisfy -- the
+ Protocols in :mod:`nodes.speech._protocol` check shape, not ancestry.
+ """
+
+ provider_name: str = ""
+
+ def __init__(
+ self,
+ api_key: str,
+ *,
+ base_url: str = "",
+ provider_name: Optional[str] = None,
+ timeout: float = DEFAULT_TIMEOUT_SECONDS,
+ ):
+ # provider_name may arrive via client_kwargs so one class can serve
+ # several registry entries (openai and groq share a transcription
+ # implementation and differ only in base URL and model list).
+ self.provider_name = provider_name or type(self).provider_name
+ self.api_key = api_key
+ self._base_url = (
+ base_url or speech_config.base_url(self.provider_name)
+ ).rstrip("/")
+ self._timeout = timeout
+ self._client: Optional[httpx.AsyncClient] = None
+
+ # ------------------------------------------------------------------
+ # transport
+ # ------------------------------------------------------------------
+
+ def auth_headers(self) -> Dict[str, str]:
+ """Headers that authenticate a request. Override per provider."""
+ return {"Authorization": f"Bearer {self.api_key}"}
+
+ @property
+ def client(self) -> httpx.AsyncClient:
+ """Lazily-built client, reused for this provider's lifetime.
+
+ Built on first use rather than in ``__init__`` so constructing a
+ provider (which the unifier does inside a lock) never touches the
+ network stack.
+ """
+ if self._client is None:
+ self._client = httpx.AsyncClient(
+ base_url=self._base_url,
+ timeout=self._timeout,
+ headers=self.auth_headers(),
+ )
+ return self._client
+
+ async def request(
+ self,
+ method: str,
+ path: str,
+ *,
+ json_body: Optional[Dict[str, Any]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ data: Optional[Dict[str, Any]] = None,
+ files: Optional[Any] = None,
+ content: Optional[bytes] = None,
+ headers: Optional[Dict[str, str]] = None,
+ ) -> httpx.Response:
+ """Issue one request and raise ``httpx.HTTPStatusError`` on 4xx/5xx.
+
+ ``params`` is not an afterthought: ElevenLabs takes ``output_format``
+ and Deepgram takes its entire option set as query string. Passing
+ those in a JSON body is accepted and silently ignored, which is far
+ worse than an error, so the distinction is explicit at every call
+ site.
+ """
+ response = await self.client.request(
+ method,
+ path,
+ json=json_body,
+ params=params,
+ data=data,
+ files=files,
+ content=content,
+ headers=headers,
+ )
+ response.raise_for_status()
+ return response
+
+ async def aclose(self) -> None:
+ """Close the underlying client. Probed via ``getattr`` by the unifier."""
+ if self._client is not None:
+ await self._client.aclose()
+ self._client = None
+
+
+def drop_none(payload: Dict[str, Any]) -> Dict[str, Any]:
+ """Strip ``None`` values from a body or query dict.
+
+ Several of these APIs reject an explicit null where they would happily
+ accept an absent key -- Sarvam 422s, Deepgram treats the literal string
+ "None" as an option value. Building the full dict and filtering once is
+ less error-prone than conditional insertion at a dozen call sites.
+ """
+ return {k: v for k, v in payload.items() if v is not None}
+
+
+def query_flags(payload: Dict[str, Any]) -> Dict[str, Any]:
+ """Render a query dict for APIs that want lowercase boolean literals.
+
+ httpx encodes ``True`` as ``"true"`` already, but only for actual bools;
+ this makes the conversion explicit and survives values arriving as
+ strings from JSON parameters.
+ """
+ rendered: Dict[str, Any] = {}
+ for key, value in drop_none(payload).items():
+ if isinstance(value, bool):
+ rendered[key] = "true" if value else "false"
+ else:
+ rendered[key] = value
+ return rendered
+
+
+__all__ = [
+ "DEFAULT_TIMEOUT_SECONDS",
+ "HttpSpeechProvider",
+ "drop_none",
+ "query_flags",
+]
diff --git a/server/nodes/speech/_providers/_openai_compat.py b/server/nodes/speech/_providers/_openai_compat.py
new file mode 100644
index 00000000..48122910
--- /dev/null
+++ b/server/nodes/speech/_providers/_openai_compat.py
@@ -0,0 +1,313 @@
+"""OpenAI's speech wire format, and everyone who speaks it.
+
+One class serves OpenAI (both directions) and Groq (transcription only).
+Groq hosts the identical route at ``/openai/v1/audio/transcriptions``, so the
+difference between them is a base URL and a model list -- both of which live
+in ``speech_defaults.json``. Adding another OpenAI-compatible transcription
+host is a JSON block plus one tuple entry at the bottom of this file, with no
+new Python.
+
+That Groq's instance also carries a ``synthesize`` method it cannot serve is
+harmless and is the point of the two-registry design: Groq is never
+registered for synthesis, so nothing can select it for synthesis, and no
+``supports_tts`` flag has to be kept honest.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Tuple
+
+from core.logging import get_logger
+from .. import _config as speech_config
+from .._protocol import (
+ AudioPayload,
+ SttRequest,
+ SttResult,
+ TranscriptWord,
+ TtsRequest,
+ TtsResult,
+ Voice,
+)
+from ._http import HttpSpeechProvider, drop_none
+from .._registry import (
+ SttProviderSpec,
+ TtsProviderSpec,
+ register_stt_provider,
+ register_tts_provider,
+)
+
+logger = get_logger(__name__)
+
+# Container -> mime. ``pcm`` is deliberately headerless: OpenAI documents it
+# as raw 24 kHz 16-bit mono, so it gets the sample rate attached on the
+# payload instead of a container mime type that would imply a header.
+_MIME_BY_FORMAT = {
+ "mp3": "audio/mpeg",
+ "opus": "audio/opus",
+ "aac": "audio/aac",
+ "flac": "audio/flac",
+ "wav": "audio/wav",
+ "pcm": "audio/L16",
+}
+
+_PCM_SAMPLE_RATE = 24000
+
+
+class OpenAISpeechProvider(HttpSpeechProvider):
+ """Speech over OpenAI's ``/audio/*`` routes."""
+
+ provider_name = "openai"
+
+ # ------------------------------------------------------------------
+ # synthesis
+ # ------------------------------------------------------------------
+
+ async def synthesize(self, req: TtsRequest) -> TtsResult:
+ model = req.model or speech_config.default_model(self.provider_name, "tts")
+ fmt = req.output_format or speech_config.capability(
+ self.provider_name, "tts", "default_output_format", default="mp3"
+ )
+
+ body: Dict[str, Any] = {
+ "input": req.text,
+ "model": model,
+ "voice": req.voice
+ or speech_config.default_voice(self.provider_name)
+ or "alloy",
+ "response_format": fmt,
+ "speed": req.speed,
+ }
+ # `instructions` is silently ignored on tts-1 / tts-1-hd rather than
+ # rejected, so it is filtered by capability instead of being sent
+ # hopefully.
+ instructions = req.provider_options.get("instructions")
+ if instructions and speech_config.supports(
+ self.provider_name, "tts", "supports_instructions", model=model
+ ):
+ body["instructions"] = instructions
+
+ body.update(
+ {
+ k: v
+ for k, v in req.provider_options.items()
+ if k not in {"instructions"}
+ }
+ )
+
+ endpoint = speech_config.endpoint(self.provider_name, "tts") or "/audio/speech"
+ response = await self.request("POST", endpoint, json_body=drop_none(body))
+
+ return TtsResult(
+ audio=[
+ AudioPayload(
+ data=response.content,
+ format=fmt,
+ mime_type=_MIME_BY_FORMAT.get(fmt, "application/octet-stream"),
+ sample_rate=_PCM_SAMPLE_RATE if fmt == "pcm" else None,
+ channels=1 if fmt == "pcm" else None,
+ )
+ ],
+ model=model,
+ voice=str(body.get("voice") or ""),
+ request_id=response.headers.get("x-request-id"),
+ billed_units=float(len(req.text)),
+ billed_unit="characters",
+ )
+
+ async def list_voices(self) -> List[Voice]:
+ """Static catalogue -- OpenAI publishes no voices endpoint.
+
+ Model-scoped because ``tts-1`` supports only nine of the thirteen
+ voices; passing the model in via ``provider_options`` is not possible
+ here, so the widest set is returned and the node validates.
+ """
+ return [
+ Voice(id=voice, name=voice.title())
+ for voice in speech_config.voices(self.provider_name)
+ ]
+
+ # ------------------------------------------------------------------
+ # transcription
+ # ------------------------------------------------------------------
+
+ async def transcribe(self, req: SttRequest) -> SttResult:
+ model = req.model or speech_config.default_model(self.provider_name, "stt")
+ response_format = self._response_format(req, model)
+
+ translate = req.translate and speech_config.supports(
+ self.provider_name, "stt", "supports_translate", model=model
+ )
+ if req.translate and not translate:
+ logger.warning(
+ "model cannot translate; transcribing in-language instead",
+ provider=self.provider_name,
+ model=model,
+ )
+
+ endpoint_key = "translate_endpoint" if translate else "endpoint"
+ endpoint = (
+ speech_config.endpoint(self.provider_name, "stt", endpoint_key)
+ or "/audio/transcriptions"
+ )
+
+ form: Dict[str, Any] = {
+ "model": model,
+ "response_format": response_format,
+ }
+ # The translation route always outputs English, so a language hint
+ # is meaningless there and rejected by some hosts.
+ if not translate and req.language and req.language != "unknown":
+ form["language"] = req.language
+ if req.prompt and speech_config.supports(
+ self.provider_name, "stt", "supports_prompt", model=model
+ ):
+ form["prompt"] = req.prompt
+ form.update(req.provider_options)
+
+ files = {"file": (req.filename, req.audio, req.mime_type)}
+ if req.timestamps and response_format == "verbose_json":
+ # Form-encoded arrays need the bracket suffix; httpx expands a
+ # list value into repeated fields.
+ form["timestamp_granularities[]"] = ["word", "segment"]
+
+ response = await self.request(
+ "POST", endpoint, data=drop_none(form), files=files
+ )
+
+ if response_format == "text":
+ # Plain-text mode carries no metadata at all -- no duration, so
+ # no billable unit either. The node falls back to probing the
+ # audio it already has on disk.
+ return SttResult(text=response.text.strip(), billed_unit="seconds")
+ return self._parse_json_transcript(response.json())
+
+ # ------------------------------------------------------------------
+ # helpers
+ # ------------------------------------------------------------------
+
+ def _response_format(self, req: SttRequest, model: str) -> str:
+ """Pick a response format this model actually accepts.
+
+ OpenAI gates the allowed set per model and returns 400 on a
+ violation, so an unsupported request is downgraded here rather than
+ failing the workflow. Word timestamps need ``verbose_json``, which
+ in practice means whisper-1.
+ """
+ allowed = speech_config.response_formats(self.provider_name, model)
+ requested = req.provider_options.get("response_format") or ""
+
+ if req.timestamps and not requested:
+ requested = "verbose_json"
+ if not requested:
+ requested = speech_config.capability(
+ self.provider_name,
+ "stt",
+ "default_response_format",
+ model=model,
+ default="json",
+ )
+
+ if allowed and requested not in allowed:
+ fallback = "json" if "json" in allowed else allowed[0]
+ logger.warning(
+ "response_format not supported by this model; downgrading",
+ provider=self.provider_name,
+ model=model,
+ requested=requested,
+ using=fallback,
+ )
+ return fallback
+ return requested
+
+ def _parse_json_transcript(self, payload: Dict[str, Any]) -> SttResult:
+ words = [
+ TranscriptWord(
+ word=str(w.get("word") or ""),
+ start=_as_float(w.get("start")),
+ end=_as_float(w.get("end")),
+ speaker=_as_str(w.get("speaker")),
+ )
+ for w in payload.get("words") or []
+ if isinstance(w, dict)
+ ]
+ duration = _as_float(payload.get("duration"))
+ return SttResult(
+ text=str(payload.get("text") or "").strip(),
+ language=_as_str(payload.get("language")),
+ duration_seconds=duration,
+ words=words,
+ segments=[s for s in payload.get("segments") or [] if isinstance(s, dict)],
+ request_id=_as_str((payload.get("x_groq") or {}).get("id"))
+ if isinstance(payload.get("x_groq"), dict)
+ else None,
+ billed_units=self._billed_seconds(duration),
+ billed_unit="seconds",
+ )
+
+ def _billed_seconds(self, duration: float | None) -> float | None:
+ """Apply a provider's minimum billed duration, if it declares one."""
+ if duration is None:
+ return None
+ floor = speech_config.capability(
+ self.provider_name, "stt", "min_billed_seconds", default=None
+ )
+ if isinstance(floor, (int, float)):
+ return max(float(duration), float(floor))
+ return float(duration)
+
+
+def _as_float(value: Any) -> float | None:
+ try:
+ return float(value) if value is not None else None
+ except (TypeError, ValueError):
+ return None
+
+
+def _as_str(value: Any) -> str | None:
+ return str(value) if value not in (None, "") else None
+
+
+# ---------------------------------------------------------------------------
+# Plugin self-registration
+# ---------------------------------------------------------------------------
+#
+# ``httpx:HTTPError`` is the common ancestor of connect, timeout and
+# status errors, so one lazy ref covers every transport failure. It stays a
+# ``"module:Class"`` string for the same reason the LLM specs do -- see
+# services/provider_registry.py.
+
+_COMPAT_STT_PROVIDERS: Tuple[str, ...] = ("openai", "groq")
+
+
+def _register() -> None:
+ register_tts_provider(
+ TtsProviderSpec(
+ name="openai",
+ factory=OpenAISpeechProvider,
+ sdk_exception_refs=("httpx:HTTPError",),
+ client_kwargs={
+ "provider_name": "openai",
+ "base_url": speech_config.base_url("openai"),
+ },
+ )
+ )
+
+ for name in _COMPAT_STT_PROVIDERS:
+ base = speech_config.base_url(name)
+ if not base:
+ logger.warning(
+ "skipping compat speech provider -- no base_url in speech_defaults.json",
+ provider=name,
+ )
+ continue
+ register_stt_provider(
+ SttProviderSpec(
+ name=name,
+ factory=OpenAISpeechProvider,
+ sdk_exception_refs=("httpx:HTTPError",),
+ client_kwargs={"provider_name": name, "base_url": base},
+ )
+ )
+
+
+_register()
diff --git a/server/nodes/speech/_providers/deepgram.py b/server/nodes/speech/_providers/deepgram.py
new file mode 100644
index 00000000..06cbd768
--- /dev/null
+++ b/server/nodes/speech/_providers/deepgram.py
@@ -0,0 +1,195 @@
+"""Deepgram speech-to-text.
+
+Diverges from every other provider here on all three axes at once, which is
+why each is called out explicitly rather than left to the reader:
+
+* **Auth** is ``Authorization: Token <key>``. Not ``Bearer`` -- Deepgram
+ documents no Bearer alternative.
+* **Options are query parameters.** There is no JSON options body. Putting
+ ``model`` or ``diarize`` in a body is ignored silently.
+* **Audio is the raw request body** with an ``audio/*`` content type. Not
+ multipart, not base64. (A remote URL is the one case that does use a JSON
+ body, and that path is deliberately not wired: the node has already read
+ the bytes under workspace containment, and letting a workflow hand
+ Deepgram an arbitrary URL would be a server-side request forgery vector.)
+
+Multi-value options are repeated keys (``keyterm=a&keyterm=b``), never comma
+lists, which httpx produces for list values.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from core.logging import get_logger
+from .. import _config as speech_config
+from .._protocol import (
+ SttRequest,
+ SttResult,
+ TranscriptWord,
+)
+from ._http import (
+ HttpSpeechProvider,
+ drop_none,
+ query_flags,
+)
+from .._registry import SttProviderSpec, register_stt_provider
+
+logger = get_logger(__name__)
+
+# Options Deepgram accepts that the neutral request already covers. Anything
+# else a caller wants goes through provider_options untouched.
+_PASSTHROUGH_OPTIONS = (
+ "smart_format",
+ "punctuate",
+ "paragraphs",
+ "utterances",
+ "filler_words",
+ "numerals",
+ "profanity_filter",
+ "redact",
+ "search",
+ "summarize",
+ "topics",
+ "intents",
+ "sentiment",
+ "detect_entities",
+ "dictation",
+ "measurements",
+ "multichannel",
+ "utt_split",
+ "encoding",
+ "sample_rate",
+ "channels",
+ "keywords",
+ "keyterm",
+ "version",
+ "tag",
+)
+
+
+class DeepgramProvider(HttpSpeechProvider):
+ provider_name = "deepgram"
+
+ def auth_headers(self) -> Dict[str, str]:
+ # "Token", not "Bearer". A Bearer prefix authenticates as nobody and
+ # returns 401 with an unhelpful body.
+ return {"Authorization": f"Token {self.api_key}"}
+
+ async def transcribe(self, req: SttRequest) -> SttResult:
+ model = req.model or speech_config.default_model(self.provider_name, "stt")
+
+ params: Dict[str, Any] = {
+ "model": model,
+ "smart_format": speech_config.capability(
+ self.provider_name, "stt", "default_smart_format", default=True
+ ),
+ }
+ if req.language and req.language != "unknown":
+ params["language"] = req.language
+ else:
+ # Deepgram auto-detects only when explicitly asked to.
+ params["detect_language"] = True
+ if req.diarize:
+ params["diarize"] = True
+ if req.translate:
+ logger.warning(
+ "Deepgram does not translate; transcribing in-language instead",
+ provider=self.provider_name,
+ )
+
+ params.update(
+ {k: v for k, v in req.provider_options.items() if k in _PASSTHROUGH_OPTIONS}
+ )
+
+ endpoint = speech_config.endpoint(self.provider_name, "stt") or "/listen"
+ response = await self.request(
+ "POST",
+ endpoint,
+ params=query_flags(params),
+ content=req.audio,
+ headers={"Content-Type": req.mime_type or "application/octet-stream"},
+ )
+ return self._parse(response.json(), req)
+
+ # ------------------------------------------------------------------
+ # response parsing
+ # ------------------------------------------------------------------
+
+ def _parse(self, payload: Dict[str, Any], req: SttRequest) -> SttResult:
+ results = payload.get("results") or {}
+ metadata = payload.get("metadata") or {}
+ channel = _first_dict(results.get("channels"))
+ alternative = _first_dict(channel.get("alternatives")) if channel else {}
+
+ duration = _as_float(metadata.get("duration"))
+ words = [
+ TranscriptWord(
+ word=str(w.get("punctuated_word") or w.get("word") or ""),
+ start=_as_float(w.get("start")),
+ end=_as_float(w.get("end")),
+ speaker=_as_str(w.get("speaker")),
+ confidence=_as_float(w.get("confidence")),
+ )
+ for w in (alternative.get("words") or [])
+ if isinstance(w, dict)
+ ]
+
+ # `paragraphs.transcript` carries speaker labels and line breaks that
+ # the flat `transcript` string loses, so it wins when present.
+ text = str(alternative.get("transcript") or "").strip()
+ paragraphs = alternative.get("paragraphs")
+ if isinstance(paragraphs, dict):
+ formatted = str(paragraphs.get("transcript") or "").strip()
+ if formatted:
+ text = formatted
+
+ utterances = [u for u in (results.get("utterances") or []) if isinstance(u, dict)]
+
+ return SttResult(
+ text=text,
+ language=_as_str(channel.get("detected_language")) or (req.language or None),
+ language_confidence=_as_float(channel.get("language_confidence")),
+ duration_seconds=duration,
+ words=words,
+ segments=utterances,
+ request_id=_as_str(metadata.get("request_id")),
+ # Deepgram prices per minute, so the unit is converted here
+ # rather than leaving central code to work out what "duration"
+ # meant for this vendor.
+ billed_units=(duration / 60.0) if duration is not None else None,
+ billed_unit="minutes",
+ )
+
+
+def _first_dict(value: Any) -> Dict[str, Any]:
+ if isinstance(value, list) and value and isinstance(value[0], dict):
+ return value[0]
+ return {}
+
+
+def _as_float(value: Any) -> Optional[float]:
+ try:
+ return float(value) if value is not None else None
+ except (TypeError, ValueError):
+ return None
+
+
+def _as_str(value: Any) -> Optional[str]:
+ return str(value) if value not in (None, "") else None
+
+
+register_stt_provider(
+ SttProviderSpec(
+ name="deepgram",
+ factory=DeepgramProvider,
+ sdk_exception_refs=("httpx:HTTPError",),
+ client_kwargs={
+ "provider_name": "deepgram",
+ "base_url": speech_config.base_url("deepgram"),
+ },
+ )
+)
+
+
+__all__: List[str] = ["DeepgramProvider"]
diff --git a/server/nodes/speech/_providers/elevenlabs.py b/server/nodes/speech/_providers/elevenlabs.py
new file mode 100644
index 00000000..466695bb
--- /dev/null
+++ b/server/nodes/speech/_providers/elevenlabs.py
@@ -0,0 +1,234 @@
+"""ElevenLabs text-to-speech.
+
+The most structurally divergent provider in the v1 set, which makes it the
+useful shape test for the abstraction. Three things differ from everyone
+else:
+
+* the voice id is a **path segment**, not a body field;
+* ``output_format`` is a **query parameter** -- put it in the JSON body and
+ the API returns the default format with no error at all;
+* auth is a bare ``xi-api-key`` header with no scheme keyword.
+
+``output_format`` also encodes container, sample rate and (for mp3) bitrate
+in one underscore-delimited string, so the container and sample rate that
+end up on the ``AudioRef`` are parsed back out of it rather than guessed.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from core.logging import get_logger
+from .. import _config as speech_config
+from .._protocol import (
+ AudioPayload,
+ TtsRequest,
+ TtsResult,
+ Voice,
+)
+from ._http import HttpSpeechProvider, drop_none
+from .._registry import TtsProviderSpec, register_tts_provider
+
+logger = get_logger(__name__)
+
+_MIME_BY_CONTAINER = {
+ "mp3": "audio/mpeg",
+ "wav": "audio/wav",
+ "pcm": "audio/L16",
+ "opus": "audio/opus",
+ "ulaw": "audio/basic",
+ "alaw": "audio/basic",
+}
+
+# Fields inside `voice_settings`. Sent only when the caller supplies them --
+# ElevenLabs' own defaults are better than anything guessed here, and
+# whether these are honoured on eleven_v3 is undocumented, so nothing is
+# sent speculatively.
+_VOICE_SETTING_KEYS = (
+ "stability",
+ "similarity_boost",
+ "style",
+ "use_speaker_boost",
+ "speed",
+)
+
+# Everything ElevenLabs takes as query string rather than body.
+_QUERY_KEYS = ("enable_logging", "optimize_streaming_latency")
+
+_VOICES_PAGE_SIZE = 100
+
+
+class ElevenLabsProvider(HttpSpeechProvider):
+ provider_name = "elevenlabs"
+
+ def auth_headers(self) -> Dict[str, str]:
+ # Bare key, no scheme keyword. `Authorization: Bearer <key>` is
+ # rejected.
+ return {"xi-api-key": self.api_key}
+
+ async def synthesize(self, req: TtsRequest) -> TtsResult:
+ voice_id = req.voice or speech_config.default_voice(self.provider_name)
+ if not voice_id:
+ from services.plugin import NodeUserError
+
+ raise NodeUserError(
+ "ElevenLabs requires a voice. Pick one from the Voice dropdown "
+ "-- there is no account-wide default."
+ )
+
+ model = req.model or speech_config.default_model(self.provider_name, "tts")
+ output_format = req.output_format or speech_config.capability(
+ self.provider_name, "tts", "default_output_format", default="mp3_44100_128"
+ )
+
+ options = dict(req.provider_options)
+ voice_settings = self._voice_settings(req, options)
+
+ body: Dict[str, Any] = {
+ "text": req.text,
+ "model_id": model,
+ "language_code": req.language or None,
+ "voice_settings": voice_settings or None,
+ }
+ # Whatever is left in options after the query keys and voice
+ # settings have been claimed is a body field (seed, previous_text,
+ # apply_text_normalization, ...).
+ params = {k: options.pop(k) for k in _QUERY_KEYS if k in options}
+ body.update(options)
+
+ params["output_format"] = output_format
+
+ path = f"/v1/text-to-speech/{voice_id}"
+ response = await self.request(
+ "POST", path, json_body=drop_none(body), params=drop_none(params)
+ )
+
+ container, sample_rate = _parse_output_format(output_format)
+ return TtsResult(
+ audio=[
+ AudioPayload(
+ data=response.content,
+ format=container,
+ mime_type=_MIME_BY_CONTAINER.get(
+ container, "application/octet-stream"
+ ),
+ sample_rate=sample_rate,
+ channels=1,
+ )
+ ],
+ model=model,
+ voice=voice_id,
+ request_id=response.headers.get("request-id"),
+ billed_units=float(len(req.text)),
+ billed_unit="characters",
+ )
+
+ async def list_voices(self) -> List[Voice]:
+ """Live catalogue via the v2 endpoint.
+
+ Cursor-paginated; the docs are explicit that ``has_more`` plus
+ ``next_page_token`` is the supported way to walk it and that
+ ``total_count`` should not be relied on. Pages are capped so a large
+ enterprise account cannot stall a dropdown load indefinitely.
+ """
+ endpoint = (
+ speech_config.capability(
+ self.provider_name, "tts", "voices_endpoint", default="/v2/voices"
+ )
+ or "/v2/voices"
+ )
+
+ voices: List[Voice] = []
+ page_token: Optional[str] = None
+ for _ in range(10):
+ params = drop_none(
+ {"page_size": _VOICES_PAGE_SIZE, "next_page_token": page_token}
+ )
+ payload = (await self.request("GET", endpoint, params=params)).json()
+ for entry in payload.get("voices") or []:
+ if not isinstance(entry, dict):
+ continue
+ voice_id = str(entry.get("voice_id") or "")
+ if not voice_id:
+ continue
+ voices.append(
+ Voice(
+ id=voice_id,
+ name=str(entry.get("name") or voice_id),
+ description=str(entry.get("description") or "")[:160],
+ preview_url=str(entry.get("preview_url") or ""),
+ )
+ )
+ if not payload.get("has_more"):
+ break
+ page_token = payload.get("next_page_token")
+ if not page_token:
+ break
+ else:
+ logger.warning(
+ "stopped paginating ElevenLabs voices at the page cap",
+ provider=self.provider_name,
+ collected=len(voices),
+ )
+ return voices
+
+ def _voice_settings(
+ self, req: TtsRequest, options: Dict[str, Any]
+ ) -> Dict[str, Any]:
+ """Collect voice_settings, clamping speed to the documented API range.
+
+ ElevenLabs publishes two different speed ranges -- the OpenAPI schema
+ allows 0.5-2.0 while the best-practices guide recommends 0.7-1.2.
+ Neither page acknowledges the other. The schema bound is enforced
+ because that is what the API actually accepts; the narrower range is
+ left as guidance for the node's help text.
+ """
+ settings = {
+ key: options.pop(key) for key in _VOICE_SETTING_KEYS if key in options
+ }
+ if req.speed is not None:
+ settings["speed"] = req.speed
+
+ speed = settings.get("speed")
+ if speed is not None:
+ bounds = speech_config.speed_range(self.provider_name) or (0.5, 2.0)
+ clamped = max(bounds[0], min(float(speed), bounds[1]))
+ if clamped != float(speed):
+ logger.warning(
+ "clamped ElevenLabs speed to the supported range",
+ requested=speed,
+ using=clamped,
+ )
+ settings["speed"] = clamped
+ return drop_none(settings)
+
+
+def _parse_output_format(output_format: str) -> tuple[str, Optional[int]]:
+ """Split ``mp3_44100_128`` into ``("mp3", 44100)``.
+
+ The container drives the file extension and mime type on the resulting
+ ``AudioRef``; the sample rate matters because ``pcm_*`` is headerless and
+ nothing downstream could otherwise work out its rate.
+ """
+ parts = str(output_format or "").split("_")
+ container = parts[0] if parts else ""
+ sample_rate: Optional[int] = None
+ if len(parts) > 1:
+ try:
+ sample_rate = int(parts[1])
+ except ValueError:
+ sample_rate = None
+ return container, sample_rate
+
+
+register_tts_provider(
+ TtsProviderSpec(
+ name="elevenlabs",
+ factory=ElevenLabsProvider,
+ sdk_exception_refs=("httpx:HTTPError",),
+ client_kwargs={
+ "provider_name": "elevenlabs",
+ "base_url": speech_config.base_url("elevenlabs"),
+ },
+ )
+)
diff --git a/server/nodes/speech/_providers/sarvam.py b/server/nodes/speech/_providers/sarvam.py
new file mode 100644
index 00000000..751befba
--- /dev/null
+++ b/server/nodes/speech/_providers/sarvam.py
@@ -0,0 +1,271 @@
+"""Sarvam AI speech, in both directions. Indic-first.
+
+Ported from the two vendor-locked nodes this layer replaces
+(``sarvamTextToSpeech`` / ``sarvamSpeechToText``), preserving their wire
+behaviour exactly. Two properties of this API shape the code:
+
+* **Synthesis returns base64 inside JSON**, not raw bytes, and the ``audios``
+ key is an **array**. Long input comes back as several standalone clips.
+ They are not byte-concatenatable -- each carries its own container header
+ -- so the array is surfaced as several payloads and the node writes one
+ ``AudioRef`` per clip. Flattening them into one file would produce audio
+ that plays only its first chunk.
+* **Explicit nulls are rejected.** Sending ``"pitch": null`` is a 422 where
+ omitting the key is fine, so every body goes through ``drop_none``.
+
+Parameter support also splits hard by model: ``bulbul:v3`` takes
+``temperature`` and rejects ``pitch`` / ``loudness`` / ``enable_preprocessing``
+outright, while ``bulbul:v2`` is the mirror image. That gating lives here
+because it is wire-format knowledge; the user-facing validation (character
+caps, speaker membership) lives in the node where ``NodeUserError`` reads
+naturally.
+"""
+
+from __future__ import annotations
+
+import base64
+import binascii
+from typing import Any, Dict, List, Optional
+
+from core.logging import get_logger
+from .. import _config as speech_config
+from .._protocol import (
+ AudioPayload,
+ SpeechError,
+ SpeechErrorCategory,
+ SttRequest,
+ SttResult,
+ TtsRequest,
+ TtsResult,
+ Voice,
+)
+from ._http import HttpSpeechProvider, drop_none
+from .._registry import (
+ SttProviderSpec,
+ TtsProviderSpec,
+ register_stt_provider,
+ register_tts_provider,
+)
+
+logger = get_logger(__name__)
+
+# Codec name -> file extension. Three of these differ from the codec string,
+# which matters because the extension ends up in the workspace filename.
+_CODEC_EXT = {"linear16": "pcm", "mulaw": "ulaw", "alaw": "alaw"}
+
+_MIME_BY_CODEC = {
+ "wav": "audio/wav",
+ "mp3": "audio/mpeg",
+ "flac": "audio/flac",
+ "aac": "audio/aac",
+ "opus": "audio/opus",
+ "linear16": "audio/L16",
+ "mulaw": "audio/basic",
+ "alaw": "audio/basic",
+}
+
+# Body fields only bulbul:v2 accepts. v3 rejects them rather than ignoring.
+_V2_ONLY = ("pitch", "loudness", "enable_preprocessing")
+# Body fields only bulbul:v3 accepts.
+_V3_ONLY = ("temperature",)
+
+
+class SarvamSpeechProvider(HttpSpeechProvider):
+ provider_name = "sarvam"
+
+ def auth_headers(self) -> Dict[str, str]:
+ # Sarvam's speech routes accept only the native header. (Bearer works
+ # on their OpenAI-compatible chat route, which is a different
+ # surface and not this one.)
+ return {"api-subscription-key": self.api_key}
+
+ # ------------------------------------------------------------------
+ # synthesis
+ # ------------------------------------------------------------------
+
+ async def synthesize(self, req: TtsRequest) -> TtsResult:
+ model = req.model or speech_config.default_model(self.provider_name, "tts")
+ is_v3 = model.endswith("v3")
+ codec = req.output_format or speech_config.capability(
+ self.provider_name, "tts", "default_output_format", default="wav"
+ )
+
+ body: Dict[str, Any] = {
+ "text": req.text,
+ "target_language_code": req.language
+ or speech_config.capability(
+ self.provider_name, "tts", "default_language", default="hi-IN"
+ ),
+ "model": model,
+ "speaker": req.voice or speech_config.default_voice(self.provider_name),
+ "pace": req.speed,
+ "speech_sample_rate": req.sample_rate
+ or speech_config.capability(
+ self.provider_name, "tts", "default_sample_rate", default=24000
+ ),
+ "output_audio_codec": codec,
+ }
+
+ # Whatever this model rejects is dropped rather than forwarded; the
+ # opposite model's exclusive fields would 400 the whole request.
+ # Fields it accepts but the caller omitted simply stay absent, and
+ # Sarvam's own defaults apply.
+ rejected = _V2_ONLY if is_v3 else _V3_ONLY
+ for key, value in req.provider_options.items():
+ if key in rejected:
+ logger.debug(
+ "dropping parameter the selected Sarvam model rejects",
+ model=model,
+ parameter=key,
+ )
+ continue
+ body[key] = value
+
+ endpoint = (
+ speech_config.endpoint(self.provider_name, "tts") or "/text-to-speech"
+ )
+ payload = (
+ await self.request("POST", endpoint, json_body=drop_none(body))
+ ).json()
+
+ chunks = payload.get("audios") or []
+ if not chunks:
+ raise SpeechError(
+ message="Sarvam returned no audio for this request.",
+ provider=self.provider_name,
+ category=SpeechErrorCategory.SERVER,
+ )
+
+ ext = _CODEC_EXT.get(codec, codec)
+ mime = _MIME_BY_CODEC.get(codec, "application/octet-stream")
+ sample_rate = body.get("speech_sample_rate")
+
+ audio = [
+ AudioPayload(
+ data=_decode(chunk, index),
+ format=ext,
+ mime_type=mime,
+ sample_rate=int(sample_rate) if sample_rate else None,
+ channels=1,
+ )
+ for index, chunk in enumerate(chunks)
+ ]
+
+ return TtsResult(
+ audio=audio,
+ model=model,
+ voice=str(body.get("speaker") or ""),
+ request_id=payload.get("request_id"),
+ billed_units=float(len(req.text)),
+ billed_unit="characters",
+ )
+
+ async def list_voices(self) -> List[Voice]:
+ """Static per-model catalogue -- Sarvam publishes no voices endpoint.
+
+ The v2 and v3 speaker sets are disjoint, so this returns the default
+ (v3) set and the node re-reads config for the selected model. Sending
+ a v2 speaker to v3 is a 400.
+ """
+ return [
+ Voice(id=speaker, name=speaker.title())
+ for speaker in speech_config.voices(self.provider_name)
+ ]
+
+ # ------------------------------------------------------------------
+ # transcription
+ # ------------------------------------------------------------------
+
+ async def transcribe(self, req: SttRequest) -> SttResult:
+ model = req.model or speech_config.default_model(self.provider_name, "stt")
+ mode = req.provider_options.get("mode") or (
+ "translate" if req.translate else "transcribe"
+ )
+
+ endpoint_key = (
+ "translate_endpoint"
+ if req.translate and mode == "translate"
+ else "endpoint"
+ )
+ endpoint = (
+ speech_config.endpoint(self.provider_name, "stt", endpoint_key)
+ or "/speech-to-text"
+ )
+
+ form = drop_none(
+ {
+ "model": model,
+ "language_code": req.language or "unknown",
+ "mode": mode,
+ "input_audio_codec": req.provider_options.get("input_audio_codec"),
+ }
+ )
+ # Bytes rather than a handle: Connection-style auth retry replays the
+ # same kwargs, and a consumed file object would replay as empty.
+ files = {"file": (req.filename, req.audio, req.mime_type)}
+
+ payload = (
+ await self.request(
+ "POST",
+ endpoint,
+ data={k: str(v) for k, v in form.items()},
+ files=files,
+ )
+ ).json()
+
+ return SttResult(
+ text=str(payload.get("transcript") or "").strip(),
+ language=payload.get("language_code"),
+ language_confidence=_as_float(payload.get("language_probability")),
+ # The synchronous endpoint never returns timestamps or
+ # diarization -- those exist only on the batch job API -- so no
+ # duration comes back either. The node fills billed_units from
+ # the audio it already probed, which is what replaced the flat
+ # 30-second charge the old node applied to every clip.
+ duration_seconds=None,
+ request_id=payload.get("request_id"),
+ billed_units=None,
+ billed_unit="seconds",
+ )
+
+
+def _decode(chunk: Any, index: int) -> bytes:
+ try:
+ return base64.b64decode(str(chunk), validate=True)
+ except (binascii.Error, ValueError) as exc:
+ raise SpeechError(
+ message=f"Sarvam audio chunk {index + 1} was not valid base64: {exc}",
+ provider="sarvam",
+ category=SpeechErrorCategory.SERVER,
+ ) from exc
+
+
+def _as_float(value: Any) -> Optional[float]:
+ try:
+ return float(value) if value is not None else None
+ except (TypeError, ValueError):
+ return None
+
+
+_CLIENT_KWARGS = {
+ "provider_name": "sarvam",
+ "base_url": speech_config.base_url("sarvam"),
+}
+
+register_tts_provider(
+ TtsProviderSpec(
+ name="sarvam",
+ factory=SarvamSpeechProvider,
+ sdk_exception_refs=("httpx:HTTPError",),
+ client_kwargs=_CLIENT_KWARGS,
+ )
+)
+
+register_stt_provider(
+ SttProviderSpec(
+ name="sarvam",
+ factory=SarvamSpeechProvider,
+ sdk_exception_refs=("httpx:HTTPError",),
+ client_kwargs=_CLIENT_KWARGS,
+ )
+)
diff --git a/server/nodes/speech/_registry.py b/server/nodes/speech/_registry.py
new file mode 100644
index 00000000..cb1bdc87
--- /dev/null
+++ b/server/nodes/speech/_registry.py
@@ -0,0 +1,89 @@
+"""Two provider registries — one per direction.
+
+The mechanism lives in :mod:`services.provider_registry`, shared with
+``services/llm``. This module is the speech-specific pair of instances plus
+the free-function API provider modules import.
+
+Why two rather than one registry with a capability flag: **membership is the
+capability**. A synthesis-only vendor registers into ``tts`` and simply never
+appears in the transcription node's dropdown, because that dropdown is
+literally ``stt_providers()``. There is no ``supports_stt`` boolean to fall
+out of sync with what the provider can actually do, and no way to select a
+provider for a direction it cannot serve.
+"""
+
+from __future__ import annotations
+
+from typing import List
+
+from services.provider_registry import (
+ ProviderRegistry,
+ ProviderSpec as _ProviderSpec,
+)
+from ._protocol import SttProvider, TtsProvider
+
+# Specialized to each direction's protocol so ``ProviderSpec(factory=...)``
+# type-checks against the right shape.
+TtsProviderSpec = _ProviderSpec[TtsProvider]
+SttProviderSpec = _ProviderSpec[SttProvider]
+
+_TTS_REGISTRY: ProviderRegistry[TtsProvider] = ProviderRegistry("text-to-speech")
+_STT_REGISTRY: ProviderRegistry[SttProvider] = ProviderRegistry("speech-to-text")
+
+
+def register_tts_provider(spec: TtsProviderSpec) -> None:
+ """Register a synthesis provider. Idempotent for identical specs."""
+ _TTS_REGISTRY.register(spec)
+
+
+def register_stt_provider(spec: SttProviderSpec) -> None:
+ """Register a transcription provider. Idempotent for identical specs."""
+ _STT_REGISTRY.register(spec)
+
+
+def get_tts_provider(name: str) -> TtsProviderSpec:
+ """Look up a synthesis spec; raises ``NodeUserError`` on a miss."""
+ return _TTS_REGISTRY.get(name)
+
+
+def get_stt_provider(name: str) -> SttProviderSpec:
+ """Look up a transcription spec; raises ``NodeUserError`` on a miss."""
+ return _STT_REGISTRY.get(name)
+
+
+def tts_providers() -> List[str]:
+ """Sorted synthesis provider names — the TTS node's provider enum."""
+ return _TTS_REGISTRY.all()
+
+
+def stt_providers() -> List[str]:
+ """Sorted transcription provider names — the STT node's provider enum."""
+ return _STT_REGISTRY.all()
+
+
+def has_tts_provider(name: str) -> bool:
+ return _TTS_REGISTRY.has(name)
+
+
+def has_stt_provider(name: str) -> bool:
+ return _STT_REGISTRY.has(name)
+
+
+def _reset_for_tests() -> None:
+ """Test-only — drop every registered provider in both directions."""
+ _TTS_REGISTRY.reset_for_tests()
+ _STT_REGISTRY.reset_for_tests()
+
+
+__all__ = [
+ "SttProviderSpec",
+ "TtsProviderSpec",
+ "get_stt_provider",
+ "get_tts_provider",
+ "has_stt_provider",
+ "has_tts_provider",
+ "register_stt_provider",
+ "register_tts_provider",
+ "stt_providers",
+ "tts_providers",
+]
diff --git a/server/nodes/speech/_unifier.py b/server/nodes/speech/_unifier.py
new file mode 100644
index 00000000..2bc292c0
--- /dev/null
+++ b/server/nodes/speech/_unifier.py
@@ -0,0 +1,192 @@
+"""Dispatch and error translation for the two speech nodes.
+
+Reads the registries to route a call to the right provider, and normalizes
+every provider failure into ``NodeUserError`` at one catch site. The nodes
+never touch a provider class.
+
+**No client cache here, deliberately.** ``ChatUnifier`` caches SDK clients
+because an agent loop makes many model calls inside one node execution, so
+connection reuse is worth a lease-counted LRU. Speech is the opposite shape:
+a node execution issues exactly one HTTP request, and the client setup cost
+is invisible next to a multi-second synthesis. Caching would buy nothing and
+cost a process-wide singleton plus a shutdown hook wired into ``main.py`` --
+core coupling this plugin has no reason to need. A client is built per call
+and closed in ``finally``.
+
+That also settles the question left open when the generic registry was
+extracted: there is no second consumer for a shared client cache, so none
+was built.
+"""
+
+from __future__ import annotations
+
+from typing import Awaitable, Callable, List, Optional, TypeVar
+
+from core.logging import get_logger
+from services.plugin import NodeUserError
+from services.provider_registry import ProviderSpec
+
+from ._protocol import (
+ SpeechError,
+ SpeechErrorCategory,
+ SttRequest,
+ SttResult,
+ TtsRequest,
+ TtsResult,
+ Voice,
+)
+from ._registry import get_stt_provider, get_tts_provider
+
+logger = get_logger(__name__)
+
+T = TypeVar("T")
+
+
+async def synthesize(
+ *,
+ provider: str,
+ api_key: str,
+ request: TtsRequest,
+ translate_errors: bool = True,
+) -> TtsResult:
+ """Synthesize speech via the named provider."""
+ spec = get_tts_provider(provider)
+ return await _dispatch(
+ spec,
+ provider,
+ api_key,
+ lambda client: client.synthesize(request),
+ operation="synthesis",
+ translate_errors=translate_errors,
+ )
+
+
+async def transcribe(
+ *,
+ provider: str,
+ api_key: str,
+ request: SttRequest,
+ translate_errors: bool = True,
+) -> SttResult:
+ """Transcribe audio via the named provider."""
+ spec = get_stt_provider(provider)
+ return await _dispatch(
+ spec,
+ provider,
+ api_key,
+ lambda client: client.transcribe(request),
+ operation="transcription",
+ translate_errors=translate_errors,
+ )
+
+
+async def list_voices(*, provider: str, api_key: str) -> List[Voice]:
+ """List selectable voices for a synthesis provider.
+
+ Providers without a live catalogue return their configured static list,
+ so the dropdown loader never branches on whether an endpoint exists.
+ """
+ spec = get_tts_provider(provider)
+ return await _dispatch(
+ spec,
+ provider,
+ api_key,
+ lambda client: client.list_voices(),
+ operation="voice listing",
+ translate_errors=True,
+ )
+
+
+async def _dispatch(
+ spec: ProviderSpec,
+ provider: str,
+ api_key: str,
+ call: Callable[[object], Awaitable[T]],
+ *,
+ operation: str,
+ translate_errors: bool,
+) -> T:
+ """Build a client, run ``call``, normalize failures, always close.
+
+ One catch site for all three entry points. ``translate_errors=False``
+ re-raises the structured :class:`SpeechError` instead of a flattened
+ user message, which is what a Temporal retry policy needs to read
+ ``retryable``.
+ """
+ client: Optional[object] = None
+ try:
+ client = spec.factory(api_key=api_key, **spec.client_kwargs)
+ return await call(client)
+ except spec.sdk_exception_types as exc:
+ error = SpeechError.from_exception(provider, exc)
+ if not translate_errors:
+ raise error from exc
+ logger.warning(
+ "speech provider request failed",
+ provider=error.provider,
+ operation=operation,
+ category=error.category.value,
+ retryable=error.retryable,
+ status_code=error.status_code,
+ provider_code=error.provider_code,
+ request_id=error.request_id,
+ )
+ raise NodeUserError(error.user_message) from error
+ except SpeechError as exc:
+ # A provider classified a failure itself -- typically a documented
+ # vendor error that arrived with HTTP 200, which both Deepgram and
+ # Sarvam can do. Already structured, so just surface it.
+ if not translate_errors:
+ raise
+ logger.warning(
+ "speech provider returned a failure payload",
+ provider=exc.provider,
+ operation=operation,
+ category=exc.category.value,
+ )
+ raise NodeUserError(exc.user_message) from exc
+ except (ValueError, TypeError, OSError) as exc:
+ # Only normalize failures raised while *constructing* the client.
+ # Once one exists, an unexpected exception from provider logic is a
+ # programming error and keeps its traceback.
+ if client is not None:
+ raise
+ category = (
+ SpeechErrorCategory.CONNECTION
+ if isinstance(exc, OSError)
+ else SpeechErrorCategory.INVALID_REQUEST
+ )
+ error = SpeechError(
+ message=str(exc),
+ provider=provider,
+ category=category,
+ retryable=category == SpeechErrorCategory.CONNECTION,
+ )
+ if not translate_errors:
+ raise error from exc
+ logger.warning(
+ "speech client construction failed",
+ provider=error.provider,
+ operation=operation,
+ category=error.category.value,
+ )
+ raise NodeUserError(error.user_message) from error
+ finally:
+ if client is not None:
+ await _close(client)
+
+
+async def _close(client: object) -> None:
+ """Close a provider if it holds anything. Probed, not required."""
+ close = getattr(client, "aclose", None)
+ if close is None:
+ return
+ try:
+ result = close()
+ if hasattr(result, "__await__"):
+ await result
+ except Exception as exc:
+ logger.warning("failed to close speech client", error=str(exc))
+
+
+__all__ = ["list_voices", "synthesize", "transcribe"]
diff --git a/server/nodes/speech/icon_speechToText.svg b/server/nodes/speech/icon_speechToText.svg
new file mode 100644
index 00000000..147620a7
--- /dev/null
+++ b/server/nodes/speech/icon_speechToText.svg
@@ -0,0 +1,9 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
+ <rect x="8" y="2" width="6" height="11" rx="3"/>
+ <path d="M5 10a6 6 0 0 0 12 0"/>
+ <path d="M11 16v3"/>
+ <path d="M8 22h6"/>
+ <path d="M19 6h3"/>
+ <path d="M19 10h3"/>
+ <path d="M19 14h3"/>
+</svg>
diff --git a/server/nodes/speech/icon_textToSpeech.svg b/server/nodes/speech/icon_textToSpeech.svg
new file mode 100644
index 00000000..2a385424
--- /dev/null
+++ b/server/nodes/speech/icon_textToSpeech.svg
@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
+ <path d="M11 5 6 9H2v6h4l5 4z"/>
+ <path d="M15.5 8.5a5 5 0 0 1 0 7"/>
+ <path d="M19 5a9 9 0 0 1 0 14"/>
+</svg>
diff --git a/server/nodes/speech/meta.json b/server/nodes/speech/meta.json
new file mode 100644
index 00000000..100645a1
--- /dev/null
+++ b/server/nodes/speech/meta.json
@@ -0,0 +1,4 @@
+{
+ "_comment": "Matches the 'language' palette group colour. The plugin is vendor-neutral, so it takes the group's accent rather than any one provider's brand.",
+ "color": "#6A88E2"
+}
diff --git a/server/nodes/speech/speech_to_text.py b/server/nodes/speech/speech_to_text.py
new file mode 100644
index 00000000..26047fd2
--- /dev/null
+++ b/server/nodes/speech/speech_to_text.py
@@ -0,0 +1,242 @@
+"""Speech to Text — one node, many providers.
+
+Replaces the vendor-locked ``sarvamSpeechToText``, and closes two of its
+defects in the process:
+
+* **Path traversal.** The old node joined a user-supplied path onto the
+ workspace root with no check, so ``audio_file="../../credentials.db"``
+ read the encrypted credential store and uploaded it to the provider.
+ Every input shape here routes through ``services.media``, which resolves
+ under containment.
+* **Billing.** The old node charged every transcription as 30 seconds
+ because it never measured the clip. This one probes the real duration and
+ bills that, or bills nothing when the duration genuinely cannot be
+ determined.
+
+The model field is ``stt_model``, not ``model`` -- see the sibling module's
+docstring for why that naming is load-bearing.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from pydantic import BaseModel, Field
+
+from services.plugin import ActionNode, NodeContext, Operation, TaskQueue
+
+from ..model._credentials import GroqCredential, OpenAICredential, SarvamCredential
+from . import _config as speech_config
+from . import _unifier
+from ._base import (
+ measure_seconds,
+ provider_api_key,
+ read_audio_input,
+ require_provider,
+ track_usage,
+)
+from ._credentials import DeepgramCredential
+from ._protocol import SttRequest
+from ._registry import stt_providers
+
+from . import _providers # noqa: F401 isort:skip
+
+_PROVIDERS: List[str] = stt_providers()
+
+
+class SpeechToTextParams(BaseModel):
+ tool_name: str = Field(
+ default="speech_to_text",
+ description="Override name shown to the LLM when used as a tool.",
+ )
+ tool_description: str = Field(
+ default=(
+ "Transcribe an audio file to text. Accepts a workspace path or an "
+ "audio reference produced by an upstream node. Use when the user "
+ "asks what was said in a recording, or wants a transcript."
+ ),
+ description="Override description shown to the LLM when used as a tool.",
+ json_schema_extra={"rows": 3},
+ )
+
+ provider: str = Field(
+ default=_PROVIDERS[0] if _PROVIDERS else "openai",
+ description="Speech provider to transcribe with.",
+ json_schema_extra={"enum": _PROVIDERS},
+ )
+ # Three shapes reach this field: an AudioRef from an upstream node or the
+ # upload route, a bare workspace path, or the legacy base64 envelope the
+ # file widget used to emit. ``read_audio_input`` handles all three.
+ audio_file: Any = Field(
+ default="",
+ description="Audio file to transcribe — upload one, or point at a workspace path.",
+ json_schema_extra={
+ "widget": "file",
+ "accept": "audio/*,.wav,.mp3,.m4a,.ogg,.opus,.flac,.aac,.webm,.amr",
+ },
+ )
+ stt_model: str = Field(
+ default="",
+ description="Provider model id. Leave blank for the provider default.",
+ json_schema_extra={"loadOptionsMethod": "speechModels"},
+ )
+ language: str = Field(
+ default="",
+ description="Language hint (e.g. en, hi-IN). Leave blank to auto-detect.",
+ )
+ translate: bool = Field(
+ default=False,
+ description="Translate to English instead of transcribing in-language.",
+ )
+ diarize: bool = Field(
+ default=False,
+ description="Label speakers, where the provider and model support it.",
+ )
+ timestamps: bool = Field(
+ default=False,
+ description="Return per-word timing, where the provider and model support it.",
+ )
+ provider_options: Dict[str, Any] = Field(
+ default_factory=dict,
+ description=(
+ "Vendor-specific options passed through untouched — e.g. Deepgram "
+ "keyterm/redact, Sarvam mode, OpenAI response_format."
+ ),
+ )
+
+ model_config = {"extra": "ignore"}
+
+
+class SpeechToTextOutput(BaseModel):
+ transcript: str = ""
+ language: Optional[str] = None
+ language_confidence: Optional[float] = None
+ duration_seconds: Optional[float] = None
+ words: List[Dict[str, Any]] = Field(default_factory=list)
+ segments: List[Dict[str, Any]] = Field(default_factory=list)
+ provider: str = ""
+ stt_model: str = ""
+ request_id: Optional[str] = None
+
+ model_config = {"extra": "allow"}
+
+
+class SpeechToTextNode(ActionNode):
+ type = "speechToText"
+ display_name = "Speech to Text"
+ subtitle = "Transcribe Audio"
+ group = ("language", "tool")
+ description = "Transcribe audio to text using any configured speech provider"
+ component_kind = "square"
+ tool_name = "speech_to_text"
+ tool_description = (
+ "Transcribe an audio file to text. Accepts a workspace path or an audio "
+ "reference from an upstream node."
+ )
+ handles = (
+ {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},
+ {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},
+ )
+ credentials = (
+ OpenAICredential,
+ GroqCredential,
+ DeepgramCredential,
+ SarvamCredential,
+ )
+ annotations = {"destructive": False, "readonly": True, "open_world": True}
+ task_queue = TaskQueue.REST_API
+ usable_as_tool = True
+ hide_input_handle = False
+ hide_output_handle = False
+
+ Params = SpeechToTextParams
+ Output = SpeechToTextOutput
+
+ @Operation("transcribe")
+ async def transcribe(
+ self, ctx: NodeContext, params: SpeechToTextParams
+ ) -> SpeechToTextOutput:
+ provider = require_provider(params.provider, _PROVIDERS, "speech-to-text")
+ model = params.stt_model or speech_config.default_model(provider, "stt")
+
+ max_bytes = speech_config.capability(
+ provider, "stt", "max_upload_bytes", default=None
+ )
+ filename, blob, path = read_audio_input(
+ params.audio_file, ctx, max_bytes=max_bytes
+ )
+
+ api_key = await provider_api_key(ctx, provider)
+ result = await _unifier.transcribe(
+ provider=provider,
+ api_key=api_key,
+ request=SttRequest(
+ audio=blob,
+ filename=filename,
+ mime_type=_guess_mime(filename),
+ model=model,
+ language=params.language,
+ translate=params.translate,
+ diarize=params.diarize,
+ timestamps=params.timestamps,
+ provider_options=dict(params.provider_options),
+ ),
+ )
+
+ # Prefer what the provider reported; fall back to probing the file we
+ # already have. Either way the figure is measured, never assumed.
+ duration = result.duration_seconds
+ if duration is None:
+ duration = measure_seconds(path, declared_format=filename.rsplit(".", 1)[-1])
+
+ await track_usage(
+ ctx,
+ provider=provider,
+ operation="speech_to_text",
+ units=_billable(result.billed_units, result.billed_unit, duration),
+ unit=result.billed_unit or "seconds",
+ )
+
+ return SpeechToTextOutput(
+ transcript=result.text,
+ language=result.language,
+ language_confidence=result.language_confidence,
+ duration_seconds=duration,
+ words=[
+ {
+ "word": w.word,
+ "start": w.start,
+ "end": w.end,
+ "speaker": w.speaker,
+ "confidence": w.confidence,
+ }
+ for w in result.words
+ ],
+ segments=result.segments,
+ provider=provider,
+ stt_model=model,
+ request_id=result.request_id,
+ )
+
+
+def _billable(
+ reported: Optional[float], unit: str, duration: Optional[float]
+) -> Optional[float]:
+ """Resolve the billable quantity in the provider's own unit.
+
+ A provider that reported its own figure wins. Otherwise the measured
+ duration is converted into that provider's unit. When nothing could be
+ measured this returns ``None`` and no metric is written -- an
+ under-count is honest, a fabricated 30 seconds was not.
+ """
+ if reported is not None:
+ return reported
+ if duration is None:
+ return None
+ return duration / 60.0 if unit == "minutes" else duration
+
+
+def _guess_mime(filename: str) -> str:
+ import mimetypes
+
+ return mimetypes.guess_type(filename)[0] or "application/octet-stream"
diff --git a/server/nodes/speech/text_to_speech.py b/server/nodes/speech/text_to_speech.py
new file mode 100644
index 00000000..488d9348
--- /dev/null
+++ b/server/nodes/speech/text_to_speech.py
@@ -0,0 +1,247 @@
+"""Text to Speech — one node, many providers.
+
+Replaces the vendor-locked ``sarvamTextToSpeech``. The provider is a
+parameter rather than a node type, so adding a vendor never adds a node.
+
+Two things about this node are unusual for the codebase and both are
+deliberate:
+
+* It declares **several credentials** and picks one at runtime via
+ ``ctx.connection(<credential_id>)``. That is why every operation here is
+ imperative: the declarative ``routing=`` DSL resolves ``credentials[0]``
+ and would silently authenticate every provider with the first key in the
+ tuple.
+* Its model field is called ``tts_model``, not ``model``. A parameter panel
+ effect keyed on the literal names ``model`` and ``api_key`` overwrites
+ them with chat-model data whenever a sibling ``provider`` field is
+ present, and it does not check that the provider is an LLM provider --
+ so a field named ``model`` here would be cleared the moment the user
+ picked ElevenLabs.
+
+Audio never leaves this node as bytes. Clips are written into the workflow
+workspace and the node returns ``AudioRef`` values, for the reasons in
+``services/media/limits.py``.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from pydantic import BaseModel, Field
+
+from services.media import AudioRef, write_audio
+from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue
+
+from ..model._credentials import OpenAICredential, SarvamCredential
+from . import _config as speech_config
+from . import _unifier
+from ._base import provider_api_key, require_provider, track_usage
+from ._credentials import ElevenLabsCredential
+from ._protocol import TtsRequest
+from ._registry import tts_providers
+
+# Import for the registration side effect so the provider list below is
+# populated before the schema is built.
+from . import _providers # noqa: F401 isort:skip
+
+_PROVIDERS: List[str] = tts_providers()
+
+
+class TextToSpeechParams(BaseModel):
+ tool_name: str = Field(
+ default="text_to_speech",
+ description="Override name shown to the LLM when used as a tool.",
+ )
+ tool_description: str = Field(
+ default=(
+ "Convert text into spoken audio. Returns a reference to an audio "
+ "file saved in the workspace, not the audio itself. Use when the "
+ "user asks for speech, narration, a voiceover or an audio version "
+ "of some text."
+ ),
+ description="Override description shown to the LLM when used as a tool.",
+ json_schema_extra={"rows": 3},
+ )
+
+ provider: str = Field(
+ default=_PROVIDERS[0] if _PROVIDERS else "openai",
+ description="Speech provider to synthesize with.",
+ json_schema_extra={"enum": _PROVIDERS},
+ )
+ text: str = Field(
+ ...,
+ min_length=1,
+ description="Text to speak.",
+ json_schema_extra={"rows": 4},
+ )
+ # NOT `model` -- see the module docstring.
+ tts_model: str = Field(
+ default="",
+ description="Provider model id. Leave blank for the provider default.",
+ json_schema_extra={"loadOptionsMethod": "speechModels"},
+ )
+ voice: str = Field(
+ default="",
+ description="Voice id. Leave blank for the provider default.",
+ json_schema_extra={"loadOptionsMethod": "speechVoices"},
+ )
+ language: str = Field(
+ default="",
+ description=(
+ "Language or locale code. Required by Sarvam (e.g. hi-IN); "
+ "auto-detected by the others."
+ ),
+ )
+ speed: Optional[float] = Field(
+ default=None,
+ description=(
+ "Playback rate. Supported ranges differ per provider and values "
+ "outside them are clamped."
+ ),
+ )
+ output_format: str = Field(
+ default="",
+ description="Audio format. Leave blank for the provider default.",
+ )
+ sample_rate: Optional[int] = Field(
+ default=None,
+ description="Output sample rate in Hz, where the provider supports it.",
+ )
+ provider_options: Dict[str, Any] = Field(
+ default_factory=dict,
+ description=(
+ "Vendor-specific options passed through untouched — e.g. "
+ "ElevenLabs stability/similarity_boost, Sarvam pitch/loudness, "
+ "OpenAI instructions."
+ ),
+ )
+
+ model_config = {"extra": "ignore"}
+
+
+class TextToSpeechOutput(BaseModel):
+ audio: Optional[AudioRef] = Field(
+ default=None, description="The generated clip; the first when several."
+ )
+ files: List[AudioRef] = Field(
+ default_factory=list,
+ description="Every generated clip. More than one when the provider split long input.",
+ )
+ chunk_count: int = 0
+ provider: str = ""
+ tts_model: str = ""
+ voice: str = ""
+ request_id: Optional[str] = None
+ note: Optional[str] = None
+
+ model_config = {"extra": "allow"}
+
+
+class TextToSpeechNode(ActionNode):
+ type = "textToSpeech"
+ display_name = "Text to Speech"
+ subtitle = "Synthesize Speech"
+ group = ("language", "tool")
+ description = "Convert text to spoken audio using any configured speech provider"
+ component_kind = "square"
+ tool_name = "text_to_speech"
+ tool_description = (
+ "Convert text into spoken audio. Returns a reference to an audio file "
+ "saved in the workspace. Pick the provider and voice that suit the "
+ "language and tone requested."
+ )
+ handles = (
+ {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},
+ {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},
+ )
+ # First multi-credential node in the repo. ``ctx.connection(id)`` is a
+ # lookup over this tuple, so every provider's key is reachable and the
+ # selected one decides which is used.
+ credentials = (OpenAICredential, ElevenLabsCredential, SarvamCredential)
+ annotations = {"destructive": False, "readonly": False, "open_world": True}
+ task_queue = TaskQueue.REST_API
+ usable_as_tool = True
+ # `usable_as_tool` otherwise auto-hides both handles, which would break
+ # chaining this into Speech to Text on the canvas.
+ hide_input_handle = False
+ hide_output_handle = False
+ ui_hints = {"outputMode": "audio"}
+
+ Params = TextToSpeechParams
+ Output = TextToSpeechOutput
+
+ @Operation("synthesize")
+ async def synthesize(
+ self, ctx: NodeContext, params: TextToSpeechParams
+ ) -> TextToSpeechOutput:
+ provider = require_provider(params.provider, _PROVIDERS, "text-to-speech")
+ model = params.tts_model or speech_config.default_model(provider, "tts")
+
+ self._check_length(provider, model, params.text)
+
+ api_key = await provider_api_key(ctx, provider)
+ result = await _unifier.synthesize(
+ provider=provider,
+ api_key=api_key,
+ request=TtsRequest(
+ text=params.text,
+ model=model,
+ voice=params.voice,
+ language=params.language,
+ speed=params.speed,
+ output_format=params.output_format,
+ sample_rate=params.sample_rate,
+ provider_options=dict(params.provider_options),
+ ),
+ )
+
+ refs = [
+ write_audio(
+ payload.data,
+ ctx=ctx,
+ stem=params.text[:32],
+ ext=payload.format or "wav",
+ mime_type=payload.mime_type,
+ sample_rate=payload.sample_rate,
+ channels=payload.channels or 1,
+ )
+ for payload in result.audio
+ ]
+
+ await track_usage(
+ ctx,
+ provider=provider,
+ operation="text_to_speech",
+ units=result.billed_units,
+ unit=result.billed_unit,
+ )
+
+ note = None
+ if len(refs) > 1:
+ # Each clip carries its own container header, so concatenating
+ # the bytes produces a file that plays only the first chunk.
+ note = (
+ f"{provider} split this text into {len(refs)} clips. They are "
+ "separate playable files, not parts of one stream."
+ )
+
+ return TextToSpeechOutput(
+ audio=refs[0] if refs else None,
+ files=refs,
+ chunk_count=len(refs),
+ provider=provider,
+ tts_model=result.model or model,
+ voice=result.voice or params.voice,
+ request_id=result.request_id,
+ note=note,
+ )
+
+ @staticmethod
+ def _check_length(provider: str, model: str, text: str) -> None:
+ """Refuse over-long input before spending a paid call on it."""
+ cap = speech_config.max_input_chars(provider, model)
+ if cap and len(text) > cap:
+ raise NodeUserError(
+ f"Text is {len(text)} characters; {provider}/{model} accepts at "
+ f"most {cap}. Shorten it or split it across several runs."
+ )
diff --git a/server/routers/workspace.py b/server/routers/workspace.py
new file mode 100644
index 00000000..20f34189
--- /dev/null
+++ b/server/routers/workspace.py
@@ -0,0 +1,200 @@
+"""Per-workflow workspace file access.
+
+Two routes: serve a file out of a workflow's workspace, and accept an
+upload into it. Both exist because audio cannot travel through the workflow
+engine as bytes (see ``services/media/limits.py``) — the UI needs a URL to
+play generated audio from, and the file parameter needs somewhere to put an
+uploaded clip other than base64 inside the node's parameters.
+
+**The id/slug asymmetry is the thing to understand here.** The URL carries
+``workflow_id`` because an ``AudioRef`` deliberately stores the immutable id,
+so a reference keeps working after the workflow is renamed. But the
+directory on disk is named by ``Workflow.slug``, which the rename path
+moves. This router owns the id → slug lookup, because it is the layer that
+has a database; ``services.media`` stays synchronous and takes the resolved
+directory.
+
+Containment is delegated to ``resolve_within``, which rejects ``..`` / ``~``
+/ drive-prefixed input before touching the filesystem and re-checks
+containment after resolution so a symlink or Windows junction cannot
+redirect the result outside the root.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import mimetypes
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Optional
+
+from fastapi import APIRouter, Depends, File, HTTPException, Path as PathParam, UploadFile
+from fastapi.responses import FileResponse
+
+from core.container import container
+from core.database import Database
+from core.logging import get_logger
+from core.paths import workspaces_dir
+from services.media import MEDIA_MAX_UPLOAD_BYTES, AudioRef, write_audio
+from services.media.workspace import UPLOAD_SUBDIR
+
+logger = get_logger(__name__)
+
+router = APIRouter(prefix="/api/workspace", tags=["workspace"])
+
+# Types a browser may render in place. Everything else downloads.
+#
+# `text/html` and `image/svg+xml` are excluded deliberately and must stay
+# excluded: `shell`, `fileDownloader` and `fileModify` can all write
+# arbitrary files into a workspace, so serving attacker-authored markup
+# inline from the app origin would be stored XSS with access to the session
+# cookie. Both types are script-bearing.
+_INLINE_PREFIXES = ("audio/", "image/", "video/")
+_NEVER_INLINE = {"image/svg+xml", "text/html", "text/xml", "application/xhtml+xml"}
+
+# Bounded so a hostile Content-Length cannot make us allocate; the running
+# total is what enforces the cap, not the declared length.
+_UPLOAD_CHUNK_BYTES = 1024 * 1024
+
+
+def _db() -> Database:
+ return container.database()
+
+
+async def _workspace_root(workflow_id: str, database: Database) -> Path:
+ """Resolve a workflow id to its on-disk workspace directory.
+
+ Mirrors ``WorkflowService._resolve_workflow_slug`` including the
+ ``"default"`` fallback, which is the anonymous workspace a one-off run
+ without a saved row writes into.
+ """
+ slug: Optional[str] = None
+ try:
+ workflow = await database.get_workflow(workflow_id)
+ slug = getattr(workflow, "slug", None) if workflow else None
+ except Exception as exc:
+ logger.warning(
+ "workspace slug lookup failed", workflow_id=workflow_id, error=str(exc)
+ )
+ return workspaces_dir() / (slug or "default")
+
+
+def _resolve(root: Path, rel_path: str) -> Path:
+ """Contain ``rel_path`` under ``root`` or 404.
+
+ 404 rather than 403 throughout: a different status for "exists but
+ forbidden" would confirm the existence of files outside the workspace.
+ """
+ from nodes.filesystem._backend import resolve_within
+
+ try:
+ target = resolve_within(root, rel_path)
+ except ValueError:
+ raise HTTPException(status_code=404, detail="Not found")
+ if not target.is_file():
+ raise HTTPException(status_code=404, detail="Not found")
+ return target
+
+
[email protected]("/{workflow_id}/files/{file_path:path}")
+async def serve_workspace_file(
+ workflow_id: str = PathParam(..., min_length=1, max_length=128),
+ file_path: str = PathParam(..., min_length=1),
+ database: Database = Depends(_db),
+) -> FileResponse:
+ """Serve one file from a workflow's workspace.
+
+ Range requests work without any code here: Starlette's ``FileResponse``
+ already implements ``Accept-Ranges``, ``206``, ``Content-Range``,
+ ``If-Range`` and ``416``. Wrapping this in a ``StreamingResponse`` would
+ lose all of that and break seeking in an ``<audio>`` element.
+ """
+ root = await _workspace_root(workflow_id, database)
+ target = _resolve(root, file_path)
+
+ media_type = mimetypes.guess_type(target.name)[0] or "application/octet-stream"
+ inline = media_type.startswith(_INLINE_PREFIXES) and media_type not in _NEVER_INLINE
+ disposition = "inline" if inline else "attachment"
+
+ return FileResponse(
+ target,
+ media_type=media_type,
+ headers={
+ "Content-Disposition": f'{disposition}; filename="{target.name}"',
+ # Workspace files are mutable, so no immutable caching. The
+ # filename already carries a random suffix, which makes a short
+ # revalidating cache safe.
+ "Cache-Control": "private, max-age=0, must-revalidate",
+ "X-Content-Type-Options": "nosniff",
+ },
+ )
+
+
[email protected]("/{workflow_id}/uploads")
+async def upload_workspace_file(
+ workflow_id: str = PathParam(..., min_length=1, max_length=128),
+ file: UploadFile = File(...),
+ database: Database = Depends(_db),
+) -> AudioRef:
+ """Accept a file into the workflow's workspace and return a reference.
+
+ Read in bounded chunks with a running total rather than
+ ``await file.read()``: the declared ``Content-Length`` is attacker
+ controlled, so the only trustworthy limit is what has actually been
+ read. Exceeding the cap aborts immediately, before the whole body has
+ been received.
+
+ Returns an ``AudioRef`` — the shape ``coerce_file_param`` already
+ documents as "what the upload route now returns", so the file parameter
+ on a node accepts it with no further plumbing.
+ """
+ root = await _workspace_root(workflow_id, database)
+
+ digest = hashlib.sha256()
+ chunks: list[bytes] = []
+ total = 0
+ while True:
+ chunk = await file.read(_UPLOAD_CHUNK_BYTES)
+ if not chunk:
+ break
+ total += len(chunk)
+ if total > MEDIA_MAX_UPLOAD_BYTES:
+ raise HTTPException(
+ status_code=413,
+ detail=(
+ f"Upload exceeds the {MEDIA_MAX_UPLOAD_BYTES // (1024 * 1024)} MB "
+ "limit."
+ ),
+ )
+ digest.update(chunk)
+ chunks.append(chunk)
+
+ if not total:
+ raise HTTPException(status_code=400, detail="Uploaded file is empty.")
+
+ payload = b"".join(chunks)
+ name = Path(file.filename or "upload.bin").name
+ stem, _, ext = name.rpartition(".")
+
+ # write_audio owns filename sanitization (Windows reserved device names,
+ # slugging, a random suffix so nothing collides or overwrites) and the
+ # atomic write. Reimplementing any of that here would be a second, worse
+ # copy.
+ ref = write_audio(
+ payload,
+ ctx=SimpleNamespace(
+ workspace_dir=str(root), node_id="upload", workflow_id=workflow_id
+ ),
+ stem=stem or name,
+ ext=ext or "bin",
+ mime_type=file.content_type or None,
+ subdir=UPLOAD_SUBDIR,
+ )
+
+ logger.info(
+ "workspace upload stored",
+ workflow_id=workflow_id,
+ path=ref.path,
+ size_bytes=ref.size_bytes,
+ )
+ return ref
diff --git a/server/services/media/workspace.py b/server/services/media/workspace.py
index 3bb2ee38..3617b13c 100644
--- a/server/services/media/workspace.py
+++ b/server/services/media/workspace.py
@@ -51,15 +51,37 @@ def _slugify(value: str, *, limit: int = 32) -> str:
return slug
-def workspace_root(ctx: Any = None, *, workflow_id: Optional[str] = None) -> Path:
- """Resolve the workspace directory for a context or workflow id.
-
- Prefers the executor-injected ``ctx.workspace_dir``; falls back to
- composing it from the workflow's slug so the HTTP routes -- which have
- no NodeContext -- can resolve the same directory.
+def workspace_root(
+ ctx: Any = None,
+ *,
+ workspace_dir: Optional[str] = None,
+ workflow_id: Optional[str] = None,
+) -> Path:
+ """Resolve the workspace directory for a context.
+
+ Order: an explicit ``workspace_dir``, then the executor-injected
+ ``ctx.workspace_dir``. Callers with neither get an error rather than a
+ guess.
+
+ ``workflow_id`` is accepted for call-site readability but is **not**
+ sufficient on its own, and that is deliberate. Workspace directories are
+ named by ``Workflow.slug`` (see ``WorkflowService._get_workspace_dir``),
+ while an ``AudioRef`` deliberately carries the immutable ``workflow_id``
+ so a reference survives a rename. Turning one into the other needs a
+ database read, which cannot happen in a sync helper -- so the caller
+ that has a database (the workspace HTTP route) does the lookup and
+ passes the resolved directory in via ``workspace_dir``.
+
+ An earlier version composed ``workspaces/<workflow_id>/`` here. That
+ silently produced a path that never exists, because the directory on
+ disk is ``workspaces/<slug>/``. It never fired in practice only because
+ every caller happened to supply a ctx.
"""
from services.plugin import NodeUserError
+ if workspace_dir:
+ return Path(workspace_dir)
+
direct = getattr(ctx, "workspace_dir", None) if ctx is not None else None
if not direct and ctx is not None:
raw = getattr(ctx, "raw", None)
@@ -70,20 +92,18 @@ def workspace_root(ctx: Any = None, *, workflow_id: Optional[str] = None) -> Pat
if workflow_id is None and ctx is not None:
workflow_id = getattr(ctx, "workflow_id", None)
- if not workflow_id:
- raise NodeUserError(
- "No workspace is available for this execution, so media cannot be "
- "read or written. Save the workflow and run it again."
- )
- from core.paths import workspace_dir as _workspace_dir
-
- return Path(_workspace_dir(workflow_id))
+ raise NodeUserError(
+ "No workspace is available for this execution, so media cannot be "
+ "read or written. Save the workflow and run it again."
+ + (f" (workflow {workflow_id})" if workflow_id else "")
+ )
def resolve_media(
ref: AudioRef | str,
*,
ctx: Any = None,
+ workspace_dir: Optional[str] = None,
workflow_id: Optional[str] = None,
) -> Path:
"""Resolve a ref or a path string to a contained absolute path."""
@@ -97,20 +117,19 @@ def resolve_media(
if isinstance(ref, AudioRef) and ref.workflow_id and workflow_id is None:
workflow_id = ref.workflow_id
+ root = workspace_root(ctx, workspace_dir=workspace_dir, workflow_id=workflow_id)
+
candidate = Path(key)
if candidate.is_absolute():
# Tolerated for back-compat with nodes that stored absolute paths
# before AudioRef existed, but still contained: an absolute path
# outside the workspace is refused rather than read.
- root = workspace_root(ctx, workflow_id=workflow_id)
try:
key = candidate.resolve(strict=False).relative_to(root).as_posix()
except ValueError as exc:
raise NodeUserError(
f"'{candidate}' is outside this workflow's workspace."
) from exc
- else:
- root = workspace_root(ctx, workflow_id=workflow_id)
try:
return resolve_within(root, key)
@@ -122,13 +141,16 @@ def read_media_bytes(
ref: AudioRef | str,
*,
ctx: Any = None,
+ workspace_dir: Optional[str] = None,
workflow_id: Optional[str] = None,
max_bytes: int = MEDIA_MAX_READ_BYTES,
) -> Tuple[str, bytes]:
"""Return ``(filename, bytes)`` for a contained media reference."""
from services.plugin import NodeUserError
- target = resolve_media(ref, ctx=ctx, workflow_id=workflow_id)
+ target = resolve_media(
+ ref, ctx=ctx, workspace_dir=workspace_dir, workflow_id=workflow_id
+ )
if not target.is_file():
raise NodeUserError(f"Media file not found: {target.name}")
diff --git a/server/skills/language_agent/sarvam-speech-skill/SKILL.md b/server/skills/language_agent/sarvam-speech-skill/SKILL.md
deleted file mode 100644
index 8b22ce0d..00000000
--- a/server/skills/language_agent/sarvam-speech-skill/SKILL.md
+++ /dev/null
@@ -1,121 +0,0 @@
----
-name: sarvam-speech-skill
-description: Transcribe speech to text and synthesize natural speech in Indian languages using Sarvam AI's Saaras and Bulbul models.
-allowed-tools: sarvam_speech_to_text sarvam_text_to_speech
-metadata:
- author: opencompany
- version: "1.0"
- category: language
-
----
-
-# Sarvam Speech Skill
-
-Speech-to-text across 23 Indian languages (Saaras) and text-to-speech with
-37 voices (Bulbul).
-
-## How It Works
-
-This skill provides instructions and context. To execute, connect the
-**Sarvam Speech to Text** and/or **Sarvam Text to Speech** nodes to the
-agent's `input-tools` handle.
-
-## sarvam_speech_to_text
-
-| Field | Type | Required | Description |
-|---|---|---|---|
-| audio_file | string | Yes | Path to an audio file |
-| model | enum | No | `saaras:v3` (default, 23 languages) or `saarika:v2.5` (legacy) |
-| language_code | string | No | BCP-47 code, or `unknown` (default) to auto-detect |
-| mode | enum | No | `transcribe` (default), `translate`, `verbatim`, `translit`, `codemix` |
-
-Modes:
-- `transcribe` — native script, cleaned up
-- `translate` — English output regardless of the spoken language
-- `verbatim` — keeps disfluencies ("um", repetitions); use for compliance
- or verbatim-record work
-- `translit` — Roman script
-- `codemix` — preserves mixed-language speech as spoken
-
-Everything except `transcribe` and `translate` requires `saaras:v3`.
-
-```json
-{
- "transcript": "नमस्ते, आप कैसे हैं?",
- "language_code": "hi-IN",
- "language_probability": 0.97
-}
-```
-
-**Hard limit: clips under 30 seconds.** This node calls the synchronous
-endpoint. Longer recordings need Sarvam's Batch API, which is not wired
-here — split the audio first, or tell the user it is out of scope rather
-than retrying.
-
-Formats: WAV, MP3, AAC, AIFF, OGG, OPUS, FLAC, MP4/M4A, AMR, WMA, WebM,
-PCM. Accuracy is best at a 16 kHz sample rate.
-
-## sarvam_text_to_speech
-
-| Field | Type | Required | Description |
-|---|---|---|---|
-| text | string | Yes | Text to speak (max 2500 chars on v3, 1500 on v2) |
-| target_language_code | enum | No | Defaults to `hi-IN` |
-| model | enum | No | `bulbul:v3` (default) or `bulbul:v2` |
-| speaker | string | No | Voice name, lowercase. Defaults to `shubh` |
-| pace | number | No | v3: 0.5–2.0, v2: 0.3–3.0. Default 1.0 |
-| speech_sample_rate | enum | No | 8000–48000 Hz, default 24000 |
-| output_audio_codec | enum | No | `wav` (default), `mp3`, `flac`, `aac`, `opus`, `linear16`, `mulaw`, `alaw` |
-| return_audio | enum | No | `file` (default) or `base64` |
-
-Voices are **per model** and do not overlap:
-- `bulbul:v3` — shubh, aditya, ritu, priya, neha, rahul, pooja, rohan,
- simran, kavya, amit, dev, ishita, shreya, ratan, varun, manan, sumit,
- roopa, kabir, aayan, ashutosh, advait, anand, tanya, tarun, sunny, mani,
- gokul, vijay, shruti, suhani, mohit, kavitha, rehan, soham, rupali
-- `bulbul:v2` — anushka, manisha, vidya, arya, abhilash, karun, hitesh
-
-Asking for a v2 voice on v3 is an error, not a fallback.
-
-```json
-{
- "file_path": "/…/workspaces/My_Flow_1/audio/namaste-a1b2c3d4-9f8e7d.wav",
- "files": ["/…/namaste-a1b2c3d4-9f8e7d.wav"],
- "chunk_count": 1,
- "audio_format": "wav"
-}
-```
-
-**The audio is written to a file, not returned inline.** Pass `file_path`
-along to whatever consumes it. Only set `return_audio: "base64"` for very
-short clips you must inline — a full-length request is roughly 12 MB of
-base64, which bloats the workflow output and your own context.
-
-For long text, Sarvam may return several chunks. Each is a standalone audio
-file with its own header, so `files` lists them all and `file_path` points
-at the first. Concatenating them byte-wise produces an unplayable file.
-
-## When to Use
-
-- Transcribing voice notes, IVR recordings or short call clips
-- Generating voice prompts and announcements in Indian languages
-- Building a voice loop: transcribe → reason → synthesize a reply
-
-## When NOT to Use
-
-- Recordings longer than 30 seconds (Batch API territory)
-- Speaker diarization — only the Batch API returns it
-- Real-time streaming — this is the request/response endpoint
-- Languages outside the Indian set plus English
-
-## API Details
-
-- **Speech to Text**: POST `https://api.sarvam.ai/speech-to-text` (multipart) — 30 INR/hour
-- **Text to Speech**: POST `https://api.sarvam.ai/text-to-speech` — 30 INR / 10K chars (v3)
-- **Auth**: `api-subscription-key` header
-
-## Setup Requirements
-
-A single Sarvam API key from [dashboard.sarvam.ai](https://dashboard.sarvam.ai),
-entered once in the Credentials modal under **Sarvam AI**. The same key
-covers speech, translation and the Sarvam chat models.
diff --git a/server/skills/language_agent/sarvam-translate-skill/SKILL.md b/server/skills/language_agent/sarvam-translate-skill/SKILL.md
index 0ee8462f..b3fb302f 100644
--- a/server/skills/language_agent/sarvam-translate-skill/SKILL.md
+++ b/server/skills/language_agent/sarvam-translate-skill/SKILL.md
@@ -99,7 +99,7 @@ language was detected.
- Localising content into Indian languages
- Normalising user input typed in Roman script into native script
- Routing a message to the right handler based on its language
-- Preparing text for `sarvam_text_to_speech`, which needs a target language
+- Preparing text for `text_to_speech`, which needs a target language
## When NOT to Use
diff --git a/server/skills/language_agent/speech-skill/SKILL.md b/server/skills/language_agent/speech-skill/SKILL.md
new file mode 100644
index 00000000..8a494f19
--- /dev/null
+++ b/server/skills/language_agent/speech-skill/SKILL.md
@@ -0,0 +1,111 @@
+---
+name: speech-skill
+description: Convert text to spoken audio and transcribe audio to text, across multiple speech providers (OpenAI, ElevenLabs, Deepgram, Groq, Sarvam AI).
+allowed-tools: text_to_speech speech_to_text
+metadata:
+ author: opencompany
+ version: "2.0"
+ category: language
+
+---
+
+# Speech Skill
+
+Two tools, several providers behind each. Pick the provider that fits the
+job rather than defaulting to one.
+
+## How It Works
+
+Connect the **Text to Speech** and/or **Speech to Text** nodes to the
+agent's `input-tools` handle. Each node has a `provider` parameter, and the
+credential for the chosen provider must be configured in the Credentials
+modal.
+
+Audio never travels as data. Synthesis writes a file into the workflow
+workspace and returns a **reference** to it; transcription accepts a path or
+a reference produced upstream. So Text to Speech can be wired straight into
+Speech to Text, and neither tool ever puts audio bytes into the
+conversation.
+
+## text_to_speech
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| text | string | Yes | The text to speak |
+| provider | enum | No | `openai` (default), `elevenlabs`, `sarvam` |
+| tts_model | string | No | Provider model id; blank uses the provider default |
+| voice | string | No | Voice id; blank uses the provider default |
+| language | string | No | Locale code. Required for Sarvam (e.g. `hi-IN`); auto-detected elsewhere |
+| speed | number | No | Playback rate. Clamped to whatever the provider allows |
+| output_format | string | No | Audio format; blank uses the provider default |
+| provider_options | object | No | Vendor-specific extras, passed through untouched |
+
+Returns `audio` — a reference carrying `path`, `url`, `duration_seconds` and
+`mime_type` — plus `files` and `chunk_count`.
+
+**When several clips come back**, they are separate playable files, not
+parts of one stream. Each carries its own container header, so concatenating
+them produces audio that plays only the first chunk.
+
+### Choosing a provider
+
+- **openai** — solid general-purpose quality, 13 voices, inexpensive. Put
+ `instructions` in `provider_options` to steer tone, but note it works only
+ on `gpt-4o-mini-tts` and is ignored on `tts-1`.
+- **elevenlabs** — the most natural and controllable. Requires an explicit
+ `voice`; there is no account-wide default. `stability`,
+ `similarity_boost`, `style` and `use_speaker_boost` go in
+ `provider_options`.
+- **sarvam** — Indian languages: 11 locales, 37 voices on `bulbul:v3`.
+ `language` is required. `pitch` / `loudness` work on `bulbul:v2` only and
+ `temperature` on `bulbul:v3` only; the wrong one for the model is dropped
+ rather than sent.
+
+## speech_to_text
+
+| Field | Type | Required | Description |
+|---|---|---|---|
+| audio_file | string / reference | Yes | Workspace path, or a reference from an upstream node |
+| provider | enum | No | `openai` (default), `deepgram`, `groq`, `sarvam` |
+| stt_model | string | No | Provider model id; blank uses the provider default |
+| language | string | No | Language hint; blank auto-detects |
+| translate | boolean | No | Translate to English instead of transcribing in-language |
+| diarize | boolean | No | Label speakers, where supported |
+| timestamps | boolean | No | Per-word timing, where supported |
+| provider_options | object | No | Vendor-specific extras, passed through untouched |
+
+Returns `transcript`, `language`, `duration_seconds`, and — when asked for
+and supported — `words` and `segments`.
+
+### Choosing a provider
+
+- **openai** — reliable default. Word timestamps require `whisper-1`; the
+ `gpt-4o-transcribe` models return plain JSON only, so a timestamp request
+ is quietly downgraded rather than failing.
+- **deepgram** — best for long recordings, diarization and keyword boosting.
+ Billed per minute.
+- **groq** — fastest and cheapest in bulk, but it bills a 10-second minimum
+ per request, so many tiny clips cost more than their duration suggests.
+ `whisper-large-v3-turbo` cannot translate; use `whisper-large-v3` when
+ `translate` is set.
+- **sarvam** — Indian languages. Timestamps and diarization are unavailable
+ on its synchronous endpoint and come back empty.
+
+## When to Use
+
+- Narration, voiceovers, or "read this out"
+- Working out what was said in a recording
+- A voice loop: transcribe, reason over the text, synthesize a reply
+
+## When NOT to Use
+
+- Real-time or streaming speech — these tools are batch only
+- Translating text you already have — use a translation tool
+- Very long recordings on a synchronous provider; split them first
+
+## Setup Requirements
+
+One API key per provider you intend to use, added in the Credentials modal.
+Nothing is shared between them: an ElevenLabs key does not enable Deepgram.
+The `openai`, `groq` and `sarvam` keys are the same ones their chat models
+use, so if those are already configured, speech works with no extra setup.
diff --git a/server/tests/media/test_audio_ref.py b/server/tests/media/test_audio_ref.py
index a3a266bc..67e32311 100644
--- a/server/tests/media/test_audio_ref.py
+++ b/server/tests/media/test_audio_ref.py
@@ -164,6 +164,45 @@ class TestContainment:
read_media_bytes(ref, ctx=_ctx(beta, workflow_id="beta"))
+class TestWorkspaceRootRefusesToGuess:
+ """Regression: a workflow_id is not a workspace directory name.
+
+ Directories are named by ``Workflow.slug``, which changes on rename,
+ while an ``AudioRef`` carries the immutable ``workflow_id`` on purpose.
+ Composing ``workspaces/<workflow_id>/`` produced a path that never
+ exists. It stayed invisible because every caller happened to pass a
+ ctx; the workspace HTTP route is the first that cannot.
+ """
+
+ def test_workflow_id_alone_is_refused(self):
+ from services.media.workspace import workspace_root
+ from services.plugin import NodeUserError
+
+ with pytest.raises(NodeUserError):
+ workspace_root(workflow_id="019f99e2-dc99-7cf3-9019-3ad3e6260de1")
+
+ def test_no_ctx_and_no_hint_is_refused(self):
+ from services.media.workspace import workspace_root
+ from services.plugin import NodeUserError
+
+ with pytest.raises(NodeUserError):
+ workspace_root()
+
+ def test_explicit_workspace_dir_wins(self, tmp_path):
+ """How the HTTP route passes a directory it resolved via the DB."""
+ from services.media.workspace import workspace_root
+
+ assert workspace_root(workspace_dir=str(tmp_path)) == tmp_path
+
+ def test_explicit_dir_lets_a_ref_resolve_without_a_ctx(self, tmp_path):
+ ctx = _ctx(tmp_path)
+ ref = write_audio(_wav_bytes(), ctx=ctx, stem="x", ext="wav")
+
+ name, blob = read_media_bytes(ref, workspace_dir=str(tmp_path))
+ assert blob == read_media_bytes(ref, ctx=ctx)[1]
+ assert name.endswith(".wav")
+
+
class TestCoerceFileParam:
def test_accepts_an_audio_ref(self, tmp_path):
ctx = _ctx(tmp_path)
diff --git a/server/tests/nodes/test_sarvam.py b/server/tests/nodes/test_sarvam.py
index bc666591..715441c1 100644
--- a/server/tests/nodes/test_sarvam.py
+++ b/server/tests/nodes/test_sarvam.py
@@ -1,12 +1,15 @@
-"""Contract tests for the five Sarvam service nodes.
+"""Contract tests for the three Sarvam text service nodes.
-sarvamTranslate / sarvamTransliterate / sarvamDetectLanguage /
-sarvamSpeechToText / sarvamTextToSpeech.
+sarvamTranslate / sarvamTransliterate / sarvamDetectLanguage.
-All five authenticate through ``ctx.connection("sarvam")``, which injects
+All three authenticate through ``ctx.connection("sarvam")``, which injects
the ``api-subscription-key`` header from the shared credential — the same
key the OpenAI-compatible ``sarvamChatModel`` uses. httpx is mocked with
respx so no network is touched.
+
+Sarvam's speech endpoints are covered by ``test_speech.py`` instead: they
+are reached through the provider-abstracted ``textToSpeech`` /
+``speechToText`` nodes now, with Sarvam as one provider among several.
"""
from __future__ import annotations
@@ -237,293 +240,6 @@ class TestSarvamDetectLanguage:
assert "1000" in result["error"]
-# ============================================================================
-# sarvamSpeechToText
-# ============================================================================
-
-
-class TestSarvamSpeechToText:
- URL = f"{_BASE}/speech-to-text"
-
- @respx.mock
- async def test_accepts_a_file_path(self, harness, tmp_path):
- clip = tmp_path / "clip.wav"
- clip.write_bytes(b"RIFF....WAVE")
- respx.post(self.URL).mock(
- return_value=httpx.Response(
- 200,
- json={
- "transcript": "hello",
- "language_code": "en-IN",
- "language_probability": 0.98,
- },
- )
- )
-
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamSpeechToText", {"audio_file": str(clip)}
- )
-
- harness.assert_envelope(result, success=True)
- assert result["result"]["transcript"] == "hello"
-
- sent = respx.calls.last.request
- assert sent.headers["content-type"].startswith("multipart/form-data")
- assert b"RIFF" in sent.content
-
- @respx.mock
- async def test_accepts_the_file_widget_upload_envelope(self, harness):
- """The file widget sends {type, data, filename, mimeType}, not a path."""
- respx.post(self.URL).mock(
- return_value=httpx.Response(200, json={"transcript": "ok"})
- )
-
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamSpeechToText",
- {
- "audio_file": {
- "type": "upload",
- "data": base64.b64encode(b"AUDIOBYTES").decode(),
- "filename": "voice.mp3",
- "mimeType": "audio/mpeg",
- }
- },
- )
-
- harness.assert_envelope(result, success=True)
- assert b"AUDIOBYTES" in respx.calls.last.request.content
-
- @respx.mock
- async def test_relative_path_resolves_against_the_workspace(self, harness, tmp_path):
- (tmp_path / "audio").mkdir()
- (tmp_path / "audio" / "a.wav").write_bytes(b"WAV")
- respx.post(self.URL).mock(
- return_value=httpx.Response(200, json={"transcript": "ok"})
- )
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamSpeechToText", {"audio_file": "audio/a.wav"}, context=ctx
- )
-
- harness.assert_envelope(result, success=True)
-
- async def test_missing_file_is_a_user_error(self, harness):
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamSpeechToText", {"audio_file": "/nope/missing.wav"}
- )
-
- harness.assert_envelope(result, success=False)
- assert "not found" in result["error"].lower()
-
- async def test_empty_audio_file_param_is_a_user_error(self, harness):
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute("sarvamSpeechToText", {})
-
- harness.assert_envelope(result, success=False)
- assert "audio_file is required" in result["error"]
-
- async def test_advanced_mode_requires_saaras_v3(self, harness, tmp_path):
- clip = tmp_path / "c.wav"
- clip.write_bytes(b"WAV")
-
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamSpeechToText",
- {
- "audio_file": str(clip),
- "model": "saarika:v2.5",
- "mode": "codemix",
- },
- )
-
- harness.assert_envelope(result, success=False)
- assert "saaras:v3" in result["error"]
-
-
-# ============================================================================
-# sarvamTextToSpeech
-# ============================================================================
-
-
-class TestSarvamTextToSpeech:
- URL = f"{_BASE}/text-to-speech"
-
- @staticmethod
- def _audio_response(chunks: int = 1) -> httpx.Response:
- blob = base64.b64encode(b"RIFFfakeWAVEdata").decode()
- return httpx.Response(
- 200, json={"request_id": "r", "audios": [blob] * chunks}
- )
-
- @respx.mock
- async def test_writes_to_the_workspace_by_default(self, harness, tmp_path):
- respx.post(self.URL).mock(return_value=self._audio_response())
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech",
- {"text": "Namaste", "target_language_code": "hi-IN"},
- context=ctx,
- )
-
- harness.assert_envelope(result, success=True)
- payload = result["result"]
- written = Path(payload["file_path"])
- assert written.is_file()
- assert written.parent == tmp_path / "audio"
- assert written.read_bytes() == b"RIFFfakeWAVEdata"
- assert payload["chunk_count"] == 1
- assert payload["files"] == [str(written)]
- # The DB/WS bloat guard: no inline base64 unless asked for.
- assert payload.get("audio_base64") is None
-
- @respx.mock
- async def test_base64_mode_returns_inline_and_writes_nothing(self, harness, tmp_path):
- respx.post(self.URL).mock(return_value=self._audio_response())
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech",
- {"text": "Namaste", "return_audio": "base64"},
- context=ctx,
- )
-
- payload = result["result"]
- assert payload["audio_base64"] == base64.b64encode(b"RIFFfakeWAVEdata").decode()
- assert payload.get("file_path") is None
- assert not (tmp_path / "audio").exists()
-
- @respx.mock
- async def test_multi_chunk_writes_one_file_per_chunk(self, harness, tmp_path):
- respx.post(self.URL).mock(return_value=self._audio_response(chunks=3))
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech", {"text": "long text"}, context=ctx
- )
-
- payload = result["result"]
- assert payload["chunk_count"] == 3
- assert len(payload["files"]) == 3
- assert all(Path(p).is_file() for p in payload["files"])
- assert payload["file_path"] == payload["files"][0]
- assert "3 audio chunks" in payload["note"]
-
- @respx.mock
- async def test_oversize_inline_audio_is_suppressed_with_a_note(
- self, harness, tmp_path, monkeypatch
- ):
- """Never silently truncate — say why and point at the file mode."""
- monkeypatch.setattr(
- "nodes.sarvam.sarvam_text_to_speech._MAX_INLINE_B64", 4
- )
- respx.post(self.URL).mock(return_value=self._audio_response())
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech",
- {"text": "hi", "return_audio": "base64"},
- context=ctx,
- )
-
- payload = result["result"]
- assert payload["audio_base64"] is None
- assert "return_audio='file'" in payload["note"]
-
- @respx.mock
- async def test_v2_only_params_are_not_sent_for_v3(self, harness, tmp_path):
- import json
-
- respx.post(self.URL).mock(return_value=self._audio_response())
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- await harness.execute(
- "sarvamTextToSpeech",
- {"text": "hi", "model": "bulbul:v3", "speaker": "shubh"},
- context=ctx,
- )
-
- body = json.loads(respx.calls.last.request.content)
- assert "pitch" not in body
- assert "loudness" not in body
- assert "enable_preprocessing" not in body
-
- @respx.mock
- async def test_v2_params_are_sent_for_v2(self, harness, tmp_path):
- import json
-
- respx.post(self.URL).mock(return_value=self._audio_response())
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- await harness.execute(
- "sarvamTextToSpeech",
- {
- "text": "hi",
- "model": "bulbul:v2",
- "speaker": "anushka",
- "pitch": 0.2,
- },
- context=ctx,
- )
-
- body = json.loads(respx.calls.last.request.content)
- assert body["pitch"] == 0.2
- assert "temperature" not in body
-
- async def test_speaker_from_the_wrong_model_is_a_user_error(self, harness):
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech",
- {"text": "hi", "model": "bulbul:v3", "speaker": "anushka"},
- )
-
- harness.assert_envelope(result, success=False)
- assert "not a bulbul:v3 voice" in result["error"]
-
- async def test_text_over_model_cap_is_a_user_error(self, harness):
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech",
- {"text": "x" * 1501, "model": "bulbul:v2", "speaker": "anushka"},
- )
-
- harness.assert_envelope(result, success=False)
- assert "1500" in result["error"]
-
- async def test_v3_pace_out_of_range_is_a_user_error(self, harness):
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech", {"text": "hi", "pace": 2.5}
- )
-
- harness.assert_envelope(result, success=False)
- assert "0.5-2.0" in result["error"]
-
- @respx.mock
- async def test_empty_audio_array_is_a_user_error(self, harness, tmp_path):
- respx.post(self.URL).mock(return_value=httpx.Response(200, json={"audios": []}))
-
- ctx = harness.build_context(workspace_dir=str(tmp_path))
- with patched_container(auth_api_keys=_KEYS), patched_pricing():
- result = await harness.execute(
- "sarvamTextToSpeech", {"text": "hi"}, context=ctx
- )
-
- harness.assert_envelope(result, success=False)
- assert "no audio" in result["error"].lower()
-
-
# ============================================================================
# Cross-node contracts
# ============================================================================
@@ -534,8 +250,6 @@ class TestSarvamPluginContract:
"sarvamTranslate",
"sarvamTransliterate",
"sarvamDetectLanguage",
- "sarvamSpeechToText",
- "sarvamTextToSpeech",
)
@pytest.mark.parametrize("node_type", NODE_TYPES)
diff --git a/server/tests/nodes/test_speech.py b/server/tests/nodes/test_speech.py
new file mode 100644
index 00000000..2e55ac4f
--- /dev/null
+++ b/server/tests/nodes/test_speech.py
@@ -0,0 +1,641 @@
+"""Contract for the provider-abstracted speech nodes.
+
+The interesting tests here are the wire-shape ones. Every provider in the v1
+set diverges from the others on auth, request transport and response
+transport, and three of those divergences fail *silently* when got wrong:
+ElevenLabs ignores a body-placed ``output_format`` and returns the default,
+Deepgram ignores body-placed options entirely, and Sarvam accepts a request
+whose base64 array is then mis-parsed. None of them produce an error, so
+only an assertion on the outgoing request catches a regression.
+"""
+
+from __future__ import annotations
+
+import base64
+import struct
+import subprocess
+import sys
+import wave
+from pathlib import Path
+
+import httpx
+import pytest
+import respx
+
+from ._mocks import patched_container, patched_pricing
+
+pytestmark = pytest.mark.node_contract
+
+SERVER_DIR = Path(__file__).resolve().parents[2]
+
+_KEYS = {
+ "openai": "sk-test",
+ "groq": "gsk-test",
+ "elevenlabs": "el-test",
+ "deepgram": "dg-test",
+ "sarvam": "sv-test",
+}
+
+
+def _wav(seconds: float = 0.5, rate: int = 8000) -> bytes:
+ import io
+
+ buffer = io.BytesIO()
+ with wave.open(buffer, "wb") as handle:
+ handle.setnchannels(1)
+ handle.setsampwidth(2)
+ handle.setframerate(rate)
+ handle.writeframes(struct.pack("<h", 0) * int(rate * seconds))
+ return buffer.getvalue()
+
+
+def _clip(harness, tmp_path, *, payload: bytes | None = None, name: str = "a.wav"):
+ """Put a clip inside the workspace and return ``(context, relative_name)``.
+
+ Audio has to live *in* the workspace to be readable: containment refuses
+ absolute paths outside it, which is the whole point of the design. Tests
+ that hand the node a bare ``tmp_path`` are testing the guard, not the
+ provider.
+ """
+ (tmp_path / name).write_bytes(payload if payload is not None else _wav())
+ return harness.build_context(workspace_dir=str(tmp_path)), name
+
+
+# ============================================================================
+# Registry — direction membership IS the capability
+# ============================================================================
+
+
+class TestDirectionMembership:
+ def test_transcription_only_provider_is_absent_from_synthesis(self):
+ from nodes.speech._registry import tts_providers
+
+ assert "deepgram" not in tts_providers()
+ assert "groq" not in tts_providers()
+
+ def test_synthesis_only_provider_is_absent_from_transcription(self):
+ from nodes.speech._registry import stt_providers
+
+ assert "elevenlabs" not in stt_providers()
+
+ def test_dual_direction_provider_is_in_both(self):
+ from nodes.speech._registry import stt_providers, tts_providers
+
+ for provider in ("openai", "sarvam"):
+ assert provider in tts_providers()
+ assert provider in stt_providers()
+
+ def test_every_provider_resolves_to_a_registered_credential(self):
+ """The layering contract: config names credential *ids*, not classes."""
+ from services.plugin.credential import CREDENTIAL_REGISTRY
+
+ from nodes.speech import _config
+ from nodes.speech._registry import stt_providers, tts_providers
+
+ for provider in set(tts_providers()) | set(stt_providers()):
+ credential = _config.credential_id(provider)
+ assert credential in CREDENTIAL_REGISTRY, (
+ f"{provider} maps to credential {credential!r}, which is not registered"
+ )
+
+
+# ============================================================================
+# Config — the per-model override ladder
+# ============================================================================
+
+
+class TestCapabilityResolution:
+ def test_exact_model_override_wins(self):
+ from nodes.speech import _config
+
+ assert "verbose_json" in _config.response_formats("openai", "whisper-1")
+ assert _config.response_formats("openai", "gpt-4o-transcribe") == [
+ "json",
+ "text",
+ ]
+
+ def test_prefix_match_covers_dated_snapshots(self):
+ """A dated release must not silently fall through to _default."""
+ from nodes.speech import _config
+
+ assert _config.response_formats(
+ "openai", "gpt-4o-mini-transcribe-2025-12-15"
+ ) == ["json", "text"]
+
+ def test_declared_null_is_distinguishable_from_absent(self):
+ """Deepgram declares no upload cap; that is a fact, not a gap."""
+ from nodes.speech import _config
+
+ assert (
+ _config.capability("deepgram", "stt", "max_upload_bytes", default="ABSENT")
+ is None
+ )
+ assert (
+ _config.capability("deepgram", "stt", "nonexistent", default="ABSENT")
+ == "ABSENT"
+ )
+
+ def test_boolean_capabilities_default_permissive(self):
+ from nodes.speech import _config
+
+ assert _config.supports("openai", "tts", "never_declared_flag") is True
+
+ def test_disjoint_voice_sets_per_model(self):
+ from nodes.speech import _config
+
+ v2 = set(_config.voices("sarvam", model="bulbul:v2"))
+ v3 = set(_config.voices("sarvam", model="bulbul:v3"))
+ assert v2 and v3 and not (v2 & v3)
+
+
+# ============================================================================
+# Wire shapes — the silent-failure guards
+# ============================================================================
+
+
+class TestElevenLabsWireShape:
+ @respx.mock
+ async def test_output_format_rides_the_query_string_not_the_body(
+ self, harness, tmp_path
+ ):
+ """A body-placed output_format is ignored and the default returned."""
+ route = respx.post(
+ url__regex=r"https://api\.elevenlabs\.io/v1/text-to-speech/.*"
+ ).mock(return_value=httpx.Response(200, content=b"ID3audio"))
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "textToSpeech",
+ {
+ "provider": "elevenlabs",
+ "text": "hello",
+ "voice": "voice123",
+ "output_format": "mp3_22050_32",
+ },
+ context=harness.build_context(workspace_dir=str(tmp_path)),
+ )
+
+ harness.assert_envelope(result, success=True)
+ request = route.calls.last.request
+ assert request.url.params["output_format"] == "mp3_22050_32"
+ assert b"output_format" not in request.content
+
+ @respx.mock
+ async def test_voice_is_a_path_segment_and_auth_has_no_scheme(self, harness):
+ route = respx.post(
+ url__regex=r"https://api\.elevenlabs\.io/v1/text-to-speech/.*"
+ ).mock(return_value=httpx.Response(200, content=b"audio"))
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute(
+ "textToSpeech",
+ {"provider": "elevenlabs", "text": "hi", "voice": "abc123"},
+ )
+
+ request = route.calls.last.request
+ assert request.url.path.endswith("/v1/text-to-speech/abc123")
+ assert request.headers["xi-api-key"] == "el-test"
+ assert "authorization" not in request.headers
+
+ @respx.mock
+ async def test_speed_is_clamped_to_the_api_range(self, harness):
+ route = respx.post(
+ url__regex=r"https://api\.elevenlabs\.io/v1/text-to-speech/.*"
+ ).mock(return_value=httpx.Response(200, content=b"audio"))
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute(
+ "textToSpeech",
+ {
+ "provider": "elevenlabs",
+ "text": "hi",
+ "voice": "v",
+ "speed": 9.0,
+ },
+ )
+
+ import json as _json
+
+ body = _json.loads(route.calls.last.request.content)
+ assert body["voice_settings"]["speed"] == 2.0
+
+ async def test_missing_voice_is_a_user_error(self, harness):
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "textToSpeech", {"provider": "elevenlabs", "text": "hi"}
+ )
+
+ harness.assert_envelope(result, success=False)
+ assert "voice" in result["error"].lower()
+
+
+class TestDeepgramWireShape:
+ URL = "https://api.deepgram.com/v1/listen"
+
+ def _response(self, transcript: str = "hello there", duration: float = 12.0):
+ return httpx.Response(
+ 200,
+ json={
+ "metadata": {"request_id": "dg-1", "duration": duration},
+ "results": {
+ "channels": [
+ {
+ "detected_language": "en",
+ "language_confidence": 0.99,
+ "alternatives": [
+ {
+ "transcript": transcript,
+ "words": [
+ {
+ "word": "hello",
+ "start": 0.1,
+ "end": 0.4,
+ "confidence": 0.9,
+ }
+ ],
+ }
+ ],
+ }
+ ]
+ },
+ },
+ )
+
+ @respx.mock
+ async def test_auth_uses_token_not_bearer(self, harness, tmp_path):
+ ctx, name = _clip(harness, tmp_path)
+ route = respx.post(self.URL).mock(return_value=self._response())
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute(
+ "speechToText",
+ {"provider": "deepgram", "audio_file": name},
+ context=ctx,
+ )
+
+ assert route.calls.last.request.headers["authorization"] == "Token dg-test"
+
+ @respx.mock
+ async def test_options_are_query_params_and_audio_is_the_raw_body(
+ self, harness, tmp_path
+ ):
+ payload = _wav()
+ ctx, name = _clip(harness, tmp_path, payload=payload)
+ route = respx.post(self.URL).mock(return_value=self._response())
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute(
+ "speechToText",
+ {
+ "provider": "deepgram",
+ "audio_file": name,
+ "stt_model": "nova-3",
+ "diarize": True,
+ },
+ context=ctx,
+ )
+
+ request = route.calls.last.request
+ assert request.url.params["model"] == "nova-3"
+ assert request.url.params["diarize"] == "true"
+ # Raw body, not multipart -- no boundary, and the bytes are verbatim.
+ assert not request.headers["content-type"].startswith("multipart/")
+ assert request.content == payload
+
+ @respx.mock
+ async def test_transcript_is_read_from_the_nested_path(self, harness, tmp_path):
+ ctx, name = _clip(harness, tmp_path)
+ respx.post(self.URL).mock(return_value=self._response("the actual transcript"))
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "speechToText",
+ {"provider": "deepgram", "audio_file": name},
+ context=ctx,
+ )
+
+ harness.assert_envelope(result, success=True)
+ assert result["result"]["transcript"] == "the actual transcript"
+ assert result["result"]["language"] == "en"
+ assert result["result"]["words"][0]["word"] == "hello"
+
+
+class TestSarvamWireShape:
+ URL = "https://api.sarvam.ai/text-to-speech"
+
+ @respx.mock
+ async def test_base64_array_becomes_one_file_per_chunk(self, harness, tmp_path):
+ clip = base64.b64encode(_wav()).decode()
+ respx.post(self.URL).mock(
+ return_value=httpx.Response(
+ 200, json={"request_id": "sv-1", "audios": [clip, clip, clip]}
+ )
+ )
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "textToSpeech",
+ {"provider": "sarvam", "text": "namaste"},
+ context=harness.build_context(workspace_dir=str(tmp_path)),
+ )
+
+ harness.assert_envelope(result, success=True)
+ payload = result["result"]
+ assert payload["chunk_count"] == 3
+ assert len(payload["files"]) == 3
+ # Distinct paths -- concatenating them would be wrong, so they must
+ # not collide either.
+ assert len({f["path"] for f in payload["files"]}) == 3
+ assert "3 clips" in (payload["note"] or "")
+
+ @respx.mock
+ async def test_v2_only_params_are_not_sent_to_v3(self, harness):
+ respx.post(self.URL).mock(
+ return_value=httpx.Response(
+ 200, json={"audios": [base64.b64encode(_wav()).decode()]}
+ )
+ )
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute(
+ "textToSpeech",
+ {
+ "provider": "sarvam",
+ "text": "hi",
+ "tts_model": "bulbul:v3",
+ "provider_options": {"pitch": 0.5, "temperature": 0.7},
+ },
+ )
+
+ import json as _json
+
+ body = _json.loads(respx.calls.last.request.content)
+ assert "pitch" not in body
+ assert body["temperature"] == 0.7
+
+ @respx.mock
+ async def test_auth_uses_the_native_subscription_header(self, harness):
+ respx.post(self.URL).mock(
+ return_value=httpx.Response(
+ 200, json={"audios": [base64.b64encode(_wav()).decode()]}
+ )
+ )
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute("textToSpeech", {"provider": "sarvam", "text": "hi"})
+
+ assert respx.calls.last.request.headers["api-subscription-key"] == "sv-test"
+
+ async def test_over_cap_text_is_refused_before_the_call(self, harness):
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "textToSpeech",
+ {"provider": "sarvam", "text": "x" * 3000, "tts_model": "bulbul:v3"},
+ )
+
+ harness.assert_envelope(result, success=False)
+ assert "2500" in result["error"]
+
+
+class TestOpenAIWireShape:
+ @respx.mock
+ async def test_synthesis_returns_raw_bytes_and_writes_one_ref(
+ self, harness, tmp_path
+ ):
+ respx.post("https://api.openai.com/v1/audio/speech").mock(
+ return_value=httpx.Response(200, content=b"ID3rawaudio")
+ )
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "textToSpeech",
+ {"provider": "openai", "text": "hello"},
+ context=harness.build_context(workspace_dir=str(tmp_path)),
+ )
+
+ harness.assert_envelope(result, success=True)
+ audio = result["result"]["audio"]
+ assert audio["kind"] == "audio"
+ assert audio["path"].startswith("audio/")
+ assert audio["size_bytes"] == len(b"ID3rawaudio")
+
+ @respx.mock
+ async def test_response_format_downgrades_on_a_gated_model(self, harness, tmp_path):
+ """verbose_json on a 4o model is a 400; the node must not send it."""
+ ctx, name = _clip(harness, tmp_path)
+ route = respx.post("https://api.openai.com/v1/audio/transcriptions").mock(
+ return_value=httpx.Response(200, json={"text": "hi"})
+ )
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ await harness.execute(
+ "speechToText",
+ {
+ "provider": "openai",
+ "audio_file": name,
+ "stt_model": "gpt-4o-transcribe",
+ "timestamps": True,
+ },
+ context=ctx,
+ )
+
+ body = route.calls.last.request.content
+ assert b"verbose_json" not in body
+ assert b"timestamp_granularities" not in body
+
+ @respx.mock
+ async def test_groq_transcription_hits_its_own_host(self, harness, tmp_path):
+ ctx, name = _clip(harness, tmp_path)
+ route = respx.post(
+ "https://api.groq.com/openai/v1/audio/transcriptions"
+ ).mock(return_value=httpx.Response(200, json={"text": "hi", "duration": 3.0}))
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "speechToText",
+ {"provider": "groq", "audio_file": name},
+ context=ctx,
+ )
+
+ harness.assert_envelope(result, success=True)
+ assert route.called
+ assert route.calls.last.request.headers["authorization"] == "Bearer gsk-test"
+
+
+# ============================================================================
+# Node-level contract
+# ============================================================================
+
+
+class TestNodeContract:
+ def test_neither_node_names_a_field_model_or_api_key(self):
+ """Regression guard for the parameter-panel clobber.
+
+ ParameterRenderer overwrites any field literally named ``model`` or
+ ``api_key`` with chat-model data whenever a sibling ``provider``
+ field exists, and it never checks that the provider is an LLM
+ provider. A field with either name here would be cleared the moment
+ a user picked a speech provider.
+ """
+ from nodes.speech.speech_to_text import SpeechToTextParams
+ from nodes.speech.text_to_speech import TextToSpeechParams
+
+ for params in (TextToSpeechParams, SpeechToTextParams):
+ assert "model" not in params.model_fields
+ assert "api_key" not in params.model_fields
+
+ def test_neither_node_uses_declarative_routing(self):
+ """``routing=`` resolves credentials[0], which ignores `provider`."""
+ from nodes.speech.speech_to_text import SpeechToTextNode
+ from nodes.speech.text_to_speech import TextToSpeechNode
+
+ for node in (TextToSpeechNode, SpeechToTextNode):
+ for spec in node._operations.values():
+ assert spec.routing is None, (
+ f"{node.type}.{spec.name} declares routing=, which would pin "
+ "every provider to the first credential in the tuple"
+ )
+
+ def test_canvas_handles_stay_visible(self):
+ """`usable_as_tool` auto-hides both handles unless declared False."""
+ from nodes.speech.speech_to_text import SpeechToTextNode
+ from nodes.speech.text_to_speech import TextToSpeechNode
+
+ for node in (TextToSpeechNode, SpeechToTextNode):
+ assert node.hide_input_handle is False
+ assert node.hide_output_handle is False
+
+ def test_declared_credentials_cover_every_registered_provider(self):
+ from nodes.speech import _config
+ from nodes.speech._registry import stt_providers, tts_providers
+ from nodes.speech.speech_to_text import SpeechToTextNode
+ from nodes.speech.text_to_speech import TextToSpeechNode
+
+ for node, providers in (
+ (TextToSpeechNode, tts_providers()),
+ (SpeechToTextNode, stt_providers()),
+ ):
+ declared = {c.id for c in node.credentials}
+ for provider in providers:
+ assert _config.credential_id(provider) in declared, (
+ f"{node.type} cannot authenticate {provider}"
+ )
+
+ async def test_unknown_provider_is_a_user_error_listing_the_real_ones(
+ self, harness
+ ):
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "textToSpeech", {"provider": "nope", "text": "hi"}
+ )
+
+ harness.assert_envelope(result, success=False)
+ assert "elevenlabs" in result["error"]
+
+
+class TestAudioInputContainment:
+ """The Sarvam STT node read the credential store. This must not."""
+
+ @pytest.mark.parametrize(
+ "attack", ["../../credentials.db", "..\\..\\credentials.db", "~/secrets"]
+ )
+ async def test_traversal_is_refused(self, harness, tmp_path, attack):
+ (tmp_path.parent / "credentials.db").write_bytes(b"SECRET")
+ ctx = harness.build_context(workspace_dir=str(tmp_path))
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "speechToText",
+ {"provider": "openai", "audio_file": attack},
+ context=ctx,
+ )
+
+ harness.assert_envelope(result, success=False)
+
+ async def test_missing_file_is_a_user_error(self, harness):
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "speechToText",
+ {"provider": "openai", "audio_file": "definitely/absent.wav"},
+ )
+
+ harness.assert_envelope(result, success=False)
+
+
+class TestBilling:
+ @respx.mock
+ async def test_duration_is_measured_not_assumed(self, harness, tmp_path):
+ """The old Sarvam node billed every clip as a flat 30 seconds."""
+ ctx, name = _clip(harness, tmp_path, payload=_wav(seconds=2.0, rate=8000))
+ respx.post("https://api.sarvam.ai/speech-to-text").mock(
+ return_value=httpx.Response(200, json={"transcript": "ok"})
+ )
+
+ with patched_container(auth_api_keys=_KEYS), patched_pricing():
+ result = await harness.execute(
+ "speechToText",
+ {"provider": "sarvam", "audio_file": name},
+ context=ctx,
+ )
+
+ harness.assert_envelope(result, success=True)
+ duration = result["result"]["duration_seconds"]
+ assert duration == pytest.approx(2.0, abs=0.1)
+ assert duration != 30
+
+
+# ============================================================================
+# Boot-path purity
+# ============================================================================
+
+
+class TestLazySdkImports:
+ """Registering speech providers must not import a heavy SDK.
+
+ Ported from ``tests/llm/test_lazy_sdk_imports.py``. Runs in a clean
+ interpreter because the pytest process already has SDKs loaded by other
+ tests. This is the only thing preventing a future provider from
+ reintroducing the eager-import-at-boot anti-pattern.
+ """
+
+ HEAVY = ("openai", "anthropic", "google.genai")
+
+ def _probe(self, code: str) -> str:
+ result = subprocess.run(
+ [sys.executable, "-c", code],
+ cwd=SERVER_DIR,
+ capture_output=True,
+ text=True,
+ timeout=120,
+ )
+ assert result.returncode == 0, (
+ f"probe subprocess failed (rc={result.returncode}):\n{result.stderr}"
+ )
+ return result.stdout
+
+ def test_registering_providers_imports_no_sdk(self):
+ out = self._probe(
+ "import sys\n"
+ "import nodes.speech\n"
+ f"leaked = [m for m in {self.HEAVY!r} if m in sys.modules]\n"
+ "print('LEAKED=' + ','.join(leaked))\n"
+ )
+ assert out.strip().endswith("LEAKED="), (
+ f"speech provider registration imported heavy SDKs: {out!r}"
+ )
+
+ def test_every_declared_exception_ref_resolves(self):
+ """Typo guard — a bad ref only fails when an error occurs otherwise."""
+ import nodes.speech # noqa: F401
+
+ from nodes.speech._registry import (
+ get_stt_provider,
+ get_tts_provider,
+ stt_providers,
+ tts_providers,
+ )
+
+ for name in tts_providers():
+ assert get_tts_provider(name).sdk_exception_types
+ for name in stt_providers():
+ assert get_stt_provider(name).sdk_exception_types
diff --git a/server/tests/routers/__init__.py b/server/tests/routers/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/server/tests/routers/test_workspace.py b/server/tests/routers/test_workspace.py
new file mode 100644
index 00000000..b5810508
--- /dev/null
+++ b/server/tests/routers/test_workspace.py
@@ -0,0 +1,226 @@
+"""Contract for the workspace file routes.
+
+Mounted on a bare FastAPI app rather than importing ``main`` — the app's
+lifespan starts the container, the database and the Temporal runtime, none
+of which this router needs. The router is the unit under test.
+
+The security assertions here are not defensive padding. ``shell``,
+``fileDownloader`` and ``fileModify`` can all write arbitrary files into a
+workspace, so this route serves attacker-influenced content from the app
+origin. Inline HTML would be stored XSS with session-cookie access.
+"""
+
+from __future__ import annotations
+
+import struct
+import wave
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import httpx
+import pytest
+from fastapi import FastAPI
+
+from routers import workspace as workspace_router
+
+pytestmark = pytest.mark.unit
+
+SLUG = "My_Workflow_1"
+WORKFLOW_ID = "019f99e2dc997cf390193ad3e6260de1"
+
+
+def _wav(seconds: float = 0.25, rate: int = 8000) -> bytes:
+ import io
+
+ buffer = io.BytesIO()
+ with wave.open(buffer, "wb") as handle:
+ handle.setnchannels(1)
+ handle.setsampwidth(2)
+ handle.setframerate(rate)
+ handle.writeframes(struct.pack("<h", 0) * int(rate * seconds))
+ return buffer.getvalue()
+
+
+def workspace(tmp_path, monkeypatch):
+ """A workspaces root containing one workflow directory, keyed by SLUG."""
+ root = tmp_path / "workspaces"
+ (root / SLUG / "audio").mkdir(parents=True)
+ monkeypatch.setattr(workspace_router, "workspaces_dir", lambda: root)
+ return root / SLUG
+
+
+def client(workspace):
+ app = FastAPI()
+ app.include_router(workspace_router.router)
+
+ database = AsyncMock()
+ database.get_workflow.return_value = SimpleNamespace(slug=SLUG)
+ app.dependency_overrides[workspace_router._db] = lambda: database
+
+ transport = httpx.ASGITransport(app=app)
+ return httpx.AsyncClient(transport=transport, base_url="http://test")
+
+
+class TestServeFile:
+ async def test_serves_audio_inline(self, client, workspace):
+ payload = _wav()
+ (workspace / "audio" / "clip.wav").write_bytes(payload)
+
+ async with client as http:
+ response = await http.get(
+ f"/api/workspace/{WORKFLOW_ID}/files/audio/clip.wav"
+ )
+
+ assert response.status_code == 200
+ assert response.content == payload
+ assert response.headers["content-type"].startswith("audio/")
+ assert response.headers["content-disposition"].startswith("inline")
+ assert response.headers["x-content-type-options"] == "nosniff"
+
+ async def test_url_carries_the_id_while_the_directory_uses_the_slug(
+ self, client, workspace
+ ):
+ """The whole point of the id/slug split: refs survive a rename."""
+ (workspace / "audio" / "clip.wav").write_bytes(_wav())
+
+ async with client as http:
+ response = await http.get(
+ f"/api/workspace/{WORKFLOW_ID}/files/audio/clip.wav"
+ )
+
+ assert response.status_code == 200
+ assert WORKFLOW_ID not in str(workspace)
+ assert workspace.name == SLUG
+
+ @pytest.mark.parametrize(
+ "name,expected",
+ [
+ ("page.html", "attachment"),
+ ("vector.svg", "attachment"),
+ ("clip.wav", "inline"),
+ ("cover.png", "inline"),
+ ("notes.txt", "attachment"),
+ ],
+ )
+ async def test_only_safe_media_renders_inline(
+ self, client, workspace, name, expected
+ ):
+ """Script-bearing types must download, never render on our origin."""
+ (workspace / name).write_bytes(b"<html><script>alert(1)</script></html>")
+
+ async with client as http:
+ response = await http.get(f"/api/workspace/{WORKFLOW_ID}/files/{name}")
+
+ assert response.status_code == 200
+ assert response.headers["content-disposition"].startswith(expected)
+
+ @pytest.mark.parametrize(
+ "attack",
+ [
+ "../../../etc/passwd",
+ "..%2f..%2fsecret.txt",
+ "....//secret.txt",
+ ],
+ )
+ async def test_traversal_is_refused_as_404(self, client, workspace, attack):
+ (workspace.parent.parent / "secret.txt").write_bytes(b"SECRET")
+
+ async with client as http:
+ response = await http.get(f"/api/workspace/{WORKFLOW_ID}/files/{attack}")
+
+ assert response.status_code == 404
+ assert b"SECRET" not in response.content
+
+ async def test_missing_file_is_404_not_403(self, client):
+ """A distinct status would confirm what exists outside the workspace."""
+ async with client as http:
+ response = await http.get(
+ f"/api/workspace/{WORKFLOW_ID}/files/audio/nope.wav"
+ )
+
+ assert response.status_code == 404
+
+ async def test_range_requests_work_without_router_code(self, client, workspace):
+ """Starlette's FileResponse already implements 206; seeking depends on it."""
+ payload = _wav()
+ (workspace / "audio" / "clip.wav").write_bytes(payload)
+
+ async with client as http:
+ response = await http.get(
+ f"/api/workspace/{WORKFLOW_ID}/files/audio/clip.wav",
+ headers={"Range": "bytes=0-99"},
+ )
+
+ assert response.status_code == 206
+ assert response.headers["accept-ranges"] == "bytes"
+ assert response.headers["content-range"].startswith("bytes 0-99/")
+ assert response.content == payload[:100]
+
+
+class TestUpload:
+ async def test_stores_the_file_and_returns_a_ref(self, client, workspace):
+ payload = _wav()
+
+ async with client as http:
+ response = await http.post(
+ f"/api/workspace/{WORKFLOW_ID}/uploads",
+ files={"file": ("My Clip.wav", payload, "audio/wav")},
+ )
+
+ assert response.status_code == 200
+ ref = response.json()
+ assert ref["kind"] == "audio"
+ assert ref["path"].startswith("uploads/")
+ assert ref["size_bytes"] == len(payload)
+ assert ref["workflow_id"] == WORKFLOW_ID
+ assert (workspace / ref["path"]).read_bytes() == payload
+
+ async def test_returned_ref_is_immediately_servable(self, client, workspace):
+ async with client as http:
+ ref = (
+ await http.post(
+ f"/api/workspace/{WORKFLOW_ID}/uploads",
+ files={"file": ("clip.wav", _wav(), "audio/wav")},
+ )
+ ).json()
+
+ # The ref's own url must round-trip through the GET route.
+ response = await http.get(ref["url"])
+
+ assert response.status_code == 200
+
+ async def test_over_cap_upload_is_413(self, client, monkeypatch):
+ monkeypatch.setattr(workspace_router, "MEDIA_MAX_UPLOAD_BYTES", 1024)
+
+ async with client as http:
+ response = await http.post(
+ f"/api/workspace/{WORKFLOW_ID}/uploads",
+ files={"file": ("big.wav", b"x" * 4096, "audio/wav")},
+ )
+
+ assert response.status_code == 413
+
+ async def test_empty_upload_is_400(self, client):
+ async with client as http:
+ response = await http.post(
+ f"/api/workspace/{WORKFLOW_ID}/uploads",
+ files={"file": ("empty.wav", b"", "audio/wav")},
+ )
+
+ assert response.status_code == 400
+
+ async def test_filename_is_sanitized_not_trusted(self, client, workspace):
+ """A traversal filename must not escape the uploads directory."""
+ async with client as http:
+ response = await http.post(
+ f"/api/workspace/{WORKFLOW_ID}/uploads",
+ files={"file": ("../../evil.wav", _wav(), "audio/wav")},
+ )
+
+ assert response.status_code == 200
+ stored = Path(response.json()["path"])
+ assert stored.parts[0] == "uploads"
+ assert ".." not in stored.parts