Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(speech): provider-abstracted text-to-speech and speech-to-text nodes
trohitga29cdb7e45ec
diff --git a/CLAUDE.md b/CLAUDE.mdindex 952164c1..f37bcbf9 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -32,7 +32,8 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[GitHub Service](./docs-internal/github_service.md)** | GitHub CLI integration — `githubAction` (repo_clone / pr_create / pr_list / pr_merge / issue_create / issue_list + `custom` passthrough; AI tool `github`). **Strict Stripe pattern: the gh CLI owns its own auth** — no token stored/injected by OpenCompany, no auth pre-flight (gh's own error surfaces), marker OAuth row for the modal badge, fieldless catalogue entry. Login = spawned `gh auth login --hostname github.com --git-protocol https --web` — source-verified: headless takes the `isInteractive=false` branch (no Press-Enter), prints one-time code + `github.com/login/device` URL on stderr; the modal shows the code via the generic `verificationCode` plumbing (`useCredentialPanel` → `OAuthConnect`). `login_env()` strips `GH_TOKEN`/`GITHUB_TOKEN` (login aborts otherwise). Success gate = `gh auth status` exit 0, then best-effort **`gh auth setup-git`** (official bridge — the future git node needs zero auth code). gh binary: **project-local, pooch-driven** (temporal `_install.py` idiom) — pinned release under `package_dir("gh")`, system gh never consulted (auth state still shared: gh config paths are user-level). Palette group `vcs`. || **[Cloudflare Service](./docs-internal/cloudflare_service.md)** | Cloudflare CLI integration — `cloudflareAction` (whoami / zones_list / dns_records CRUD / `graphql_query` / `custom` passthrough; AI tool `cloudflare`) wrapping the official `cf` CLI (npm `[email protected]`, technical preview — "the next version of Wrangler"). **Two cf-specific variants of the CLI-managed-auth pattern**: (1) the CLI opens the browser ITSELF (loopback PKCE callback on fixed port 8877) so the login handler proxies nothing — single-flight guard (concurrent logins collide on the fixed port) + never-kill-the-shim invariant (killing the npm `.cmd` wrapper orphans the node child holding 8877); success gate = `cf auth whoami` JSON (`authenticated: true` — cf exits 0 in BOTH auth states). (2) **Fixed 86-scope OAuth grant with no `--scopes` flag** — only `dns_analytics:read` for analytics; no RUM/zone-analytics OAuth scope exists anywhere, so the optional `cloudflare_api_token` field (injected as `CLOUDFLARE_API_TOKEN`, cf's documented first-priority credential) is the only path to Web Analytics/RUM + the GraphQL Analytics API. `graphql_query` op POSTs the official `client/v4/graphql` endpoint directly (outside the OpenAPI schema cf/SDKs are generated from) — the replacement for the sunsetted Zone Analytics REST API; needs `Account > Account Analytics > Read`. Pinned npm install into the shared packages tree, system cf never consulted (preview argv surface drifts across versions). Palette group `deployment`. || **[Google Cloud Service](./docs-internal/gcloud_service.md)** | Google Cloud CLI integration — `gcloudAction` (auth/config/projects + set_project / Compute Engine instances list-start-stop-describe / Cloud Run deploy-list-describe / Cloud Storage ls-cp-rm / `custom` passthrough; AI tool `gcloud`) wrapping the official gcloud CLI (pinned versioned archive via pooch into `package_dir("gcloud")` — NOT npm; the Windows bundled-python zip keeps the legacy `google-cloud-sdk-` filename prefix, the `-cli-` spelling 404s, locked by test). Cloudflare login variant (CLI opens the browser itself, random loopback port — no fixed-port hazard) + the gh marker-token pattern; success gate = `gcloud auth list --filter=status:ACTIVE --format=json` non-empty (exit codes never trusted). **Config isolation divergence from gh**: every invocation pins `CLOUDSDK_CONFIG=<DATA_DIR>/gcloud/`, so terminal logins against the operator's global gcloud are deliberately NOT visible to the node. ADC note: node login mints gcloud USER creds only (CLI commands never read ADC); client-library code gets ADC via the `custom` op `auth application-default login` (lands isolated in the pinned dir). Cold install ~100 MB → login answers `pending: True` within the 22 s WS budget. Provider id `gcloud` is orthogonal to the Workspace `google` OAuth2 provider. Palette group `deployment`. |-| **[Sarvam AI Service](./docs-internal/sarvam_service.md)** | Sarvam AI (Indic-first) — `sarvamChatModel` registered as an **OpenAI-compat provider** (one `_COMPAT_PROVIDERS` entry + a JSON block; no provider subclass) plus five REST nodes under `nodes/sarvam/` (`sarvamTranslate` / `sarvamTransliterate` / `sarvamDetectLanguage` / `sarvamSpeechToText` / `sarvamTextToSpeech`, palette group `language`). **One stored key spans two auth styles** — the chat route takes `Authorization: Bearer` (what the openai SDK sends) while every other endpoint takes only `api-subscription-key`, so `SarvamCredential` declares the native header and both surfaces share it. Introduces the generic **`supports_model_listing: false`** flag: Sarvam ships no `/v1/models` route, and unhandled that 404 becomes a `NodeUserError` that `AIService.fetch_models` re-raises *before* its curated fallback, breaking key validation and the model dropdown — so `OpenAIProvider.fetch_models` serves the curated list and probes with a one-token completion instead (defaults `true`; the other 12 providers are untouched, locked by test). Reasoning needs zero Python (`thinking_type: "effort"` + `reasoning_content` are already handled); `thinking_default_on` and `reasoning_models` are deliberately omitted. Also added generic **`Connection.request(files=)`** multipart support for the STT node (threaded through the auth-retry rebuild). TTS writes audio to `<workspace>/audio/` by default — inline base64 is opt-in and size-capped because a full request is ~12 MB. |+| **[Sarvam AI Service](./docs-internal/sarvam_service.md)** | Sarvam AI (Indic-first) — `sarvamChatModel` registered as an **OpenAI-compat provider** (one `_COMPAT_PROVIDERS` entry + a JSON block; no provider subclass) plus three text REST nodes under `nodes/sarvam/` (`sarvamTranslate` / `sarvamTransliterate` / `sarvamDetectLanguage`, palette group `language`). Sarvam's speech endpoints are NOT nodes here — they are one provider inside the abstracted `textToSpeech` / `speechToText` nodes (see **[Speech Provider RFC](./docs-internal/speech_provider_rfc.md)**). **One stored key spans two auth styles** — the chat route takes `Authorization: Bearer` (what the openai SDK sends) while every other endpoint takes only `api-subscription-key`, so `SarvamCredential` declares the native header and every surface shares it. Introduces the generic **`supports_model_listing: false`** flag: Sarvam ships no `/v1/models` route, and unhandled that 404 becomes a `NodeUserError` that `AIService.fetch_models` re-raises *before* its curated fallback, breaking key validation and the model dropdown — so `OpenAIProvider.fetch_models` serves the curated list and probes with a one-token completion instead (defaults `true`; the other 12 providers are untouched, locked by test). Reasoning needs zero Python (`thinking_type: "effort"` + `reasoning_content` are already handled); `thinking_default_on` and `reasoning_models` are deliberately omitted. Also added generic **`Connection.request(files=)`** multipart support (threaded through the auth-retry rebuild), which the speech plugin still relies on. |+| **[Speech Provider RFC](./docs-internal/speech_provider_rfc.md)** | Provider-abstracted speech — two nodes (`textToSpeech`, `speechToText`) with a `provider` dropdown, backed by **two registries, one per direction** (registry membership IS the capability: a synthesis-only vendor simply never appears in the transcription enum). Everything vendor-specific lives in the plugin folder [`server/nodes/speech/`](./server/nodes/speech/) — protocol, registries, dispatch, per-vendor modules; only the generic `services/provider_registry.py` and the vendor-neutral `services/media/` are shared. v1 providers: OpenAI (both directions), ElevenLabs (TTS), Deepgram + Groq (STT), Sarvam (both). Capabilities are JSON (`server/config/speech_defaults.json`) with per-model overrides, so no shared code branches on a provider name. **Audio never travels as bytes** — nodes return an `AudioRef` (a ~400 B reference) because a 12 MB base64 result blows Temporal's 2 MiB blob limit and retries 3× re-billing the provider each time. Backing routes: `GET/POST /api/workspace/{workflow_id}/…` in [`routers/workspace.py`](./server/routers/workspace.py). || **[CI/CD Pipeline](./docs-internal/ci_cd.md)** | GitHub Actions workflows, predeploy validation, release publishing, and composite setup action || **[Performance](./docs-internal/performance.md)** | Cold-start measurements (Application startup complete 2.90 s warm, 21.5 s cold post-`company clean` — was 71 s before the 2026-07-14 boot-delay fixes; first WS connect 8.29 s) + warm and cold per-phase timelines + optimisation history (lazy LangChain imports, lazy LLM-SDK exception refs, esbuild sidecar bundle, bytecode-at-build, PartySocket reconnect, TanStack Query auth bootstrap, `manualChunks`, Vite dep-cache preservation, off-loop Temporal build-id hash) + bottleneck inventory + reproduction commands + anti-patterns to never reintroduce (incl. eager SDK import at LLM provider registration, unconditional `.vite` wipe, synchronous `Worker()` on the event loop). Pairs with the build-time pipeline doc below. || **[Release Build Pipeline](./docs-internal/release_build_pipeline.md)** | npm-distribution build pipeline: `tsgo` for type-check (5× faster than tsc), Vite `manualChunks` + `target: 'es2022'`, esbuild Node.js sidecar bundle (drops `tsx` interpreter cost), bytecode pre-compile in two halves — `[tool.uv] compile-bytecode = true` (uv sync compiles `.venv/` site-packages) + `python -m compileall` for project source (plain `.pyc`, **no `-O`** — per PEP 488 the non-`-O` runtimes never load `.opt-1.pyc`; excludes `.venv/`, `tests/`). Single source of truth for the bytecode-compile path list: `cli.commands.build.COMPILEALL_SOURCE_DIRS`. |@@ -618,7 +619,7 @@ The project uses WebSocket as the primary communication method between frontend> The authoritative, per-node reference is the backend plugin registry plus the per-node "logic-flow" cards under [docs-internal/node-logic-flows/](./docs-internal/node-logic-flows/) (one card per node, grouped by category, with handles / params / outputs / side-effects / edge-cases). Live node list = the plugin registry (one folder per node under `server/nodes/<group>/`, 131 node types today via `len(services.node_registry.NODE_METADATA)`; a bare `__init__.py` glob overcounts by also matching the group packages). Do NOT maintain a per-node catalogue here — it drifts on every plugin add.-Node groups (palette categories): agent, model, skill, tool, trigger, workflow, search, google, android, whatsapp, telegram, twitter, social, email, proxy, chat, scheduler, text, code, document, location, utility, browser, scraper, filesystem, stripe, vercel, github (palette group `vcs`), cloudflare and gcloud (palette group `deployment`), sarvam (palette group `language` — translate / transliterate / detect-language / speech-to-text / text-to-speech, sharing one API key with `sarvamChatModel`). See [docs-internal/node-logic-flows/](./docs-internal/node-logic-flows/) for the card index.+Node groups (palette categories): agent, model, skill, tool, trigger, workflow, search, google, android, whatsapp, telegram, twitter, social, email, proxy, chat, scheduler, text, code, document, location, utility, browser, scraper, filesystem, stripe, vercel, github (palette group `vcs`), cloudflare and gcloud (palette group `deployment`), sarvam (palette group `language` — translate / transliterate / detect-language, sharing one API key with `sarvamChatModel`), speech (palette group `language` — provider-abstracted `textToSpeech` / `speechToText`). See [docs-internal/node-logic-flows/](./docs-internal/node-logic-flows/) for the card index.## Backend Servicesdiff --git a/client/src/components/ParameterRenderer.tsx b/client/src/components/ParameterRenderer.tsxindex 9fa15144..86faa1f0 100644--- a/client/src/components/ParameterRenderer.tsx+++ b/client/src/components/ParameterRenderer.tsx@@ -33,6 +33,7 @@ import { shouldShowParameter } from '../utils/parameterVisibility';import { AI_MODEL_PROVIDER_MAP } from '../lib/aiModelProviders';import { resolveNodeDescription } from '../lib/nodeSpec';+import { uploadToWorkspace } from '../lib/workspaceUpload';/*** Radix reserves the empty string to clear a Select, so raw node option@@ -1627,16 +1628,43 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({);case 'file':+ // Hooks inside a switch case are a pre-existing pattern in this+ // renderer (`fileInputRef` predates this change). Kept consistent+ // rather than restructuring the whole component here.const fileInputRef = React.useRef<HTMLInputElement>(null);-- const handleFileUpload = (e: React.ChangeEvent<HTMLInputElement>) => {+ const [isUploading, setIsUploading] = useState(false);+ const [uploadError, setUploadError] = useState<string | null>(null);++ // Uploads go to the workspace and the parameter stores a ~400 byte+ // reference. The legacy path — base64 straight into the parameter —+ // is kept only as a fallback for a workflow that has never been+ // saved, because it has no workspace to upload into yet. That path+ // is a known hazard at media sizes: the blob rides the WebSocket, is+ // persisted, is re-broadcast to every client, and then exceeds+ // Temporal's payload limit. `coerce_file_param` still accepts it+ // server-side so existing saved rows keep working.+ const handleFileUpload = async (e: React.ChangeEvent<HTMLInputElement>) => {const file = e.target.files?.[0];if (!file) return;+ const workflowId = useAppStore.getState().currentWorkflow?.id;+ if (workflowId) {+ try {+ setUploadError(null);+ setIsUploading(true);+ onChange(await uploadToWorkspace(file, workflowId));+ return;+ } catch (err) {+ setUploadError(err instanceof Error ? err.message : 'Upload failed.');+ return;+ } finally {+ setIsUploading(false);+ }+ }+const reader = new FileReader();reader.onload = () => {const base64 = (reader.result as string).split(',')[1]; // Remove data:mime;base64, prefix- // Store as object with base64 data, filename, and mime typeonChange({type: 'upload',data: base64,@@ -1648,6 +1676,7 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({};const isUploadedFile = currentValue && typeof currentValue === 'object' && currentValue.type === 'upload';+ const isFileRef = currentValue && typeof currentValue === 'object' && currentValue.kind === 'audio';// Determine file accept type based on context (e.g., message_type for WhatsApp)const getFileAcceptType = () => {@@ -1677,14 +1706,20 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({<div className="flex items-center gap-2"><ShadcnInputtype="text"- value={isUploadedFile ? `[Uploaded] ${currentValue.filename}` : (currentValue || '')}+ value={+ isFileRef+ ? currentValue.filename+ : isUploadedFile+ ? `[Uploaded] ${currentValue.filename}`+ : (currentValue || '')+ }onChange={(e) => onChange(e.target.value)}placeholder={parameter.placeholder || 'Enter file path or upload'}onDragOver={handleDragOver}onDragLeave={handleDragLeave}onDrop={handleDrop}- readOnly={isUploadedFile}- className={`flex-1 font-mono ${isDragOver ? 'border-ring ring-3 ring-ring/50 bg-accent/10' : ''} ${isUploadedFile ? 'bg-muted text-success' : (isFileTemplate ? 'text-info' : '')}`}+ readOnly={isUploadedFile || isFileRef}+ className={`flex-1 font-mono ${isDragOver ? 'border-ring ring-3 ring-ring/50 bg-accent/10' : ''} ${isUploadedFile || isFileRef ? 'bg-muted text-success' : (isFileTemplate ? 'text-info' : '')}`}/><inputkey={`file-input-${allParameters?.message_type || 'default'}`}@@ -1697,15 +1732,17 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({<ActionButtonintent="config"onClick={() => fileInputRef.current?.click()}+ disabled={isUploading}title="Upload file">- Upload+ {isUploading ? 'Uploading…' : 'Upload'}</ActionButton>- {isUploadedFile && (+ {(isUploadedFile || isFileRef) && (<ActionButtonintent="stop"onClick={() => {onChange('');+ setUploadError(null);if (fileInputRef.current) fileInputRef.current.value = '';}}title="Clear uploaded file"@@ -1714,10 +1751,16 @@ const ParameterRenderer: React.FC<ParameterRendererProps> = ({</ActionButton>)}</div>- <div className="mt-1 text-[11px] italic text-muted-foreground">- {isUploadedFile- ? `Size: ${(currentValue.data.length * 0.75 / 1024).toFixed(1)} KB | Type: ${currentValue.mimeType}`- : 'Enter server path or click Upload to select a file'}+ <div+ className={`mt-1 text-[11px] italic ${uploadError ? 'text-destructive' : 'text-muted-foreground'}`}+ >+ {uploadError+ ? uploadError+ : isFileRef+ ? `Saved to the workspace: ${currentValue.path}`+ : isUploadedFile+ ? `Size: ${(currentValue.data.length * 0.75 / 1024).toFixed(1)} KB | Type: ${currentValue.mimeType}`+ : 'Enter server path or click Upload to select a file'}</div></div>);diff --git a/client/src/components/__tests__/OutputPanel.test.tsx b/client/src/components/__tests__/OutputPanel.test.tsxindex 4064f75b..fb17b8ca 100644--- a/client/src/components/__tests__/OutputPanel.test.tsx+++ b/client/src/components/__tests__/OutputPanel.test.tsx@@ -94,3 +94,83 @@ describe('OutputPanel response routing', () => {expect(views.some(v => v.textContent?.includes('42'))).toBe(true);});});++/**+ * Audio results.+ *+ * An AudioRef is an object, so without a dedicated branch it falls into+ * the JSON tree and the user gets a wall of metadata instead of something+ * they can play. Detection is structural (`kind === 'audio'`) rather than+ * hint-driven, so a ref reaching this panel from anywhere still renders.+ */+describe('OutputPanel — audio', () => {+ const SPEECH_NODE = { id: 'tts-1', type: 'textToSpeech' } as any;++ const audioRef = (overrides: Record<string, any> = {}) => ({+ kind: 'audio',+ path: 'audio/greeting-abc123.wav',+ workflow_id: 'wf-1',+ filename: 'greeting-abc123.wav',+ mime_type: 'audio/wav',+ format: 'wav',+ size_bytes: 24000,+ duration_seconds: 1.5,+ url: '/api/workspace/wf-1/files/audio/greeting-abc123.wav',+ ...overrides,+ });++ const makeSpeechResult = (outputs: Record<string, any>) =>+ [{ nodeId: 'tts-1', success: true, executionTime: 900, outputs }] as any;++ it('renders a player instead of dumping the reference as JSON', () => {+ specState.spec = { uiHints: { outputMode: 'audio' } };+ const { container } = renderWithProviders(+ <OutputPanel+ results={makeSpeechResult({ audio: audioRef(), files: [audioRef()], chunk_count: 1 })}+ selectedNode={SPEECH_NODE}+ />,+ );++ const player = container.querySelector('audio');+ expect(player).toBeTruthy();+ expect(player?.getAttribute('src')).toContain(+ '/api/workspace/wf-1/files/audio/greeting-abc123.wav',+ );+ expect(screen.getByText('greeting-abc123.wav')).toBeTruthy();+ });++ it('renders one player per clip when a provider split the input', () => {+ specState.spec = { uiHints: { outputMode: 'audio' } };+ const { container } = renderWithProviders(+ <OutputPanel+ results={makeSpeechResult({+ audio: audioRef({ path: 'audio/a.wav', filename: 'a.wav', sha256: 'aa' }),+ files: [+ audioRef({ path: 'audio/a.wav', filename: 'a.wav', sha256: 'aa' }),+ audioRef({ path: 'audio/b.wav', filename: 'b.wav', sha256: 'bb' }),+ ],+ chunk_count: 2,+ })}+ selectedNode={SPEECH_NODE}+ />,+ );++ expect(container.querySelectorAll('audio')).toHaveLength(2);+ // The clips are separate files, not parts of one stream — the UI has to+ // say so, because concatenating them produces unplayable audio.+ expect(screen.getByText(/separate files/i)).toBeTruthy();+ });++ it('does not hijack a non-audio result', () => {+ specState.spec = null;+ const { container } = renderWithProviders(+ <OutputPanel+ results={makeSpeechResult({ response: 'plain text' })}+ selectedNode={SPEECH_NODE}+ />,+ );++ expect(container.querySelector('audio')).toBeNull();+ expect(screen.getByTestId('markdown')).toBeTruthy();+ });+});diff --git a/client/src/components/output/AudioPreview.tsx b/client/src/components/output/AudioPreview.tsxnew file mode 100644index 00000000..61cc1499--- /dev/null+++ b/client/src/components/output/AudioPreview.tsx@@ -0,0 +1,108 @@+import { useState } from 'react';+import { AlertCircle } from 'lucide-react';+import { buildApiUrl } from '../../config/api';++/**+ * An AudioRef as the backend serializes it (`services/media/refs.py`).+ * Deliberately has no bytes/base64 field — audio travels as a reference,+ * never as data. See `services/media/limits.py` for the measured reason.+ */+export interface AudioRef {+ kind: 'audio';+ path: string;+ workflow_id?: string | null;+ filename: string;+ mime_type?: string;+ format?: string;+ size_bytes?: number;+ duration_seconds?: number | null;+ sample_rate?: number | null;+ channels?: number | null;+ sha256?: string | null;+ url?: string | null;+}++/** Structural check — cheaper and more honest than trusting a uiHint alone. */+export const isAudioRef = (value: unknown): value is AudioRef =>+ !!value &&+ typeof value === 'object' &&+ (value as AudioRef).kind === 'audio' &&+ typeof (value as AudioRef).path === 'string';++const formatDuration = (seconds?: number | null): string | null => {+ if (seconds == null || !Number.isFinite(seconds)) return null;+ const total = Math.round(seconds);+ return `${Math.floor(total / 60)}:${String(total % 60).padStart(2, '0')}`;+};++const formatSize = (bytes?: number): string | null => {+ if (!bytes) return null;+ return bytes < 1024 * 1024+ ? `${(bytes / 1024).toFixed(0)} KB`+ : `${(bytes / (1024 * 1024)).toFixed(1)} MB`;+};++function AudioClip({ audio }: { audio: AudioRef }) {+ const [failed, setFailed] = useState(false);++ // `url` is path-only so a remote backend can be prefixed; fall back to+ // composing it when an older ref predates the field.+ const src = buildApiUrl(+ audio.url || `/api/workspace/${audio.workflow_id ?? ''}/files/${audio.path}`+ );++ const meta = [formatDuration(audio.duration_seconds), formatSize(audio.size_bytes), audio.format]+ .filter(Boolean)+ .join(' · ');++ return (+ <div className="rounded-md border border-border bg-card p-3">+ <div className="mb-2 flex items-baseline justify-between gap-2">+ <span className="truncate font-mono text-xs text-foreground">{audio.filename}</span>+ {meta && <span className="shrink-0 text-xs text-muted-foreground">{meta}</span>}+ </div>++ {failed ? (+ // A 401 surfaces to <audio> as a silent `error` event, so an+ // unauthenticated session would otherwise show a dead player with+ // no explanation.+ <div className="flex items-center gap-2 text-xs text-destructive">+ <AlertCircle className="h-3.5 w-3.5 shrink-0" />+ <span>Could not load this clip. It may have been cleaned up, or the session expired.</span>+ </div>+ ) : (+ <audio+ controls+ preload="metadata"+ src={src}+ onError={() => setFailed(true)}+ className="w-full"+ />+ )}+ </div>+ );+}++/**+ * Renders one or more generated clips.+ *+ * Several clips are not one stream: providers that split long input return+ * standalone files, each with its own container header, so they are shown+ * as separate players rather than concatenated.+ */+export default function AudioPreview({ clips }: { clips: AudioRef[] }) {+ if (!clips.length) return null;++ return (+ <div className="flex flex-col gap-2">+ {clips.length > 1 && (+ <span className="text-xs text-muted-foreground">+ {clips.length} clips — separate files, not parts of one stream.+ </span>+ )}+ {clips.map((clip, index) => (+ <AudioClip key={clip.sha256 || clip.path || index} audio={clip} />+ ))}+ </div>+ );+}diff --git a/client/src/components/output/OutputPanel.tsx b/client/src/components/output/OutputPanel.tsxindex 1194db8d..2eaa00a9 100644--- a/client/src/components/output/OutputPanel.tsx+++ b/client/src/components/output/OutputPanel.tsx@@ -23,6 +23,7 @@ import { ExecutionResult } from '@/services/executionService';import { useNodeSpec } from '@/lib/nodeSpec';import { copyToClipboard, tryParseJson } from '@/utils/formatters';import { cn } from '@/lib/utils';+import AudioPreview, { AudioRef, isAudioRef } from './AudioPreview';/** Extract output data from ExecutionResult. */const getData = (r: ExecutionResult) =>@@ -136,6 +137,14 @@ export default function OutputPanel({ results, onClear, selectedNode }: Props) {// Terminal nodes sometimes emit JSON on stdout anyway — a tree view// beats a wall of braces (stdlib JSON.parse via shared formatter).const stdoutJson = isTerminal && typeof response === 'string' ? tryParseJson(response) : null;+ // Audio results are AudioRef objects, which would otherwise fall into the+ // JSON tree above. Detected structurally rather than by uiHint alone, so a+ // ref reaching this panel from anywhere still renders as a player.+ const audioClips: AudioRef[] = (+ Array.isArray(data?.files) && data.files.some(isAudioRef)+ ? data.files+ : [data?.audio ?? response]+ ).filter(isAudioRef);const thinking = data?.thinking;const metaTags = ['model', 'provider', 'agent_type'].filter(k => data?.[k]);@@ -173,7 +182,13 @@ export default function OutputPanel({ results, onClear, selectedNode }: Props) {)}<div className="space-y-0">- {response && (+ {audioClips.length > 0 && (+ <Section label="Audio" defaultOpen>+ <AudioPreview clips={audioClips} />+ </Section>+ )}++ {response && audioClips.length === 0 && (<Section label="Response" defaultOpen>{typeof response !== 'string' ? (// Object/array response — themed JSON viewer (same --code-*diff --git a/client/src/lib/workspaceUpload.ts b/client/src/lib/workspaceUpload.tsnew file mode 100644index 00000000..01eda1f7--- /dev/null+++ b/client/src/lib/workspaceUpload.ts@@ -0,0 +1,59 @@+import { buildApiUrl } from '../config/api';+import type { AudioRef } from '../components/output/AudioPreview';++/**+ * Mirrors `MEDIA_MAX_UPLOAD_BYTES` in `server/services/media/limits.py`.+ * The server is the authority and returns 413 regardless; checking here+ * only buys a better message before 25 MB goes over the wire.+ */+export const MEDIA_MAX_UPLOAD_BYTES = 25 * 1024 * 1024;++export class UploadError extends Error {}++/**+ * POST a file into the workflow's workspace and return the stored reference.+ *+ * This exists because the alternative — base64 inside the node's parameters —+ * is actively harmful at audio sizes. That path JSON-stringifies the blob onto+ * the WebSocket, persists it in the `node_parameters` row, re-broadcasts it to+ * every connected client, and then dies at Temporal's 2 MiB payload limit,+ * burning three retries on the way. A ~1.5 MB clip is already ~2 MB of base64.+ *+ * The reference that comes back is ~400 bytes.+ */+export async function uploadToWorkspace(+ file: File,+ workflowId: string+): Promise<AudioRef> {+ if (!workflowId) {+ throw new UploadError('Save the workflow before uploading a file to it.');+ }+ if (file.size > MEDIA_MAX_UPLOAD_BYTES) {+ throw new UploadError(+ `${file.name} is ${(file.size / (1024 * 1024)).toFixed(1)} MB; the limit is ` ++ `${MEDIA_MAX_UPLOAD_BYTES / (1024 * 1024)} MB.`+ );+ }++ const body = new FormData();+ body.append('file', file, file.name);++ // No explicit Content-Type: the browser must set it so the multipart+ // boundary is generated. Setting it by hand produces a body the server+ // cannot parse.+ const response = await fetch(buildApiUrl(`/api/workspace/${workflowId}/uploads`), {+ method: 'POST',+ body,+ credentials: 'include',+ });++ if (!response.ok) {+ const detail = await response+ .json()+ .then((payload) => payload?.detail)+ .catch(() => null);+ throw new UploadError(detail || `Upload failed (${response.status}).`);+ }++ return response.json();+}diff --git a/client/src/types/INodeProperties.ts b/client/src/types/INodeProperties.tsindex 568deb63..ead88fde 100644--- a/client/src/types/INodeProperties.ts+++ b/client/src/types/INodeProperties.ts@@ -224,8 +224,13 @@ export interface INodeUIHints {/** OutputPanel: how the node's textual output renders. `"terminal"`* = preformatted CLI text (never markdown — `#` would become* headings and indentation would collapse). Declared by CLI-wrapper- * plugins (githubAction, vercelAction, shell). */- outputMode?: 'terminal';+ * plugins (githubAction, vercelAction, shell). `"audio"` = render an+ * `<audio>` player for a result carrying an AudioRef (`kind:+ * "audio"`), declared by textToSpeech; without it the ref falls+ * through to the JSON tree. Widening this union needs no change to+ * the backend's `test_ui_hints_only_carry_known_flags` — that+ * invariant checks flag *names*, not values. */+ outputMode?: 'terminal' | 'audio';}export interface INodeTypeDescription {diff --git a/docs-internal/node-logic-flows/README.md b/docs-internal/node-logic-flows/README.mdindex 9331d123..28a0dc62 100644--- a/docs-internal/node-logic-flows/README.md+++ b/docs-internal/node-logic-flows/README.md@@ -137,8 +137,8 @@ Sarvam AI's non-chat REST APIs. The chat models live under `ai_chat_models`and share the same credential.- [Sarvam Detect Language (`sarvamDetectLanguage`)](./language/sarvamDetectLanguage.md)-- [Sarvam Speech to Text (`sarvamSpeechToText`)](./language/sarvamSpeechToText.md)-- [Sarvam Text to Speech (`sarvamTextToSpeech`)](./language/sarvamTextToSpeech.md)+- [Speech to Text (`speechToText`)](./language/speechToText.md)+- [Text to Speech (`textToSpeech`)](./language/textToSpeech.md)- [Sarvam Translate (`sarvamTranslate`)](./language/sarvamTranslate.md)- [Sarvam Transliterate (`sarvamTransliterate`)](./language/sarvamTransliterate.md)diff --git a/docs-internal/node-logic-flows/language/sarvamSpeechToText.md b/docs-internal/node-logic-flows/language/sarvamSpeechToText.mddeleted file mode 100644index bf1add32..00000000--- a/docs-internal/node-logic-flows/language/sarvamSpeechToText.md+++ /dev/null@@ -1,119 +0,0 @@-# Sarvam Speech to Text (`sarvamSpeechToText`)--| Field | Value |-|------|-------|-| **Category** | language |-| **Backend handler** | [`server/nodes/sarvam/sarvam_speech_to_text.py`](../../../server/nodes/sarvam/sarvam_speech_to_text.py) |-| **Shared helpers** | [`server/nodes/sarvam/_base.py`](../../../server/nodes/sarvam/_base.py) (`post_multipart`, `track_sarvam_usage`) |-| **Tests** | [`server/tests/nodes/test_sarvam.py`](../../../server/tests/nodes/test_sarvam.py), [`server/tests/services/test_connection_multipart.py`](../../../server/tests/services/test_connection_multipart.py) |-| **Skill (if any)** | [`server/skills/language_agent/sarvam-speech-skill/SKILL.md`](../../../server/skills/language_agent/sarvam-speech-skill/SKILL.md) |-| **Dual-purpose tool** | yes — tool name `sarvam_speech_to_text` |--## Purpose--Transcribe short audio clips across 23 Indian languages using Sarvam's Saaras-models, with optional direct translation to English. This is the repo's first-`multipart/form-data` node.--## Inputs (handles)--| Handle | Connection type | Required | Purpose |-|--------|-----------------|----------|---------|-| `input-main` | main | no | Upstream data injected into params (commonly a file path) |--## Parameters--| Name | Type | Default | Required | displayOptions.show | Description |-|------|------|---------|----------|---------------------|-------------|-| `audio_file` | string \| object | `""` | yes | - | Path, **or** the file-widget upload envelope. `widget: file`, `accept: audio/*` |-| `model` | enum | `saaras:v3` | no | - | `saaras:v3` (23 langs) / `saarika:v2.5` (legacy) |-| `language_code` | string | `unknown` | no | - | BCP-47, or `unknown` to auto-detect |-| `mode` | enum | `transcribe` | no | - | `transcribe` / `translate` / `verbatim` / `translit` / `codemix` |-| `input_audio_codec` | enum\|null | `null` | no | - | `pcm_s16le` / `pcm_l16` / `pcm_raw` — headerless PCM only |--## Outputs (handles)--| Handle | Shape | Description |-|--------|-------|-------------|-| `output-main` | object | Standard envelope payload |--### Output payload--```ts-{- transcript: string;- language_code: string | null;- language_probability: number | null; // null when language_code was supplied- timestamps: object | null; // conditional- diarized_transcript: object | null; // Batch API only — always null here- request_id: string | null;-}-```--## Logic Flow--```mermaid-flowchart TD- A[Receive params] --> B{mode advanced && model != saaras:v3?}- B -- yes --> X[NodeUserError]- B -- no --> C[_read_audio: dict upload or path]- C -- dict, bad base64 --> X- C -- path missing --> X- C -- empty or oversize --> X- C -- ok --> D[guess mime from filename]- D --> E[ctx.connection sarvam -> POST /speech-to-text<br/>files= + data=, no Content-Type]- E -- status >= 400 --> Y[NodeUserError w/ Sarvam message]- E -- ok --> F[track_sarvam_usage speech_to_text, 30s]- F --> G[Return SarvamSpeechToTextOutput]-```--## Decision Logic--- **Mode/model compatibility**: `verbatim` / `translit` / `codemix` require- `saaras:v3`; asking for them on `saarika:v2.5` fails before the call.-- **Dual-shaped `audio_file`** (`_read_audio`): the frontend file widget sends- `{"type": "upload", "data": "<base64>", "filename", "mimeType"}` when the- user picks a file, and a bare string when they type a path or drag an- upstream value in. The param is typed `Union[str, Dict]` for exactly this- reason — `whatsapp_send` declares the same field as `str` and only survives- because its handler reads the raw dict.-- **Path resolution**: relative paths resolve against `ctx.workspace_dir`, so- they line up with `fileDownloader` and `sarvamTextToSpeech` output.-- **Size guard**: empty files and payloads over 24 MB fail with a message that- names the 30-second limit and points at the Batch API.-- **Billing**: `resource_count` is the documented 30-second ceiling; the node- does not decode the clip to measure real duration.--## Side Effects--- **Database writes**: one `api_usage_metrics` row per call (seconds).-- **Broadcasts**: standard node status only.-- **External API calls**: `POST https://api.sarvam.ai/speech-to-text` as `multipart/form-data`, `api-subscription-key` header.-- **File I/O**: reads the audio file when given a path. Writes nothing.-- **Subprocess**: none.--## External Dependencies--- **Credentials**: `SarvamCredential` via `ctx.connection("sarvam")`.-- **Framework**: `Connection.request(files=)` — added generically for this- node and threaded through the auth-retry rebuild, so a 401 replays the file- parts rather than an empty body. Pass **bytes**, never an open handle.-- **Python packages**: `httpx` (through `Connection`), stdlib `base64` / `mimetypes`.--## Edge cases & known limits--- **30-second hard limit.** This is the synchronous endpoint; longer audio- needs Sarvam's Batch API, which is a separate job-based surface and is not- wired. Tell the user rather than retrying.-- **Diarization is Batch-only.** `diarized_transcript` is declared on the- output model for shape stability but will always be `null` here.-- Accuracy is best at 16 kHz; PCM inputs are limited to 16 kHz.-- `language_probability` is `null` whenever `language_code` was supplied- explicitly (nothing was inferred).-- No streaming — the WebSocket transcription surface is not wired.--## Related--- **Skills using this as a tool**: `sarvam-speech-skill`.-- **Peer nodes**: [`sarvamTextToSpeech`](./sarvamTextToSpeech.md) (the other half of a voice loop).-- **Architecture docs**: [Sarvam AI Service](../../sarvam_service.md).diff --git a/docs-internal/node-logic-flows/language/sarvamTextToSpeech.md b/docs-internal/node-logic-flows/language/sarvamTextToSpeech.mddeleted file mode 100644index 88291663..00000000--- a/docs-internal/node-logic-flows/language/sarvamTextToSpeech.md+++ /dev/null@@ -1,141 +0,0 @@-# Sarvam Text to Speech (`sarvamTextToSpeech`)--| Field | Value |-|------|-------|-| **Category** | language |-| **Backend handler** | [`server/nodes/sarvam/sarvam_text_to_speech.py`](../../../server/nodes/sarvam/sarvam_text_to_speech.py) |-| **Shared helpers** | [`server/nodes/sarvam/_base.py`](../../../server/nodes/sarvam/_base.py) (`post_json`, `track_sarvam_usage`) |-| **Tests** | [`server/tests/nodes/test_sarvam.py`](../../../server/tests/nodes/test_sarvam.py) |-| **Skill (if any)** | [`server/skills/language_agent/sarvam-speech-skill/SKILL.md`](../../../server/skills/language_agent/sarvam-speech-skill/SKILL.md) |-| **Dual-purpose tool** | yes — tool name `sarvam_text_to_speech` |--## Purpose--Synthesize natural speech in 11 Indian languages using Sarvam's Bulbul voices-(37 on v3, 7 on v2). Writes audio into the workflow workspace and returns a-path; inline base64 is opt-in and size-capped.--## Inputs (handles)--| Handle | Connection type | Required | Purpose |-|--------|-----------------|----------|---------|-| `input-main` | main | no | Upstream data injected into params |--## Parameters--| Name | Type | Default | Required | displayOptions.show | Description |-|------|------|---------|----------|---------------------|-------------|-| `text` | string | - | yes | - | Text to speak (2500 chars v3, 1500 v2) |-| `target_language_code` | enum | `hi-IN` | no | - | 11 core codes |-| `model` | enum | `bulbul:v3` | no | - | `bulbul:v3` / `bulbul:v2` |-| `speaker` | string | `shubh` | no | - | Voice name, lowercase. Per-model sets |-| `pace` | number | `1.0` | no | - | v3 0.5-2.0, v2 0.3-3.0 |-| `speech_sample_rate` | enum | `24000` | no | - | 8000 / 16000 / 22050 / 24000 / 32000 / 44100 / 48000 |-| `output_audio_codec` | enum | `wav` | no | - | wav / mp3 / linear16 / mulaw / alaw / opus / flac / aac |-| `temperature` | number\|null | `null` | no | `model: [bulbul:v3]` | 0.01-2.0, v3 only |-| `pitch` | number\|null | `null` | no | `model: [bulbul:v2]` | -0.75 to 0.75, v2 only |-| `loudness` | number\|null | `null` | no | `model: [bulbul:v2]` | 0.3-3.0, v2 only |-| `enable_preprocessing` | boolean\|null | `null` | no | `model: [bulbul:v2]` | v2 only (v3 always preprocesses) |-| `return_audio` | enum | `file` | no | - | `file` / `base64` |--## Outputs (handles)--| Handle | Shape | Description |-|--------|-------|-------------|-| `output-main` | object | Standard envelope payload |--### Output payload--```ts-{- file_path: string | null; // chunk 1; null in base64 mode- files: string[]; // one entry per chunk- chunk_count: number;- audio_format: string; // echoes output_audio_codec- audio_base64: string | null; // opt-in, size-capped- note: string | null; // multi-chunk or suppression explanation- request_id: string | null;-}-```--## Logic Flow--```mermaid-flowchart TD- A[Receive params] --> B{len text > model cap?}- B -- yes --> X[NodeUserError]- B -- no --> C{speaker in model's voice set?}- C -- no --> X- C -- yes --> D{v3 && pace outside 0.5-2.0?}- D -- yes --> X- D -- no --> E[Build body: v3 -> temperature; v2 -> pitch/loudness/preprocessing]- E --> F[ctx.connection sarvam -> POST /text-to-speech]- F -- status >= 400 --> Y[NodeUserError w/ Sarvam message]- F -- ok --> G{audios empty?}- G -- yes --> X- G -- no --> H{return_audio}- H -- file --> I[_write_chunks -> workspace/audio, one file per chunk]- H -- base64 --> J{chunk 1 over _MAX_INLINE_B64?}- J -- yes --> K[audio_base64 null + note]- J -- no --> L[audio_base64 = chunk 1]- I --> M[track usage v3 or v2 rate]- K --> M- L --> M- M --> N[Return SarvamTextToSpeechOutput]-```--## Decision Logic--- **Per-model validation**: character cap, voice-set membership and (for v3)- pace range all fail before the HTTP call. A v2 voice on v3 is an error, not- a silent fallback.-- **Version-gated body**: v3 sends `temperature`; v2 sends `pitch` /- `loudness` / `enable_preprocessing`. They are mutually exclusive because v3- rejects the v2 fields outright — `displayOptions` hides them in the UI and- the body builder filters them regardless of what the caller sent.-- **Default is file, not base64.** A 2500-character v3 request is roughly- 12 MB of base64, which would land in the `node_outputs` JSON column, ride a- status-WebSocket broadcast, and — because this node is tool-exposed — be- serialized into an LLM message.-- **Inline cap**: `_MAX_INLINE_B64 = 1_000_000`. Over it, `audio_base64` stays- `None` and `note` says why and names `return_audio='file'`. Never a silent- truncation.-- **Multi-chunk**: Sarvam may split long text across `audios[]`. Each chunk is- a standalone container with its own header, so byte-wise concatenation- produces an unplayable file. One file per chunk; `file_path` is chunk 1;- `note` explains.-- **Billing**: `bulbul:v2` is half the v3 character rate, so the tracked- action differs (`text_to_speech` vs `text_to_speech_v2`).--## Side Effects--- **Database writes**: one `api_usage_metrics` row per call (characters).-- **Broadcasts**: standard node status only.-- **External API calls**: `POST https://api.sarvam.ai/text-to-speech`, `api-subscription-key` header.-- **File I/O**: writes `<workspace_dir>/audio/<slug>-<node8>-<rand6>[-N].<ext>` in `file` mode; creates the directory if absent. Falls back to a relative `audio/` dir when no workspace is set.-- **Subprocess**: none.--## External Dependencies--- **Credentials**: `SarvamCredential` via `ctx.connection("sarvam")`.-- **Python packages**: `httpx` (through `Connection`), stdlib `base64` / `uuid` / `re`.--## Edge cases & known limits--- **Audio is a file, not bytes.** Downstream nodes should consume `file_path`.- Only reach for `return_audio: "base64"` for very short clips that must be- inlined (e.g. chaining into `whatsappSend` with `media_source: "base64"`).-- Voice sets do not overlap between v2 and v3.-- `pitch` / `loudness` are silently unavailable on v3 — that is Sarvam's- limitation, surfaced through `displayOptions` rather than an error.-- Sample rates above 24000 are v3-REST-only.-- No streaming — the HTTP-stream and WebSocket TTS surfaces are not wired.-- Filenames embed a random suffix, so repeated runs never collide or- silently overwrite prior audio.--## Related--- **Skills using this as a tool**: `sarvam-speech-skill`.-- **Upstream nodes**: [`sarvamTransliterate`](./sarvamTransliterate.md) with `spoken_form: true` produces speech-ready text.-- **Peer nodes**: [`sarvamSpeechToText`](./sarvamSpeechToText.md).-- **Architecture docs**: [Sarvam AI Service](../../sarvam_service.md).diff --git a/docs-internal/node-logic-flows/language/sarvamTransliterate.md b/docs-internal/node-logic-flows/language/sarvamTransliterate.mdindex 69000258..6f4cc7d3 100644--- a/docs-internal/node-logic-flows/language/sarvamTransliterate.md+++ b/docs-internal/node-logic-flows/language/sarvamTransliterate.md@@ -15,7 +15,7 @@ Convert text between scripts while preserving pronunciation — romanised Hindi("namaste") to Devanagari ("नमस्ते") and back. **This is not translation**:the words and meaning are unchanged, only the writing system. Optionallyexpands numerals and abbreviations into spoken form, which is useful when the-output feeds `sarvamTextToSpeech`.+output feeds `textToSpeech`.## Inputs (handles)@@ -98,5 +98,5 @@ flowchart TD## Related- **Skills using this as a tool**: `sarvam-translate-skill`.-- **Peer nodes**: [`sarvamTranslate`](./sarvamTranslate.md), [`sarvamTextToSpeech`](./sarvamTextToSpeech.md) (a common downstream consumer of spoken-form output).+- **Peer nodes**: [`sarvamTranslate`](./sarvamTranslate.md), [`textToSpeech`](./textToSpeech.md) (a common downstream consumer of spoken-form output).- **Architecture docs**: [Sarvam AI Service](../../sarvam_service.md).diff --git a/docs-internal/node-logic-flows/language/speechToText.md b/docs-internal/node-logic-flows/language/speechToText.mdnew file mode 100644index 00000000..31dcc47e--- /dev/null+++ b/docs-internal/node-logic-flows/language/speechToText.md@@ -0,0 +1,84 @@+# Speech to Text (`speechToText`)++| Field | Value |+|------|-------|+| **Category** | language |+| **Backend handler** | [`server/nodes/speech/speech_to_text.py`](../../../server/nodes/speech/speech_to_text.py) |+| **Shared helpers** | [`server/nodes/speech/_base.py`](../../../server/nodes/speech/_base.py), [`_unifier.py`](../../../server/nodes/speech/_unifier.py), [`_providers/`](../../../server/nodes/speech/_providers/) |+| **Capabilities config** | [`server/config/speech_defaults.json`](../../../server/config/speech_defaults.json) |+| **Tests** | [`server/tests/nodes/test_speech.py`](../../../server/tests/nodes/test_speech.py) |+| **Skill (if any)** | [`server/skills/language_agent/speech-skill/SKILL.md`](../../../server/skills/language_agent/speech-skill/SKILL.md) |+| **Dual-purpose tool** | yes — tool name `speech_to_text` |++## Purpose++Transcribe audio through any configured provider. Replaces the vendor-locked+`sarvamSpeechToText`, and fixes two of its defects along the way (see+**Edge cases**).++## Inputs (handles)++| Handle | Connection type | Required | Purpose |+|--------|-----------------|----------|---------|+| `input-main` | main | no | Upstream data injected into params |++## Parameters++| Name | Type | Default | Required | Description |+|------|------|---------|----------|-------------|+| `provider` | enum | `deepgram` | no | Registry-driven: the enum is `stt_providers()` |+| `audio_file` | string \| AudioRef \| upload | `""` | yes | Widget `file`. Three shapes accepted — see below |+| `stt_model` | string | `""` | no | Blank uses the provider default. **Not** named `model` |+| `language` | string | `""` | no | Blank auto-detects |+| `translate` | boolean | `false` | no | Translate to English rather than transcribe |+| `diarize` | boolean | `false` | no | Speaker labels, where supported |+| `timestamps` | boolean | `false` | no | Per-word timing, where supported |+| `provider_options` | object | `{}` | no | Vendor-specific keys, passed through untouched |++`audio_file` accepts an `AudioRef` (from an upstream node or the upload+route), a bare workspace path, or the legacy `{type: "upload", data: base64}`+envelope the file widget used to emit. All three route through+`services.media`, which resolves under containment.++## Outputs++| Field | Type | Description |+|-------|------|-------------|+| `transcript` | string | The text |+| `language` / `language_confidence` | string / float | Detected or echoed |+| `duration_seconds` | float | Provider-reported, else probed from the file |+| `words` | object[] | `word` / `start` / `end` / `speaker` / `confidence` |+| `segments` | object[] | Utterances or segments, provider-shaped |+| `provider` / `stt_model` / `request_id` | string | What actually ran |++## Side effects++- Records an `APIUsageMetric` keyed on the provider id, in that provider's+ own billing unit — seconds for OpenAI / Groq / Sarvam, **minutes** for+ Deepgram (converted by the provider module, not by shared code)++## Edge cases++- **Path traversal — fixed.** The predecessor joined a user-supplied path+ onto the workspace root unchecked, so `audio_file="../../credentials.db"`+ read the encrypted credential store and uploaded it to the provider.+ Every input shape now resolves through `services.media`, which rejects+ `..` / `~` / drive prefixes and re-checks containment after resolution.+- **Billing — fixed.** The predecessor charged every transcription as a flat+ 30 seconds because it never measured the clip. Duration is now taken from+ the provider when reported, otherwise probed with `inspect_audio`; when+ neither is possible no metric is written, because an under-count is honest+ and a fabricated one is not.+- **Capability downgrades**: `verbose_json` is a 400 on `gpt-4o-transcribe`,+ so a timestamp request on a gated model is downgraded with a WARN rather+ than failing.+- **Groq**: bills a 10-second floor per request; the provider applies it so+ cost reporting matches the invoice. `whisper-large-v3-turbo` cannot+ translate and falls back with a WARN.+- **Sarvam**: timestamps and diarization are batch-API-only and come back+ empty on the synchronous endpoint.++## Related++- **Peer nodes**: [`textToSpeech`](./textToSpeech.md) — the other half of a voice loop.+- **RFC**: [`speech_provider_rfc.md`](../../speech_provider_rfc.md)diff --git a/docs-internal/node-logic-flows/language/textToSpeech.md b/docs-internal/node-logic-flows/language/textToSpeech.mdnew file mode 100644index 00000000..82c5b6ff--- /dev/null+++ b/docs-internal/node-logic-flows/language/textToSpeech.md@@ -0,0 +1,84 @@+# Text to Speech (`textToSpeech`)++| Field | Value |+|------|-------|+| **Category** | language |+| **Backend handler** | [`server/nodes/speech/text_to_speech.py`](../../../server/nodes/speech/text_to_speech.py) |+| **Shared helpers** | [`server/nodes/speech/_base.py`](../../../server/nodes/speech/_base.py), [`_unifier.py`](../../../server/nodes/speech/_unifier.py), [`_providers/`](../../../server/nodes/speech/_providers/) |+| **Capabilities config** | [`server/config/speech_defaults.json`](../../../server/config/speech_defaults.json) |+| **Tests** | [`server/tests/nodes/test_speech.py`](../../../server/tests/nodes/test_speech.py) |+| **Skill (if any)** | [`server/skills/language_agent/speech-skill/SKILL.md`](../../../server/skills/language_agent/speech-skill/SKILL.md) |+| **Dual-purpose tool** | yes — tool name `text_to_speech` |++## Purpose++Synthesize speech through any configured provider. Replaces the vendor-locked+`sarvamTextToSpeech`: the provider is a parameter, so switching vendors never+means swapping the node out of a workflow.++## Inputs (handles)++| Handle | Connection type | Required | Purpose |+|--------|-----------------|----------|---------|+| `input-main` | main | no | Upstream data injected into params |++Handles are declared visible explicitly (`hide_input_handle = False` /+`hide_output_handle = False`) because `usable_as_tool = True` otherwise+auto-hides both, which would break chaining into `speechToText`.++## Parameters++| Name | Type | Default | Required | Description |+|------|------|---------|----------|-------------|+| `provider` | enum | `elevenlabs` | no | Registry-driven: the enum is `tts_providers()` |+| `text` | string | - | yes | Text to speak; capped per provider/model |+| `tts_model` | string | `""` | no | Blank uses the provider default. **Not** named `model` — see below |+| `voice` | string | `""` | no | Loader `speechVoices`; live catalogue on ElevenLabs, config elsewhere |+| `language` | string | `""` | no | Required by Sarvam (e.g. `hi-IN`), auto-detected elsewhere |+| `speed` | number | `null` | no | Clamped to the provider's documented range |+| `output_format` | string | `""` | no | Blank uses the provider default |+| `sample_rate` | number | `null` | no | Where the provider supports it |+| `provider_options` | object | `{}` | no | Vendor-specific keys, passed through untouched |++**`tts_model`, not `model`.** [`ParameterRenderer.tsx:866`](../../../client/src/components/ParameterRenderer.tsx#L866)+overwrites any field literally named `model` or `api_key` with chat-model data+whenever a sibling `provider` field exists — and it never checks that the+provider is an LLM provider. A field named `model` here would be cleared the+moment a user picked ElevenLabs.++## Outputs++| Field | Type | Description |+|-------|------|-------------|+| `audio` | AudioRef | The generated clip; the first when several |+| `files` | AudioRef[] | Every clip |+| `chunk_count` | int | >1 when the provider split long input |+| `provider` / `tts_model` / `voice` | string | What actually ran |+| `request_id` | string | Provider request id, where returned |+| `note` | string | Set when several clips came back |++`AudioRef` carries a path and metadata, never bytes — see+[`services/media/limits.py`](../../../server/services/media/limits.py) for the+measured reason.++## Side effects++- Writes one file per clip into `<workspace>/audio/` via `write_audio`+ (atomic, random-suffixed filename, no collisions across runs)+- Records an `APIUsageMetric` keyed on the provider id, in that provider's+ own billing unit (characters for all three current TTS providers)++## Edge cases++- **Several clips**: Sarvam splits long input into standalone files. Each has+ its own container header, so byte-concatenation produces audio that plays+ only the first chunk. `note` says so explicitly.+- **ElevenLabs without a voice**: refused up front — there is no+ account-wide default voice.+- **Over-cap text**: refused before the paid call, naming the cap.+- **Unsupported provider**: `NodeUserError` listing the registered ones.++## Related++- **Peer nodes**: [`speechToText`](./speechToText.md) — the other half of a voice loop.+- **RFC**: [`speech_provider_rfc.md`](../../speech_provider_rfc.md)diff --git a/docs-internal/sarvam_service.md b/docs-internal/sarvam_service.mdindex 7f08f2ce..37fc52c7 100644--- a/docs-internal/sarvam_service.md+++ b/docs-internal/sarvam_service.md@@ -144,8 +144,13 @@ nothing here owns long-lived state, so the folder is helpers plus node files.| `sarvamTranslate` | `sarvam_translate` | `POST /translate` | characters || `sarvamTransliterate` | `sarvam_transliterate` | `POST /transliterate` | characters || `sarvamDetectLanguage` | `sarvam_detect_language` | `POST /text-lid` | characters |-| `sarvamSpeechToText` | `sarvam_speech_to_text` | `POST /speech-to-text` (multipart) | seconds |-| `sarvamTextToSpeech` | `sarvam_text_to_speech` | `POST /text-to-speech` | characters |++Sarvam's two speech endpoints are no longer nodes in this folder. They are+reached through the provider-abstracted `textToSpeech` / `speechToText` nodes+in [`server/nodes/speech/`](../server/nodes/speech/), where Sarvam is one+provider among several; the wire logic was ported verbatim into+[`_providers/sarvam.py`](../server/nodes/speech/_providers/sarvam.py). See+[Speech Provider RFC](./speech_provider_rfc.md).All authenticate through `ctx.connection("sarvam")`, which injects`api-subscription-key` via `ApiKeyCredential.inject` — no per-node auth code.@@ -161,84 +166,30 @@ Per-model input caps are enforced client-side (`mayura:v1` 1000 chars,`sarvam-translate:v1` 2000, `/text-lid` 1000) so the LLM gets an actionablemessage instead of an opaque 422.-### Speech-to-text: multipart and the 30-second wall+### Speech: moved to the provider-abstracted nodes-This is the repo's first multipart upload. Rather than bypass the authed-facade, `Connection.request` gained a generic `files=` parameter-([`connection.py`](../server/services/plugin/connection.py)) — threaded into-**both** the initial request and the auth-retry rebuild, so a 401 replays the+The multipart plumbing this folder introduced is still load-bearing and still+generic: `Connection.request` gained a `files=` parameter+([`connection.py`](../server/services/plugin/connection.py)), threaded into+**both** the initial request and the auth-retry rebuild so a 401 replays thefile parts instead of an empty body. Pass file *bytes*, never an open handle,for exactly that reason.-`audio_file` accepts **either** shape the frontend can produce: a path string-(absolute, or relative to the workflow workspace) or the file widget's-`{"type": "upload", "data": "<base64>", "filename", "mimeType"}` envelope.-Typing it as `Union[str, Dict]` is deliberate — `whatsapp_send` declares the-same field as `str` and only survives because its handler reads the raw dict.--**The synchronous endpoint caps at 30 seconds of audio.** Longer recordings-need Sarvam's Batch API, a separate job-based surface that is out of scope.-Diarization is Batch-only too, so `diarized_transcript` is declared on the-output model but will be `None` here.--### Text-to-speech: files, not base64--Sarvam returns audio as base64 inside the JSON body. A 2500-character v3-request is roughly **12 MB of base64** — which would land in the-`node_outputs` JSON column, ride a status-WebSocket broadcast, and (because the-node is tool-exposed) be serialized into an LLM message.--So the default is `return_audio: "file"`: audio is written under-`<workspace>/audio/` and the node returns `file_path`. `return_audio: "base64"`-is opt-in and capped at `_MAX_INLINE_B64 = 1_000_000`; over the cap-`audio_base64` stays `None` and `note` explains why and points at the file-mode. It never silently truncates.--Sarvam may split long text into several chunks in `audios[]`. Each chunk is a-standalone container with its own header, so **concatenating them byte-wise-produces an unplayable file**. The node writes one file per chunk, points-`file_path` at the first, lists all of them in `files`, and sets `note`.--Voices are per model and do not overlap — 37 for `bulbul:v3`, 7 for-`bulbul:v2`. Asking for a v2 voice on v3 is a `NodeUserError`, not a silent-fallback. v2-only params (`pitch`, `loudness`, `enable_preprocessing`) and-v3-only params (`temperature`, `dict_id`) are gated by `displayOptions` in the-UI *and* filtered from the request body, since v3 rejects the v2 fields.-------## Cost tracking--`@Operation(cost=...)` is declarative metadata that nothing reads at runtime,-so attribution is an explicit `track_sarvam_usage()` call per operation —-the same shape as [`nodes/twitter/_base.py`](../server/nodes/twitter/_base.py).-It never raises: a metrics failure must not fail a successful API call.--Sarvam publishes INR list prices. [`pricing.json`](../server/config/pricing.json)-stores USD converted at **1 INR = 0.0113 USD (~88.5 INR/USD, 2026-07)**, with-the rate and the source figures recorded in a `_note` key. Both an-`api.sarvam` block and an `operation_map.sarvam` block are required —-`calculate_api_cost` returns 0 without the latter.--Token pricing for the chat models lives separately under `llm.sarvam`.-------## Icons--`@lobehub/icons` has no Sarvam brand (278 entries, none matching), so the-usual `lobehub:<brand>` route in `visuals.json` is unavailable. Instead the-mark ships as SVG and is served by the backend:--- `server/nodes/model/sarvam_chat_model/icon.svg` and `server/nodes/sarvam/icon.svg` → `GET /api/schemas/nodes/<type>/icon`-- `server/credentials/icons/sarvam.svg` → `GET /api/schemas/credentials/sarvam/icon`--The frontend's `/api/` branch in-[`assets/icons/index.ts`](../client/src/assets/icons/index.ts) already resolves-that wire format, so no icon-registry edit was needed — only an `<img>` wrapper-in `AIProviderIcons.tsx` for the direct-FC consumers. Skills inherit the same-icons automatically: `_parse_skill_metadata` tries `get_plugin_icon_path`-before `visuals.json`.+Two behaviours that were discovered here and are preserved in the speech+plugin:++- **Sarvam returns TTS audio as base64 inside JSON, and `audios` is an+ array.** Long input comes back as several standalone clips, each with its+ own container header, so concatenating them byte-wise produces audio that+ plays only the first chunk. The node writes one file per clip and says so.+- **Timestamps and diarization are Batch-API-only.** The synchronous endpoint+ returns them as `null` regardless of what is requested.++One claim from the original implementation did **not** survive verification:+it asserted the synchronous endpoint "caps at 30 seconds of audio", and billed+every transcription at that ceiling. Sarvam's only 30-second reference is+about *response latency*, not audio duration. The speech node measures real+duration instead.### The mark itselfdiff --git a/docs-internal/speech_provider_rfc.md b/docs-internal/speech_provider_rfc.mdindex 46f0629a..4f32aa0d 100644--- a/docs-internal/speech_provider_rfc.md+++ b/docs-internal/speech_provider_rfc.md@@ -117,8 +117,14 @@ hardcodes `self.credentials[0]` ([`base.py:566`](../server/services/plugin/base.provider-agnostic. Copying it into `services/speech/` would fork the boot-time-import-avoidancelogic that exists specifically to keep ~7 s (warm) / ~45 s (cold) of SDK imports off startup.-`services/provider_registry.py` + `services/provider_clients.py`; `services/llm/registry.py`-becomes a shim. **Success criterion: `server/tests/llm/` passes untouched.**+`services/provider_registry.py`; `services/llm/registry.py` becomes a shim.+**Success criterion: `server/tests/llm/` passes untouched.** Met — 168 tests, no edits. `_REGISTRY`+survives as an alias bound to the *same dict object* the registry mutates, because several tests+swap provider factories in place through it.++A companion `services/provider_clients.py` (the lease-counted client cache) was scoped here and+deliberately **not** built. See §4: speech makes one HTTP call per node execution, so the cache has+no second consumer and would have earned only a shutdown hook in `main.py`.### D7 — `tinytag` for duration and format@@ -159,42 +165,86 @@ retrying again.## 4. Architecture+Speech is a **plugin**, not a service. Everything vendor-specific lives in the plugin folder;+only genuinely shared machinery sits under `services/`.+```-services/provider_registry.py generic ProviderSpec + registry (extracted from llm/)-services/provider_clients.py generic lease-counted client cache (extracted from llm/)+services/provider_registry.py generic ProviderSpec + registry (shared with services/llm)-services/media/ audio transport — vendor-neutral, reusable for image/video+services/media/ audio transport -- vendor-neutral, reusable for image/videorefs.py AudioRefworkspace.py write_audio / resolve_media / read_media_bytes / coerce_file_paraminspect.py tinytag -> wave -> PCM arithmetic; never raiseslimits.py every size constant, one place-services/speech/ provider abstraction — mirrors services/llm/- protocol.py TtsProvider / SttProvider, requests, results, SpeechError- registry.py two registries over the generic one- config.py reads server/config/speech_defaults.json- unifier.py SpeechUnifier — dispatch + typed-error -> NodeUserError- providers/ _http.py, _openai_compat.py (openai+groq STT), elevenlabs.py,- sarvam.py, deepgram.py, gemini.py--nodes/speech/ the two nodes+nodes/speech/ the whole speech surface+ text_to_speech.py the two nodes+ speech_to_text.py+ _protocol.py TtsProvider / SttProvider, requests, results, SpeechError+ _registry.py two registries over the generic one+ _config.py reads server/config/speech_defaults.json+ _unifier.py dispatch + typed-error -> NodeUserError+ _providers/ _http.py, _openai_compat.py (openai+groq), elevenlabs.py,+ deepgram.py, sarvam.py+ _credentials.py ElevenLabs + Deepgram (the rest are shared with nodes/model)+ _base.py, _option_loaders.py++config/speech_defaults.json operator-editable capabilitiesrouters/workspace.py GET file (Range-capable) + POST upload```-**Layering rule.** `services/speech` takes credential **id strings**, never `Credential` classes.-The classes live under `nodes/`, and a `services → nodes` import inverts the layering and breaks-`test_plugin_self_containment.py`. A test locks the two sides agree.+**Why not `services/speech/`.** An earlier draft put the provider layer under `services/`, mirroring+`services/llm/`. That mirror is misleading: `services/llm` earns its place because agents, chat+models and a dozen nodes all consume it, whereas speech has exactly two consumers and both live in+`nodes/speech/`. Per-vendor code under `services/` is the pattern Wave 11.I explicitly retired --+`services.whatsapp_service`, `services.maps` and `services.nodejs_client` all moved into plugin+folders, and `test_plugin_self_containment.py` names them in a forbidden-import list. Speech would+have been the next entry on it.++**Layering rule.** The speech layer takes credential **id strings**, never `Credential` classes.+`nodes/speech/_config.credential_id(provider)` reads the id from JSON, and a test asserts every+registered provider's id resolves in `CREDENTIAL_REGISTRY`. Cross-plugin *credential* imports are+fine and idiomatic (`from ..model._credentials import OpenAICredential`, exactly what `nodes/sarvam/`+already does); what stays out is any `services/` module knowing a vendor name.++**No client cache.** `ChatUnifier` keeps a lease-counted LRU because an agent loop makes many model+calls inside one node execution. Speech is the opposite shape -- one HTTP request per execution,+where client setup is invisible next to a multi-second synthesis. Caching would buy nothing and cost+a process-wide singleton plus a shutdown hook wired into `main.py`. This also settles the question+left open when the registry was extracted: no second consumer materialised, so no shared client+cache was built.----## 5. Two bugs this work closes--1. **Path traversal in `sarvam_speech_to_text`.** `_read_audio` does `Path(workspace_dir) / raw`- with no containment check, so `audio_file="../../credentials.db"` reads the encrypted credential- store and uploads it to the provider. `coerce_file_param` closes it by construction.-2. **Workflow rename orphans every workspace file.** `rename_workflow` is a single-row `UPDATE`; the+## 5. Bugs this work closes++1. **Path traversal in `sarvam_speech_to_text`.** `_read_audio` did `Path(workspace_dir) / raw`+ with no containment check, so `audio_file="../../credentials.db"` read the encrypted credential+ store and uploaded it to the provider. `coerce_file_param` closes it by construction, and the+ node that carried the bug no longer exists.+2. **Flat 30-second billing.** The same node charged every transcription as 30 seconds, with the+ comment "we do not decode the clip to measure it". It also justified the figure as a documented+ endpoint limit, which the docs do not actually say (§7). Duration is now measured.+3. **`workspace_root()` resolved by id where the directory is named by slug.** Introduced by this+ work's own first wave: the ctx-less fallback composed `workspaces/<workflow_id>/`, a path that+ never exists, because `WorkflowService._get_workspace_dir` names those directories after+ `Workflow.slug`. It stayed invisible because every caller happened to pass a `NodeContext` — the+ workspace HTTP route is the first that cannot. The fallback now raises rather than guessing, and+ the route does the id → slug lookup itself, since that needs a database read and `services.media`+ is synchronous by contract. `core.paths.workspace_dir`'s parameter was renamed to+ `workflow_slug` so the next reader is not misled the same way.+4. **Workflow rename orphans every workspace file.** `rename_workflow` is a single-row `UPDATE`; thedirectory is never moved, and `_get_workspace_dir` keys on the mutable slug. Pre-existing and notaudio-specific, but `AudioRef` makes it fixable — a best-effort `os.replace` in the rename path.+ Still outstanding.++### On migrating existing workflows++None was written. The two Sarvam speech nodes shipped only days before this work and no saved+workflow references them, so a `workflow_migrations` entry would have been dead code guarding+against a case that does not exist. Should one appear, the graph rewrite is the easy half; the+parameter half is best-effort, because only 2 of the 5 migration call sites pass `node_parameters`+and neither of those is the load path.---@@ -212,33 +262,66 @@ The classes live under `nodes/`, and a `services → nodes` import inverts the l## 7. Verified provider surface-Confirmed against live documentation on 2026-07-25. Anything not listed here was **not** verified-and must be checked before it is coded.+Confirmed against live vendor documentation on 2026-07-25/26, for the **v1 provider set only**+(OpenAI, Groq, ElevenLabs, Deepgram, Sarvam). Gemini, Azure, AssemblyAI and Cartesia are out of+scope and deliberately unverified.-**OpenAI** — TTS `POST /v1/audio/speech`, models `gpt-4o-mini-tts` / `tts-1` / `tts-1-hd`, voices-`alloy ash ballad coral echo fable nova onyx sage shimmer verse marin cedar` (subset for tts-1),-`response_format` ∈ `mp3 opus aac flac wav pcm`, plus `instructions` and `speed`. Raw binary-response. STT `POST /v1/audio/transcriptions`, multipart.+The set diverges on every axis, which is what makes it a real test of the abstraction rather than+four variations of one shape:-**ElevenLabs** — TTS `POST /v1/text-to-speech/{voice_id}`, header `xi-api-key`, body `text`,-`model_id` (default `eleven_multilingual_v2`), `voice_settings{stability, similarity_boost, style,-use_speaker_boost, speed}`, `language_code`, `seed`, `apply_text_normalization` ∈ `auto|on|off`;-**query** `output_format` (default `mp3_44100_128`, 27 values). Binary response. Models via-`GET /v1/models`, voices via `GET /v1/voices`. STT `POST /v1/speech-to-text`, multipart,-`model_id` ∈ `scribe_v2|scribe_v1`, `language_code`, `tag_audio_events` (default true),-`num_speakers` (≤32), `timestamps_granularity` ∈ `none|word|character`, `diarize` (default false).-Response `{language_code, language_probability, text, words[]}`.--**Groq** — STT `POST https://api.groq.com/openai/v1/audio/transcriptions`, OpenAI-compatible,-`whisper-large-v3` / `whisper-large-v3-turbo`. This is why `_openai_compat.py` covers both vendors-off one factory.--**Deepgram** — TTS `POST /v1/speak`, header `Authorization: Token <key>`, model names of the form-`aura-2-<voice>-<lang>`. STT `POST /v1/listen`. Query-parameter driven; exact parameter set to be-confirmed at implementation time.--**Sarvam** — already implemented; see [`sarvam_service.md`](./sarvam_service.md). TTS returns-base64 inside JSON (`{audios: […]}`), the one provider that does.--**Gemini** — TTS via `generateContent` with an audio response modality. Shape **not yet verified**;-must be confirmed before coding.+| | auth header | request transport | response transport |+|---|---|---|---|+| OpenAI TTS | `Authorization: Bearer` | JSON body | **raw audio bytes** |+| OpenAI / Groq STT | `Authorization: Bearer` | multipart | JSON |+| ElevenLabs TTS | **`xi-api-key`** (no scheme) | JSON body + **query params** | **raw audio bytes** |+| Deepgram STT | **`Authorization: Token`** | **raw body bytes** + **query params** | JSON |+| Sarvam TTS | `api-subscription-key` | JSON body | **base64 in a JSON array** |+| Sarvam STT | `api-subscription-key` | multipart | JSON |++**OpenAI** -- TTS `POST /v1/audio/speech`; models `gpt-4o-mini-tts` (default), `tts-1`, `tts-1-hd`;+13 voices, of which `tts-1` / `tts-1-hd` support only 9 (`ballad`, `verse`, `marin`, `cedar` are+excluded); `response_format` in `mp3 opus aac flac wav pcm`; `speed` 0.25-4.0; input <=4096 chars;+`instructions` works only on `gpt-4o-mini-tts`. STT `POST /v1/audio/transcriptions` (multipart), and+**`response_format` is model-gated**: `whisper-1` allows `json text srt verbose_json vtt`, the+`gpt-4o-*-transcribe` models allow only `json text`. Since `timestamp_granularities` requires+`verbose_json`, word timestamps are effectively whisper-1 only, and requesting them elsewhere is a+400 -- so the node downgrades rather than sends.++**Groq** -- STT `POST https://api.groq.com/openai/v1/audio/transcriptions`, OpenAI-compatible;+`whisper-large-v3-turbo` (default) and `whisper-large-v3`. **Turbo omits translation entirely.**+`prompt` is capped at 224 tokens, and Groq bills a **10-second minimum per request** regardless of+clip length.++**ElevenLabs** -- TTS `POST /v1/text-to-speech/{voice_id}`; header `xi-api-key`, bare, no scheme+keyword; body `text`, `model_id` (default `eleven_multilingual_v2`), `language_code`,+`voice_settings`, `seed`, `apply_text_normalization`; **`output_format` is a query parameter**+(default `mp3_44100_128`) and is silently ignored if placed in the body. Voices via+`GET /v2/voices`, cursor-paginated (`has_more` + `next_page_token`; the docs say not to rely on+`total_count`). `speed` has **two conflicting official ranges** -- schema 0.5-2.0, best-practices+prose 0.7-1.2, never reconciled; the schema bound is what the API accepts and is what is enforced.+Whether `voice_settings` are honoured on `eleven_v3` is **undocumented**.++**Deepgram** -- STT `POST /v1/listen`; `Authorization: Token <key>`, **not** Bearer; **all options+are query parameters** (`punctuate`, `diarize`, `smart_format`, `detect_language`, `paragraphs`,+`utterances`, ...), with multi-value options as repeated keys rather than comma lists; audio is the+**raw request body** with an `audio/*` content type, not multipart. Transcript at+`results.channels[N].alternatives[N].transcript`. The documented default model is `base-general`, so+an explicit default is configured instead. **No cloud file-size or duration cap is published** -- the+25 MB figure in circulation belongs to their self-hosted SageMaker docs and is not asserted here.++**Sarvam** -- TTS `POST /text-to-speech` returns **base64 inside JSON**, and `audios` is an **array**+of standalone clips. Explicit nulls are rejected where an absent key is accepted. `bulbul:v3` takes+`temperature` and rejects `pitch` / `loudness` / `enable_preprocessing`; `bulbul:v2` is the mirror+image. STT `POST /speech-to-text` (multipart); timestamps and diarization are **batch-API-only** and+come back null here.++### Corrections to earlier drafts++- An earlier version of this section listed an ElevenLabs STT endpoint and a Deepgram TTS endpoint.+ Both products exist, but neither is in the v1 set and neither was re-verified, so neither is+ implemented.+- The Sarvam node asserted that its synchronous STT endpoint "caps at 30 seconds of audio", and+ billed every transcription at that ceiling. The docs' only 30-second reference is about **response+ latency**, not audio duration. The claim was wrong and the billing with it.+- Sarvam's own docs contradict themselves on `speech_sample_rate` (schema `22050`, prose `24000`).+ The configured default is `24000`, matching the prose and the previous implementation.diff --git a/server/config/pricing.json b/server/config/pricing.jsonindex 6fb9b517..7b5741da 100644--- a/server/config/pricing.json+++ b/server/config/pricing.json@@ -264,6 +264,35 @@"text_to_speech_v2": 0.00001695,"speech_to_text": 0.00009417,"_default": 0.0000226+ },+ "elevenlabs": {+ "_description": "Text-to-speech, billed per CHARACTER of input. ElevenLabs prices by subscription tier rather than by a flat published per-character rate, so this is the Creator-tier effective rate (~$0.30 per 1,000 characters) and is an ESTIMATE. Operators on another tier should edit this value — the node reports the character count truthfully either way.",+ "_source": "https://elevenlabs.io/pricing",+ "_updated": "2026-07",+ "text_to_speech": 0.0003,+ "_default": 0.0003+ },+ "deepgram": {+ "_description": "Speech-to-text, billed per MINUTE of audio. Pay-as-you-go rate for the nova-3 family. The node converts measured seconds to minutes before recording, so resource_count here is minutes, not seconds.",+ "_source": "https://deepgram.com/pricing",+ "_updated": "2026-07",+ "speech_to_text": 0.0043,+ "_default": 0.0043+ },+ "openai": {+ "_description": "OpenAI's /audio/* routes only. These are NOT token-priced, so they live here rather than under `llm.openai` — the two blocks are separate namespaces and do not collide. text_to_speech bills per CHARACTER of input ($0.60 per 1M characters for gpt-4o-mini-tts, the default model; tts-1 is ~25x that, and this table has no per-model axis, so a tts-1-heavy install should raise this value). speech_to_text bills per SECOND ($0.006/minute for whisper-1).",+ "_source": "https://openai.com/api/pricing/",+ "_updated": "2026-07",+ "text_to_speech": 0.0000006,+ "speech_to_text": 0.0001,+ "_default": 0.0001+ },+ "groq": {+ "_description": "Groq transcription only (token pricing lives under `llm.groq`). Billed per SECOND of audio: $0.04/hour for whisper-large-v3-turbo, the default model. Groq bills a 10-second floor per request, which the provider module applies before reporting the count.",+ "_source": "https://groq.com/pricing/",+ "_updated": "2026-07",+ "speech_to_text": 0.0000111,+ "_default": 0.0000111}},@@ -368,6 +397,20 @@"text_to_speech": "text_to_speech","text_to_speech_v2": "text_to_speech_v2","speech_to_text": "speech_to_text"+ },+ "_comment_speech": "Every speech provider needs an entry here as well as in api_pricing: calculate_api_cost() returns a zero cost SILENTLY when an action has no operation_map entry, so a missing line reads as 'free' rather than as an error.",+ "elevenlabs": {+ "text_to_speech": "text_to_speech"+ },+ "deepgram": {+ "speech_to_text": "speech_to_text"+ },+ "openai": {+ "text_to_speech": "text_to_speech",+ "speech_to_text": "speech_to_text"+ },+ "groq": {+ "speech_to_text": "speech_to_text"}},diff --git a/server/config/speech_defaults.json b/server/config/speech_defaults.jsonnew file mode 100644index 00000000..a70ac8d9--- /dev/null+++ b/server/config/speech_defaults.json@@ -0,0 +1,206 @@+{+ "_comment": "Per-provider speech capabilities. Sibling of llm_defaults.json and read the same way: services/speech/config.py loads this once at import and every capability lookup goes through it, so shared code never branches on a provider name. Values here were verified against live official docs; see docs-internal/speech_provider_rfc.md section 7 for the sources and for the handful of items the docs do not state. Per-model overrides use the same {'model-id': value, '_default': value} shape as llm_defaults.json's max_output_tokens.",+ "version": "1.0",+ "providers": {+ "openai": {+ "base_url": "https://api.openai.com/v1",+ "credential_id": "openai",+ "tts": {+ "endpoint": "/audio/speech",+ "default_model": "gpt-4o-mini-tts",+ "models": ["gpt-4o-mini-tts", "tts-1", "tts-1-hd"],+ "_voices_comment": "13 voices exist, but tts-1 and tts-1-hd support only the first 9. Per-model override rather than a flat list so the node can validate before the provider 400s.",+ "voices": {+ "tts-1": ["alloy", "ash", "coral", "echo", "sage", "shimmer", "fable", "onyx", "nova"],+ "tts-1-hd": ["alloy", "ash", "coral", "echo", "sage", "shimmer", "fable", "onyx", "nova"],+ "_default": ["alloy", "ash", "ballad", "coral", "echo", "sage", "shimmer", "verse", "marin", "cedar", "fable", "onyx", "nova"]+ },+ "default_voice": "alloy",+ "output_formats": ["mp3", "opus", "aac", "flac", "wav", "pcm"],+ "default_output_format": "mp3",+ "speed_range": [0.25, 4.0],+ "max_input_chars": 4096,+ "billed_unit": "characters",+ "_instructions_comment": "Steerability via the `instructions` body field is gpt-4o-mini-tts only; it is silently ignored on tts-1 / tts-1-hd.",+ "supports_instructions": {+ "tts-1": false,+ "tts-1-hd": false,+ "_default": true+ }+ },+ "stt": {+ "endpoint": "/audio/transcriptions",+ "translate_endpoint": "/audio/translations",+ "default_model": "whisper-1",+ "models": ["whisper-1", "gpt-4o-transcribe", "gpt-4o-mini-transcribe", "gpt-4o-transcribe-diarize"],+ "_response_formats_comment": "Model-gated, and getting this wrong is a 400 rather than a degraded result. Because timestamp_granularities requires verbose_json, word timestamps are effectively whisper-1 only.",+ "response_formats": {+ "whisper-1": ["json", "text", "srt", "verbose_json", "vtt"],+ "gpt-4o-transcribe-diarize": ["json", "text", "diarized_json"],+ "_default": ["json", "text"]+ },+ "default_response_format": "json",+ "supports_timestamps": {+ "whisper-1": true,+ "_default": false+ },+ "supports_diarization": {+ "gpt-4o-transcribe-diarize": true,+ "_default": false+ },+ "supports_prompt": {+ "gpt-4o-transcribe-diarize": false,+ "_default": true+ },+ "max_upload_bytes": 26214400,+ "input_formats": ["flac", "mp3", "mp4", "mpeg", "mpga", "m4a", "ogg", "wav", "webm"],+ "billed_unit": "seconds"+ }+ },+ "groq": {+ "_comment": "Serves OpenAI's exact transcription wire format at its own base URL, so it rides the same provider class. Transcription-only: no TTS product.",+ "base_url": "https://api.groq.com/openai/v1",+ "credential_id": "groq",+ "stt": {+ "endpoint": "/audio/transcriptions",+ "translate_endpoint": "/audio/translations",+ "default_model": "whisper-large-v3-turbo",+ "models": ["whisper-large-v3-turbo", "whisper-large-v3"],+ "_translate_comment": "The turbo model omits translation entirely; only whisper-large-v3 can translate.",+ "supports_translate": {+ "whisper-large-v3-turbo": false,+ "_default": true+ },+ "response_formats": {+ "_default": ["json", "text", "verbose_json"]+ },+ "default_response_format": "json",+ "supports_timestamps": {+ "_default": true+ },+ "supports_diarization": {+ "_default": false+ },+ "max_prompt_tokens": 224,+ "max_upload_bytes": 26214400,+ "billed_unit": "seconds",+ "_min_billed_comment": "Groq bills a 10-second floor regardless of clip length; the provider reports max(duration, 10).",+ "min_billed_seconds": 10+ }+ },+ "elevenlabs": {+ "base_url": "https://api.elevenlabs.io",+ "credential_id": "elevenlabs",+ "tts": {+ "_endpoint_comment": "voice_id is a PATH segment, and output_format is a QUERY parameter rather than a body field. Sending output_format in the JSON body yields the default silently instead of an error.",+ "endpoint": "/v1/text-to-speech/{voice_id}",+ "voices_endpoint": "/v2/voices",+ "default_model": "eleven_multilingual_v2",+ "models": ["eleven_multilingual_v2", "eleven_v3", "eleven_flash_v2_5", "eleven_flash_v2", "eleven_ttv_v3"],+ "default_voice": "",+ "output_formats": [+ "mp3_22050_32", "mp3_24000_48", "mp3_44100_32", "mp3_44100_64",+ "mp3_44100_96", "mp3_44100_128", "mp3_44100_192",+ "opus_48000_32", "opus_48000_64", "opus_48000_96", "opus_48000_128", "opus_48000_192",+ "pcm_8000", "pcm_16000", "pcm_22050", "pcm_24000", "pcm_32000", "pcm_44100", "pcm_48000",+ "wav_8000", "wav_16000", "wav_22050", "wav_24000", "wav_32000", "wav_44100", "wav_48000",+ "alaw_8000", "ulaw_8000"+ ],+ "default_output_format": "mp3_44100_128",+ "_tier_gated_comment": "mp3 at 192 kbps needs Creator+; PCM and WAV at 44.1 kHz need Pro+. A tier miss surfaces as a permission error, which is why the panel does not pre-filter.",+ "_speed_comment": "The OpenAPI schema allows 0.5-2.0 while the best-practices guide recommends 0.7-1.2. Both are official and were never reconciled, so the API bound is enforced and the narrower range is guidance only.",+ "speed_range": [0.5, 2.0],+ "voice_settings_defaults": {+ "stability": 0.5,+ "similarity_boost": 0.75,+ "style": 0.0,+ "use_speaker_boost": true+ },+ "billed_unit": "characters"+ }+ },+ "deepgram": {+ "_comment": "Options are QUERY parameters, audio is sent as a RAW body with an audio/* Content-Type (not multipart), and auth is 'Authorization: Token <key>' rather than Bearer. All three differ from every other provider here.",+ "base_url": "https://api.deepgram.com/v1",+ "credential_id": "deepgram",+ "stt": {+ "endpoint": "/listen",+ "_default_model_comment": "Deepgram's own documented default is base-general, which is not what anyone wants in 2026, so an explicit default is set here rather than inherited.",+ "default_model": "nova-3",+ "models": [+ "nova-3", "nova-3-general", "nova-3-medical",+ "nova-2", "nova-2-meeting", "nova-2-phonecall", "nova-2-voicemail",+ "nova-2-finance", "nova-2-conversationalai", "nova-2-video", "nova-2-medical",+ "nova", "nova-general", "nova-phonecall",+ "whisper-large", "whisper-medium", "whisper-small", "whisper-base", "whisper-tiny"+ ],+ "supports_timestamps": {+ "_default": true+ },+ "supports_diarization": {+ "_default": true+ },+ "_smart_format_comment": "smart_format supersedes punctuate and additionally applies paragraphing plus number/date/currency formatting.",+ "default_smart_format": true,+ "_max_upload_comment": "Deepgram publishes no cloud file-size or duration cap. The 25 MB figure in circulation belongs to their self-hosted SageMaker docs and does not apply here, so no limit is asserted; services.media's read cap is the only ceiling.",+ "max_upload_bytes": null,+ "billed_unit": "minutes"+ }+ },+ "sarvam": {+ "_comment": "Indic-first. TTS returns base64 inside a JSON array rather than raw bytes, and long input comes back as several standalone clips that must not be byte-concatenated.",+ "base_url": "https://api.sarvam.ai",+ "credential_id": "sarvam",+ "tts": {+ "endpoint": "/text-to-speech",+ "default_model": "bulbul:v3",+ "models": ["bulbul:v3", "bulbul:v2"],+ "voices": {+ "bulbul:v2": ["anushka", "abhilash", "manisha", "vidya", "arya", "karun", "hitesh"],+ "_default": [+ "shubh", "aditya", "ritu", "priya", "neha", "rahul", "pooja", "rohan",+ "simran", "kavya", "amit", "dev", "ishita", "shreya", "ratan", "varun",+ "manan", "sumit", "roopa", "kabir", "aayan", "ashutosh", "advait", "anand",+ "tanya", "tarun", "sunny", "mani", "gokul", "vijay", "shruti", "suhani",+ "mohit", "kavitha", "rehan", "soham", "rupali"+ ]+ },+ "default_voice": "shubh",+ "languages": ["bn-IN", "en-IN", "gu-IN", "hi-IN", "kn-IN", "ml-IN", "mr-IN", "od-IN", "pa-IN", "ta-IN", "te-IN"],+ "default_language": "hi-IN",+ "output_formats": ["wav", "mp3", "linear16", "mulaw", "alaw", "opus", "flac", "aac"],+ "default_output_format": "wav",+ "max_input_chars": {+ "bulbul:v2": 1500,+ "_default": 2500+ },+ "speed_range": {+ "bulbul:v2": [0.3, 3.0],+ "_default": [0.5, 2.0]+ },+ "_sample_rate_comment": "Sarvam's own docs disagree with themselves here: the OpenAPI schema says 22050 while the field description says 24000. 24000 matches what the previous node shipped and what the description states.",+ "sample_rates": [8000, 16000, 22050, 24000, 32000, 44100, 48000],+ "default_sample_rate": 24000,+ "billed_unit": "characters"+ },+ "stt": {+ "endpoint": "/speech-to-text",+ "translate_endpoint": "/speech-to-text-translate",+ "default_model": "saaras:v3",+ "models": ["saaras:v3", "saarika:v2.5"],+ "_modes_comment": "Modes beyond transcribe/translate require saaras:v3.",+ "modes": ["transcribe", "translate", "verbatim", "translit", "codemix"],+ "_timestamps_comment": "Timestamps and diarization exist only on the batch job API, never on this synchronous endpoint. The response fields are always null here.",+ "supports_timestamps": {+ "_default": false+ },+ "supports_diarization": {+ "_default": false+ },+ "default_language": "unknown",+ "max_upload_bytes": 25165824,+ "billed_unit": "seconds"+ }+ }+ }+}diff --git a/server/core/paths.py b/server/core/paths.pyindex fe57ba57..8186effd 100644--- a/server/core/paths.py+++ b/server/core/paths.py@@ -262,8 +262,15 @@ def workspaces_dir() -> Path:return data_path(Settings().workspace_base_dir)-def workspace_dir(workflow_id: str) -> Path:- """Per-workflow workspace at ``<opencompany_root>/workspaces/<workflow_id>/``.+def workspace_dir(workflow_slug: str) -> Path:+ """Per-workflow workspace at ``<opencompany_root>/workspaces/<slug>/``.++ Keyed by ``Workflow.slug``, NOT by ``Workflow.id`` — the parameter was+ named ``workflow_id`` until it was noticed that every real caller+ passes a slug (see ``WorkflowService._get_workspace_dir``, which is+ what actually creates these directories, and the rename path in+ ``workflow_storage/handlers.py`` which moves them when the slug+ changes). Passing an id here yields a directory that does not exist.The workflow executor injects this into the execution context as``ctx.raw["workspace_dir"]`` and the cli_agent service splices it@@ -272,7 +279,7 @@ def workspace_dir(workflow_id: str) -> Path:executors) + materialise its connected skills under``<workspace_dir>/.claude/skills/``."""- return workspaces_dir() / workflow_id+ return workspaces_dir() / workflow_slugdef example_workflows_dir() -> Path:diff --git a/server/main.py b/server/main.pyindex 8fb8f4f0..d104ff4e 100644--- a/server/main.py+++ b/server/main.py@@ -59,7 +59,7 @@ _startup_log("Importing DI container + all services...")from core.container import container_startup_log("Importing routers...")-from routers import workflow, database, websocket, webhook, auth, credentials, schemas+from routers import workflow, database, websocket, webhook, auth, credentials, schemas, workspace_startup_log("All imports complete")@@ -612,6 +612,7 @@ app.include_router(database.router)app.include_router(websocket.router)app.include_router(credentials.router) # Credentials panel - lazy per-tile icon endpoint (n8n pattern)app.include_router(schemas.router) # Per-node output schema endpoint (GET /api/schemas/nodes/{type}.json)+app.include_router(workspace.router) # Per-workflow workspace file serving + uploads# Routers awaiting migration into their plugin folders. As each plugin# moves to the self-contained pattern (nodes/<plugin>/_router.py +diff --git a/server/nodes/sarvam/__init__.py b/server/nodes/sarvam/__init__.pyindex bc7c7543..005afb15 100644--- a/server/nodes/sarvam/__init__.py+++ b/server/nodes/sarvam/__init__.py@@ -1,15 +1,22 @@"""Sarvam AI service plugins.-Five stateless REST nodes over ``api.sarvam.ai`` — translate,-transliterate, language ID, speech-to-text and text-to-speech — sharing-one ``api-subscription-key`` credential. Node classes register themselves-on import via ``BaseNode.__init_subclass__``; the only explicit wiring-here is the output-schema registry, which the parameter panel reads to-render "Input Data & Variables" for downstream nodes.+Three stateless REST nodes over ``api.sarvam.ai`` — translate,+transliterate and language ID — sharing one ``api-subscription-key``+credential. Node classes register themselves on import via+``BaseNode.__init_subclass__``; the only explicit wiring here is the+output-schema registry, which the parameter panel reads to render+"Input Data & Variables" for downstream nodes.++Speech used to live here too, as ``sarvamTextToSpeech`` and+``sarvamSpeechToText``. Both were retired in favour of the+provider-abstracted ``textToSpeech`` / ``speechToText`` nodes in+``nodes/speech/``, where Sarvam is one provider among several and the+same canvas node can switch vendors without being replaced. The Sarvam+wire logic was ported verbatim into ``nodes/speech/_providers/sarvam.py``.Sarvam's OpenAI-compatible chat endpoint is a separate plugin(``nodes/model/sarvam_chat_model``) registered through the standard-LLM-provider path, and both surfaces share ``SarvamCredential``.+LLM-provider path, and every surface shares ``SarvamCredential``."""from services.node_output_schemas import register_output_schema@@ -18,14 +25,6 @@ from .sarvam_detect_language import (SarvamDetectLanguageNode,SarvamDetectLanguageOutput,)-from .sarvam_speech_to_text import (- SarvamSpeechToTextNode,- SarvamSpeechToTextOutput,-)-from .sarvam_text_to_speech import (- SarvamTextToSpeechNode,- SarvamTextToSpeechOutput,-)from .sarvam_translate import SarvamTranslateNode, SarvamTranslateOutputfrom .sarvam_transliterate import (SarvamTransliterateNode,@@ -35,13 +34,9 @@ from .sarvam_transliterate import (register_output_schema(SarvamTranslateNode.type, SarvamTranslateOutput)register_output_schema(SarvamTransliterateNode.type, SarvamTransliterateOutput)register_output_schema(SarvamDetectLanguageNode.type, SarvamDetectLanguageOutput)-register_output_schema(SarvamSpeechToTextNode.type, SarvamSpeechToTextOutput)-register_output_schema(SarvamTextToSpeechNode.type, SarvamTextToSpeechOutput)__all__ = ["SarvamDetectLanguageNode",- "SarvamSpeechToTextNode",- "SarvamTextToSpeechNode","SarvamTranslateNode","SarvamTransliterateNode",]diff --git a/server/nodes/sarvam/sarvam_speech_to_text.py b/server/nodes/sarvam/sarvam_speech_to_text.pydeleted file mode 100644index a80370d5..00000000--- a/server/nodes/sarvam/sarvam_speech_to_text.py+++ /dev/null@@ -1,201 +0,0 @@-"""Sarvam Speech to Text — POST /speech-to-text (multipart transcription)."""--from __future__ import annotations--import base64-import binascii-import mimetypes-from pathlib import Path-from typing import Any, Dict, List, Literal, Optional, Union--from pydantic import BaseModel, Field--from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue--from ._base import post_multipart, track_sarvam_usage-from ..model._credentials import SarvamCredential--# The synchronous REST endpoint is documented for clips under 30 seconds;-# anything longer needs Sarvam's Batch API, which is a different job-based-# surface and out of scope for this node.-_MAX_SECONDS = 30-# Guard against a caller pasting a multi-hundred-MB blob into the param.-_MAX_BYTES = 24 * 1024 * 1024---class SarvamSpeechToTextParams(BaseModel):- tool_name: str = Field(- default="sarvam_speech_to_text",- description="Override name shown to the LLM when used as a tool.",- )- tool_description: str = Field(- default=(- "Transcribe a short audio clip (under 30 seconds) in any of 23 Indian "- "languages. Can also translate speech directly into English."- ),- description="Override description shown to the LLM when used as a tool.",- json_schema_extra={"rows": 3},- )- # Accepts either a workspace/absolute path OR the upload envelope the- # file widget emits ({type, data, filename, mimeType}) — the frontend- # sends whichever the user chose, so the field must tolerate both.- audio_file: Union[str, Dict[str, Any]] = Field(- default="",- description="Path to an audio file, or an uploaded file.",- json_schema_extra={- "widget": "file",- "accept": "audio/*,.wav,.mp3,.m4a,.ogg,.opus,.flac,.aac,.webm,.amr",- },- )- model: Literal["saaras:v3", "saarika:v2.5"] = Field(- default="saaras:v3",- description="saaras:v3 covers 23 languages; saarika:v2.5 is the legacy 11-language model.",- )- language_code: str = Field(- default="unknown",- description="BCP-47 code of the spoken language, or 'unknown' to auto-detect.",- )- mode: Literal["transcribe", "translate", "verbatim", "translit", "codemix"] = Field(- default="transcribe",- description=(- "transcribe = native script; translate = English output; verbatim keeps "- "disfluencies; translit = Roman script; codemix preserves mixed-language input. "- "Everything except transcribe/translate requires saaras:v3."- ),- )- input_audio_codec: Optional[Literal["pcm_s16le", "pcm_l16", "pcm_raw"]] = Field(- default=None,- description="Only needed for headerless PCM; every other format is auto-detected.",- )-- model_config = {"extra": "ignore"}---class SarvamSpeechToTextOutput(BaseModel):- transcript: str = ""- language_code: Optional[str] = None- language_probability: Optional[float] = None- # Present only when Sarvam returns them; the sync endpoint omits- # diarization entirely (Batch API only).- timestamps: Optional[Dict[str, Any]] = None- diarized_transcript: Optional[Dict[str, Any]] = None- request_id: Optional[str] = None-- model_config = {"extra": "allow"}---class SarvamSpeechToTextNode(ActionNode):- type = "sarvamSpeechToText"- display_name = "Sarvam Speech to Text"- subtitle = "Transcribe Audio"- group = ("language", "tool")- description = "Transcribe or translate short audio clips across 23 Indian languages"- component_kind = "square"- tool_name = "sarvam_speech_to_text"- tool_description = (- "Transcribe a short audio clip (under 30 seconds) in any of 23 Indian languages. "- "Pass a file path in `audio_file`. Set mode='translate' to get English text from "- "non-English speech. Leave language_code='unknown' to auto-detect. Returns the "- "transcript and the detected language."- )- handles = (- {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},- {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},- )- credentials = (SarvamCredential,)- annotations = {"destructive": False, "readonly": True, "open_world": True}- task_queue = TaskQueue.REST_API- usable_as_tool = True-- Params = SarvamSpeechToTextParams- Output = SarvamSpeechToTextOutput-- @Operation("transcribe", cost={"service": "sarvam", "action": "speech_to_text", "count": 1})- async def transcribe(- self, ctx: NodeContext, params: SarvamSpeechToTextParams- ) -> SarvamSpeechToTextOutput:- if params.mode not in ("transcribe", "translate") and params.model != "saaras:v3":- raise NodeUserError(- f"mode='{params.mode}' requires model='saaras:v3'; "- f"'{params.model}' supports only transcribe and translate."- )-- filename, blob = _read_audio(params.audio_file, ctx.workspace_dir)- mime = mimetypes.guess_type(filename)[0] or "application/octet-stream"-- data = await post_multipart(- ctx,- "/speech-to-text",- files={"file": (filename, blob, mime)},- data={- "model": params.model,- "language_code": params.language_code,- "mode": params.mode,- "input_audio_codec": params.input_audio_codec,- },- )-- # Billed per second of audio; we do not decode the clip to measure- # it, so charge the documented sync-endpoint ceiling.- await track_sarvam_usage(ctx, "speech_to_text", _MAX_SECONDS)- return SarvamSpeechToTextOutput(- transcript=data.get("transcript", ""),- language_code=data.get("language_code"),- language_probability=data.get("language_probability"),- timestamps=data.get("timestamps"),- diarized_transcript=data.get("diarized_transcript"),- request_id=data.get("request_id"),- )---def _read_audio(- audio_file: Union[str, Dict[str, Any]], workspace_dir: Optional[str]-) -> tuple[str, bytes]:- """Resolve the dual-shaped ``audio_file`` param to ``(filename, bytes)``.-- The file widget emits ``{"type": "upload", "data": "<base64>", ...}``- when the user picks a local file, and a bare path string when they type- one or drag an upstream value in. Relative paths resolve against the- per-workflow workspace so they line up with ``fileDownloader`` output.- """- if isinstance(audio_file, dict):- if audio_file.get("type") != "upload" or not audio_file.get("data"):- raise NodeUserError(- "audio_file is an object but not a file upload. Provide a file "- "path or re-select the file."- )- try:- blob = base64.b64decode(audio_file["data"], validate=True)- except (binascii.Error, ValueError) as e:- raise NodeUserError(f"Uploaded audio is not valid base64: {e}") from e- name = str(audio_file.get("filename") or "audio.wav")- else:- raw = (audio_file or "").strip()- if not raw:- raise NodeUserError(- "audio_file is required — provide a path to an audio file or upload one."- )- path = Path(raw)- if not path.is_absolute() and workspace_dir:- path = Path(workspace_dir) / path- if not path.is_file():- raise NodeUserError(f"Audio file not found: {path}")- blob = path.read_bytes()- name = path.name-- if not blob:- raise NodeUserError("Audio file is empty.")- if len(blob) > _MAX_BYTES:- raise NodeUserError(- f"Audio is {len(blob) // (1024 * 1024)} MB. The synchronous endpoint takes "- f"clips under {_MAX_SECONDS} seconds — trim it, or use Sarvam's Batch API "- "for long recordings."- )- return name, blob---__all__: List[str] = [- "SarvamSpeechToTextNode",- "SarvamSpeechToTextOutput",- "SarvamSpeechToTextParams",-]diff --git a/server/nodes/sarvam/sarvam_text_to_speech.py b/server/nodes/sarvam/sarvam_text_to_speech.pydeleted file mode 100644index a26cf82d..00000000--- a/server/nodes/sarvam/sarvam_text_to_speech.py+++ /dev/null@@ -1,284 +0,0 @@-"""Sarvam Text to Speech — POST /text-to-speech (Bulbul v2/v3)."""--from __future__ import annotations--import base64-import binascii-import re-import uuid-from pathlib import Path-from typing import List, Literal, Optional--from pydantic import BaseModel, Field--from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue--from ._base import post_json, track_sarvam_usage-from ..model._credentials import SarvamCredential--_MAX_TEXT = {"bulbul:v3": 2500, "bulbul:v2": 1500}--# Sarvam returns audio as base64 inside the JSON body. A 2500-character v3-# request is roughly 12 MB of base64, and node outputs are persisted to the-# DB, broadcast over the status WebSocket, and — for a tool-exposed node —-# serialized into an LLM message. Writing to the workspace is therefore the-# default; inline base64 is opt-in and capped.-_MAX_INLINE_B64 = 1_000_000--_V3_SPEAKERS = (- "shubh", "aditya", "ritu", "priya", "neha", "rahul", "pooja", "rohan",- "simran", "kavya", "amit", "dev", "ishita", "shreya", "ratan", "varun",- "manan", "sumit", "roopa", "kabir", "aayan", "ashutosh", "advait",- "anand", "tanya", "tarun", "sunny", "mani", "gokul", "vijay", "shruti",- "suhani", "mohit", "kavitha", "rehan", "soham", "rupali",-)-_V2_SPEAKERS = (- "anushka", "manisha", "vidya", "arya", "abhilash", "karun", "hitesh",-)--_CODEC_EXT = {- "wav": "wav", "mp3": "mp3", "flac": "flac", "aac": "aac",- "opus": "opus", "linear16": "pcm", "mulaw": "ulaw", "alaw": "alaw",-}---class SarvamTextToSpeechParams(BaseModel):- tool_name: str = Field(- default="sarvam_text_to_speech",- description="Override name shown to the LLM when used as a tool.",- )- tool_description: str = Field(- default=(- "Synthesize natural speech from text in 11 Indian languages using Sarvam's "- "Bulbul voices. Writes an audio file and returns its path."- ),- description="Override description shown to the LLM when used as a tool.",- json_schema_extra={"rows": 3},- )- text: str = Field(- ...,- min_length=1,- description="Text to speak.",- json_schema_extra={"rows": 4},- )- target_language_code: Literal[- "bn-IN", "en-IN", "gu-IN", "hi-IN", "kn-IN", "ml-IN",- "mr-IN", "od-IN", "pa-IN", "ta-IN", "te-IN",- ] = Field(default="hi-IN", description="Language of the spoken output.")- model: Literal["bulbul:v3", "bulbul:v2"] = Field(- default="bulbul:v3",- description="bulbul:v3 is the current model (37 voices, 2500 chars).",- )- speaker: str = Field(- default="shubh",- description="Voice name (lowercase). v3 and v2 have separate voice sets.",- )- pace: float = Field(- default=1.0, ge=0.3, le=3.0, description="Speaking rate. v3 allows 0.5-2.0; v2 allows 0.3-3.0."- )- speech_sample_rate: Literal[8000, 16000, 22050, 24000, 32000, 44100, 48000] = Field(- default=24000, description="Output sample rate in Hz."- )- output_audio_codec: Literal[- "wav", "mp3", "linear16", "mulaw", "alaw", "opus", "flac", "aac"- ] = Field(default="wav", description="Audio container/codec.")- temperature: Optional[float] = Field(- default=None,- ge=0.01,- le=2.0,- description="Expressiveness. bulbul:v3 only.",- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v3"]}}},- )- pitch: Optional[float] = Field(- default=None,- ge=-0.75,- le=0.75,- description="Pitch shift. bulbul:v2 only.",- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v2"]}}},- )- loudness: Optional[float] = Field(- default=None,- ge=0.3,- le=3.0,- description="Output gain. bulbul:v2 only.",- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v2"]}}},- )- enable_preprocessing: Optional[bool] = Field(- default=None,- description="Normalise numbers and abbreviations. bulbul:v2 only (v3 always preprocesses).",- json_schema_extra={"displayOptions": {"show": {"model": ["bulbul:v2"]}}},- )- return_audio: Literal["file", "base64"] = Field(- default="file",- description=(- "'file' writes the audio into the workflow workspace and returns its path "- "(recommended). 'base64' returns the bytes inline — only for short clips."- ),- )-- model_config = {"extra": "ignore"}---class SarvamTextToSpeechOutput(BaseModel):- file_path: Optional[str] = None- files: List[str] = Field(default_factory=list)- chunk_count: int = 0- audio_format: str = "wav"- audio_base64: Optional[str] = None- note: Optional[str] = None- request_id: Optional[str] = None-- model_config = {"extra": "allow"}---class SarvamTextToSpeechNode(ActionNode):- type = "sarvamTextToSpeech"- display_name = "Sarvam Text to Speech"- subtitle = "Synthesize Speech"- group = ("language", "tool")- description = "Generate natural speech in 11 Indian languages with Sarvam Bulbul voices"- component_kind = "square"- tool_name = "sarvam_text_to_speech"- tool_description = (- "Synthesize natural speech from text in 11 Indian languages using Sarvam's Bulbul "- "voices. Writes an audio file into the workflow workspace and returns its path in "- "`file_path` — pass that path to other nodes rather than expecting audio bytes. "- "Keep `text` under 2500 characters."- )- handles = (- {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},- {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},- )- credentials = (SarvamCredential,)- annotations = {"destructive": False, "readonly": False, "open_world": True}- task_queue = TaskQueue.REST_API- usable_as_tool = True-- Params = SarvamTextToSpeechParams- Output = SarvamTextToSpeechOutput-- @Operation("synthesize", cost={"service": "sarvam", "action": "text_to_speech", "count": 1})- async def synthesize(- self, ctx: NodeContext, params: SarvamTextToSpeechParams- ) -> SarvamTextToSpeechOutput:- is_v3 = params.model == "bulbul:v3"- cap = _MAX_TEXT[params.model]- if len(params.text) > cap:- raise NodeUserError(- f"text is {len(params.text)} characters; {params.model} accepts at most "- f"{cap}. Split the text across multiple calls."- )-- speakers = _V3_SPEAKERS if is_v3 else _V2_SPEAKERS- speaker = params.speaker.strip().lower()- if speaker not in speakers:- raise NodeUserError(- f"'{params.speaker}' is not a {params.model} voice. Available: "- + ", ".join(speakers)- )- if is_v3 and not (0.5 <= params.pace <= 2.0):- raise NodeUserError(- f"pace {params.pace} is out of range for bulbul:v3 (0.5-2.0)."- )-- body = {- "text": params.text,- "target_language_code": params.target_language_code,- "model": params.model,- "speaker": speaker,- "pace": params.pace,- "speech_sample_rate": params.speech_sample_rate,- "output_audio_codec": params.output_audio_codec,- }- if is_v3:- body["temperature"] = params.temperature- else:- # v3 rejects these outright, so they are only sent for v2.- body["pitch"] = params.pitch- body["loudness"] = params.loudness- body["enable_preprocessing"] = params.enable_preprocessing-- data = await post_json(ctx, "/text-to-speech", body)-- chunks = [c for c in (data.get("audios") or []) if c]- if not chunks:- raise NodeUserError("Sarvam returned no audio for this request.")-- out = SarvamTextToSpeechOutput(- chunk_count=len(chunks),- audio_format=params.output_audio_codec,- request_id=data.get("request_id"),- )-- if params.return_audio == "file":- out.files = _write_chunks(- chunks,- workspace_dir=ctx.workspace_dir,- node_id=ctx.node_id,- codec=params.output_audio_codec,- text=params.text,- )- out.file_path = out.files[0]- if len(chunks) > 1:- out.note = (- f"Sarvam split this request into {len(chunks)} audio chunks. "- "file_path is the first; see `files` for all of them. Each chunk "- "is a standalone file — concatenating them byte-wise is invalid."- )- else:- first = chunks[0]- if len(first) > _MAX_INLINE_B64:- out.note = (- f"Inline audio suppressed: {len(first)} base64 characters exceeds the "- f"{_MAX_INLINE_B64} limit. Re-run with return_audio='file'."- )- else:- out.audio_base64 = first- if len(chunks) > 1:- out.note = (- f"audio_base64 is chunk 1 of {len(chunks)}. Use "- "return_audio='file' to capture every chunk."- )-- # bulbul:v2 is billed at half the v3 character rate.- await track_sarvam_usage(- ctx,- "text_to_speech" if is_v3 else "text_to_speech_v2",- len(params.text),- )- return out---def _write_chunks(- chunks: List[str],- *,- workspace_dir: Optional[str],- node_id: str,- codec: str,- text: str,-) -> List[str]:- """Decode each base64 chunk to its own file under ``<workspace>/audio``.-- One file per chunk: every chunk carries its own container header, so- concatenating them produces an unplayable file.- """- base = Path(workspace_dir) / "audio" if workspace_dir else Path("audio")- base.mkdir(parents=True, exist_ok=True)-- slug = re.sub(r"[^a-z0-9]+", "-", text[:32].lower()).strip("-") or "speech"- stem = f"{slug}-{node_id[:8]}-{uuid.uuid4().hex[:6]}"- ext = _CODEC_EXT.get(codec, "wav")-- paths: List[str] = []- for index, chunk in enumerate(chunks, start=1):- try:- blob = base64.b64decode(chunk, validate=True)- except (binascii.Error, ValueError) as e:- raise NodeUserError(- f"Sarvam returned audio chunk {index} that is not valid base64: {e}"- ) from e- suffix = "" if len(chunks) == 1 else f"-{index}"- target = base / f"{stem}{suffix}.{ext}"- target.write_bytes(blob)- paths.append(str(target))- return pathsdiff --git a/server/nodes/speech/__init__.py b/server/nodes/speech/__init__.pynew file mode 100644index 00000000..49d75353--- /dev/null+++ b/server/nodes/speech/__init__.py@@ -0,0 +1,57 @@+"""Speech plugin — provider-abstracted text-to-speech and speech-to-text.++Two nodes, one provider dropdown each. Everything a vendor needs lives in+this folder: the wire protocol, the two registries, the JSON-backed+capability lookups, dispatch, and the provider implementations. Nothing+under ``services/`` knows a speech provider's name.++What this folder deliberately does *not* own, because it is genuinely+shared:++``services/provider_registry.py``+ The registry mechanism, shared with ``services/llm``.+``services/media/``+ Audio transport and workspace containment — vendor-neutral, and usable+ by any node that produces or consumes media.+``server/config/speech_defaults.json``+ Operator-editable capability data, alongside ``llm_defaults.json`` and+ ``email_providers.json``.++Layout::++ text_to_speech.py / speech_to_text.py the two nodes+ _protocol.py requests, results, errors, protocols+ _registry.py one registry per direction+ _config.py speech_defaults.json lookups+ _unifier.py dispatch + error translation+ _providers/ one module per vendor+ _credentials.py speech-only credentials+ _base.py shared node helpers+ _option_loaders.py dropdown loaders+"""++from __future__ import annotations++from services.node_output_schemas import register_output_schema+from services.ws_handler_registry import register_option_loader++from ._option_loaders import load_speech_models, load_speech_voices+from .speech_to_text import SpeechToTextNode, SpeechToTextOutput+from .text_to_speech import TextToSpeechNode, TextToSpeechOutput++# Runtime output shapes, so the parameter panel can offer these fields as+# drag sources for downstream nodes.+register_output_schema(TextToSpeechNode.type, TextToSpeechOutput)+register_output_schema(SpeechToTextNode.type, SpeechToTextOutput)++# Dropdown loaders. The names must match the `loadOptionsMethod` strings on+# the Params fields or `test_node_spec.py` fails.+register_option_loader("speechModels", load_speech_models)+register_option_loader("speechVoices", load_speech_voices)++__all__ = [+ "SpeechToTextNode",+ "SpeechToTextOutput",+ "TextToSpeechNode",+ "TextToSpeechOutput",+]diff --git a/server/nodes/speech/_base.py b/server/nodes/speech/_base.pynew file mode 100644index 00000000..bd919800--- /dev/null+++ b/server/nodes/speech/_base.py@@ -0,0 +1,159 @@+"""Shared helpers for the two speech nodes.++Credential resolution, audio input handling, and cost attribution -- the+parts both nodes need and neither should own.+"""++from __future__ import annotations++from pathlib import Path+from typing import Any, Dict, Optional, Sequence, Tuple++from core.logging import get_logger+from services.media import coerce_file_param, read_media_bytes, resolve_media+from services.media.inspect import inspect_audio+from services.plugin import NodeContext, NodeUserError++from . import _config as speech_config++logger = get_logger(__name__)+++async def provider_api_key(ctx: NodeContext, provider: str) -> str:+ """Resolve the stored key for ``provider`` through its credential.++ Goes through ``ctx.connection(...)`` rather than reading the auth+ service directly so a missing key raises the framework's annotated+ ``PermissionError`` -- which ``BaseNode.execute`` turns into a+ credential envelope plus a CloudEvents broadcast, so the Credentials+ modal lights up the right provider.+ """+ credential_id = speech_config.credential_id(provider)+ async with ctx.connection(credential_id) as conn:+ secrets = await conn.credentials()+ api_key = str(secrets.get("api_key") or "")+ if not api_key:+ raise NodeUserError(+ f"No API key stored for '{credential_id}'. Add one in the "+ "Credentials modal."+ )+ return api_key+++def require_provider(provider: str, available: Sequence[str], direction: str) -> str:+ """Validate a provider selection against what is actually registered."""+ if provider in available:+ return provider+ raise NodeUserError(+ f"'{provider}' is not a {direction} provider. Available: "+ f"{', '.join(available)}."+ )+++def read_audio_input(+ value: Any, ctx: NodeContext, *, max_bytes: Optional[int] = None+) -> Tuple[str, bytes, Optional[Path]]:+ """Resolve an audio parameter to ``(filename, bytes, path_or_None)``.++ The path comes back when one exists because it is what makes real+ duration billing possible -- ``inspect_audio`` needs a file. A legacy+ base64 upload has no path, and rather than inventing a duration for it+ the node bills nothing. That is the fix for the old Sarvam node, which+ charged every clip as 30 seconds because it never measured.++ All three input shapes route through ``services.media``, so the+ traversal that let ``audio_file="../../credentials.db"`` read the+ credential store is closed here by construction.+ """+ kwargs: Dict[str, Any] = {"ctx": ctx}+ if max_bytes:+ kwargs["max_bytes"] = max_bytes++ path: Optional[Path] = None+ if isinstance(value, str) and value.strip():+ # A path or workspace-relative string: resolve it so we keep the+ # location for probing, then read through the contained reader.+ path = resolve_media(value, ctx=ctx)+ filename, blob = read_media_bytes(value, **kwargs)+ elif isinstance(value, dict) and value.get("kind") == "audio":+ from services.media import AudioRef++ ref = AudioRef.model_validate(value)+ path = resolve_media(ref, ctx=ctx)+ filename, blob = read_media_bytes(ref, **kwargs)+ else:+ filename, blob = coerce_file_param(value, **kwargs)++ return filename, blob, path+++def measure_seconds(path: Optional[Path], declared_format: str = "") -> Optional[float]:+ """Real duration for billing, or ``None`` when it cannot be measured.++ Never raises and never guesses: ``inspect_audio`` degrades to an empty+ probe on an unknown container, and a missing duration means the caller+ skips per-second attribution rather than inventing a figure.+ """+ if path is None:+ return None+ return inspect_audio(path, declared_format=declared_format).duration_seconds+++async def track_usage(+ ctx: NodeContext,+ *,+ provider: str,+ operation: str,+ units: Optional[float],+ unit: str,+) -> None:+ """Record an API usage metric. Never raises.++ ``service`` is the provider id rather than a generic ``"speech"`` so+ per-provider dashboards group correctly, and the unit is whatever that+ provider actually bills in -- characters, seconds or minutes. Both come+ from the provider module, because only it knows.++ A provider with no ``operation_map`` entry in ``pricing.json`` yields a+ zero cost silently, which is why every provider added here needs both+ an ``api_pricing`` and an ``operation_map`` block.+ """+ if units is None or units <= 0:+ return+ try:+ from services.plugin.deps import get_database+ from services.pricing import get_pricing_service++ cost_data = get_pricing_service().calculate_api_cost(+ provider, operation, units+ )+ await get_database().save_api_usage_metric(+ {+ "session_id": ctx.raw.get("session_id", "default"),+ "node_id": ctx.node_id,+ "workflow_id": ctx.workflow_id,+ "service": provider,+ "operation": cost_data.get("operation", operation),+ "endpoint": operation,+ "resource_count": units,+ "cost": cost_data.get("total_cost", 0.0),+ }+ )+ except Exception as exc:+ # Cost attribution must never fail a workflow that already did the+ # paid work.+ logger.warning(+ "failed to record speech usage",+ provider=provider,+ operation=operation,+ error=str(exc),+ )+++__all__ = [+ "measure_seconds",+ "provider_api_key",+ "read_audio_input",+ "require_provider",+ "track_usage",+]diff --git a/server/nodes/speech/_config.py b/server/nodes/speech/_config.pynew file mode 100644index 00000000..7d0a4c84--- /dev/null+++ b/server/nodes/speech/_config.py@@ -0,0 +1,244 @@+"""Capability lookups backed by ``server/config/speech_defaults.json``.++Same contract as :mod:`services.llm.config`: the JSON is loaded once at+import into a module-level dict, failure is soft, and every capability+question is answered from here so no shared code ever branches on a provider+name.++The one idea worth understanding is :func:`capability`. A capability value in+the JSON is either a plain scalar (applies to every model) or a mapping of+model id to value with a ``_default`` fallback. Callers never care which was+written -- they ask for ``(provider, direction, key, model)`` and get a value.+That is what lets OpenAI declare "``response_formats`` is ``json|text`` except+on whisper-1 where it is five formats" without a line of Python.++Lookup order inside a mapping is exact match, then longest prefix match, then+``_default``. Prefix matching earns its keep on dated snapshots -- a key of+``gpt-4o-mini-transcribe`` also answers for+``gpt-4o-mini-transcribe-2025-12-15`` without the JSON tracking every release.+"""++from __future__ import annotations++import json+from pathlib import Path+from typing import Any, Dict, List, Optional++from core.logging import get_logger++logger = get_logger(__name__)++TTS = "tts"+STT = "stt"++_CONFIG_PATH = Path(__file__).parent.parent.parent / "config" / "speech_defaults.json"+++def _load_speech_defaults() -> Dict[str, Any]:+ try:+ with open(_CONFIG_PATH, encoding="utf-8") as handle:+ return json.load(handle)+ except Exception as exc:+ # Soft-fail, matching services/llm/config.py: a malformed config+ # degrades capability lookups to their permissive defaults rather+ # than preventing the process from booting.+ logger.warning("Could not load speech_defaults.json", error=str(exc))+ return {"providers": {}}+++SPEECH_DEFAULTS: Dict[str, Any] = _load_speech_defaults()+++def reload_defaults() -> None:+ """Re-read the JSON from disk. Mutates in place.++ In-place rather than rebinding because callers may hold a reference to+ the dict (the unifier does), and rebinding would leave them on the stale+ object -- a trap ``services/llm/config.reload_defaults`` still has.+ """+ fresh = _load_speech_defaults()+ SPEECH_DEFAULTS.clear()+ SPEECH_DEFAULTS.update(fresh)+++# ---------------------------------------------------------------------------+# Raw block access+# ---------------------------------------------------------------------------+++def provider_block(provider: str) -> Dict[str, Any]:+ """The whole ``providers.<name>`` block, or ``{}``."""+ block = SPEECH_DEFAULTS.get("providers", {}).get(provider)+ return block if isinstance(block, dict) else {}+++def direction_block(provider: str, direction: str) -> Dict[str, Any]:+ """The ``providers.<name>.<tts|stt>`` block, or ``{}``."""+ block = provider_block(provider).get(direction)+ return block if isinstance(block, dict) else {}+++def supports_direction(provider: str, direction: str) -> bool:+ """Whether the JSON declares this provider for this direction.++ Advisory only -- registry membership is the authority (a provider that+ never calls ``register_tts_provider`` cannot be selected regardless of+ what the JSON says). Useful for config-consistency tests.+ """+ return bool(direction_block(provider, direction))+++def base_url(provider: str) -> str:+ return str(provider_block(provider).get("base_url") or "")+++def credential_id(provider: str) -> str:+ """Credential id string for this provider.++ A *string*, never a ``Credential`` class: the classes live under+ ``nodes/`` and importing one here would invert the layering and break+ ``test_plugin_self_containment.py``. Falls back to the provider name,+ which is the convention every current provider follows.+ """+ return str(provider_block(provider).get("credential_id") or provider)+++# ---------------------------------------------------------------------------+# Capability resolution+# ---------------------------------------------------------------------------+++def capability(+ provider: str,+ direction: str,+ key: str,+ *,+ model: Optional[str] = None,+ default: Any = None,+) -> Any:+ """Resolve one capability, honouring per-model overrides.++ Returns ``default`` when the key is absent entirely. A declared ``null``+ is returned as ``None`` and is meaningful -- Deepgram's+ ``max_upload_bytes`` is explicitly null because no cap is documented,+ which is different from "not configured".+ """+ block = direction_block(provider, direction)+ if key not in block:+ return default++ value = block[key]+ if not isinstance(value, dict):+ return value++ # A mapping means per-model overrides. Without a model we can still+ # answer from _default, which is what the panel does before the user+ # has picked one.+ if model:+ if model in value:+ return value[model]+ prefix_matches = [k for k in value if k != "_default" and model.startswith(k)]+ if prefix_matches:+ return value[max(prefix_matches, key=len)]+ if "_default" in value:+ return value["_default"]+ return default+++def default_model(provider: str, direction: str) -> str:+ return str(capability(provider, direction, "default_model", default="") or "")+++def models(provider: str, direction: str) -> List[str]:+ value = capability(provider, direction, "models", default=[])+ return list(value or [])+++def voices(provider: str, *, model: Optional[str] = None) -> List[str]:+ """Static voice ids from config.++ Only a fallback. Providers that expose a live voices endpoint+ (ElevenLabs) answer the dropdown from the API instead, because their+ catalogue is per-account and changes without a release.+ """+ value = capability(provider, TTS, "voices", model=model, default=[])+ return list(value or [])+++def default_voice(provider: str) -> str:+ return str(capability(provider, TTS, "default_voice", default="") or "")+++def output_formats(provider: str, direction: str = TTS) -> List[str]:+ return list(capability(provider, direction, "output_formats", default=[]) or [])+++def billed_unit(provider: str, direction: str) -> str:+ """``characters`` / ``seconds`` / ``minutes`` -- for cost attribution."""+ return str(capability(provider, direction, "billed_unit", default="") or "")+++def endpoint(provider: str, direction: str, key: str = "endpoint") -> str:+ return str(capability(provider, direction, key, default="") or "")+++def max_input_chars(provider: str, model: Optional[str] = None) -> Optional[int]:+ value = capability(provider, TTS, "max_input_chars", model=model)+ return int(value) if isinstance(value, (int, float)) else None+++def speed_range(+ provider: str, model: Optional[str] = None+) -> Optional[tuple[float, float]]:+ value = capability(provider, TTS, "speed_range", model=model)+ if isinstance(value, (list, tuple)) and len(value) == 2:+ return float(value[0]), float(value[1])+ return None+++def supports(+ provider: str, direction: str, key: str, *, model: Optional[str] = None+) -> bool:+ """Boolean capability probe.++ Defaults to ``True`` -- permissive, matching ``supports_model_listing``+ in the LLM layer. A provider opts *out* explicitly; forgetting to+ declare a flag never silently disables a working feature.+ """+ return bool(capability(provider, direction, key, model=model, default=True))+++def response_formats(provider: str, model: Optional[str] = None) -> List[str]:+ """Allowed ``response_format`` values for a transcription model.++ Model-gated on OpenAI and a 400 when violated, which is why this is+ config rather than something the node discovers at runtime.+ """+ return list(+ capability(provider, STT, "response_formats", model=model, default=[]) or []+ )+++__all__ = [+ "SPEECH_DEFAULTS",+ "STT",+ "TTS",+ "base_url",+ "billed_unit",+ "capability",+ "credential_id",+ "default_model",+ "default_voice",+ "direction_block",+ "endpoint",+ "max_input_chars",+ "models",+ "output_formats",+ "provider_block",+ "reload_defaults",+ "response_formats",+ "speed_range",+ "supports",+ "supports_direction",+ "voices",+]diff --git a/server/nodes/speech/_credentials.py b/server/nodes/speech/_credentials.pynew file mode 100644index 00000000..984b8e1f--- /dev/null+++ b/server/nodes/speech/_credentials.py@@ -0,0 +1,63 @@+"""Credentials for speech-only providers.++Only two live here. OpenAI, Groq and Sarvam already have credentials under+``nodes/model/_credentials.py`` because the same stored key authenticates+their chat surfaces, and a second class for the same ``id`` would collide in+``CREDENTIAL_REGISTRY``. The speech nodes import those rather than+redeclaring them -- the same cross-plugin import ``nodes/sarvam/`` already+does.+"""++from __future__ import annotations++from typing import Any, Dict++from services.plugin.credential import ApiKeyCredential+++class ElevenLabsCredential(ApiKeyCredential):+ """ElevenLabs API key.++ Sent as a bare ``xi-api-key`` header with no scheme keyword;+ ``Authorization: Bearer`` is rejected. The probe lists one voice, which+ is the cheapest authenticated call the API offers.+ """++ id = "elevenlabs"+ display_name = "ElevenLabs"+ category = "AI"+ key_name = "xi-api-key"+ key_location = "header"+ docs_url = "https://elevenlabs.io/app/settings/api-keys"+ probe_url = "https://api.elevenlabs.io/v2/voices"+ probe_params = {"page_size": 1}+++class DeepgramCredential(ApiKeyCredential):+ """Deepgram API key.++ Deepgram's scheme keyword is ``Token``, not ``Bearer``, which none of+ the three built-in ``key_location`` modes can express: ``header`` would+ send the raw key with no keyword and ``bearer`` would send the wrong+ one. Hence the :meth:`inject` override -- and it is load-bearing for the+ declarative probe too, which builds its request through the same method.+ """++ id = "deepgram"+ display_name = "Deepgram"+ category = "AI"+ key_name = "Authorization"+ key_location = "header"+ docs_url = "https://console.deepgram.com"+ probe_url = "https://api.deepgram.com/v1/projects"++ @classmethod+ def inject(+ cls, secrets: Dict[str, Any], request: Dict[str, Any]+ ) -> Dict[str, Any]:+ headers = dict(request.get("headers") or {})+ headers["Authorization"] = f"Token {secrets.get('api_key', '')}"+ return {**request, "headers": headers}+++__all__ = ["DeepgramCredential", "ElevenLabsCredential"]diff --git a/server/nodes/speech/_option_loaders.py b/server/nodes/speech/_option_loaders.pynew file mode 100644index 00000000..4d922024--- /dev/null+++ b/server/nodes/speech/_option_loaders.py@@ -0,0 +1,101 @@+"""Dynamic dropdown loaders for the speech nodes.++Both loaders read ``provider`` out of the parameter dict they are handed+rather than declaring ``loadOptionsDependsOn: ["provider"]``. That attribute+is lifted by the client adapter but has no usages anywhere in the codebase,+so it is untested machinery; the parameter dict, by contrast, is how every+existing loader in the repo works and it re-fires when parameters change.+"""++from __future__ import annotations++from typing import Any, Dict, List++from core.logging import get_logger++from . import _config as speech_config+from ._registry import stt_providers, tts_providers++logger = get_logger(__name__)+++def _direction_for(params: Dict[str, Any]) -> str:+ """Infer direction from which node asked.++ The two nodes have distinct model field names, which doubles as the+ signal for which direction's catalogue to serve.+ """+ if "stt_model" in params or params.get("node_type") == "speechToText":+ return speech_config.STT+ return speech_config.TTS+++def _selected_provider(params: Dict[str, Any], direction: str) -> str:+ provider = str(params.get("provider") or "")+ available = tts_providers() if direction == speech_config.TTS else stt_providers()+ if provider in available:+ return provider+ return available[0] if available else ""+++async def load_speech_models(params: Dict[str, Any]) -> List[Dict[str, Any]]:+ """Model ids for the selected provider and direction."""+ direction = _direction_for(params)+ provider = _selected_provider(params, direction)+ if not provider:+ return []++ default = speech_config.default_model(provider, direction)+ options: List[Dict[str, Any]] = []+ for model in speech_config.models(provider, direction):+ option: Dict[str, Any] = {"value": model, "label": model}+ if model == default:+ option["description"] = "Provider default"+ options.append(option)+ return options+++async def load_speech_voices(params: Dict[str, Any]) -> List[Dict[str, Any]]:+ """Voices for the selected synthesis provider.++ Providers with a live catalogue (ElevenLabs) are queried, because their+ voice list is per-account and changes without a release. Everyone else+ answers from config. A live lookup that fails degrades to the configured+ list rather than leaving the user with an empty dropdown and no way to+ proceed.+ """+ provider = _selected_provider(params, speech_config.TTS)+ if not provider:+ return []++ model = str(params.get("tts_model") or "")+ static = [+ {"value": voice, "label": voice.replace("_", " ").title()}+ for voice in speech_config.voices(provider, model=model or None)+ ]++ if not speech_config.capability(provider, speech_config.TTS, "voices_endpoint"):+ return static++ try:+ from services.plugin.deps import get_auth_service++ from . import _unifier++ api_key = await get_auth_service().get_api_key(+ speech_config.credential_id(provider)+ )+ if not api_key:+ return static+ voices = await _unifier.list_voices(provider=provider, api_key=api_key)+ return [voice.as_option() for voice in voices] or static+ except Exception as exc:+ logger.warning(+ "live voice lookup failed; falling back to the configured list",+ provider=provider,+ error=str(exc),+ )+ return static+++__all__ = ["load_speech_models", "load_speech_voices"]diff --git a/server/nodes/speech/_protocol.py b/server/nodes/speech/_protocol.pynew file mode 100644index 00000000..bd0a0aa9--- /dev/null+++ b/server/nodes/speech/_protocol.py@@ -0,0 +1,444 @@+"""Provider-neutral speech types — the contract every provider implements.++Mirrors :mod:`services.llm.protocol`: stdlib dataclasses only, no Pydantic,+no SDK objects. These values cross the Temporal boundary and land in node+results, so everything here must stay JSON-safe.++The one place this deliberately diverges from the LLM layer is the error+taxonomy. ``LLMError`` carries ``CONTEXT_LENGTH``, which is meaningless for+speech, and speech has failure modes the LLM layer has never seen+(``AUDIO_TOO_LONG``, ``UNSUPPORTED_FORMAT``). Sharing one enum would mean+half the members raising "can't happen here" questions on both sides, so+each layer owns its own. The HTTP status ladder underneath is similar by+coincidence of HTTP, not by shared domain.++Audio bytes appear in exactly two places here — :class:`AudioPayload` on the+way out of a TTS provider, and :attr:`SttRequest.audio` on the way in. Both+are consumed inside the node before it returns; what leaves the node is an+``AudioRef`` (see :mod:`services.media.refs`). Nothing in this module is+allowed into a node ``Output`` model.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from enum import Enum+from typing import Any, Dict, List, Mapping, Optional, Protocol, runtime_checkable+++# ---------------------------------------------------------------------------+# Requests+# ---------------------------------------------------------------------------+++@dataclass+class TtsRequest:+ """One synthesis call, in provider-neutral terms.++ ``provider_options`` is the escape hatch for vendor-specific keys that+ do not generalise (ElevenLabs' ``stability``, Sarvam's ``pitch``). It is+ passed through untouched, which keeps the common surface small without+ making any provider's full capability unreachable.+ """++ text: str+ model: str = ""+ voice: str = ""+ language: str = ""+ speed: Optional[float] = None+ output_format: str = ""+ sample_rate: Optional[int] = None+ provider_options: Dict[str, Any] = field(default_factory=dict)+++@dataclass+class SttRequest:+ """One transcription call.++ ``audio`` is raw bytes rather than a path or a file handle: providers+ disagree on transport (multipart vs. raw body), and a handle cannot be+ replayed by ``Connection``'s auth-retry. The caller has already applied+ containment via ``services.media.coerce_file_param``.+ """++ audio: bytes+ filename: str = "audio.wav"+ mime_type: str = "application/octet-stream"+ model: str = ""+ language: str = ""+ prompt: str = ""+ translate: bool = False+ diarize: bool = False+ timestamps: bool = False+ provider_options: Dict[str, Any] = field(default_factory=dict)+++# ---------------------------------------------------------------------------+# Results+# ---------------------------------------------------------------------------+++@dataclass+class AudioPayload:+ """One synthesized audio blob.++ Providers return a *list* of these. Most emit exactly one; Sarvam splits+ long input into several standalone clips, and those are not+ byte-concatenatable, so the plural is the honest shape rather than a+ convenience.+ """++ data: bytes+ format: str = ""+ mime_type: str = ""+ sample_rate: Optional[int] = None+ channels: Optional[int] = None+++@dataclass+class TtsResult:+ audio: List[AudioPayload] = field(default_factory=list)+ model: str = ""+ voice: str = ""+ request_id: Optional[str] = None+ # Billing units differ per provider (characters / seconds / minutes), so+ # the provider that made the call reports both the count and its unit.+ # Central code never guesses.+ billed_units: Optional[float] = None+ billed_unit: str = ""+++@dataclass+class TranscriptWord:+ word: str+ start: Optional[float] = None+ end: Optional[float] = None+ speaker: Optional[str] = None+ confidence: Optional[float] = None+++@dataclass+class SttResult:+ text: str = ""+ language: Optional[str] = None+ language_confidence: Optional[float] = None+ duration_seconds: Optional[float] = None+ words: List[TranscriptWord] = field(default_factory=list)+ segments: List[Dict[str, Any]] = field(default_factory=list)+ request_id: Optional[str] = None+ billed_units: Optional[float] = None+ billed_unit: str = ""+++@dataclass+class Voice:+ """One selectable voice, for the node's dropdown loader."""++ id: str+ name: str = ""+ description: str = ""+ language: str = ""+ preview_url: str = ""++ def as_option(self) -> Dict[str, str]:+ """Shape the ``loadOptionsMethod`` contract expects."""+ option = {"value": self.id, "label": self.name or self.id}+ if self.description:+ option["description"] = self.description+ return option+++# ---------------------------------------------------------------------------+# Errors+# ---------------------------------------------------------------------------+++class SpeechErrorCategory(str, Enum):+ """Stable categories used by retry and user-error policies."""++ AUTHENTICATION = "authentication"+ PERMISSION = "permission"+ RATE_LIMIT = "rate_limit"+ INVALID_REQUEST = "invalid_request"+ NOT_FOUND = "not_found"+ AUDIO_TOO_LONG = "audio_too_long"+ UNSUPPORTED_FORMAT = "unsupported_format"+ TIMEOUT = "timeout"+ CONNECTION = "connection"+ SERVER = "server"+ UNKNOWN = "unknown"+++_RETRYABLE = {+ SpeechErrorCategory.RATE_LIMIT,+ SpeechErrorCategory.TIMEOUT,+ SpeechErrorCategory.CONNECTION,+ SpeechErrorCategory.SERVER,+}++_PROVIDER_NAMES = {+ "openai": "OpenAI",+ "groq": "Groq",+ "elevenlabs": "ElevenLabs",+ "deepgram": "Deepgram",+ "sarvam": "Sarvam AI",+}+++@dataclass+class SpeechError(Exception):+ """Provider-independent structured speech failure."""++ message: str+ provider: str+ category: SpeechErrorCategory = SpeechErrorCategory.UNKNOWN+ retryable: bool = False+ status_code: Optional[int] = None+ provider_code: Optional[str] = None+ request_id: Optional[str] = None+ retry_after: Optional[float] = None+ retry_after_raw: Optional[str] = None++ def __post_init__(self) -> None:+ Exception.__init__(self, self.message)++ @property+ def user_message(self) -> str:+ """A category-based message that is safe for public surfaces.++ ``message`` retains the original provider text for operator+ diagnostics and exception chaining. Provider bodies routinely echo+ request fragments and internal endpoint URLs, so execution+ boundaries must expose this property instead.+ """+ known = str(self.provider or "").strip().lower() in _PROVIDER_NAMES+ provider = _PROVIDER_NAMES.get(+ str(self.provider or "").strip().lower(), "The speech provider"+ )+ provider_object = provider if known else "the speech provider"+ category = (+ self.category.value+ if isinstance(self.category, SpeechErrorCategory)+ else str(self.category or SpeechErrorCategory.UNKNOWN.value)+ )+ messages = {+ SpeechErrorCategory.AUTHENTICATION.value: (+ f"{provider} authentication failed. Check the configured API key."+ ),+ SpeechErrorCategory.PERMISSION.value: (+ f"{provider} denied this request. Check account, plan tier and "+ "model access."+ ),+ SpeechErrorCategory.RATE_LIMIT.value: (+ f"{provider} is rate-limiting requests. Retry after a short delay."+ ),+ SpeechErrorCategory.INVALID_REQUEST.value: (+ f"{provider} rejected the request configuration. Check the "+ "selected model, voice and output format."+ ),+ SpeechErrorCategory.NOT_FOUND.value: (+ f"The configured {provider_object} model, voice or endpoint was "+ "not found."+ ),+ SpeechErrorCategory.AUDIO_TOO_LONG.value: (+ f"The audio is longer or larger than {provider_object} accepts "+ "on this endpoint. Trim or split the clip."+ ),+ SpeechErrorCategory.UNSUPPORTED_FORMAT.value: (+ f"{provider} does not accept this audio format. Convert the "+ "file and try again."+ ),+ SpeechErrorCategory.TIMEOUT.value: (+ f"The request to {provider_object} timed out."+ ),+ SpeechErrorCategory.CONNECTION.value: (+ f"Could not connect to {provider_object}."+ ),+ SpeechErrorCategory.SERVER.value: (+ f"{provider} is temporarily unavailable."+ ),+ SpeechErrorCategory.UNKNOWN.value: f"{provider} request failed.",+ }+ return messages.get(category, messages[SpeechErrorCategory.UNKNOWN.value])++ @classmethod+ def from_exception(cls, provider: str, exc: BaseException) -> "SpeechError":+ """Normalize an httpx / SDK exception into a structured error.++ Duck-typed rather than isinstance-based so a provider may raise its+ own SDK error without this module importing that SDK.+ """+ response = getattr(exc, "response", None)+ status = _optional_int(+ getattr(exc, "status_code", None)+ or getattr(response, "status_code", None)+ )+ body_text = _response_text(response)+ code = _provider_code(response, body_text)+ request_id = (+ getattr(exc, "request_id", None)+ or _header(response, "x-request-id")+ or _header(response, "request-id")+ or _header(response, "dg-request-id")+ )+ retry_after_raw = _header(response, "retry-after")+ category = classify(exc, status, body_text)+ return cls(+ message=_detail(exc, body_text),+ provider=provider,+ category=category,+ retryable=category in _RETRYABLE,+ status_code=status,+ provider_code=str(code) if code is not None else None,+ request_id=str(request_id) if request_id is not None else None,+ retry_after=_optional_float(retry_after_raw),+ retry_after_raw=(+ str(retry_after_raw) if retry_after_raw is not None else None+ ),+ )+++def classify(+ exc: BaseException, status: Optional[int], body_text: str = ""+) -> SpeechErrorCategory:+ """Map an exception + HTTP status onto a category.++ Status wins where it is unambiguous; name and body substrings resolve+ the rest. 413 and 415 are the two speech-specific rungs — every provider+ in the v1 set uses them for oversize and unsupported-codec respectively.+ """+ name = type(exc).__name__.lower()+ haystack = f"{exc} {body_text}".lower()++ if status == 401 or "authentication" in name or "api key" in haystack:+ return SpeechErrorCategory.AUTHENTICATION+ if status == 403 or "permission" in name:+ return SpeechErrorCategory.PERMISSION+ if status == 429 or "ratelimit" in name or "rate limit" in haystack:+ return SpeechErrorCategory.RATE_LIMIT+ if status == 404 or "notfound" in name:+ return SpeechErrorCategory.NOT_FOUND+ if status == 413 or "too large" in haystack or "too long" in haystack:+ return SpeechErrorCategory.AUDIO_TOO_LONG+ if status == 415 or "unsupported" in haystack or "invalid audio" in haystack:+ return SpeechErrorCategory.UNSUPPORTED_FORMAT+ if status in {400, 409, 422} or "badrequest" in name:+ return SpeechErrorCategory.INVALID_REQUEST+ if status == 408 or "timeout" in name:+ return SpeechErrorCategory.TIMEOUT+ if "connect" in name:+ return SpeechErrorCategory.CONNECTION+ if status is not None and status >= 500:+ return SpeechErrorCategory.SERVER+ return SpeechErrorCategory.UNKNOWN+++# ---------------------------------------------------------------------------+# Small helpers (duck-typed; never import a provider SDK)+# ---------------------------------------------------------------------------+++def _optional_int(value: Any) -> Optional[int]:+ try:+ return int(value) if value is not None else None+ except (TypeError, ValueError, OverflowError):+ return None+++def _optional_float(value: Any) -> Optional[float]:+ try:+ return float(value) if value is not None else None+ except (TypeError, ValueError, OverflowError):+ return None+++def _header(response: Any, name: str) -> Optional[str]:+ headers = getattr(response, "headers", None)+ if headers is None:+ return None+ try:+ return headers.get(name)+ except (AttributeError, TypeError):+ return None+++def _response_text(response: Any) -> str:+ """Best-effort body text, capped. Never raises."""+ if response is None:+ return ""+ try:+ return str(getattr(response, "text", "") or "")[:600]+ except Exception:+ return ""+++def _provider_code(response: Any, body_text: str) -> Optional[str]:+ """Pull a vendor error code out of a JSON body if one is there."""+ if response is None:+ return None+ try:+ payload = response.json()+ except Exception:+ return None+ if not isinstance(payload, Mapping):+ return None+ error = payload.get("error")+ if isinstance(error, Mapping):+ return error.get("code") or error.get("type")+ return payload.get("code") or payload.get("err_code")+++def _detail(exc: BaseException, body_text: str) -> str:+ """Operator-facing message: exception text plus body when it adds signal."""+ base = str(exc)+ if body_text and body_text not in base:+ return f"{base} :: {body_text}"+ return base+++# ---------------------------------------------------------------------------+# Provider protocols (structural typing)+# ---------------------------------------------------------------------------+#+# Two protocols, not one. AssemblyAI is transcription-only and Cartesia is+# synthesis-only; a single protocol would force dead methods that raise at+# runtime *after* the provider has already appeared in a dropdown. With one+# registry per direction, capability is registry membership and there is no+# ``supports_tts`` flag to keep in sync with reality.+#+# ``aclose`` is deliberately absent from both. The unifier probes for it with+# ``getattr`` — every provider here happens to hold an httpx client and+# implement it, but keeping it out of the protocol means a stateless provider+# never has to declare an empty one.+++@runtime_checkable+class TtsProvider(Protocol):+ provider_name: str++ async def synthesize(self, req: TtsRequest) -> TtsResult: ...++ async def list_voices(self) -> List[Voice]: ...+++@runtime_checkable+class SttProvider(Protocol):+ provider_name: str++ async def transcribe(self, req: SttRequest) -> SttResult: ...+++__all__ = [+ "AudioPayload",+ "SpeechError",+ "SpeechErrorCategory",+ "SttProvider",+ "SttRequest",+ "SttResult",+ "TranscriptWord",+ "TtsProvider",+ "TtsRequest",+ "TtsResult",+ "Voice",+ "classify",+]diff --git a/server/nodes/speech/_providers/__init__.py b/server/nodes/speech/_providers/__init__.pynew file mode 100644index 00000000..429e7b18--- /dev/null+++ b/server/nodes/speech/_providers/__init__.py@@ -0,0 +1,35 @@+"""Speech provider plugins — importing this package registers them all.++Adding a provider:++1. Add a ``providers.<name>`` block to ``server/config/speech_defaults.json``+ with at least ``base_url`` and a ``tts`` and/or ``stt`` sub-block. Which+ sub-blocks exist declares which directions it serves.+2. Write ``<name>.py`` here implementing the matching protocol(s) from+ :mod:`nodes.speech._protocol`, subclassing+ :class:`nodes.speech._providers._http.HttpSpeechProvider` if it is a+ plain HTTP API.+3. Call ``register_tts_provider`` / ``register_stt_provider`` at the bottom+ of that module.+4. Add the side-effect import below.+5. Add the credential id to ``server/config/credential_providers.json`` and a+ ``Credential`` subclass under ``nodes/speech/_credentials.py``.++If the provider speaks OpenAI's ``/audio/*`` wire format, steps 2-4 collapse+into adding its name to ``_COMPAT_STT_PROVIDERS`` in ``_openai_compat.py``.++**Never import a provider SDK at module level.** ``sdk_exception_refs`` are+lazy ``"module:ClassName"`` strings precisely so registration stays free;+``tests/speech/test_lazy_sdk_imports.py`` enforces it from a clean+interpreter.+"""++from __future__ import annotations++# Side-effect imports — each module registers into one or both registries at+# the bottom of the file. Order is documentation only; Python's module cache+# makes duplicate registration impossible.+from . import _openai_compat # noqa: F401+from . import elevenlabs # noqa: F401+from . import deepgram # noqa: F401+from . import sarvam # noqa: F401diff --git a/server/nodes/speech/_providers/_http.py b/server/nodes/speech/_providers/_http.pynew file mode 100644index 00000000..3ff69322--- /dev/null+++ b/server/nodes/speech/_providers/_http.py@@ -0,0 +1,168 @@+"""Shared httpx plumbing for speech providers.++None of the v1 speech providers ship a Python SDK worth taking a dependency+on, so every one of them is a plain HTTP client. This base holds the parts+that are genuinely identical -- lazy client construction, timeouts, status+handling -- and leaves the parts that are not to subclasses.++Authentication is the interesting one. All four providers use a different+scheme, and none of them is plain Bearer across the board:++=========== ==========================================+openai/groq ``Authorization: Bearer <key>``+elevenlabs ``xi-api-key: <key>`` (no scheme)+deepgram ``Authorization: Token <key>`` (not Bearer)+sarvam ``api-subscription-key: <key>``+=========== ==========================================++So :meth:`HttpSpeechProvider.auth_headers` is the single override point, and+nothing above this layer ever has to know.++Timeouts are generous by speech standards: synthesis of a few thousand+characters routinely takes tens of seconds, and transcription of a long clip+longer still. The default here is well under the Temporal activity+``start_to_close`` so a hung provider surfaces as a provider error rather+than an activity timeout.+"""++from __future__ import annotations++from typing import Any, Dict, Optional++import httpx++from core.logging import get_logger+from .. import _config as speech_config++logger = get_logger(__name__)++DEFAULT_TIMEOUT_SECONDS = 180.0+++class HttpSpeechProvider:+ """Base for HTTP-backed speech providers.++ Subclasses set :attr:`provider_name` and implement whichever of+ ``synthesize`` / ``transcribe`` / ``list_voices`` their direction+ requires. Structural typing means there is no ABC to satisfy -- the+ Protocols in :mod:`nodes.speech._protocol` check shape, not ancestry.+ """++ provider_name: str = ""++ def __init__(+ self,+ api_key: str,+ *,+ base_url: str = "",+ provider_name: Optional[str] = None,+ timeout: float = DEFAULT_TIMEOUT_SECONDS,+ ):+ # provider_name may arrive via client_kwargs so one class can serve+ # several registry entries (openai and groq share a transcription+ # implementation and differ only in base URL and model list).+ self.provider_name = provider_name or type(self).provider_name+ self.api_key = api_key+ self._base_url = (+ base_url or speech_config.base_url(self.provider_name)+ ).rstrip("/")+ self._timeout = timeout+ self._client: Optional[httpx.AsyncClient] = None++ # ------------------------------------------------------------------+ # transport+ # ------------------------------------------------------------------++ def auth_headers(self) -> Dict[str, str]:+ """Headers that authenticate a request. Override per provider."""+ return {"Authorization": f"Bearer {self.api_key}"}++ @property+ def client(self) -> httpx.AsyncClient:+ """Lazily-built client, reused for this provider's lifetime.++ Built on first use rather than in ``__init__`` so constructing a+ provider (which the unifier does inside a lock) never touches the+ network stack.+ """+ if self._client is None:+ self._client = httpx.AsyncClient(+ base_url=self._base_url,+ timeout=self._timeout,+ headers=self.auth_headers(),+ )+ return self._client++ async def request(+ self,+ method: str,+ path: str,+ *,+ json_body: Optional[Dict[str, Any]] = None,+ params: Optional[Dict[str, Any]] = None,+ data: Optional[Dict[str, Any]] = None,+ files: Optional[Any] = None,+ content: Optional[bytes] = None,+ headers: Optional[Dict[str, str]] = None,+ ) -> httpx.Response:+ """Issue one request and raise ``httpx.HTTPStatusError`` on 4xx/5xx.++ ``params`` is not an afterthought: ElevenLabs takes ``output_format``+ and Deepgram takes its entire option set as query string. Passing+ those in a JSON body is accepted and silently ignored, which is far+ worse than an error, so the distinction is explicit at every call+ site.+ """+ response = await self.client.request(+ method,+ path,+ json=json_body,+ params=params,+ data=data,+ files=files,+ content=content,+ headers=headers,+ )+ response.raise_for_status()+ return response++ async def aclose(self) -> None:+ """Close the underlying client. Probed via ``getattr`` by the unifier."""+ if self._client is not None:+ await self._client.aclose()+ self._client = None+++def drop_none(payload: Dict[str, Any]) -> Dict[str, Any]:+ """Strip ``None`` values from a body or query dict.++ Several of these APIs reject an explicit null where they would happily+ accept an absent key -- Sarvam 422s, Deepgram treats the literal string+ "None" as an option value. Building the full dict and filtering once is+ less error-prone than conditional insertion at a dozen call sites.+ """+ return {k: v for k, v in payload.items() if v is not None}+++def query_flags(payload: Dict[str, Any]) -> Dict[str, Any]:+ """Render a query dict for APIs that want lowercase boolean literals.++ httpx encodes ``True`` as ``"true"`` already, but only for actual bools;+ this makes the conversion explicit and survives values arriving as+ strings from JSON parameters.+ """+ rendered: Dict[str, Any] = {}+ for key, value in drop_none(payload).items():+ if isinstance(value, bool):+ rendered[key] = "true" if value else "false"+ else:+ rendered[key] = value+ return rendered+++__all__ = [+ "DEFAULT_TIMEOUT_SECONDS",+ "HttpSpeechProvider",+ "drop_none",+ "query_flags",+]diff --git a/server/nodes/speech/_providers/_openai_compat.py b/server/nodes/speech/_providers/_openai_compat.pynew file mode 100644index 00000000..48122910--- /dev/null+++ b/server/nodes/speech/_providers/_openai_compat.py@@ -0,0 +1,313 @@+"""OpenAI's speech wire format, and everyone who speaks it.++One class serves OpenAI (both directions) and Groq (transcription only).+Groq hosts the identical route at ``/openai/v1/audio/transcriptions``, so the+difference between them is a base URL and a model list -- both of which live+in ``speech_defaults.json``. Adding another OpenAI-compatible transcription+host is a JSON block plus one tuple entry at the bottom of this file, with no+new Python.++That Groq's instance also carries a ``synthesize`` method it cannot serve is+harmless and is the point of the two-registry design: Groq is never+registered for synthesis, so nothing can select it for synthesis, and no+``supports_tts`` flag has to be kept honest.+"""++from __future__ import annotations++from typing import Any, Dict, List, Tuple++from core.logging import get_logger+from .. import _config as speech_config+from .._protocol import (+ AudioPayload,+ SttRequest,+ SttResult,+ TranscriptWord,+ TtsRequest,+ TtsResult,+ Voice,+)+from ._http import HttpSpeechProvider, drop_none+from .._registry import (+ SttProviderSpec,+ TtsProviderSpec,+ register_stt_provider,+ register_tts_provider,+)++logger = get_logger(__name__)++# Container -> mime. ``pcm`` is deliberately headerless: OpenAI documents it+# as raw 24 kHz 16-bit mono, so it gets the sample rate attached on the+# payload instead of a container mime type that would imply a header.+_MIME_BY_FORMAT = {+ "mp3": "audio/mpeg",+ "opus": "audio/opus",+ "aac": "audio/aac",+ "flac": "audio/flac",+ "wav": "audio/wav",+ "pcm": "audio/L16",+}++_PCM_SAMPLE_RATE = 24000+++class OpenAISpeechProvider(HttpSpeechProvider):+ """Speech over OpenAI's ``/audio/*`` routes."""++ provider_name = "openai"++ # ------------------------------------------------------------------+ # synthesis+ # ------------------------------------------------------------------++ async def synthesize(self, req: TtsRequest) -> TtsResult:+ model = req.model or speech_config.default_model(self.provider_name, "tts")+ fmt = req.output_format or speech_config.capability(+ self.provider_name, "tts", "default_output_format", default="mp3"+ )++ body: Dict[str, Any] = {+ "input": req.text,+ "model": model,+ "voice": req.voice+ or speech_config.default_voice(self.provider_name)+ or "alloy",+ "response_format": fmt,+ "speed": req.speed,+ }+ # `instructions` is silently ignored on tts-1 / tts-1-hd rather than+ # rejected, so it is filtered by capability instead of being sent+ # hopefully.+ instructions = req.provider_options.get("instructions")+ if instructions and speech_config.supports(+ self.provider_name, "tts", "supports_instructions", model=model+ ):+ body["instructions"] = instructions++ body.update(+ {+ k: v+ for k, v in req.provider_options.items()+ if k not in {"instructions"}+ }+ )++ endpoint = speech_config.endpoint(self.provider_name, "tts") or "/audio/speech"+ response = await self.request("POST", endpoint, json_body=drop_none(body))++ return TtsResult(+ audio=[+ AudioPayload(+ data=response.content,+ format=fmt,+ mime_type=_MIME_BY_FORMAT.get(fmt, "application/octet-stream"),+ sample_rate=_PCM_SAMPLE_RATE if fmt == "pcm" else None,+ channels=1 if fmt == "pcm" else None,+ )+ ],+ model=model,+ voice=str(body.get("voice") or ""),+ request_id=response.headers.get("x-request-id"),+ billed_units=float(len(req.text)),+ billed_unit="characters",+ )++ async def list_voices(self) -> List[Voice]:+ """Static catalogue -- OpenAI publishes no voices endpoint.++ Model-scoped because ``tts-1`` supports only nine of the thirteen+ voices; passing the model in via ``provider_options`` is not possible+ here, so the widest set is returned and the node validates.+ """+ return [+ Voice(id=voice, name=voice.title())+ for voice in speech_config.voices(self.provider_name)+ ]++ # ------------------------------------------------------------------+ # transcription+ # ------------------------------------------------------------------++ async def transcribe(self, req: SttRequest) -> SttResult:+ model = req.model or speech_config.default_model(self.provider_name, "stt")+ response_format = self._response_format(req, model)++ translate = req.translate and speech_config.supports(+ self.provider_name, "stt", "supports_translate", model=model+ )+ if req.translate and not translate:+ logger.warning(+ "model cannot translate; transcribing in-language instead",+ provider=self.provider_name,+ model=model,+ )++ endpoint_key = "translate_endpoint" if translate else "endpoint"+ endpoint = (+ speech_config.endpoint(self.provider_name, "stt", endpoint_key)+ or "/audio/transcriptions"+ )++ form: Dict[str, Any] = {+ "model": model,+ "response_format": response_format,+ }+ # The translation route always outputs English, so a language hint+ # is meaningless there and rejected by some hosts.+ if not translate and req.language and req.language != "unknown":+ form["language"] = req.language+ if req.prompt and speech_config.supports(+ self.provider_name, "stt", "supports_prompt", model=model+ ):+ form["prompt"] = req.prompt+ form.update(req.provider_options)++ files = {"file": (req.filename, req.audio, req.mime_type)}+ if req.timestamps and response_format == "verbose_json":+ # Form-encoded arrays need the bracket suffix; httpx expands a+ # list value into repeated fields.+ form["timestamp_granularities[]"] = ["word", "segment"]++ response = await self.request(+ "POST", endpoint, data=drop_none(form), files=files+ )++ if response_format == "text":+ # Plain-text mode carries no metadata at all -- no duration, so+ # no billable unit either. The node falls back to probing the+ # audio it already has on disk.+ return SttResult(text=response.text.strip(), billed_unit="seconds")+ return self._parse_json_transcript(response.json())++ # ------------------------------------------------------------------+ # helpers+ # ------------------------------------------------------------------++ def _response_format(self, req: SttRequest, model: str) -> str:+ """Pick a response format this model actually accepts.++ OpenAI gates the allowed set per model and returns 400 on a+ violation, so an unsupported request is downgraded here rather than+ failing the workflow. Word timestamps need ``verbose_json``, which+ in practice means whisper-1.+ """+ allowed = speech_config.response_formats(self.provider_name, model)+ requested = req.provider_options.get("response_format") or ""++ if req.timestamps and not requested:+ requested = "verbose_json"+ if not requested:+ requested = speech_config.capability(+ self.provider_name,+ "stt",+ "default_response_format",+ model=model,+ default="json",+ )++ if allowed and requested not in allowed:+ fallback = "json" if "json" in allowed else allowed[0]+ logger.warning(+ "response_format not supported by this model; downgrading",+ provider=self.provider_name,+ model=model,+ requested=requested,+ using=fallback,+ )+ return fallback+ return requested++ def _parse_json_transcript(self, payload: Dict[str, Any]) -> SttResult:+ words = [+ TranscriptWord(+ word=str(w.get("word") or ""),+ start=_as_float(w.get("start")),+ end=_as_float(w.get("end")),+ speaker=_as_str(w.get("speaker")),+ )+ for w in payload.get("words") or []+ if isinstance(w, dict)+ ]+ duration = _as_float(payload.get("duration"))+ return SttResult(+ text=str(payload.get("text") or "").strip(),+ language=_as_str(payload.get("language")),+ duration_seconds=duration,+ words=words,+ segments=[s for s in payload.get("segments") or [] if isinstance(s, dict)],+ request_id=_as_str((payload.get("x_groq") or {}).get("id"))+ if isinstance(payload.get("x_groq"), dict)+ else None,+ billed_units=self._billed_seconds(duration),+ billed_unit="seconds",+ )++ def _billed_seconds(self, duration: float | None) -> float | None:+ """Apply a provider's minimum billed duration, if it declares one."""+ if duration is None:+ return None+ floor = speech_config.capability(+ self.provider_name, "stt", "min_billed_seconds", default=None+ )+ if isinstance(floor, (int, float)):+ return max(float(duration), float(floor))+ return float(duration)+++def _as_float(value: Any) -> float | None:+ try:+ return float(value) if value is not None else None+ except (TypeError, ValueError):+ return None+++def _as_str(value: Any) -> str | None:+ return str(value) if value not in (None, "") else None+++# ---------------------------------------------------------------------------+# Plugin self-registration+# ---------------------------------------------------------------------------+#+# ``httpx:HTTPError`` is the common ancestor of connect, timeout and+# status errors, so one lazy ref covers every transport failure. It stays a+# ``"module:Class"`` string for the same reason the LLM specs do -- see+# services/provider_registry.py.++_COMPAT_STT_PROVIDERS: Tuple[str, ...] = ("openai", "groq")+++def _register() -> None:+ register_tts_provider(+ TtsProviderSpec(+ name="openai",+ factory=OpenAISpeechProvider,+ sdk_exception_refs=("httpx:HTTPError",),+ client_kwargs={+ "provider_name": "openai",+ "base_url": speech_config.base_url("openai"),+ },+ )+ )++ for name in _COMPAT_STT_PROVIDERS:+ base = speech_config.base_url(name)+ if not base:+ logger.warning(+ "skipping compat speech provider -- no base_url in speech_defaults.json",+ provider=name,+ )+ continue+ register_stt_provider(+ SttProviderSpec(+ name=name,+ factory=OpenAISpeechProvider,+ sdk_exception_refs=("httpx:HTTPError",),+ client_kwargs={"provider_name": name, "base_url": base},+ )+ )+++_register()diff --git a/server/nodes/speech/_providers/deepgram.py b/server/nodes/speech/_providers/deepgram.pynew file mode 100644index 00000000..06cbd768--- /dev/null+++ b/server/nodes/speech/_providers/deepgram.py@@ -0,0 +1,195 @@+"""Deepgram speech-to-text.++Diverges from every other provider here on all three axes at once, which is+why each is called out explicitly rather than left to the reader:++* **Auth** is ``Authorization: Token <key>``. Not ``Bearer`` -- Deepgram+ documents no Bearer alternative.+* **Options are query parameters.** There is no JSON options body. Putting+ ``model`` or ``diarize`` in a body is ignored silently.+* **Audio is the raw request body** with an ``audio/*`` content type. Not+ multipart, not base64. (A remote URL is the one case that does use a JSON+ body, and that path is deliberately not wired: the node has already read+ the bytes under workspace containment, and letting a workflow hand+ Deepgram an arbitrary URL would be a server-side request forgery vector.)++Multi-value options are repeated keys (``keyterm=a&keyterm=b``), never comma+lists, which httpx produces for list values.+"""++from __future__ import annotations++from typing import Any, Dict, List, Optional++from core.logging import get_logger+from .. import _config as speech_config+from .._protocol import (+ SttRequest,+ SttResult,+ TranscriptWord,+)+from ._http import (+ HttpSpeechProvider,+ drop_none,+ query_flags,+)+from .._registry import SttProviderSpec, register_stt_provider++logger = get_logger(__name__)++# Options Deepgram accepts that the neutral request already covers. Anything+# else a caller wants goes through provider_options untouched.+_PASSTHROUGH_OPTIONS = (+ "smart_format",+ "punctuate",+ "paragraphs",+ "utterances",+ "filler_words",+ "numerals",+ "profanity_filter",+ "redact",+ "search",+ "summarize",+ "topics",+ "intents",+ "sentiment",+ "detect_entities",+ "dictation",+ "measurements",+ "multichannel",+ "utt_split",+ "encoding",+ "sample_rate",+ "channels",+ "keywords",+ "keyterm",+ "version",+ "tag",+)+++class DeepgramProvider(HttpSpeechProvider):+ provider_name = "deepgram"++ def auth_headers(self) -> Dict[str, str]:+ # "Token", not "Bearer". A Bearer prefix authenticates as nobody and+ # returns 401 with an unhelpful body.+ return {"Authorization": f"Token {self.api_key}"}++ async def transcribe(self, req: SttRequest) -> SttResult:+ model = req.model or speech_config.default_model(self.provider_name, "stt")++ params: Dict[str, Any] = {+ "model": model,+ "smart_format": speech_config.capability(+ self.provider_name, "stt", "default_smart_format", default=True+ ),+ }+ if req.language and req.language != "unknown":+ params["language"] = req.language+ else:+ # Deepgram auto-detects only when explicitly asked to.+ params["detect_language"] = True+ if req.diarize:+ params["diarize"] = True+ if req.translate:+ logger.warning(+ "Deepgram does not translate; transcribing in-language instead",+ provider=self.provider_name,+ )++ params.update(+ {k: v for k, v in req.provider_options.items() if k in _PASSTHROUGH_OPTIONS}+ )++ endpoint = speech_config.endpoint(self.provider_name, "stt") or "/listen"+ response = await self.request(+ "POST",+ endpoint,+ params=query_flags(params),+ content=req.audio,+ headers={"Content-Type": req.mime_type or "application/octet-stream"},+ )+ return self._parse(response.json(), req)++ # ------------------------------------------------------------------+ # response parsing+ # ------------------------------------------------------------------++ def _parse(self, payload: Dict[str, Any], req: SttRequest) -> SttResult:+ results = payload.get("results") or {}+ metadata = payload.get("metadata") or {}+ channel = _first_dict(results.get("channels"))+ alternative = _first_dict(channel.get("alternatives")) if channel else {}++ duration = _as_float(metadata.get("duration"))+ words = [+ TranscriptWord(+ word=str(w.get("punctuated_word") or w.get("word") or ""),+ start=_as_float(w.get("start")),+ end=_as_float(w.get("end")),+ speaker=_as_str(w.get("speaker")),+ confidence=_as_float(w.get("confidence")),+ )+ for w in (alternative.get("words") or [])+ if isinstance(w, dict)+ ]++ # `paragraphs.transcript` carries speaker labels and line breaks that+ # the flat `transcript` string loses, so it wins when present.+ text = str(alternative.get("transcript") or "").strip()+ paragraphs = alternative.get("paragraphs")+ if isinstance(paragraphs, dict):+ formatted = str(paragraphs.get("transcript") or "").strip()+ if formatted:+ text = formatted++ utterances = [u for u in (results.get("utterances") or []) if isinstance(u, dict)]++ return SttResult(+ text=text,+ language=_as_str(channel.get("detected_language")) or (req.language or None),+ language_confidence=_as_float(channel.get("language_confidence")),+ duration_seconds=duration,+ words=words,+ segments=utterances,+ request_id=_as_str(metadata.get("request_id")),+ # Deepgram prices per minute, so the unit is converted here+ # rather than leaving central code to work out what "duration"+ # meant for this vendor.+ billed_units=(duration / 60.0) if duration is not None else None,+ billed_unit="minutes",+ )+++def _first_dict(value: Any) -> Dict[str, Any]:+ if isinstance(value, list) and value and isinstance(value[0], dict):+ return value[0]+ return {}+++def _as_float(value: Any) -> Optional[float]:+ try:+ return float(value) if value is not None else None+ except (TypeError, ValueError):+ return None+++def _as_str(value: Any) -> Optional[str]:+ return str(value) if value not in (None, "") else None+++register_stt_provider(+ SttProviderSpec(+ name="deepgram",+ factory=DeepgramProvider,+ sdk_exception_refs=("httpx:HTTPError",),+ client_kwargs={+ "provider_name": "deepgram",+ "base_url": speech_config.base_url("deepgram"),+ },+ )+)+++__all__: List[str] = ["DeepgramProvider"]diff --git a/server/nodes/speech/_providers/elevenlabs.py b/server/nodes/speech/_providers/elevenlabs.pynew file mode 100644index 00000000..466695bb--- /dev/null+++ b/server/nodes/speech/_providers/elevenlabs.py@@ -0,0 +1,234 @@+"""ElevenLabs text-to-speech.++The most structurally divergent provider in the v1 set, which makes it the+useful shape test for the abstraction. Three things differ from everyone+else:++* the voice id is a **path segment**, not a body field;+* ``output_format`` is a **query parameter** -- put it in the JSON body and+ the API returns the default format with no error at all;+* auth is a bare ``xi-api-key`` header with no scheme keyword.++``output_format`` also encodes container, sample rate and (for mp3) bitrate+in one underscore-delimited string, so the container and sample rate that+end up on the ``AudioRef`` are parsed back out of it rather than guessed.+"""++from __future__ import annotations++from typing import Any, Dict, List, Optional++from core.logging import get_logger+from .. import _config as speech_config+from .._protocol import (+ AudioPayload,+ TtsRequest,+ TtsResult,+ Voice,+)+from ._http import HttpSpeechProvider, drop_none+from .._registry import TtsProviderSpec, register_tts_provider++logger = get_logger(__name__)++_MIME_BY_CONTAINER = {+ "mp3": "audio/mpeg",+ "wav": "audio/wav",+ "pcm": "audio/L16",+ "opus": "audio/opus",+ "ulaw": "audio/basic",+ "alaw": "audio/basic",+}++# Fields inside `voice_settings`. Sent only when the caller supplies them --+# ElevenLabs' own defaults are better than anything guessed here, and+# whether these are honoured on eleven_v3 is undocumented, so nothing is+# sent speculatively.+_VOICE_SETTING_KEYS = (+ "stability",+ "similarity_boost",+ "style",+ "use_speaker_boost",+ "speed",+)++# Everything ElevenLabs takes as query string rather than body.+_QUERY_KEYS = ("enable_logging", "optimize_streaming_latency")++_VOICES_PAGE_SIZE = 100+++class ElevenLabsProvider(HttpSpeechProvider):+ provider_name = "elevenlabs"++ def auth_headers(self) -> Dict[str, str]:+ # Bare key, no scheme keyword. `Authorization: Bearer <key>` is+ # rejected.+ return {"xi-api-key": self.api_key}++ async def synthesize(self, req: TtsRequest) -> TtsResult:+ voice_id = req.voice or speech_config.default_voice(self.provider_name)+ if not voice_id:+ from services.plugin import NodeUserError++ raise NodeUserError(+ "ElevenLabs requires a voice. Pick one from the Voice dropdown "+ "-- there is no account-wide default."+ )++ model = req.model or speech_config.default_model(self.provider_name, "tts")+ output_format = req.output_format or speech_config.capability(+ self.provider_name, "tts", "default_output_format", default="mp3_44100_128"+ )++ options = dict(req.provider_options)+ voice_settings = self._voice_settings(req, options)++ body: Dict[str, Any] = {+ "text": req.text,+ "model_id": model,+ "language_code": req.language or None,+ "voice_settings": voice_settings or None,+ }+ # Whatever is left in options after the query keys and voice+ # settings have been claimed is a body field (seed, previous_text,+ # apply_text_normalization, ...).+ params = {k: options.pop(k) for k in _QUERY_KEYS if k in options}+ body.update(options)++ params["output_format"] = output_format++ path = f"/v1/text-to-speech/{voice_id}"+ response = await self.request(+ "POST", path, json_body=drop_none(body), params=drop_none(params)+ )++ container, sample_rate = _parse_output_format(output_format)+ return TtsResult(+ audio=[+ AudioPayload(+ data=response.content,+ format=container,+ mime_type=_MIME_BY_CONTAINER.get(+ container, "application/octet-stream"+ ),+ sample_rate=sample_rate,+ channels=1,+ )+ ],+ model=model,+ voice=voice_id,+ request_id=response.headers.get("request-id"),+ billed_units=float(len(req.text)),+ billed_unit="characters",+ )++ async def list_voices(self) -> List[Voice]:+ """Live catalogue via the v2 endpoint.++ Cursor-paginated; the docs are explicit that ``has_more`` plus+ ``next_page_token`` is the supported way to walk it and that+ ``total_count`` should not be relied on. Pages are capped so a large+ enterprise account cannot stall a dropdown load indefinitely.+ """+ endpoint = (+ speech_config.capability(+ self.provider_name, "tts", "voices_endpoint", default="/v2/voices"+ )+ or "/v2/voices"+ )++ voices: List[Voice] = []+ page_token: Optional[str] = None+ for _ in range(10):+ params = drop_none(+ {"page_size": _VOICES_PAGE_SIZE, "next_page_token": page_token}+ )+ payload = (await self.request("GET", endpoint, params=params)).json()+ for entry in payload.get("voices") or []:+ if not isinstance(entry, dict):+ continue+ voice_id = str(entry.get("voice_id") or "")+ if not voice_id:+ continue+ voices.append(+ Voice(+ id=voice_id,+ name=str(entry.get("name") or voice_id),+ description=str(entry.get("description") or "")[:160],+ preview_url=str(entry.get("preview_url") or ""),+ )+ )+ if not payload.get("has_more"):+ break+ page_token = payload.get("next_page_token")+ if not page_token:+ break+ else:+ logger.warning(+ "stopped paginating ElevenLabs voices at the page cap",+ provider=self.provider_name,+ collected=len(voices),+ )+ return voices++ def _voice_settings(+ self, req: TtsRequest, options: Dict[str, Any]+ ) -> Dict[str, Any]:+ """Collect voice_settings, clamping speed to the documented API range.++ ElevenLabs publishes two different speed ranges -- the OpenAPI schema+ allows 0.5-2.0 while the best-practices guide recommends 0.7-1.2.+ Neither page acknowledges the other. The schema bound is enforced+ because that is what the API actually accepts; the narrower range is+ left as guidance for the node's help text.+ """+ settings = {+ key: options.pop(key) for key in _VOICE_SETTING_KEYS if key in options+ }+ if req.speed is not None:+ settings["speed"] = req.speed++ speed = settings.get("speed")+ if speed is not None:+ bounds = speech_config.speed_range(self.provider_name) or (0.5, 2.0)+ clamped = max(bounds[0], min(float(speed), bounds[1]))+ if clamped != float(speed):+ logger.warning(+ "clamped ElevenLabs speed to the supported range",+ requested=speed,+ using=clamped,+ )+ settings["speed"] = clamped+ return drop_none(settings)+++def _parse_output_format(output_format: str) -> tuple[str, Optional[int]]:+ """Split ``mp3_44100_128`` into ``("mp3", 44100)``.++ The container drives the file extension and mime type on the resulting+ ``AudioRef``; the sample rate matters because ``pcm_*`` is headerless and+ nothing downstream could otherwise work out its rate.+ """+ parts = str(output_format or "").split("_")+ container = parts[0] if parts else ""+ sample_rate: Optional[int] = None+ if len(parts) > 1:+ try:+ sample_rate = int(parts[1])+ except ValueError:+ sample_rate = None+ return container, sample_rate+++register_tts_provider(+ TtsProviderSpec(+ name="elevenlabs",+ factory=ElevenLabsProvider,+ sdk_exception_refs=("httpx:HTTPError",),+ client_kwargs={+ "provider_name": "elevenlabs",+ "base_url": speech_config.base_url("elevenlabs"),+ },+ )+)diff --git a/server/nodes/speech/_providers/sarvam.py b/server/nodes/speech/_providers/sarvam.pynew file mode 100644index 00000000..751befba--- /dev/null+++ b/server/nodes/speech/_providers/sarvam.py@@ -0,0 +1,271 @@+"""Sarvam AI speech, in both directions. Indic-first.++Ported from the two vendor-locked nodes this layer replaces+(``sarvamTextToSpeech`` / ``sarvamSpeechToText``), preserving their wire+behaviour exactly. Two properties of this API shape the code:++* **Synthesis returns base64 inside JSON**, not raw bytes, and the ``audios``+ key is an **array**. Long input comes back as several standalone clips.+ They are not byte-concatenatable -- each carries its own container header+ -- so the array is surfaced as several payloads and the node writes one+ ``AudioRef`` per clip. Flattening them into one file would produce audio+ that plays only its first chunk.+* **Explicit nulls are rejected.** Sending ``"pitch": null`` is a 422 where+ omitting the key is fine, so every body goes through ``drop_none``.++Parameter support also splits hard by model: ``bulbul:v3`` takes+``temperature`` and rejects ``pitch`` / ``loudness`` / ``enable_preprocessing``+outright, while ``bulbul:v2`` is the mirror image. That gating lives here+because it is wire-format knowledge; the user-facing validation (character+caps, speaker membership) lives in the node where ``NodeUserError`` reads+naturally.+"""++from __future__ import annotations++import base64+import binascii+from typing import Any, Dict, List, Optional++from core.logging import get_logger+from .. import _config as speech_config+from .._protocol import (+ AudioPayload,+ SpeechError,+ SpeechErrorCategory,+ SttRequest,+ SttResult,+ TtsRequest,+ TtsResult,+ Voice,+)+from ._http import HttpSpeechProvider, drop_none+from .._registry import (+ SttProviderSpec,+ TtsProviderSpec,+ register_stt_provider,+ register_tts_provider,+)++logger = get_logger(__name__)++# Codec name -> file extension. Three of these differ from the codec string,+# which matters because the extension ends up in the workspace filename.+_CODEC_EXT = {"linear16": "pcm", "mulaw": "ulaw", "alaw": "alaw"}++_MIME_BY_CODEC = {+ "wav": "audio/wav",+ "mp3": "audio/mpeg",+ "flac": "audio/flac",+ "aac": "audio/aac",+ "opus": "audio/opus",+ "linear16": "audio/L16",+ "mulaw": "audio/basic",+ "alaw": "audio/basic",+}++# Body fields only bulbul:v2 accepts. v3 rejects them rather than ignoring.+_V2_ONLY = ("pitch", "loudness", "enable_preprocessing")+# Body fields only bulbul:v3 accepts.+_V3_ONLY = ("temperature",)+++class SarvamSpeechProvider(HttpSpeechProvider):+ provider_name = "sarvam"++ def auth_headers(self) -> Dict[str, str]:+ # Sarvam's speech routes accept only the native header. (Bearer works+ # on their OpenAI-compatible chat route, which is a different+ # surface and not this one.)+ return {"api-subscription-key": self.api_key}++ # ------------------------------------------------------------------+ # synthesis+ # ------------------------------------------------------------------++ async def synthesize(self, req: TtsRequest) -> TtsResult:+ model = req.model or speech_config.default_model(self.provider_name, "tts")+ is_v3 = model.endswith("v3")+ codec = req.output_format or speech_config.capability(+ self.provider_name, "tts", "default_output_format", default="wav"+ )++ body: Dict[str, Any] = {+ "text": req.text,+ "target_language_code": req.language+ or speech_config.capability(+ self.provider_name, "tts", "default_language", default="hi-IN"+ ),+ "model": model,+ "speaker": req.voice or speech_config.default_voice(self.provider_name),+ "pace": req.speed,+ "speech_sample_rate": req.sample_rate+ or speech_config.capability(+ self.provider_name, "tts", "default_sample_rate", default=24000+ ),+ "output_audio_codec": codec,+ }++ # Whatever this model rejects is dropped rather than forwarded; the+ # opposite model's exclusive fields would 400 the whole request.+ # Fields it accepts but the caller omitted simply stay absent, and+ # Sarvam's own defaults apply.+ rejected = _V2_ONLY if is_v3 else _V3_ONLY+ for key, value in req.provider_options.items():+ if key in rejected:+ logger.debug(+ "dropping parameter the selected Sarvam model rejects",+ model=model,+ parameter=key,+ )+ continue+ body[key] = value++ endpoint = (+ speech_config.endpoint(self.provider_name, "tts") or "/text-to-speech"+ )+ payload = (+ await self.request("POST", endpoint, json_body=drop_none(body))+ ).json()++ chunks = payload.get("audios") or []+ if not chunks:+ raise SpeechError(+ message="Sarvam returned no audio for this request.",+ provider=self.provider_name,+ category=SpeechErrorCategory.SERVER,+ )++ ext = _CODEC_EXT.get(codec, codec)+ mime = _MIME_BY_CODEC.get(codec, "application/octet-stream")+ sample_rate = body.get("speech_sample_rate")++ audio = [+ AudioPayload(+ data=_decode(chunk, index),+ format=ext,+ mime_type=mime,+ sample_rate=int(sample_rate) if sample_rate else None,+ channels=1,+ )+ for index, chunk in enumerate(chunks)+ ]++ return TtsResult(+ audio=audio,+ model=model,+ voice=str(body.get("speaker") or ""),+ request_id=payload.get("request_id"),+ billed_units=float(len(req.text)),+ billed_unit="characters",+ )++ async def list_voices(self) -> List[Voice]:+ """Static per-model catalogue -- Sarvam publishes no voices endpoint.++ The v2 and v3 speaker sets are disjoint, so this returns the default+ (v3) set and the node re-reads config for the selected model. Sending+ a v2 speaker to v3 is a 400.+ """+ return [+ Voice(id=speaker, name=speaker.title())+ for speaker in speech_config.voices(self.provider_name)+ ]++ # ------------------------------------------------------------------+ # transcription+ # ------------------------------------------------------------------++ async def transcribe(self, req: SttRequest) -> SttResult:+ model = req.model or speech_config.default_model(self.provider_name, "stt")+ mode = req.provider_options.get("mode") or (+ "translate" if req.translate else "transcribe"+ )++ endpoint_key = (+ "translate_endpoint"+ if req.translate and mode == "translate"+ else "endpoint"+ )+ endpoint = (+ speech_config.endpoint(self.provider_name, "stt", endpoint_key)+ or "/speech-to-text"+ )++ form = drop_none(+ {+ "model": model,+ "language_code": req.language or "unknown",+ "mode": mode,+ "input_audio_codec": req.provider_options.get("input_audio_codec"),+ }+ )+ # Bytes rather than a handle: Connection-style auth retry replays the+ # same kwargs, and a consumed file object would replay as empty.+ files = {"file": (req.filename, req.audio, req.mime_type)}++ payload = (+ await self.request(+ "POST",+ endpoint,+ data={k: str(v) for k, v in form.items()},+ files=files,+ )+ ).json()++ return SttResult(+ text=str(payload.get("transcript") or "").strip(),+ language=payload.get("language_code"),+ language_confidence=_as_float(payload.get("language_probability")),+ # The synchronous endpoint never returns timestamps or+ # diarization -- those exist only on the batch job API -- so no+ # duration comes back either. The node fills billed_units from+ # the audio it already probed, which is what replaced the flat+ # 30-second charge the old node applied to every clip.+ duration_seconds=None,+ request_id=payload.get("request_id"),+ billed_units=None,+ billed_unit="seconds",+ )+++def _decode(chunk: Any, index: int) -> bytes:+ try:+ return base64.b64decode(str(chunk), validate=True)+ except (binascii.Error, ValueError) as exc:+ raise SpeechError(+ message=f"Sarvam audio chunk {index + 1} was not valid base64: {exc}",+ provider="sarvam",+ category=SpeechErrorCategory.SERVER,+ ) from exc+++def _as_float(value: Any) -> Optional[float]:+ try:+ return float(value) if value is not None else None+ except (TypeError, ValueError):+ return None+++_CLIENT_KWARGS = {+ "provider_name": "sarvam",+ "base_url": speech_config.base_url("sarvam"),+}++register_tts_provider(+ TtsProviderSpec(+ name="sarvam",+ factory=SarvamSpeechProvider,+ sdk_exception_refs=("httpx:HTTPError",),+ client_kwargs=_CLIENT_KWARGS,+ )+)++register_stt_provider(+ SttProviderSpec(+ name="sarvam",+ factory=SarvamSpeechProvider,+ sdk_exception_refs=("httpx:HTTPError",),+ client_kwargs=_CLIENT_KWARGS,+ )+)diff --git a/server/nodes/speech/_registry.py b/server/nodes/speech/_registry.pynew file mode 100644index 00000000..cb1bdc87--- /dev/null+++ b/server/nodes/speech/_registry.py@@ -0,0 +1,89 @@+"""Two provider registries — one per direction.++The mechanism lives in :mod:`services.provider_registry`, shared with+``services/llm``. This module is the speech-specific pair of instances plus+the free-function API provider modules import.++Why two rather than one registry with a capability flag: **membership is the+capability**. A synthesis-only vendor registers into ``tts`` and simply never+appears in the transcription node's dropdown, because that dropdown is+literally ``stt_providers()``. There is no ``supports_stt`` boolean to fall+out of sync with what the provider can actually do, and no way to select a+provider for a direction it cannot serve.+"""++from __future__ import annotations++from typing import List++from services.provider_registry import (+ ProviderRegistry,+ ProviderSpec as _ProviderSpec,+)+from ._protocol import SttProvider, TtsProvider++# Specialized to each direction's protocol so ``ProviderSpec(factory=...)``+# type-checks against the right shape.+TtsProviderSpec = _ProviderSpec[TtsProvider]+SttProviderSpec = _ProviderSpec[SttProvider]++_TTS_REGISTRY: ProviderRegistry[TtsProvider] = ProviderRegistry("text-to-speech")+_STT_REGISTRY: ProviderRegistry[SttProvider] = ProviderRegistry("speech-to-text")+++def register_tts_provider(spec: TtsProviderSpec) -> None:+ """Register a synthesis provider. Idempotent for identical specs."""+ _TTS_REGISTRY.register(spec)+++def register_stt_provider(spec: SttProviderSpec) -> None:+ """Register a transcription provider. Idempotent for identical specs."""+ _STT_REGISTRY.register(spec)+++def get_tts_provider(name: str) -> TtsProviderSpec:+ """Look up a synthesis spec; raises ``NodeUserError`` on a miss."""+ return _TTS_REGISTRY.get(name)+++def get_stt_provider(name: str) -> SttProviderSpec:+ """Look up a transcription spec; raises ``NodeUserError`` on a miss."""+ return _STT_REGISTRY.get(name)+++def tts_providers() -> List[str]:+ """Sorted synthesis provider names — the TTS node's provider enum."""+ return _TTS_REGISTRY.all()+++def stt_providers() -> List[str]:+ """Sorted transcription provider names — the STT node's provider enum."""+ return _STT_REGISTRY.all()+++def has_tts_provider(name: str) -> bool:+ return _TTS_REGISTRY.has(name)+++def has_stt_provider(name: str) -> bool:+ return _STT_REGISTRY.has(name)+++def _reset_for_tests() -> None:+ """Test-only — drop every registered provider in both directions."""+ _TTS_REGISTRY.reset_for_tests()+ _STT_REGISTRY.reset_for_tests()+++__all__ = [+ "SttProviderSpec",+ "TtsProviderSpec",+ "get_stt_provider",+ "get_tts_provider",+ "has_stt_provider",+ "has_tts_provider",+ "register_stt_provider",+ "register_tts_provider",+ "stt_providers",+ "tts_providers",+]diff --git a/server/nodes/speech/_unifier.py b/server/nodes/speech/_unifier.pynew file mode 100644index 00000000..2bc292c0--- /dev/null+++ b/server/nodes/speech/_unifier.py@@ -0,0 +1,192 @@+"""Dispatch and error translation for the two speech nodes.++Reads the registries to route a call to the right provider, and normalizes+every provider failure into ``NodeUserError`` at one catch site. The nodes+never touch a provider class.++**No client cache here, deliberately.** ``ChatUnifier`` caches SDK clients+because an agent loop makes many model calls inside one node execution, so+connection reuse is worth a lease-counted LRU. Speech is the opposite shape:+a node execution issues exactly one HTTP request, and the client setup cost+is invisible next to a multi-second synthesis. Caching would buy nothing and+cost a process-wide singleton plus a shutdown hook wired into ``main.py`` --+core coupling this plugin has no reason to need. A client is built per call+and closed in ``finally``.++That also settles the question left open when the generic registry was+extracted: there is no second consumer for a shared client cache, so none+was built.+"""++from __future__ import annotations++from typing import Awaitable, Callable, List, Optional, TypeVar++from core.logging import get_logger+from services.plugin import NodeUserError+from services.provider_registry import ProviderSpec++from ._protocol import (+ SpeechError,+ SpeechErrorCategory,+ SttRequest,+ SttResult,+ TtsRequest,+ TtsResult,+ Voice,+)+from ._registry import get_stt_provider, get_tts_provider++logger = get_logger(__name__)++T = TypeVar("T")+++async def synthesize(+ *,+ provider: str,+ api_key: str,+ request: TtsRequest,+ translate_errors: bool = True,+) -> TtsResult:+ """Synthesize speech via the named provider."""+ spec = get_tts_provider(provider)+ return await _dispatch(+ spec,+ provider,+ api_key,+ lambda client: client.synthesize(request),+ operation="synthesis",+ translate_errors=translate_errors,+ )+++async def transcribe(+ *,+ provider: str,+ api_key: str,+ request: SttRequest,+ translate_errors: bool = True,+) -> SttResult:+ """Transcribe audio via the named provider."""+ spec = get_stt_provider(provider)+ return await _dispatch(+ spec,+ provider,+ api_key,+ lambda client: client.transcribe(request),+ operation="transcription",+ translate_errors=translate_errors,+ )+++async def list_voices(*, provider: str, api_key: str) -> List[Voice]:+ """List selectable voices for a synthesis provider.++ Providers without a live catalogue return their configured static list,+ so the dropdown loader never branches on whether an endpoint exists.+ """+ spec = get_tts_provider(provider)+ return await _dispatch(+ spec,+ provider,+ api_key,+ lambda client: client.list_voices(),+ operation="voice listing",+ translate_errors=True,+ )+++async def _dispatch(+ spec: ProviderSpec,+ provider: str,+ api_key: str,+ call: Callable[[object], Awaitable[T]],+ *,+ operation: str,+ translate_errors: bool,+) -> T:+ """Build a client, run ``call``, normalize failures, always close.++ One catch site for all three entry points. ``translate_errors=False``+ re-raises the structured :class:`SpeechError` instead of a flattened+ user message, which is what a Temporal retry policy needs to read+ ``retryable``.+ """+ client: Optional[object] = None+ try:+ client = spec.factory(api_key=api_key, **spec.client_kwargs)+ return await call(client)+ except spec.sdk_exception_types as exc:+ error = SpeechError.from_exception(provider, exc)+ if not translate_errors:+ raise error from exc+ logger.warning(+ "speech provider request failed",+ provider=error.provider,+ operation=operation,+ category=error.category.value,+ retryable=error.retryable,+ status_code=error.status_code,+ provider_code=error.provider_code,+ request_id=error.request_id,+ )+ raise NodeUserError(error.user_message) from error+ except SpeechError as exc:+ # A provider classified a failure itself -- typically a documented+ # vendor error that arrived with HTTP 200, which both Deepgram and+ # Sarvam can do. Already structured, so just surface it.+ if not translate_errors:+ raise+ logger.warning(+ "speech provider returned a failure payload",+ provider=exc.provider,+ operation=operation,+ category=exc.category.value,+ )+ raise NodeUserError(exc.user_message) from exc+ except (ValueError, TypeError, OSError) as exc:+ # Only normalize failures raised while *constructing* the client.+ # Once one exists, an unexpected exception from provider logic is a+ # programming error and keeps its traceback.+ if client is not None:+ raise+ category = (+ SpeechErrorCategory.CONNECTION+ if isinstance(exc, OSError)+ else SpeechErrorCategory.INVALID_REQUEST+ )+ error = SpeechError(+ message=str(exc),+ provider=provider,+ category=category,+ retryable=category == SpeechErrorCategory.CONNECTION,+ )+ if not translate_errors:+ raise error from exc+ logger.warning(+ "speech client construction failed",+ provider=error.provider,+ operation=operation,+ category=error.category.value,+ )+ raise NodeUserError(error.user_message) from error+ finally:+ if client is not None:+ await _close(client)+++async def _close(client: object) -> None:+ """Close a provider if it holds anything. Probed, not required."""+ close = getattr(client, "aclose", None)+ if close is None:+ return+ try:+ result = close()+ if hasattr(result, "__await__"):+ await result+ except Exception as exc:+ logger.warning("failed to close speech client", error=str(exc))+++__all__ = ["list_voices", "synthesize", "transcribe"]diff --git a/server/nodes/speech/icon_speechToText.svg b/server/nodes/speech/icon_speechToText.svgnew file mode 100644index 00000000..147620a7--- /dev/null+++ b/server/nodes/speech/icon_speechToText.svg@@ -0,0 +1,9 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">+ <rect x="8" y="2" width="6" height="11" rx="3"/>+ <path d="M5 10a6 6 0 0 0 12 0"/>+ <path d="M11 16v3"/>+ <path d="M8 22h6"/>+ <path d="M19 6h3"/>+ <path d="M19 10h3"/>+ <path d="M19 14h3"/>+</svg>diff --git a/server/nodes/speech/icon_textToSpeech.svg b/server/nodes/speech/icon_textToSpeech.svgnew file mode 100644index 00000000..2a385424--- /dev/null+++ b/server/nodes/speech/icon_textToSpeech.svg@@ -0,0 +1,5 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">+ <path d="M11 5 6 9H2v6h4l5 4z"/>+ <path d="M15.5 8.5a5 5 0 0 1 0 7"/>+ <path d="M19 5a9 9 0 0 1 0 14"/>+</svg>diff --git a/server/nodes/speech/meta.json b/server/nodes/speech/meta.jsonnew file mode 100644index 00000000..100645a1--- /dev/null+++ b/server/nodes/speech/meta.json@@ -0,0 +1,4 @@+{+ "_comment": "Matches the 'language' palette group colour. The plugin is vendor-neutral, so it takes the group's accent rather than any one provider's brand.",+ "color": "#6A88E2"+}diff --git a/server/nodes/speech/speech_to_text.py b/server/nodes/speech/speech_to_text.pynew file mode 100644index 00000000..26047fd2--- /dev/null+++ b/server/nodes/speech/speech_to_text.py@@ -0,0 +1,242 @@+"""Speech to Text — one node, many providers.++Replaces the vendor-locked ``sarvamSpeechToText``, and closes two of its+defects in the process:++* **Path traversal.** The old node joined a user-supplied path onto the+ workspace root with no check, so ``audio_file="../../credentials.db"``+ read the encrypted credential store and uploaded it to the provider.+ Every input shape here routes through ``services.media``, which resolves+ under containment.+* **Billing.** The old node charged every transcription as 30 seconds+ because it never measured the clip. This one probes the real duration and+ bills that, or bills nothing when the duration genuinely cannot be+ determined.++The model field is ``stt_model``, not ``model`` -- see the sibling module's+docstring for why that naming is load-bearing.+"""++from __future__ import annotations++from typing import Any, Dict, List, Optional++from pydantic import BaseModel, Field++from services.plugin import ActionNode, NodeContext, Operation, TaskQueue++from ..model._credentials import GroqCredential, OpenAICredential, SarvamCredential+from . import _config as speech_config+from . import _unifier+from ._base import (+ measure_seconds,+ provider_api_key,+ read_audio_input,+ require_provider,+ track_usage,+)+from ._credentials import DeepgramCredential+from ._protocol import SttRequest+from ._registry import stt_providers++from . import _providers # noqa: F401 isort:skip++_PROVIDERS: List[str] = stt_providers()+++class SpeechToTextParams(BaseModel):+ tool_name: str = Field(+ default="speech_to_text",+ description="Override name shown to the LLM when used as a tool.",+ )+ tool_description: str = Field(+ default=(+ "Transcribe an audio file to text. Accepts a workspace path or an "+ "audio reference produced by an upstream node. Use when the user "+ "asks what was said in a recording, or wants a transcript."+ ),+ description="Override description shown to the LLM when used as a tool.",+ json_schema_extra={"rows": 3},+ )++ provider: str = Field(+ default=_PROVIDERS[0] if _PROVIDERS else "openai",+ description="Speech provider to transcribe with.",+ json_schema_extra={"enum": _PROVIDERS},+ )+ # Three shapes reach this field: an AudioRef from an upstream node or the+ # upload route, a bare workspace path, or the legacy base64 envelope the+ # file widget used to emit. ``read_audio_input`` handles all three.+ audio_file: Any = Field(+ default="",+ description="Audio file to transcribe — upload one, or point at a workspace path.",+ json_schema_extra={+ "widget": "file",+ "accept": "audio/*,.wav,.mp3,.m4a,.ogg,.opus,.flac,.aac,.webm,.amr",+ },+ )+ stt_model: str = Field(+ default="",+ description="Provider model id. Leave blank for the provider default.",+ json_schema_extra={"loadOptionsMethod": "speechModels"},+ )+ language: str = Field(+ default="",+ description="Language hint (e.g. en, hi-IN). Leave blank to auto-detect.",+ )+ translate: bool = Field(+ default=False,+ description="Translate to English instead of transcribing in-language.",+ )+ diarize: bool = Field(+ default=False,+ description="Label speakers, where the provider and model support it.",+ )+ timestamps: bool = Field(+ default=False,+ description="Return per-word timing, where the provider and model support it.",+ )+ provider_options: Dict[str, Any] = Field(+ default_factory=dict,+ description=(+ "Vendor-specific options passed through untouched — e.g. Deepgram "+ "keyterm/redact, Sarvam mode, OpenAI response_format."+ ),+ )++ model_config = {"extra": "ignore"}+++class SpeechToTextOutput(BaseModel):+ transcript: str = ""+ language: Optional[str] = None+ language_confidence: Optional[float] = None+ duration_seconds: Optional[float] = None+ words: List[Dict[str, Any]] = Field(default_factory=list)+ segments: List[Dict[str, Any]] = Field(default_factory=list)+ provider: str = ""+ stt_model: str = ""+ request_id: Optional[str] = None++ model_config = {"extra": "allow"}+++class SpeechToTextNode(ActionNode):+ type = "speechToText"+ display_name = "Speech to Text"+ subtitle = "Transcribe Audio"+ group = ("language", "tool")+ description = "Transcribe audio to text using any configured speech provider"+ component_kind = "square"+ tool_name = "speech_to_text"+ tool_description = (+ "Transcribe an audio file to text. Accepts a workspace path or an audio "+ "reference from an upstream node."+ )+ handles = (+ {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},+ {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},+ )+ credentials = (+ OpenAICredential,+ GroqCredential,+ DeepgramCredential,+ SarvamCredential,+ )+ annotations = {"destructive": False, "readonly": True, "open_world": True}+ task_queue = TaskQueue.REST_API+ usable_as_tool = True+ hide_input_handle = False+ hide_output_handle = False++ Params = SpeechToTextParams+ Output = SpeechToTextOutput++ @Operation("transcribe")+ async def transcribe(+ self, ctx: NodeContext, params: SpeechToTextParams+ ) -> SpeechToTextOutput:+ provider = require_provider(params.provider, _PROVIDERS, "speech-to-text")+ model = params.stt_model or speech_config.default_model(provider, "stt")++ max_bytes = speech_config.capability(+ provider, "stt", "max_upload_bytes", default=None+ )+ filename, blob, path = read_audio_input(+ params.audio_file, ctx, max_bytes=max_bytes+ )++ api_key = await provider_api_key(ctx, provider)+ result = await _unifier.transcribe(+ provider=provider,+ api_key=api_key,+ request=SttRequest(+ audio=blob,+ filename=filename,+ mime_type=_guess_mime(filename),+ model=model,+ language=params.language,+ translate=params.translate,+ diarize=params.diarize,+ timestamps=params.timestamps,+ provider_options=dict(params.provider_options),+ ),+ )++ # Prefer what the provider reported; fall back to probing the file we+ # already have. Either way the figure is measured, never assumed.+ duration = result.duration_seconds+ if duration is None:+ duration = measure_seconds(path, declared_format=filename.rsplit(".", 1)[-1])++ await track_usage(+ ctx,+ provider=provider,+ operation="speech_to_text",+ units=_billable(result.billed_units, result.billed_unit, duration),+ unit=result.billed_unit or "seconds",+ )++ return SpeechToTextOutput(+ transcript=result.text,+ language=result.language,+ language_confidence=result.language_confidence,+ duration_seconds=duration,+ words=[+ {+ "word": w.word,+ "start": w.start,+ "end": w.end,+ "speaker": w.speaker,+ "confidence": w.confidence,+ }+ for w in result.words+ ],+ segments=result.segments,+ provider=provider,+ stt_model=model,+ request_id=result.request_id,+ )+++def _billable(+ reported: Optional[float], unit: str, duration: Optional[float]+) -> Optional[float]:+ """Resolve the billable quantity in the provider's own unit.++ A provider that reported its own figure wins. Otherwise the measured+ duration is converted into that provider's unit. When nothing could be+ measured this returns ``None`` and no metric is written -- an+ under-count is honest, a fabricated 30 seconds was not.+ """+ if reported is not None:+ return reported+ if duration is None:+ return None+ return duration / 60.0 if unit == "minutes" else duration+++def _guess_mime(filename: str) -> str:+ import mimetypes++ return mimetypes.guess_type(filename)[0] or "application/octet-stream"diff --git a/server/nodes/speech/text_to_speech.py b/server/nodes/speech/text_to_speech.pynew file mode 100644index 00000000..488d9348--- /dev/null+++ b/server/nodes/speech/text_to_speech.py@@ -0,0 +1,247 @@+"""Text to Speech — one node, many providers.++Replaces the vendor-locked ``sarvamTextToSpeech``. The provider is a+parameter rather than a node type, so adding a vendor never adds a node.++Two things about this node are unusual for the codebase and both are+deliberate:++* It declares **several credentials** and picks one at runtime via+ ``ctx.connection(<credential_id>)``. That is why every operation here is+ imperative: the declarative ``routing=`` DSL resolves ``credentials[0]``+ and would silently authenticate every provider with the first key in the+ tuple.+* Its model field is called ``tts_model``, not ``model``. A parameter panel+ effect keyed on the literal names ``model`` and ``api_key`` overwrites+ them with chat-model data whenever a sibling ``provider`` field is+ present, and it does not check that the provider is an LLM provider --+ so a field named ``model`` here would be cleared the moment the user+ picked ElevenLabs.++Audio never leaves this node as bytes. Clips are written into the workflow+workspace and the node returns ``AudioRef`` values, for the reasons in+``services/media/limits.py``.+"""++from __future__ import annotations++from typing import Any, Dict, List, Optional++from pydantic import BaseModel, Field++from services.media import AudioRef, write_audio+from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue++from ..model._credentials import OpenAICredential, SarvamCredential+from . import _config as speech_config+from . import _unifier+from ._base import provider_api_key, require_provider, track_usage+from ._credentials import ElevenLabsCredential+from ._protocol import TtsRequest+from ._registry import tts_providers++# Import for the registration side effect so the provider list below is+# populated before the schema is built.+from . import _providers # noqa: F401 isort:skip++_PROVIDERS: List[str] = tts_providers()+++class TextToSpeechParams(BaseModel):+ tool_name: str = Field(+ default="text_to_speech",+ description="Override name shown to the LLM when used as a tool.",+ )+ tool_description: str = Field(+ default=(+ "Convert text into spoken audio. Returns a reference to an audio "+ "file saved in the workspace, not the audio itself. Use when the "+ "user asks for speech, narration, a voiceover or an audio version "+ "of some text."+ ),+ description="Override description shown to the LLM when used as a tool.",+ json_schema_extra={"rows": 3},+ )++ provider: str = Field(+ default=_PROVIDERS[0] if _PROVIDERS else "openai",+ description="Speech provider to synthesize with.",+ json_schema_extra={"enum": _PROVIDERS},+ )+ text: str = Field(+ ...,+ min_length=1,+ description="Text to speak.",+ json_schema_extra={"rows": 4},+ )+ # NOT `model` -- see the module docstring.+ tts_model: str = Field(+ default="",+ description="Provider model id. Leave blank for the provider default.",+ json_schema_extra={"loadOptionsMethod": "speechModels"},+ )+ voice: str = Field(+ default="",+ description="Voice id. Leave blank for the provider default.",+ json_schema_extra={"loadOptionsMethod": "speechVoices"},+ )+ language: str = Field(+ default="",+ description=(+ "Language or locale code. Required by Sarvam (e.g. hi-IN); "+ "auto-detected by the others."+ ),+ )+ speed: Optional[float] = Field(+ default=None,+ description=(+ "Playback rate. Supported ranges differ per provider and values "+ "outside them are clamped."+ ),+ )+ output_format: str = Field(+ default="",+ description="Audio format. Leave blank for the provider default.",+ )+ sample_rate: Optional[int] = Field(+ default=None,+ description="Output sample rate in Hz, where the provider supports it.",+ )+ provider_options: Dict[str, Any] = Field(+ default_factory=dict,+ description=(+ "Vendor-specific options passed through untouched — e.g. "+ "ElevenLabs stability/similarity_boost, Sarvam pitch/loudness, "+ "OpenAI instructions."+ ),+ )++ model_config = {"extra": "ignore"}+++class TextToSpeechOutput(BaseModel):+ audio: Optional[AudioRef] = Field(+ default=None, description="The generated clip; the first when several."+ )+ files: List[AudioRef] = Field(+ default_factory=list,+ description="Every generated clip. More than one when the provider split long input.",+ )+ chunk_count: int = 0+ provider: str = ""+ tts_model: str = ""+ voice: str = ""+ request_id: Optional[str] = None+ note: Optional[str] = None++ model_config = {"extra": "allow"}+++class TextToSpeechNode(ActionNode):+ type = "textToSpeech"+ display_name = "Text to Speech"+ subtitle = "Synthesize Speech"+ group = ("language", "tool")+ description = "Convert text to spoken audio using any configured speech provider"+ component_kind = "square"+ tool_name = "text_to_speech"+ tool_description = (+ "Convert text into spoken audio. Returns a reference to an audio file "+ "saved in the workspace. Pick the provider and voice that suit the "+ "language and tone requested."+ )+ handles = (+ {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},+ {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},+ )+ # First multi-credential node in the repo. ``ctx.connection(id)`` is a+ # lookup over this tuple, so every provider's key is reachable and the+ # selected one decides which is used.+ credentials = (OpenAICredential, ElevenLabsCredential, SarvamCredential)+ annotations = {"destructive": False, "readonly": False, "open_world": True}+ task_queue = TaskQueue.REST_API+ usable_as_tool = True+ # `usable_as_tool` otherwise auto-hides both handles, which would break+ # chaining this into Speech to Text on the canvas.+ hide_input_handle = False+ hide_output_handle = False+ ui_hints = {"outputMode": "audio"}++ Params = TextToSpeechParams+ Output = TextToSpeechOutput++ @Operation("synthesize")+ async def synthesize(+ self, ctx: NodeContext, params: TextToSpeechParams+ ) -> TextToSpeechOutput:+ provider = require_provider(params.provider, _PROVIDERS, "text-to-speech")+ model = params.tts_model or speech_config.default_model(provider, "tts")++ self._check_length(provider, model, params.text)++ api_key = await provider_api_key(ctx, provider)+ result = await _unifier.synthesize(+ provider=provider,+ api_key=api_key,+ request=TtsRequest(+ text=params.text,+ model=model,+ voice=params.voice,+ language=params.language,+ speed=params.speed,+ output_format=params.output_format,+ sample_rate=params.sample_rate,+ provider_options=dict(params.provider_options),+ ),+ )++ refs = [+ write_audio(+ payload.data,+ ctx=ctx,+ stem=params.text[:32],+ ext=payload.format or "wav",+ mime_type=payload.mime_type,+ sample_rate=payload.sample_rate,+ channels=payload.channels or 1,+ )+ for payload in result.audio+ ]++ await track_usage(+ ctx,+ provider=provider,+ operation="text_to_speech",+ units=result.billed_units,+ unit=result.billed_unit,+ )++ note = None+ if len(refs) > 1:+ # Each clip carries its own container header, so concatenating+ # the bytes produces a file that plays only the first chunk.+ note = (+ f"{provider} split this text into {len(refs)} clips. They are "+ "separate playable files, not parts of one stream."+ )++ return TextToSpeechOutput(+ audio=refs[0] if refs else None,+ files=refs,+ chunk_count=len(refs),+ provider=provider,+ tts_model=result.model or model,+ voice=result.voice or params.voice,+ request_id=result.request_id,+ note=note,+ )++ @staticmethod+ def _check_length(provider: str, model: str, text: str) -> None:+ """Refuse over-long input before spending a paid call on it."""+ cap = speech_config.max_input_chars(provider, model)+ if cap and len(text) > cap:+ raise NodeUserError(+ f"Text is {len(text)} characters; {provider}/{model} accepts at "+ f"most {cap}. Shorten it or split it across several runs."+ )diff --git a/server/routers/workspace.py b/server/routers/workspace.pynew file mode 100644index 00000000..20f34189--- /dev/null+++ b/server/routers/workspace.py@@ -0,0 +1,200 @@+"""Per-workflow workspace file access.++Two routes: serve a file out of a workflow's workspace, and accept an+upload into it. Both exist because audio cannot travel through the workflow+engine as bytes (see ``services/media/limits.py``) — the UI needs a URL to+play generated audio from, and the file parameter needs somewhere to put an+uploaded clip other than base64 inside the node's parameters.++**The id/slug asymmetry is the thing to understand here.** The URL carries+``workflow_id`` because an ``AudioRef`` deliberately stores the immutable id,+so a reference keeps working after the workflow is renamed. But the+directory on disk is named by ``Workflow.slug``, which the rename path+moves. This router owns the id → slug lookup, because it is the layer that+has a database; ``services.media`` stays synchronous and takes the resolved+directory.++Containment is delegated to ``resolve_within``, which rejects ``..`` / ``~``+/ drive-prefixed input before touching the filesystem and re-checks+containment after resolution so a symlink or Windows junction cannot+redirect the result outside the root.+"""++from __future__ import annotations++import hashlib+import mimetypes+from pathlib import Path+from types import SimpleNamespace+from typing import Optional++from fastapi import APIRouter, Depends, File, HTTPException, Path as PathParam, UploadFile+from fastapi.responses import FileResponse++from core.container import container+from core.database import Database+from core.logging import get_logger+from core.paths import workspaces_dir+from services.media import MEDIA_MAX_UPLOAD_BYTES, AudioRef, write_audio+from services.media.workspace import UPLOAD_SUBDIR++logger = get_logger(__name__)++router = APIRouter(prefix="/api/workspace", tags=["workspace"])++# Types a browser may render in place. Everything else downloads.+#+# `text/html` and `image/svg+xml` are excluded deliberately and must stay+# excluded: `shell`, `fileDownloader` and `fileModify` can all write+# arbitrary files into a workspace, so serving attacker-authored markup+# inline from the app origin would be stored XSS with access to the session+# cookie. Both types are script-bearing.+_INLINE_PREFIXES = ("audio/", "image/", "video/")+_NEVER_INLINE = {"image/svg+xml", "text/html", "text/xml", "application/xhtml+xml"}++# Bounded so a hostile Content-Length cannot make us allocate; the running+# total is what enforces the cap, not the declared length.+_UPLOAD_CHUNK_BYTES = 1024 * 1024+++def _db() -> Database:+ return container.database()+++async def _workspace_root(workflow_id: str, database: Database) -> Path:+ """Resolve a workflow id to its on-disk workspace directory.++ Mirrors ``WorkflowService._resolve_workflow_slug`` including the+ ``"default"`` fallback, which is the anonymous workspace a one-off run+ without a saved row writes into.+ """+ slug: Optional[str] = None+ try:+ workflow = await database.get_workflow(workflow_id)+ slug = getattr(workflow, "slug", None) if workflow else None+ except Exception as exc:+ logger.warning(+ "workspace slug lookup failed", workflow_id=workflow_id, error=str(exc)+ )+ return workspaces_dir() / (slug or "default")+++def _resolve(root: Path, rel_path: str) -> Path:+ """Contain ``rel_path`` under ``root`` or 404.++ 404 rather than 403 throughout: a different status for "exists but+ forbidden" would confirm the existence of files outside the workspace.+ """+ from nodes.filesystem._backend import resolve_within++ try:+ target = resolve_within(root, rel_path)+ except ValueError:+ raise HTTPException(status_code=404, detail="Not found")+ if not target.is_file():+ raise HTTPException(status_code=404, detail="Not found")+ return target++[email protected]("/{workflow_id}/files/{file_path:path}")+async def serve_workspace_file(+ workflow_id: str = PathParam(..., min_length=1, max_length=128),+ file_path: str = PathParam(..., min_length=1),+ database: Database = Depends(_db),+) -> FileResponse:+ """Serve one file from a workflow's workspace.++ Range requests work without any code here: Starlette's ``FileResponse``+ already implements ``Accept-Ranges``, ``206``, ``Content-Range``,+ ``If-Range`` and ``416``. Wrapping this in a ``StreamingResponse`` would+ lose all of that and break seeking in an ``<audio>`` element.+ """+ root = await _workspace_root(workflow_id, database)+ target = _resolve(root, file_path)++ media_type = mimetypes.guess_type(target.name)[0] or "application/octet-stream"+ inline = media_type.startswith(_INLINE_PREFIXES) and media_type not in _NEVER_INLINE+ disposition = "inline" if inline else "attachment"++ return FileResponse(+ target,+ media_type=media_type,+ headers={+ "Content-Disposition": f'{disposition}; filename="{target.name}"',+ # Workspace files are mutable, so no immutable caching. The+ # filename already carries a random suffix, which makes a short+ # revalidating cache safe.+ "Cache-Control": "private, max-age=0, must-revalidate",+ "X-Content-Type-Options": "nosniff",+ },+ )++[email protected]("/{workflow_id}/uploads")+async def upload_workspace_file(+ workflow_id: str = PathParam(..., min_length=1, max_length=128),+ file: UploadFile = File(...),+ database: Database = Depends(_db),+) -> AudioRef:+ """Accept a file into the workflow's workspace and return a reference.++ Read in bounded chunks with a running total rather than+ ``await file.read()``: the declared ``Content-Length`` is attacker+ controlled, so the only trustworthy limit is what has actually been+ read. Exceeding the cap aborts immediately, before the whole body has+ been received.++ Returns an ``AudioRef`` — the shape ``coerce_file_param`` already+ documents as "what the upload route now returns", so the file parameter+ on a node accepts it with no further plumbing.+ """+ root = await _workspace_root(workflow_id, database)++ digest = hashlib.sha256()+ chunks: list[bytes] = []+ total = 0+ while True:+ chunk = await file.read(_UPLOAD_CHUNK_BYTES)+ if not chunk:+ break+ total += len(chunk)+ if total > MEDIA_MAX_UPLOAD_BYTES:+ raise HTTPException(+ status_code=413,+ detail=(+ f"Upload exceeds the {MEDIA_MAX_UPLOAD_BYTES // (1024 * 1024)} MB "+ "limit."+ ),+ )+ digest.update(chunk)+ chunks.append(chunk)++ if not total:+ raise HTTPException(status_code=400, detail="Uploaded file is empty.")++ payload = b"".join(chunks)+ name = Path(file.filename or "upload.bin").name+ stem, _, ext = name.rpartition(".")++ # write_audio owns filename sanitization (Windows reserved device names,+ # slugging, a random suffix so nothing collides or overwrites) and the+ # atomic write. Reimplementing any of that here would be a second, worse+ # copy.+ ref = write_audio(+ payload,+ ctx=SimpleNamespace(+ workspace_dir=str(root), node_id="upload", workflow_id=workflow_id+ ),+ stem=stem or name,+ ext=ext or "bin",+ mime_type=file.content_type or None,+ subdir=UPLOAD_SUBDIR,+ )++ logger.info(+ "workspace upload stored",+ workflow_id=workflow_id,+ path=ref.path,+ size_bytes=ref.size_bytes,+ )+ return refdiff --git a/server/services/media/workspace.py b/server/services/media/workspace.pyindex 3bb2ee38..3617b13c 100644--- a/server/services/media/workspace.py+++ b/server/services/media/workspace.py@@ -51,15 +51,37 @@ def _slugify(value: str, *, limit: int = 32) -> str:return slug-def workspace_root(ctx: Any = None, *, workflow_id: Optional[str] = None) -> Path:- """Resolve the workspace directory for a context or workflow id.-- Prefers the executor-injected ``ctx.workspace_dir``; falls back to- composing it from the workflow's slug so the HTTP routes -- which have- no NodeContext -- can resolve the same directory.+def workspace_root(+ ctx: Any = None,+ *,+ workspace_dir: Optional[str] = None,+ workflow_id: Optional[str] = None,+) -> Path:+ """Resolve the workspace directory for a context.++ Order: an explicit ``workspace_dir``, then the executor-injected+ ``ctx.workspace_dir``. Callers with neither get an error rather than a+ guess.++ ``workflow_id`` is accepted for call-site readability but is **not**+ sufficient on its own, and that is deliberate. Workspace directories are+ named by ``Workflow.slug`` (see ``WorkflowService._get_workspace_dir``),+ while an ``AudioRef`` deliberately carries the immutable ``workflow_id``+ so a reference survives a rename. Turning one into the other needs a+ database read, which cannot happen in a sync helper -- so the caller+ that has a database (the workspace HTTP route) does the lookup and+ passes the resolved directory in via ``workspace_dir``.++ An earlier version composed ``workspaces/<workflow_id>/`` here. That+ silently produced a path that never exists, because the directory on+ disk is ``workspaces/<slug>/``. It never fired in practice only because+ every caller happened to supply a ctx."""from services.plugin import NodeUserError+ if workspace_dir:+ return Path(workspace_dir)+direct = getattr(ctx, "workspace_dir", None) if ctx is not None else Noneif not direct and ctx is not None:raw = getattr(ctx, "raw", None)@@ -70,20 +92,18 @@ def workspace_root(ctx: Any = None, *, workflow_id: Optional[str] = None) -> Patif workflow_id is None and ctx is not None:workflow_id = getattr(ctx, "workflow_id", None)- if not workflow_id:- raise NodeUserError(- "No workspace is available for this execution, so media cannot be "- "read or written. Save the workflow and run it again."- )- from core.paths import workspace_dir as _workspace_dir-- return Path(_workspace_dir(workflow_id))+ raise NodeUserError(+ "No workspace is available for this execution, so media cannot be "+ "read or written. Save the workflow and run it again."+ + (f" (workflow {workflow_id})" if workflow_id else "")+ )def resolve_media(ref: AudioRef | str,*,ctx: Any = None,+ workspace_dir: Optional[str] = None,workflow_id: Optional[str] = None,) -> Path:"""Resolve a ref or a path string to a contained absolute path."""@@ -97,20 +117,19 @@ def resolve_media(if isinstance(ref, AudioRef) and ref.workflow_id and workflow_id is None:workflow_id = ref.workflow_id+ root = workspace_root(ctx, workspace_dir=workspace_dir, workflow_id=workflow_id)+candidate = Path(key)if candidate.is_absolute():# Tolerated for back-compat with nodes that stored absolute paths# before AudioRef existed, but still contained: an absolute path# outside the workspace is refused rather than read.- root = workspace_root(ctx, workflow_id=workflow_id)try:key = candidate.resolve(strict=False).relative_to(root).as_posix()except ValueError as exc:raise NodeUserError(f"'{candidate}' is outside this workflow's workspace.") from exc- else:- root = workspace_root(ctx, workflow_id=workflow_id)try:return resolve_within(root, key)@@ -122,13 +141,16 @@ def read_media_bytes(ref: AudioRef | str,*,ctx: Any = None,+ workspace_dir: Optional[str] = None,workflow_id: Optional[str] = None,max_bytes: int = MEDIA_MAX_READ_BYTES,) -> Tuple[str, bytes]:"""Return ``(filename, bytes)`` for a contained media reference."""from services.plugin import NodeUserError- target = resolve_media(ref, ctx=ctx, workflow_id=workflow_id)+ target = resolve_media(+ ref, ctx=ctx, workspace_dir=workspace_dir, workflow_id=workflow_id+ )if not target.is_file():raise NodeUserError(f"Media file not found: {target.name}")diff --git a/server/skills/language_agent/sarvam-speech-skill/SKILL.md b/server/skills/language_agent/sarvam-speech-skill/SKILL.mddeleted file mode 100644index 8b22ce0d..00000000--- a/server/skills/language_agent/sarvam-speech-skill/SKILL.md+++ /dev/null@@ -1,121 +0,0 @@-----name: sarvam-speech-skill-description: Transcribe speech to text and synthesize natural speech in Indian languages using Sarvam AI's Saaras and Bulbul models.-allowed-tools: sarvam_speech_to_text sarvam_text_to_speech-metadata:- author: opencompany- version: "1.0"- category: language-------# Sarvam Speech Skill--Speech-to-text across 23 Indian languages (Saaras) and text-to-speech with-37 voices (Bulbul).--## How It Works--This skill provides instructions and context. To execute, connect the-**Sarvam Speech to Text** and/or **Sarvam Text to Speech** nodes to the-agent's `input-tools` handle.--## sarvam_speech_to_text--| Field | Type | Required | Description |-|---|---|---|---|-| audio_file | string | Yes | Path to an audio file |-| model | enum | No | `saaras:v3` (default, 23 languages) or `saarika:v2.5` (legacy) |-| language_code | string | No | BCP-47 code, or `unknown` (default) to auto-detect |-| mode | enum | No | `transcribe` (default), `translate`, `verbatim`, `translit`, `codemix` |--Modes:-- `transcribe` — native script, cleaned up-- `translate` — English output regardless of the spoken language-- `verbatim` — keeps disfluencies ("um", repetitions); use for compliance- or verbatim-record work-- `translit` — Roman script-- `codemix` — preserves mixed-language speech as spoken--Everything except `transcribe` and `translate` requires `saaras:v3`.--```json-{- "transcript": "नमस्ते, आप कैसे हैं?",- "language_code": "hi-IN",- "language_probability": 0.97-}-```--**Hard limit: clips under 30 seconds.** This node calls the synchronous-endpoint. Longer recordings need Sarvam's Batch API, which is not wired-here — split the audio first, or tell the user it is out of scope rather-than retrying.--Formats: WAV, MP3, AAC, AIFF, OGG, OPUS, FLAC, MP4/M4A, AMR, WMA, WebM,-PCM. Accuracy is best at a 16 kHz sample rate.--## sarvam_text_to_speech--| Field | Type | Required | Description |-|---|---|---|---|-| text | string | Yes | Text to speak (max 2500 chars on v3, 1500 on v2) |-| target_language_code | enum | No | Defaults to `hi-IN` |-| model | enum | No | `bulbul:v3` (default) or `bulbul:v2` |-| speaker | string | No | Voice name, lowercase. Defaults to `shubh` |-| pace | number | No | v3: 0.5–2.0, v2: 0.3–3.0. Default 1.0 |-| speech_sample_rate | enum | No | 8000–48000 Hz, default 24000 |-| output_audio_codec | enum | No | `wav` (default), `mp3`, `flac`, `aac`, `opus`, `linear16`, `mulaw`, `alaw` |-| return_audio | enum | No | `file` (default) or `base64` |--Voices are **per model** and do not overlap:-- `bulbul:v3` — shubh, aditya, ritu, priya, neha, rahul, pooja, rohan,- simran, kavya, amit, dev, ishita, shreya, ratan, varun, manan, sumit,- roopa, kabir, aayan, ashutosh, advait, anand, tanya, tarun, sunny, mani,- gokul, vijay, shruti, suhani, mohit, kavitha, rehan, soham, rupali-- `bulbul:v2` — anushka, manisha, vidya, arya, abhilash, karun, hitesh--Asking for a v2 voice on v3 is an error, not a fallback.--```json-{- "file_path": "/…/workspaces/My_Flow_1/audio/namaste-a1b2c3d4-9f8e7d.wav",- "files": ["/…/namaste-a1b2c3d4-9f8e7d.wav"],- "chunk_count": 1,- "audio_format": "wav"-}-```--**The audio is written to a file, not returned inline.** Pass `file_path`-along to whatever consumes it. Only set `return_audio: "base64"` for very-short clips you must inline — a full-length request is roughly 12 MB of-base64, which bloats the workflow output and your own context.--For long text, Sarvam may return several chunks. Each is a standalone audio-file with its own header, so `files` lists them all and `file_path` points-at the first. Concatenating them byte-wise produces an unplayable file.--## When to Use--- Transcribing voice notes, IVR recordings or short call clips-- Generating voice prompts and announcements in Indian languages-- Building a voice loop: transcribe → reason → synthesize a reply--## When NOT to Use--- Recordings longer than 30 seconds (Batch API territory)-- Speaker diarization — only the Batch API returns it-- Real-time streaming — this is the request/response endpoint-- Languages outside the Indian set plus English--## API Details--- **Speech to Text**: POST `https://api.sarvam.ai/speech-to-text` (multipart) — 30 INR/hour-- **Text to Speech**: POST `https://api.sarvam.ai/text-to-speech` — 30 INR / 10K chars (v3)-- **Auth**: `api-subscription-key` header--## Setup Requirements--A single Sarvam API key from [dashboard.sarvam.ai](https://dashboard.sarvam.ai),-entered once in the Credentials modal under **Sarvam AI**. The same key-covers speech, translation and the Sarvam chat models.diff --git a/server/skills/language_agent/sarvam-translate-skill/SKILL.md b/server/skills/language_agent/sarvam-translate-skill/SKILL.mdindex 0ee8462f..b3fb302f 100644--- a/server/skills/language_agent/sarvam-translate-skill/SKILL.md+++ b/server/skills/language_agent/sarvam-translate-skill/SKILL.md@@ -99,7 +99,7 @@ language was detected.- Localising content into Indian languages- Normalising user input typed in Roman script into native script- Routing a message to the right handler based on its language-- Preparing text for `sarvam_text_to_speech`, which needs a target language+- Preparing text for `text_to_speech`, which needs a target language## When NOT to Usediff --git a/server/skills/language_agent/speech-skill/SKILL.md b/server/skills/language_agent/speech-skill/SKILL.mdnew file mode 100644index 00000000..8a494f19--- /dev/null+++ b/server/skills/language_agent/speech-skill/SKILL.md@@ -0,0 +1,111 @@+---+name: speech-skill+description: Convert text to spoken audio and transcribe audio to text, across multiple speech providers (OpenAI, ElevenLabs, Deepgram, Groq, Sarvam AI).+allowed-tools: text_to_speech speech_to_text+metadata:+ author: opencompany+ version: "2.0"+ category: language++---++# Speech Skill++Two tools, several providers behind each. Pick the provider that fits the+job rather than defaulting to one.++## How It Works++Connect the **Text to Speech** and/or **Speech to Text** nodes to the+agent's `input-tools` handle. Each node has a `provider` parameter, and the+credential for the chosen provider must be configured in the Credentials+modal.++Audio never travels as data. Synthesis writes a file into the workflow+workspace and returns a **reference** to it; transcription accepts a path or+a reference produced upstream. So Text to Speech can be wired straight into+Speech to Text, and neither tool ever puts audio bytes into the+conversation.++## text_to_speech++| Field | Type | Required | Description |+|---|---|---|---|+| text | string | Yes | The text to speak |+| provider | enum | No | `openai` (default), `elevenlabs`, `sarvam` |+| tts_model | string | No | Provider model id; blank uses the provider default |+| voice | string | No | Voice id; blank uses the provider default |+| language | string | No | Locale code. Required for Sarvam (e.g. `hi-IN`); auto-detected elsewhere |+| speed | number | No | Playback rate. Clamped to whatever the provider allows |+| output_format | string | No | Audio format; blank uses the provider default |+| provider_options | object | No | Vendor-specific extras, passed through untouched |++Returns `audio` — a reference carrying `path`, `url`, `duration_seconds` and+`mime_type` — plus `files` and `chunk_count`.++**When several clips come back**, they are separate playable files, not+parts of one stream. Each carries its own container header, so concatenating+them produces audio that plays only the first chunk.++### Choosing a provider++- **openai** — solid general-purpose quality, 13 voices, inexpensive. Put+ `instructions` in `provider_options` to steer tone, but note it works only+ on `gpt-4o-mini-tts` and is ignored on `tts-1`.+- **elevenlabs** — the most natural and controllable. Requires an explicit+ `voice`; there is no account-wide default. `stability`,+ `similarity_boost`, `style` and `use_speaker_boost` go in+ `provider_options`.+- **sarvam** — Indian languages: 11 locales, 37 voices on `bulbul:v3`.+ `language` is required. `pitch` / `loudness` work on `bulbul:v2` only and+ `temperature` on `bulbul:v3` only; the wrong one for the model is dropped+ rather than sent.++## speech_to_text++| Field | Type | Required | Description |+|---|---|---|---|+| audio_file | string / reference | Yes | Workspace path, or a reference from an upstream node |+| provider | enum | No | `openai` (default), `deepgram`, `groq`, `sarvam` |+| stt_model | string | No | Provider model id; blank uses the provider default |+| language | string | No | Language hint; blank auto-detects |+| translate | boolean | No | Translate to English instead of transcribing in-language |+| diarize | boolean | No | Label speakers, where supported |+| timestamps | boolean | No | Per-word timing, where supported |+| provider_options | object | No | Vendor-specific extras, passed through untouched |++Returns `transcript`, `language`, `duration_seconds`, and — when asked for+and supported — `words` and `segments`.++### Choosing a provider++- **openai** — reliable default. Word timestamps require `whisper-1`; the+ `gpt-4o-transcribe` models return plain JSON only, so a timestamp request+ is quietly downgraded rather than failing.+- **deepgram** — best for long recordings, diarization and keyword boosting.+ Billed per minute.+- **groq** — fastest and cheapest in bulk, but it bills a 10-second minimum+ per request, so many tiny clips cost more than their duration suggests.+ `whisper-large-v3-turbo` cannot translate; use `whisper-large-v3` when+ `translate` is set.+- **sarvam** — Indian languages. Timestamps and diarization are unavailable+ on its synchronous endpoint and come back empty.++## When to Use++- Narration, voiceovers, or "read this out"+- Working out what was said in a recording+- A voice loop: transcribe, reason over the text, synthesize a reply++## When NOT to Use++- Real-time or streaming speech — these tools are batch only+- Translating text you already have — use a translation tool+- Very long recordings on a synchronous provider; split them first++## Setup Requirements++One API key per provider you intend to use, added in the Credentials modal.+Nothing is shared between them: an ElevenLabs key does not enable Deepgram.+The `openai`, `groq` and `sarvam` keys are the same ones their chat models+use, so if those are already configured, speech works with no extra setup.diff --git a/server/tests/media/test_audio_ref.py b/server/tests/media/test_audio_ref.pyindex a3a266bc..67e32311 100644--- a/server/tests/media/test_audio_ref.py+++ b/server/tests/media/test_audio_ref.py@@ -164,6 +164,45 @@ class TestContainment:read_media_bytes(ref, ctx=_ctx(beta, workflow_id="beta"))+class TestWorkspaceRootRefusesToGuess:+ """Regression: a workflow_id is not a workspace directory name.++ Directories are named by ``Workflow.slug``, which changes on rename,+ while an ``AudioRef`` carries the immutable ``workflow_id`` on purpose.+ Composing ``workspaces/<workflow_id>/`` produced a path that never+ exists. It stayed invisible because every caller happened to pass a+ ctx; the workspace HTTP route is the first that cannot.+ """++ def test_workflow_id_alone_is_refused(self):+ from services.media.workspace import workspace_root+ from services.plugin import NodeUserError++ with pytest.raises(NodeUserError):+ workspace_root(workflow_id="019f99e2-dc99-7cf3-9019-3ad3e6260de1")++ def test_no_ctx_and_no_hint_is_refused(self):+ from services.media.workspace import workspace_root+ from services.plugin import NodeUserError++ with pytest.raises(NodeUserError):+ workspace_root()++ def test_explicit_workspace_dir_wins(self, tmp_path):+ """How the HTTP route passes a directory it resolved via the DB."""+ from services.media.workspace import workspace_root++ assert workspace_root(workspace_dir=str(tmp_path)) == tmp_path++ def test_explicit_dir_lets_a_ref_resolve_without_a_ctx(self, tmp_path):+ ctx = _ctx(tmp_path)+ ref = write_audio(_wav_bytes(), ctx=ctx, stem="x", ext="wav")++ name, blob = read_media_bytes(ref, workspace_dir=str(tmp_path))+ assert blob == read_media_bytes(ref, ctx=ctx)[1]+ assert name.endswith(".wav")++class TestCoerceFileParam:def test_accepts_an_audio_ref(self, tmp_path):ctx = _ctx(tmp_path)diff --git a/server/tests/nodes/test_sarvam.py b/server/tests/nodes/test_sarvam.pyindex bc666591..715441c1 100644--- a/server/tests/nodes/test_sarvam.py+++ b/server/tests/nodes/test_sarvam.py@@ -1,12 +1,15 @@-"""Contract tests for the five Sarvam service nodes.+"""Contract tests for the three Sarvam text service nodes.-sarvamTranslate / sarvamTransliterate / sarvamDetectLanguage /-sarvamSpeechToText / sarvamTextToSpeech.+sarvamTranslate / sarvamTransliterate / sarvamDetectLanguage.-All five authenticate through ``ctx.connection("sarvam")``, which injects+All three authenticate through ``ctx.connection("sarvam")``, which injectsthe ``api-subscription-key`` header from the shared credential — the samekey the OpenAI-compatible ``sarvamChatModel`` uses. httpx is mocked withrespx so no network is touched.++Sarvam's speech endpoints are covered by ``test_speech.py`` instead: they+are reached through the provider-abstracted ``textToSpeech`` /+``speechToText`` nodes now, with Sarvam as one provider among several."""from __future__ import annotations@@ -237,293 +240,6 @@ class TestSarvamDetectLanguage:assert "1000" in result["error"]-# ============================================================================-# sarvamSpeechToText-# ============================================================================---class TestSarvamSpeechToText:- URL = f"{_BASE}/speech-to-text"-- @respx.mock- async def test_accepts_a_file_path(self, harness, tmp_path):- clip = tmp_path / "clip.wav"- clip.write_bytes(b"RIFF....WAVE")- respx.post(self.URL).mock(- return_value=httpx.Response(- 200,- json={- "transcript": "hello",- "language_code": "en-IN",- "language_probability": 0.98,- },- )- )-- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamSpeechToText", {"audio_file": str(clip)}- )-- harness.assert_envelope(result, success=True)- assert result["result"]["transcript"] == "hello"-- sent = respx.calls.last.request- assert sent.headers["content-type"].startswith("multipart/form-data")- assert b"RIFF" in sent.content-- @respx.mock- async def test_accepts_the_file_widget_upload_envelope(self, harness):- """The file widget sends {type, data, filename, mimeType}, not a path."""- respx.post(self.URL).mock(- return_value=httpx.Response(200, json={"transcript": "ok"})- )-- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamSpeechToText",- {- "audio_file": {- "type": "upload",- "data": base64.b64encode(b"AUDIOBYTES").decode(),- "filename": "voice.mp3",- "mimeType": "audio/mpeg",- }- },- )-- harness.assert_envelope(result, success=True)- assert b"AUDIOBYTES" in respx.calls.last.request.content-- @respx.mock- async def test_relative_path_resolves_against_the_workspace(self, harness, tmp_path):- (tmp_path / "audio").mkdir()- (tmp_path / "audio" / "a.wav").write_bytes(b"WAV")- respx.post(self.URL).mock(- return_value=httpx.Response(200, json={"transcript": "ok"})- )-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamSpeechToText", {"audio_file": "audio/a.wav"}, context=ctx- )-- harness.assert_envelope(result, success=True)-- async def test_missing_file_is_a_user_error(self, harness):- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamSpeechToText", {"audio_file": "/nope/missing.wav"}- )-- harness.assert_envelope(result, success=False)- assert "not found" in result["error"].lower()-- async def test_empty_audio_file_param_is_a_user_error(self, harness):- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute("sarvamSpeechToText", {})-- harness.assert_envelope(result, success=False)- assert "audio_file is required" in result["error"]-- async def test_advanced_mode_requires_saaras_v3(self, harness, tmp_path):- clip = tmp_path / "c.wav"- clip.write_bytes(b"WAV")-- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamSpeechToText",- {- "audio_file": str(clip),- "model": "saarika:v2.5",- "mode": "codemix",- },- )-- harness.assert_envelope(result, success=False)- assert "saaras:v3" in result["error"]---# ============================================================================-# sarvamTextToSpeech-# ============================================================================---class TestSarvamTextToSpeech:- URL = f"{_BASE}/text-to-speech"-- @staticmethod- def _audio_response(chunks: int = 1) -> httpx.Response:- blob = base64.b64encode(b"RIFFfakeWAVEdata").decode()- return httpx.Response(- 200, json={"request_id": "r", "audios": [blob] * chunks}- )-- @respx.mock- async def test_writes_to_the_workspace_by_default(self, harness, tmp_path):- respx.post(self.URL).mock(return_value=self._audio_response())-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech",- {"text": "Namaste", "target_language_code": "hi-IN"},- context=ctx,- )-- harness.assert_envelope(result, success=True)- payload = result["result"]- written = Path(payload["file_path"])- assert written.is_file()- assert written.parent == tmp_path / "audio"- assert written.read_bytes() == b"RIFFfakeWAVEdata"- assert payload["chunk_count"] == 1- assert payload["files"] == [str(written)]- # The DB/WS bloat guard: no inline base64 unless asked for.- assert payload.get("audio_base64") is None-- @respx.mock- async def test_base64_mode_returns_inline_and_writes_nothing(self, harness, tmp_path):- respx.post(self.URL).mock(return_value=self._audio_response())-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech",- {"text": "Namaste", "return_audio": "base64"},- context=ctx,- )-- payload = result["result"]- assert payload["audio_base64"] == base64.b64encode(b"RIFFfakeWAVEdata").decode()- assert payload.get("file_path") is None- assert not (tmp_path / "audio").exists()-- @respx.mock- async def test_multi_chunk_writes_one_file_per_chunk(self, harness, tmp_path):- respx.post(self.URL).mock(return_value=self._audio_response(chunks=3))-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech", {"text": "long text"}, context=ctx- )-- payload = result["result"]- assert payload["chunk_count"] == 3- assert len(payload["files"]) == 3- assert all(Path(p).is_file() for p in payload["files"])- assert payload["file_path"] == payload["files"][0]- assert "3 audio chunks" in payload["note"]-- @respx.mock- async def test_oversize_inline_audio_is_suppressed_with_a_note(- self, harness, tmp_path, monkeypatch- ):- """Never silently truncate — say why and point at the file mode."""- monkeypatch.setattr(- "nodes.sarvam.sarvam_text_to_speech._MAX_INLINE_B64", 4- )- respx.post(self.URL).mock(return_value=self._audio_response())-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech",- {"text": "hi", "return_audio": "base64"},- context=ctx,- )-- payload = result["result"]- assert payload["audio_base64"] is None- assert "return_audio='file'" in payload["note"]-- @respx.mock- async def test_v2_only_params_are_not_sent_for_v3(self, harness, tmp_path):- import json-- respx.post(self.URL).mock(return_value=self._audio_response())-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- await harness.execute(- "sarvamTextToSpeech",- {"text": "hi", "model": "bulbul:v3", "speaker": "shubh"},- context=ctx,- )-- body = json.loads(respx.calls.last.request.content)- assert "pitch" not in body- assert "loudness" not in body- assert "enable_preprocessing" not in body-- @respx.mock- async def test_v2_params_are_sent_for_v2(self, harness, tmp_path):- import json-- respx.post(self.URL).mock(return_value=self._audio_response())-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- await harness.execute(- "sarvamTextToSpeech",- {- "text": "hi",- "model": "bulbul:v2",- "speaker": "anushka",- "pitch": 0.2,- },- context=ctx,- )-- body = json.loads(respx.calls.last.request.content)- assert body["pitch"] == 0.2- assert "temperature" not in body-- async def test_speaker_from_the_wrong_model_is_a_user_error(self, harness):- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech",- {"text": "hi", "model": "bulbul:v3", "speaker": "anushka"},- )-- harness.assert_envelope(result, success=False)- assert "not a bulbul:v3 voice" in result["error"]-- async def test_text_over_model_cap_is_a_user_error(self, harness):- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech",- {"text": "x" * 1501, "model": "bulbul:v2", "speaker": "anushka"},- )-- harness.assert_envelope(result, success=False)- assert "1500" in result["error"]-- async def test_v3_pace_out_of_range_is_a_user_error(self, harness):- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech", {"text": "hi", "pace": 2.5}- )-- harness.assert_envelope(result, success=False)- assert "0.5-2.0" in result["error"]-- @respx.mock- async def test_empty_audio_array_is_a_user_error(self, harness, tmp_path):- respx.post(self.URL).mock(return_value=httpx.Response(200, json={"audios": []}))-- ctx = harness.build_context(workspace_dir=str(tmp_path))- with patched_container(auth_api_keys=_KEYS), patched_pricing():- result = await harness.execute(- "sarvamTextToSpeech", {"text": "hi"}, context=ctx- )-- harness.assert_envelope(result, success=False)- assert "no audio" in result["error"].lower()--# ============================================================================# Cross-node contracts# ============================================================================@@ -534,8 +250,6 @@ class TestSarvamPluginContract:"sarvamTranslate","sarvamTransliterate","sarvamDetectLanguage",- "sarvamSpeechToText",- "sarvamTextToSpeech",)@pytest.mark.parametrize("node_type", NODE_TYPES)diff --git a/server/tests/nodes/test_speech.py b/server/tests/nodes/test_speech.pynew file mode 100644index 00000000..2e55ac4f--- /dev/null+++ b/server/tests/nodes/test_speech.py@@ -0,0 +1,641 @@+"""Contract for the provider-abstracted speech nodes.++The interesting tests here are the wire-shape ones. Every provider in the v1+set diverges from the others on auth, request transport and response+transport, and three of those divergences fail *silently* when got wrong:+ElevenLabs ignores a body-placed ``output_format`` and returns the default,+Deepgram ignores body-placed options entirely, and Sarvam accepts a request+whose base64 array is then mis-parsed. None of them produce an error, so+only an assertion on the outgoing request catches a regression.+"""++from __future__ import annotations++import base64+import struct+import subprocess+import sys+import wave+from pathlib import Path++import httpx+import pytest+import respx++from ._mocks import patched_container, patched_pricing++pytestmark = pytest.mark.node_contract++SERVER_DIR = Path(__file__).resolve().parents[2]++_KEYS = {+ "openai": "sk-test",+ "groq": "gsk-test",+ "elevenlabs": "el-test",+ "deepgram": "dg-test",+ "sarvam": "sv-test",+}+++def _wav(seconds: float = 0.5, rate: int = 8000) -> bytes:+ import io++ buffer = io.BytesIO()+ with wave.open(buffer, "wb") as handle:+ handle.setnchannels(1)+ handle.setsampwidth(2)+ handle.setframerate(rate)+ handle.writeframes(struct.pack("<h", 0) * int(rate * seconds))+ return buffer.getvalue()+++def _clip(harness, tmp_path, *, payload: bytes | None = None, name: str = "a.wav"):+ """Put a clip inside the workspace and return ``(context, relative_name)``.++ Audio has to live *in* the workspace to be readable: containment refuses+ absolute paths outside it, which is the whole point of the design. Tests+ that hand the node a bare ``tmp_path`` are testing the guard, not the+ provider.+ """+ (tmp_path / name).write_bytes(payload if payload is not None else _wav())+ return harness.build_context(workspace_dir=str(tmp_path)), name+++# ============================================================================+# Registry — direction membership IS the capability+# ============================================================================+++class TestDirectionMembership:+ def test_transcription_only_provider_is_absent_from_synthesis(self):+ from nodes.speech._registry import tts_providers++ assert "deepgram" not in tts_providers()+ assert "groq" not in tts_providers()++ def test_synthesis_only_provider_is_absent_from_transcription(self):+ from nodes.speech._registry import stt_providers++ assert "elevenlabs" not in stt_providers()++ def test_dual_direction_provider_is_in_both(self):+ from nodes.speech._registry import stt_providers, tts_providers++ for provider in ("openai", "sarvam"):+ assert provider in tts_providers()+ assert provider in stt_providers()++ def test_every_provider_resolves_to_a_registered_credential(self):+ """The layering contract: config names credential *ids*, not classes."""+ from services.plugin.credential import CREDENTIAL_REGISTRY++ from nodes.speech import _config+ from nodes.speech._registry import stt_providers, tts_providers++ for provider in set(tts_providers()) | set(stt_providers()):+ credential = _config.credential_id(provider)+ assert credential in CREDENTIAL_REGISTRY, (+ f"{provider} maps to credential {credential!r}, which is not registered"+ )+++# ============================================================================+# Config — the per-model override ladder+# ============================================================================+++class TestCapabilityResolution:+ def test_exact_model_override_wins(self):+ from nodes.speech import _config++ assert "verbose_json" in _config.response_formats("openai", "whisper-1")+ assert _config.response_formats("openai", "gpt-4o-transcribe") == [+ "json",+ "text",+ ]++ def test_prefix_match_covers_dated_snapshots(self):+ """A dated release must not silently fall through to _default."""+ from nodes.speech import _config++ assert _config.response_formats(+ "openai", "gpt-4o-mini-transcribe-2025-12-15"+ ) == ["json", "text"]++ def test_declared_null_is_distinguishable_from_absent(self):+ """Deepgram declares no upload cap; that is a fact, not a gap."""+ from nodes.speech import _config++ assert (+ _config.capability("deepgram", "stt", "max_upload_bytes", default="ABSENT")+ is None+ )+ assert (+ _config.capability("deepgram", "stt", "nonexistent", default="ABSENT")+ == "ABSENT"+ )++ def test_boolean_capabilities_default_permissive(self):+ from nodes.speech import _config++ assert _config.supports("openai", "tts", "never_declared_flag") is True++ def test_disjoint_voice_sets_per_model(self):+ from nodes.speech import _config++ v2 = set(_config.voices("sarvam", model="bulbul:v2"))+ v3 = set(_config.voices("sarvam", model="bulbul:v3"))+ assert v2 and v3 and not (v2 & v3)+++# ============================================================================+# Wire shapes — the silent-failure guards+# ============================================================================+++class TestElevenLabsWireShape:+ @respx.mock+ async def test_output_format_rides_the_query_string_not_the_body(+ self, harness, tmp_path+ ):+ """A body-placed output_format is ignored and the default returned."""+ route = respx.post(+ url__regex=r"https://api\.elevenlabs\.io/v1/text-to-speech/.*"+ ).mock(return_value=httpx.Response(200, content=b"ID3audio"))++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "textToSpeech",+ {+ "provider": "elevenlabs",+ "text": "hello",+ "voice": "voice123",+ "output_format": "mp3_22050_32",+ },+ context=harness.build_context(workspace_dir=str(tmp_path)),+ )++ harness.assert_envelope(result, success=True)+ request = route.calls.last.request+ assert request.url.params["output_format"] == "mp3_22050_32"+ assert b"output_format" not in request.content++ @respx.mock+ async def test_voice_is_a_path_segment_and_auth_has_no_scheme(self, harness):+ route = respx.post(+ url__regex=r"https://api\.elevenlabs\.io/v1/text-to-speech/.*"+ ).mock(return_value=httpx.Response(200, content=b"audio"))++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute(+ "textToSpeech",+ {"provider": "elevenlabs", "text": "hi", "voice": "abc123"},+ )++ request = route.calls.last.request+ assert request.url.path.endswith("/v1/text-to-speech/abc123")+ assert request.headers["xi-api-key"] == "el-test"+ assert "authorization" not in request.headers++ @respx.mock+ async def test_speed_is_clamped_to_the_api_range(self, harness):+ route = respx.post(+ url__regex=r"https://api\.elevenlabs\.io/v1/text-to-speech/.*"+ ).mock(return_value=httpx.Response(200, content=b"audio"))++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute(+ "textToSpeech",+ {+ "provider": "elevenlabs",+ "text": "hi",+ "voice": "v",+ "speed": 9.0,+ },+ )++ import json as _json++ body = _json.loads(route.calls.last.request.content)+ assert body["voice_settings"]["speed"] == 2.0++ async def test_missing_voice_is_a_user_error(self, harness):+ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "textToSpeech", {"provider": "elevenlabs", "text": "hi"}+ )++ harness.assert_envelope(result, success=False)+ assert "voice" in result["error"].lower()+++class TestDeepgramWireShape:+ URL = "https://api.deepgram.com/v1/listen"++ def _response(self, transcript: str = "hello there", duration: float = 12.0):+ return httpx.Response(+ 200,+ json={+ "metadata": {"request_id": "dg-1", "duration": duration},+ "results": {+ "channels": [+ {+ "detected_language": "en",+ "language_confidence": 0.99,+ "alternatives": [+ {+ "transcript": transcript,+ "words": [+ {+ "word": "hello",+ "start": 0.1,+ "end": 0.4,+ "confidence": 0.9,+ }+ ],+ }+ ],+ }+ ]+ },+ },+ )++ @respx.mock+ async def test_auth_uses_token_not_bearer(self, harness, tmp_path):+ ctx, name = _clip(harness, tmp_path)+ route = respx.post(self.URL).mock(return_value=self._response())++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute(+ "speechToText",+ {"provider": "deepgram", "audio_file": name},+ context=ctx,+ )++ assert route.calls.last.request.headers["authorization"] == "Token dg-test"++ @respx.mock+ async def test_options_are_query_params_and_audio_is_the_raw_body(+ self, harness, tmp_path+ ):+ payload = _wav()+ ctx, name = _clip(harness, tmp_path, payload=payload)+ route = respx.post(self.URL).mock(return_value=self._response())++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute(+ "speechToText",+ {+ "provider": "deepgram",+ "audio_file": name,+ "stt_model": "nova-3",+ "diarize": True,+ },+ context=ctx,+ )++ request = route.calls.last.request+ assert request.url.params["model"] == "nova-3"+ assert request.url.params["diarize"] == "true"+ # Raw body, not multipart -- no boundary, and the bytes are verbatim.+ assert not request.headers["content-type"].startswith("multipart/")+ assert request.content == payload++ @respx.mock+ async def test_transcript_is_read_from_the_nested_path(self, harness, tmp_path):+ ctx, name = _clip(harness, tmp_path)+ respx.post(self.URL).mock(return_value=self._response("the actual transcript"))++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "speechToText",+ {"provider": "deepgram", "audio_file": name},+ context=ctx,+ )++ harness.assert_envelope(result, success=True)+ assert result["result"]["transcript"] == "the actual transcript"+ assert result["result"]["language"] == "en"+ assert result["result"]["words"][0]["word"] == "hello"+++class TestSarvamWireShape:+ URL = "https://api.sarvam.ai/text-to-speech"++ @respx.mock+ async def test_base64_array_becomes_one_file_per_chunk(self, harness, tmp_path):+ clip = base64.b64encode(_wav()).decode()+ respx.post(self.URL).mock(+ return_value=httpx.Response(+ 200, json={"request_id": "sv-1", "audios": [clip, clip, clip]}+ )+ )++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "textToSpeech",+ {"provider": "sarvam", "text": "namaste"},+ context=harness.build_context(workspace_dir=str(tmp_path)),+ )++ harness.assert_envelope(result, success=True)+ payload = result["result"]+ assert payload["chunk_count"] == 3+ assert len(payload["files"]) == 3+ # Distinct paths -- concatenating them would be wrong, so they must+ # not collide either.+ assert len({f["path"] for f in payload["files"]}) == 3+ assert "3 clips" in (payload["note"] or "")++ @respx.mock+ async def test_v2_only_params_are_not_sent_to_v3(self, harness):+ respx.post(self.URL).mock(+ return_value=httpx.Response(+ 200, json={"audios": [base64.b64encode(_wav()).decode()]}+ )+ )++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute(+ "textToSpeech",+ {+ "provider": "sarvam",+ "text": "hi",+ "tts_model": "bulbul:v3",+ "provider_options": {"pitch": 0.5, "temperature": 0.7},+ },+ )++ import json as _json++ body = _json.loads(respx.calls.last.request.content)+ assert "pitch" not in body+ assert body["temperature"] == 0.7++ @respx.mock+ async def test_auth_uses_the_native_subscription_header(self, harness):+ respx.post(self.URL).mock(+ return_value=httpx.Response(+ 200, json={"audios": [base64.b64encode(_wav()).decode()]}+ )+ )++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute("textToSpeech", {"provider": "sarvam", "text": "hi"})++ assert respx.calls.last.request.headers["api-subscription-key"] == "sv-test"++ async def test_over_cap_text_is_refused_before_the_call(self, harness):+ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "textToSpeech",+ {"provider": "sarvam", "text": "x" * 3000, "tts_model": "bulbul:v3"},+ )++ harness.assert_envelope(result, success=False)+ assert "2500" in result["error"]+++class TestOpenAIWireShape:+ @respx.mock+ async def test_synthesis_returns_raw_bytes_and_writes_one_ref(+ self, harness, tmp_path+ ):+ respx.post("https://api.openai.com/v1/audio/speech").mock(+ return_value=httpx.Response(200, content=b"ID3rawaudio")+ )++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "textToSpeech",+ {"provider": "openai", "text": "hello"},+ context=harness.build_context(workspace_dir=str(tmp_path)),+ )++ harness.assert_envelope(result, success=True)+ audio = result["result"]["audio"]+ assert audio["kind"] == "audio"+ assert audio["path"].startswith("audio/")+ assert audio["size_bytes"] == len(b"ID3rawaudio")++ @respx.mock+ async def test_response_format_downgrades_on_a_gated_model(self, harness, tmp_path):+ """verbose_json on a 4o model is a 400; the node must not send it."""+ ctx, name = _clip(harness, tmp_path)+ route = respx.post("https://api.openai.com/v1/audio/transcriptions").mock(+ return_value=httpx.Response(200, json={"text": "hi"})+ )++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ await harness.execute(+ "speechToText",+ {+ "provider": "openai",+ "audio_file": name,+ "stt_model": "gpt-4o-transcribe",+ "timestamps": True,+ },+ context=ctx,+ )++ body = route.calls.last.request.content+ assert b"verbose_json" not in body+ assert b"timestamp_granularities" not in body++ @respx.mock+ async def test_groq_transcription_hits_its_own_host(self, harness, tmp_path):+ ctx, name = _clip(harness, tmp_path)+ route = respx.post(+ "https://api.groq.com/openai/v1/audio/transcriptions"+ ).mock(return_value=httpx.Response(200, json={"text": "hi", "duration": 3.0}))++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "speechToText",+ {"provider": "groq", "audio_file": name},+ context=ctx,+ )++ harness.assert_envelope(result, success=True)+ assert route.called+ assert route.calls.last.request.headers["authorization"] == "Bearer gsk-test"+++# ============================================================================+# Node-level contract+# ============================================================================+++class TestNodeContract:+ def test_neither_node_names_a_field_model_or_api_key(self):+ """Regression guard for the parameter-panel clobber.++ ParameterRenderer overwrites any field literally named ``model`` or+ ``api_key`` with chat-model data whenever a sibling ``provider``+ field exists, and it never checks that the provider is an LLM+ provider. A field with either name here would be cleared the moment+ a user picked a speech provider.+ """+ from nodes.speech.speech_to_text import SpeechToTextParams+ from nodes.speech.text_to_speech import TextToSpeechParams++ for params in (TextToSpeechParams, SpeechToTextParams):+ assert "model" not in params.model_fields+ assert "api_key" not in params.model_fields++ def test_neither_node_uses_declarative_routing(self):+ """``routing=`` resolves credentials[0], which ignores `provider`."""+ from nodes.speech.speech_to_text import SpeechToTextNode+ from nodes.speech.text_to_speech import TextToSpeechNode++ for node in (TextToSpeechNode, SpeechToTextNode):+ for spec in node._operations.values():+ assert spec.routing is None, (+ f"{node.type}.{spec.name} declares routing=, which would pin "+ "every provider to the first credential in the tuple"+ )++ def test_canvas_handles_stay_visible(self):+ """`usable_as_tool` auto-hides both handles unless declared False."""+ from nodes.speech.speech_to_text import SpeechToTextNode+ from nodes.speech.text_to_speech import TextToSpeechNode++ for node in (TextToSpeechNode, SpeechToTextNode):+ assert node.hide_input_handle is False+ assert node.hide_output_handle is False++ def test_declared_credentials_cover_every_registered_provider(self):+ from nodes.speech import _config+ from nodes.speech._registry import stt_providers, tts_providers+ from nodes.speech.speech_to_text import SpeechToTextNode+ from nodes.speech.text_to_speech import TextToSpeechNode++ for node, providers in (+ (TextToSpeechNode, tts_providers()),+ (SpeechToTextNode, stt_providers()),+ ):+ declared = {c.id for c in node.credentials}+ for provider in providers:+ assert _config.credential_id(provider) in declared, (+ f"{node.type} cannot authenticate {provider}"+ )++ async def test_unknown_provider_is_a_user_error_listing_the_real_ones(+ self, harness+ ):+ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "textToSpeech", {"provider": "nope", "text": "hi"}+ )++ harness.assert_envelope(result, success=False)+ assert "elevenlabs" in result["error"]+++class TestAudioInputContainment:+ """The Sarvam STT node read the credential store. This must not."""++ @pytest.mark.parametrize(+ "attack", ["../../credentials.db", "..\\..\\credentials.db", "~/secrets"]+ )+ async def test_traversal_is_refused(self, harness, tmp_path, attack):+ (tmp_path.parent / "credentials.db").write_bytes(b"SECRET")+ ctx = harness.build_context(workspace_dir=str(tmp_path))++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "speechToText",+ {"provider": "openai", "audio_file": attack},+ context=ctx,+ )++ harness.assert_envelope(result, success=False)++ async def test_missing_file_is_a_user_error(self, harness):+ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "speechToText",+ {"provider": "openai", "audio_file": "definitely/absent.wav"},+ )++ harness.assert_envelope(result, success=False)+++class TestBilling:+ @respx.mock+ async def test_duration_is_measured_not_assumed(self, harness, tmp_path):+ """The old Sarvam node billed every clip as a flat 30 seconds."""+ ctx, name = _clip(harness, tmp_path, payload=_wav(seconds=2.0, rate=8000))+ respx.post("https://api.sarvam.ai/speech-to-text").mock(+ return_value=httpx.Response(200, json={"transcript": "ok"})+ )++ with patched_container(auth_api_keys=_KEYS), patched_pricing():+ result = await harness.execute(+ "speechToText",+ {"provider": "sarvam", "audio_file": name},+ context=ctx,+ )++ harness.assert_envelope(result, success=True)+ duration = result["result"]["duration_seconds"]+ assert duration == pytest.approx(2.0, abs=0.1)+ assert duration != 30+++# ============================================================================+# Boot-path purity+# ============================================================================+++class TestLazySdkImports:+ """Registering speech providers must not import a heavy SDK.++ Ported from ``tests/llm/test_lazy_sdk_imports.py``. Runs in a clean+ interpreter because the pytest process already has SDKs loaded by other+ tests. This is the only thing preventing a future provider from+ reintroducing the eager-import-at-boot anti-pattern.+ """++ HEAVY = ("openai", "anthropic", "google.genai")++ def _probe(self, code: str) -> str:+ result = subprocess.run(+ [sys.executable, "-c", code],+ cwd=SERVER_DIR,+ capture_output=True,+ text=True,+ timeout=120,+ )+ assert result.returncode == 0, (+ f"probe subprocess failed (rc={result.returncode}):\n{result.stderr}"+ )+ return result.stdout++ def test_registering_providers_imports_no_sdk(self):+ out = self._probe(+ "import sys\n"+ "import nodes.speech\n"+ f"leaked = [m for m in {self.HEAVY!r} if m in sys.modules]\n"+ "print('LEAKED=' + ','.join(leaked))\n"+ )+ assert out.strip().endswith("LEAKED="), (+ f"speech provider registration imported heavy SDKs: {out!r}"+ )++ def test_every_declared_exception_ref_resolves(self):+ """Typo guard — a bad ref only fails when an error occurs otherwise."""+ import nodes.speech # noqa: F401++ from nodes.speech._registry import (+ get_stt_provider,+ get_tts_provider,+ stt_providers,+ tts_providers,+ )++ for name in tts_providers():+ assert get_tts_provider(name).sdk_exception_types+ for name in stt_providers():+ assert get_stt_provider(name).sdk_exception_typesdiff --git a/server/tests/routers/__init__.py b/server/tests/routers/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/server/tests/routers/test_workspace.py b/server/tests/routers/test_workspace.pynew file mode 100644index 00000000..b5810508--- /dev/null+++ b/server/tests/routers/test_workspace.py@@ -0,0 +1,226 @@+"""Contract for the workspace file routes.++Mounted on a bare FastAPI app rather than importing ``main`` — the app's+lifespan starts the container, the database and the Temporal runtime, none+of which this router needs. The router is the unit under test.++The security assertions here are not defensive padding. ``shell``,+``fileDownloader`` and ``fileModify`` can all write arbitrary files into a+workspace, so this route serves attacker-influenced content from the app+origin. Inline HTML would be stored XSS with session-cookie access.+"""++from __future__ import annotations++import struct+import wave+from pathlib import Path+from types import SimpleNamespace+from unittest.mock import AsyncMock++import httpx+import pytest+from fastapi import FastAPI++from routers import workspace as workspace_router++pytestmark = pytest.mark.unit++SLUG = "My_Workflow_1"+WORKFLOW_ID = "019f99e2dc997cf390193ad3e6260de1"+++def _wav(seconds: float = 0.25, rate: int = 8000) -> bytes:+ import io++ buffer = io.BytesIO()+ with wave.open(buffer, "wb") as handle:+ handle.setnchannels(1)+ handle.setsampwidth(2)+ handle.setframerate(rate)+ handle.writeframes(struct.pack("<h", 0) * int(rate * seconds))+ return buffer.getvalue()+++def workspace(tmp_path, monkeypatch):+ """A workspaces root containing one workflow directory, keyed by SLUG."""+ root = tmp_path / "workspaces"+ (root / SLUG / "audio").mkdir(parents=True)+ monkeypatch.setattr(workspace_router, "workspaces_dir", lambda: root)+ return root / SLUG+++def client(workspace):+ app = FastAPI()+ app.include_router(workspace_router.router)++ database = AsyncMock()+ database.get_workflow.return_value = SimpleNamespace(slug=SLUG)+ app.dependency_overrides[workspace_router._db] = lambda: database++ transport = httpx.ASGITransport(app=app)+ return httpx.AsyncClient(transport=transport, base_url="http://test")+++class TestServeFile:+ async def test_serves_audio_inline(self, client, workspace):+ payload = _wav()+ (workspace / "audio" / "clip.wav").write_bytes(payload)++ async with client as http:+ response = await http.get(+ f"/api/workspace/{WORKFLOW_ID}/files/audio/clip.wav"+ )++ assert response.status_code == 200+ assert response.content == payload+ assert response.headers["content-type"].startswith("audio/")+ assert response.headers["content-disposition"].startswith("inline")+ assert response.headers["x-content-type-options"] == "nosniff"++ async def test_url_carries_the_id_while_the_directory_uses_the_slug(+ self, client, workspace+ ):+ """The whole point of the id/slug split: refs survive a rename."""+ (workspace / "audio" / "clip.wav").write_bytes(_wav())++ async with client as http:+ response = await http.get(+ f"/api/workspace/{WORKFLOW_ID}/files/audio/clip.wav"+ )++ assert response.status_code == 200+ assert WORKFLOW_ID not in str(workspace)+ assert workspace.name == SLUG++ @pytest.mark.parametrize(+ "name,expected",+ [+ ("page.html", "attachment"),+ ("vector.svg", "attachment"),+ ("clip.wav", "inline"),+ ("cover.png", "inline"),+ ("notes.txt", "attachment"),+ ],+ )+ async def test_only_safe_media_renders_inline(+ self, client, workspace, name, expected+ ):+ """Script-bearing types must download, never render on our origin."""+ (workspace / name).write_bytes(b"<html><script>alert(1)</script></html>")++ async with client as http:+ response = await http.get(f"/api/workspace/{WORKFLOW_ID}/files/{name}")++ assert response.status_code == 200+ assert response.headers["content-disposition"].startswith(expected)++ @pytest.mark.parametrize(+ "attack",+ [+ "../../../etc/passwd",+ "..%2f..%2fsecret.txt",+ "....//secret.txt",+ ],+ )+ async def test_traversal_is_refused_as_404(self, client, workspace, attack):+ (workspace.parent.parent / "secret.txt").write_bytes(b"SECRET")++ async with client as http:+ response = await http.get(f"/api/workspace/{WORKFLOW_ID}/files/{attack}")++ assert response.status_code == 404+ assert b"SECRET" not in response.content++ async def test_missing_file_is_404_not_403(self, client):+ """A distinct status would confirm what exists outside the workspace."""+ async with client as http:+ response = await http.get(+ f"/api/workspace/{WORKFLOW_ID}/files/audio/nope.wav"+ )++ assert response.status_code == 404++ async def test_range_requests_work_without_router_code(self, client, workspace):+ """Starlette's FileResponse already implements 206; seeking depends on it."""+ payload = _wav()+ (workspace / "audio" / "clip.wav").write_bytes(payload)++ async with client as http:+ response = await http.get(+ f"/api/workspace/{WORKFLOW_ID}/files/audio/clip.wav",+ headers={"Range": "bytes=0-99"},+ )++ assert response.status_code == 206+ assert response.headers["accept-ranges"] == "bytes"+ assert response.headers["content-range"].startswith("bytes 0-99/")+ assert response.content == payload[:100]+++class TestUpload:+ async def test_stores_the_file_and_returns_a_ref(self, client, workspace):+ payload = _wav()++ async with client as http:+ response = await http.post(+ f"/api/workspace/{WORKFLOW_ID}/uploads",+ files={"file": ("My Clip.wav", payload, "audio/wav")},+ )++ assert response.status_code == 200+ ref = response.json()+ assert ref["kind"] == "audio"+ assert ref["path"].startswith("uploads/")+ assert ref["size_bytes"] == len(payload)+ assert ref["workflow_id"] == WORKFLOW_ID+ assert (workspace / ref["path"]).read_bytes() == payload++ async def test_returned_ref_is_immediately_servable(self, client, workspace):+ async with client as http:+ ref = (+ await http.post(+ f"/api/workspace/{WORKFLOW_ID}/uploads",+ files={"file": ("clip.wav", _wav(), "audio/wav")},+ )+ ).json()++ # The ref's own url must round-trip through the GET route.+ response = await http.get(ref["url"])++ assert response.status_code == 200++ async def test_over_cap_upload_is_413(self, client, monkeypatch):+ monkeypatch.setattr(workspace_router, "MEDIA_MAX_UPLOAD_BYTES", 1024)++ async with client as http:+ response = await http.post(+ f"/api/workspace/{WORKFLOW_ID}/uploads",+ files={"file": ("big.wav", b"x" * 4096, "audio/wav")},+ )++ assert response.status_code == 413++ async def test_empty_upload_is_400(self, client):+ async with client as http:+ response = await http.post(+ f"/api/workspace/{WORKFLOW_ID}/uploads",+ files={"file": ("empty.wav", b"", "audio/wav")},+ )++ assert response.status_code == 400++ async def test_filename_is_sanitized_not_trusted(self, client, workspace):+ """A traversal filename must not escape the uploads directory."""+ async with client as http:+ response = await http.post(+ f"/api/workspace/{WORKFLOW_ID}/uploads",+ files={"file": ("../../evil.wav", _wav(), "audio/wav")},+ )++ assert response.status_code == 200+ stored = Path(response.json()["path"])+ assert stored.parts[0] == "uploads"+ assert ".." not in stored.parts