Skip to content

Self-improving AI Employees for everyone

feat(speech): RFC + audio-transport and provider-registry foundations

trohitg18460e02096a
diff --git a/docs-internal/speech_provider_rfc.md b/docs-internal/speech_provider_rfc.md
new file mode 100644
index 00000000..46f0629a
--- /dev/null
+++ b/docs-internal/speech_provider_rfc.md
@@ -0,0 +1,244 @@
+# RFC: Multi-provider Speech (TTS / STT)
+
+**Status:** accepted, implementation in progress
+**Date:** 2026-07-25
+**Supersedes:** the vendor-locked `sarvamTextToSpeech` / `sarvamSpeechToText` nodes
+
+---
+
+## 1. Problem
+
+Speech exists in exactly one place today: two Sarvam nodes. The pattern does not generalise — a
+second vendor means a second pair of nodes, a third a third pair, each with its own params, its own
+credential, its own skill file. Chat models solved this years ago with `services/llm/`; speech has
+no equivalent.
+
+Adding one is not just a refactor, because **audio is big and the execution engine is not built for
+big**. That constraint, measured rather than assumed, drives nearly every decision below.
+
+---
+
+## 2. The measured constraint
+
+| Limit | Value | Source |
+|---|---|---|
+| Temporal blob **error** — activity result, activity **input**, workflow result | **2,097,152 B** | server default; no custom `DataConverter` or `PayloadCodec` anywhere in the repo |
+| Temporal blob **warning** | 524,288 B → `PayloadSizeWarning` | not covered by `pyproject.toml` `filterwarnings`, so it surfaces |
+| Retries burned before the failure is reported | **3** | `_PayloadSizeError` absent from `NON_RETRYABLE_ERROR_TYPES` ([`_retry_policies.py:51`](../server/services/temporal/_retry_policies.py)) |
+| Legacy `execute_node_activity` internal WS | 4 MiB inbound (aiohttp default, `max_msg_size` unset) | [`activities.py:242`](../server/services/temporal/activities.py) |
+| `node_outputs.data` | no cap; written **3×** per store (`output_main`/`output_top`/`output_0`) | [`activities.py:486`](../server/services/temporal/activities.py) |
+| WS broadcast | no size guard; retained in `_status` **forever** and replayed to every newly connecting client | [`status_broadcaster.py:87,688`](../server/services/status_broadcaster.py) |
+| In-memory `_outputs` | never evicted except by `clear_all_outputs` | [`workflow.py:577`](../server/services/workflow.py) |
+
+### What actually happens to a 12 MB base64 TTS result
+
+1. The activity result exceeds 2 MiB → temporalio raises `_PayloadSizeError` at the converter.
+2. It is not marked non-retryable → **3 attempts**, each re-invoking the provider and re-billing it.
+3. The user sees a generic activity failure, not "audio too large".
+
+And the same limit applies to activity **inputs**, so a ~2 MB audio upload fails before the node
+runs at all — today's only file-input path is base64-through-parameters.
+
+### The multiplication
+
+One audio payload is copied at least six ways: `_serialize_result` → `node_outputs` ×3 → WS
+broadcast ×2 → `_status` cache (forever) → Temporal activity result → `MachinaWorkflow` aggregate →
+every downstream activity input → and, because these nodes are `usable_as_tool`, verbatim into an
+LLM message.
+
+**Conclusion: audio cannot travel as bytes.** Not "should not" — the engine will not carry it.
+
+---
+
+## 3. Decisions
+
+### D1 — `AudioRef` is structurally incapable of carrying bytes
+
+A Pydantic model with `extra="forbid"`, a workspace-**relative** path, and no base64 field. Ever.
+It serializes to ~400 B, i.e. ~5,200 refs before approaching the Temporal error limit.
+
+Rejected alternative: a capped inline-base64 opt-in, as the current Sarvam TTS node has
+(`_MAX_INLINE_B64 = 1_000_000`). The decisive argument is `usable_as_tool` — `execute_as_tool`
+returns the flat `Output` dict straight into an LLM message and there is no mechanism to tell a
+model "skip this field". A capped opt-in is *sometimes* catastrophic, which is worse than
+always-broken because it passes review.
+
+Relative paths (not absolute) because absolute paths embed the mutable workflow slug, leak the
+operator's home directory into the DB / WebSocket / LLM context, and cannot be safely served over
+HTTP.
+
+### D2 — Two Protocols and two registries, not one `SpeechProvider`
+
+AssemblyAI is STT-only; Cartesia is TTS-only. A single Protocol forces those vendors to ship a dead
+method that raises — invisible to the node layer, so the provider would appear in the dropdown and
+fail at runtime.
+
+With `TtsProvider` and `SttProvider` registries, **direction capability is registry membership**:
+the TTS node's provider enum is literally `tts_registry.all_providers()`. Zero extra machinery.
+
+### D3 — Capability-driven common params + a `provider_options` escape hatch
+
+The verified provider surface diverges much harder than the chat-completions surface does:
+
+| | OpenAI | ElevenLabs | Deepgram |
+|---|---|---|---|
+| Auth header | `Authorization: Bearer` | `xi-api-key` | `Authorization: Token` |
+| Voice selection | body param `voice` | **URL path segment** | encoded in the model name (`aura-2-thalia-en`) |
+| Output format | `response_format` enum (`mp3\|opus\|aac\|flac\|wav\|pcm`) | **query** `output_format=mp3_44100_128` (container+rate+bitrate in one token) | separate `encoding` + `container` + `sample_rate` |
+| Tuning | `instructions` (free text), `speed` | `voice_settings{stability, similarity_boost, style, use_speaker_boost, speed}` | — |
+
+A per-provider `displayOptions` matrix over 8 providers explodes combinatorially and *still* cannot
+express "ElevenLabs `stability` applies only to `eleven_multilingual_v2`". So: ~11 common fields
+covering the 90 % case, normalised by each provider adapter, plus a `provider_options: dict`
+passed through untouched for vendor specifics.
+
+Keeps the tool schema flat — no `$defs` / `$ref`, which `usable_as_tool` requires.
+
+### D4 — Capabilities declared in JSON, not code
+
+`server/config/speech_defaults.json`, one block per provider per direction, overridable per model —
+the same shape `llm_defaults.json` uses for `max_output_tokens`. Drives the provider dropdown, the
+voice loader, and validation from one place. `test_plugin_shape.py` bans `if provider == "x"`
+branches in shared code; this is how we avoid needing any.
+
+### D5 — First multi-credential node in the repo
+
+`credentials = (OpenAICredential, ElevenLabsCredential, …)` with `ctx.connection(params.provider)`.
+Already supported: `_make_connection_factory` ([`base.py:870`](../server/services/plugin/base.py))
+builds a dict over **all** declared credentials and raises only for undeclared ids. No node uses
+this today.
+
+Consequence: the node must use imperative `@Operation` bodies — the declarative `routing=` path
+hardcodes `self.credentials[0]` ([`base.py:566`](../server/services/plugin/base.py)).
+
+### D6 — Extract the generic registry rather than copy it
+
+`ProviderSpec` + lazy `"module:Class"` `sdk_exception_refs` + idempotent registration is entirely
+provider-agnostic. Copying it into `services/speech/` would fork the boot-time-import-avoidance
+logic that exists specifically to keep ~7 s (warm) / ~45 s (cold) of SDK imports off startup.
+
+`services/provider_registry.py` + `services/provider_clients.py`; `services/llm/registry.py`
+becomes a shim. **Success criterion: `server/tests/llm/` passes untouched.**
+
+### D7 — `tinytag` for duration and format
+
+| Candidate | Native deps | Windows | License | Verdict |
+|---|---|---|---|---|
+| stdlib `wave` | none | ✅ | PSF | WAV only — insufficient alone |
+| **`tinytag`** | **none** | ✅ | **MIT** | **chosen** |
+| `mutagen` | none | ✅ | **GPL-2.0** | rejected — this repo is MIT; a licensing call, not a technical one |
+| `pydub` | ffmpeg on PATH | ❌ | MIT | rejected |
+| `ffprobe` | external binary | ❌ | LGPL/GPL | rejected — a pooch-managed binary for a 4-line metadata read |
+
+Fallback chain: `tinytag` → stdlib `wave` → raw-PCM arithmetic → give up.
+
+### D8 — Inspection never fails a workflow
+
+Unknown format returns an all-`None` probe, logs at DEBUG, and still produces a valid `AudioRef`.
+Duration-based **rejection only fires when duration is known**; unknown duration falls through to
+the byte cap.
+
+A metadata-parser miss on some codec variant must never turn a working workflow into a failing one.
+Degrading a billing *estimate* is acceptable; hard-failing a valid file is not.
+
+### D9 — The result-size guard ships WARN-only first
+
+A generic size check in `BaseNode._serialize_result` is the highest-value safety net available: it
+converts a 31-second silent triple-retry into an immediate, actionable error. But flipping straight
+to hard-fail risks breaking a node that legitimately returns a large result today (a document
+parser emitting 3 MB of extracted text).
+
+Ship the warning, read one release of logs, then flip. Both thresholds behind `Settings`.
+
+Implementation note: raise **`NodeUserError`** — it is already in `NON_RETRYABLE_ERROR_TYPES`, so
+the failure is immediate with zero new registrations. Do **not** subclass it: Temporal matches
+`non_retryable_error_types` on the exception **type-name string**, so a subclass silently starts
+retrying again.
+
+---
+
+## 4. Architecture
+
+```
+services/provider_registry.py generic ProviderSpec + registry (extracted from llm/)
+services/provider_clients.py generic lease-counted client cache (extracted from llm/)
+
+services/media/ audio transport — vendor-neutral, reusable for image/video
+ refs.py AudioRef
+ workspace.py write_audio / resolve_media / read_media_bytes / coerce_file_param
+ inspect.py tinytag -> wave -> PCM arithmetic; never raises
+ limits.py every size constant, one place
+
+services/speech/ provider abstraction — mirrors services/llm/
+ protocol.py TtsProvider / SttProvider, requests, results, SpeechError
+ registry.py two registries over the generic one
+ config.py reads server/config/speech_defaults.json
+ unifier.py SpeechUnifier — dispatch + typed-error -> NodeUserError
+ providers/ _http.py, _openai_compat.py (openai+groq STT), elevenlabs.py,
+ sarvam.py, deepgram.py, gemini.py
+
+nodes/speech/ the two nodes
+routers/workspace.py GET file (Range-capable) + POST upload
+```
+
+**Layering rule.** `services/speech` takes credential **id strings**, never `Credential` classes.
+The classes live under `nodes/`, and a `services → nodes` import inverts the layering and breaks
+`test_plugin_self_containment.py`. A test locks the two sides agree.
+
+---
+
+## 5. Two bugs this work closes
+
+1. **Path traversal in `sarvam_speech_to_text`.** `_read_audio` does `Path(workspace_dir) / raw`
+ with no containment check, so `audio_file="../../credentials.db"` reads the encrypted credential
+ store and uploads it to the provider. `coerce_file_param` closes it by construction.
+2. **Workflow rename orphans every workspace file.** `rename_workflow` is a single-row `UPDATE`; the
+ directory is never moved, and `_get_workspace_dir` keys on the mutable slug. Pre-existing and not
+ audio-specific, but `AudioRef` makes it fixable — a best-effort `os.replace` in the rename path.
+
+---
+
+## 6. Explicitly out of scope
+
+| Not doing | Why |
+|---|---|
+| Streaming / realtime speech | `@Operation` returns one Pydantic model; no `AsyncGenerator` support, no SSE anywhere in the repo. A genuinely new operation kind. |
+| Keying workspaces on `workflow_id` | Changes the on-disk layout for every existing install, invalidates `--add-dir` paths baked into live Claude sessions, and reverses a documented decision. Best-effort directory rename instead. |
+| Deduping the 3× `store_node_output` write | Touches `ParameterResolver`, the `socialReceive` four-handle case, and `store_agent_output` to save ~1.2 KB once refs replace blobs. |
+| Removing the double `node_status` + `node_output` broadcast | Frontend-visible contract change. Elide the cached payload instead so the duplication is bounded. |
+| Fixing `MachinaWorkflow.run`'s aggregate result | 20 nodes × 900 KB still exceeds 2 MiB even when every individual node passes. Structural Temporal change; the retry-policy fix makes it fail fast instead of slowly. |
+
+---
+
+## 7. Verified provider surface
+
+Confirmed against live documentation on 2026-07-25. Anything not listed here was **not** verified
+and must be checked before it is coded.
+
+**OpenAI** — TTS `POST /v1/audio/speech`, models `gpt-4o-mini-tts` / `tts-1` / `tts-1-hd`, voices
+`alloy ash ballad coral echo fable nova onyx sage shimmer verse marin cedar` (subset for tts-1),
+`response_format` ∈ `mp3 opus aac flac wav pcm`, plus `instructions` and `speed`. Raw binary
+response. STT `POST /v1/audio/transcriptions`, multipart.
+
+**ElevenLabs** — TTS `POST /v1/text-to-speech/{voice_id}`, header `xi-api-key`, body `text`,
+`model_id` (default `eleven_multilingual_v2`), `voice_settings{stability, similarity_boost, style,
+use_speaker_boost, speed}`, `language_code`, `seed`, `apply_text_normalization` ∈ `auto|on|off`;
+**query** `output_format` (default `mp3_44100_128`, 27 values). Binary response. Models via
+`GET /v1/models`, voices via `GET /v1/voices`. STT `POST /v1/speech-to-text`, multipart,
+`model_id` ∈ `scribe_v2|scribe_v1`, `language_code`, `tag_audio_events` (default true),
+`num_speakers` (≤32), `timestamps_granularity` ∈ `none|word|character`, `diarize` (default false).
+Response `{language_code, language_probability, text, words[]}`.
+
+**Groq** — STT `POST https://api.groq.com/openai/v1/audio/transcriptions`, OpenAI-compatible,
+`whisper-large-v3` / `whisper-large-v3-turbo`. This is why `_openai_compat.py` covers both vendors
+off one factory.
+
+**Deepgram** — TTS `POST /v1/speak`, header `Authorization: Token <key>`, model names of the form
+`aura-2-<voice>-<lang>`. STT `POST /v1/listen`. Query-parameter driven; exact parameter set to be
+confirmed at implementation time.
+
+**Sarvam** — already implemented; see [`sarvam_service.md`](./sarvam_service.md). TTS returns
+base64 inside JSON (`{audios: […]}`), the one provider that does.
+
+**Gemini** — TTS via `generateContent` with an audio response modality. Shape **not yet verified**;
+must be confirmed before coding.
diff --git a/server/nodes/filesystem/_backend.py b/server/nodes/filesystem/_backend.py
index 290423b1..8b5ed4e6 100644
--- a/server/nodes/filesystem/_backend.py
+++ b/server/nodes/filesystem/_backend.py
@@ -160,13 +160,26 @@ def _verify_posix_directory(
os.close(current_fd)
-def _write_utf8_to_fd(
+def _write_payload_to_fd(
file_fd: int,
- content: str,
+ payload: str | bytes,
existing_mode: Optional[int],
) -> None:
- with os.fdopen(file_fd, "w", encoding="utf-8", newline="") as handle:
- handle.write(content)
+ """Write ``payload`` to an already-opened fd, then fsync.
+
+ Accepts ``str`` (UTF-8, newline-preserving) or ``bytes``. The binary
+ branch exists so generated media — audio from a TTS provider, an
+ uploaded clip — reaches disk through the same anchored-write path that
+ refuses to follow a mutable symlink or Windows junction. Those bytes
+ are later served over HTTP, so the containment matters more than for
+ ordinary text writes.
+ """
+ if isinstance(payload, bytes):
+ mode, kwargs = "wb", {}
+ else:
+ mode, kwargs = "w", {"encoding": "utf-8", "newline": ""}
+ with os.fdopen(file_fd, mode, **kwargs) as handle:
+ handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
if existing_mode is not None and hasattr(os, "fchmod"):
@@ -186,10 +199,10 @@ def _replace_posix_entry(
)
-def _atomic_write_text_posix(
+def _atomic_write_posix(
root: Path,
relative: Path,
- content: str,
+ payload: str | bytes,
) -> None:
parent_components = tuple(relative.parts[:-1])
target_name = relative.parts[-1]
@@ -250,7 +263,7 @@ def _atomic_write_text_posix(
"Could not allocate a unique atomic-write temporary file"
)
- _write_utf8_to_fd(temporary_fd, content, existing_mode)
+ _write_payload_to_fd(temporary_fd, payload, existing_mode)
_verify_posix_directory(root, parent_components, parent_fd)
_replace_posix_entry(
temporary_name,
@@ -323,10 +336,10 @@ def _verify_windows_temp(
)
-def _atomic_write_text_windows(
+def _atomic_write_windows(
root: Path,
relative: Path,
- content: str,
+ payload: str | bytes,
) -> None:
"""Best-effort contained atomic replacement for Windows.
@@ -393,7 +406,7 @@ def _atomic_write_text_windows(
write_fd = temporary_fd
temporary_fd = -1
- _write_utf8_to_fd(write_fd, content, existing_mode)
+ _write_payload_to_fd(write_fd, payload, existing_mode)
if existing_mode is not None:
# Windows has no ``os.fchmod``. Apply its supported mode bits
# (notably the read-only flag) to the still-private temporary
@@ -459,19 +472,11 @@ def _atomic_write_text_windows(
pass
-def atomic_write_text(
+def _atomic_write(
path: os.PathLike[str] | str,
- content: str,
- *,
- root_dir: os.PathLike[str] | str | None = None,
+ payload: str | bytes,
+ root_dir: os.PathLike[str] | str | None,
) -> None:
- """Atomically replace UTF-8 text without traversing mutable symlinks.
-
- Production callers pass the workspace root. The optional default keeps
- direct helper callers compatible by treating the existing parent as the
- root, but it does not establish a broader containment boundary.
- """
-
target = Path(os.path.abspath(os.fspath(path)))
root = Path(
os.path.abspath(
@@ -488,9 +493,41 @@ def atomic_write_text(
raise IsADirectoryError(f"Cannot replace workspace root: {target}")
if _anchored_write_supported():
- _atomic_write_text_posix(root, relative, content)
+ _atomic_write_posix(root, relative, payload)
else:
- _atomic_write_text_windows(root, relative, content)
+ _atomic_write_windows(root, relative, payload)
+
+
+def atomic_write_text(
+ path: os.PathLike[str] | str,
+ content: str,
+ *,
+ root_dir: os.PathLike[str] | str | None = None,
+) -> None:
+ """Atomically replace UTF-8 text without traversing mutable symlinks.
+
+ Production callers pass the workspace root. The optional default keeps
+ direct helper callers compatible by treating the existing parent as the
+ root, but it does not establish a broader containment boundary.
+ """
+ _atomic_write(path, content, root_dir)
+
+
+def atomic_write_bytes(
+ path: os.PathLike[str] | str,
+ payload: bytes,
+ *,
+ root_dir: os.PathLike[str] | str | None = None,
+) -> None:
+ """Atomically replace binary content, with the same containment rules.
+
+ Used by ``services.media`` for generated and uploaded audio. Atomicity
+ is load-bearing here rather than merely tidy: the frontend renders an
+ ``<audio src>`` the moment it sees a reference, so a half-written file
+ is a broken player, and Temporal retries a synthesis activity against
+ the *same* target path.
+ """
+ _atomic_write(path, payload, root_dir)
def _validate_virtual_path(path: str) -> str:
@@ -508,6 +545,30 @@ def _validate_virtual_path(path: str) -> str:
return normalized
+def resolve_within(root: os.PathLike[str] | str, key: str) -> Path:
+ """Resolve ``key`` under ``root``, refusing anything that escapes it.
+
+ Two layers: reject ``..`` / ``~`` / drive-prefixed inputs *before*
+ touching the filesystem, then resolve and re-check containment so a
+ symlink or Windows junction cannot redirect the result outside the
+ root.
+
+ Module-level rather than a ``WorkspaceBackend`` method because
+ ``services.media`` and the workspace HTTP route both need it without
+ paying for a backend instance (whose ``__init__`` mkdirs, copies the
+ environment, and mints a sandbox id).
+ """
+ root_path = Path(os.path.abspath(os.fspath(root)))
+ virtual = _validate_virtual_path(key)
+ candidate = root_path / virtual.lstrip("/")
+ try:
+ resolved = candidate.resolve(strict=False)
+ resolved.relative_to(root_path)
+ except (OSError, RuntimeError, ValueError) as exc:
+ raise ValueError(f"Path '{key}' resolves outside workspace root") from exc
+ return resolved
+
+
def normalize_virtual_path(path: str) -> str:
"""Normalize user paths to canonical workspace-relative virtual paths."""
from services.plugin import NodeUserError
@@ -611,16 +672,7 @@ class WorkspaceBackend:
return self._sandbox_id
def _resolve_path(self, key: str) -> Path:
- virtual = _validate_virtual_path(key)
- candidate = self.cwd / virtual.lstrip("/")
- try:
- resolved = candidate.resolve(strict=False)
- resolved.relative_to(self.cwd)
- except (OSError, RuntimeError, ValueError) as exc:
- raise ValueError(
- f"Path '{key}' resolves outside workspace root"
- ) from exc
- return resolved
+ return resolve_within(self.cwd, key)
def _to_virtual_path(self, path: Path) -> str:
resolved = path.resolve(strict=False)
diff --git a/server/pyproject.toml b/server/pyproject.toml
index a885eb10..fd088a97 100644
--- a/server/pyproject.toml
+++ b/server/pyproject.toml
@@ -40,6 +40,16 @@ dependencies = [
# https://temporal.download/cli/archive/latest).
"pooch>=1.8.2",
+ # Media metadata (services/media/inspect.py). Pure Python, zero native
+ # deps, MIT. Chosen over mutagen (GPL-2.0, incompatible with this
+ # project's MIT licence) and over pydub / ffprobe, which both require
+ # an ffmpeg binary on PATH that nothing installs on Windows.
+ "tinytag>=2.0.0",
+ # Multipart parsing for the workspace upload route. Already present
+ # transitively via mcp; declared explicitly so it survives that
+ # dependency changing.
+ "python-multipart>=0.0.20",
+
# Auth
"bcrypt>=4.2.0",
# pyjwt replaced python-jose (2026-06): python-jose drags in python-ecdsa,
diff --git a/server/services/llm/registry.py b/server/services/llm/registry.py
index 8fc47b7f..c32fb76a 100644
--- a/server/services/llm/registry.py
+++ b/server/services/llm/registry.py
@@ -5,141 +5,69 @@ calls ``register_provider(ProviderSpec(...))`` at top level, and the unifier
reads this registry to dispatch chat / fetch_models calls and to translate
typed SDK exceptions into ``NodeUserError``. No per-provider Python lives
inside ``services/ai.py`` after this layer is wired.
+
+The mechanism itself now lives in :mod:`services.provider_registry`, shared
+with ``services/speech`` — this module is the LLM-specific instance plus the
+free-function API that provider modules and tests already import. The public
+surface (``ProviderSpec``, ``register_provider``, ``get_provider``,
+``all_providers``, ``has_provider``, ``_reset_for_tests``) is unchanged.
"""
from __future__ import annotations
-import functools
-import pkgutil
-from dataclasses import dataclass, field
-from typing import Any, Callable, Dict, List, Tuple, Type
+from typing import List
-from core.logging import get_logger
from services.llm.protocol import LLMProvider
-from services.plugin import NodeUserError
-
-logger = get_logger(__name__)
-
-
[email protected]_cache(maxsize=None)
-def _resolve_exc(ref: str) -> Type[BaseException]:
- """Resolve a ``"module:ClassName"`` ref to the exception class.
-
- Called lazily from ``ProviderSpec.sdk_exception_types`` — by the time
- anything needs the resolved classes (an ``except`` clause matching a
- raised SDK error, or a test reading the tuple), the provider factory
- has already imported the SDK to build its client, so this is a
- ``sys.modules`` cache hit, never a cold import on the boot path.
- """
- obj = pkgutil.resolve_name(ref)
- if not (isinstance(obj, type) and issubclass(obj, BaseException)):
- raise TypeError(f"{ref!r} did not resolve to an exception class: {obj!r}")
- return obj
-
-
-@dataclass(frozen=True)
-class ProviderSpec:
- """Declarative spec for a chat-model provider plugin.
-
- Fields:
- name: registry key — matches the ``provider`` string used by
- ai.py, the chat-model node ``provider`` field, and the
- ``providers.<name>`` block in ``llm_defaults.json``.
- factory: callable returning an ``LLMProvider`` instance.
- Invoked as ``factory(api_key=..., proxy_url=..., **client_kwargs)``.
- sdk_exception_refs: tuple of ``"module:ClassName"`` refs (the
- packaging entry-points format, resolved via
- ``pkgutil.resolve_name``) naming the typed error classes
- raised by this provider's SDK. Refs — not classes — so that
- registering a provider at boot never imports its SDK
- (docs-internal/performance.md anti-pattern; cost ~7s warm /
- ~45s cold across openai + anthropic + google.genai). The
- resolved classes are exposed by ``sdk_exception_types``.
- client_kwargs: static keyword arguments passed to ``factory`` on
- every instantiation. Used by OpenAI-compatible providers
- (xai / deepseek / kimi / mistral / ollama / lmstudio / groq /
- cerebras) to pin ``base_url`` from JSON config.
- """
-
- name: str
- factory: Callable[..., LLMProvider]
- sdk_exception_refs: Tuple[str, ...]
- client_kwargs: Dict[str, Any] = field(default_factory=dict)
-
- @property
- def sdk_exception_types(self) -> Tuple[Type[BaseException], ...]:
- """Resolved typed SDK error classes.
-
- The unifier wraps a single ``except spec.sdk_exception_types``
- around every call site so user-correctable errors surface as
- ``NodeUserError`` without a per-provider branch in ai.py.
- """
- return tuple(_resolve_exc(ref) for ref in self.sdk_exception_refs)
-
-
-_REGISTRY: Dict[str, ProviderSpec] = {}
+from services.provider_registry import (
+ ProviderRegistry,
+ ProviderSpec as _ProviderSpec,
+ resolve_exception_ref as _resolve_exc, # noqa: F401 — re-exported for tests
+)
+
+# Re-exported under the historical name. Generic in the provider type, so
+# ``ProviderSpec(factory=...)`` still type-checks against ``LLMProvider``.
+ProviderSpec = _ProviderSpec[LLMProvider]
+
+_LLM_REGISTRY: ProviderRegistry[LLMProvider] = ProviderRegistry("LLM")
+
+# Compatibility alias bound to the SAME dict object the registry mutates.
+# Several tests swap a provider's factory in place via ``_REGISTRY[name] = spec``
+# to exercise typed-error translation and client-cache policy without touching
+# the real SDKs. Keeping the name (and the identity) means this extraction is
+# invisible to them — the success criterion for the refactor was that
+# ``tests/llm/`` passes untouched.
+_REGISTRY = _LLM_REGISTRY._specs
def register_provider(spec: ProviderSpec) -> None:
- """Register a provider plugin.
-
- Called from each provider module's top level on import. The unifier
- expects every registered provider to declare at least one typed SDK
- exception so the wrapper try/except can translate it.
-
- Re-registration with an identical spec is a no-op (handles uvicorn
- reload cycles). Re-registration with a different spec raises so
- accidental shadowing is loud.
- """
- # Check the raw refs, NOT the resolving property — reading the
- # property here would import every SDK at registration time and
- # defeat the lazy-ref design.
- if not spec.sdk_exception_refs:
- raise ValueError(
- f"ProviderSpec({spec.name!r}) declares empty sdk_exception_refs. "
- "Every provider must surface its typed SDK error class so the "
- "unifier can translate it into NodeUserError."
- )
- existing = _REGISTRY.get(spec.name)
- if existing is not None and existing != spec:
- raise ValueError(
- f"Provider {spec.name!r} already registered with a different spec. "
- f"Existing: {existing!r}; new: {spec!r}."
- )
- _REGISTRY[spec.name] = spec
- logger.debug("registered LLM provider", provider=spec.name)
+ """Register a provider plugin. Idempotent for identical specs."""
+ _LLM_REGISTRY.register(spec)
def get_provider(name: str) -> ProviderSpec:
- """Look up a provider spec by name.
-
- Raises ``NodeUserError`` on unknown providers so the failure surfaces
- cleanly through ``BaseNode.execute()`` as a single WARN line — same
- contract as every other user-correctable error in the framework.
- """
- spec = _REGISTRY.get(name)
- if spec is None:
- raise NodeUserError(
- f"Unknown LLM provider: {name!r}. "
- f"Registered providers: {sorted(_REGISTRY)}"
- )
- return spec
+ """Look up a provider spec by name; raises ``NodeUserError`` on a miss."""
+ return _LLM_REGISTRY.get(name)
def all_providers() -> List[str]:
"""Return sorted list of registered provider names."""
- return sorted(_REGISTRY)
+ return _LLM_REGISTRY.all()
def has_provider(name: str) -> bool:
"""Cheap membership probe — does not raise on miss."""
- return name in _REGISTRY
+ return _LLM_REGISTRY.has(name)
def _reset_for_tests() -> None:
- """Test-only — drop every registered provider.
+ """Test-only — drop every registered provider."""
+ _LLM_REGISTRY.reset_for_tests()
+
- Lets unit tests around the registry control the state instead of
- inheriting the production population. Never call from runtime code.
- """
- _REGISTRY.clear()
+__all__ = [
+ "ProviderSpec",
+ "all_providers",
+ "get_provider",
+ "has_provider",
+ "register_provider",
+]
diff --git a/server/services/media/__init__.py b/server/services/media/__init__.py
new file mode 100644
index 00000000..3752ee3e
--- /dev/null
+++ b/server/services/media/__init__.py
@@ -0,0 +1,50 @@
+"""Media transport — references, not bytes.
+
+Vendor-neutral and deliberately not part of ``services/speech``: the same
+envelope is consumed by the file widget, the workspace HTTP routes, and
+any node that produces or accepts a file. Owning it from a provider
+package would invert the dependency.
+
+The contract in one line: **a node returns an**
+:class:`~services.media.refs.AudioRef`, **never audio bytes.** See
+:mod:`services.media.limits` for the measured reason.
+"""
+
+from services.media.inspect import AudioProbe, inspect_audio
+from services.media.limits import (
+ MEDIA_MAX_AUDIO_SECONDS,
+ MEDIA_MAX_READ_BYTES,
+ MEDIA_MAX_UPLOAD_BYTES,
+ TEMPORAL_PAYLOAD_ERROR_BYTES,
+ TEMPORAL_PAYLOAD_WARN_BYTES,
+)
+from services.media.refs import AudioRef
+from services.media.workspace import (
+ AUDIO_SUBDIR,
+ UPLOAD_SUBDIR,
+ coerce_file_param,
+ read_media_bytes,
+ resolve_media,
+ workspace_file_url,
+ workspace_root,
+ write_audio,
+)
+
+__all__ = [
+ "AUDIO_SUBDIR",
+ "UPLOAD_SUBDIR",
+ "AudioProbe",
+ "AudioRef",
+ "MEDIA_MAX_AUDIO_SECONDS",
+ "MEDIA_MAX_READ_BYTES",
+ "MEDIA_MAX_UPLOAD_BYTES",
+ "TEMPORAL_PAYLOAD_ERROR_BYTES",
+ "TEMPORAL_PAYLOAD_WARN_BYTES",
+ "coerce_file_param",
+ "inspect_audio",
+ "read_media_bytes",
+ "resolve_media",
+ "workspace_file_url",
+ "workspace_root",
+ "write_audio",
+]
diff --git a/server/services/media/inspect.py b/server/services/media/inspect.py
new file mode 100644
index 00000000..4d5580de
--- /dev/null
+++ b/server/services/media/inspect.py
@@ -0,0 +1,158 @@
+"""Audio metadata probing — duration, sample rate, channels.
+
+Three fallbacks, and a hard rule: **this never raises and never fails a
+workflow.** An unknown container returns an empty probe, logs at DEBUG,
+and the caller still gets a valid :class:`~services.media.AudioRef`.
+
+The rule matters because the alternative is worse in exactly the wrong
+direction. Degrading a *billing estimate* when a codec variant confuses
+the parser is acceptable; hard-failing a file the provider would have
+accepted is not.
+
+``tinytag`` was chosen over the alternatives on two filters. License:
+``mutagen`` is GPL-2.0 and this repository is MIT, which is a legal
+decision rather than a technical one. Portability: ``pydub`` and
+``ffprobe`` both require an ffmpeg binary on PATH, and this repository is
+developed on Windows where nothing installs one.
+"""
+
+from __future__ import annotations
+
+import wave
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional
+
+from core.logging import get_logger
+
+logger = get_logger(__name__)
+
+# Bytes per sample frame for the raw, header-less encodings a TTS provider
+# may hand back. Used only when the caller tells us the sample rate --
+# there is nothing in the file itself to read.
+_PCM_BYTES_PER_SAMPLE = {
+ "pcm": 2,
+ "linear16": 2,
+ "pcm_s16le": 2,
+ "pcm_l16": 2,
+ "pcm_raw": 2,
+ "mulaw": 1,
+ "ulaw": 1,
+ "alaw": 1,
+}
+
+
+@dataclass(frozen=True)
+class AudioProbe:
+ """What we managed to learn. Every field may legitimately be None."""
+
+ duration_seconds: Optional[float] = None
+ sample_rate: Optional[int] = None
+ channels: Optional[int] = None
+ format: Optional[str] = None
+
+
+def inspect_audio(
+ path: Path | str,
+ *,
+ declared_format: Optional[str] = None,
+ pcm_sample_rate: Optional[int] = None,
+ pcm_channels: int = 1,
+) -> AudioProbe:
+ """Best-effort metadata for an audio file. Never raises."""
+ target = Path(path)
+
+ probe = _probe_tinytag(target)
+ if probe.duration_seconds is not None:
+ return probe
+
+ probe = _probe_wave(target)
+ if probe.duration_seconds is not None:
+ return probe
+
+ probe = _probe_raw_pcm(
+ target,
+ declared_format=declared_format,
+ sample_rate=pcm_sample_rate,
+ channels=pcm_channels,
+ )
+ if probe.duration_seconds is not None:
+ return probe
+
+ logger.debug(
+ "audio metadata unavailable",
+ path=str(target),
+ declared_format=declared_format,
+ )
+ return AudioProbe(format=declared_format or None)
+
+
+def _probe_tinytag(target: Path) -> AudioProbe:
+ """mp3 / m4a / flac / ogg / opus / wav / aiff / wma."""
+ try:
+ from tinytag import TinyTag
+
+ tag = TinyTag.get(str(target))
+ except Exception as exc: # unsupported container, corrupt header, missing dep
+ logger.debug("tinytag declined", path=str(target), error=str(exc))
+ return AudioProbe()
+
+ duration = getattr(tag, "duration", None)
+ return AudioProbe(
+ duration_seconds=float(duration) if duration else None,
+ sample_rate=getattr(tag, "samplerate", None),
+ channels=getattr(tag, "channels", None),
+ format=target.suffix.lstrip(".").lower() or None,
+ )
+
+
+def _probe_wave(target: Path) -> AudioProbe:
+ """stdlib fallback for WAV variants tinytag declines."""
+ try:
+ with wave.open(str(target), "rb") as handle:
+ frames = handle.getnframes()
+ rate = handle.getframerate()
+ channels = handle.getnchannels()
+ except Exception as exc:
+ logger.debug("wave declined", path=str(target), error=str(exc))
+ return AudioProbe()
+
+ if not rate:
+ return AudioProbe()
+ return AudioProbe(
+ duration_seconds=frames / float(rate),
+ sample_rate=rate,
+ channels=channels,
+ format="wav",
+ )
+
+
+def _probe_raw_pcm(
+ target: Path,
+ *,
+ declared_format: Optional[str],
+ sample_rate: Optional[int],
+ channels: int,
+) -> AudioProbe:
+ """Header-less PCM: arithmetic, only when the caller knows the rate."""
+ if not declared_format or not sample_rate:
+ return AudioProbe()
+ width = _PCM_BYTES_PER_SAMPLE.get(declared_format.lower())
+ if not width:
+ return AudioProbe()
+ try:
+ size = target.stat().st_size
+ except OSError:
+ return AudioProbe()
+ divisor = sample_rate * width * max(1, channels)
+ if divisor <= 0:
+ return AudioProbe()
+ return AudioProbe(
+ duration_seconds=size / divisor,
+ sample_rate=sample_rate,
+ channels=channels,
+ format=declared_format.lower(),
+ )
+
+
+__all__ = ["AudioProbe", "inspect_audio"]
diff --git a/server/services/media/limits.py b/server/services/media/limits.py
new file mode 100644
index 00000000..4dfdc130
--- /dev/null
+++ b/server/services/media/limits.py
@@ -0,0 +1,49 @@
+"""Every media size constant, in one place.
+
+These exist because the execution engine multiplies whatever a node
+returns. One payload is copied at least six ways: ``_serialize_result``
+-> ``node_outputs`` x3 -> WebSocket broadcast x2 -> the status cache
+(retained for the process lifetime) -> the Temporal activity result ->
+``MachinaWorkflow``'s aggregate -> every downstream activity input ->
+and, for a ``usable_as_tool`` node, verbatim into an LLM message.
+
+Temporal's blob limits are the binding constraint. They are server
+defaults; nothing in this repo configures a ``DataConverter`` or a
+payload codec, so they apply as-is:
+
+ warning 524_288 B -> PayloadSizeWarning (not filtered, so visible)
+ error 2_097_152 B -> _PayloadSizeError, and the activity RETRIES
+ three times before reporting a generic failure
+
+That is why :class:`services.media.AudioRef` carries no bytes at all.
+"""
+
+from __future__ import annotations
+
+# Temporal's own thresholds, restated so call sites can reason about
+# headroom without importing temporalio.
+TEMPORAL_PAYLOAD_WARN_BYTES = 524_288
+TEMPORAL_PAYLOAD_ERROR_BYTES = 2_097_152
+
+# Largest upload accepted by the workspace upload route. Deliberately far
+# above the Temporal limit: uploaded bytes land on disk and only a ~400 B
+# reference ever enters a payload.
+MEDIA_MAX_UPLOAD_BYTES = 25 * 1024 * 1024
+
+# Largest file a node will pull back off disk into memory to hand to a
+# provider. Bounds one activity's peak RSS; the bytes are discarded as
+# soon as the HTTP call returns.
+MEDIA_MAX_READ_BYTES = 25 * 1024 * 1024
+
+# Longest clip accepted when the duration is actually known. Unknown
+# duration falls through to the byte cap rather than failing -- see
+# services.media.inspect for why inspection must never be load-bearing.
+MEDIA_MAX_AUDIO_SECONDS = 60 * 60
+
+__all__ = [
+ "MEDIA_MAX_AUDIO_SECONDS",
+ "MEDIA_MAX_READ_BYTES",
+ "MEDIA_MAX_UPLOAD_BYTES",
+ "TEMPORAL_PAYLOAD_ERROR_BYTES",
+ "TEMPORAL_PAYLOAD_WARN_BYTES",
+]
diff --git a/server/services/media/refs.py b/server/services/media/refs.py
new file mode 100644
index 00000000..fa6d7fb6
--- /dev/null
+++ b/server/services/media/refs.py
@@ -0,0 +1,62 @@
+"""``AudioRef`` — a reference to audio, never the audio itself.
+
+The one rule this module exists to enforce: **audio bytes do not travel
+through the workflow engine.** See :mod:`services.media.limits` for the
+measured reason. This model has no bytes field, no base64 field, and
+``extra="forbid"`` so that adding one is a validation error rather than a
+silent regression.
+"""
+
+from __future__ import annotations
+
+from typing import Literal, Optional
+
+from pydantic import BaseModel, ConfigDict, Field
+
+
+class AudioRef(BaseModel):
+ """A pointer to an audio file inside a workflow workspace.
+
+ Serializes to roughly 400 bytes, i.e. about 5,200 refs before
+ approaching Temporal's 2 MiB error limit -- the envelope is
+ structurally incapable of getting near it.
+
+ ``path`` is workspace-**relative** POSIX, never an absolute host path.
+ Absolute paths embed the mutable workflow slug, leak the operator's
+ home directory into the database / WebSocket broadcasts / LLM context,
+ and cannot be safely turned into an HTTP URL.
+ """
+
+ kind: Literal["audio"] = "audio"
+
+ path: str = Field(
+ description="Workspace-relative POSIX path, no leading slash "
+ "(e.g. 'audio/greeting-1a2b3c.wav').",
+ )
+ # Stable across renames -- the workspace directory is keyed on the
+ # slug, which changes, while the id does not. Lets the resolver and
+ # the file-serving route find the workspace without a NodeContext.
+ workflow_id: Optional[str] = None
+
+ filename: str = Field(description="Display name. Never used for resolution.")
+ mime_type: str = "application/octet-stream"
+ format: str = Field(default="", description="Container/codec: wav, mp3, opus, ...")
+ size_bytes: int = 0
+
+ # None whenever the container could not be inspected. Never guessed --
+ # a fabricated duration would silently mis-bill per-second providers.
+ duration_seconds: Optional[float] = None
+ sample_rate: Optional[int] = None
+ channels: Optional[int] = None
+
+ sha256: Optional[str] = None
+
+ # Path-only, no scheme or host, so the frontend can prepend its own
+ # base via buildApiUrl() when it points at a remote backend.
+ # Advisory: `path` + `workflow_id` remain canonical.
+ url: Optional[str] = None
+
+ model_config = ConfigDict(extra="forbid")
+
+
+__all__ = ["AudioRef"]
diff --git a/server/services/media/workspace.py b/server/services/media/workspace.py
new file mode 100644
index 00000000..3bb2ee38
--- /dev/null
+++ b/server/services/media/workspace.py
@@ -0,0 +1,287 @@
+"""Reading and writing media inside a workflow workspace.
+
+Everything here funnels through :func:`nodes.filesystem._backend.resolve_within`,
+which rejects ``..`` / ``~`` / drive-prefixed inputs before touching the
+filesystem and then re-checks containment after resolution so a symlink or
+Windows junction cannot redirect the result outside the workspace.
+
+That containment is not decorative. Before this module existed, the Sarvam
+speech-to-text node joined a user-supplied path onto the workspace root with
+no check at all, so ``audio_file="../../credentials.db"`` read the encrypted
+credential store and uploaded it to the provider. :func:`coerce_file_param`
+closes that by construction, for every node that adopts it.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import mimetypes
+import re
+from pathlib import Path
+from typing import Any, Optional, Tuple
+from uuid import uuid4
+
+from core.logging import get_logger
+from services.media.inspect import inspect_audio
+from services.media.limits import MEDIA_MAX_READ_BYTES
+from services.media.refs import AudioRef
+
+logger = get_logger(__name__)
+
+AUDIO_SUBDIR = "audio"
+UPLOAD_SUBDIR = "uploads"
+
+_SLUG_RE = re.compile(r"[^a-z0-9]+")
+
+# Windows treats these as devices no matter the extension, so a write to
+# "CON.wav" is a write to the console. Every generated name carries a uuid
+# suffix, which makes a bare reserved name unreachable, but the check stays
+# as defence for callers that pass an unusual stem.
+_WINDOWS_RESERVED = {
+ "con", "prn", "aux", "nul",
+ *(f"com{i}" for i in range(1, 10)),
+ *(f"lpt{i}" for i in range(1, 10)),
+}
+
+
+def _slugify(value: str, *, limit: int = 32) -> str:
+ slug = _SLUG_RE.sub("-", (value or "").lower()).strip("-")[:limit].strip("-")
+ if not slug or slug in _WINDOWS_RESERVED:
+ return "audio"
+ return slug
+
+
+def workspace_root(ctx: Any = None, *, workflow_id: Optional[str] = None) -> Path:
+ """Resolve the workspace directory for a context or workflow id.
+
+ Prefers the executor-injected ``ctx.workspace_dir``; falls back to
+ composing it from the workflow's slug so the HTTP routes -- which have
+ no NodeContext -- can resolve the same directory.
+ """
+ from services.plugin import NodeUserError
+
+ direct = getattr(ctx, "workspace_dir", None) if ctx is not None else None
+ if not direct and ctx is not None:
+ raw = getattr(ctx, "raw", None)
+ if isinstance(raw, dict):
+ direct = raw.get("workspace_dir")
+ if direct:
+ return Path(direct)
+
+ if workflow_id is None and ctx is not None:
+ workflow_id = getattr(ctx, "workflow_id", None)
+ if not workflow_id:
+ raise NodeUserError(
+ "No workspace is available for this execution, so media cannot be "
+ "read or written. Save the workflow and run it again."
+ )
+ from core.paths import workspace_dir as _workspace_dir
+
+ return Path(_workspace_dir(workflow_id))
+
+
+def resolve_media(
+ ref: AudioRef | str,
+ *,
+ ctx: Any = None,
+ workflow_id: Optional[str] = None,
+) -> Path:
+ """Resolve a ref or a path string to a contained absolute path."""
+ from nodes.filesystem._backend import resolve_within
+ from services.plugin import NodeUserError
+
+ key = ref.path if isinstance(ref, AudioRef) else str(ref or "")
+ if not key.strip():
+ raise NodeUserError("No media path was provided.")
+
+ if isinstance(ref, AudioRef) and ref.workflow_id and workflow_id is None:
+ workflow_id = ref.workflow_id
+
+ candidate = Path(key)
+ if candidate.is_absolute():
+ # Tolerated for back-compat with nodes that stored absolute paths
+ # before AudioRef existed, but still contained: an absolute path
+ # outside the workspace is refused rather than read.
+ root = workspace_root(ctx, workflow_id=workflow_id)
+ try:
+ key = candidate.resolve(strict=False).relative_to(root).as_posix()
+ except ValueError as exc:
+ raise NodeUserError(
+ f"'{candidate}' is outside this workflow's workspace."
+ ) from exc
+ else:
+ root = workspace_root(ctx, workflow_id=workflow_id)
+
+ try:
+ return resolve_within(root, key)
+ except ValueError as exc:
+ raise NodeUserError(f"'{key}' is outside this workflow's workspace.") from exc
+
+
+def read_media_bytes(
+ ref: AudioRef | str,
+ *,
+ ctx: Any = None,
+ workflow_id: Optional[str] = None,
+ max_bytes: int = MEDIA_MAX_READ_BYTES,
+) -> Tuple[str, bytes]:
+ """Return ``(filename, bytes)`` for a contained media reference."""
+ from services.plugin import NodeUserError
+
+ target = resolve_media(ref, ctx=ctx, workflow_id=workflow_id)
+ if not target.is_file():
+ raise NodeUserError(f"Media file not found: {target.name}")
+
+ size = target.stat().st_size
+ if size == 0:
+ raise NodeUserError(f"Media file is empty: {target.name}")
+ if size > max_bytes:
+ raise NodeUserError(
+ f"{target.name} is {size // (1024 * 1024)} MB; the limit is "
+ f"{max_bytes // (1024 * 1024)} MB."
+ )
+ return target.name, target.read_bytes()
+
+
+def write_audio(
+ payload: bytes,
+ *,
+ ctx: Any,
+ stem: str,
+ ext: str,
+ mime_type: Optional[str] = None,
+ subdir: str = AUDIO_SUBDIR,
+ inspect: bool = True,
+ sample_rate: Optional[int] = None,
+ channels: int = 1,
+) -> AudioRef:
+ """Atomically write audio into the workspace and return a reference.
+
+ The filename shape (``<stem>-<node8>-<rand6>.<ext>``) matches what the
+ Sarvam TTS node produced before this helper existed, so porting it is
+ behaviour-preserving. The random suffix also means retries and repeated
+ runs never collide or silently overwrite earlier audio.
+ """
+ from nodes.filesystem._backend import atomic_write_bytes
+ from services.plugin import NodeUserError
+
+ if not payload:
+ raise NodeUserError("Refusing to write an empty audio file.")
+
+ root = workspace_root(ctx)
+ node_id = str(getattr(ctx, "node_id", "") or "node")
+ name = f"{_slugify(stem)}-{node_id[:8]}-{uuid4().hex[:6]}.{ext.lstrip('.')}"
+ rel = f"{subdir}/{name}"
+
+ target = resolve_media(rel, ctx=ctx)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ atomic_write_bytes(target, payload, root_dir=root)
+
+ probe = (
+ inspect_audio(
+ target,
+ declared_format=ext,
+ pcm_sample_rate=sample_rate,
+ pcm_channels=channels,
+ )
+ if inspect
+ else None
+ )
+ workflow_id = getattr(ctx, "workflow_id", None)
+
+ return AudioRef(
+ path=rel,
+ workflow_id=workflow_id,
+ filename=name,
+ mime_type=mime_type or mimetypes.guess_type(name)[0] or "application/octet-stream",
+ format=ext.lstrip(".").lower(),
+ size_bytes=len(payload),
+ duration_seconds=probe.duration_seconds if probe else None,
+ sample_rate=(probe.sample_rate if probe else None) or sample_rate,
+ channels=(probe.channels if probe else None),
+ sha256=hashlib.sha256(payload).hexdigest(),
+ url=workspace_file_url(workflow_id, rel),
+ )
+
+
+def workspace_file_url(workflow_id: Optional[str], rel_path: str) -> Optional[str]:
+ """Path-only URL for the workspace file route. No scheme, no host."""
+ if not workflow_id:
+ return None
+ from urllib.parse import quote
+
+ return f"/api/workspace/{quote(str(workflow_id))}/files/{quote(rel_path)}"
+
+
+def coerce_file_param(
+ value: Any,
+ *,
+ ctx: Any = None,
+ max_bytes: int = MEDIA_MAX_READ_BYTES,
+) -> Tuple[str, bytes]:
+ """Read a file parameter in any of the three shapes the UI can produce.
+
+ 1. A serialized :class:`AudioRef` -- what the upload route now returns.
+ 2. The legacy ``{"type": "upload", "data": "<base64>"}`` envelope the
+ file widget used to emit. Still accepted indefinitely: saved
+ workflow rows carry it and are not migrated. Logs one warning
+ naming the node so operators can find and re-save them.
+ 3. A bare path string, typed by the user or dragged from an upstream
+ node -- now resolved with containment.
+ """
+ import base64
+ import binascii
+
+ from services.plugin import NodeUserError
+
+ if isinstance(value, dict) and value.get("kind") == "audio":
+ ref = AudioRef.model_validate(value)
+ return read_media_bytes(ref, ctx=ctx, max_bytes=max_bytes)
+
+ if isinstance(value, dict) and value.get("type") == "upload":
+ data = value.get("data") or ""
+ if not data:
+ raise NodeUserError(
+ "The uploaded file carried no data. Re-select it in the "
+ "parameter panel."
+ )
+ logger.warning(
+ "legacy base64 upload parameter; re-select the file to migrate it",
+ node_id=str(getattr(ctx, "node_id", "") or "unknown"),
+ characters=len(data),
+ )
+ try:
+ blob = base64.b64decode(data, validate=True)
+ except (binascii.Error, ValueError) as exc:
+ raise NodeUserError(f"Uploaded file is not valid base64: {exc}") from exc
+ if len(blob) > max_bytes:
+ raise NodeUserError(
+ f"Uploaded file is {len(blob) // (1024 * 1024)} MB; the limit "
+ f"is {max_bytes // (1024 * 1024)} MB."
+ )
+ if not blob:
+ raise NodeUserError("Uploaded file is empty.")
+ return str(value.get("filename") or "upload.bin"), blob
+
+ if isinstance(value, dict):
+ raise NodeUserError(
+ "File parameter is an object but not a recognised file reference. "
+ "Re-select the file in the parameter panel."
+ )
+
+ text = str(value or "").strip()
+ if not text:
+ raise NodeUserError("No file was provided.")
+ return read_media_bytes(text, ctx=ctx, max_bytes=max_bytes)
+
+
+__all__ = [
+ "AUDIO_SUBDIR",
+ "UPLOAD_SUBDIR",
+ "coerce_file_param",
+ "read_media_bytes",
+ "resolve_media",
+ "workspace_file_url",
+ "workspace_root",
+ "write_audio",
+]
diff --git a/server/services/provider_registry.py b/server/services/provider_registry.py
new file mode 100644
index 00000000..477b8825
--- /dev/null
+++ b/server/services/provider_registry.py
@@ -0,0 +1,139 @@
+"""Generic provider-plugin registry, shared by every provider layer.
+
+Extracted from ``services/llm/registry.py`` when ``services/speech`` needed
+the same machinery. Nothing here knows what a provider *does* — it holds a
+name, a factory, and the typed SDK exceptions that layer's dispatcher must
+translate. ``services/llm`` and ``services/speech`` each own a
+:class:`ProviderRegistry` instance; speech owns two, one per direction.
+
+The one subtle piece is ``sdk_exception_refs``. They are
+``"module:ClassName"`` strings rather than classes so that *registering* a
+provider never imports its SDK. That is load-bearing for startup time:
+eagerly importing openai + anthropic + google.genai at registration cost
+roughly 7 s warm and 45 s cold, and is recorded as an anti-pattern in
+docs-internal/performance.md. By the time anything reads the resolved
+classes, the factory has already built a client, so resolution is a
+``sys.modules`` hit.
+"""
+
+from __future__ import annotations
+
+import functools
+import pkgutil
+from dataclasses import dataclass, field
+from typing import Any, Callable, Dict, Generic, List, Tuple, Type, TypeVar
+
+from core.logging import get_logger
+from services.plugin import NodeUserError
+
+logger = get_logger(__name__)
+
+ProviderT = TypeVar("ProviderT")
+
+
[email protected]_cache(maxsize=None)
+def resolve_exception_ref(ref: str) -> Type[BaseException]:
+ """Resolve a ``"module:ClassName"`` ref to the exception class."""
+ obj = pkgutil.resolve_name(ref)
+ if not (isinstance(obj, type) and issubclass(obj, BaseException)):
+ raise TypeError(f"{ref!r} did not resolve to an exception class: {obj!r}")
+ return obj
+
+
+@dataclass(frozen=True)
+class ProviderSpec(Generic[ProviderT]):
+ """Declarative spec for one provider plugin.
+
+ Fields:
+ name: registry key — matches the ``provider`` string used by nodes
+ and the ``providers.<name>`` block in the layer's JSON config.
+ factory: callable returning a provider instance, invoked as
+ ``factory(api_key=..., **client_kwargs)``.
+ sdk_exception_refs: ``"module:ClassName"`` refs naming the typed
+ errors this provider's SDK raises. Refs, not classes — see the
+ module docstring.
+ client_kwargs: static keyword arguments merged into every
+ instantiation. How OpenAI-compatible providers pin ``base_url``
+ from JSON instead of from Python.
+ """
+
+ name: str
+ factory: Callable[..., ProviderT]
+ sdk_exception_refs: Tuple[str, ...]
+ client_kwargs: Dict[str, Any] = field(default_factory=dict)
+
+ @property
+ def sdk_exception_types(self) -> Tuple[Type[BaseException], ...]:
+ """Resolved typed SDK error classes, for one ``except`` clause."""
+ return tuple(resolve_exception_ref(ref) for ref in self.sdk_exception_refs)
+
+
+class ProviderRegistry(Generic[ProviderT]):
+ """One registry per provider layer and direction.
+
+ ``label`` appears in log lines and error messages, so it should read
+ naturally in "Unknown {label} provider: 'x'".
+ """
+
+ def __init__(self, label: str) -> None:
+ self._label = label
+ self._specs: Dict[str, ProviderSpec[ProviderT]] = {}
+
+ @property
+ def label(self) -> str:
+ return self._label
+
+ def register(self, spec: ProviderSpec[ProviderT]) -> None:
+ """Register a provider. Idempotent for identical specs.
+
+ Re-registration with an identical spec is a no-op so uvicorn reload
+ cycles are harmless; re-registration with a *different* spec raises
+ so accidental shadowing is loud rather than silent.
+ """
+ # Check the raw refs, NOT the resolving property — reading the
+ # property here would import every SDK at registration time and
+ # defeat the entire lazy-ref design.
+ if not spec.sdk_exception_refs:
+ raise ValueError(
+ f"ProviderSpec({spec.name!r}) declares empty sdk_exception_refs. "
+ "Every provider must surface its typed SDK error class so the "
+ "dispatcher can translate it into NodeUserError."
+ )
+ existing = self._specs.get(spec.name)
+ if existing is not None and existing != spec:
+ raise ValueError(
+ f"{self._label} provider {spec.name!r} already registered with a "
+ f"different spec. Existing: {existing!r}; new: {spec!r}."
+ )
+ self._specs[spec.name] = spec
+ logger.debug("registered provider", layer=self._label, provider=spec.name)
+
+ def get(self, name: str) -> ProviderSpec[ProviderT]:
+ """Look up a spec, raising ``NodeUserError`` on a miss.
+
+ ``NodeUserError`` rather than ``KeyError`` so an unknown provider
+ surfaces through ``BaseNode.execute()`` as one WARN line with no
+ traceback — the same contract as every other user-correctable
+ failure in the framework.
+ """
+ spec = self._specs.get(name)
+ if spec is None:
+ raise NodeUserError(
+ f"Unknown {self._label} provider: {name!r}. "
+ f"Registered providers: {sorted(self._specs)}"
+ )
+ return spec
+
+ def all(self) -> List[str]:
+ return sorted(self._specs)
+
+ def has(self, name: str) -> bool:
+ """Cheap membership probe — does not raise."""
+ return name in self._specs
+
+ def reset_for_tests(self) -> None:
+ """Test-only. Never call from runtime code."""
+ self._specs.clear()
+
+
+__all__ = ["ProviderRegistry", "ProviderSpec", "resolve_exception_ref"]
diff --git a/server/tests/media/__init__.py b/server/tests/media/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/server/tests/media/test_audio_ref.py b/server/tests/media/test_audio_ref.py
new file mode 100644
index 00000000..a3a266bc
--- /dev/null
+++ b/server/tests/media/test_audio_ref.py
@@ -0,0 +1,226 @@
+"""Contract for services.media — references, never bytes.
+
+The invariants here are not stylistic. Audio bytes inside a node result
+are copied at least six ways by the engine (see services/media/limits.py),
+and Temporal's 2 MiB blob limit turns that into a three-retry failure with
+a generic error message. ``AudioRef`` is the type that makes it impossible.
+"""
+
+from __future__ import annotations
+
+import base64
+import struct
+import wave
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+from pydantic import ValidationError
+
+from services.media import (
+ AudioRef,
+ coerce_file_param,
+ inspect_audio,
+ read_media_bytes,
+ resolve_media,
+ write_audio,
+)
+from services.media.limits import TEMPORAL_PAYLOAD_WARN_BYTES
+
+
+pytestmark = pytest.mark.unit
+
+
+def _ctx(tmp_path: Path, *, node_id: str = "tts-node-1", workflow_id: str = "wf-1"):
+ return SimpleNamespace(
+ node_id=node_id,
+ workflow_id=workflow_id,
+ workspace_dir=str(tmp_path),
+ raw={"workspace_dir": str(tmp_path)},
+ )
+
+
+def _wav_bytes(seconds: float = 0.25, rate: int = 8000) -> bytes:
+ import io
+
+ buffer = io.BytesIO()
+ with wave.open(buffer, "wb") as handle:
+ handle.setnchannels(1)
+ handle.setsampwidth(2)
+ handle.setframerate(rate)
+ handle.writeframes(struct.pack("<h", 0) * int(rate * seconds))
+ return buffer.getvalue()
+
+
+class TestAudioRefCannotCarryBytes:
+ """D1 is structural, not a convention someone must remember."""
+
+ def test_has_no_bytes_or_base64_field(self):
+ forbidden = {"data", "bytes", "audio", "base64", "audio_base64", "content"}
+ assert forbidden.isdisjoint(AudioRef.model_fields)
+
+ def test_rejects_an_injected_payload_field(self):
+ with pytest.raises(ValidationError):
+ AudioRef(path="audio/a.wav", filename="a.wav", audio_base64="AAAA")
+
+ def test_serializes_far_below_the_temporal_warning_threshold(self):
+ import orjson
+
+ ref = AudioRef(
+ path="audio/some-fairly-long-generated-name-1a2b3c.wav",
+ workflow_id="019f99e2-dc99-7cf3-9019-3ad3e6260de1",
+ filename="some-fairly-long-generated-name-1a2b3c.wav",
+ mime_type="audio/wav",
+ format="wav",
+ size_bytes=12_000_000,
+ duration_seconds=312.5,
+ sample_rate=44100,
+ channels=2,
+ sha256="a" * 64,
+ url="/api/workspace/019f99e2/files/audio/some-name.wav",
+ )
+ size = len(orjson.dumps(ref.model_dump(mode="json")))
+ assert size < 1024
+ # The whole point: thousands of refs still fit inside one payload.
+ assert TEMPORAL_PAYLOAD_WARN_BYTES // size > 500
+
+
+class TestWriteAudio:
+ def test_round_trips_through_the_workspace(self, tmp_path):
+ ctx = _ctx(tmp_path)
+ payload = _wav_bytes()
+
+ ref = write_audio(payload, ctx=ctx, stem="Hello World!", ext="wav")
+
+ assert ref.path.startswith("audio/")
+ assert not Path(ref.path).is_absolute()
+ assert ref.size_bytes == len(payload)
+ # Not an exact match: mimetypes consults the Windows registry, so
+ # WAV resolves to audio/wav on some machines and audio/x-wav on others.
+ assert ref.mime_type.startswith("audio/")
+ assert read_media_bytes(ref, ctx=ctx)[1] == payload
+
+ def test_populates_duration_from_the_container(self, tmp_path):
+ ref = write_audio(_wav_bytes(seconds=0.5, rate=8000), ctx=_ctx(tmp_path), stem="clip", ext="wav")
+ assert ref.duration_seconds == pytest.approx(0.5, abs=0.05)
+ assert ref.sample_rate == 8000
+
+ def test_never_collides_across_runs(self, tmp_path):
+ ctx = _ctx(tmp_path)
+ first = write_audio(_wav_bytes(), ctx=ctx, stem="same", ext="wav")
+ second = write_audio(_wav_bytes(), ctx=ctx, stem="same", ext="wav")
+ assert first.path != second.path
+
+ def test_url_is_path_only(self, tmp_path):
+ """The frontend prepends its own base for remote backends."""
+ ref = write_audio(_wav_bytes(), ctx=_ctx(tmp_path), stem="x", ext="wav")
+ assert ref.url and ref.url.startswith("/api/workspace/")
+ assert "://" not in ref.url
+
+ def test_refuses_empty_payload(self, tmp_path):
+ from services.plugin import NodeUserError
+
+ with pytest.raises(NodeUserError):
+ write_audio(b"", ctx=_ctx(tmp_path), stem="x", ext="wav")
+
+
+class TestContainment:
+ """Regression: the Sarvam STT node read the credential store."""
+
+ @pytest.mark.parametrize(
+ "attack",
+ ["../../credentials.db", "..\\..\\credentials.db", "~/secrets", "C:/Windows/win.ini"],
+ )
+ def test_traversal_is_refused(self, tmp_path, attack):
+ from services.plugin import NodeUserError
+
+ (tmp_path.parent / "credentials.db").write_bytes(b"SECRET")
+ with pytest.raises(NodeUserError):
+ resolve_media(attack, ctx=_ctx(tmp_path))
+
+ def test_absolute_path_outside_the_workspace_is_refused(self, tmp_path):
+ from services.plugin import NodeUserError
+
+ outside = tmp_path.parent / "elsewhere.wav"
+ outside.write_bytes(b"nope")
+ with pytest.raises(NodeUserError):
+ resolve_media(str(outside), ctx=_ctx(tmp_path))
+
+ def test_absolute_path_inside_the_workspace_still_works(self, tmp_path):
+ """Back-compat for nodes that stored absolute paths pre-AudioRef."""
+ ctx = _ctx(tmp_path)
+ ref = write_audio(_wav_bytes(), ctx=ctx, stem="x", ext="wav")
+ absolute = tmp_path / ref.path
+ assert read_media_bytes(str(absolute), ctx=ctx)[1] == read_media_bytes(ref, ctx=ctx)[1]
+
+ def test_a_ref_cannot_reach_another_workflows_workspace(self, tmp_path):
+ from services.plugin import NodeUserError
+
+ alpha, beta = tmp_path / "alpha", tmp_path / "beta"
+ alpha.mkdir(), beta.mkdir()
+ ref = write_audio(_wav_bytes(), ctx=_ctx(alpha, workflow_id="alpha"), stem="x", ext="wav")
+
+ with pytest.raises(NodeUserError):
+ read_media_bytes(ref, ctx=_ctx(beta, workflow_id="beta"))
+
+
+class TestCoerceFileParam:
+ def test_accepts_an_audio_ref(self, tmp_path):
+ ctx = _ctx(tmp_path)
+ payload = _wav_bytes()
+ ref = write_audio(payload, ctx=ctx, stem="x", ext="wav")
+
+ name, blob = coerce_file_param(ref.model_dump(mode="json"), ctx=ctx)
+ assert blob == payload
+ assert name.endswith(".wav")
+
+ def test_accepts_the_legacy_base64_envelope(self, tmp_path, caplog):
+ """Saved workflow rows still carry this shape and are not migrated."""
+ payload = b"RIFFfake"
+ value = {
+ "type": "upload",
+ "data": base64.b64encode(payload).decode(),
+ "filename": "voice.mp3",
+ "mimeType": "audio/mpeg",
+ }
+ name, blob = coerce_file_param(value, ctx=_ctx(tmp_path))
+ assert (name, blob) == ("voice.mp3", payload)
+
+ def test_accepts_a_relative_path(self, tmp_path):
+ ctx = _ctx(tmp_path)
+ payload = _wav_bytes()
+ ref = write_audio(payload, ctx=ctx, stem="x", ext="wav")
+ assert coerce_file_param(ref.path, ctx=ctx)[1] == payload
+
+ def test_rejects_oversize_legacy_upload(self, tmp_path):
+ from services.plugin import NodeUserError
+
+ value = {"type": "upload", "data": base64.b64encode(b"x" * 4096).decode()}
+ with pytest.raises(NodeUserError, match="limit"):
+ coerce_file_param(value, ctx=_ctx(tmp_path), max_bytes=1024)
+
+ def test_rejects_empty_and_unknown_shapes(self, tmp_path):
+ from services.plugin import NodeUserError
+
+ for value in ("", None, {"kind": "video"}, {"type": "upload", "data": ""}):
+ with pytest.raises(NodeUserError):
+ coerce_file_param(value, ctx=_ctx(tmp_path))
+
+
+class TestInspectNeverRaises:
+ """D8: a parser miss degrades billing, it does not fail a workflow."""
+
+ def test_unknown_container_returns_an_empty_probe(self, tmp_path):
+ junk = tmp_path / "mystery.xyz"
+ junk.write_bytes(b"\x00\x01\x02not audio at all")
+ probe = inspect_audio(junk, declared_format="xyz")
+ assert probe.duration_seconds is None
+
+ def test_missing_file_returns_an_empty_probe(self, tmp_path):
+ assert inspect_audio(tmp_path / "nope.wav").duration_seconds is None
+
+ def test_raw_pcm_duration_is_computed_arithmetically(self, tmp_path):
+ raw = tmp_path / "raw.pcm"
+ raw.write_bytes(b"\x00\x00" * 16000) # 16k frames, 16-bit mono
+ probe = inspect_audio(raw, declared_format="linear16", pcm_sample_rate=16000)
+ assert probe.duration_seconds == pytest.approx(1.0)
diff --git a/server/tests/nodes/test_filesystem_backend.py b/server/tests/nodes/test_filesystem_backend.py
index a5e550ae..e852879c 100644
--- a/server/tests/nodes/test_filesystem_backend.py
+++ b/server/tests/nodes/test_filesystem_backend.py
@@ -225,7 +225,7 @@ def test_windows_fallback_revalidates_parent_before_writing_payload(
)
monkeypatch.setattr(
backend_module,
- "_write_utf8_to_fd",
+ "_write_payload_to_fd",
fail_if_payload_is_written,
)
@@ -233,7 +233,7 @@ def test_windows_fallback_revalidates_parent_before_writing_payload(
ValueError,
match="parent directory changed",
):
- backend_module._atomic_write_text_windows(
+ backend_module._atomic_write_windows(
root,
Path("docs/victim.txt"),
"replacement",