Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(speech): RFC + audio-transport and provider-registry foundations
trohitg18460e02096a
diff --git a/docs-internal/speech_provider_rfc.md b/docs-internal/speech_provider_rfc.mdnew file mode 100644index 00000000..46f0629a--- /dev/null+++ b/docs-internal/speech_provider_rfc.md@@ -0,0 +1,244 @@+# RFC: Multi-provider Speech (TTS / STT)++**Status:** accepted, implementation in progress+**Date:** 2026-07-25+**Supersedes:** the vendor-locked `sarvamTextToSpeech` / `sarvamSpeechToText` nodes++---++## 1. Problem++Speech exists in exactly one place today: two Sarvam nodes. The pattern does not generalise — a+second vendor means a second pair of nodes, a third a third pair, each with its own params, its own+credential, its own skill file. Chat models solved this years ago with `services/llm/`; speech has+no equivalent.++Adding one is not just a refactor, because **audio is big and the execution engine is not built for+big**. That constraint, measured rather than assumed, drives nearly every decision below.++---++## 2. The measured constraint++| Limit | Value | Source |+|---|---|---|+| Temporal blob **error** — activity result, activity **input**, workflow result | **2,097,152 B** | server default; no custom `DataConverter` or `PayloadCodec` anywhere in the repo |+| Temporal blob **warning** | 524,288 B → `PayloadSizeWarning` | not covered by `pyproject.toml` `filterwarnings`, so it surfaces |+| Retries burned before the failure is reported | **3** | `_PayloadSizeError` absent from `NON_RETRYABLE_ERROR_TYPES` ([`_retry_policies.py:51`](../server/services/temporal/_retry_policies.py)) |+| Legacy `execute_node_activity` internal WS | 4 MiB inbound (aiohttp default, `max_msg_size` unset) | [`activities.py:242`](../server/services/temporal/activities.py) |+| `node_outputs.data` | no cap; written **3×** per store (`output_main`/`output_top`/`output_0`) | [`activities.py:486`](../server/services/temporal/activities.py) |+| WS broadcast | no size guard; retained in `_status` **forever** and replayed to every newly connecting client | [`status_broadcaster.py:87,688`](../server/services/status_broadcaster.py) |+| In-memory `_outputs` | never evicted except by `clear_all_outputs` | [`workflow.py:577`](../server/services/workflow.py) |++### What actually happens to a 12 MB base64 TTS result++1. The activity result exceeds 2 MiB → temporalio raises `_PayloadSizeError` at the converter.+2. It is not marked non-retryable → **3 attempts**, each re-invoking the provider and re-billing it.+3. The user sees a generic activity failure, not "audio too large".++And the same limit applies to activity **inputs**, so a ~2 MB audio upload fails before the node+runs at all — today's only file-input path is base64-through-parameters.++### The multiplication++One audio payload is copied at least six ways: `_serialize_result` → `node_outputs` ×3 → WS+broadcast ×2 → `_status` cache (forever) → Temporal activity result → `MachinaWorkflow` aggregate →+every downstream activity input → and, because these nodes are `usable_as_tool`, verbatim into an+LLM message.++**Conclusion: audio cannot travel as bytes.** Not "should not" — the engine will not carry it.++---++## 3. Decisions++### D1 — `AudioRef` is structurally incapable of carrying bytes++A Pydantic model with `extra="forbid"`, a workspace-**relative** path, and no base64 field. Ever.+It serializes to ~400 B, i.e. ~5,200 refs before approaching the Temporal error limit.++Rejected alternative: a capped inline-base64 opt-in, as the current Sarvam TTS node has+(`_MAX_INLINE_B64 = 1_000_000`). The decisive argument is `usable_as_tool` — `execute_as_tool`+returns the flat `Output` dict straight into an LLM message and there is no mechanism to tell a+model "skip this field". A capped opt-in is *sometimes* catastrophic, which is worse than+always-broken because it passes review.++Relative paths (not absolute) because absolute paths embed the mutable workflow slug, leak the+operator's home directory into the DB / WebSocket / LLM context, and cannot be safely served over+HTTP.++### D2 — Two Protocols and two registries, not one `SpeechProvider`++AssemblyAI is STT-only; Cartesia is TTS-only. A single Protocol forces those vendors to ship a dead+method that raises — invisible to the node layer, so the provider would appear in the dropdown and+fail at runtime.++With `TtsProvider` and `SttProvider` registries, **direction capability is registry membership**:+the TTS node's provider enum is literally `tts_registry.all_providers()`. Zero extra machinery.++### D3 — Capability-driven common params + a `provider_options` escape hatch++The verified provider surface diverges much harder than the chat-completions surface does:++| | OpenAI | ElevenLabs | Deepgram |+|---|---|---|---|+| Auth header | `Authorization: Bearer` | `xi-api-key` | `Authorization: Token` |+| Voice selection | body param `voice` | **URL path segment** | encoded in the model name (`aura-2-thalia-en`) |+| Output format | `response_format` enum (`mp3\|opus\|aac\|flac\|wav\|pcm`) | **query** `output_format=mp3_44100_128` (container+rate+bitrate in one token) | separate `encoding` + `container` + `sample_rate` |+| Tuning | `instructions` (free text), `speed` | `voice_settings{stability, similarity_boost, style, use_speaker_boost, speed}` | — |++A per-provider `displayOptions` matrix over 8 providers explodes combinatorially and *still* cannot+express "ElevenLabs `stability` applies only to `eleven_multilingual_v2`". So: ~11 common fields+covering the 90 % case, normalised by each provider adapter, plus a `provider_options: dict`+passed through untouched for vendor specifics.++Keeps the tool schema flat — no `$defs` / `$ref`, which `usable_as_tool` requires.++### D4 — Capabilities declared in JSON, not code++`server/config/speech_defaults.json`, one block per provider per direction, overridable per model —+the same shape `llm_defaults.json` uses for `max_output_tokens`. Drives the provider dropdown, the+voice loader, and validation from one place. `test_plugin_shape.py` bans `if provider == "x"`+branches in shared code; this is how we avoid needing any.++### D5 — First multi-credential node in the repo++`credentials = (OpenAICredential, ElevenLabsCredential, …)` with `ctx.connection(params.provider)`.+Already supported: `_make_connection_factory` ([`base.py:870`](../server/services/plugin/base.py))+builds a dict over **all** declared credentials and raises only for undeclared ids. No node uses+this today.++Consequence: the node must use imperative `@Operation` bodies — the declarative `routing=` path+hardcodes `self.credentials[0]` ([`base.py:566`](../server/services/plugin/base.py)).++### D6 — Extract the generic registry rather than copy it++`ProviderSpec` + lazy `"module:Class"` `sdk_exception_refs` + idempotent registration is entirely+provider-agnostic. Copying it into `services/speech/` would fork the boot-time-import-avoidance+logic that exists specifically to keep ~7 s (warm) / ~45 s (cold) of SDK imports off startup.++`services/provider_registry.py` + `services/provider_clients.py`; `services/llm/registry.py`+becomes a shim. **Success criterion: `server/tests/llm/` passes untouched.**++### D7 — `tinytag` for duration and format++| Candidate | Native deps | Windows | License | Verdict |+|---|---|---|---|---|+| stdlib `wave` | none | ✅ | PSF | WAV only — insufficient alone |+| **`tinytag`** | **none** | ✅ | **MIT** | **chosen** |+| `mutagen` | none | ✅ | **GPL-2.0** | rejected — this repo is MIT; a licensing call, not a technical one |+| `pydub` | ffmpeg on PATH | ❌ | MIT | rejected |+| `ffprobe` | external binary | ❌ | LGPL/GPL | rejected — a pooch-managed binary for a 4-line metadata read |++Fallback chain: `tinytag` → stdlib `wave` → raw-PCM arithmetic → give up.++### D8 — Inspection never fails a workflow++Unknown format returns an all-`None` probe, logs at DEBUG, and still produces a valid `AudioRef`.+Duration-based **rejection only fires when duration is known**; unknown duration falls through to+the byte cap.++A metadata-parser miss on some codec variant must never turn a working workflow into a failing one.+Degrading a billing *estimate* is acceptable; hard-failing a valid file is not.++### D9 — The result-size guard ships WARN-only first++A generic size check in `BaseNode._serialize_result` is the highest-value safety net available: it+converts a 31-second silent triple-retry into an immediate, actionable error. But flipping straight+to hard-fail risks breaking a node that legitimately returns a large result today (a document+parser emitting 3 MB of extracted text).++Ship the warning, read one release of logs, then flip. Both thresholds behind `Settings`.++Implementation note: raise **`NodeUserError`** — it is already in `NON_RETRYABLE_ERROR_TYPES`, so+the failure is immediate with zero new registrations. Do **not** subclass it: Temporal matches+`non_retryable_error_types` on the exception **type-name string**, so a subclass silently starts+retrying again.++---++## 4. Architecture++```+services/provider_registry.py generic ProviderSpec + registry (extracted from llm/)+services/provider_clients.py generic lease-counted client cache (extracted from llm/)++services/media/ audio transport — vendor-neutral, reusable for image/video+ refs.py AudioRef+ workspace.py write_audio / resolve_media / read_media_bytes / coerce_file_param+ inspect.py tinytag -> wave -> PCM arithmetic; never raises+ limits.py every size constant, one place++services/speech/ provider abstraction — mirrors services/llm/+ protocol.py TtsProvider / SttProvider, requests, results, SpeechError+ registry.py two registries over the generic one+ config.py reads server/config/speech_defaults.json+ unifier.py SpeechUnifier — dispatch + typed-error -> NodeUserError+ providers/ _http.py, _openai_compat.py (openai+groq STT), elevenlabs.py,+ sarvam.py, deepgram.py, gemini.py++nodes/speech/ the two nodes+routers/workspace.py GET file (Range-capable) + POST upload+```++**Layering rule.** `services/speech` takes credential **id strings**, never `Credential` classes.+The classes live under `nodes/`, and a `services → nodes` import inverts the layering and breaks+`test_plugin_self_containment.py`. A test locks the two sides agree.++---++## 5. Two bugs this work closes++1. **Path traversal in `sarvam_speech_to_text`.** `_read_audio` does `Path(workspace_dir) / raw`+ with no containment check, so `audio_file="../../credentials.db"` reads the encrypted credential+ store and uploads it to the provider. `coerce_file_param` closes it by construction.+2. **Workflow rename orphans every workspace file.** `rename_workflow` is a single-row `UPDATE`; the+ directory is never moved, and `_get_workspace_dir` keys on the mutable slug. Pre-existing and not+ audio-specific, but `AudioRef` makes it fixable — a best-effort `os.replace` in the rename path.++---++## 6. Explicitly out of scope++| Not doing | Why |+|---|---|+| Streaming / realtime speech | `@Operation` returns one Pydantic model; no `AsyncGenerator` support, no SSE anywhere in the repo. A genuinely new operation kind. |+| Keying workspaces on `workflow_id` | Changes the on-disk layout for every existing install, invalidates `--add-dir` paths baked into live Claude sessions, and reverses a documented decision. Best-effort directory rename instead. |+| Deduping the 3× `store_node_output` write | Touches `ParameterResolver`, the `socialReceive` four-handle case, and `store_agent_output` to save ~1.2 KB once refs replace blobs. |+| Removing the double `node_status` + `node_output` broadcast | Frontend-visible contract change. Elide the cached payload instead so the duplication is bounded. |+| Fixing `MachinaWorkflow.run`'s aggregate result | 20 nodes × 900 KB still exceeds 2 MiB even when every individual node passes. Structural Temporal change; the retry-policy fix makes it fail fast instead of slowly. |++---++## 7. Verified provider surface++Confirmed against live documentation on 2026-07-25. Anything not listed here was **not** verified+and must be checked before it is coded.++**OpenAI** — TTS `POST /v1/audio/speech`, models `gpt-4o-mini-tts` / `tts-1` / `tts-1-hd`, voices+`alloy ash ballad coral echo fable nova onyx sage shimmer verse marin cedar` (subset for tts-1),+`response_format` ∈ `mp3 opus aac flac wav pcm`, plus `instructions` and `speed`. Raw binary+response. STT `POST /v1/audio/transcriptions`, multipart.++**ElevenLabs** — TTS `POST /v1/text-to-speech/{voice_id}`, header `xi-api-key`, body `text`,+`model_id` (default `eleven_multilingual_v2`), `voice_settings{stability, similarity_boost, style,+use_speaker_boost, speed}`, `language_code`, `seed`, `apply_text_normalization` ∈ `auto|on|off`;+**query** `output_format` (default `mp3_44100_128`, 27 values). Binary response. Models via+`GET /v1/models`, voices via `GET /v1/voices`. STT `POST /v1/speech-to-text`, multipart,+`model_id` ∈ `scribe_v2|scribe_v1`, `language_code`, `tag_audio_events` (default true),+`num_speakers` (≤32), `timestamps_granularity` ∈ `none|word|character`, `diarize` (default false).+Response `{language_code, language_probability, text, words[]}`.++**Groq** — STT `POST https://api.groq.com/openai/v1/audio/transcriptions`, OpenAI-compatible,+`whisper-large-v3` / `whisper-large-v3-turbo`. This is why `_openai_compat.py` covers both vendors+off one factory.++**Deepgram** — TTS `POST /v1/speak`, header `Authorization: Token <key>`, model names of the form+`aura-2-<voice>-<lang>`. STT `POST /v1/listen`. Query-parameter driven; exact parameter set to be+confirmed at implementation time.++**Sarvam** — already implemented; see [`sarvam_service.md`](./sarvam_service.md). TTS returns+base64 inside JSON (`{audios: […]}`), the one provider that does.++**Gemini** — TTS via `generateContent` with an audio response modality. Shape **not yet verified**;+must be confirmed before coding.diff --git a/server/nodes/filesystem/_backend.py b/server/nodes/filesystem/_backend.pyindex 290423b1..8b5ed4e6 100644--- a/server/nodes/filesystem/_backend.py+++ b/server/nodes/filesystem/_backend.py@@ -160,13 +160,26 @@ def _verify_posix_directory(os.close(current_fd)-def _write_utf8_to_fd(+def _write_payload_to_fd(file_fd: int,- content: str,+ payload: str | bytes,existing_mode: Optional[int],) -> None:- with os.fdopen(file_fd, "w", encoding="utf-8", newline="") as handle:- handle.write(content)+ """Write ``payload`` to an already-opened fd, then fsync.++ Accepts ``str`` (UTF-8, newline-preserving) or ``bytes``. The binary+ branch exists so generated media — audio from a TTS provider, an+ uploaded clip — reaches disk through the same anchored-write path that+ refuses to follow a mutable symlink or Windows junction. Those bytes+ are later served over HTTP, so the containment matters more than for+ ordinary text writes.+ """+ if isinstance(payload, bytes):+ mode, kwargs = "wb", {}+ else:+ mode, kwargs = "w", {"encoding": "utf-8", "newline": ""}+ with os.fdopen(file_fd, mode, **kwargs) as handle:+ handle.write(payload)handle.flush()os.fsync(handle.fileno())if existing_mode is not None and hasattr(os, "fchmod"):@@ -186,10 +199,10 @@ def _replace_posix_entry()-def _atomic_write_text_posix(+def _atomic_write_posix(root: Path,relative: Path,- content: str,+ payload: str | bytes,) -> None:parent_components = tuple(relative.parts[:-1])target_name = relative.parts[-1]@@ -250,7 +263,7 @@ def _atomic_write_text_posix("Could not allocate a unique atomic-write temporary file")- _write_utf8_to_fd(temporary_fd, content, existing_mode)+ _write_payload_to_fd(temporary_fd, payload, existing_mode)_verify_posix_directory(root, parent_components, parent_fd)_replace_posix_entry(temporary_name,@@ -323,10 +336,10 @@ def _verify_windows_temp()-def _atomic_write_text_windows(+def _atomic_write_windows(root: Path,relative: Path,- content: str,+ payload: str | bytes,) -> None:"""Best-effort contained atomic replacement for Windows.@@ -393,7 +406,7 @@ def _atomic_write_text_windows(write_fd = temporary_fdtemporary_fd = -1- _write_utf8_to_fd(write_fd, content, existing_mode)+ _write_payload_to_fd(write_fd, payload, existing_mode)if existing_mode is not None:# Windows has no ``os.fchmod``. Apply its supported mode bits# (notably the read-only flag) to the still-private temporary@@ -459,19 +472,11 @@ def _atomic_write_text_windows(pass-def atomic_write_text(+def _atomic_write(path: os.PathLike[str] | str,- content: str,- *,- root_dir: os.PathLike[str] | str | None = None,+ payload: str | bytes,+ root_dir: os.PathLike[str] | str | None,) -> None:- """Atomically replace UTF-8 text without traversing mutable symlinks.-- Production callers pass the workspace root. The optional default keeps- direct helper callers compatible by treating the existing parent as the- root, but it does not establish a broader containment boundary.- """-target = Path(os.path.abspath(os.fspath(path)))root = Path(os.path.abspath(@@ -488,9 +493,41 @@ def atomic_write_text(raise IsADirectoryError(f"Cannot replace workspace root: {target}")if _anchored_write_supported():- _atomic_write_text_posix(root, relative, content)+ _atomic_write_posix(root, relative, payload)else:- _atomic_write_text_windows(root, relative, content)+ _atomic_write_windows(root, relative, payload)+++def atomic_write_text(+ path: os.PathLike[str] | str,+ content: str,+ *,+ root_dir: os.PathLike[str] | str | None = None,+) -> None:+ """Atomically replace UTF-8 text without traversing mutable symlinks.++ Production callers pass the workspace root. The optional default keeps+ direct helper callers compatible by treating the existing parent as the+ root, but it does not establish a broader containment boundary.+ """+ _atomic_write(path, content, root_dir)+++def atomic_write_bytes(+ path: os.PathLike[str] | str,+ payload: bytes,+ *,+ root_dir: os.PathLike[str] | str | None = None,+) -> None:+ """Atomically replace binary content, with the same containment rules.++ Used by ``services.media`` for generated and uploaded audio. Atomicity+ is load-bearing here rather than merely tidy: the frontend renders an+ ``<audio src>`` the moment it sees a reference, so a half-written file+ is a broken player, and Temporal retries a synthesis activity against+ the *same* target path.+ """+ _atomic_write(path, payload, root_dir)def _validate_virtual_path(path: str) -> str:@@ -508,6 +545,30 @@ def _validate_virtual_path(path: str) -> str:return normalized+def resolve_within(root: os.PathLike[str] | str, key: str) -> Path:+ """Resolve ``key`` under ``root``, refusing anything that escapes it.++ Two layers: reject ``..`` / ``~`` / drive-prefixed inputs *before*+ touching the filesystem, then resolve and re-check containment so a+ symlink or Windows junction cannot redirect the result outside the+ root.++ Module-level rather than a ``WorkspaceBackend`` method because+ ``services.media`` and the workspace HTTP route both need it without+ paying for a backend instance (whose ``__init__`` mkdirs, copies the+ environment, and mints a sandbox id).+ """+ root_path = Path(os.path.abspath(os.fspath(root)))+ virtual = _validate_virtual_path(key)+ candidate = root_path / virtual.lstrip("/")+ try:+ resolved = candidate.resolve(strict=False)+ resolved.relative_to(root_path)+ except (OSError, RuntimeError, ValueError) as exc:+ raise ValueError(f"Path '{key}' resolves outside workspace root") from exc+ return resolved++def normalize_virtual_path(path: str) -> str:"""Normalize user paths to canonical workspace-relative virtual paths."""from services.plugin import NodeUserError@@ -611,16 +672,7 @@ class WorkspaceBackend:return self._sandbox_iddef _resolve_path(self, key: str) -> Path:- virtual = _validate_virtual_path(key)- candidate = self.cwd / virtual.lstrip("/")- try:- resolved = candidate.resolve(strict=False)- resolved.relative_to(self.cwd)- except (OSError, RuntimeError, ValueError) as exc:- raise ValueError(- f"Path '{key}' resolves outside workspace root"- ) from exc- return resolved+ return resolve_within(self.cwd, key)def _to_virtual_path(self, path: Path) -> str:resolved = path.resolve(strict=False)diff --git a/server/pyproject.toml b/server/pyproject.tomlindex a885eb10..fd088a97 100644--- a/server/pyproject.toml+++ b/server/pyproject.toml@@ -40,6 +40,16 @@ dependencies = [# https://temporal.download/cli/archive/latest)."pooch>=1.8.2",+ # Media metadata (services/media/inspect.py). Pure Python, zero native+ # deps, MIT. Chosen over mutagen (GPL-2.0, incompatible with this+ # project's MIT licence) and over pydub / ffprobe, which both require+ # an ffmpeg binary on PATH that nothing installs on Windows.+ "tinytag>=2.0.0",+ # Multipart parsing for the workspace upload route. Already present+ # transitively via mcp; declared explicitly so it survives that+ # dependency changing.+ "python-multipart>=0.0.20",+# Auth"bcrypt>=4.2.0",# pyjwt replaced python-jose (2026-06): python-jose drags in python-ecdsa,diff --git a/server/services/llm/registry.py b/server/services/llm/registry.pyindex 8fc47b7f..c32fb76a 100644--- a/server/services/llm/registry.py+++ b/server/services/llm/registry.py@@ -5,141 +5,69 @@ calls ``register_provider(ProviderSpec(...))`` at top level, and the unifierreads this registry to dispatch chat / fetch_models calls and to translatetyped SDK exceptions into ``NodeUserError``. No per-provider Python livesinside ``services/ai.py`` after this layer is wired.++The mechanism itself now lives in :mod:`services.provider_registry`, shared+with ``services/speech`` — this module is the LLM-specific instance plus the+free-function API that provider modules and tests already import. The public+surface (``ProviderSpec``, ``register_provider``, ``get_provider``,+``all_providers``, ``has_provider``, ``_reset_for_tests``) is unchanged."""from __future__ import annotations-import functools-import pkgutil-from dataclasses import dataclass, field-from typing import Any, Callable, Dict, List, Tuple, Type+from typing import List-from core.logging import get_loggerfrom services.llm.protocol import LLMProvider-from services.plugin import NodeUserError--logger = get_logger(__name__)--[email protected]_cache(maxsize=None)-def _resolve_exc(ref: str) -> Type[BaseException]:- """Resolve a ``"module:ClassName"`` ref to the exception class.-- Called lazily from ``ProviderSpec.sdk_exception_types`` — by the time- anything needs the resolved classes (an ``except`` clause matching a- raised SDK error, or a test reading the tuple), the provider factory- has already imported the SDK to build its client, so this is a- ``sys.modules`` cache hit, never a cold import on the boot path.- """- obj = pkgutil.resolve_name(ref)- if not (isinstance(obj, type) and issubclass(obj, BaseException)):- raise TypeError(f"{ref!r} did not resolve to an exception class: {obj!r}")- return obj---@dataclass(frozen=True)-class ProviderSpec:- """Declarative spec for a chat-model provider plugin.-- Fields:- name: registry key — matches the ``provider`` string used by- ai.py, the chat-model node ``provider`` field, and the- ``providers.<name>`` block in ``llm_defaults.json``.- factory: callable returning an ``LLMProvider`` instance.- Invoked as ``factory(api_key=..., proxy_url=..., **client_kwargs)``.- sdk_exception_refs: tuple of ``"module:ClassName"`` refs (the- packaging entry-points format, resolved via- ``pkgutil.resolve_name``) naming the typed error classes- raised by this provider's SDK. Refs — not classes — so that- registering a provider at boot never imports its SDK- (docs-internal/performance.md anti-pattern; cost ~7s warm /- ~45s cold across openai + anthropic + google.genai). The- resolved classes are exposed by ``sdk_exception_types``.- client_kwargs: static keyword arguments passed to ``factory`` on- every instantiation. Used by OpenAI-compatible providers- (xai / deepseek / kimi / mistral / ollama / lmstudio / groq /- cerebras) to pin ``base_url`` from JSON config.- """-- name: str- factory: Callable[..., LLMProvider]- sdk_exception_refs: Tuple[str, ...]- client_kwargs: Dict[str, Any] = field(default_factory=dict)-- @property- def sdk_exception_types(self) -> Tuple[Type[BaseException], ...]:- """Resolved typed SDK error classes.-- The unifier wraps a single ``except spec.sdk_exception_types``- around every call site so user-correctable errors surface as- ``NodeUserError`` without a per-provider branch in ai.py.- """- return tuple(_resolve_exc(ref) for ref in self.sdk_exception_refs)---_REGISTRY: Dict[str, ProviderSpec] = {}+from services.provider_registry import (+ ProviderRegistry,+ ProviderSpec as _ProviderSpec,+ resolve_exception_ref as _resolve_exc, # noqa: F401 — re-exported for tests+)++# Re-exported under the historical name. Generic in the provider type, so+# ``ProviderSpec(factory=...)`` still type-checks against ``LLMProvider``.+ProviderSpec = _ProviderSpec[LLMProvider]++_LLM_REGISTRY: ProviderRegistry[LLMProvider] = ProviderRegistry("LLM")++# Compatibility alias bound to the SAME dict object the registry mutates.+# Several tests swap a provider's factory in place via ``_REGISTRY[name] = spec``+# to exercise typed-error translation and client-cache policy without touching+# the real SDKs. Keeping the name (and the identity) means this extraction is+# invisible to them — the success criterion for the refactor was that+# ``tests/llm/`` passes untouched.+_REGISTRY = _LLM_REGISTRY._specsdef register_provider(spec: ProviderSpec) -> None:- """Register a provider plugin.-- Called from each provider module's top level on import. The unifier- expects every registered provider to declare at least one typed SDK- exception so the wrapper try/except can translate it.-- Re-registration with an identical spec is a no-op (handles uvicorn- reload cycles). Re-registration with a different spec raises so- accidental shadowing is loud.- """- # Check the raw refs, NOT the resolving property — reading the- # property here would import every SDK at registration time and- # defeat the lazy-ref design.- if not spec.sdk_exception_refs:- raise ValueError(- f"ProviderSpec({spec.name!r}) declares empty sdk_exception_refs. "- "Every provider must surface its typed SDK error class so the "- "unifier can translate it into NodeUserError."- )- existing = _REGISTRY.get(spec.name)- if existing is not None and existing != spec:- raise ValueError(- f"Provider {spec.name!r} already registered with a different spec. "- f"Existing: {existing!r}; new: {spec!r}."- )- _REGISTRY[spec.name] = spec- logger.debug("registered LLM provider", provider=spec.name)+ """Register a provider plugin. Idempotent for identical specs."""+ _LLM_REGISTRY.register(spec)def get_provider(name: str) -> ProviderSpec:- """Look up a provider spec by name.-- Raises ``NodeUserError`` on unknown providers so the failure surfaces- cleanly through ``BaseNode.execute()`` as a single WARN line — same- contract as every other user-correctable error in the framework.- """- spec = _REGISTRY.get(name)- if spec is None:- raise NodeUserError(- f"Unknown LLM provider: {name!r}. "- f"Registered providers: {sorted(_REGISTRY)}"- )- return spec+ """Look up a provider spec by name; raises ``NodeUserError`` on a miss."""+ return _LLM_REGISTRY.get(name)def all_providers() -> List[str]:"""Return sorted list of registered provider names."""- return sorted(_REGISTRY)+ return _LLM_REGISTRY.all()def has_provider(name: str) -> bool:"""Cheap membership probe — does not raise on miss."""- return name in _REGISTRY+ return _LLM_REGISTRY.has(name)def _reset_for_tests() -> None:- """Test-only — drop every registered provider.+ """Test-only — drop every registered provider."""+ _LLM_REGISTRY.reset_for_tests()+- Lets unit tests around the registry control the state instead of- inheriting the production population. Never call from runtime code.- """- _REGISTRY.clear()+__all__ = [+ "ProviderSpec",+ "all_providers",+ "get_provider",+ "has_provider",+ "register_provider",+]diff --git a/server/services/media/__init__.py b/server/services/media/__init__.pynew file mode 100644index 00000000..3752ee3e--- /dev/null+++ b/server/services/media/__init__.py@@ -0,0 +1,50 @@+"""Media transport — references, not bytes.++Vendor-neutral and deliberately not part of ``services/speech``: the same+envelope is consumed by the file widget, the workspace HTTP routes, and+any node that produces or accepts a file. Owning it from a provider+package would invert the dependency.++The contract in one line: **a node returns an**+:class:`~services.media.refs.AudioRef`, **never audio bytes.** See+:mod:`services.media.limits` for the measured reason.+"""++from services.media.inspect import AudioProbe, inspect_audio+from services.media.limits import (+ MEDIA_MAX_AUDIO_SECONDS,+ MEDIA_MAX_READ_BYTES,+ MEDIA_MAX_UPLOAD_BYTES,+ TEMPORAL_PAYLOAD_ERROR_BYTES,+ TEMPORAL_PAYLOAD_WARN_BYTES,+)+from services.media.refs import AudioRef+from services.media.workspace import (+ AUDIO_SUBDIR,+ UPLOAD_SUBDIR,+ coerce_file_param,+ read_media_bytes,+ resolve_media,+ workspace_file_url,+ workspace_root,+ write_audio,+)++__all__ = [+ "AUDIO_SUBDIR",+ "UPLOAD_SUBDIR",+ "AudioProbe",+ "AudioRef",+ "MEDIA_MAX_AUDIO_SECONDS",+ "MEDIA_MAX_READ_BYTES",+ "MEDIA_MAX_UPLOAD_BYTES",+ "TEMPORAL_PAYLOAD_ERROR_BYTES",+ "TEMPORAL_PAYLOAD_WARN_BYTES",+ "coerce_file_param",+ "inspect_audio",+ "read_media_bytes",+ "resolve_media",+ "workspace_file_url",+ "workspace_root",+ "write_audio",+]diff --git a/server/services/media/inspect.py b/server/services/media/inspect.pynew file mode 100644index 00000000..4d5580de--- /dev/null+++ b/server/services/media/inspect.py@@ -0,0 +1,158 @@+"""Audio metadata probing — duration, sample rate, channels.++Three fallbacks, and a hard rule: **this never raises and never fails a+workflow.** An unknown container returns an empty probe, logs at DEBUG,+and the caller still gets a valid :class:`~services.media.AudioRef`.++The rule matters because the alternative is worse in exactly the wrong+direction. Degrading a *billing estimate* when a codec variant confuses+the parser is acceptable; hard-failing a file the provider would have+accepted is not.++``tinytag`` was chosen over the alternatives on two filters. License:+``mutagen`` is GPL-2.0 and this repository is MIT, which is a legal+decision rather than a technical one. Portability: ``pydub`` and+``ffprobe`` both require an ffmpeg binary on PATH, and this repository is+developed on Windows where nothing installs one.+"""++from __future__ import annotations++import wave+from dataclasses import dataclass+from pathlib import Path+from typing import Optional++from core.logging import get_logger++logger = get_logger(__name__)++# Bytes per sample frame for the raw, header-less encodings a TTS provider+# may hand back. Used only when the caller tells us the sample rate --+# there is nothing in the file itself to read.+_PCM_BYTES_PER_SAMPLE = {+ "pcm": 2,+ "linear16": 2,+ "pcm_s16le": 2,+ "pcm_l16": 2,+ "pcm_raw": 2,+ "mulaw": 1,+ "ulaw": 1,+ "alaw": 1,+}+++@dataclass(frozen=True)+class AudioProbe:+ """What we managed to learn. Every field may legitimately be None."""++ duration_seconds: Optional[float] = None+ sample_rate: Optional[int] = None+ channels: Optional[int] = None+ format: Optional[str] = None+++def inspect_audio(+ path: Path | str,+ *,+ declared_format: Optional[str] = None,+ pcm_sample_rate: Optional[int] = None,+ pcm_channels: int = 1,+) -> AudioProbe:+ """Best-effort metadata for an audio file. Never raises."""+ target = Path(path)++ probe = _probe_tinytag(target)+ if probe.duration_seconds is not None:+ return probe++ probe = _probe_wave(target)+ if probe.duration_seconds is not None:+ return probe++ probe = _probe_raw_pcm(+ target,+ declared_format=declared_format,+ sample_rate=pcm_sample_rate,+ channels=pcm_channels,+ )+ if probe.duration_seconds is not None:+ return probe++ logger.debug(+ "audio metadata unavailable",+ path=str(target),+ declared_format=declared_format,+ )+ return AudioProbe(format=declared_format or None)+++def _probe_tinytag(target: Path) -> AudioProbe:+ """mp3 / m4a / flac / ogg / opus / wav / aiff / wma."""+ try:+ from tinytag import TinyTag++ tag = TinyTag.get(str(target))+ except Exception as exc: # unsupported container, corrupt header, missing dep+ logger.debug("tinytag declined", path=str(target), error=str(exc))+ return AudioProbe()++ duration = getattr(tag, "duration", None)+ return AudioProbe(+ duration_seconds=float(duration) if duration else None,+ sample_rate=getattr(tag, "samplerate", None),+ channels=getattr(tag, "channels", None),+ format=target.suffix.lstrip(".").lower() or None,+ )+++def _probe_wave(target: Path) -> AudioProbe:+ """stdlib fallback for WAV variants tinytag declines."""+ try:+ with wave.open(str(target), "rb") as handle:+ frames = handle.getnframes()+ rate = handle.getframerate()+ channels = handle.getnchannels()+ except Exception as exc:+ logger.debug("wave declined", path=str(target), error=str(exc))+ return AudioProbe()++ if not rate:+ return AudioProbe()+ return AudioProbe(+ duration_seconds=frames / float(rate),+ sample_rate=rate,+ channels=channels,+ format="wav",+ )+++def _probe_raw_pcm(+ target: Path,+ *,+ declared_format: Optional[str],+ sample_rate: Optional[int],+ channels: int,+) -> AudioProbe:+ """Header-less PCM: arithmetic, only when the caller knows the rate."""+ if not declared_format or not sample_rate:+ return AudioProbe()+ width = _PCM_BYTES_PER_SAMPLE.get(declared_format.lower())+ if not width:+ return AudioProbe()+ try:+ size = target.stat().st_size+ except OSError:+ return AudioProbe()+ divisor = sample_rate * width * max(1, channels)+ if divisor <= 0:+ return AudioProbe()+ return AudioProbe(+ duration_seconds=size / divisor,+ sample_rate=sample_rate,+ channels=channels,+ format=declared_format.lower(),+ )+++__all__ = ["AudioProbe", "inspect_audio"]diff --git a/server/services/media/limits.py b/server/services/media/limits.pynew file mode 100644index 00000000..4dfdc130--- /dev/null+++ b/server/services/media/limits.py@@ -0,0 +1,49 @@+"""Every media size constant, in one place.++These exist because the execution engine multiplies whatever a node+returns. One payload is copied at least six ways: ``_serialize_result``+-> ``node_outputs`` x3 -> WebSocket broadcast x2 -> the status cache+(retained for the process lifetime) -> the Temporal activity result ->+``MachinaWorkflow``'s aggregate -> every downstream activity input ->+and, for a ``usable_as_tool`` node, verbatim into an LLM message.++Temporal's blob limits are the binding constraint. They are server+defaults; nothing in this repo configures a ``DataConverter`` or a+payload codec, so they apply as-is:++ warning 524_288 B -> PayloadSizeWarning (not filtered, so visible)+ error 2_097_152 B -> _PayloadSizeError, and the activity RETRIES+ three times before reporting a generic failure++That is why :class:`services.media.AudioRef` carries no bytes at all.+"""++from __future__ import annotations++# Temporal's own thresholds, restated so call sites can reason about+# headroom without importing temporalio.+TEMPORAL_PAYLOAD_WARN_BYTES = 524_288+TEMPORAL_PAYLOAD_ERROR_BYTES = 2_097_152++# Largest upload accepted by the workspace upload route. Deliberately far+# above the Temporal limit: uploaded bytes land on disk and only a ~400 B+# reference ever enters a payload.+MEDIA_MAX_UPLOAD_BYTES = 25 * 1024 * 1024++# Largest file a node will pull back off disk into memory to hand to a+# provider. Bounds one activity's peak RSS; the bytes are discarded as+# soon as the HTTP call returns.+MEDIA_MAX_READ_BYTES = 25 * 1024 * 1024++# Longest clip accepted when the duration is actually known. Unknown+# duration falls through to the byte cap rather than failing -- see+# services.media.inspect for why inspection must never be load-bearing.+MEDIA_MAX_AUDIO_SECONDS = 60 * 60++__all__ = [+ "MEDIA_MAX_AUDIO_SECONDS",+ "MEDIA_MAX_READ_BYTES",+ "MEDIA_MAX_UPLOAD_BYTES",+ "TEMPORAL_PAYLOAD_ERROR_BYTES",+ "TEMPORAL_PAYLOAD_WARN_BYTES",+]diff --git a/server/services/media/refs.py b/server/services/media/refs.pynew file mode 100644index 00000000..fa6d7fb6--- /dev/null+++ b/server/services/media/refs.py@@ -0,0 +1,62 @@+"""``AudioRef`` — a reference to audio, never the audio itself.++The one rule this module exists to enforce: **audio bytes do not travel+through the workflow engine.** See :mod:`services.media.limits` for the+measured reason. This model has no bytes field, no base64 field, and+``extra="forbid"`` so that adding one is a validation error rather than a+silent regression.+"""++from __future__ import annotations++from typing import Literal, Optional++from pydantic import BaseModel, ConfigDict, Field+++class AudioRef(BaseModel):+ """A pointer to an audio file inside a workflow workspace.++ Serializes to roughly 400 bytes, i.e. about 5,200 refs before+ approaching Temporal's 2 MiB error limit -- the envelope is+ structurally incapable of getting near it.++ ``path`` is workspace-**relative** POSIX, never an absolute host path.+ Absolute paths embed the mutable workflow slug, leak the operator's+ home directory into the database / WebSocket broadcasts / LLM context,+ and cannot be safely turned into an HTTP URL.+ """++ kind: Literal["audio"] = "audio"++ path: str = Field(+ description="Workspace-relative POSIX path, no leading slash "+ "(e.g. 'audio/greeting-1a2b3c.wav').",+ )+ # Stable across renames -- the workspace directory is keyed on the+ # slug, which changes, while the id does not. Lets the resolver and+ # the file-serving route find the workspace without a NodeContext.+ workflow_id: Optional[str] = None++ filename: str = Field(description="Display name. Never used for resolution.")+ mime_type: str = "application/octet-stream"+ format: str = Field(default="", description="Container/codec: wav, mp3, opus, ...")+ size_bytes: int = 0++ # None whenever the container could not be inspected. Never guessed --+ # a fabricated duration would silently mis-bill per-second providers.+ duration_seconds: Optional[float] = None+ sample_rate: Optional[int] = None+ channels: Optional[int] = None++ sha256: Optional[str] = None++ # Path-only, no scheme or host, so the frontend can prepend its own+ # base via buildApiUrl() when it points at a remote backend.+ # Advisory: `path` + `workflow_id` remain canonical.+ url: Optional[str] = None++ model_config = ConfigDict(extra="forbid")+++__all__ = ["AudioRef"]diff --git a/server/services/media/workspace.py b/server/services/media/workspace.pynew file mode 100644index 00000000..3bb2ee38--- /dev/null+++ b/server/services/media/workspace.py@@ -0,0 +1,287 @@+"""Reading and writing media inside a workflow workspace.++Everything here funnels through :func:`nodes.filesystem._backend.resolve_within`,+which rejects ``..`` / ``~`` / drive-prefixed inputs before touching the+filesystem and then re-checks containment after resolution so a symlink or+Windows junction cannot redirect the result outside the workspace.++That containment is not decorative. Before this module existed, the Sarvam+speech-to-text node joined a user-supplied path onto the workspace root with+no check at all, so ``audio_file="../../credentials.db"`` read the encrypted+credential store and uploaded it to the provider. :func:`coerce_file_param`+closes that by construction, for every node that adopts it.+"""++from __future__ import annotations++import hashlib+import mimetypes+import re+from pathlib import Path+from typing import Any, Optional, Tuple+from uuid import uuid4++from core.logging import get_logger+from services.media.inspect import inspect_audio+from services.media.limits import MEDIA_MAX_READ_BYTES+from services.media.refs import AudioRef++logger = get_logger(__name__)++AUDIO_SUBDIR = "audio"+UPLOAD_SUBDIR = "uploads"++_SLUG_RE = re.compile(r"[^a-z0-9]+")++# Windows treats these as devices no matter the extension, so a write to+# "CON.wav" is a write to the console. Every generated name carries a uuid+# suffix, which makes a bare reserved name unreachable, but the check stays+# as defence for callers that pass an unusual stem.+_WINDOWS_RESERVED = {+ "con", "prn", "aux", "nul",+ *(f"com{i}" for i in range(1, 10)),+ *(f"lpt{i}" for i in range(1, 10)),+}+++def _slugify(value: str, *, limit: int = 32) -> str:+ slug = _SLUG_RE.sub("-", (value or "").lower()).strip("-")[:limit].strip("-")+ if not slug or slug in _WINDOWS_RESERVED:+ return "audio"+ return slug+++def workspace_root(ctx: Any = None, *, workflow_id: Optional[str] = None) -> Path:+ """Resolve the workspace directory for a context or workflow id.++ Prefers the executor-injected ``ctx.workspace_dir``; falls back to+ composing it from the workflow's slug so the HTTP routes -- which have+ no NodeContext -- can resolve the same directory.+ """+ from services.plugin import NodeUserError++ direct = getattr(ctx, "workspace_dir", None) if ctx is not None else None+ if not direct and ctx is not None:+ raw = getattr(ctx, "raw", None)+ if isinstance(raw, dict):+ direct = raw.get("workspace_dir")+ if direct:+ return Path(direct)++ if workflow_id is None and ctx is not None:+ workflow_id = getattr(ctx, "workflow_id", None)+ if not workflow_id:+ raise NodeUserError(+ "No workspace is available for this execution, so media cannot be "+ "read or written. Save the workflow and run it again."+ )+ from core.paths import workspace_dir as _workspace_dir++ return Path(_workspace_dir(workflow_id))+++def resolve_media(+ ref: AudioRef | str,+ *,+ ctx: Any = None,+ workflow_id: Optional[str] = None,+) -> Path:+ """Resolve a ref or a path string to a contained absolute path."""+ from nodes.filesystem._backend import resolve_within+ from services.plugin import NodeUserError++ key = ref.path if isinstance(ref, AudioRef) else str(ref or "")+ if not key.strip():+ raise NodeUserError("No media path was provided.")++ if isinstance(ref, AudioRef) and ref.workflow_id and workflow_id is None:+ workflow_id = ref.workflow_id++ candidate = Path(key)+ if candidate.is_absolute():+ # Tolerated for back-compat with nodes that stored absolute paths+ # before AudioRef existed, but still contained: an absolute path+ # outside the workspace is refused rather than read.+ root = workspace_root(ctx, workflow_id=workflow_id)+ try:+ key = candidate.resolve(strict=False).relative_to(root).as_posix()+ except ValueError as exc:+ raise NodeUserError(+ f"'{candidate}' is outside this workflow's workspace."+ ) from exc+ else:+ root = workspace_root(ctx, workflow_id=workflow_id)++ try:+ return resolve_within(root, key)+ except ValueError as exc:+ raise NodeUserError(f"'{key}' is outside this workflow's workspace.") from exc+++def read_media_bytes(+ ref: AudioRef | str,+ *,+ ctx: Any = None,+ workflow_id: Optional[str] = None,+ max_bytes: int = MEDIA_MAX_READ_BYTES,+) -> Tuple[str, bytes]:+ """Return ``(filename, bytes)`` for a contained media reference."""+ from services.plugin import NodeUserError++ target = resolve_media(ref, ctx=ctx, workflow_id=workflow_id)+ if not target.is_file():+ raise NodeUserError(f"Media file not found: {target.name}")++ size = target.stat().st_size+ if size == 0:+ raise NodeUserError(f"Media file is empty: {target.name}")+ if size > max_bytes:+ raise NodeUserError(+ f"{target.name} is {size // (1024 * 1024)} MB; the limit is "+ f"{max_bytes // (1024 * 1024)} MB."+ )+ return target.name, target.read_bytes()+++def write_audio(+ payload: bytes,+ *,+ ctx: Any,+ stem: str,+ ext: str,+ mime_type: Optional[str] = None,+ subdir: str = AUDIO_SUBDIR,+ inspect: bool = True,+ sample_rate: Optional[int] = None,+ channels: int = 1,+) -> AudioRef:+ """Atomically write audio into the workspace and return a reference.++ The filename shape (``<stem>-<node8>-<rand6>.<ext>``) matches what the+ Sarvam TTS node produced before this helper existed, so porting it is+ behaviour-preserving. The random suffix also means retries and repeated+ runs never collide or silently overwrite earlier audio.+ """+ from nodes.filesystem._backend import atomic_write_bytes+ from services.plugin import NodeUserError++ if not payload:+ raise NodeUserError("Refusing to write an empty audio file.")++ root = workspace_root(ctx)+ node_id = str(getattr(ctx, "node_id", "") or "node")+ name = f"{_slugify(stem)}-{node_id[:8]}-{uuid4().hex[:6]}.{ext.lstrip('.')}"+ rel = f"{subdir}/{name}"++ target = resolve_media(rel, ctx=ctx)+ target.parent.mkdir(parents=True, exist_ok=True)+ atomic_write_bytes(target, payload, root_dir=root)++ probe = (+ inspect_audio(+ target,+ declared_format=ext,+ pcm_sample_rate=sample_rate,+ pcm_channels=channels,+ )+ if inspect+ else None+ )+ workflow_id = getattr(ctx, "workflow_id", None)++ return AudioRef(+ path=rel,+ workflow_id=workflow_id,+ filename=name,+ mime_type=mime_type or mimetypes.guess_type(name)[0] or "application/octet-stream",+ format=ext.lstrip(".").lower(),+ size_bytes=len(payload),+ duration_seconds=probe.duration_seconds if probe else None,+ sample_rate=(probe.sample_rate if probe else None) or sample_rate,+ channels=(probe.channels if probe else None),+ sha256=hashlib.sha256(payload).hexdigest(),+ url=workspace_file_url(workflow_id, rel),+ )+++def workspace_file_url(workflow_id: Optional[str], rel_path: str) -> Optional[str]:+ """Path-only URL for the workspace file route. No scheme, no host."""+ if not workflow_id:+ return None+ from urllib.parse import quote++ return f"/api/workspace/{quote(str(workflow_id))}/files/{quote(rel_path)}"+++def coerce_file_param(+ value: Any,+ *,+ ctx: Any = None,+ max_bytes: int = MEDIA_MAX_READ_BYTES,+) -> Tuple[str, bytes]:+ """Read a file parameter in any of the three shapes the UI can produce.++ 1. A serialized :class:`AudioRef` -- what the upload route now returns.+ 2. The legacy ``{"type": "upload", "data": "<base64>"}`` envelope the+ file widget used to emit. Still accepted indefinitely: saved+ workflow rows carry it and are not migrated. Logs one warning+ naming the node so operators can find and re-save them.+ 3. A bare path string, typed by the user or dragged from an upstream+ node -- now resolved with containment.+ """+ import base64+ import binascii++ from services.plugin import NodeUserError++ if isinstance(value, dict) and value.get("kind") == "audio":+ ref = AudioRef.model_validate(value)+ return read_media_bytes(ref, ctx=ctx, max_bytes=max_bytes)++ if isinstance(value, dict) and value.get("type") == "upload":+ data = value.get("data") or ""+ if not data:+ raise NodeUserError(+ "The uploaded file carried no data. Re-select it in the "+ "parameter panel."+ )+ logger.warning(+ "legacy base64 upload parameter; re-select the file to migrate it",+ node_id=str(getattr(ctx, "node_id", "") or "unknown"),+ characters=len(data),+ )+ try:+ blob = base64.b64decode(data, validate=True)+ except (binascii.Error, ValueError) as exc:+ raise NodeUserError(f"Uploaded file is not valid base64: {exc}") from exc+ if len(blob) > max_bytes:+ raise NodeUserError(+ f"Uploaded file is {len(blob) // (1024 * 1024)} MB; the limit "+ f"is {max_bytes // (1024 * 1024)} MB."+ )+ if not blob:+ raise NodeUserError("Uploaded file is empty.")+ return str(value.get("filename") or "upload.bin"), blob++ if isinstance(value, dict):+ raise NodeUserError(+ "File parameter is an object but not a recognised file reference. "+ "Re-select the file in the parameter panel."+ )++ text = str(value or "").strip()+ if not text:+ raise NodeUserError("No file was provided.")+ return read_media_bytes(text, ctx=ctx, max_bytes=max_bytes)+++__all__ = [+ "AUDIO_SUBDIR",+ "UPLOAD_SUBDIR",+ "coerce_file_param",+ "read_media_bytes",+ "resolve_media",+ "workspace_file_url",+ "workspace_root",+ "write_audio",+]diff --git a/server/services/provider_registry.py b/server/services/provider_registry.pynew file mode 100644index 00000000..477b8825--- /dev/null+++ b/server/services/provider_registry.py@@ -0,0 +1,139 @@+"""Generic provider-plugin registry, shared by every provider layer.++Extracted from ``services/llm/registry.py`` when ``services/speech`` needed+the same machinery. Nothing here knows what a provider *does* — it holds a+name, a factory, and the typed SDK exceptions that layer's dispatcher must+translate. ``services/llm`` and ``services/speech`` each own a+:class:`ProviderRegistry` instance; speech owns two, one per direction.++The one subtle piece is ``sdk_exception_refs``. They are+``"module:ClassName"`` strings rather than classes so that *registering* a+provider never imports its SDK. That is load-bearing for startup time:+eagerly importing openai + anthropic + google.genai at registration cost+roughly 7 s warm and 45 s cold, and is recorded as an anti-pattern in+docs-internal/performance.md. By the time anything reads the resolved+classes, the factory has already built a client, so resolution is a+``sys.modules`` hit.+"""++from __future__ import annotations++import functools+import pkgutil+from dataclasses import dataclass, field+from typing import Any, Callable, Dict, Generic, List, Tuple, Type, TypeVar++from core.logging import get_logger+from services.plugin import NodeUserError++logger = get_logger(__name__)++ProviderT = TypeVar("ProviderT")++[email protected]_cache(maxsize=None)+def resolve_exception_ref(ref: str) -> Type[BaseException]:+ """Resolve a ``"module:ClassName"`` ref to the exception class."""+ obj = pkgutil.resolve_name(ref)+ if not (isinstance(obj, type) and issubclass(obj, BaseException)):+ raise TypeError(f"{ref!r} did not resolve to an exception class: {obj!r}")+ return obj+++@dataclass(frozen=True)+class ProviderSpec(Generic[ProviderT]):+ """Declarative spec for one provider plugin.++ Fields:+ name: registry key — matches the ``provider`` string used by nodes+ and the ``providers.<name>`` block in the layer's JSON config.+ factory: callable returning a provider instance, invoked as+ ``factory(api_key=..., **client_kwargs)``.+ sdk_exception_refs: ``"module:ClassName"`` refs naming the typed+ errors this provider's SDK raises. Refs, not classes — see the+ module docstring.+ client_kwargs: static keyword arguments merged into every+ instantiation. How OpenAI-compatible providers pin ``base_url``+ from JSON instead of from Python.+ """++ name: str+ factory: Callable[..., ProviderT]+ sdk_exception_refs: Tuple[str, ...]+ client_kwargs: Dict[str, Any] = field(default_factory=dict)++ @property+ def sdk_exception_types(self) -> Tuple[Type[BaseException], ...]:+ """Resolved typed SDK error classes, for one ``except`` clause."""+ return tuple(resolve_exception_ref(ref) for ref in self.sdk_exception_refs)+++class ProviderRegistry(Generic[ProviderT]):+ """One registry per provider layer and direction.++ ``label`` appears in log lines and error messages, so it should read+ naturally in "Unknown {label} provider: 'x'".+ """++ def __init__(self, label: str) -> None:+ self._label = label+ self._specs: Dict[str, ProviderSpec[ProviderT]] = {}++ @property+ def label(self) -> str:+ return self._label++ def register(self, spec: ProviderSpec[ProviderT]) -> None:+ """Register a provider. Idempotent for identical specs.++ Re-registration with an identical spec is a no-op so uvicorn reload+ cycles are harmless; re-registration with a *different* spec raises+ so accidental shadowing is loud rather than silent.+ """+ # Check the raw refs, NOT the resolving property — reading the+ # property here would import every SDK at registration time and+ # defeat the entire lazy-ref design.+ if not spec.sdk_exception_refs:+ raise ValueError(+ f"ProviderSpec({spec.name!r}) declares empty sdk_exception_refs. "+ "Every provider must surface its typed SDK error class so the "+ "dispatcher can translate it into NodeUserError."+ )+ existing = self._specs.get(spec.name)+ if existing is not None and existing != spec:+ raise ValueError(+ f"{self._label} provider {spec.name!r} already registered with a "+ f"different spec. Existing: {existing!r}; new: {spec!r}."+ )+ self._specs[spec.name] = spec+ logger.debug("registered provider", layer=self._label, provider=spec.name)++ def get(self, name: str) -> ProviderSpec[ProviderT]:+ """Look up a spec, raising ``NodeUserError`` on a miss.++ ``NodeUserError`` rather than ``KeyError`` so an unknown provider+ surfaces through ``BaseNode.execute()`` as one WARN line with no+ traceback — the same contract as every other user-correctable+ failure in the framework.+ """+ spec = self._specs.get(name)+ if spec is None:+ raise NodeUserError(+ f"Unknown {self._label} provider: {name!r}. "+ f"Registered providers: {sorted(self._specs)}"+ )+ return spec++ def all(self) -> List[str]:+ return sorted(self._specs)++ def has(self, name: str) -> bool:+ """Cheap membership probe — does not raise."""+ return name in self._specs++ def reset_for_tests(self) -> None:+ """Test-only. Never call from runtime code."""+ self._specs.clear()+++__all__ = ["ProviderRegistry", "ProviderSpec", "resolve_exception_ref"]diff --git a/server/tests/media/__init__.py b/server/tests/media/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/server/tests/media/test_audio_ref.py b/server/tests/media/test_audio_ref.pynew file mode 100644index 00000000..a3a266bc--- /dev/null+++ b/server/tests/media/test_audio_ref.py@@ -0,0 +1,226 @@+"""Contract for services.media — references, never bytes.++The invariants here are not stylistic. Audio bytes inside a node result+are copied at least six ways by the engine (see services/media/limits.py),+and Temporal's 2 MiB blob limit turns that into a three-retry failure with+a generic error message. ``AudioRef`` is the type that makes it impossible.+"""++from __future__ import annotations++import base64+import struct+import wave+from pathlib import Path+from types import SimpleNamespace++import pytest+from pydantic import ValidationError++from services.media import (+ AudioRef,+ coerce_file_param,+ inspect_audio,+ read_media_bytes,+ resolve_media,+ write_audio,+)+from services.media.limits import TEMPORAL_PAYLOAD_WARN_BYTES+++pytestmark = pytest.mark.unit+++def _ctx(tmp_path: Path, *, node_id: str = "tts-node-1", workflow_id: str = "wf-1"):+ return SimpleNamespace(+ node_id=node_id,+ workflow_id=workflow_id,+ workspace_dir=str(tmp_path),+ raw={"workspace_dir": str(tmp_path)},+ )+++def _wav_bytes(seconds: float = 0.25, rate: int = 8000) -> bytes:+ import io++ buffer = io.BytesIO()+ with wave.open(buffer, "wb") as handle:+ handle.setnchannels(1)+ handle.setsampwidth(2)+ handle.setframerate(rate)+ handle.writeframes(struct.pack("<h", 0) * int(rate * seconds))+ return buffer.getvalue()+++class TestAudioRefCannotCarryBytes:+ """D1 is structural, not a convention someone must remember."""++ def test_has_no_bytes_or_base64_field(self):+ forbidden = {"data", "bytes", "audio", "base64", "audio_base64", "content"}+ assert forbidden.isdisjoint(AudioRef.model_fields)++ def test_rejects_an_injected_payload_field(self):+ with pytest.raises(ValidationError):+ AudioRef(path="audio/a.wav", filename="a.wav", audio_base64="AAAA")++ def test_serializes_far_below_the_temporal_warning_threshold(self):+ import orjson++ ref = AudioRef(+ path="audio/some-fairly-long-generated-name-1a2b3c.wav",+ workflow_id="019f99e2-dc99-7cf3-9019-3ad3e6260de1",+ filename="some-fairly-long-generated-name-1a2b3c.wav",+ mime_type="audio/wav",+ format="wav",+ size_bytes=12_000_000,+ duration_seconds=312.5,+ sample_rate=44100,+ channels=2,+ sha256="a" * 64,+ url="/api/workspace/019f99e2/files/audio/some-name.wav",+ )+ size = len(orjson.dumps(ref.model_dump(mode="json")))+ assert size < 1024+ # The whole point: thousands of refs still fit inside one payload.+ assert TEMPORAL_PAYLOAD_WARN_BYTES // size > 500+++class TestWriteAudio:+ def test_round_trips_through_the_workspace(self, tmp_path):+ ctx = _ctx(tmp_path)+ payload = _wav_bytes()++ ref = write_audio(payload, ctx=ctx, stem="Hello World!", ext="wav")++ assert ref.path.startswith("audio/")+ assert not Path(ref.path).is_absolute()+ assert ref.size_bytes == len(payload)+ # Not an exact match: mimetypes consults the Windows registry, so+ # WAV resolves to audio/wav on some machines and audio/x-wav on others.+ assert ref.mime_type.startswith("audio/")+ assert read_media_bytes(ref, ctx=ctx)[1] == payload++ def test_populates_duration_from_the_container(self, tmp_path):+ ref = write_audio(_wav_bytes(seconds=0.5, rate=8000), ctx=_ctx(tmp_path), stem="clip", ext="wav")+ assert ref.duration_seconds == pytest.approx(0.5, abs=0.05)+ assert ref.sample_rate == 8000++ def test_never_collides_across_runs(self, tmp_path):+ ctx = _ctx(tmp_path)+ first = write_audio(_wav_bytes(), ctx=ctx, stem="same", ext="wav")+ second = write_audio(_wav_bytes(), ctx=ctx, stem="same", ext="wav")+ assert first.path != second.path++ def test_url_is_path_only(self, tmp_path):+ """The frontend prepends its own base for remote backends."""+ ref = write_audio(_wav_bytes(), ctx=_ctx(tmp_path), stem="x", ext="wav")+ assert ref.url and ref.url.startswith("/api/workspace/")+ assert "://" not in ref.url++ def test_refuses_empty_payload(self, tmp_path):+ from services.plugin import NodeUserError++ with pytest.raises(NodeUserError):+ write_audio(b"", ctx=_ctx(tmp_path), stem="x", ext="wav")+++class TestContainment:+ """Regression: the Sarvam STT node read the credential store."""++ @pytest.mark.parametrize(+ "attack",+ ["../../credentials.db", "..\\..\\credentials.db", "~/secrets", "C:/Windows/win.ini"],+ )+ def test_traversal_is_refused(self, tmp_path, attack):+ from services.plugin import NodeUserError++ (tmp_path.parent / "credentials.db").write_bytes(b"SECRET")+ with pytest.raises(NodeUserError):+ resolve_media(attack, ctx=_ctx(tmp_path))++ def test_absolute_path_outside_the_workspace_is_refused(self, tmp_path):+ from services.plugin import NodeUserError++ outside = tmp_path.parent / "elsewhere.wav"+ outside.write_bytes(b"nope")+ with pytest.raises(NodeUserError):+ resolve_media(str(outside), ctx=_ctx(tmp_path))++ def test_absolute_path_inside_the_workspace_still_works(self, tmp_path):+ """Back-compat for nodes that stored absolute paths pre-AudioRef."""+ ctx = _ctx(tmp_path)+ ref = write_audio(_wav_bytes(), ctx=ctx, stem="x", ext="wav")+ absolute = tmp_path / ref.path+ assert read_media_bytes(str(absolute), ctx=ctx)[1] == read_media_bytes(ref, ctx=ctx)[1]++ def test_a_ref_cannot_reach_another_workflows_workspace(self, tmp_path):+ from services.plugin import NodeUserError++ alpha, beta = tmp_path / "alpha", tmp_path / "beta"+ alpha.mkdir(), beta.mkdir()+ ref = write_audio(_wav_bytes(), ctx=_ctx(alpha, workflow_id="alpha"), stem="x", ext="wav")++ with pytest.raises(NodeUserError):+ read_media_bytes(ref, ctx=_ctx(beta, workflow_id="beta"))+++class TestCoerceFileParam:+ def test_accepts_an_audio_ref(self, tmp_path):+ ctx = _ctx(tmp_path)+ payload = _wav_bytes()+ ref = write_audio(payload, ctx=ctx, stem="x", ext="wav")++ name, blob = coerce_file_param(ref.model_dump(mode="json"), ctx=ctx)+ assert blob == payload+ assert name.endswith(".wav")++ def test_accepts_the_legacy_base64_envelope(self, tmp_path, caplog):+ """Saved workflow rows still carry this shape and are not migrated."""+ payload = b"RIFFfake"+ value = {+ "type": "upload",+ "data": base64.b64encode(payload).decode(),+ "filename": "voice.mp3",+ "mimeType": "audio/mpeg",+ }+ name, blob = coerce_file_param(value, ctx=_ctx(tmp_path))+ assert (name, blob) == ("voice.mp3", payload)++ def test_accepts_a_relative_path(self, tmp_path):+ ctx = _ctx(tmp_path)+ payload = _wav_bytes()+ ref = write_audio(payload, ctx=ctx, stem="x", ext="wav")+ assert coerce_file_param(ref.path, ctx=ctx)[1] == payload++ def test_rejects_oversize_legacy_upload(self, tmp_path):+ from services.plugin import NodeUserError++ value = {"type": "upload", "data": base64.b64encode(b"x" * 4096).decode()}+ with pytest.raises(NodeUserError, match="limit"):+ coerce_file_param(value, ctx=_ctx(tmp_path), max_bytes=1024)++ def test_rejects_empty_and_unknown_shapes(self, tmp_path):+ from services.plugin import NodeUserError++ for value in ("", None, {"kind": "video"}, {"type": "upload", "data": ""}):+ with pytest.raises(NodeUserError):+ coerce_file_param(value, ctx=_ctx(tmp_path))+++class TestInspectNeverRaises:+ """D8: a parser miss degrades billing, it does not fail a workflow."""++ def test_unknown_container_returns_an_empty_probe(self, tmp_path):+ junk = tmp_path / "mystery.xyz"+ junk.write_bytes(b"\x00\x01\x02not audio at all")+ probe = inspect_audio(junk, declared_format="xyz")+ assert probe.duration_seconds is None++ def test_missing_file_returns_an_empty_probe(self, tmp_path):+ assert inspect_audio(tmp_path / "nope.wav").duration_seconds is None++ def test_raw_pcm_duration_is_computed_arithmetically(self, tmp_path):+ raw = tmp_path / "raw.pcm"+ raw.write_bytes(b"\x00\x00" * 16000) # 16k frames, 16-bit mono+ probe = inspect_audio(raw, declared_format="linear16", pcm_sample_rate=16000)+ assert probe.duration_seconds == pytest.approx(1.0)diff --git a/server/tests/nodes/test_filesystem_backend.py b/server/tests/nodes/test_filesystem_backend.pyindex a5e550ae..e852879c 100644--- a/server/tests/nodes/test_filesystem_backend.py+++ b/server/tests/nodes/test_filesystem_backend.py@@ -225,7 +225,7 @@ def test_windows_fallback_revalidates_parent_before_writing_payload()monkeypatch.setattr(backend_module,- "_write_utf8_to_fd",+ "_write_payload_to_fd",fail_if_payload_is_written,)@@ -233,7 +233,7 @@ def test_windows_fallback_revalidates_parent_before_writing_payload(ValueError,match="parent directory changed",):- backend_module._atomic_write_text_windows(+ backend_module._atomic_write_windows(root,Path("docs/victim.txt"),"replacement",