Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
feat(integrations): add the pipecat-threews avatar processor, fix a cross-turn transcript leak
nirholas0cee019fb3bc
diff --git a/integrations/pipecat/pipecat-threews/CHANGELOG.md b/integrations/pipecat/pipecat-threews/CHANGELOG.mdnew file mode 100644index 000000000..d04996d5f--- /dev/null+++ b/integrations/pipecat/pipecat-threews/CHANGELOG.md@@ -0,0 +1,19 @@+# Changelog++All notable changes to `pipecat-threews` are documented in this file.++The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),+and this project adheres to [Semantic Versioning](https://semver.org/).++## [0.1.0] - 2026-10-13++### Added++- `ThreeWsAvatarProcessor`, a Pipecat frame processor that republishes a+ pipeline's TTS transcript, best-effort gesture cues, and user speech+ boundaries as JSON transport messages for three.ws's `<agent-3d>` web+ component to render client-side.+- `gesture_for_text`, the text heuristic used to decide a gesture cue.+- Foundational example (`examples/avatar_bot.py`) joining a local LiveKit+ room and speaking a scripted demo through a real local `espeak-ng` TTS+ service.diff --git a/integrations/pipecat/pipecat-threews/README.md b/integrations/pipecat/pipecat-threews/README.mdnew file mode 100644index 000000000..36f519b1f--- /dev/null+++ b/integrations/pipecat/pipecat-threews/README.md@@ -0,0 +1,140 @@+# pipecat-threews++A [Pipecat](https://github.com/pipecat-ai/pipecat) frame processor that publishes+[three.ws](https://three.ws) 3D avatar motion data from any Pipecat pipeline.++**Maintainer:** three.ws ([github.com/nirholas/three.ws](https://github.com/nirholas/three.ws)).+This is a community integration maintained by the three.ws team, not by Daily+(the maintainers of Pipecat).++## What it does++three.ws avatars (the `<agent-3d>` web component) render entirely on the+visitor's own device: the 3D model, its rig and its animation clip library+already live in the browser. `ThreeWsAvatarProcessor` never synthesizes video+or audio of its own. Instead it taps the TTS and user-turn frames already+flowing through your pipeline and republishes them as small JSON messages+over the active transport's data channel:++```json+{"type": "transcript", "text": "Hi, how can I help today?"}+{"type": "gesture", "name": "wave"}+{"type": "speech_started"}+{"type": "speech_ended", "transcript": "what can you do"}+```++Point your frontend at+[`<agent-3d voice="livekit">`](https://github.com/nirholas/three.ws/blob/main/src/element.js)+(or any embed that loads+[`src/runtime/livekit-voice.js`](https://github.com/nirholas/three.ws/blob/main/src/runtime/livekit-voice.js)),+which already parses this message shape over LiveKit's data channel. Any+transport whose Pipecat implementation routes `OutputTransportMessageFrame`/+`OutputTransportMessageUrgentFrame` to a real data channel (LiveKit, Daily)+works the same way.++## Installation++```bash+pip install pipecat-threews+```++## Usage with a Pipecat Pipeline++Add `ThreeWsAvatarProcessor()` after your TTS service and before+`transport.output()`, so it observes both the assistant's `TTSTextFrame`s and+the user's `UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame` /+`TranscriptionFrame`s on their way out:++```python+from pipecat.pipeline.pipeline import Pipeline+from pipecat_threews import ThreeWsAvatarProcessor++pipeline = Pipeline(+ [+ transport.input(),+ stt,+ context_aggregator.user(),+ llm,+ tts,+ ThreeWsAvatarProcessor(),+ transport.output(),+ context_aggregator.assistant(),+ ]+)+```++No API key, account or environment variable is required: the processor reads+state from frames already in your pipeline and republishes it as motion data+on whatever room or call the transport is already connected to.++**Your TTS service must be constructed with `push_start_frame=True` and+`push_stop_frames=True`.** `ThreeWsAvatarProcessor` buffers `TTSTextFrame`+text between the `TTSStartedFrame`/`TTSStoppedFrame` pair and publishes on+`TTSStoppedFrame`. Pipecat's `TTSService` base class defaults both flags to+`False`, so a TTS service built without them never emits a `TTSStoppedFrame`+and the processor silently never publishes. See `examples/avatar_bot.py`'s+`EspeakTTSService` for the correct construction.++### How it decides what to show++- **Lip sync** comes from every finished assistant turn: the full text is+ sent once (buffered between `TTSStartedFrame` and `TTSStoppedFrame`), and+ the browser times it into visemes the same way it already does for every+ other voice path in three.ws (`src/runtime/lipsync.js`), so there is+ nothing provider specific for the browser to learn.+- **Gestures** are a best-effort read of that same text+ (`pipecat_threews.gesture_for_text`): a greeting or farewell triggers+ `wave`, an agreement triggers `nod`, a disagreement triggers `shrug`. A+ miss just means no gesture fires, never a wrong one.+- **Listening pose** comes from `UserStartedSpeakingFrame` /+ `UserStoppedSpeakingFrame`, with the last finalized `TranscriptionFrame`+ attached to the `speech_ended` message.++## How to run the example++[`examples/avatar_bot.py`](examples/avatar_bot.py) is a complete,+runnable foundational example: a bot that joins a LiveKit room and speaks a+scripted demo sequence through a real local `espeak-ng` text-to-speech+service (no cloud TTS, no API keys), with `ThreeWsAvatarProcessor` wired in+to publish the matching transcript and gesture data.++1. Install `espeak-ng` (`apt install espeak-ng` or equivalent).+2. Run a local, open-source LiveKit server:++ ```bash+ docker run --rm -p 7880:7880 -p 7881:7881 -p 7882:7882/udp \+ -e LIVEKIT_KEYS="devkey: devsecret" \+ livekit/livekit-server --dev --bind 0.0.0.0+ ```++3. Install this package and its example dependencies:++ ```bash+ pip install -e ".[dev]"+ ```++4. Run the bot:++ ```bash+ LIVEKIT_URL=ws://localhost:7880 \+ LIVEKIT_API_KEY=devkey \+ LIVEKIT_API_SECRET=devsecret \+ LIVEKIT_ROOM_NAME=pipecat-threews-demo \+ python examples/avatar_bot.py+ ```++5. Join `pipecat-threews-demo` with any LiveKit client (or three.ws's own+ `<agent-3d voice="livekit">` pointed at the same server) to hear the bot+ speak and watch the avatar data arrive on the room's data channel.++## Pipecat version compatibility++Tested with Pipecat v1.12.0.++## License++Apache-2.0, matching the three.ws monorepo this package ships inside. See+the repository [LICENSE](../../../LICENSE). Apache-2.0 is a permissive+OSI-approved license and satisfies Pipecat's community-integration+requirement of "a permissive license (BSD-2 like Pipecat, or equivalent+open source terms)".diff --git a/integrations/pipecat/pipecat-threews/docs-submission/docs-json-navigation.md b/integrations/pipecat/pipecat-threews/docs-submission/docs-json-navigation.mdnew file mode 100644index 000000000..ebfc4078d--- /dev/null+++ b/integrations/pipecat/pipecat-threews/docs-submission/docs-json-navigation.md@@ -0,0 +1,23 @@+# Docs PR: docs.json navigation entry++In `docs.json`, the `Extensions` group lives at+`navigation.tabs[4].groups[1].pages[3]` and is a flat, alphabetically sorted+list of page paths. Add `"api-reference/server/extensions/pipecat-threews"`+after `pipecat-duplexjev` and before `voicemail`:++```json+{+ "group": "Extensions",+ "pages": [+ "api-reference/server/extensions/ivr",+ "api-reference/server/extensions/pipecat-backchannel",+ "api-reference/server/extensions/pipecat-duplexjev",+ "api-reference/server/extensions/pipecat-threews",+ "api-reference/server/extensions/voicemail"+ ]+}+```++No `redirects` entry is needed: this is a brand-new page with no prior URL,+and the other community extension pages (`pipecat-backchannel`,+`pipecat-duplexjev`) have none either.diff --git a/integrations/pipecat/pipecat-threews/docs-submission/pipecat-threews.mdx b/integrations/pipecat/pipecat-threews/docs-submission/pipecat-threews.mdxnew file mode 100644index 000000000..677ec7b26--- /dev/null+++ b/integrations/pipecat/pipecat-threews/docs-submission/pipecat-threews.mdx@@ -0,0 +1,115 @@+---+title: "three.ws Avatar"+description: "Publishes three.ws 3D avatar motion data (lip-sync transcript and gesture cues) from any Pipecat pipeline"+---++import { CommunityMaintained } from "/snippets/community-maintained.mdx";++<CommunityMaintained+ maintainer="nirholas"+ maintainerUrl="https://github.com/nirholas"+ repo="https://github.com/nirholas/three.ws"+/>++## Overview++`ThreeWsAvatarProcessor` drives a [three.ws](https://three.ws) 3D avatar+(the `<agent-3d>` web component) from a Pipecat pipeline. three.ws avatars+render entirely on the visitor's own device: the 3D model, its rig, and its+animation clip library already live in the browser. This processor never+synthesizes video or audio of its own. Instead it taps the TTS and+user-turn frames already flowing through your pipeline and republishes them+as small JSON messages over the active transport's data channel (lip-sync+transcript, best-effort gesture cues, and user speech boundaries), which the+three.ws client already knows how to render.++<CardGroup cols={2}>+ <Card+ title="Source Repository"+ icon="github"+ href="https://github.com/nirholas/three.ws/tree/main/integrations/pipecat/pipecat-threews"+ >+ Source code, the foundational example, and the three.ws platform this+ integration drives+ </Card>+ <Card+ title="PyPI Package"+ icon="cube"+ href="https://pypi.org/project/pipecat-threews/"+ >+ The `pipecat-threews` package on PyPI+ </Card>+</CardGroup>++## Installation++This is a community-maintained package distributed separately from+`pipecat-ai`:++```bash+pip install pipecat-threews+```++## Prerequisites++No account or API key of its own. The processor only reads frames already+flowing through your pipeline.++- **Pipecat 1.0.0 or later**: `pipecat-ai>=1.0.0`+- **A TTS service in the pipeline, constructed with `push_start_frame=True`+ and `push_stop_frames=True`**: the processor buffers `TTSTextFrame` text+ between the `TTSStartedFrame`/`TTSStoppedFrame` pair and publishes on+ `TTSStoppedFrame`. Pipecat's `TTSService` base class defaults both flags+ to `False`, so a TTS service built without them never emits a+ `TTSStoppedFrame` and the processor never publishes anything, silently.+ See `examples/avatar_bot.py`'s `EspeakTTSService` for the correct+ construction.+- **A transport with a data channel** (LiveKit, Daily): the processor+ publishes via `OutputTransportMessageUrgentFrame`, which only reaches+ clients when the transport routes transport-message frames to a real+ data channel.+- **A three.ws `<agent-3d>` embed** (or any client reading the same JSON+ shape) to render the avatar. See+ [`src/runtime/livekit-voice.js`](https://github.com/nirholas/three.ws/blob/main/src/runtime/livekit-voice.js)+ in the three.ws repository.++## Configuration++Constructor arguments for `ThreeWsAvatarProcessor`.++<ParamField path="topic" type="str" default="threews.avatar">+ The transport data-channel topic to publish on. Only meaningful for+ transports (like LiveKit) whose data messages carry a topic string;+ transports without one ignore it.+</ParamField>++## Usage++Place one instance after your TTS service and before `transport.output()`,+so it observes both the assistant's `TTSTextFrame`s and the user's+`UserStartedSpeakingFrame` / `UserStoppedSpeakingFrame` /+`TranscriptionFrame`s on their way out:++```python+from pipecat.pipeline.pipeline import Pipeline+from pipecat_threews import ThreeWsAvatarProcessor++pipeline = Pipeline(+ [+ transport.input(),+ stt,+ context_aggregator.user(),+ llm,+ tts,+ ThreeWsAvatarProcessor(),+ transport.output(),+ context_aggregator.assistant(),+ ]+)+```++## Compatibility++Tested with Pipecat v1.12.0. Check the [source+repository](https://github.com/nirholas/three.ws/tree/main/integrations/pipecat/pipecat-threews)+for the latest tested version and changelog.diff --git a/integrations/pipecat/pipecat-threews/docs-submission/supported-services-row.md b/integrations/pipecat/pipecat-threews/docs-submission/supported-services-row.mdnew file mode 100644index 000000000..9e68d185b--- /dev/null+++ b/integrations/pipecat/pipecat-threews/docs-submission/supported-services-row.md@@ -0,0 +1,21 @@+# Docs PR: supported-services.mdx row++Add this row to the `## Extensions` table in+`api-reference/server/services/supported-services.mdx` (alphabetical by+link text, after "Pipecat Backchannel"):++```+| [three.ws Avatar](/api-reference/server/extensions/pipecat-threews) | `uv add pipecat-threews` | Community |+```++Full table after the addition:++```+| Extension | Setup | Maintainer |+| --------------------------------------------------------------------------- | ---------------------------- | ---------- |+| [DuplexJev Turn Analyzer](/api-reference/server/extensions/pipecat-duplexjev) | `uv add pipecat-duplexjev` | Community |+| [IVRNavigator](/api-reference/server/extensions/ivr) | Built in | Pipecat |+| [Pipecat Backchannel](/api-reference/server/extensions/pipecat-backchannel) | `uv add pipecat-backchannel` | Community |+| [three.ws Avatar](/api-reference/server/extensions/pipecat-threews) | `uv add pipecat-threews` | Community |+| [VoicemailDetector](/api-reference/server/extensions/voicemail) | Built in | Pipecat |+```diff --git a/integrations/pipecat/pipecat-threews/examples/avatar_bot.py b/integrations/pipecat/pipecat-threews/examples/avatar_bot.pynew file mode 100644index 000000000..2f7b5d52b--- /dev/null+++ b/integrations/pipecat/pipecat-threews/examples/avatar_bot.py@@ -0,0 +1,166 @@+#+# Copyright (c) 2026, three.ws+#+# SPDX-License-Identifier: Apache-2.0+#++"""Foundational example: a Pipecat bot publishing three.ws avatar data.++Speech-to-text and the LLM are skipped on purpose: this example proves the+``ThreeWsAvatarProcessor`` end to end (a real local TTS speaks, the+processor republishes synced transcript and gesture data over the room's+data channel) without needing a cloud account for any leg of the pipeline.++Text-to-speech is a thin wrapper around the ``espeak-ng`` command line+(``EspeakTTSService`` below), a real open-source offline synthesizer, not a+mock: every audio frame the room receives came out of an actual+``espeak-ng`` subprocess.++Run it against a local open-source ``livekit-server``+(see ``../../../livekit/livekit-plugins-threews/README.md`` for the docker+command) with ``LIVEKIT_URL``, ``LIVEKIT_API_KEY`` and ``LIVEKIT_API_SECRET``+set in the environment:++ python avatar_bot.py+"""++import asyncio+import io+import shutil+import wave+from collections.abc import AsyncGenerator++from loguru import logger+from pipecat.frames.frames import EndFrame, ErrorFrame, Frame, TTSAudioRawFrame, TTSSpeakFrame+from pipecat.pipeline.pipeline import Pipeline+from pipecat.pipeline.runner import PipelineRunner+from pipecat.pipeline.task import PipelineParams, PipelineTask+from pipecat.runner.livekit import configure+from pipecat.services.tts_service import TTSService+from pipecat.transports.livekit.transport import LiveKitParams, LiveKitTransport++from pipecat_threews import ThreeWsAvatarProcessor++ESPEAK_SAMPLE_RATE = 22050+DEMO_LINES = [+ "Hello! I'm the three.ws avatar, speaking through Pipecat.",+ "Yes, this processor publishes a gesture cue with that line.",+ "Goodbye, and thanks for watching the demo!",+]+++class EspeakTTSService(TTSService):+ """Offline TTS backed by the ``espeak-ng`` binary.++ No network call, no API key: every utterance is synthesized locally by+ shelling out to ``espeak-ng --stdout``, which writes a WAV file this+ service decodes into the PCM frames Pipecat's audio pipeline expects.+ """++ def __init__(self, *, voice: str = "en-us", rate_wpm: int = 165, **kwargs) -> None:+ """Initialize the service.++ Args:+ voice: The ``espeak-ng`` voice name, forwarded to its ``-v`` flag.+ rate_wpm: Speech rate in words per minute, forwarded to ``-s``.+ **kwargs: Additional arguments passed to the parent ``TTSService``.+ """+ super().__init__(+ sample_rate=ESPEAK_SAMPLE_RATE,+ push_start_frame=True,+ push_stop_frames=True,+ **kwargs,+ )+ if shutil.which("espeak-ng") is None:+ raise RuntimeError(+ "espeak-ng is not installed; install it (e.g. `apt install espeak-ng`) "+ "to run this example"+ )+ self._voice = voice+ self._rate_wpm = rate_wpm++ def can_generate_metrics(self) -> bool:+ """Check if this service can generate processing metrics.++ Returns:+ True, as this service supports metrics.+ """+ return True++ async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]:+ """Synthesize ``text`` with ``espeak-ng`` and yield PCM audio frames.++ Args:+ text: The text to synthesize into speech.+ context_id: The context ID for tracking audio frames.++ Yields:+ Frame: Audio frames containing the synthesized speech.+ """+ try:+ proc = await asyncio.create_subprocess_exec(+ "espeak-ng",+ "-v",+ self._voice,+ "-s",+ str(self._rate_wpm),+ "--stdout",+ text,+ stdout=asyncio.subprocess.PIPE,+ stderr=asyncio.subprocess.PIPE,+ )+ wav_bytes, stderr = await proc.communicate()+ if proc.returncode != 0:+ raise RuntimeError(f"espeak-ng failed: {stderr.decode(errors='replace')}")++ with wave.open(io.BytesIO(wav_bytes), "rb") as wav_file:+ pcm = wav_file.readframes(wav_file.getnframes())++ await self.start_tts_usage_metrics(text)++ chunk_size = self.sample_rate // 5 * 2 # ~200ms of 16-bit mono PCM per chunk+ for offset in range(0, len(pcm), chunk_size):+ chunk = pcm[offset : offset + chunk_size]+ if chunk:+ await self.stop_ttfb_metrics()+ yield TTSAudioRawFrame(chunk, self.sample_rate, 1, context_id=context_id)+ except Exception as error:+ logger.exception("espeak-ng synthesis failed")+ yield ErrorFrame(error=f"espeak-ng TTS error: {error}")+++async def main() -> None:+ """Join a local LiveKit room and speak a scripted demo sequence."""+ url, token, room_name = await configure()++ transport = LiveKitTransport(+ url,+ token,+ room_name,+ params=LiveKitParams(audio_out_enabled=True),+ )++ tts = EspeakTTSService()+ avatar = ThreeWsAvatarProcessor()++ pipeline = Pipeline([tts, avatar, transport.output()])+ task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))++ async def speak_demo_lines() -> None:+ # Queuing every line at once would let the TTS service (near-instant+ # for espeak-ng) race through all three turns before any audio+ # actually plays, publishing avatar data for the whole script in a+ # single burst. A real conversational bot is never this instant: a+ # turn takes as long as it takes to actually speak before the next+ # one starts. Pace it the same way so the demo matches real usage.+ for line in DEMO_LINES:+ await task.queue_frames([TTSSpeakFrame(line)])+ await asyncio.sleep(2)+ await task.queue_frames([EndFrame()])++ runner = PipelineRunner()+ await asyncio.gather(runner.run(task), speak_demo_lines())+++if __name__ == "__main__":+ asyncio.run(main())diff --git a/integrations/pipecat/pipecat-threews/pyproject.toml b/integrations/pipecat/pipecat-threews/pyproject.tomlnew file mode 100644index 000000000..3b415c79a--- /dev/null+++ b/integrations/pipecat/pipecat-threews/pyproject.toml@@ -0,0 +1,68 @@+[build-system]+requires = ["setuptools>=68"]+build-backend = "setuptools.build_meta"++[project]+name = "pipecat-threews"+version = "0.1.0"+description = "Pipecat frame processor publishing three.ws 3D avatar motion data"+readme = "README.md"+license = { text = "Apache-2.0" }+requires-python = ">=3.10"+authors = [{ name = "three.ws", email = "[email protected]" }]+keywords = ["pipecat", "voice", "ai", "realtime", "avatar", "3d", "threews"]+classifiers = [+ "Intended Audience :: Developers",+ "License :: OSI Approved :: Apache Software License",+ "Topic :: Multimedia :: Sound/Audio",+ "Topic :: Scientific/Engineering :: Artificial Intelligence",+ "Programming Language :: Python :: 3",+ "Programming Language :: Python :: 3.10",+ "Programming Language :: Python :: 3 :: Only",+]+dependencies = [+ "pipecat-ai>=1.0.0",+]++[project.urls]+Documentation = "https://three.ws/docs"+Website = "https://three.ws/"+Source = "https://github.com/nirholas/three.ws"+Changelog = "https://github.com/nirholas/three.ws/blob/main/integrations/pipecat/pipecat-threews/CHANGELOG.md"++[tool.setuptools.packages.find]+where = ["src"]++# Mirrors pipecat-ai/pipecat's own [tool.ruff] block (their pyproject.toml), the+# config their CONTRIBUTING.md asks every community integration to pass.+[tool.ruff]+exclude = [".git"]+line-length = 100++[tool.ruff.lint]+select = [+ "ASYNC", # Blocking calls and other misuse inside async functions+ "B", # Likely bugs and design problems+ "D", # Docstring rules+ "F", # Pyflakes+ "I", # Import rules+ "RUF006", # asyncio tasks nothing holds a reference to+ "UP", # Modern Python+]+ignore = [+ "ASYNC109",+ "ASYNC240",+ "ASYNC110",+ "B024",+ "B027",+ "B007",+ "B904",+ "B905",+ "D105",+]++[tool.ruff.lint.per-file-ignores]+"examples/*" = ["D"]++[tool.ruff.lint.pydocstyle]+convention = "google"diff --git a/integrations/pipecat/pipecat-threews/src/pipecat_threews/__init__.py b/integrations/pipecat/pipecat-threews/src/pipecat_threews/__init__.pynew file mode 100644index 000000000..4840dc924--- /dev/null+++ b/integrations/pipecat/pipecat-threews/src/pipecat_threews/__init__.py@@ -0,0 +1,13 @@+#+# Copyright (c) 2026, three.ws+#+# SPDX-License-Identifier: Apache-2.0+#++"""Pipecat frame processor publishing three.ws 3D avatar motion data."""++from .avatar import ThreeWsAvatarProcessor, gesture_for_text++__all__ = ["ThreeWsAvatarProcessor", "gesture_for_text"]++__version__ = "0.1.0"diff --git a/integrations/pipecat/pipecat-threews/src/pipecat_threews/avatar.py b/integrations/pipecat/pipecat-threews/src/pipecat_threews/avatar.pynew file mode 100644index 000000000..2bb7d8033--- /dev/null+++ b/integrations/pipecat/pipecat-threews/src/pipecat_threews/avatar.py@@ -0,0 +1,166 @@+#+# Copyright (c) 2026, three.ws+#+# SPDX-License-Identifier: Apache-2.0+#++"""Frame processor publishing three.ws avatar motion data from a pipeline."""++import re++from pipecat.frames.frames import (+ Frame,+ OutputTransportMessageUrgentFrame,+ TranscriptionFrame,+ TTSStartedFrame,+ TTSStoppedFrame,+ TTSTextFrame,+ UserStartedSpeakingFrame,+ UserStoppedSpeakingFrame,+)+from pipecat.processors.frame_processor import FrameDirection, FrameProcessor++DEFAULT_TOPIC = "threews.avatar"++# Gesture cues are a best-effort read of the assistant's own final text, the+# same heuristic family three.ws's browser widget already uses for lip sync+# (src/runtime/lipsync.js) and the one its LiveKit plugin uses+# (livekit-plugins-threews): no NLU model, just the opening/closing words a+# turn is likely to use. A miss just means no gesture fires, never a wrong+# one, so the pattern favors precision over recall.+_GREETING_RE = re.compile(+ r"^\s*(hi|hello|hey there|hey|greetings|good (morning|afternoon|evening))\b",+ re.IGNORECASE,+)+_FAREWELL_RE = re.compile(+ r"\b(goodbye|bye for now|take care|farewell)[.!]?\s*$",+ re.IGNORECASE,+)+_AGREEMENT_RE = re.compile(+ r"^\s*(yes|yeah|yep|absolutely|definitely|of course)\b",+ re.IGNORECASE,+)+_DISAGREEMENT_RE = re.compile(+ r"^\s*(no|nope|not quite|not really|unfortunately)\b",+ re.IGNORECASE,+)+++def gesture_for_text(text: str) -> str | None:+ """Best-effort gesture slot name for a line of assistant speech.++ Args:+ text: The assistant's finished turn, as spoken by the TTS service.++ Returns:+ One of the animation slot names three.ws avatars already know+ (``wave``, ``nod``, ``shrug``), matched against+ ``src/runtime/animation-slots.js`` in the three.ws client, or+ ``None`` if the text does not read like a greeting, farewell,+ agreement or disagreement.+ """+ if _GREETING_RE.search(text) or _FAREWELL_RE.search(text):+ return "wave"+ if _AGREEMENT_RE.search(text):+ return "nod"+ if _DISAGREEMENT_RE.search(text):+ return "shrug"+ return None+++class ThreeWsAvatarProcessor(FrameProcessor):+ """Publishes three.ws avatar motion data alongside a Pipecat pipeline.++ three.ws avatars (the ``<agent-3d>`` web component) render entirely on+ the visitor's own device: the 3D model, its rig and its animation clip+ library already live in the browser. This processor never synthesizes+ video or audio of its own. Instead it taps the TTS and user-turn frames+ already flowing through the pipeline and republishes them as small JSON+ messages via ``OutputTransportMessageUrgentFrame``, which any transport+ with a data channel (LiveKit, Daily, a WebSocket transport) delivers to+ connected clients:++ - ``{"type": "transcript", "text": "..."}`` for every finished assistant+ turn, which the browser times into visemes locally+ (``src/runtime/lipsync.js``).+ - ``{"type": "gesture", "name": "wave" | "nod" | "shrug"}`` when the+ turn's text reads like a greeting, farewell, agreement or+ disagreement.+ - ``{"type": "speech_started"}`` / ``{"type": "speech_ended", "transcript": "..."}``+ around the user's own turns, so the avatar can switch to a listening+ pose.++ Place one instance late in the pipeline, after the TTS service and+ before ``transport.output()``, so it observes both the assistant's+ ``TTSTextFrame`` frames and the user's ``UserStartedSpeakingFrame``,+ ``UserStoppedSpeakingFrame`` and ``TranscriptionFrame`` frames on their+ way out.++ Example::++ pipeline = Pipeline(+ [+ transport.input(),+ stt,+ context_aggregator.user(),+ llm,+ tts,+ ThreeWsAvatarProcessor(),+ transport.output(),+ context_aggregator.assistant(),+ ]+ )+ """++ def __init__(self, *, topic: str = DEFAULT_TOPIC) -> None:+ """Initialize the processor.++ Args:+ topic: The transport data-channel topic to publish on. Defaults+ to ``"threews.avatar"``, the topic three.ws's LiveKit client+ (``src/runtime/livekit-voice.js``) already subscribes to.+ """+ super().__init__()+ self._topic = topic+ self._tts_text_buffer = ""+ self._speaking_tts = False+ self._last_user_transcript = ""++ async def process_frame(self, frame: Frame, direction: FrameDirection) -> None:+ """Observe TTS and user-turn frames, then forward every frame unchanged.++ Args:+ frame: The frame being processed.+ direction: The direction the frame is flowing in the pipeline.+ """+ await super().process_frame(frame, direction)++ if isinstance(frame, TTSStartedFrame):+ self._speaking_tts = True+ self._tts_text_buffer = ""+ elif isinstance(frame, TTSTextFrame) and self._speaking_tts:+ self._tts_text_buffer += frame.text+ elif isinstance(frame, TTSStoppedFrame) and self._speaking_tts:+ self._speaking_tts = False+ text = self._tts_text_buffer.strip()+ self._tts_text_buffer = ""+ if text:+ await self._publish({"type": "transcript", "text": text})+ gesture = gesture_for_text(text)+ if gesture:+ await self._publish({"type": "gesture", "name": gesture})+ elif isinstance(frame, TranscriptionFrame) and frame.finalized:+ self._last_user_transcript = frame.text+ elif isinstance(frame, UserStartedSpeakingFrame):+ self._last_user_transcript = ""+ await self._publish({"type": "speech_started"})+ elif isinstance(frame, UserStoppedSpeakingFrame):+ await self._publish({"type": "speech_ended", "transcript": self._last_user_transcript})++ await self.push_frame(frame, direction)++ async def _publish(self, payload: dict) -> None:+ await self.push_frame(+ OutputTransportMessageUrgentFrame(message=payload),+ FrameDirection.DOWNSTREAM,+ )diff --git a/integrations/pipecat/pipecat-threews/tests/__init__.py b/integrations/pipecat/pipecat-threews/tests/__init__.pynew file mode 100644index 000000000..fc16b7f6a--- /dev/null+++ b/integrations/pipecat/pipecat-threews/tests/__init__.py@@ -0,0 +1 @@+"""Tests for pipecat-threews."""diff --git a/integrations/pipecat/pipecat-threews/tests/test_avatar.py b/integrations/pipecat/pipecat-threews/tests/test_avatar.pynew file mode 100644index 000000000..582979616--- /dev/null+++ b/integrations/pipecat/pipecat-threews/tests/test_avatar.py@@ -0,0 +1,153 @@+# Copyright 2026 three.ws+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Regression test for ``ThreeWsAvatarProcessor``'s TTS-turn publish contract.++A real deployment ships a demo bot (``examples/avatar_bot.py``) that was+found, via a real local LiveKit run, to publish an assistant turn's gesture+cue only if the TTS service is constructed with ``push_stop_frames=True``+(Pipecat's ``TTSService`` defaults it to ``False``, so ``TTSStoppedFrame``+never fires and the processor never publishes, silently). This test drives+the processor directly with a real ``TTSStartedFrame`` /+``TTSTextFrame`` / ``TTSStoppedFrame`` sequence through Pipecat's own test+pipeline (``pipecat.tests.utils.run_test``), so a regression in the+processor's own frame handling, independent of any particular TTS service's+construction, fails this test rather than silently publishing nothing.+"""++from __future__ import annotations++import pytest+from pipecat.frames.frames import (+ AggregationType,+ OutputTransportMessageUrgentFrame,+ TranscriptionFrame,+ TTSStartedFrame,+ TTSStoppedFrame,+ TTSTextFrame,+ UserStartedSpeakingFrame,+ UserStoppedSpeakingFrame,+)+from pipecat.tests.utils import SleepFrame, run_test++from pipecat_threews.avatar import ThreeWsAvatarProcessor+++async def test_finished_tts_turn_publishes_transcript_and_gesture():+ """A full TTSStarted/TTSText/TTSStopped turn must publish both messages."""+ processor = ThreeWsAvatarProcessor()++ down_frames, _ = await run_test(+ processor,+ frames_to_send=[+ TTSStartedFrame(),+ TTSTextFrame("Hello! ", aggregated_by=AggregationType.SENTENCE),+ TTSTextFrame("I'm the three.ws avatar.", aggregated_by=AggregationType.SENTENCE),+ TTSStoppedFrame(),+ ],+ )++ published = [f.message for f in down_frames if isinstance(f, OutputTransportMessageUrgentFrame)]+ assert published == [+ {"type": "transcript", "text": "Hello! I'm the three.ws avatar."},+ {"type": "gesture", "name": "wave"},+ ]+++async def test_tts_text_outside_a_turn_is_not_published():+ """Without a TTSStoppedFrame, no buffered text is ever published."""+ processor = ThreeWsAvatarProcessor()++ down_frames, _ = await run_test(+ processor,+ frames_to_send=[+ TTSStartedFrame(),+ TTSTextFrame("Hello there", aggregated_by=AggregationType.SENTENCE),+ ],+ )++ published = [f for f in down_frames if isinstance(f, OutputTransportMessageUrgentFrame)]+ assert published == []+++async def test_user_speech_boundaries_publish_listen_events():+ """User VAD boundaries publish speech_started/speech_ended with the transcript.++ ``UserStartedSpeakingFrame``/``UserStoppedSpeakingFrame`` are Pipecat+ ``SystemFrame``s, processed immediately on a fast path that bypasses the+ processor's regular frame queue; ``TranscriptionFrame`` is not, so it+ queues behind other work on a separately-scheduled task. Sent back to+ back with no gap, ``UserStoppedSpeakingFrame`` can finish processing+ before the ``TranscriptionFrame`` just ahead of it does, so the+ ``SleepFrame`` here (Pipecat's own test-only frame for this exact+ purpose) gives the regular queue a chance to drain first, matching how a+ real STT service's finalized transcript has normally already landed by+ the time VAD reports the user stopped talking.+ """+ processor = ThreeWsAvatarProcessor()++ down_frames, _ = await run_test(+ processor,+ frames_to_send=[+ UserStartedSpeakingFrame(),+ TranscriptionFrame(+ text="what can you do", user_id="user-1", timestamp="", finalized=True+ ),+ SleepFrame(sleep=0.05),+ UserStoppedSpeakingFrame(),+ ],+ )++ published = [f.message for f in down_frames if isinstance(f, OutputTransportMessageUrgentFrame)]+ assert published == [+ {"type": "speech_started"},+ {"type": "speech_ended", "transcript": "what can you do"},+ ]+++async def test_user_stopped_speaking_does_not_leak_prior_turn_transcript():+ """A turn with no finalized transcript yet must not report the previous turn's.++ ``_last_user_transcript`` is set whenever a finalized ``TranscriptionFrame``+ arrives and, before this test's fix, was never cleared. If a second+ turn's own transcript is slow to finalize (the STT race described in+ ``test_user_speech_boundaries_publish_listen_events``), the processor+ must not attribute the first turn's leftover text to the second turn's+ ``speech_ended``.+ """+ processor = ThreeWsAvatarProcessor()++ down_frames, _ = await run_test(+ processor,+ frames_to_send=[+ UserStartedSpeakingFrame(),+ TranscriptionFrame(text="first turn", user_id="user-1", timestamp="", finalized=True),+ SleepFrame(sleep=0.05),+ UserStoppedSpeakingFrame(),+ UserStartedSpeakingFrame(),+ UserStoppedSpeakingFrame(),+ ],+ )++ published = [f.message for f in down_frames if isinstance(f, OutputTransportMessageUrgentFrame)]+ assert published == [+ {"type": "speech_started"},+ {"type": "speech_ended", "transcript": "first turn"},+ {"type": "speech_started"},+ {"type": "speech_ended", "transcript": ""},+ ]diff --git a/tests/e2e/agent-3d-pipecat-avatar.spec.js b/tests/e2e/agent-3d-pipecat-avatar.spec.jsnew file mode 100644index 000000000..3b0baa92a--- /dev/null+++ b/tests/e2e/agent-3d-pipecat-avatar.spec.js@@ -0,0 +1,152 @@+import { test, expect } from '@playwright/test';+import { spawn } from 'node:child_process';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { SignJWT } from 'jose';++// Work order 087: proves the Pipecat `pipecat-threews` package end to end,+// browser side, the same way agent-3d-livekit-avatar.spec.js proves the+// LiveKit plugin. A real Pipecat pipeline (integrations/pipecat/+// pipecat-threews/examples/avatar_bot.py) joins the same kind of real local+// open-source livekit-server, speaks three lines through a real espeak-ng+// TTS service, and `ThreeWsAvatarProcessor` republishes them as+// `transcript`/`gesture` data-channel messages that a real Chromium loading+// `<agent-3d voice="livekit">` renders through the exact same browser code+// path (LiveKitVoice never knows which backend published the message).+//+// `video: 'on'` is set for this spec only so the run also produces the demo+// video pipecat-ai/pipecat's COMMUNITY_INTEGRATIONS.md asks every community+// integration PR to link (saved under the gitignored test-results/, not+// committed; the submission doc records the local path for the owner to+// attach when the PR is actually opened).++const ROOM = `e2e-pipecat-${Date.now()}`;+const LIVEKIT_URL = process.env.LIVEKIT_URL || 'ws://localhost:7880';+const LIVEKIT_API_KEY = process.env.LIVEKIT_API_KEY || 'devkey';+const LIVEKIT_API_SECRET = process.env.LIVEKIT_API_SECRET || 'devsecret';++const HERE = path.dirname(fileURLToPath(import.meta.url));+const EXAMPLES_DIR = path.join(+ HERE,+ '..',+ '..',+ 'integrations',+ 'pipecat',+ 'pipecat-threews',+ 'examples',+);++async function mintBrowserToken(room) {+ const secret = new TextEncoder().encode(LIVEKIT_API_SECRET);+ const now = Math.floor(Date.now() / 1000);+ return new SignJWT({+ video: { roomJoin: true, room, canPublish: true, canSubscribe: true },+ sub: 'e2e-browser',+ iss: LIVEKIT_API_KEY,+ nbf: now,+ exp: now + 3600,+ })+ .setProtectedHeader({ alg: 'HS256' })+ .sign(secret);+}++function spawnBot(room) {+ const child = spawn('python3', ['-u', 'avatar_bot.py', '--room', room], {+ cwd: EXAMPLES_DIR,+ env: {+ ...process.env,+ LIVEKIT_URL,+ LIVEKIT_API_KEY,+ LIVEKIT_API_SECRET,+ },+ stdio: ['ignore', 'pipe', 'pipe'],+ });+ let log = '';+ child.stdout.on('data', (d) => (log += d.toString()));+ child.stderr.on('data', (d) => (log += d.toString()));+ return { child, getLog: () => log };+}++test.use({ video: 'on' });++test.describe('agent-3d Pipecat avatar processor (real local livekit-server)', () => {+ test.setTimeout(120_000);++ test('a real Pipecat bot speaking in the room drives the browser avatar lipsync + gesture', async ({+ page,+ }) => {+ // The bot speaks its scripted lines immediately on connecting, with no+ // wait for a participant to join (unlike the LiveKit AgentSession demo,+ // whose session startup latency happens to give the browser time to+ // join first). So the browser must be fully joined to the room, not+ // just past protocol-object creation, before the bot is spawned at all,+ // or its first lines race the browser's own room.connect() and are lost.+ let bot;+ try {+ const token = await mintBrowserToken(ROOM);+ await page.route('**/api/agents/*/livekit-token', (route) =>+ route.fulfill({+ status: 200,+ contentType: 'application/json',+ body: JSON.stringify({ token, serverUrl: LIVEKIT_URL }),+ }),+ );++ const errors = [];+ page.on('pageerror', (err) => errors.push(String(err?.message || err)));++ await page.goto('/agent-3d-livekit-harness.html', { waitUntil: 'domcontentloaded' });++ await expect+ .poll(() => page.evaluate(() => document.getElementById('a3d')?._livekitVoice?.connected === true), {+ timeout: 60_000,+ })+ .toBe(true);++ // Attach the collectors with an awaited evaluate (not a long-lived+ // in-page Promise) so this call only resolves once the listeners are+ // actually registered, guaranteeing they're in place before the bot+ // (spawned right after) can publish anything.+ await page.evaluate(() => {+ window.__threewsEvents = [];+ const protocol = document.getElementById('a3d')._avatar.protocol;+ protocol.on('speak', (action) =>+ window.__threewsEvents.push({ type: 'speak', text: action.payload?.text }),+ );+ protocol.on('gesture', (action) =>+ window.__threewsEvents.push({ type: 'gesture', name: action.payload?.name }),+ );+ });++ bot = spawnBot(ROOM);++ await expect+ .poll(+ () =>+ page.evaluate(+ () => window.__threewsEvents.filter((e) => e.type === 'speak').length,+ ),+ { timeout: 60_000 },+ )+ .toBeGreaterThanOrEqual(3);++ // Hold on the final frame for a moment so the saved video has a+ // visible tail after the last gesture instead of cutting off mid-clip.+ await page.waitForTimeout(2000);++ const events = await page.evaluate(() => window.__threewsEvents);+ const speaks = events.filter((e) => e.type === 'speak').map((e) => e.text);+ const gestures = events.filter((e) => e.type === 'gesture').map((e) => e.name);++ expect(speaks, `bot log:\n${bot.getLog()}`).toEqual([+ "Hello! I'm the three.ws avatar, speaking through Pipecat.",+ 'Yes, this processor publishes a gesture cue with that line.',+ 'Goodbye, and thanks for watching the demo!',+ ]);+ expect(gestures).toEqual(['wave', 'nod']);+ expect(errors, `page errors: ${errors.join(' | ')}`).toEqual([]);+ } finally {+ bot?.child.kill('SIGTERM');+ }+ });+});