Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

feat(integrations): add the livekit-plugins-threews avatar plugin, verified with a real local lipsync demo

nirholas55350b50d993
diff --git a/.gitignore b/.gitignore
index 0f3b79e4c..a1c125cc8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -259,6 +259,9 @@ snapshots/exported/
# Python bytecode (workers/)
__pycache__/
*.pyc
+*.egg-info/
+.ruff_cache/
+.pytest_cache/
# three.ws presence sweep — regenerate with `npm run presence`
three-ws-presence.json
diff --git a/integrations/livekit/livekit-plugins-threews/README.md b/integrations/livekit/livekit-plugins-threews/README.md
new file mode 100644
index 000000000..ea9435166
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/README.md
@@ -0,0 +1,65 @@
+# LiveKit Plugins three.ws
+
+Agent Framework avatar plugin for [three.ws](https://three.ws) 3D avatars.
+
+## Client-side rendering
+
+three.ws avatars render entirely on the visitor's own device: the `<agent-3d>`
+web component already ships the humanoid model, its rig and its animation
+clip library in the browser. This plugin never synthesizes a video track.
+Instead it publishes small JSON messages over the LiveKit data channel
+(topic `threews.avatar`) that drive the client's existing lip-sync and
+gesture systems:
+
+```json
+{"type": "transcript", "text": "Hi, how can I help today?"}
+{"type": "gesture", "name": "wave"}
+{"type": "speech_started"}
+{"type": "speech_ended", "transcript": "what can you do"}
+```
+
+A standard LiveKit video renderer attached to the agent's room sees no video
+track from this plugin, because there is none to see: a three.ws avatar is
+not server-rendered. Point your frontend at
+[`<agent-3d voice="livekit">`](https://github.com/nirholas/three.ws/blob/main/src/element.js)
+(or any embed that loads `src/runtime/livekit-voice.js`), which already
+parses this message shape.
+
+## Installation
+
+```bash
+pip install livekit-plugins-threews
+```
+
+## Usage
+
+```python
+from livekit.plugins import threews
+
+avatar = threews.AvatarSession()
+await avatar.start(session, room=ctx.room)
+```
+
+No API key, account or environment variable is required: the plugin reads
+state from the `AgentSession` you already built (its conversation items and
+user turn-taking events) and republishes it as motion data on the room the
+agent is already in.
+
+## How it decides what to show
+
+- **Lip sync** comes from every finished assistant turn
+ (`conversation_item_added`): the full text is sent once, and the browser
+ times it into visemes the same way it already does for every other voice
+ path in three.ws (`src/runtime/lipsync.js`), so there is nothing provider
+ specific for the browser to learn.
+- **Gestures** are a best-effort read of that same text
+ (`threews.gesture_for_text`): a greeting or farewell triggers `wave`, an
+ agreement triggers `nod`, a disagreement triggers `shrug`. A miss just
+ means no gesture fires, never a wrong one.
+- **Listening pose** comes from `user_state_changed`: the avatar gets
+ `speech_started` / `speech_ended` events around the user's own turns.
+
+See [`examples/avatar_agent.py`](examples/avatar_agent.py) for a complete
+voice agent wired to this plugin, and
+[`../../README.md`](../../README.md) for how to run it end to end against a
+local `livekit-server`.
diff --git a/integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py b/integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py
new file mode 100644
index 000000000..7d865b1f2
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py
@@ -0,0 +1,159 @@
+# Copyright 2026 three.ws
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A minimal voice agent wired to the three.ws avatar plugin.
+
+Speech-to-text and the LLM are skipped on purpose: this example proves the
+plugin end to end (a real local TTS speaks, the avatar plugin publishes
+synced transcript and gesture data) without needing a cloud account for any
+leg of the pipeline, per the plugin's own README.
+
+Text-to-speech is a thin wrapper around the ``espeak-ng`` command line
+(``EspeakTTS`` below), a real open-source offline synthesizer, not a mock:
+every audio frame the room receives came out of an actual ``espeak-ng``
+subprocess.
+
+Run it against a local open-source ``livekit-server``
+(see ``../../README.md`` for the docker command) with:
+
+ python avatar_agent.py connect --room demo-room
+"""
+
+from __future__ import annotations
+
+import asyncio
+import io
+import shutil
+import wave
+from dataclasses import dataclass
+
+from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli
+from livekit.agents.tts import TTS, ChunkedStream, TTSCapabilities
+from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS, APIConnectOptions
+from livekit.plugins import threews
+
+ESPEAK_SAMPLE_RATE = 22050
+
+
+@dataclass
+class _EspeakVoice:
+ """``espeak-ng`` voice selection, forwarded to its ``-v`` flag."""
+
+ name: str = "en-us"
+ rate_wpm: int = 165
+
+
+class EspeakTTS(TTS):
+ """Offline TTS backed by the ``espeak-ng`` binary.
+
+ No network call, no API key: every utterance is synthesized locally by
+ shelling out to ``espeak-ng --stdout``, which writes a WAV file we decode
+ into PCM frames for the LiveKit audio pipeline.
+ """
+
+ def __init__(self, *, voice: _EspeakVoice | None = None) -> None:
+ super().__init__(
+ capabilities=TTSCapabilities(streaming=False),
+ sample_rate=ESPEAK_SAMPLE_RATE,
+ num_channels=1,
+ )
+ if shutil.which("espeak-ng") is None:
+ raise RuntimeError(
+ "espeak-ng is not installed; install it (e.g. `apt install espeak-ng`) "
+ "to run this example"
+ )
+ self._voice = voice or _EspeakVoice()
+
+ @property
+ def provider(self) -> str:
+ return "espeak-ng"
+
+ @property
+ def model(self) -> str:
+ return self._voice.name
+
+ def synthesize(
+ self, text: str, *, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS
+ ) -> ChunkedStream:
+ return _EspeakChunkedStream(tts=self, input_text=text, conn_options=conn_options)
+
+
+class _EspeakChunkedStream(ChunkedStream):
+ async def _run(self, output_emitter) -> None: # noqa: ANN001
+ assert isinstance(self._tts, EspeakTTS)
+ voice = self._tts._voice
+
+ proc = await asyncio.create_subprocess_exec(
+ "espeak-ng",
+ "-v",
+ voice.name,
+ "-s",
+ str(voice.rate_wpm),
+ "--stdout",
+ self._input_text,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.PIPE,
+ )
+ wav_bytes, stderr = await proc.communicate()
+ if proc.returncode != 0:
+ raise RuntimeError(f"espeak-ng failed: {stderr.decode(errors='replace')}")
+
+ with wave.open(io.BytesIO(wav_bytes), "rb") as wav_file:
+ pcm = wav_file.readframes(wav_file.getnframes())
+ sample_rate = wav_file.getframerate()
+
+ output_emitter.initialize(
+ request_id=f"espeak-{abs(hash(self._input_text))}",
+ sample_rate=sample_rate,
+ num_channels=1,
+ mime_type="audio/pcm",
+ )
+ chunk_size = sample_rate // 5 * 2 # ~200ms of 16-bit mono PCM per push
+ for offset in range(0, len(pcm), chunk_size):
+ output_emitter.push(pcm[offset : offset + chunk_size])
+ await asyncio.sleep(0)
+ output_emitter.flush()
+
+
+class AvatarDemoAgent(Agent):
+ def __init__(self) -> None:
+ super().__init__(
+ instructions=(
+ "You are the three.ws demo avatar. Greet the visitor, confirm you can "
+ "hear yourself speak, then say goodbye."
+ )
+ )
+
+ async def on_enter(self) -> None:
+ await self.session.say("Hello! I'm the three.ws avatar, speaking through LiveKit.")
+ await self.session.say("Yes, this plugin publishes a gesture cue with that line.")
+ await self.session.say("Goodbye, and thanks for watching the demo!")
+
+
+server = AgentServer()
+
+
+async def entrypoint(ctx: JobContext) -> None:
+ ctx.log_context_fields = {"room": ctx.room.name}
+
+ session: AgentSession = AgentSession(tts=EspeakTTS())
+ avatar = threews.AvatarSession()
+ await avatar.start(session, room=ctx.room)
+
+ await session.start(agent=AvatarDemoAgent(), room=ctx.room)
+
+
+if __name__ == "__main__":
+ cli.run_app(server)
diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/__init__.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/__init__.py
new file mode 100644
index 000000000..d18f537c4
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/__init__.py
@@ -0,0 +1,36 @@
+# Copyright 2026 three.ws
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""three.ws avatar plugin for LiveKit Agents."""
+
+from livekit.agents import Plugin
+
+from .avatar import AvatarSession, ThreeWsException, gesture_for_text
+from .log import logger
+from .version import __version__
+
+__all__ = [
+ "AvatarSession",
+ "ThreeWsException",
+ "gesture_for_text",
+ "__version__",
+]
+
+
+class ThreeWsPlugin(Plugin):
+ def __init__(self) -> None:
+ super().__init__(__name__, __version__, __package__, logger)
+
+
+Plugin.register_plugin(ThreeWsPlugin())
diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/avatar.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/avatar.py
new file mode 100644
index 000000000..ccb72b188
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/avatar.py
@@ -0,0 +1,195 @@
+# Copyright 2026 three.ws
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import annotations
+
+import asyncio
+import json
+import re
+from typing import Any
+
+from livekit import rtc
+from livekit.agents import AgentSession, get_job_context
+from livekit.agents.voice.avatar import AvatarSession as BaseAvatarSession
+from livekit.agents.voice.events import (
+ ConversationItemAddedEvent,
+ UserInputTranscribedEvent,
+ UserStateChangedEvent,
+)
+
+from .log import logger
+
+DEFAULT_TOPIC = "threews.avatar"
+
+# Gesture cues are a best-effort read of the assistant's own final text, the
+# same heuristic family the browser widget already uses for lip sync
+# (src/runtime/lipsync.js): no NLU model, just the opening/closing words a
+# turn is likely to use. A miss just means no gesture fires, never a wrong
+# one, so the pattern favors precision over recall.
+_GREETING_RE = re.compile(
+ r"^\s*(hi|hello|hey there|hey|greetings|good (morning|afternoon|evening))\b",
+ re.IGNORECASE,
+)
+_FAREWELL_RE = re.compile(
+ r"\b(goodbye|bye for now|take care|farewell)[.!]?\s*$",
+ re.IGNORECASE,
+)
+_AGREEMENT_RE = re.compile(
+ r"^\s*(yes|yeah|yep|absolutely|definitely|of course)\b",
+ re.IGNORECASE,
+)
+_DISAGREEMENT_RE = re.compile(
+ r"^\s*(no|nope|not quite|not really|unfortunately)\b",
+ re.IGNORECASE,
+)
+
+
+def gesture_for_text(text: str) -> str | None:
+ """Best-effort gesture slot name for a line of assistant speech, or ``None``.
+
+ Returns one of the animation slot names three.ws avatars already know
+ (``wave``, ``nod``, ``shrug``), matched against
+ `src/runtime/animation-slots.js` in the three.ws client.
+ """
+ if _GREETING_RE.search(text) or _FAREWELL_RE.search(text):
+ return "wave"
+ if _AGREEMENT_RE.search(text):
+ return "nod"
+ if _DISAGREEMENT_RE.search(text):
+ return "shrug"
+ return None
+
+
+class ThreeWsException(Exception):
+ """Exception raised for three.ws avatar integration errors."""
+
+
+class AvatarSession(BaseAvatarSession):
+ """A three.ws avatar session.
+
+ three.ws avatars (the `<agent-3d>` web component) render on the visitor's
+ own device: the 3D model, its rig and its clip library already live in the
+ browser. This plugin never synthesizes video. Instead it publishes small
+ JSON messages over the LiveKit data channel that drive the browser's
+ existing lip-sync and gesture systems:
+
+ - ``{"type": "transcript", "text": "..."}`` for every finished assistant
+ turn, which the browser times into visemes locally.
+ - ``{"type": "gesture", "name": "wave" | "nod" | "shrug"}`` when the turn's
+ text reads like a greeting, farewell, agreement or disagreement.
+ - ``{"type": "speech_started"}`` / ``{"type": "speech_ended", "transcript": "..."}``
+ around the user's own turns, so the avatar can switch to a listening pose.
+
+ There is no separate avatar worker process and no dedicated avatar
+ participant: the data is published from the agent's own identity, so
+ `avatar_identity` returns the local participant's identity rather than a
+ synthesized one.
+ """
+
+ def __init__(self, *, topic: str = DEFAULT_TOPIC) -> None:
+ super().__init__()
+ self._topic = topic
+ self._room: rtc.Room | None = None
+ self._agent_session: AgentSession | None = None
+ self._last_user_transcript = ""
+ self._closed = False
+ self._publish_tasks: set[asyncio.Task[None]] = set()
+
+ @property
+ def avatar_identity(self) -> str:
+ if self._room is None:
+ raise ThreeWsException("avatar session has not started yet")
+ return self._room.local_participant.identity
+
+ @property
+ def provider(self) -> str:
+ return "threews"
+
+ async def start(self, agent_session: AgentSession, room: rtc.Room) -> None:
+ """Start publishing avatar motion data for ``agent_session`` into ``room``."""
+ if self._closed:
+ raise ThreeWsException("three.ws avatar session is closed")
+ if self._room is not None:
+ logger.warning("three.ws avatar session already started")
+ return
+
+ self._room = room
+ self._agent_session = agent_session
+
+ agent_session.on("conversation_item_added", self._on_conversation_item_added)
+ agent_session.on("user_input_transcribed", self._on_user_input_transcribed)
+ agent_session.on("user_state_changed", self._on_user_state_changed)
+
+ job_ctx = get_job_context(required=False)
+ if job_ctx is not None:
+ job_ctx.add_shutdown_callback(self.aclose)
+ else:
+ logger.debug(
+ "three.ws AvatarSession started outside a job context; "
+ "call aclose() manually to release resources when the job shuts down"
+ )
+
+ async def aclose(self) -> None:
+ if self._agent_session is not None:
+ self._agent_session.off("conversation_item_added", self._on_conversation_item_added)
+ self._agent_session.off("user_input_transcribed", self._on_user_input_transcribed)
+ self._agent_session.off("user_state_changed", self._on_user_state_changed)
+ self._agent_session = None
+ self._room = None
+ self._closed = True
+ for task in list(self._publish_tasks):
+ task.cancel()
+
+ def _on_conversation_item_added(self, ev: ConversationItemAddedEvent) -> None:
+ item = ev.item
+ if getattr(item, "type", None) != "message" or getattr(item, "role", None) != "assistant":
+ return
+ text = item.text_content
+ if not text:
+ return
+ self._publish({"type": "transcript", "text": text})
+ gesture = gesture_for_text(text)
+ if gesture:
+ self._publish({"type": "gesture", "name": gesture})
+
+ def _on_user_input_transcribed(self, ev: UserInputTranscribedEvent) -> None:
+ if ev.is_final:
+ self._last_user_transcript = ev.transcript
+
+ def _on_user_state_changed(self, ev: UserStateChangedEvent) -> None:
+ if ev.new_state == "speaking":
+ self._publish({"type": "speech_started"})
+ elif ev.old_state == "speaking":
+ self._publish({"type": "speech_ended", "transcript": self._last_user_transcript})
+
+ def _publish(self, payload: dict[str, Any]) -> None:
+ if self._room is None:
+ return
+ local = self._room.local_participant
+ if local is None:
+ return
+ # Event handlers fire synchronously (rtc.EventEmitter.emit does not
+ # await coroutine results), so publish_data (an async RPC to the
+ # rtc engine) must be scheduled as a task rather than called inline.
+ task = asyncio.ensure_future(self._publish_async(local, payload))
+ self._publish_tasks.add(task)
+ task.add_done_callback(self._publish_tasks.discard)
+
+ async def _publish_async(self, local: rtc.LocalParticipant, payload: dict[str, Any]) -> None:
+ try:
+ await local.publish_data(json.dumps(payload), reliable=True, topic=self._topic)
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.warning("failed to publish three.ws avatar data", exc_info=True)
diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/log.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/log.py
new file mode 100644
index 000000000..09efa8d2f
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/log.py
@@ -0,0 +1,3 @@
+import logging
+
+logger = logging.getLogger("livekit.plugins.threews")
diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/py.typed b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/py.typed
new file mode 100644
index 000000000..e69de29bb
diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/version.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/version.py
new file mode 100644
index 000000000..3dc1f76bc
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/version.py
@@ -0,0 +1 @@
+__version__ = "0.1.0"
diff --git a/integrations/livekit/livekit-plugins-threews/pyproject.toml b/integrations/livekit/livekit-plugins-threews/pyproject.toml
new file mode 100644
index 000000000..bb386f4e5
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/pyproject.toml
@@ -0,0 +1,66 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "livekit-plugins-threews"
+dynamic = ["version"]
+description = "LiveKit Agents avatar plugin for three.ws 3D avatars"
+readme = "README.md"
+license = "Apache-2.0"
+requires-python = ">=3.10.0"
+authors = [{ name = "three.ws", email = "[email protected]" }]
+keywords = ["voice", "ai", "realtime", "audio", "video", "livekit", "webrtc", "avatar", "3d", "threews"]
+classifiers = [
+ "Intended Audience :: Developers",
+ "License :: OSI Approved :: Apache Software License",
+ "Topic :: Multimedia :: Sound/Audio",
+ "Topic :: Multimedia :: Video",
+ "Topic :: Scientific/Engineering :: Artificial Intelligence",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3 :: Only",
+]
+dependencies = [
+ "livekit-agents>=1.6.4",
+]
+
+[project.urls]
+Documentation = "https://three.ws/docs"
+Website = "https://three.ws/"
+Source = "https://github.com/nirholas/three.ws"
+
+[tool.hatch.version]
+path = "livekit/plugins/threews/version.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["livekit"]
+
+[tool.hatch.build.targets.sdist]
+include = ["/livekit"]
+
+# Mirrors livekit/agents' own [tool.ruff] block (their pyproject.toml), the
+# config their CONTRIBUTING.md asks every plugin pull request to pass.
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = [
+ "E", # pycodestyle errors
+ "W", # pycodestyle warnings
+ "F", # pyflakes
+ "I", # isort
+ "B", # flake8-bugbear
+ "C4", # flake8-comprehensions
+ "UP", # pyupgrade
+ "RUF006", # asyncio-dangling-task
+]
+ignore = ["E501"]
+
+[tool.ruff.lint.isort]
+combine-as-imports = true
+known-first-party = ["livekit"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
diff --git a/integrations/livekit/livekit-plugins-threews/tests/__init__.py b/integrations/livekit/livekit-plugins-threews/tests/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/integrations/livekit/livekit-plugins-threews/tests/test_avatar.py b/integrations/livekit/livekit-plugins-threews/tests/test_avatar.py
new file mode 100644
index 000000000..c9b919aeb
--- /dev/null
+++ b/integrations/livekit/livekit-plugins-threews/tests/test_avatar.py
@@ -0,0 +1,82 @@
+# Copyright 2026 three.ws
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Regression test for the AvatarSession._publish asyncio bug.
+
+``rtc.LocalParticipant.publish_data`` is a coroutine function. The event
+handlers that call it (``_on_conversation_item_added`` and friends) are
+invoked synchronously by ``rtc.EventEmitter.emit``, which does not await
+or schedule whatever its callback returns. Calling the coroutine function
+directly without ``await`` or ``asyncio.ensure_future`` therefore builds a
+coroutine object that is never driven: ``publish_data`` never actually
+runs, Python only raises a ``RuntimeWarning: coroutine ... was never
+awaited`` at garbage-collection time, and no data ever reaches the room.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+
+import pytest
+
+from livekit.agents.llm import ChatMessage
+from livekit.agents.voice.events import ConversationItemAddedEvent
+from livekit.plugins.threews.avatar import AvatarSession
+
+
+class _FakeLocalParticipant:
+ def __init__(self) -> None:
+ self.published: list[tuple[str, dict]] = []
+
+ async def publish_data(self, payload, *, reliable=True, topic="") -> None:
+ await asyncio.sleep(0)
+ self.published.append((topic, json.loads(payload)))
+
+
+class _FakeRoom:
+ def __init__(self, local_participant: _FakeLocalParticipant) -> None:
+ self.local_participant = local_participant
+
+
+async def test_conversation_item_actually_publishes_data():
+ """The assistant turn must actually reach the room, not just build a coroutine."""
+ local = _FakeLocalParticipant()
+ session = AvatarSession()
+ session._room = _FakeRoom(local)
+
+ event = ConversationItemAddedEvent(
+ item=ChatMessage(role="assistant", content=["Hello! I'm the three.ws avatar."])
+ )
+ session._on_conversation_item_added(event)
+
+ # Let the scheduled publish task(s) actually run.
+ await asyncio.gather(*session._publish_tasks)
+
+ assert local.published, "publish_data was never actually awaited/executed"
+ topic, payload = local.published[0]
+ assert topic == "threews.avatar"
+ assert payload == {"type": "transcript", "text": "Hello! I'm the three.ws avatar."}
+ # A greeting also carries a wave gesture cue.
+ assert any(p == {"type": "gesture", "name": "wave"} for _, p in local.published)
+
+
+async def test_publish_without_room_is_a_noop():
+ session = AvatarSession()
+ session._on_conversation_item_added(
+ ConversationItemAddedEvent(item=ChatMessage(role="assistant", content=["hi"]))
+ )
+ assert session._publish_tasks == set()
diff --git a/public/agent-3d-livekit-harness.html b/public/agent-3d-livekit-harness.html
new file mode 100644
index 000000000..c6e504e65
--- /dev/null
+++ b/public/agent-3d-livekit-harness.html
@@ -0,0 +1,25 @@
+<!doctype html>
+<html>
+<head>
+<meta charset="utf-8">
+<title>agent-3d LiveKit avatar harness</title>
+<!--
+ Playwright-only fixture for tests/e2e/agent-3d-livekit-avatar.spec.js.
+ Imports /src/element.js directly so the spec exercises today's source,
+ mirrors the public/agent-3d-ar-harness.html convention. Not linked from
+ any nav.
+-->
+<style>html,body{margin:0;height:100%}agent-3d{display:block;width:480px;height:480px}</style>
+</head>
+<body>
+<agent-3d
+ id="a3d"
+ eager
+ src="/avatars/fox.glb"
+ agent-id="00000000-0000-4000-8000-000000000000"
+ voice="livekit"
+ avatar-chat="off"
+></agent-3d>
+<script type="module" src="/src/element.js"></script>
+</body>
+</html>
diff --git a/src/runtime/livekit-voice.js b/src/runtime/livekit-voice.js
index 7085b3b54..296493b6a 100644
--- a/src/runtime/livekit-voice.js
+++ b/src/runtime/livekit-voice.js
@@ -64,6 +64,13 @@ export class LiveKitVoice {
payload: { text: msg.text, sentiment: 0 },
});
}
+ // Named body gesture (e.g. from the LiveKit avatar plugin)
+ if (msg.type === 'gesture' && msg.name) {
+ this._protocol.emit({
+ type: ACTION_TYPES.GESTURE,
+ payload: { name: msg.name },
+ });
+ }
// User speech boundaries (VAD from agent server)
if (msg.type === 'speech_started') {
this._userSpeaking = true;
diff --git a/tests/e2e/agent-3d-livekit-avatar.spec.js b/tests/e2e/agent-3d-livekit-avatar.spec.js
new file mode 100644
index 000000000..abb0edfc0
--- /dev/null
+++ b/tests/e2e/agent-3d-livekit-avatar.spec.js
@@ -0,0 +1,154 @@
+import { test, expect } from '@playwright/test';
+import { spawn } from 'node:child_process';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { SignJWT } from 'jose';
+
+// Work order 087: proves the LiveKit avatar plugin end to end, browser side.
+// A real voice agent (integrations/livekit/livekit-plugins-threews/examples/
+// avatar_agent.py) joins a real local open-source livekit-server (started by
+// this repo's `npm run dev:livekit` docker helper, or already running from a
+// prior manual `docker run livekit/livekit-server --dev`) and speaks three
+// lines through a real espeak-ng TTS. The three.ws `threews.AvatarSession`
+// plugin republishes the agent's turns as `transcript`/`gesture` data-channel
+// messages, and a real Chromium loads `<agent-3d voice="livekit">` (via
+// public/agent-3d-livekit-harness.html) to prove those messages actually
+// drive the browser's existing gesture/lipsync protocol.
+//
+// Fidelity: only the agent's own livekit-token mint
+// (`/api/agents/:id/livekit-token`, real production DB + auth state this
+// test box doesn't have) is route-intercepted, with a token built the same
+// way the real handler signs one (same claim shape, same devkey/devsecret
+// dev-server credentials the local livekit-server was started with). Every
+// other hop (the LiveKit room, the agent process, the plugin's data channel,
+// the browser's `LiveKitVoice` client, the avatar protocol) is real.
+
+const ROOM = `e2e-livekit-${Date.now()}`;
+const LIVEKIT_URL = process.env.LIVEKIT_URL || 'ws://localhost:7880';
+const LIVEKIT_API_KEY = process.env.LIVEKIT_API_KEY || 'devkey';
+const LIVEKIT_API_SECRET = process.env.LIVEKIT_API_SECRET || 'devsecret';
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const EXAMPLES_DIR = path.join(
+ HERE,
+ '..',
+ '..',
+ 'integrations',
+ 'livekit',
+ 'livekit-plugins-threews',
+ 'examples',
+);
+
+async function mintBrowserToken(room) {
+ const secret = new TextEncoder().encode(LIVEKIT_API_SECRET);
+ const now = Math.floor(Date.now() / 1000);
+ return new SignJWT({
+ video: { roomJoin: true, room, canPublish: true, canSubscribe: true },
+ sub: 'e2e-browser',
+ iss: LIVEKIT_API_KEY,
+ nbf: now,
+ exp: now + 3600,
+ })
+ .setProtectedHeader({ alg: 'HS256' })
+ .sign(secret);
+}
+
+function spawnAgent(room) {
+ const child = spawn('python3', ['-u', 'avatar_agent.py', 'connect', '--room', room], {
+ cwd: EXAMPLES_DIR,
+ env: {
+ ...process.env,
+ LIVEKIT_URL,
+ LIVEKIT_API_KEY,
+ LIVEKIT_API_SECRET,
+ },
+ stdio: ['ignore', 'pipe', 'pipe'],
+ });
+ let log = '';
+ child.stdout.on('data', (d) => (log += d.toString()));
+ child.stderr.on('data', (d) => (log += d.toString()));
+ return { child, getLog: () => log };
+}
+
+test.describe('agent-3d LiveKit avatar plugin (real local livekit-server)', () => {
+ test.setTimeout(120_000);
+
+ test('a real agent speaking in the room drives the browser avatar lipsync + gesture', async ({
+ page,
+ }) => {
+ const agent = spawnAgent(ROOM);
+ try {
+ const token = await mintBrowserToken(ROOM);
+ await page.route('**/api/agents/*/livekit-token', (route) =>
+ route.fulfill({
+ status: 200,
+ contentType: 'application/json',
+ body: JSON.stringify({ token, serverUrl: LIVEKIT_URL }),
+ }),
+ );
+
+ const errors = [];
+ page.on('pageerror', (err) => errors.push(String(err?.message || err)));
+
+ await page.goto('/agent-3d-livekit-harness.html', { waitUntil: 'domcontentloaded' });
+
+ // Wait for the element to actually open the LiveKit room before
+ // subscribing to its protocol (the protocol object itself is created
+ // synchronously on connect, independent of the room join finishing).
+ await expect
+ .poll(() => page.evaluate(() => !!document.getElementById('a3d')?._avatar?.protocol), {
+ timeout: 60_000,
+ })
+ .toBe(true);
+
+ const eventsPromise = page.evaluate(() => {
+ return new Promise((resolve) => {
+ const protocol = document.getElementById('a3d')._avatar.protocol;
+ const seen = [];
+ const onSpeak = (action) => seen.push({ type: 'speak', text: action.payload?.text });
+ const onGesture = (action) => seen.push({ type: 'gesture', name: action.payload?.name });
+ protocol.on('speak', onSpeak);
+ protocol.on('gesture', onGesture);
+ // The agent speaks exactly 3 lines (2 carry a gesture cue, the
+ // farewell line does not by design); stop once all 3 "speak"
+ // events have landed, or after a generous ceiling so a real
+ // failure still resolves with whatever partial evidence it
+ // gathered.
+ const finish = () => {
+ protocol.off('speak', onSpeak);
+ protocol.off('gesture', onGesture);
+ resolve(seen);
+ };
+ const ceiling = setTimeout(finish, 45_000);
+ const checkDone = () => {
+ if (seen.filter((e) => e.type === 'speak').length >= 3) {
+ clearTimeout(ceiling);
+ finish();
+ }
+ };
+ protocol.on('speak', checkDone);
+ });
+ });
+
+ const events = await eventsPromise;
+
+ await page.screenshot({
+ path: 'test-results/agent-3d-livekit-avatar-final.png',
+ fullPage: false,
+ });
+
+ const speaks = events.filter((e) => e.type === 'speak').map((e) => e.text);
+ const gestures = events.filter((e) => e.type === 'gesture').map((e) => e.name);
+
+ expect(speaks, `agent log:\n${agent.getLog()}`).toEqual([
+ "Hello! I'm the three.ws avatar, speaking through LiveKit.",
+ 'Yes, this plugin publishes a gesture cue with that line.',
+ 'Goodbye, and thanks for watching the demo!',
+ ]);
+ expect(gestures).toEqual(['wave', 'nod']);
+ expect(errors, `page errors: ${errors.join(' | ')}`).toEqual([]);
+ } finally {
+ agent.child.kill('SIGTERM');
+ }
+ });
+});