Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
feat(integrations): add the livekit-plugins-threews avatar plugin, verified with a real local lipsync demo
nirholas55350b50d993
diff --git a/.gitignore b/.gitignoreindex 0f3b79e4c..a1c125cc8 100644--- a/.gitignore+++ b/.gitignore@@ -259,6 +259,9 @@ snapshots/exported/# Python bytecode (workers/)__pycache__/*.pyc+*.egg-info/+.ruff_cache/+.pytest_cache/# three.ws presence sweep — regenerate with `npm run presence`three-ws-presence.jsondiff --git a/integrations/livekit/livekit-plugins-threews/README.md b/integrations/livekit/livekit-plugins-threews/README.mdnew file mode 100644index 000000000..ea9435166--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/README.md@@ -0,0 +1,65 @@+# LiveKit Plugins three.ws++Agent Framework avatar plugin for [three.ws](https://three.ws) 3D avatars.++## Client-side rendering++three.ws avatars render entirely on the visitor's own device: the `<agent-3d>`+web component already ships the humanoid model, its rig and its animation+clip library in the browser. This plugin never synthesizes a video track.+Instead it publishes small JSON messages over the LiveKit data channel+(topic `threews.avatar`) that drive the client's existing lip-sync and+gesture systems:++```json+{"type": "transcript", "text": "Hi, how can I help today?"}+{"type": "gesture", "name": "wave"}+{"type": "speech_started"}+{"type": "speech_ended", "transcript": "what can you do"}+```++A standard LiveKit video renderer attached to the agent's room sees no video+track from this plugin, because there is none to see: a three.ws avatar is+not server-rendered. Point your frontend at+[`<agent-3d voice="livekit">`](https://github.com/nirholas/three.ws/blob/main/src/element.js)+(or any embed that loads `src/runtime/livekit-voice.js`), which already+parses this message shape.++## Installation++```bash+pip install livekit-plugins-threews+```++## Usage++```python+from livekit.plugins import threews++avatar = threews.AvatarSession()+await avatar.start(session, room=ctx.room)+```++No API key, account or environment variable is required: the plugin reads+state from the `AgentSession` you already built (its conversation items and+user turn-taking events) and republishes it as motion data on the room the+agent is already in.++## How it decides what to show++- **Lip sync** comes from every finished assistant turn+ (`conversation_item_added`): the full text is sent once, and the browser+ times it into visemes the same way it already does for every other voice+ path in three.ws (`src/runtime/lipsync.js`), so there is nothing provider+ specific for the browser to learn.+- **Gestures** are a best-effort read of that same text+ (`threews.gesture_for_text`): a greeting or farewell triggers `wave`, an+ agreement triggers `nod`, a disagreement triggers `shrug`. A miss just+ means no gesture fires, never a wrong one.+- **Listening pose** comes from `user_state_changed`: the avatar gets+ `speech_started` / `speech_ended` events around the user's own turns.++See [`examples/avatar_agent.py`](examples/avatar_agent.py) for a complete+voice agent wired to this plugin, and+[`../../README.md`](../../README.md) for how to run it end to end against a+local `livekit-server`.diff --git a/integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py b/integrations/livekit/livekit-plugins-threews/examples/avatar_agent.pynew file mode 100644index 000000000..7d865b1f2--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py@@ -0,0 +1,159 @@+# Copyright 2026 three.ws+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""A minimal voice agent wired to the three.ws avatar plugin.++Speech-to-text and the LLM are skipped on purpose: this example proves the+plugin end to end (a real local TTS speaks, the avatar plugin publishes+synced transcript and gesture data) without needing a cloud account for any+leg of the pipeline, per the plugin's own README.++Text-to-speech is a thin wrapper around the ``espeak-ng`` command line+(``EspeakTTS`` below), a real open-source offline synthesizer, not a mock:+every audio frame the room receives came out of an actual ``espeak-ng``+subprocess.++Run it against a local open-source ``livekit-server``+(see ``../../README.md`` for the docker command) with:++ python avatar_agent.py connect --room demo-room+"""++from __future__ import annotations++import asyncio+import io+import shutil+import wave+from dataclasses import dataclass++from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli+from livekit.agents.tts import TTS, ChunkedStream, TTSCapabilities+from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS, APIConnectOptions+from livekit.plugins import threews++ESPEAK_SAMPLE_RATE = 22050+++@dataclass+class _EspeakVoice:+ """``espeak-ng`` voice selection, forwarded to its ``-v`` flag."""++ name: str = "en-us"+ rate_wpm: int = 165+++class EspeakTTS(TTS):+ """Offline TTS backed by the ``espeak-ng`` binary.++ No network call, no API key: every utterance is synthesized locally by+ shelling out to ``espeak-ng --stdout``, which writes a WAV file we decode+ into PCM frames for the LiveKit audio pipeline.+ """++ def __init__(self, *, voice: _EspeakVoice | None = None) -> None:+ super().__init__(+ capabilities=TTSCapabilities(streaming=False),+ sample_rate=ESPEAK_SAMPLE_RATE,+ num_channels=1,+ )+ if shutil.which("espeak-ng") is None:+ raise RuntimeError(+ "espeak-ng is not installed; install it (e.g. `apt install espeak-ng`) "+ "to run this example"+ )+ self._voice = voice or _EspeakVoice()++ @property+ def provider(self) -> str:+ return "espeak-ng"++ @property+ def model(self) -> str:+ return self._voice.name++ def synthesize(+ self, text: str, *, conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS+ ) -> ChunkedStream:+ return _EspeakChunkedStream(tts=self, input_text=text, conn_options=conn_options)+++class _EspeakChunkedStream(ChunkedStream):+ async def _run(self, output_emitter) -> None: # noqa: ANN001+ assert isinstance(self._tts, EspeakTTS)+ voice = self._tts._voice++ proc = await asyncio.create_subprocess_exec(+ "espeak-ng",+ "-v",+ voice.name,+ "-s",+ str(voice.rate_wpm),+ "--stdout",+ self._input_text,+ stdout=asyncio.subprocess.PIPE,+ stderr=asyncio.subprocess.PIPE,+ )+ wav_bytes, stderr = await proc.communicate()+ if proc.returncode != 0:+ raise RuntimeError(f"espeak-ng failed: {stderr.decode(errors='replace')}")++ with wave.open(io.BytesIO(wav_bytes), "rb") as wav_file:+ pcm = wav_file.readframes(wav_file.getnframes())+ sample_rate = wav_file.getframerate()++ output_emitter.initialize(+ request_id=f"espeak-{abs(hash(self._input_text))}",+ sample_rate=sample_rate,+ num_channels=1,+ mime_type="audio/pcm",+ )+ chunk_size = sample_rate // 5 * 2 # ~200ms of 16-bit mono PCM per push+ for offset in range(0, len(pcm), chunk_size):+ output_emitter.push(pcm[offset : offset + chunk_size])+ await asyncio.sleep(0)+ output_emitter.flush()+++class AvatarDemoAgent(Agent):+ def __init__(self) -> None:+ super().__init__(+ instructions=(+ "You are the three.ws demo avatar. Greet the visitor, confirm you can "+ "hear yourself speak, then say goodbye."+ )+ )++ async def on_enter(self) -> None:+ await self.session.say("Hello! I'm the three.ws avatar, speaking through LiveKit.")+ await self.session.say("Yes, this plugin publishes a gesture cue with that line.")+ await self.session.say("Goodbye, and thanks for watching the demo!")+++server = AgentServer()++[email protected]_session()+async def entrypoint(ctx: JobContext) -> None:+ ctx.log_context_fields = {"room": ctx.room.name}++ session: AgentSession = AgentSession(tts=EspeakTTS())+ avatar = threews.AvatarSession()+ await avatar.start(session, room=ctx.room)++ await session.start(agent=AvatarDemoAgent(), room=ctx.room)+++if __name__ == "__main__":+ cli.run_app(server)diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/__init__.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/__init__.pynew file mode 100644index 000000000..d18f537c4--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/__init__.py@@ -0,0 +1,36 @@+# Copyright 2026 three.ws+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""three.ws avatar plugin for LiveKit Agents."""++from livekit.agents import Plugin++from .avatar import AvatarSession, ThreeWsException, gesture_for_text+from .log import logger+from .version import __version__++__all__ = [+ "AvatarSession",+ "ThreeWsException",+ "gesture_for_text",+ "__version__",+]+++class ThreeWsPlugin(Plugin):+ def __init__(self) -> None:+ super().__init__(__name__, __version__, __package__, logger)+++Plugin.register_plugin(ThreeWsPlugin())diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/avatar.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/avatar.pynew file mode 100644index 000000000..ccb72b188--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/avatar.py@@ -0,0 +1,195 @@+# Copyright 2026 three.ws+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++from __future__ import annotations++import asyncio+import json+import re+from typing import Any++from livekit import rtc+from livekit.agents import AgentSession, get_job_context+from livekit.agents.voice.avatar import AvatarSession as BaseAvatarSession+from livekit.agents.voice.events import (+ ConversationItemAddedEvent,+ UserInputTranscribedEvent,+ UserStateChangedEvent,+)++from .log import logger++DEFAULT_TOPIC = "threews.avatar"++# Gesture cues are a best-effort read of the assistant's own final text, the+# same heuristic family the browser widget already uses for lip sync+# (src/runtime/lipsync.js): no NLU model, just the opening/closing words a+# turn is likely to use. A miss just means no gesture fires, never a wrong+# one, so the pattern favors precision over recall.+_GREETING_RE = re.compile(+ r"^\s*(hi|hello|hey there|hey|greetings|good (morning|afternoon|evening))\b",+ re.IGNORECASE,+)+_FAREWELL_RE = re.compile(+ r"\b(goodbye|bye for now|take care|farewell)[.!]?\s*$",+ re.IGNORECASE,+)+_AGREEMENT_RE = re.compile(+ r"^\s*(yes|yeah|yep|absolutely|definitely|of course)\b",+ re.IGNORECASE,+)+_DISAGREEMENT_RE = re.compile(+ r"^\s*(no|nope|not quite|not really|unfortunately)\b",+ re.IGNORECASE,+)+++def gesture_for_text(text: str) -> str | None:+ """Best-effort gesture slot name for a line of assistant speech, or ``None``.++ Returns one of the animation slot names three.ws avatars already know+ (``wave``, ``nod``, ``shrug``), matched against+ `src/runtime/animation-slots.js` in the three.ws client.+ """+ if _GREETING_RE.search(text) or _FAREWELL_RE.search(text):+ return "wave"+ if _AGREEMENT_RE.search(text):+ return "nod"+ if _DISAGREEMENT_RE.search(text):+ return "shrug"+ return None+++class ThreeWsException(Exception):+ """Exception raised for three.ws avatar integration errors."""+++class AvatarSession(BaseAvatarSession):+ """A three.ws avatar session.++ three.ws avatars (the `<agent-3d>` web component) render on the visitor's+ own device: the 3D model, its rig and its clip library already live in the+ browser. This plugin never synthesizes video. Instead it publishes small+ JSON messages over the LiveKit data channel that drive the browser's+ existing lip-sync and gesture systems:++ - ``{"type": "transcript", "text": "..."}`` for every finished assistant+ turn, which the browser times into visemes locally.+ - ``{"type": "gesture", "name": "wave" | "nod" | "shrug"}`` when the turn's+ text reads like a greeting, farewell, agreement or disagreement.+ - ``{"type": "speech_started"}`` / ``{"type": "speech_ended", "transcript": "..."}``+ around the user's own turns, so the avatar can switch to a listening pose.++ There is no separate avatar worker process and no dedicated avatar+ participant: the data is published from the agent's own identity, so+ `avatar_identity` returns the local participant's identity rather than a+ synthesized one.+ """++ def __init__(self, *, topic: str = DEFAULT_TOPIC) -> None:+ super().__init__()+ self._topic = topic+ self._room: rtc.Room | None = None+ self._agent_session: AgentSession | None = None+ self._last_user_transcript = ""+ self._closed = False+ self._publish_tasks: set[asyncio.Task[None]] = set()++ @property+ def avatar_identity(self) -> str:+ if self._room is None:+ raise ThreeWsException("avatar session has not started yet")+ return self._room.local_participant.identity++ @property+ def provider(self) -> str:+ return "threews"++ async def start(self, agent_session: AgentSession, room: rtc.Room) -> None:+ """Start publishing avatar motion data for ``agent_session`` into ``room``."""+ if self._closed:+ raise ThreeWsException("three.ws avatar session is closed")+ if self._room is not None:+ logger.warning("three.ws avatar session already started")+ return++ self._room = room+ self._agent_session = agent_session++ agent_session.on("conversation_item_added", self._on_conversation_item_added)+ agent_session.on("user_input_transcribed", self._on_user_input_transcribed)+ agent_session.on("user_state_changed", self._on_user_state_changed)++ job_ctx = get_job_context(required=False)+ if job_ctx is not None:+ job_ctx.add_shutdown_callback(self.aclose)+ else:+ logger.debug(+ "three.ws AvatarSession started outside a job context; "+ "call aclose() manually to release resources when the job shuts down"+ )++ async def aclose(self) -> None:+ if self._agent_session is not None:+ self._agent_session.off("conversation_item_added", self._on_conversation_item_added)+ self._agent_session.off("user_input_transcribed", self._on_user_input_transcribed)+ self._agent_session.off("user_state_changed", self._on_user_state_changed)+ self._agent_session = None+ self._room = None+ self._closed = True+ for task in list(self._publish_tasks):+ task.cancel()++ def _on_conversation_item_added(self, ev: ConversationItemAddedEvent) -> None:+ item = ev.item+ if getattr(item, "type", None) != "message" or getattr(item, "role", None) != "assistant":+ return+ text = item.text_content+ if not text:+ return+ self._publish({"type": "transcript", "text": text})+ gesture = gesture_for_text(text)+ if gesture:+ self._publish({"type": "gesture", "name": gesture})++ def _on_user_input_transcribed(self, ev: UserInputTranscribedEvent) -> None:+ if ev.is_final:+ self._last_user_transcript = ev.transcript++ def _on_user_state_changed(self, ev: UserStateChangedEvent) -> None:+ if ev.new_state == "speaking":+ self._publish({"type": "speech_started"})+ elif ev.old_state == "speaking":+ self._publish({"type": "speech_ended", "transcript": self._last_user_transcript})++ def _publish(self, payload: dict[str, Any]) -> None:+ if self._room is None:+ return+ local = self._room.local_participant+ if local is None:+ return+ # Event handlers fire synchronously (rtc.EventEmitter.emit does not+ # await coroutine results), so publish_data (an async RPC to the+ # rtc engine) must be scheduled as a task rather than called inline.+ task = asyncio.ensure_future(self._publish_async(local, payload))+ self._publish_tasks.add(task)+ task.add_done_callback(self._publish_tasks.discard)++ async def _publish_async(self, local: rtc.LocalParticipant, payload: dict[str, Any]) -> None:+ try:+ await local.publish_data(json.dumps(payload), reliable=True, topic=self._topic)+ except asyncio.CancelledError:+ raise+ except Exception:+ logger.warning("failed to publish three.ws avatar data", exc_info=True)diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/log.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/log.pynew file mode 100644index 000000000..09efa8d2f--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/log.py@@ -0,0 +1,3 @@+import logging++logger = logging.getLogger("livekit.plugins.threews")diff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/py.typed b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/py.typednew file mode 100644index 000000000..e69de29bbdiff --git a/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/version.py b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/version.pynew file mode 100644index 000000000..3dc1f76bc--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/livekit/plugins/threews/version.py@@ -0,0 +1 @@+__version__ = "0.1.0"diff --git a/integrations/livekit/livekit-plugins-threews/pyproject.toml b/integrations/livekit/livekit-plugins-threews/pyproject.tomlnew file mode 100644index 000000000..bb386f4e5--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/pyproject.toml@@ -0,0 +1,66 @@+[build-system]+requires = ["hatchling"]+build-backend = "hatchling.build"++[project]+name = "livekit-plugins-threews"+dynamic = ["version"]+description = "LiveKit Agents avatar plugin for three.ws 3D avatars"+readme = "README.md"+license = "Apache-2.0"+requires-python = ">=3.10.0"+authors = [{ name = "three.ws", email = "[email protected]" }]+keywords = ["voice", "ai", "realtime", "audio", "video", "livekit", "webrtc", "avatar", "3d", "threews"]+classifiers = [+ "Intended Audience :: Developers",+ "License :: OSI Approved :: Apache Software License",+ "Topic :: Multimedia :: Sound/Audio",+ "Topic :: Multimedia :: Video",+ "Topic :: Scientific/Engineering :: Artificial Intelligence",+ "Programming Language :: Python :: 3",+ "Programming Language :: Python :: 3.10",+ "Programming Language :: Python :: 3 :: Only",+]+dependencies = [+ "livekit-agents>=1.6.4",+]++[project.urls]+Documentation = "https://three.ws/docs"+Website = "https://three.ws/"+Source = "https://github.com/nirholas/three.ws"++[tool.hatch.version]+path = "livekit/plugins/threews/version.py"++[tool.hatch.build.targets.wheel]+packages = ["livekit"]++[tool.hatch.build.targets.sdist]+include = ["/livekit"]++# Mirrors livekit/agents' own [tool.ruff] block (their pyproject.toml), the+# config their CONTRIBUTING.md asks every plugin pull request to pass.+[tool.ruff]+line-length = 100+target-version = "py310"++[tool.ruff.lint]+select = [+ "E", # pycodestyle errors+ "W", # pycodestyle warnings+ "F", # pyflakes+ "I", # isort+ "B", # flake8-bugbear+ "C4", # flake8-comprehensions+ "UP", # pyupgrade+ "RUF006", # asyncio-dangling-task+]+ignore = ["E501"]++[tool.ruff.lint.isort]+combine-as-imports = true+known-first-party = ["livekit"]++[tool.ruff.lint.pydocstyle]+convention = "google"diff --git a/integrations/livekit/livekit-plugins-threews/tests/__init__.py b/integrations/livekit/livekit-plugins-threews/tests/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/integrations/livekit/livekit-plugins-threews/tests/test_avatar.py b/integrations/livekit/livekit-plugins-threews/tests/test_avatar.pynew file mode 100644index 000000000..c9b919aeb--- /dev/null+++ b/integrations/livekit/livekit-plugins-threews/tests/test_avatar.py@@ -0,0 +1,82 @@+# Copyright 2026 three.ws+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++"""Regression test for the AvatarSession._publish asyncio bug.++``rtc.LocalParticipant.publish_data`` is a coroutine function. The event+handlers that call it (``_on_conversation_item_added`` and friends) are+invoked synchronously by ``rtc.EventEmitter.emit``, which does not await+or schedule whatever its callback returns. Calling the coroutine function+directly without ``await`` or ``asyncio.ensure_future`` therefore builds a+coroutine object that is never driven: ``publish_data`` never actually+runs, Python only raises a ``RuntimeWarning: coroutine ... was never+awaited`` at garbage-collection time, and no data ever reaches the room.+"""++from __future__ import annotations++import asyncio+import json++import pytest++from livekit.agents.llm import ChatMessage+from livekit.agents.voice.events import ConversationItemAddedEvent+from livekit.plugins.threews.avatar import AvatarSession+++class _FakeLocalParticipant:+ def __init__(self) -> None:+ self.published: list[tuple[str, dict]] = []++ async def publish_data(self, payload, *, reliable=True, topic="") -> None:+ await asyncio.sleep(0)+ self.published.append((topic, json.loads(payload)))+++class _FakeRoom:+ def __init__(self, local_participant: _FakeLocalParticipant) -> None:+ self.local_participant = local_participant+++async def test_conversation_item_actually_publishes_data():+ """The assistant turn must actually reach the room, not just build a coroutine."""+ local = _FakeLocalParticipant()+ session = AvatarSession()+ session._room = _FakeRoom(local)++ event = ConversationItemAddedEvent(+ item=ChatMessage(role="assistant", content=["Hello! I'm the three.ws avatar."])+ )+ session._on_conversation_item_added(event)++ # Let the scheduled publish task(s) actually run.+ await asyncio.gather(*session._publish_tasks)++ assert local.published, "publish_data was never actually awaited/executed"+ topic, payload = local.published[0]+ assert topic == "threews.avatar"+ assert payload == {"type": "transcript", "text": "Hello! I'm the three.ws avatar."}+ # A greeting also carries a wave gesture cue.+ assert any(p == {"type": "gesture", "name": "wave"} for _, p in local.published)+++async def test_publish_without_room_is_a_noop():+ session = AvatarSession()+ session._on_conversation_item_added(+ ConversationItemAddedEvent(item=ChatMessage(role="assistant", content=["hi"]))+ )+ assert session._publish_tasks == set()diff --git a/public/agent-3d-livekit-harness.html b/public/agent-3d-livekit-harness.htmlnew file mode 100644index 000000000..c6e504e65--- /dev/null+++ b/public/agent-3d-livekit-harness.html@@ -0,0 +1,25 @@+<!doctype html>+<html>+<head>+<meta charset="utf-8">+<title>agent-3d LiveKit avatar harness</title>+<!--+ Playwright-only fixture for tests/e2e/agent-3d-livekit-avatar.spec.js.+ Imports /src/element.js directly so the spec exercises today's source,+ mirrors the public/agent-3d-ar-harness.html convention. Not linked from+ any nav.+-->+<style>html,body{margin:0;height:100%}agent-3d{display:block;width:480px;height:480px}</style>+</head>+<body>+<agent-3d+ id="a3d"+ eager+ src="/avatars/fox.glb"+ agent-id="00000000-0000-4000-8000-000000000000"+ voice="livekit"+ avatar-chat="off"+></agent-3d>+<script type="module" src="/src/element.js"></script>+</body>+</html>diff --git a/src/runtime/livekit-voice.js b/src/runtime/livekit-voice.jsindex 7085b3b54..296493b6a 100644--- a/src/runtime/livekit-voice.js+++ b/src/runtime/livekit-voice.js@@ -64,6 +64,13 @@ export class LiveKitVoice {payload: { text: msg.text, sentiment: 0 },});}+ // Named body gesture (e.g. from the LiveKit avatar plugin)+ if (msg.type === 'gesture' && msg.name) {+ this._protocol.emit({+ type: ACTION_TYPES.GESTURE,+ payload: { name: msg.name },+ });+ }// User speech boundaries (VAD from agent server)if (msg.type === 'speech_started') {this._userSpeaking = true;diff --git a/tests/e2e/agent-3d-livekit-avatar.spec.js b/tests/e2e/agent-3d-livekit-avatar.spec.jsnew file mode 100644index 000000000..abb0edfc0--- /dev/null+++ b/tests/e2e/agent-3d-livekit-avatar.spec.js@@ -0,0 +1,154 @@+import { test, expect } from '@playwright/test';+import { spawn } from 'node:child_process';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { SignJWT } from 'jose';++// Work order 087: proves the LiveKit avatar plugin end to end, browser side.+// A real voice agent (integrations/livekit/livekit-plugins-threews/examples/+// avatar_agent.py) joins a real local open-source livekit-server (started by+// this repo's `npm run dev:livekit` docker helper, or already running from a+// prior manual `docker run livekit/livekit-server --dev`) and speaks three+// lines through a real espeak-ng TTS. The three.ws `threews.AvatarSession`+// plugin republishes the agent's turns as `transcript`/`gesture` data-channel+// messages, and a real Chromium loads `<agent-3d voice="livekit">` (via+// public/agent-3d-livekit-harness.html) to prove those messages actually+// drive the browser's existing gesture/lipsync protocol.+//+// Fidelity: only the agent's own livekit-token mint+// (`/api/agents/:id/livekit-token`, real production DB + auth state this+// test box doesn't have) is route-intercepted, with a token built the same+// way the real handler signs one (same claim shape, same devkey/devsecret+// dev-server credentials the local livekit-server was started with). Every+// other hop (the LiveKit room, the agent process, the plugin's data channel,+// the browser's `LiveKitVoice` client, the avatar protocol) is real.++const ROOM = `e2e-livekit-${Date.now()}`;+const LIVEKIT_URL = process.env.LIVEKIT_URL || 'ws://localhost:7880';+const LIVEKIT_API_KEY = process.env.LIVEKIT_API_KEY || 'devkey';+const LIVEKIT_API_SECRET = process.env.LIVEKIT_API_SECRET || 'devsecret';++const HERE = path.dirname(fileURLToPath(import.meta.url));+const EXAMPLES_DIR = path.join(+ HERE,+ '..',+ '..',+ 'integrations',+ 'livekit',+ 'livekit-plugins-threews',+ 'examples',+);++async function mintBrowserToken(room) {+ const secret = new TextEncoder().encode(LIVEKIT_API_SECRET);+ const now = Math.floor(Date.now() / 1000);+ return new SignJWT({+ video: { roomJoin: true, room, canPublish: true, canSubscribe: true },+ sub: 'e2e-browser',+ iss: LIVEKIT_API_KEY,+ nbf: now,+ exp: now + 3600,+ })+ .setProtectedHeader({ alg: 'HS256' })+ .sign(secret);+}++function spawnAgent(room) {+ const child = spawn('python3', ['-u', 'avatar_agent.py', 'connect', '--room', room], {+ cwd: EXAMPLES_DIR,+ env: {+ ...process.env,+ LIVEKIT_URL,+ LIVEKIT_API_KEY,+ LIVEKIT_API_SECRET,+ },+ stdio: ['ignore', 'pipe', 'pipe'],+ });+ let log = '';+ child.stdout.on('data', (d) => (log += d.toString()));+ child.stderr.on('data', (d) => (log += d.toString()));+ return { child, getLog: () => log };+}++test.describe('agent-3d LiveKit avatar plugin (real local livekit-server)', () => {+ test.setTimeout(120_000);++ test('a real agent speaking in the room drives the browser avatar lipsync + gesture', async ({+ page,+ }) => {+ const agent = spawnAgent(ROOM);+ try {+ const token = await mintBrowserToken(ROOM);+ await page.route('**/api/agents/*/livekit-token', (route) =>+ route.fulfill({+ status: 200,+ contentType: 'application/json',+ body: JSON.stringify({ token, serverUrl: LIVEKIT_URL }),+ }),+ );++ const errors = [];+ page.on('pageerror', (err) => errors.push(String(err?.message || err)));++ await page.goto('/agent-3d-livekit-harness.html', { waitUntil: 'domcontentloaded' });++ // Wait for the element to actually open the LiveKit room before+ // subscribing to its protocol (the protocol object itself is created+ // synchronously on connect, independent of the room join finishing).+ await expect+ .poll(() => page.evaluate(() => !!document.getElementById('a3d')?._avatar?.protocol), {+ timeout: 60_000,+ })+ .toBe(true);++ const eventsPromise = page.evaluate(() => {+ return new Promise((resolve) => {+ const protocol = document.getElementById('a3d')._avatar.protocol;+ const seen = [];+ const onSpeak = (action) => seen.push({ type: 'speak', text: action.payload?.text });+ const onGesture = (action) => seen.push({ type: 'gesture', name: action.payload?.name });+ protocol.on('speak', onSpeak);+ protocol.on('gesture', onGesture);+ // The agent speaks exactly 3 lines (2 carry a gesture cue, the+ // farewell line does not by design); stop once all 3 "speak"+ // events have landed, or after a generous ceiling so a real+ // failure still resolves with whatever partial evidence it+ // gathered.+ const finish = () => {+ protocol.off('speak', onSpeak);+ protocol.off('gesture', onGesture);+ resolve(seen);+ };+ const ceiling = setTimeout(finish, 45_000);+ const checkDone = () => {+ if (seen.filter((e) => e.type === 'speak').length >= 3) {+ clearTimeout(ceiling);+ finish();+ }+ };+ protocol.on('speak', checkDone);+ });+ });++ const events = await eventsPromise;++ await page.screenshot({+ path: 'test-results/agent-3d-livekit-avatar-final.png',+ fullPage: false,+ });++ const speaks = events.filter((e) => e.type === 'speak').map((e) => e.text);+ const gestures = events.filter((e) => e.type === 'gesture').map((e) => e.name);++ expect(speaks, `agent log:\n${agent.getLog()}`).toEqual([+ "Hello! I'm the three.ws avatar, speaking through LiveKit.",+ 'Yes, this plugin publishes a gesture cue with that line.',+ 'Goodbye, and thanks for watching the demo!',+ ]);+ expect(gestures).toEqual(['wave', 'nod']);+ expect(errors, `page errors: ${errors.join(' | ')}`).toEqual([]);+ } finally {+ agent.child.kill('SIGTERM');+ }+ });+});