Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

4.4 KB
Raw Blame

LiveKit and Pipecat avatar integrations

A three.ws <agent-3d> avatar renders entirely on the visitor's own device: the 3D model, its rig, and its animation clip library already live in the browser. When your voice agent runs on LiveKit Agents or Pipecat, you don't need a server that renders video. Instead, a small package republishes the TTS and user-turn frames your pipeline already has as JSON messages over the room's data channel, and <agent-3d voice="livekit"> lipsyncs and gestures to them client-side.

On this page: Message contract · LiveKit · Pipecat · Run the examples locally


Message contract

Both packages publish the same shape, so the browser side never knows which backend sent it:

{"type": "transcript", "text": "Hi, how can I help today?"}
{"type": "gesture", "name": "wave"}
{"type": "speech_started"}
{"type": "speech_ended", "transcript": "what can you do"}
  • transcript fires once per finished assistant turn; the browser times it into visemes locally (src/runtime/lipsync.js).
  • gesture is a best-effort read of that same turn's text: a greeting or farewell triggers wave, an agreement triggers nod, a disagreement triggers shrug. A miss just means no gesture fires, never a wrong one.
  • speech_started / speech_ended bracket the user's own turn, so the avatar can switch to a listening pose. speech_ended.transcript is the last finalized transcript captured since the matching speech_started; it is empty if the turn's STT result hasn't finalized yet.

Point your frontend at <agent-3d voice="livekit"> (or any embed that loads src/runtime/livekit-voice.js), which already subscribes to this message shape over LiveKit's data channel. Any transport whose implementation routes output messages to a real data channel (LiveKit, Daily) works the same way.

LiveKit Agents

livekit-plugins-threews is an AvatarSession plugin: drop it into a LiveKit AgentSession and every say()/TTS turn and VAD boundary republishes as avatar motion data.

pip install livekit-plugins-threews
from livekit.plugins import threews

avatar = threews.AvatarSession()
await avatar.start(session, room=ctx.room)

Full reference: integrations/livekit/livekit-plugins-threews/README.md.

Pipecat

pipecat-threews is a FrameProcessor: place it after your TTS service and before transport.output() in any Pipecat Pipeline.

pip install pipecat-threews
from pipecat.pipeline.pipeline import Pipeline
from pipecat_threews import ThreeWsAvatarProcessor

pipeline = Pipeline([
    transport.input(), stt, context_aggregator.user(), llm, tts,
    ThreeWsAvatarProcessor(),
    transport.output(), context_aggregator.assistant(),
])

Your TTS service must be constructed with push_start_frame=True and push_stop_frames=True (Pipecat's TTSService base class defaults both to False); otherwise ThreeWsAvatarProcessor never sees a TTSStoppedFrame and silently never publishes. Full reference: integrations/pipecat/pipecat-threews/README.md.

Run the examples locally

Both packages ship a runnable foundational example against a real local, open-source livekit-server, no cloud account or API key required:

docker run --rm -p 7880:7880 -p 7881:7881 -p 7882:7882/udp \
  -e LIVEKIT_KEYS="devkey: devsecret" \
  livekit/livekit-server --dev --bind 0.0.0.0
  • LiveKit: integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py
  • Pipecat: integrations/pipecat/pipecat-threews/examples/avatar_bot.py

Each example speaks a short scripted sequence through a real local TTS (espeak-ng for Pipecat's example), so you can point <agent-3d voice="livekit"> at the same room and watch the avatar lipsync and gesture to a real agent with no cloud dependency on either side.