Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(voice-integrations): add the LiveKit/Pipecat avatar guide, wire it into start-here, STRUCTURE.md and the changelog
nirholasaf2261e685ce
diff --git a/STRUCTURE.md b/STRUCTURE.mdindex 9ee7282dd..e10bb29c1 100644--- a/STRUCTURE.md+++ b/STRUCTURE.md@@ -354,6 +354,7 @@ bottom.| Desktop local agent runtime | [apps/desktop/src/runtime/](apps/desktop/src/runtime) (runtime, keystore, feed, live executor), [src/main/local-runtime.js](apps/desktop/src/main/local-runtime.js), console view [src/console/views/local.js](apps/desktop/src/console/views/local.js), smoke [scripts/smoke-runtime.mjs](apps/desktop/scripts/smoke-runtime.mjs) | In-repo; Linux AppImage and deb built, Mac and Windows builds and signing are owner actions | Runs agents, paper and live strategies, limit orders and automations on the user's machine with keys in the OS keychain, native payload-hash approvals, a tray menu, and a mascot that shows agent state. Docs: [docs/desktop-local-runtime.md](docs/desktop-local-runtime.md). Tests: tests/desktop-runtime.test.js || Desktop menu bar panel | [apps/desktop/src/main/panel.js](apps/desktop/src/main/panel.js), pure logic [src/panel/model.js](apps/desktop/src/panel/model.js), renderer [src/panel/](apps/desktop/src/panel), preload [src/panel-preload.cjs](apps/desktop/src/panel-preload.cjs) | In-repo; ships with the desktop app | A popover under the tray icon: the selected agent alive in 3D (the published walk embed, gesture tied to local runtime state), SOL balance, start and stop, a pending-approvals card that opens the native review dialog (the panel has no approve call), alerts, companion switch and a one-line ask box. Docs: [docs/desktop-menu-bar-panel.md](docs/desktop-menu-bar-panel.md). Tests: tests/desktop-panel.test.js || DCC tool integrations (Blender, ComfyUI, Modly) | [integrations/](integrations): [integrations/blender/](integrations/blender), [integrations/comfyui/](integrations/comfyui), Modly extensions [integrations/modly/](integrations/modly) (`three-ws/` cloud nodes, `three-ws-publish/` publish node), shared Python client [integrations/_pyclient/](integrations/_pyclient) | In-repo; Modly extensions install by linking the folder, one-click GitHub install is owner-gated ([SUBMITTING.md](integrations/modly/SUBMITTING.md)) | First-party plugins that drive the public Forge pipeline (`/api/forge`, rig, remesh, publish) from inside the tools artists already use, so a model is generated without leaving the DCC. Docs: [integrations/README.md](integrations/README.md), [integrations/modly/README.md](integrations/modly/README.md) |+| LiveKit and Pipecat avatar integrations | [integrations/livekit/livekit-plugins-threews/](integrations/livekit/livekit-plugins-threews) (published as `livekit-plugins-threews`), [integrations/pipecat/pipecat-threews/](integrations/pipecat/pipecat-threews) (published as `pipecat-threews`) | In-repo; PyPI publish and the upstream community-integration PRs are owner actions | A LiveKit Agents `AvatarSession` plugin and a Pipecat `FrameProcessor` that both republish a running voice pipeline's TTS and user-turn frames as the same `transcript`/`gesture`/`speech_started`/`speech_ended` JSON message shape over the room's data channel, which `<agent-3d voice="livekit">` ([src/runtime/livekit-voice.js](src/runtime/livekit-voice.js)) already renders client-side, no server-side video rendering. Each ships a runnable local example against a real open-source `livekit-server`, no cloud account needed. Docs: [docs/voice-framework-integrations.md](docs/voice-framework-integrations.md). Tests: `integrations/livekit/livekit-plugins-threews/tests/`, `integrations/pipecat/pipecat-threews/tests/`, browser E2E [tests/e2e/agent-3d-livekit-avatar.spec.js](tests/e2e/agent-3d-livekit-avatar.spec.js), [tests/e2e/agent-3d-pipecat-avatar.spec.js](tests/e2e/agent-3d-pipecat-avatar.spec.js) || Base58 difficulty model (the honest math) | [src/solana/vanity/base58-distribution.js](src/solana/vanity/base58-distribution.js), dispatch in [src/solana/vanity/validation.js](src/solana/vanity/validation.js); ported copies in [solana-agent-sdk/src/vanity/verify.ts](solana-agent-sdk/src/vanity/verify.ts) and [mcp-server/src/lib/base58-distribution.js](mcp-server/src/lib/base58-distribution.js), held in agreement by [tests/vanity-model-parity.test.js](tests/vanity-model-parity.test.js) | In-repo | The exact probability that a random Solana address matches a vanity pattern, and the single source of truth behind every quote, price, rarity tier, bounty payout and signed receipt. Base58 is a positional encoding of a 256-bit integer, so the *leading* character is not uniform: the 40 symbols `K`-`z` are 17.2x harder to lead with than `5`-`H`, a 58x spread the old flat 58ⁿ model erased. Models are versioned (`base58-exact/v2`, superseding `58^effectiveLength`) and receipts name theirs, so attestations issued before the correction still verify. Spec: [docs/PROTOCOL-vanity.md](docs/PROTOCOL-vanity.md#41-difficulty-model-versioning) || Vanity address grinder (Rust → WASM) | [crates/vanity-grinder/](crates/vanity-grinder) → compiled to [src/solana/vanity/wasm/](src/solana/vanity/wasm); consumed by [src/agent-vanity-grinder.js](src/agent-vanity-grinder.js), [src/agent-eth-vanity-card.js](src/agent-eth-vanity-card.js) | In-repo | The Rust source behind in-browser Solana/EVM vanity-address mining. Compiled to WebAssembly so grinding runs on the visitor's machine, never the server. Surfaced as the `vanity_grinder` MCP tool and the premium pre-ground inventory (`vanity_premium`). || x402 payment modal (drop-in UI) | [x402-payment-modal/](x402-payment-modal) | Published as `@three-ws/x402-payment-modal` | A drop-in payment modal for any x402 paid endpoint — one ES import turns a 402 challenge into a wallet-connect + pay flow. Has a build step, so it sits outside the zero-dependency `packages/*` SDK family. |diff --git a/data/changelog.json b/data/changelog.jsonindex 1cc9dbf50..09b951ffb 100644--- a/data/changelog.json+++ b/data/changelog.json@@ -1,6 +1,17 @@{"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security, desktop (desktop entries become the desktop app's release notes). `link` is optional and must be a live three.ws path.","entries": [+ {+ "date": "2026-10-13",+ "title": "A three.ws avatar now lipsyncs to any LiveKit Agents or Pipecat voice pipeline",+ "link": "/docs/voice-framework-integrations",+ "summary": "Two new packages, livekit-plugins-threews and pipecat-threews, republish a running voice agent's TTS and user-turn frames as the same transcript/gesture/speech_started/speech_ended message shape over the room's data channel, which <agent-3d voice=\"livekit\"> already renders client-side: no server-side video rendering, no change to the browser code at all. The LiveKit package is an AvatarSession plugin dropped into an AgentSession; the Pipecat package is a FrameProcessor placed after the TTS service. Each ships a runnable local example against a real open-source livekit-server with a real local TTS, verified end to end with real browser automation driving the actual avatar lipsync and gesture cues off a live bot in both frameworks. Docs: /docs/voice-framework-integrations.",+ "tags": [+ "feature",+ "sdk",+ "docs"+ ]+ },{"date": "2026-10-13","title": "Open source credits: every dependency, model, and asset source three.ws runs on, in one place",diff --git a/data/pages.json b/data/pages.jsonindex 4f8731365..be6dd8833 100644--- a/data/pages.json+++ b/data/pages.json@@ -6698,6 +6698,14 @@"changefreq": "monthly","added": "2026-07-30"},+ {+ "path": "/docs/voice-framework-integrations",+ "title": "Docs: LiveKit and Pipecat avatar integrations",+ "description": "How to drive a three.ws <agent-3d> avatar's lipsync and gestures from a LiveKit Agents or Pipecat voice pipeline, with the livekit-plugins-threews and pipecat-threews packages, runnable local examples, and the data-channel message contract both share.",+ "priority": 0.7,+ "changefreq": "monthly",+ "added": "2026-10-13"+ },{"path": "/docs/rig-doctor","title": "Rig Doctor reference",diff --git a/docs/start-here.md b/docs/start-here.mdindex 83b266994..771e8ba91 100644--- a/docs/start-here.md+++ b/docs/start-here.md@@ -212,6 +212,7 @@ Want the model in your hands rather than on a screen? [Materialize](./materializ- **Teaching another AI to use three.ws?** → [The Agent Skills pack](./agent-skills.md) — portable `SKILL.md` folders that give any Claude surface (Claude Code, the Claude apps, the Agent SDK) three.ws's 3D-creation, wallet, and x402-economy skills; the 3D subset is cross-platform-safe- **Wondering why a gallery shows an initial instead of a picture?** → [Avatar thumbnails](./avatar-thumbnails.md) — where an avatar's preview image comes from, the one rule every code path obeys (never publish a thumbnail URL whose object doesn't exist), the two crons that keep coverage at 100%, and how to run the backfill- **Adding 3D to a page, or wondering why one feels slow?** → [Page performance](./page-performance.md): the rules a three.ws page plays by, a model preview runs only while someone is looking at it, a model loads when it is wanted, anything that arrives late reserves its box first, and images are fetched at the size they are shown+- **Running your voice agent on LiveKit or Pipecat?** → [LiveKit and Pipecat avatar integrations](./voice-framework-integrations.md): drop `livekit-plugins-threews` or `pipecat-threews` into your pipeline and a real `<agent-3d>` avatar lipsyncs and gestures to it, no server-side rendering, no cloud account required to try it against a local `livekit-server`---diff --git a/docs/voice-framework-integrations.md b/docs/voice-framework-integrations.mdnew file mode 100644index 000000000..1cddd0492--- /dev/null+++ b/docs/voice-framework-integrations.md@@ -0,0 +1,107 @@+# LiveKit and Pipecat avatar integrations++A three.ws `<agent-3d>` avatar renders entirely on the visitor's own device: the+3D model, its rig, and its animation clip library already live in the browser.+When your voice agent runs on [LiveKit Agents](https://github.com/livekit/agents)+or [Pipecat](https://github.com/pipecat-ai/pipecat), you don't need a server+that renders video. Instead, a small package republishes the TTS and+user-turn frames your pipeline already has as JSON messages over the room's+data channel, and `<agent-3d voice="livekit">` lipsyncs and gestures to them+client-side.++**On this page:** [Message contract](#message-contract) · [LiveKit](#livekit-agents) · [Pipecat](#pipecat) · [Run the examples locally](#run-the-examples-locally)++---++## Message contract++Both packages publish the same shape, so the browser side never knows which+backend sent it:++```json+{"type": "transcript", "text": "Hi, how can I help today?"}+{"type": "gesture", "name": "wave"}+{"type": "speech_started"}+{"type": "speech_ended", "transcript": "what can you do"}+```++- `transcript` fires once per finished assistant turn; the browser times it+ into visemes locally (`src/runtime/lipsync.js`).+- `gesture` is a best-effort read of that same turn's text: a greeting or+ farewell triggers `wave`, an agreement triggers `nod`, a disagreement+ triggers `shrug`. A miss just means no gesture fires, never a wrong one.+- `speech_started` / `speech_ended` bracket the user's own turn, so the+ avatar can switch to a listening pose. `speech_ended.transcript` is the+ last finalized transcript captured since the matching `speech_started`;+ it is empty if the turn's STT result hasn't finalized yet.++Point your frontend at `<agent-3d voice="livekit">` (or any embed that loads+[`src/runtime/livekit-voice.js`](https://github.com/nirholas/three.ws/blob/main/src/runtime/livekit-voice.js)),+which already subscribes to this message shape over LiveKit's data channel.+Any transport whose implementation routes output messages to a real data+channel (LiveKit, Daily) works the same way.++## LiveKit Agents++[`livekit-plugins-threews`](https://github.com/nirholas/three.ws/tree/main/integrations/livekit/livekit-plugins-threews)+is an `AvatarSession` plugin: drop it into a LiveKit `AgentSession` and every+`say()`/TTS turn and VAD boundary republishes as avatar motion data.++```bash+pip install livekit-plugins-threews+```++```python+from livekit.plugins import threews++avatar = threews.AvatarSession()+await avatar.start(session, room=ctx.room)+```++Full reference: [`integrations/livekit/livekit-plugins-threews/README.md`](https://github.com/nirholas/three.ws/blob/main/integrations/livekit/livekit-plugins-threews/README.md).++## Pipecat++[`pipecat-threews`](https://github.com/nirholas/three.ws/tree/main/integrations/pipecat/pipecat-threews)+is a `FrameProcessor`: place it after your TTS service and before+`transport.output()` in any Pipecat `Pipeline`.++```bash+pip install pipecat-threews+```++```python+from pipecat.pipeline.pipeline import Pipeline+from pipecat_threews import ThreeWsAvatarProcessor++pipeline = Pipeline([+ transport.input(), stt, context_aggregator.user(), llm, tts,+ ThreeWsAvatarProcessor(),+ transport.output(), context_aggregator.assistant(),+])+```++Your TTS service must be constructed with `push_start_frame=True` and+`push_stop_frames=True` (Pipecat's `TTSService` base class defaults both to+`False`); otherwise `ThreeWsAvatarProcessor` never sees a `TTSStoppedFrame`+and silently never publishes. Full reference:+[`integrations/pipecat/pipecat-threews/README.md`](https://github.com/nirholas/three.ws/blob/main/integrations/pipecat/pipecat-threews/README.md).++## Run the examples locally++Both packages ship a runnable foundational example against a real local,+open-source `livekit-server`, no cloud account or API key required:++```bash+docker run --rm -p 7880:7880 -p 7881:7881 -p 7882:7882/udp \+ -e LIVEKIT_KEYS="devkey: devsecret" \+ livekit/livekit-server --dev --bind 0.0.0.0+```++- LiveKit: `integrations/livekit/livekit-plugins-threews/examples/avatar_agent.py`+- Pipecat: `integrations/pipecat/pipecat-threews/examples/avatar_bot.py`++Each example speaks a short scripted sequence through a real local TTS+(`espeak-ng` for Pipecat's example), so you can point `<agent-3d+voice="livekit">` at the same room and watch the avatar lipsync and gesture+to a real agent with no cloud dependency on either side.