Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(agent-system): document companion presence, add the changelog entry

nirholase25cdc318dd5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b5139a7b3..48f8941e1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,7 @@ Public history for [three.ws](https://three.ws), newest first. New pages come fr
- **Docs · Engine QA** (`/docs/engine-qa`): A free real-time-engine fitness grade for any 3D model: draw calls, material count, texel density, a per-level LOD triangle budget, and whether the mesh is cheap and closed enough to use as its own physics collider.
- **<agent-3d> gets model-viewer's AR contract: ar-modes, variants, hotspots, and /ar drops the third-party CDN**: <agent-3d> now matches the AR/viewer contract the market expects from model-viewer (Apache-2.0): ar-modes picks which launch paths to try (WebXR, Scene Viewer, Quick Look), ar-placement anchors floor or wall, ios-src supplies a pre-baked USDZ or the element generates a best-effort one on device, variant-name switches a GLB's real KHR_materials_variants materials live, and any element with slot="hotspot" tracks a 3D position and flies the camera to a saved orbit on click. /avatars/:id/ar and /agents/:id/ar (pages/ar.html) now run entirely on <agent-3d ar>; the page no longer loads a third-party viewer from a CDN. (`/avatars`) `[feature]`
- **Animation Studio: trim, loop-blend, and save any generated clip to your library**: The Animation Studio's prompt-to-clip flow (type a prompt, generate motion on the self-hosted GPU worker, preview it retargeted onto the loaded avatar) now has an editing and persistence half. The transport grows trim start/end inputs, a loop-blend duration field, and a loop toggle, all recomputed through a non-destructive edit path so the live preview, the GLB export, and the saved clip always agree on the current edit. A Save to library button collects a name and visibility and posts the canonical, pre-retarget clip to the shared clip library, so it plays back correctly on any future rig, not just the avatar it was previewed on. A separate bug that crashed every generated clip's playback the moment it reached the browser (AnimationClip.parse() expected userData as a JSON string, the server was sending an object) is also fixed. (`/pose`) `[feature, fix]`
+- **Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync**: Agent avatars carry more of what makes a companion feel present. setMood(valence, arousal) sets a sustained resting bias (a soft smile or a worried brow that drifts in over about a second) on top of the existing per-utterance emotion blend, and a reply that is nothing but emoji (no English sentiment words at all) now drives a real expression change instead of staying neutral, because the built-in sentiment scorer reads emoji as direct valence/arousal signal. The LLM brain gets a general-purpose `gesture` tool covering all 19 body-language slots (nod, shake, shrug, bow, point, think, and the rest), so it can react physically mid-turn instead of only through a skill or an emotion threshold. While a real VAD/STT signal says the user is talking, the avatar nods on a fixed cadence to show it is listening, and stops the moment they stop. ElevenLabs text-to-speech now drives lip-sync from the provider's real per-character timestamps (any language the voice speaks), falling back to the energy-based analyser only if that call fails, so mouth shapes track what is actually being said instead of approximating it from amplitude alone. (`/docs/agent-system`) `[feature, improvement]`
- **Engine QA: a free real-time-engine fitness grade, plus real LOD chains**: A new free grade answers the question a game engine's import step asks that neither glTF validation nor simulation readiness does: draw calls, material count, texel density against the mesh's real-world size, a per-level triangle budget for a LOD chain, and whether the mesh is cheap and closed enough to use as its own physics collider. GET/POST /api/3d/engine-qa grades any public GLB (format=stl exports a watertight mesh for printing); the same browser-only checks run with zero network cost in Rig Doctor, and the full grade with a one-click upgrade in the GLB viewer, through a shared <engine-qa> panel component. The underlying LOD builder (packages/glb-tools's buildLodChain, real meshoptimizer decimation tagged with MSFT_lod) is now also available on generation: send { "lods": true } to POST /api/forge and the finished creation gets a combined LOD-chain variant alongside its delivered mesh, the same opt-in, fire-and-forget pattern as the existing web-delivery variant. (`/docs/engine-qa`) `[feature]`
- **Every generated and purchased model now ships with a signed provenance credential, free, by default**: Content credentials (specs/PROVENANCE_3D.md) used to be an on-demand, Claude-track action: you had to call anchor_provenance to get one. Now every GLB the Forge produces, across every generation lane (free NIM, Forge Pro, rigged avatars, self-host TRELLIS/Hunyuan, async GCP workers), gets an ed25519-signed content credential the moment it's materialized, and every first download of a paid asset from the x402 asset bazaar gets one too, with the hash and a verify link returned right in the download response. Signing and storage are free and local, no Solana write and no spend, so there's nothing to gate; the separate paid anchor_provenance upgrade, which additionally commits the hash on-chain, is untouched and still available. A credential is keyed by content hash, so it's issued exactly once per file no matter how many times a lane or endpoint touches it. Model pages now show the same verified / tampered / unverified badge the 3D viewer already had, and a third party can check any model for free with GET /api/provenance. The owner can turn automatic issuance off platform-wide with PROVENANCE_AUTO_DISABLE=true; nothing else about verification changes if they do. (`/docs/provenance`) `[feature, security]`
- **Real VRM avatars: spring bones, expressions, and .vrma clips via @pixiv/three-vrm**: The hand-rolled VRM loader is gone. vrm-boot.js now registers the real VRMLoaderPlugin (@pixiv/three-vrm) on the three.ws GLTFLoader, so VRM 0.x and 1.0 avatars get genuine MToon materials, spring-bone physics, node constraints, look-at, and all 18 expression presets, driven live by vrm.update(delta) in AnimationManager and mapped onto the ARKit-52 face-capture driver. VRMAnimationLoaderPlugin loads .vrma clips into the same retargeting pipeline that drives non-VRM humanoids, so a single clip plays on either. The Ready Player Me import page (/import-rpm) now accepts .vrm files alongside .glb, and a long-standing bug that 400'd every import at the final save step (it sent an invalid source value) is fixed. docs/avatar-engines.md upgrades three-vrm from interop to live. (`/docs/avatar-engines`) `[feature, fix]`
diff --git a/data/changelog.json b/data/changelog.json
index 309c2455d..2b4271a60 100644
--- a/data/changelog.json
+++ b/data/changelog.json
@@ -1,6 +1,16 @@
{
"$comment": "Curated changelog entries for three.ws, the editorial layer of the public changelog. New PAGE launches are tracked automatically via the `added` field in data/pages.json; this file is for everything else holders care about: improvements, fixes, SDK releases, trading/payments capabilities, security work. scripts/build-page-index.mjs merges both sources into CHANGELOG.md, public/changelog.json, and public/changelog.xml. Newest first. Allowed tags: feature, improvement, fix, sdk, infra, docs, security, desktop (desktop entries become the desktop app's release notes). `link` is optional and must be a live three.ws path.",
"entries": [
+ {
+ "date": "2026-10-12",
+ "title": "Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync",
+ "link": "/docs/agent-system",
+ "summary": "Agent avatars carry more of what makes a companion feel present. setMood(valence, arousal) sets a sustained resting bias (a soft smile or a worried brow that drifts in over about a second) on top of the existing per-utterance emotion blend, and a reply that is nothing but emoji (no English sentiment words at all) now drives a real expression change instead of staying neutral, because the built-in sentiment scorer reads emoji as direct valence/arousal signal. The LLM brain gets a general-purpose `gesture` tool covering all 19 body-language slots (nod, shake, shrug, bow, point, think, and the rest), so it can react physically mid-turn instead of only through a skill or an emotion threshold. While a real VAD/STT signal says the user is talking, the avatar nods on a fixed cadence to show it is listening, and stops the moment they stop. ElevenLabs text-to-speech now drives lip-sync from the provider's real per-character timestamps (any language the voice speaks), falling back to the energy-based analyser only if that call fails, so mouth shapes track what is actually being said instead of approximating it from amplitude alone.",
+ "tags": [
+ "feature",
+ "improvement"
+ ]
+ },
{
"date": "2026-10-12",
"title": "Scene Studio: an Agent tab that edits your open scene by prompt, plus a TSL node material editor",
diff --git a/docs/agent-system.md b/docs/agent-system.md
index 216e35cd5..366029227 100644
--- a/docs/agent-system.md
+++ b/docs/agent-system.md
@@ -46,6 +46,8 @@ Every event detail has the shape `{ type, payload, timestamp, agentId, sourceSki
| `load-end` | `{ uri, error? }` | Avatar (concern or celebration) |
| `validate` | `{ errors, warnings }` | Avatar, identity |
| `presence` | `{ state }` | Agent home UI |
+| `listen-start` | `{}` | Avatar (attentive gaze, arms the backchannel nod cadence) |
+| `listen-end` | `{ text }` | Avatar (releases attentive gaze, stops nodding) |
The identity module records the following event types to the backend (fire-and-forget via `POST /api/agent-actions`): `speak`, `remember`, `sign`, `skill-done`, `validate`, `load-end`.
@@ -124,6 +126,14 @@ Triggers the wave gesture animation on the avatar. Use for greetings, farewells,
|-----|------|---------|-------------|
| `duration_ms` | integer | 1500 | Gesture duration in milliseconds (500–5000) |
+### `gesture`
+
+The general-purpose body-language tool: any of the 19 slots in `runtime/animation-slots.js` (`nod`, `shake`, `wave`, `shrug`, `bow`, `point`, `celebrate`, `concern`, `think`, `inspect`, `present`, `sign`, `curiosity`, `patience`, `fidget`, `dance`, `manipulate`, `conjure`, every slot except the resting `idle`), by name, resolved through the agent's animation override map (`meta.edits.animations`) before falling back to the default clip. The LLM picks this when it wants to react physically mid-turn rather than only when a skill or the emotion-threshold triggers fire one for it. Plays once; the body returns to its resting pose.
+
+| Arg | Type | Required | Description |
+|-----|------|----------|-------------|
+| `name` | string (enum) | Yes | Which gesture to perform |
+
### `lookAt`
Changes the agent's gaze direction.
@@ -202,6 +212,7 @@ Six named emotions, each a float in [0, 1]:
| `patience` | 0.035/s | ~20 seconds |
| `curiosity` | 0.12/s | ~8 seconds |
| `empathy` | 0.055/s | ~13 seconds |
+| `uncertain` | 0.1/s | ~7 seconds |
| `neutral` | computed | 1 − sum(others) |
Every frame, each emotion decays linearly by its rate × delta-time. `neutral` is always `max(0, 1 − sum)`. Stimuli are added with `_injectStimulus(emotion, weight)`, which clamps the result to 1.
@@ -224,7 +235,15 @@ Events on the protocol bus automatically inject stimuli:
| `validate` (errors > 0) | `concern += 0.4 + errors × 0.1` |
| `validate` (clean) | `celebration += 0.85` |
-The vocabulary-based sentiment analysis runs entirely in-browser with no external API. It scans text for keyword buckets (`concern: ['error', 'failed', ...]`, `celebration: ['success', 'complete', ...]`) and computes a valence (−1..1) and arousal (0..1) score.
+The vocabulary-based sentiment analysis runs entirely in-browser with no external API. It scans text for keyword buckets (`concern: ['error', 'failed', ...]`, `celebration: ['success', 'complete', ...]`) and computes a valence (−1..1) and arousal (0..1) score. Emoji in the text contribute to the same valence/arousal totals independent of any English keywords (an `EMOJI_SENTIMENT` map in `agent-avatar.js`, matched as substrings so an emoji glued to punctuation with no word boundary still registers), so a reply that is nothing but "🎉" or "😢" still drives a visible expression change.
+
+### Sustained mood
+
+Beyond the per-utterance stimulus blend above, an agent can carry a **sustained mood**: a resting valence/arousal bias that persists until explicitly changed, set with `el.setMood(valence, arousal, { reducedMotion? })` (`-1..1`, `0..1`). `AgentAvatar._applyMoodLayer()` lerps the applied mood toward the target over about a second (so a mood change drifts in rather than snapping) and layers it on top of the transient emotion targets every frame: positive valence adds a soft resting smile and cheek squint, negative valence adds a frown and worried inner brow, arousal above the ~0.4 baseline widens the eyes (alert), below it droops the lids (relaxed). Mood also nudges the idle head tilt and gaze. A mood set before the empathy layer attaches is queued (`_pendingMood`) and applied on boot, so it is safe to call from a manifest's init step.
+
+### Active listening
+
+While a real VAD/STT signal says the user is talking (`listen-start` on the protocol bus, never a fabricated timer), the avatar holds a slightly raised, attentive gaze and nods on a fixed ~2.8s cadence to backchannel that it is tracking the user, the same way a person nods along in conversation. A one-shot gesture or a performing routine already in flight always takes priority; the nod timer simply holds at zero until the avatar is free again. `listen-end` releases the gaze bias and stops the cadence immediately. `LiveKitVoice` emits both events from the voice server's VAD signal; any other STT integration can emit them the same way to get the same behavior.
### Morph target control
@@ -364,6 +383,8 @@ For the full skill bundle format, see [SKILL_SPEC.md](../specs/SKILL_SPEC.md).
An ElevenLabs config also carries `agentId`, the agent the voice belongs to. `Runtime` fills it in from its own agent id (an explicit manifest or `voiceConfig` value still wins) and `ElevenLabsTTS` forwards it to the proxy, because `/api/tts/eleven` only serves a voice cloned on the owner's own ElevenLabs key when the request names its agent; that is what lets a bound voice speak to embed visitors and signed-out listeners, not just the signed-in owner. `agentVoiceConfig(agent)` in `runtime/speech.js` turns a public agent record (`voice_provider`, `voice_id`, `voice_model`, `voice_settings`) into that config, and returns `null` when no ElevenLabs voice is bound so a caller can render a "no voice configured" state instead of guessing.
+**Lip-sync accuracy:** when a scene is passed to `speak()`, `ElevenLabsTTS` calls ElevenLabs' `/v1/text-to-speech/{voiceId}/with-timestamps` endpoint instead of the plain streaming endpoint. That returns real per-character timestamps (`alignment`) alongside the audio, which `alignmentToSequence()` (`runtime/lipsync.js`) turns into a `{viseme, startMs, endMs}` sequence driven straight onto the mouth morphs via `playVisemeSequence()`, correct for any language the voice speaks (a character with no mapped viseme still carries its real timing, falling back to a generic open-mouth shape rather than dropping the beat). If the timestamped call fails for any reason (network, an older proxy), `speak()` falls back to the original streaming endpoint with the energy-based `LipSyncAnalyser` path (amplitude-driven, language-agnostic but less precise), so the mouth is never left silent.
+
### Speech-to-Text
| Provider | Notes |
diff --git a/public/changelog-recent.json b/public/changelog-recent.json
index c243f860a..a13e000bd 100644
--- a/public/changelog-recent.json
+++ b/public/changelog-recent.json
@@ -1,11 +1,11 @@
{
- "generated_at": "2026-10-12T10:51:38.414Z",
+ "generated_at": "2026-10-12T20:40:00.148Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
"url": "https://three.ws"
},
- "total": 4045,
+ "total": 4046,
"complete": false,
"full_feed": "/changelog.json",
"entries": [
@@ -60,6 +60,17 @@
"fix"
]
},
+ {
+ "date": "2026-10-12",
+ "type": "update",
+ "title": "Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync",
+ "summary": "Agent avatars carry more of what makes a companion feel present. setMood(valence, arousal) sets a sustained resting bias (a soft smile or a worried brow that drifts in over about a second) on top of the existing per-utterance emotion blend, and a reply that is nothing but emoji (no English sentiment words at all) now drives a real expression change instead of staying neutral, because the built-in sentiment scorer reads emoji as direct valence/arousal signal. The LLM brain gets a general-purpose `gesture` tool covering all 19 body-language slots (nod, shake, shrug, bow, point, think, and the rest), so it can react physically mid-turn instead of only through a skill or an emotion threshold. While a real VAD/STT signal says the user is talking, the avatar nods on a fixed cadence to show it is listening, and stops the moment they stop. ElevenLabs text-to-speech now drives lip-sync from the provider's real per-character timestamps (any language the voice speaks), falling back to the energy-based analyser only if that call fails, so mouth shapes track what is actually being said instead of approximating it from amplitude alone.",
+ "link": "/docs/agent-system",
+ "tags": [
+ "feature",
+ "improvement"
+ ]
+ },
{
"date": "2026-10-12",
"type": "update",
@@ -1538,16 +1549,6 @@
"tags": [
"launch"
]
- },
- {
- "date": "2026-10-09",
- "type": "launch",
- "title": "grok-skill.md",
- "summary": "The three.ws skill file for Grok and Grok Bot: the connector URL to add, runnable curl calls for the free 3D studio over plain HTTPS, the links every result returns, the free limits, and the spend rule. Upload it to Grok Skills or have Grok Bot read it.",
- "link": "/grok-skill.md",
- "tags": [
- "launch"
- ]
}
]
}
diff --git a/public/changelog.json b/public/changelog.json
index c2de191b9..bb0be0fbd 100644
--- a/public/changelog.json
+++ b/public/changelog.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-10-12T10:51:38.392Z",
+ "generated_at": "2026-10-12T20:40:00.126Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json + data/changelog.json",
"site": {
"name": "three.ws",
@@ -57,6 +57,17 @@
"fix"
]
},
+ {
+ "date": "2026-10-12",
+ "type": "update",
+ "title": "Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync",
+ "summary": "Agent avatars carry more of what makes a companion feel present. setMood(valence, arousal) sets a sustained resting bias (a soft smile or a worried brow that drifts in over about a second) on top of the existing per-utterance emotion blend, and a reply that is nothing but emoji (no English sentiment words at all) now drives a real expression change instead of staying neutral, because the built-in sentiment scorer reads emoji as direct valence/arousal signal. The LLM brain gets a general-purpose `gesture` tool covering all 19 body-language slots (nod, shake, shrug, bow, point, think, and the rest), so it can react physically mid-turn instead of only through a skill or an emotion threshold. While a real VAD/STT signal says the user is talking, the avatar nods on a fixed cadence to show it is listening, and stops the moment they stop. ElevenLabs text-to-speech now drives lip-sync from the provider's real per-character timestamps (any language the voice speaks), falling back to the energy-based analyser only if that call fails, so mouth shapes track what is actually being said instead of approximating it from amplitude alone.",
+ "link": "/docs/agent-system",
+ "tags": [
+ "feature",
+ "improvement"
+ ]
+ },
{
"date": "2026-10-12",
"type": "update",
diff --git a/public/changelog.xml b/public/changelog.xml
index 76052c068..b479a04e8 100644
--- a/public/changelog.xml
+++ b/public/changelog.xml
@@ -46,6 +46,14 @@
<category>feature</category>
<description>The Animation Studio's prompt-to-clip flow (type a prompt, generate motion on the self-hosted GPU worker, preview it retargeted onto the loaded avatar) now has an editing and persistence half. The transport grows trim start/end inputs, a loop-blend duration field, and a loop toggle, all recomputed through a non-destructive edit path so the live preview, the GLB export, and the saved clip always agree on the current edit. A Save to library button collects a name and visibility and posts the canonical, pre-retarget clip to the shared clip library, so it plays back correctly on any future rig, not just the avatar it was previewed on. A separate bug that crashed every generated clip's playback the moment it reached the browser (AnimationClip.parse() expected userData as a JSON string, the server was sending an object) is also fixed.</description>
</item>
+ <item>
+ <title>Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync</title>
+ <link>https://three.ws/docs/agent-system</link>
+ <guid isPermaLink="false">2026-10-12:Companion presence: sustained mood, emoji-driven expression, an LLM gesture tool, active-listening nods, and real multi-language lip-sync</guid>
+ <pubDate>Mon, 12 Oct 2026 12:00:00 GMT</pubDate>
+ <category>feature</category>
+ <description>Agent avatars carry more of what makes a companion feel present. setMood(valence, arousal) sets a sustained resting bias (a soft smile or a worried brow that drifts in over about a second) on top of the existing per-utterance emotion blend, and a reply that is nothing but emoji (no English sentiment words at all) now drives a real expression change instead of staying neutral, because the built-in sentiment scorer reads emoji as direct valence/arousal signal. The LLM brain gets a general-purpose `gesture` tool covering all 19 body-language slots (nod, shake, shrug, bow, point, think, and the rest), so it can react physically mid-turn instead of only through a skill or an emotion threshold. While a real VAD/STT signal says the user is talking, the avatar nods on a fixed cadence to show it is listening, and stops the moment they stop. ElevenLabs text-to-speech now drives lip-sync from the provider's real per-character timestamps (any language the voice speaks), falling back to the energy-based analyser only if that call fails, so mouth shapes track what is actually being said instead of approximating it from amplitude alone.</description>
+ </item>
<item>
<title>Engine QA: a free real-time-engine fitness grade, plus real LOD chains</title>
<link>https://three.ws/docs/engine-qa</link>
diff --git a/public/features.json b/public/features.json
index 05804ea91..a4239960c 100644
--- a/public/features.json
+++ b/public/features.json
@@ -1,5 +1,5 @@
{
- "generated_at": "2026-10-12T10:51:38.378Z",
+ "generated_at": "2026-10-12T20:40:00.113Z",
"generated_by": "scripts/build-page-index.mjs from data/pages.json",
"site": {
"name": "three.ws",