Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(nvidia): revise the Audio2Face forum draft to the approval pattern with latencies re-measured today
qhqf58202a639ad6
diff --git a/docs/nvidia-forum-browser-digital-human.md b/docs/nvidia-forum-browser-digital-human.mdindex 4dcacac4d..7abddd253 100644--- a/docs/nvidia-forum-browser-digital-human.md+++ b/docs/nvidia-forum-browser-digital-human.md@@ -1,33 +1,35 @@----venue: NVIDIA Developer Forums (community showcase)-account: three.ws (official)+venue: NVIDIA Developer Forums+category: AI & Data Science > NVIDIA NIM > Models (where the three approved posts live; `audio2face` is already a tag there)+account: nichxbtsuggested_title: "A digital human in a browser tab: streaming Audio2Face-3D onto whatever rig the visitor brought"-description: "How three.ws drives real-time facial animation on a WebGL avatar from NVIDIA Audio2Face-3D over NVCF gRPC, with Magpie TTS in front and Riva ASR behind it, and how we map an ARKit-52 blendshape track onto VRM and Oculus rigs that were never built for it. Measured latencies from production."-tags: [ace, audio2face, riva, magpie, nim, digital-humans, webgl, inception]-canonical: https://three.ws/docs/nvidia-forum-browser-digital-human.md-status: draft, owner approval required before posting (external-channel gate in CLAUDE.md)+description: "How three.ws drives real-time facial animation on a WebGL avatar from NVIDIA Audio2Face-3D over NVCF gRPC, with Magpie TTS in front and Riva ASR behind it, and how an ARKit-52 blendshape track is mapped onto VRM and Oculus rigs that were never built for it. Latencies measured against production on 2026-09-22."+tags: [nim, audio2face, riva]+links_in_body: 2 (the live demo, and our own retirement topic on this forum). Repo paths are plain text on purpose; see the approval pattern in nvidia-visibility-map.md.+images: 0 (a short clip of the demo would be the one allowed image, if the owner wants it)+status: revised to the approval pattern 2026-09-22, owner approval required before posting (external-channel gate in CLAUDE.md)----# A digital human in a browser tab+# A digital human in a browser tab: streaming Audio2Face-3D onto whatever rig the visitor brought-I run [three.ws](https://three.ws). You type a sentence, a rigged 3D character appears, and then it talks to you. I have written here twice before: once about [Nemotron in front of the text-to-3D generator](https://forums.developer.nvidia.com/t/how-nemotron-made-three-ws-text-to-3d-pipeline-usable/376445), once about [NIM driving a 100-language i18n pipeline](https://forums.developer.nvidia.com/t/how-three-ws-translates-a-web-app-into-100-languages-with-nvidia-nim-an-llm-powered-i18n-pipeline/377379). This one is about the face.+I run three.ws, an open-source platform where you type a sentence, a rigged 3D character appears, and then it talks to you. My last post here was about [what a retired NIM model does to a fallback chain](https://forums.developer.nvidia.com/t/nvidia-nim-model-retirements-what-a-410-gone-does-to-a-fallback-chain-and-how-we-survive-it-now/383950). This one is the opposite kind of post: the part of the stack that has worked so well we built the product's face on it.Audio2Face-3D is usually shown inside Unreal or Omniverse Kit, driving a character somebody built on purpose. We run it in a browser tab, with no install and no engine, driving a character the visitor generated ninety seconds earlier from a text prompt. That second constraint turned out to be the whole problem: the avatar's rig is not ours, and we do not get to pick its blendshape convention.-Live, no signup: [three.ws/demos/audio2face](https://three.ws/demos/audio2face). Everything below is checkable in the [open-source repository](https://github.com/nirholas/three.ws).+Live, no signup: [three.ws/demos/audio2face](https://three.ws/demos/audio2face). The repository is nirholas/three.ws on GitHub, and every file named below is in it.## The loopThree NVIDIA-hosted models on one face, all reached with one `nvapi-` key:-- **Riva ASR** takes the visitor's microphone audio and returns text ([api/_lib/asr-nvidia.js](https://github.com/nirholas/three.ws/blob/main/api/_lib/asr-nvidia.js)).-- **Nemotron**, through the NIM APIs, writes the reply, with NemoGuard content safety on any publicly reachable agent.-- **Magpie TTS multilingual** speaks it, nine languages, personas held constant across languages so a character keeps its voice when it switches ([api/_lib/tts-nvidia.js](https://github.com/nirholas/three.ws/blob/main/api/_lib/tts-nvidia.js)).-- **Audio2Face-3D** turns that exact audio into a per-frame blendshape track ([api/_lib/a2f-nvidia.js](https://github.com/nirholas/three.ws/blob/main/api/_lib/a2f-nvidia.js)).+- **Riva ASR** takes the visitor's microphone audio and returns text (api/_lib/asr-nvidia.js).+- **Nemotron 3**, through the NIM chat endpoint, writes the reply.+- **Magpie TTS multilingual** speaks it: nine languages, and personas held constant across languages so a character keeps its voice when it switches (api/_lib/tts-nvidia.js).+- **Audio2Face-3D** turns that exact audio into a per-frame blendshape track (api/_lib/a2f-nvidia.js).-All four are NVCF gRPC functions on `grpc.nvcf.nvidia.com:443`, selected by a `function-id` metadata entry with the key as a bearer `authorization` entry. Magpie has no REST surface at all, so gRPC is not a preference here, it is the only door.+ASR, TTS and A2F are NVCF gRPC functions on `grpc.nvcf.nvidia.com:443`, selected by a `function-id` metadata entry with the key as a bearer `authorization` entry. Magpie has no REST surface at all, so gRPC is not a preference here, it is the only door.-One wrinkle worth passing on if you are calling NVCF from a serverless runtime: our `api/` handlers are esbuild-bundled in place, so a `.proto` file does not reliably exist on the filesystem at call time. We vendor the ACE and Riva protos and load them from a generated JSON descriptor instead. Three separate descriptors, because the ASR, TTS, and A2F services do not share one.+One wrinkle worth passing on if you are calling NVCF from a serverless runtime: our API handlers are bundled in place, so a `.proto` file does not reliably exist on the filesystem at call time. We vendor the ACE and Riva protos and load them from a generated JSON descriptor instead. Three separate descriptors, because the ASR, TTS, and A2F services do not share one.## The audio contract, and the resampling trap@@ -35,28 +37,28 @@ A2F-3D is trained on 16 kHz mono 16-bit PCM. Magpie emits 44.1 kHz. So the serveThe trap is what you play back. It is tempting to hand the browser the 16 kHz copy you already made, because it is right there. Do not: the visitor then hears a downsampled voice for no reason. A2F returns time codes in **seconds from clip start**, independent of the sample rate it consumed, so the browser plays the original 44.1 kHz Magpie audio and samples the track by the audio element's `currentTime`. The lips track the real voice, not the copy you fed the model.-The service is a bidirectional stream (`nvidia_ace.services.a2f_controller.v1.A2FControllerService/ProcessAudioStream`): client sends an audio header, PCM chunks, and an end-of-audio marker; server sends an animation header carrying the ordered blendshape names, then animation-data frames. 30 inferences per second of audio, so 30 fps out.+The service is a bidirectional stream (`nvidia_ace.services.a2f_controller.v1.A2FControllerService/ProcessAudioStream`): the client sends an audio header, PCM chunks, and an end-of-audio marker; the server sends an animation header carrying the ordered blendshape names, then animation-data frames. 30 inferences per second of audio, so 30 fps out.-## Measured, against production, today+## Measured against production, 2026-09-22-One line of speech, 4.64 seconds of audio, from the public endpoint:+One line of dialogue, 5.5 seconds of Magpie audio, three consecutive calls each to the public endpoint:| Path | Round trip ||---|---|-| Audio in, animation out (`POST /api/a2f` with a WAV) | 1.29 s, 1.31 s, 2.26 s |-| Text in, speech and animation out (Magpie then A2F, one call) | 1.88 s, 3.35 s, 3.41 s |+| Audio in, animation out (`POST /api/a2f` with a WAV) | 1.29 s, 2.09 s, 2.19 s |+| Text in, speech and animation out (Magpie then A2F, one call) | 1.94 s, 2.90 s, 3.17 s |-So A2F-3D animates a 4.6 second clip in roughly a third of its duration, and Magpie adds around 0.6 to 1.1 seconds on top. The returned track is 140 frames at 30 fps carrying **55** blendshape weights: the ARKit-52 set plus three tongue shapes. Sixty seconds of audio is our hard ceiling per request, which is about 1.9 MB of PCM and 1800 frames, generous for a line of avatar dialogue and small enough that one request cannot pin a serverless instance.+So A2F-3D animates a 5.5 second clip in well under half its duration, and Magpie adds roughly 0.7 to 1.0 seconds in front of it. The returned track is 166 frames at 30 fps carrying **55** blendshape weights: the ARKit-52 set plus three tongue shapes. Sixty seconds of audio is our hard ceiling per request, about 1.9 MB of PCM and 1800 frames, generous for a line of avatar dialogue and small enough that one request cannot pin a serverless instance.## The part nobody warns you about: the visitor's rigOur avatars come from a text prompt, a photo, or a GLB the visitor uploaded from somewhere else entirely. That means the face we have to drive might expose ARKit-52 morphs, or VRoid's vowel shapes, or Meta's `viseme_*` set, or nothing usable at all. A2F speaks ARKit. Half of the web does not.-We had already solved the same shape of problem for skeletons: [glb-canonicalize.js](https://github.com/nirholas/three.ws/blob/main/src/glb-canonicalize.js) maps Mixamo, Avaturn, Unreal, VRM, Daz, MakeHuman and plain `.L` bone conventions onto one canonical rig so the animation library retargets onto anything humanoid. The face version lives in [src/voice/arkit-blendshapes.js](https://github.com/nirholas/three.ws/blob/main/src/voice/arkit-blendshapes.js) and [src/voice/a2f-player.js](https://github.com/nirholas/three.ws/blob/main/src/voice/a2f-player.js), and it runs in two modes per mesh:+We had already solved the same shape of problem for skeletons: src/glb-canonicalize.js maps Mixamo, Avaturn, Unreal, VRM, Daz, MakeHuman and plain `.L` bone conventions onto one canonical rig so the animation library retargets onto anything humanoid. The face version lives in src/voice/arkit-blendshapes.js and src/voice/a2f-player.js, and it runs in two modes per mesh:-**Direct.** The morph name canonicalizes to an ARKit shape (`jawOpen`, `JawOpen`, `jaw_open`, and the RPM, Avaturn, and MetaHuman spellings all land on the same canonical name). The A2F weight is written straight to it.+**Direct.** The morph name canonicalizes to an ARKit shape (`jawOpen`, `JawOpen`, `jaw_open`, and the Ready Player Me, Avaturn, and MetaHuman spellings all land on the same canonical name). The A2F weight is written straight to it.-**Derived.** The mesh only has monolithic expressions: VRM's `Aa`, `Ih`, `Ou`, `Ee`, `Oh`, or Oculus's `viseme_PP`, `viseme_FF`, `viseme_CH`. We hold a forward map from each of those to its ARKit components (20 VRM entries, 15 Oculus entries), and at playback we run that map **backwards**: an expression's activation is the normalized sum of the ARKit weights it is made of, taken from the A2F frame. A rig that only knows five vowels still lip-syncs, because the vowel is reconstructed out of the ARKit frame rather than looked up from a phoneme we never received.+**Derived.** The mesh only has monolithic expressions: VRM's `Aa`, `Ih`, `Ou`, `Ee`, `Oh`, or Oculus's `viseme_PP`, `viseme_FF`, `viseme_CH`. We hold a forward map from each of those to its ARKit components (`VRM_TO_ARKIT`, `OCULUS_TO_ARKIT`), and at playback we run that map **backwards**: an expression's activation is the normalized sum of the ARKit weights it is made of, taken from the A2F frame. A rig that only knows five vowels still lip-syncs, because the vowel is reconstructed out of the ARKit frame rather than looked up from a phoneme we never received.Two details that only show up once you run this on real uploads:@@ -75,6 +77,11 @@ And there is no allowlist. A rig we cannot map is reported as "no coverage" and## Why the browser mattered-The reason to do this in WebGL instead of an engine is distribution. Our whole embed surface is one script tag: `<agent-3d>` on any page, and the avatar on that page listens with Riva, thinks with Nemotron, speaks with Magpie, and moves its face from Audio2Face-3D. No plugin, no download, no runtime for the site owner to install. The bodies themselves come off our own NVIDIA GPU fleet on Cloud Run, mostly L4s with one RTX PRO 6000 Blackwell for the heavy image-to-3D lane, which I wrote about separately.+The reason to do this in WebGL instead of an engine is distribution. Our whole embed surface is one script tag: `<agent-3d>` on any page, and the avatar on that page listens with Riva, thinks with Nemotron, speaks with Magpie, and moves its face from Audio2Face-3D. No plugin, no download, no runtime for the site owner to install. The bodies themselves come off our own NVIDIA GPU fleet on Cloud Run, mostly L4s with one RTX PRO 6000 Blackwell for the heavy image-to-3D lane.-Happy to go deeper on any of it, particularly the derived-blendshape path, which I have not seen written up anywhere and which took the longest to get right. three.ws is an NVIDIA Inception member.+## Two things I would ask of the ACE team++1. **Publish the hosted A2F function ids somewhere stable**, with a deprecation date when one is retiring. Ours died the way the model ids in my last post died: silently, discovered by a 502. The `nvcf/functions` list is the right tool, but a documented page beats a script.+2. **A mode that returns the ARKit track for a text input directly**, so a client that already uses Magpie does not have to round-trip the audio through its own server to resample it. We do the resampling for our users today; it would be one fewer thing every ACE-on-the-web integrator has to get right.++Happy to go deeper on any of it, particularly the derived-blendshape path, which I have not seen written up anywhere and which took the longest to get right. If you are driving A2F-3D onto rigs you did not author, I would like to compare mapping tables.diff --git a/docs/nvidia-visibility-map.md b/docs/nvidia-visibility-map.mdindex cc4ee178e..68b22841b 100644--- a/docs/nvidia-visibility-map.md+++ b/docs/nvidia-visibility-map.md@@ -71,7 +71,7 @@ Everything here can be done today. Nothing waits on NVIDIA, a build, or a date.| 3 | **Portal product record correction** | `programs.nvidia.com/phoenix/application` | Gates the Showcase AND the Apps Catalog. Already specified in [nvidia-apps-catalog-listing.md](./nvidia-apps-catalog-listing.md) step 1. Nothing else in this doc lands well while the record says "CUDA consumer". || 4 | **Inception Startup Showcase** | Existing `[email protected]` thread | Requested 2026-09-04 and not directly answered. After #3, send only the focused routing question drafted in [nvidia-apps-catalog-request.md](./nvidia-apps-catalog-request.md). || 5 | **AI Podcast guest nomination** | Embedded form at [ai-podcast.nvidia.com](https://ai-podcast.nvidia.com) | Free, one form, and a founder-shaped story. **Answers drafted below: [AI Podcast nomination](#ai-podcast-nomination-paste-ready).** Re-verified 2026-09-04: the form is live and self-nomination is explicitly invited. |-| 6 | **NVIDIA Developer Forums, post 3** | [forums.developer.nvidia.com](https://forums.developer.nvidia.com) | We are 2 for 2 on posts that landed. This is the one channel with a proven track record, and it is self-serve. **Drafted 2026-09-02: [nvidia-forum-browser-digital-human.md](./nvidia-forum-browser-digital-human.md).** Owner-gated on the external-channel rule, then paste and post. See [Technical publishing](#technical-publishing-the-proven-channel). |+| 6 | **NVIDIA Developer Forums, post 4** | [forums.developer.nvidia.com](https://forums.developer.nvidia.com) | We are 3 for 3 on posts that landed. This is the one channel with a proven track record, and it is self-serve. **Revised to the approval pattern 2026-09-22: [nvidia-forum-browser-digital-human.md](./nvidia-forum-browser-digital-human.md)**, the positive follow-up to the retirement post. Owner-gated on the external-channel rule, then paste and post. See [Technical publishing](#technical-publishing-the-proven-channel). || 7 | **NVIDIA Developer Discord** | [discord.com/invite/nvidiadeveloper](https://discord.com/invite/nvidiadeveloper) | NVIDIA engineers and product staff are in there. The browser Audio2Face demo is a conversation starter with the actual ACE team, not a marketing ask. || 8 | **Fix `docs/listings.md`** | Ours | The "Startup & Credit Programs" table lists Quicknode and Google Cloud and has **no NVIDIA row at all**, which is the one canonical place a reader looks for our program memberships. Fixed in this change. || 9 | **"Maximize Your Membership" benefits webinar** | Region registration from the Inception member mailer (Americas / EMEA / APAC) | Announced to members 2026-09-02. NVIDIA's own walkthrough of requesting and managing benefits through the portal, which is the exact mechanism gating items 2, 3 and 4 on this list. The batched email went out 2026-09-04, so use the Q&A to chase asks 2 and 3 live rather than waiting on the inbox. |@@ -276,9 +276,11 @@ resampling contract and why playback must use the original audio, the derivedblendshape path that lets a VRM or Oculus-viseme rig lip-sync from an ARKit-52track, the double-stacking and 30-to-60 fps interpolation bugs that only appearon real uploads, and the function-id rotation that took the lane down once.-Latencies in it were measured against the live endpoint the day it was written,-not estimated. It is owner-gated under the external-channel rule: paste and post-when approved.+It was revised on 2026-09-22 to the approval pattern above: two links (the live+demo and our retirement topic), plain-text repo paths, the NIM > Models category,+latencies re-measured against production that day, and the NemoGuard claim+removed (it screens outbound publishing, not chat). It is owner-gated under the+external-channel rule: paste and post when approved.## Drafted, not yet posted