Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

23.0 KB
Raw Blame

NVIDIA models on three.ws — the free inference layer

If you generate a 3D model on /forge, chat with an agent, or hear an avatar speak, there is a good chance an NVIDIA-hosted model did the work. This page is for developers and operators who want to know exactly which model sits behind each feature.

three.ws runs a large share of its AI on NVIDIA's free hosted models. One key — NVIDIA_API_KEY (an nvapi-… token from build.nvidia.com) — unlocks every model on this page. There is no per-model billing, no per-seat cost, and no SLA: it is a rate-limited free tier, which is exactly why the platform treats it as a free-first lane and always keeps a fallback behind it.

This document is the canonical map of which NVIDIA-hosted model does what, where it's wired, and why. Every model and endpoint below is in production source — nothing here is aspirational.

As of July 2026, three.ws is also a member of the NVIDIA Inception program (details). The free hosted lane on this page is unchanged by that: membership adds GPU credits, hardware access, and engineering support on top of it, which is how the platform scales past free-tier rate limits without giving up the free-first design.


How the platform talks to NVIDIA

NVIDIA exposes its catalog over a few distinct surfaces. three.ws uses four:

Surface Base URL Shape What runs here
NIM (OpenAI-compatible) https://integrate.api.nvidia.com/v1 chat/completions, embeddings LLM chat, vision (VLM), embeddings, content-safety
GenAI invoke https://ai.api.nvidia.com/v1/genai/… Async (202 + poll) or sync TRELLIS text→3D, FLUX text→image
Retrieval https://ai.api.nvidia.com/v1/retrieval/nvidia/reranking Sync rerank Cross-encoder reranking
NVCF gRPC grpc.nvcf.nvidia.com:443 Riva gRPC Magpie text-to-speech
NVCF status https://api.nvcf.nvidia.com/v2/nvcf/pexec/status/{id} Poll Async job status (TRELLIS)

Because it is one key for everything, a deployment either has the whole NVIDIA layer or none of it — every consumer below degrades gracefully when NVIDIA_API_KEY is absent.


The catalog at a glance

Capability NVIDIA model(s) Wired in Free?
Text → 3D microsoft/trellis api/_providers/nvidia.js, api/_lib/forge-tiers.js ✅
Text → image black-forest-labs/flux.1-dev api/_mcp3d/text-to-image.js ✅
LLM (default lane) nvidia/nemotron-3-super-120b-a12b, then nvidia/nemotron-3-ultra-550b-a55b (step-down rung: nvidia/nemotron-3.5-lightning-30b-a3b) api/_lib/llm.js, api/_lib/chat-models.js ✅
LLM (model garden) Nemotron 3 Super 120B / Ultra 550B / 3.5 Lightning 30B, DeepSeek V4 Pro, Kimi K2.6, MiniMax M2.7 api/brain/chat.js ✅
Vision / VLM meta/llama-3.2-11b-vision-instruct, then free OpenRouter :free VLM rungs api/_lib/vision.js ✅
Embeddings nvidia/nemotron-3-embed-1b api/_lib/embeddings.js (and api/agents/_id/embed.js, which delegates to it) ✅
Reranking nvidia/rerank-qa-mistral-4b (pin verified dead, stage off by default) api/_lib/rerank.js ✅
Content safety nvidia/llama-3.1-nemoguard-8b-content-safety, meta/llama-guard-4-12b api/_lib/publish-safety.js ✅
Text-to-speech magpie-tts-multilingual (Riva) api/_lib/tts-nvidia.js ✅

1. Text → 3D — Microsoft TRELLIS

Model: microsoft/trellis · Endpoint: ai.api.nvidia.com/v1/genai/microsoft/trellis → poll api.nvcf.nvidia.com/v2/nvcf/pexec/status/{id} Source: api/_providers/nvidia.js, registered as the nvidia backend in api/_lib/forge-tiers.js.

This is the headline free model. Microsoft TRELLIS hosted on NVIDIA NVCF gives /forge a zero-vendor-cost text→3D lane that returns a textured GLB. It is the default draft/standard engine for prompt generations, per the platform's free-first policy.

Where it's used:

  • The /forge web app — draft and standard tiers default here (FREE_DEFAULT_FOR_TIERS).
  • The free forge_free MCP tool — text prompt → downloadable GLB + viewer link, no payment, no wallet, no key.
  • The IBM × three.ws x402 demo — the free generator next to the paid USDC Forge.
  • The auto-generation gallery — a fresh community avatar every minute.

How it works:

  • Async by default. Submit returns 202 + NVCF-REQID; the forge polls the NVCF status endpoint until the GLB is ready (or the job completes synchronously within a 30 s window).
  • Sampling steps are pinned to the proven budget. TRELLIS accepts 10 to 50 steps, but the hosted preview only returns inside the gateway's synchronous window at the low end, so the free lane runs 15/15 (ss_sampling_steps / slat_sampling_steps) for both draft and standard tiers. The 40/40 budget is reserved for a self-hosted TRELLIS NIM; the high tier itself defaults to the self-hosted Hunyuan3D engine, not this lane.
  • Prompts are clamped to 77 characters (TRELLIS truncates server-side) and get a , studio lighting suffix unless the caller already supplied lighting/color cues — without it TRELLIS defaults to dark, gritty output.
  • Output GLBs arrive in several shapes over time (inline base64, bare string, CDN URL, numeric-keyed object, raw bytes); the extractor normalizes all of them, then persists the bytes to R2 so three.ws owns a durable public URL.

Key constraint: text only. NVIDIA's hosted preview rejects every user-image input form (verified live 2026-06-11). So photo to 3D never routes here: it falls to the free reconstruct chain in FREE_FALLBACK_FOR_PATH (the self-hosted TRELLIS GPU worker first, then the self-hosted Hunyuan3D worker, then the free Hugging Face Spaces lane). A self-deployed TRELLIS NIM accepts real images; this is a hosted-preview limitation, not a model one.


2. Text → image: FLUX.1-dev

Model: black-forest-labs/flux.1-dev · Endpoint: ai.api.nvidia.com/v1/genai/black-forest-labs/flux.1-dev Source: api/_mcp3d/text-to-image.js.

FLUX.1-dev is the free NVIDIA text-to-image lane (the hosted FLUX.1-schnell preview was retired, so the lane moved to the dev checkpoint at 20 steps, CFG 3.5). It is a synchronous invoke: the image returns inline as base64, no poll.

Where it's used:

  • The reference-image step of the image-intermediate 3D path. When a prompt is reconstructed to a mesh, FLUX paints the reference view first, which the free reconstruction lanes (TRELLIS / Hunyuan3D) then turn into geometry. A photo-quality reference reconstructs into a far better mesh than a busy scene, so the lane steers FLUX toward a clean, centered subject.
  • General text→image wherever the platform needs a synthesized image.

Ladder order. When GOOGLE_CLOUD_PROJECT is set, Vertex Imagen leads by default because it outdraws distilled FLUX for photoreal reference images and spends the GCP credit pool the platform is funded to burn (VERTEX_IMAGEN_FIRST=0 restores the NIM-first order). After that: NVIDIA FLUX.1-dev (free) → the Hugging Face router (HF_TOKEN; fal-ai, then nscale, both serving FLUX.1-schnell in 2-5 s) → the Livepeer federation lane (behind LIVEPEER_FEDERATION_ENABLED) → Pollinations (keyless, so a fallback always exists past every keyed rung) → Replicate flux-schnell ($0.003/image, the paid backstop). The whole ladder shares one budget, TEXT_TO_IMAGE_BUDGET_MS (default 60 s): each lane gets a bounded slice of what remains so a stalled provider cannot hold a Forge submit for minutes, and exhausting the budget returns a retryable rate_limited error (retryAfter: 15) rather than a hung request.


3. LLM chat & reasoning

NVIDIA NIM hosts 100+ open-weight chat models behind the one key, all OpenAI-compatible at integrate.api.nvidia.com/v1/chat/completions. three.ws uses them two ways.

3a. The default production lane

Models: nvidia/nemotron-3-super-120b-a12b, then nvidia/nemotron-3-ultra-550b-a55b, with nvidia/nemotron-3.5-lightning-30b-a3b as the compact step-down rung Source: api/_lib/llm.js, api/_lib/chat-models.js.

The platform's general LLM helper runs a free-first ladder: Groq → Cerebras → OpenRouter → NVIDIA NIM, followed by further keyless free rungs (OVH, Gemini, Pollinations) and only at the very end a paid backstop (Anthropic/OpenAI). Every free rung was re-pointed on 2026-08-27, and every NVIDIA-routed id again on 2026-09-10 after a sweep of GET /v1/models on the live account found most of them answering 410 Gone. A dead id is worse than a missing one: a fallback chain treats a 410 like a rate limit and moves on, so nothing logged an outage while each dead entry burned one of the chain's few retry slots. The rungs the providers still serve: Groq now runs qwen/qwen3.8-27b (instant tier openai/gpt-oss-20b), Cerebras keeps llama-3.3-70b, and the NVIDIA rung runs Nemotron 3 Super 120B, an NVIDIA MoE on an independent provider (the opt-in compact rung moved off nemotron-3-nano-30b-a3b, which NIM stopped serving, to nemotron-3.5-lightning-30b-a3b), so an outage on the other free lanes still answers here. Both NVIDIA rungs are called with thinking disabled (chat_template_kwargs: { enable_thinking: false }) so a chat turn answers instead of reasoning out loud. Both are tool/function-calling capable, so they are eligible for tool-required requests.

Since 2026-10-08 the shared chain runs all three Nemotron models on the one key, as separate rungs: Super 120B, then Ultra 550B right behind it, then Lightning 30B in the step-down group near the tail. Each NIM model is its own function with its own queue and its own retirement date, and NVIDIA sizes the free rate limit by model, so one model throttling, queueing, or answering 410 no longer takes the NVIDIA lane down with it. The third-party models the same key lists (DeepSeek V4.1 Flash, Kimi K3, GLM 5.3, Gemma 4) were probed the same day and each queued past 60 seconds on the free tier, so none of them is a chain rung. The same Nemotron Ultra and Super models are also reachable keylessly through the Kilo Code gateway, which the chain uses as a separate rung on Kilo's own capacity. /api/cron/free-model-audit checks every pinned NIM id against the live catalog every 6 hours and pages ops when one is retired. Full chain: free-llm-providers.md.

This lane powers the platform's built-in AI surfaces — chat, embedded site widgets, the tutor, the fact-checker, persona tools, agent-to-agent talk, the transaction explainer — all of which lead with the free providers and only fall through to a paid model if every free lane fails.

3b. The Brain model garden

Source: api/brain/chat.js — the Brain workbench lets users pick a model. The NVIDIA-hosted options, all unlocked by the single key:

Brain label Model id Tier What it's for
Nemotron 3 Super 120B nvidia/nemotron-3-super-120b-a12b flagship NVIDIA's flagship Nemotron MoE, strong agentic reasoning
Llama-Nemotron Super 49B nvidia/nemotron-3-ultra-550b-a55b reasoning Deepest Nemotron reasoning rung: math, code, planning
Nemotron Nano 9B nvidia/nemotron-3.5-lightning-30b-a3b balanced Compact Nemotron with built-in reasoning, strong quality per token
DeepSeek V4 Pro deepseek-ai/deepseek-v4-pro reasoning Deep reasoning, hosted on NIM
Kimi K2.6 moonshotai/kimi-k2.6 flagship Moonshot long-context agentic model
Llama 4 Maverick nvidia/nemotron-3-super-120b-a12b balanced Meta's Maverick left the account's catalog; the slot now serves Nemotron 3 Super
MiniMax M2.7 minimaxai/minimax-m2.7 balanced General reasoning and chat

For anonymous (signed-out) callers, only the genuinely free tiers (the OpenRouter open-weight default plus these NVIDIA NIM models) are selectable. Each shows "unavailable" until the key is set. The three picker labels above are the Brain's own UI strings and were not renamed when the ids under them were re-pinned on 2026-09-10, so the id column is the one to trust. The routing catalog in chat-models.js lists nvidia/nemotron-3-super-120b-a12b, nvidia/nemotron-3.5-lightning-30b-a3b, and nvidia/nemotron-3-ultra-550b-a55b as the tool-capable NVIDIA models, and the Brain's own fallback chain appends nvidia/nemotron-3.5-lightning-30b-a3b on NIM behind the OpenRouter rungs when a chosen route is dead.

Where the line is: Nemotron and the nvidia/…-prefixed models are NVIDIA's own. The others in this table (DeepSeek, Kimi, Llama 4, MiniMax) are third-party open weights that NVIDIA hosts and serves free on NIM — so they ride the same key, but the model itself isn't NVIDIA's. The point of NIM is exactly this: one free key, a whole model garden.


4. Vision / VLM — image understanding

Model on NIM: meta/llama-3.2-11b-vision-instruct Source: api/_lib/vision.js.

One free NIM vision lane on the OpenAI-compatible chat host. nvidia/nemotron-nano-12b-v2-vl used to lead it for its small image-token footprint; NVIDIA retired that model on 2026-08-26 and the host now answers every request for it with a hard 410, so it was removed on 2026-09-08 rather than swapped, because probing the catalog for a replacement free NIM VLM needs a live key. Behind the NIM rung sit free OpenRouter :free VLM routes (Gemma 4, Ling 3.0 Flash VL, Nemotron Nano Omni and others), then Cloudflare Workers AI, Vertex Gemini, and the paid OpenAI backstop; the OpenRouter rung was added on 2026-09-09 after NVIDIA, OpenAI, and both Google routes failed in the same request and took every image-judging feature down at once. Only :free ids are listed there, because the spend ledger prices OpenRouter by that exact suffix. Images pass as an http(s) URL (the model server fetches it) with SSRF validation on the URL before it leaves the box.

Where it's used:

  • Forge photo pre-check — before a generation slot is spent, the uploaded photo is screened: a screenshot of text, a cluttered subject-less scene, or a too-dark image gets a heads-up and a fix (with a one-click "Generate anyway").
  • Fact Checker image evidence — reads a picture alongside a claim, transcribes any text in it, and weighs it in the verdict.
  • Avatar gallery alt-text — writes real, descriptive alt text from each avatar's thumbnail for screen-reader users.

All three fail safe: if the vision lane is unavailable, the feature quietly switches off — it never blocks or breaks the primary flow.

The chain is budgeted and remembers what hurt. Each attempt takes whatever remains on the caller's deadline minus a reserve for the next rungs (laneAttemptTimeout, never below 3.5 s, reserving for at most two further lanes), so the lane in hand spends the budget nobody else needs yet while every remaining rung keeps its floor. Dividing the deadline evenly instead starved the lane most likely to answer once the chain grew past a handful of rungs, and the pre-chain inline image fetch is capped at a quarter of the deadline and 8 s. A lane that fails is cooled for 45 s through the shared provider-health ledger (api/_lib/provider-health.js) and moved behind the healthy rungs on the next call rather than re-picked; a 429 cools every sibling lane on that host and skips them for the rest of the call, and a 401/403/402 earns the longer auth cooldown. A success clears the lane's cooldown.


5. Embeddings — semantic retrieval

Primary: nvidia/nemotron-3-embed-1b (2048-dim) · Endpoint: integrate.api.nvidia.com/v1/embeddings Source: api/_lib/embeddings.js (tag nvidia/nemotron-3-embed-1b@2048).

nv-embedqa-e5-v5 held this slot until NVIDIA ended its life on 2026-08-25. Its tag stays in the registry as NIM_EMBED_TAG_RETIRED, resolvable forever so vectors written in that space still read back, but reporting unconfigured so it is never picked for new work. Vectors are stored as jsonb, so the dimension change needed no migration.

The default embedder for new vectors: free with the one key, 2048 dimensions, hard-capped at 512 input tokens (longer inputs are rejected upstream, so callers chunk to fit). Vectors are tagged with model@dimension so a later model swap can't silently mix incompatible spaces. Powers agent memory and knowledge-widget retrieval; the paid embedding provider is demoted to backup behind it. At ingest, embedPassagesAny() walks every configured embedder in free-first order (NIM → Vertex → OpenAI, the caller's preferred tag first), cooling a lane that fails for 45 s (longer on an auth fault) so the next ingest starts on a healthy rung, and returns the tag it actually used so the document set is stamped with the space its vectors live in. Search-side, the Agent Galaxy falls back to lexical ranking when the embedder cannot answer, because a query vector from another model cannot be compared against vectors stored in a different space.

Also: the agent-embed endpoint api/agents/_id/embed.js does not carry a provider list of its own. It delegates to the registry above, so POST /api/agents/:id/embed serves this same model and returns the tag alongside the vector. It previously called baai/bge-m3 directly; NVIDIA stopped serving that model on the hosted endpoint (500 on every request), which is exactly the failure mode a single registry prevents.


6. Reranking — sharpening retrieval

Model: nvidia/rerank-qa-mistral-4b · Endpoint: ai.api.nvidia.com/v1/retrieval/nvidia/reranking Source: api/_lib/rerank.js.

Cosine-over-embeddings recall is cheap but coarse. This cross-encoder reranker re-scores the top passages so the most relevant context leads. It is opt-in (KNOWLEDGE_RERANK_ENABLED=1 plus the NVIDIA key) and strictly fail-open: any rerank error keeps the original cosine ordering. The 2026-09-10 catalog sweep found this model dead on the account and it is deliberately left pinned: the stage is off in production and fails open by contract, so a pin verified dead beats an unverified guess. Turning the stage on means re-pinning it against a live catalog first. Reranking may improve retrieval but may never break it. Used to refine knowledge-widget answers.


7. Content safety — NemoGuard

Primary: nvidia/llama-3.1-nemoguard-8b-content-safety · Drop-in alt: meta/llama-guard-4-12b Endpoint: integrate.api.nvidia.com/v1/chat/completions · Source: api/_lib/publish-safety.js.

A free content-safety classifier for what the platform publishes outward. Its only caller is the Sketchfab showcase cron, which uploads visitor-generated models to our own official account on a third party's platform. NemoGuard classifies the source prompt and returns a JSON verdict plus named risk categories (harm, self-harm, weapons, sexual content, …); the parser also accepts Llama Guard's unsafe\nS# text form, which is why the two are interchangeable. Median ~340 ms on the free tier.

Scope and posture: it screens OUTBOUND publishing only. No three.ws chat surface filters what a user may ask (owner directive 2026-08-07): whatever safety judgment the serving model makes is the only one those routes apply. It is also a content-safety classifier, not a jailbreak / prompt-injection detector, and it is fail-open: anything it can't parse returns "not flagged", so a classifier outage never stops a publish. Only a clean parsed "unsafe" verdict blocks an upload.


8. Text-to-speech — Magpie (Riva)

Model: magpie-tts-multilingual · Transport: Riva gRPC at grpc.nvcf.nvidia.com:443 Source: api/_lib/tts-nvidia.js (mirrored in packages/avatar-agent-mcp/src/lib/tts-nvidia.js).

The free NVIDIA TTS lane: Magpie multilingual on Riva, selected by an NVCF function-id, speaking over the standard Riva gRPC synthesis contract (protos shipped in api/_lib/riva-protos/ and loaded from a generated descriptor, so there's no .proto build step). Drives avatar speech with multilingual voices. Configured by the presence of NVIDIA_API_KEY; returns synthesized audio bytes, with a clear error if the lane returns empty audio.


Design principles across every lane

  1. Free-first, always. NVIDIA NIM leads its category (or sits in a free trio with Groq/OpenRouter) before any paid model is touched. Cost to the platform is $0.
  2. One key, whole layer. NVIDIA_API_KEY is the only credential. Present → the layer is live; absent → every consumer degrades to its next lane or switches off cleanly.
  3. Fail-open / fail-safe. Safety, rerank, and vision never break the primary flow — a NVIDIA outage downgrades quietly, it doesn't error the user.
  4. Independent fallbacks. Where reliability matters, the fallback is a different model family (vision) or a different provider (chat), not a retry of the same thing.
  5. It's a free tier, not an SLA. Rate-limited, no uptime guarantee — great for the default path precisely because there's always something behind it.

Environment

Variable Unlocks
NVIDIA_API_KEY Every model on this page (nvapi-… from build.nvidia.com)
KNOWLEDGE_RERANK_ENABLED=1 Turns on the rerank stage (§6)
FORGE_PREFER_FREE Free-first reconstruct ordering (default on)
TEXT_TO_IMAGE_BUDGET_MS Shared deadline for the whole text-to-image ladder (§2), default 60000
VERTEX_IMAGEN_FIRST=0 Puts the free NIM FLUX lane ahead of Vertex Imagen again (§2)
HF_TOKEN Unlocks the Hugging Face router rungs (fal-ai, nscale) in the text-to-image ladder (§2)

Related