Turn one illustration into an animated 2D mesh avatar with a coding agent and a local editor
Reference
Details behind the workflow in the README.
Projects
Open the local URL printed by Vite. Choose Open project and select a project from the local
list, or use Browse for a project folder… to select its folder. A project contains
rig.json, source.png and built/. Load rig.json only… replaces the rig without loading images.
The header shows the project name with buttons to copy its full path and open its folder.
Save rig and Ctrl/Cmd+S save a listed project directly to rig.json, keeping the previous
file as rig.json.bak. The sample is read-only, and browser-picked folders use a JSON download.
Local project access is available only through the development server on 127.0.0.1; a static
build keeps the folder picker and download workflow. Images stay on this machine.
Recent keeps up to ten opened projects in this browser and can reopen the last project
on start. Remove individual entries, clear the history, or disable automatic reopening in
the project menu. Supported browsers retain a directory handle for picked folders and may
ask for permission again; other browsers show Browse again. Image data is never stored
in this history.
See the avatar file and folder format for the file layout, manifest, layer and sprite metadata, compatibility rules, and whole-folder validation.
Build a project by hand
The layer builder requires Python 3.10+ and uv.
It runs locally with NumPy, Pillow and OpenCV; these are build tools, not browser dependencies.
The other Python tools declare dependencies inline for bare uv run. The validator and
pose renderer use Node.js 22.17+; install Node dependencies with npm ci and Chromium with
npx playwright install chromium if needed. Rig field descriptions
explain coordinates and placement.
Create a project folder and put your illustration in
source.png. A PNG with a transparent background works. Use a copy of your artwork when experimenting.Prepare
rig.draft.json. Usesamples/miko-qipao/rig.jsonas a schema example, setimage.widthandimage.heightto your image dimensions, and place the geometry in source-image pixels. Trace each eye'sopeninginside the lashes and its enclosingroiaround the lash and lid strokes. Place hair strandnodesfrom root to tip. Removehand,bunsandaccessoriesif absent; keep the required head, body, face, mouth, cheeks, mesh and view settings. The eye fieldsx0,x1,topandbotcan be omitted from the draft.From the repository root, build the layers:
uv run --with numpy --with pillow --with opencv-python-headless tools/build-layers.py /path/to/project --rig rig.draft.jsonThis writes
rig.jsonwith 24 sampled points for each eye's top and bottom curves, plusbuilt/base.png,built/hairmask.png, the foureye{i}_ball/lash/low/crease.pnglayers per eye, andbuilt/layers.json. Hand and accessory layers are generated only when present in the rig. The input draft remains unchanged. Existing generated files are replaced; optional drawn sprites are preserved. Without--rig, the input isrig.json.In the editor, choose Open project and select it from the list, or browse for the whole project folder, including
source.png,rig.jsonandbuilt/. Use Pose test to check blinking and face angles, and Sweep angles to check the full motion range.
The builder estimates eyelid skin and line colours near each eye's ROI, and hair colours near
the strand lines and buns. It follows connected colour regions within the head, buns and
strand areas, using nearby skin and body colours to exclude non-hair pixels. Bright hair
highlights or hair with several unrelated colours can leave gaps in the inferred mask.
It does not trace eyes or invent hidden artwork automatically. Incorrect
outlines can leave eye pixels behind during a blink or include skin in a moving layer. Inspect
the result and adjust the rig in the editor. For a writable project opened from the local
list, use Save and rebuild layers in the changed-outlines banner. It runs the local
builder and reloads the preview, keeping selection, zoom and undo history. A failed build
keeps the previous layers and offers a log. Browser-picked folders still require saving
the JSON, running the command without --rig, and reopening the folder. Keep the original draft separately
if you want to retain it. Drawn closed-eye and mouth variants are optional.
Accessory colour masks currently use the rig's red-dominance thresholds (color.redness and
color.minRed) inside each box. Hair and skin estimation assumes reasonably opaque pixels
around the traced regions; fully translucent artwork or unrelated colours inside an eye ROI
may need manual retouching. Inpainting approximates the artwork hidden behind hands and eyes.
Editing
- Drag a handle or select an item to edit its numeric fields.
- Double-click a polygon/polyline edge to insert a vertex; Alt-click a vertex to remove it.
- Pinch to zoom at the cursor; two-finger scrolling pans. A mouse wheel zooms by default; Mouse wheel can switch it to panning. Ctrl/Cmd-scroll always zooms.
- Drag empty space, Space-drag or middle-drag to pan. Use − / + to zoom, the percentage to return to 100% (one image pixel per screen pixel), Fit for the whole image, or Fit selected part for the current part. Double-clicking a part in the list also fits it. Zoom ranges from 10% to 1600%, while dots keep the same screen size.
- Ctrl/Cmd+plus, minus, 0 and 1 zoom in, zoom out, fit and reset to 100%. These shortcuts leave browser behavior alone when an input is focused.
- Undo / Redo restore edits. Keyboard shortcuts: Ctrl/Cmd+Z, Ctrl/Cmd+Shift+Z and Ctrl/Cmd+S.
- Save rig saves a listed project, or downloads the JSON for a browser-picked folder. Dropping a JSON file also loads it.
- Pause Idle motion for a stable pose comparison, or use Sweep angles to sweep angles.
- Switch between the Pose test and Lip sync tabs under the preview. Lip sync holds あ・い・う・え・お・ん, or plays kana text at 4–12 morae per second. Release / Stop return control to the pose sliders. Drawn mouth images are used when available; otherwise the mesh mouth animates. This check does not play audio.
Drawn variants shows eye and mouth image counts. Check the drawings you want and
copy the prepared message for Codex or Claude Code. Codex's message requests its
built-in image generation, local import and pose review, and the card states that images
will be sent to Codex image generation. Claude Code's message asks it to prepare masks
and prompts and explain where to save the drawings, then stop. The recipient is remembered. Changes in the local project's variants/ and built/sprites/ directories
reload in place, keeping unsaved outlines, selection, zoom and undo history. The sample's
request asks the agent to work on a copy under projects/, leaving the original intact.
If you prepare images yourself is collapsed by default. Open it to export local masks
and prompts with variant-requests.py, or to drop or choose full-size PNGs with the listed
filenames. The editor validates dimensions and pixels outside the mask before building
sprites; rejected imports keep the previous drawings and sprites. Manual export and import
need a writable project from the development server. The app keeps images local; the
Codex card discloses the image-generation upload before you copy its request; other
external services are excluded from that request.
Lighting
Lighting is optional and off by default. Open the collapsed Lighting section below the Pose test and Lip sync tabs, or near the bottom of the Live page, and choose Enable lighting. While the section is open, a light handle appears over the preview; drag it, or focus it and use the arrow keys (Shift moves further). The section title shows ON while lighting is enabled.
The shape used for shading is inferred from the single image: each layer's silhouette is rounded, the head is treated as an ellipsoid, and dark painted strokes add shallow relief. It is computed once per layer when lighting is first enabled. Painted shadows in the artwork are kept; the light deepens colours on the far side and keeps the painted colour on the lit side.
| Control | Effect |
|---|---|
| Light height | Low values light from the side; high values light from the front |
| Light spread | How far the light reaches before it fades |
| Intensity / Ambient light | Direct light and the even base light |
| Light color / Ambient color | Colours of the direct and ambient light |
| Strength | Blend between the original artwork and the lit result |
| Shading smoothness | Width of the transition between lit and shaded areas |
| Gloss / Rim light | Small highlights, and light along edges facing the light |
| Stroke relief | Relief taken from painted strokes such as hair lines |
| Shading | Soft or Cel (stepped, anime style) |
| Drop shadow | A soft silhouette shadow behind the avatar, opposite the light |
Reset lighting restores the defaults. Settings are presentation only: they are not saved
in rig.json, but each project remembers them in this browser. With lighting off, the avatar
renders exactly as without the feature.
Lighting presets
The Live page's Lighting section has eight preset slots, for example a normal light, a red light from below for a horror game, and a sunset colour. Save stores every current lighting setting (position, colours, shading and on or off) in that slot; saving again overwrites it. Click a saved slot, or press Shift+1–8 while the Live page is focused, to switch to it at once. The swatch shows the light colour and the ambient colour. Presets are kept per project in this browser, like the current lighting.
A preset does not change the expression, and an expression does not change the lighting, so both can be switched independently. To return to normal lighting, save it in a slot (for example 1) and switch to that slot. Switching to an empty slot changes nothing.
Other apps switch presets the same way as
expressions, with the same token: POST to
/__live/lighting (shown as Preset request URL under the presets) with a JSON body
such as {"preset": 2}. Numbers are 1–8, and "project" works as for expressions.
curl -X POST http://127.0.0.1:5173/__live/lighting \
-H "X-Studio-Token: <token>" -H "Content-Type: application/json" \
-d '{"preset": 2}'
Live and streaming
Two pages served by the development server (npm run dev):
- Live page (
/live.html?project=<name>, or Live in the editor header): camera and microphone controls, face tracking, calibration and the stream URL. - Stream view (
/stream.html): only the avatar, for OBS or other capture software.
Stream view URL
| Parameter | Values | Default |
|---|---|---|
project |
A project name from the local list, or sample-miko-qipao |
the sample |
avatar |
A folder or .mavatar URL on the same origin as the local development server; an explicit project takes precedence |
unset |
bg |
transparent, green, blue or a hex colour such as #336699 |
transparent |
fit |
contain (whole avatar) or cover (fill the frame) |
contain |
idle |
1 (idle motion, blinking, breathing, hair sway) or 0 |
1 |
light |
1 enables lighting; omit it for no lighting |
off |
lx, ly |
Light position across and down the frame, 0–1 |
0.2, 0.2 |
lz |
Light height, 0.1–2 |
0.45 |
lr |
Light spread, 0.2–3 |
1.2 |
li, la |
Intensity 0–2, ambient light 0–1 |
0.8, 0.4 |
lc, lac |
Light and ambient colours as six hex digits without # |
ffffff, eef0f8 |
ls, lf |
Strength and shading smoothness, 0–1 |
0.8, 0.45 |
lsp, lrim, ld |
Gloss, rim light and stroke relief, 0–1 |
0.12, 0.3, 0.4 |
lm |
soft or cel |
soft |
shadow |
1 adds the drop shadow |
0 |
Any other bg value falls back to transparent. Out-of-range lighting values are clamped,
and invalid ones use the default. Copy OBS URL and Open stream view include the
Live page's current lighting settings. In OBS, add the URL as a Browser Source
(for example 1080 × 1080); the transparent background needs no chroma key. Use green or blue
for software without transparency support.
avatar accepts relative URLs or HTTP(S) URLs with the same origin (scheme, host and port)
as Studio. External or invalid URLs are ignored and the sample loads instead; they do not
trigger external requests. If project is present, avatar is ignored.
How motion reaches the stream view
- The Live page sends numeric pose values (head angles, eyes, gaze, brows, mouth) over the
dev server's WebSocket, at most 60 times per second. The server accepts only well-formed
numeric messages and rebroadcasts them; each stream view applies only messages for its own
project. - When updates stop for one second, the stream view eases back to idle motion.
- Lighting changes on the Live page reach matching stream views the same way. The server accepts only complete settings made of numbers and the listed options.
- Tracking runs in a Web Worker, so it continues while the Live page is hidden: in another
tab, minimised, or behind another window. Frames come from
MediaStreamTrackProcessorwhere available, otherwise fromrequestVideoFrameCallbackor a worker timer. If tracking still stops or slows, the Live page shows a warning; bring its window to the front. - Face Landmarker uses the GPU and switches to the CPU once if the GPU fails or stays slow after warm-up.
- With the microphone on, the voice level drives how far the mouth opens and the camera keeps the mouth shape. Drawn mouth images are used automatically when the project has them.
Expressions
The collapsed Expression section on the Live page switches the face to Neutral, Smile, Shy, Surprised, Half-lidded, Angry, Sad or Wink. Keys 1–8 do the same while the Live page is focused, even with the section closed; press 1 or the Neutral button to return to Neutral. Pressing the key of the expression already shown keeps it. Keys typed into a form field are ignored. The section title shows the current expression. Browsers receive keys only for the focused page; to switch while another app is in front, use other apps.
An expression is layered over tracking: blinking, gaze and the mouth keep following the camera or microphone, while the eyes, brows and blush hold the expression. Switching fades smoothly. An expression also reaches the stream view without the camera or microphone. Brows hidden under bangs show less difference between Half-lidded, Angry and Sad.
Switching expressions from other apps
Keyboard-shortcut apps and button panels such as Stream Deck can switch expressions while another app, such as OBS, is in front. They send a request to the development server, which forwards it to the Live page; keep the Live page open.
Open Expression → Switch from other apps on the Live page to copy the request URL and the token. Each request is:
POSTto the request URL, for examplehttp://127.0.0.1:5173/__live/expression- the header
X-Studio-Token: <token>(orAuthorization: Bearer <token>) - a JSON body such as
{"expression": "smile"}. Names:neutral,smile,shy,surprise,halfLidded,angry,sad,wink. Add"project": "<name>"to reach only the Live page for that project.
Sending the expression already shown keeps it; send neutral to return to the normal face.
macOS Shortcuts
- In the Shortcuts app, open Settings → Advanced and turn on Allow Running Scripts.
- Create a shortcut, add the Run Shell Script action, and replace its text with the command below (see Command line), using your token. Name the shortcut, for example "Avatar Smile".
- Run it once with the ▶ button to check that the Live page changes.
- In the shortcut's details, choose Add Keyboard Shortcut and press the keys to use.
- Duplicate the shortcut for each expression and change
smilein the command.
macOS runs these keyboard shortcuts through the Services menu, so they do not fire in every app: in testing they worked with OBS in front but not with Finder. Turning on Use as Quick Action → Services Menu in the shortcut's details may help. For keys that work in any app, use a hotkey tool such as Raycast or Hammerspoon to run the same command, or a Stream Deck.
Stream Deck: use a plugin action that sends web requests in the background, set it to POST with the same URL, header and JSON body, and assign one button per expression. The built-in Website action opens a browser tab instead, so it does not work for this.
Command line (also usable from AutoHotkey or other tools):
curl -X POST http://127.0.0.1:5173/__live/expression \
-H "X-Studio-Token: <token>" -H "Content-Type: application/json" \
-d '{"expression": "smile"}'
The token is stored in projects/.expression-token, which is excluded from version control.
Regenerate token replaces it; shortcuts that use the old token then receive 401. Only
apps on this machine can reach the server. Websites open in the browser can neither read the
token nor send the token header.
Without a camera
The camera is optional. Turn on only the microphone and the avatar keeps its idle motion (blinking, gaze, breathing, hair sway) while your voice opens and closes the mouth. Without the camera the mouth keeps one shape and changes only how far it opens; drawn mouth images are still used when the project has them.
Full-body tracking (experimental)
Track body movement (experimental) is off by default. While it is off, Live tracks only the face, exactly as before, and the Pose Landmarker model is not loaded. Its behaviour and settings may change.
When it is on, the tracking worker also runs the local Pose Landmarker lite model on every second face frame and returns only numeric shoulder, hip, elbow and wrist positions. Live lists what the loaded avatar follows:
- Upper body (every avatar): shoulder movement drives body sway and tilt.
- Legs (avatars with
legs): visible hips drive weight shift. When the hips are out of frame, for example while seated, weight shift keeps its idle motion. - Arms (avatars with
arms): elbows and wrists drive small arm swings and elbow bends. Arms never rise; see Rig fields.
Calibrate while standing relaxed with your arms down. Small shoulder movements are ignored, and while your arms move away from the calibrated pose the body keeps its previous lean, so an arm movement is not mistaken for leaning. A joint that leaves the frame holds its pose for 500 ms and then returns to idle.
Webcam body positions are less stable than face tracking, so skirts and other cloth may sway
more than you expect. To calm them, raise Smoothing, or lower each garment's
responseGain towards 1 (physical) in Studio under Body → Cloth. The value is saved in
the rig, so the stream view uses it too. The native runtime ignores arms.
Mouth movement settings
The studio's Lip sync tab offers five styles: Relaxed, Standard, Snappy, Subtle and Expressive. Choosing one immediately previews the mouth motion with kana. Open Fine-tune to adjust movement, size and closing; adjusting a slider selects Custom, and releasing it previews the result. A user-started kana phrase keeps its current playback position while settings change. The disclosure also contains a local audio-file preview driven by loudness. Preview files stay in the browser and are not included in avatar exports.
Each movement setting accepts a finite number from 0 to 1 and defaults to 0.5:
| Setting | 0 | 0.5 (existing behavior) | 1 |
|---|---|---|---|
granularity |
Only large loudness dips separate syllables; merge short kana morae. | Existing sensitivity. | Smaller dips separate syllables; stronger kana closures. |
openness |
Scale opening by 0.55. | Original opening amount. | Scale opening by 1.25, capped at 1. |
closeSpeed |
Slow release (6/s). | Existing audio 32/s and kana 22/s release. | Quick release (45/s). |
Configure the runtime when creating it, or update settings during playback:
import { createMeshAvatar, type LipSyncSettings } from 'mesh-avatar';
const settings: LipSyncSettings = { granularity: 0, openness: 0.5, closeSpeed: 0 };
const avatar = await createMeshAvatar(canvas, { rig, assets, lipSync: settings });
avatar.setLipSync({ openness: 0.75 }); // retains the current movement and closing settings
avatar.setSpeaking(true);
avatar.setVoiceLevel(level); // normalized audio loudness, 0–1
setLipSync(settings: LipSyncSettings): void is a partial update: omitted or undefined fields
retain their current values. Empty settings and omitted creation options use the existing
behavior; explicitly setting all three numbers to 0.5 produces the same motion and random
draws. Invalid objects, non-finite numbers and values outside 0–1 throw before changing settings.
Openness also affects held-mouth previews. Low kana granularity groups adjacent short morae
and softens brief closures; long pauses still close the mouth.
Writable projects save the numbers and optional editor selection in avatar.json.lipSync;
exports include them for browser-picked folders and read-only samples too. loadMeshAvatar
applies these saved values, with explicit options.lipSync fields taking precedence. The
Live page and browser Player apply the avatar's settings to their existing microphone
loudness path. Loading an avatar alone does not start a microphone.
The element exposes element.lipSync: LipSyncSettings (get/set) and
element.setLipSync(settings: LipSyncSettings): void. Both assignment and the method apply
partial updates. The getter returns a copy of explicit overrides; these survive source or
fit changes and otherwise the avatar's saved values apply. The optional preset field
(calm, default, lively, subtle, big, custom) records the editor selection only;
playback always uses the three numbers. The React wrapper exposes the element through its
ref, so call ref.current.setLipSync(...); it has no dedicated settings prop.
Video calls
To use the avatar in Zoom, Google Meet or other video-call apps, show the stream view in OBS and start OBS's Virtual Camera:
- Add the stream URL to an OBS scene as a Browser Source. A transparent background turns black in the virtual camera, so choose a colour or add an image source behind the avatar.
- Choose Start Virtual Camera in OBS. On macOS, approve the camera extension the first time.
- In the video-call app, select OBS Virtual Camera as the camera. Select your microphone directly for audio.
Keep the Live page running during the call. Many video-call apps mirror your own preview only; the other participants see the avatar as it appears in OBS.
Privacy
Camera video and microphone audio are processed in the browser and never leave your machine.
Only the numeric pose values above travel over the local WebSocket. The MediaPipe runtime and
model are served locally; no CDN or remote model is requested, and the end-to-end tests fail if
either page makes a request outside 127.0.0.1.
@mediapipe/tasks-vision is pinned to 0.10.21 because newer releases include code that posts
usage logs to Google. Read vendor/mediapipe/README.md before
upgrading it.
Troubleshooting
- No camera image: allow camera access when the browser asks. On macOS, also check System Settings → Privacy & Security → Camera for your browser. Chrome is recommended.
- Movement is too small or jittery: raise Sensitivity, or raise Smoothing. Calibrate again while facing the camera with a relaxed face.
- Left and right are swapped: toggle Mirror.
- The avatar flickers or an expression keeps flipping: the project is probably open in two Live tabs; the Live page shows a warning when it is. The stream view follows one of them, but switches from other apps reach both. Close the tab you are not using.
- The stream view stays idle: keep the Live page open and running, and make sure both pages
use the same
project.
Verify
For headless deformation statistics and seeded motion comparisons, see deformation and motion metrics.
npm run lint
npm test
npm run build
uv run --with numpy --with pillow --with opencv-python-headless tools/test_build_layers.py
npx playwright install chromium
npm run e2e
docs/screenshots/ stores local visual verification evidence and is excluded from version control.