Skip to content

A lightweight 2.5D character player for the web, with lip sync and webcam face tracking.

A 30 KB core; types for node16 and nodenext; PNGTuber and VRM without the renderer; one file per character on the examples site; the face model served with the pages

LivLivd69d01d250e2
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 476ddeb..c6ec096 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -20,5 +20,6 @@ jobs:
- run: npm run check
- run: npm run build
- run: npm test
+ - run: npm run test:types
- run: npx playwright-core install --with-deps chromium-headless-shell
- run: npm run test:browser
diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml
index be2f994..f7ee8a8 100644
--- a/.github/workflows/pages.yml
+++ b/.github/workflows/pages.yml
@@ -32,6 +32,15 @@ jobs:
cp -r dist example characters media _site/
cp example/pages-index.html _site/index.html
touch _site/.nojekyll
+ # the face model's files, served with the pages: the camera demo then needs neither jsDelivr nor Google's bucket
+ # (some networks reach neither). The checksum is of the model the player was tested with
+ mkdir -p _site/mediapipe/wasm
+ cp node_modules/@mediapipe/tasks-vision/vision_bundle.mjs _site/mediapipe/
+ cp node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_internal.* node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_nosimd_internal.* _site/mediapipe/wasm/
+ curl -fsSL --retry 3 -o _site/mediapipe/face_landmarker.task https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task
+ echo "64184e229b263107bc2b804c6625db1341ff2bb731874b0bcc2fe6544e0bc9ff _site/mediapipe/face_landmarker.task" | sha256sum -c -
+ sed -i 's#https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@[^/]*/vision_bundle.mjs#../mediapipe/vision_bundle.mjs#' _site/example/demo.html _site/example/png.html _site/example/vrm.html
+ grep -q '"../mediapipe/vision_bundle.mjs"' _site/example/demo.html
- uses: actions/configure-pages@v5
- uses: actions/upload-pages-artifact@v3
with:
diff --git a/.gitignore b/.gitignore
index b711778..90e5b3a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,3 +3,5 @@ dist/
.DS_Store
# your own unzipped model package, to try the examples with
example/model/
+# made by the build (scripts/pack-characters.mjs)
+characters/*.livliv
diff --git a/API.md b/API.md
index c43ba93..ed21a4c 100644
--- a/API.md
+++ b/API.md
@@ -2,7 +2,7 @@
A lightweight 2.5D character player for the web. `Player` animates a LivLiv model (a `.livliv` package, or its folder:
`model.json` and `textures/`) on a `<canvas>` with WebGL2; VRM and PNGTuber characters play on the same API through
-`@livliv/player/body`. The core has no dependencies (~98 KB minified, ~34 KB gzipped, the renderer and the idle motion included).
+`@livliv/player/body`. The core has no dependencies (~75 KB minified, ~30 KB gzipped, the renderer and the idle motion included).
Lip sync and face tracking are separate imports, so you only ship them if you use them.
> Not on npm yet. Build it from this repository (see [Building](#building)) and import `dist/index.js`.
@@ -40,6 +40,8 @@ load, as they are ([FORMAT-v3.md](FORMAT-v3.md)).
- `load()` takes the URL of a `.livliv` file, of `model.json`, or of its folder. Image paths are read relative to
`model.json`, so the folder can live anywhere: your server, a CDN, another origin.
+- **Prefer the `.livliv` on a page people wait for.** It is one request, where the folder is one for each image (30-70):
+ on a slow path that saved more than half the wait in our measurements, and a browser's cache has only one file to check.
- **Another origin** (a CDN, `cdn.example.com` while the page is on `example.com`): the server must send
`Access-Control-Allow-Origin` for `model.json` and every image (WebGL can only read images that allow it). Without it
`load()` rejects with an error saying so.
@@ -62,6 +64,9 @@ new Player(canvas: HTMLCanvasElement, options?: PlayerOptions)
`idle` and `followPointer` can also be changed later on the player.
+`new Player()` throws where WebGL2 cannot start: the browser has none or has it turned off, or the canvas already draws
+in 2D. `Player.isSupported()` asks first, so a page can say so or show a picture instead.
+
### `load(source): Promise<ModelInfo>`
Loads a model and shows it in place of the one before (loads run one after another; a failed load keeps the old one).
@@ -133,7 +138,8 @@ only with `idle` off (or use a [driver](#drivers) to add to them). `mouthOpen`,
- `play()`, `pause()`, `playing`. The loop also skips drawing while the canvas is scrolled out of view.
- `resize()`: draws now at the canvas's current size. Only needed after resizing a paused player.
- `snapshot()`: the current frame as a PNG data URL (with the mark).
-- `dispose()`: stops, removes drivers, frees GPU memory. The canvas can take a new `Player` afterwards.
+- `dispose()`: stops, removes drivers, frees GPU memory. A `load()` still running rejects, and a camera or microphone still
+ starting (`startFaceTracking`, `startLipSync`) is let go as soon as it opens. The canvas can take a new `Player` afterwards.
- `puppet`: the renderer underneath, for what this API does not cover. Its fields can change between versions.
### The mark
@@ -294,8 +300,10 @@ await vrm.load('/avatars/me.vrm') // a URL, a File, or files picked
control. Without a bundler, map `three`, `three/examples/jsm/` and `@pixiv/three-vrm` in an import map, as
`example/vrm.html` does.
- **Your own**: anything with `load(files)`, `render(canvas, pose, w, h, dt)`, `head()` and `dispose()` (the `Body`
- type). `pose` has the params after the idle motion and the drivers, the picture's `move` and each expression's weight.
+ type), and if it cares which canvas it gets, `attach(canvas)`. `pose` has the params after the idle motion and the
+ drivers, the picture's `move` and each expression's weight.
+A failed `load()` keeps the character before, as on a Player. A canvas has one kind of context for good: a PNGTuber draws in 2D, a Player and a VRM in WebGL2, so a PNGTuber on a canvas that was a Player's (or the other way round) is refused with an error; give it its own.
A BodyPlayer draws no mark and has no `puppet`; `view.head` is not used. Live2D models (`.moc3`) are not supported:
reading them takes Live2D's proprietary Cubism Core, under its own licence.
diff --git a/README.ja.md b/README.ja.md
index 8fa23a5..8923b50 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -14,7 +14,7 @@
<p align="center"><img src="media/demo.gif" alt="厳選キャラクターが呼吸し、まばたきし、振り向き、話す様子" width="720"></p>
-- WebGL2、依存なし、コアは gzip で約 34 KB
+- WebGL2、依存なし、コアは gzip で約 30 KB
- リップシンク(マイク、音声、動画)と Web カメラのフェイストラッキング(MediaPipe)はブラウザ内で動作し、必要なときだけ読み込めます
- LivLiv・VRM・PNGTuber のキャラクターを同じ API で再生
- [`characters/`](characters/) に厳選キャラクター 21 体。すぐに使えます
diff --git a/README.md b/README.md
index 7d10040..a694aa8 100644
--- a/README.md
+++ b/README.md
@@ -14,7 +14,7 @@
<p align="center"><img src="media/demo.gif" alt="Featured characters breathing, blinking, turning their heads and talking" width="720"></p>
-- WebGL2, no dependencies, about 34 KB gzipped for the core
+- WebGL2, no dependencies, about 30 KB gzipped for the core
- Lip sync (microphone, audio, video) and webcam face tracking (MediaPipe), both in the browser, imported only when you need them
- Plays LivLiv, VRM and PNGTuber characters through the same API
- 21 featured characters in [`characters/`](characters/), ready to use
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 148f5ba..5bca6ca 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -16,7 +16,7 @@
特点:
-- WebGL2,无依赖,核心 gzip 后约 34 KB
+- WebGL2,无依赖,核心 gzip 后约 30 KB
- 口型同步(麦克风、音频、视频)和摄像头面捕(MediaPipe),都在浏览器里运行,按需引入
- 支持 LivLiv、VRM 和 PNGTuber 三种角色,用同一套接口
- [`characters/`](characters/) 里有 21 个精选角色,可以直接用
diff --git a/example/camera.js b/example/camera.js
index 5d29e4e..74e37d5 100644
--- a/example/camera.js
+++ b/example/camera.js
@@ -7,6 +7,12 @@ const NOTES = {
NotReadableError: 'The camera is busy in another app.',
}
+// The face model's files: on the examples' site they are served with the pages (../mediapipe/), else they come from
+// MediaPipe's own hosts, the player's default. On your own site, host them and pass these two the same way.
+const here = (p) => new URL(`../mediapipe/${p}`, location.href).href
+const faceModelFiles = () => fetch(here('face_landmarker.task'), { method: 'HEAD' })
+ .then((r) => (r.ok ? { wasmUrl: here('wasm'), modelUrl: here('face_landmarker.task') } : {}), () => ({}))
+
const style = document.createElement('style')
style.textContent = `
.camera-card { position: fixed; z-index: 2; box-sizing: border-box; width: min(320px, calc(100vw - 24px)); padding: 12px;
@@ -56,6 +62,7 @@ export function cameraButton(button, player, status) {
const { startFaceTracking } = await import('../dist/tracking.js')
running = await startFaceTracking(player, {
trace,
+ ...(trace ? {} : await faceModelFiles()),
onProgress: (s) => { if (asking) status.textContent = s.step === 'download' ? `Face model ${Math.round(s.progress * 100)}%` : `${s.step}…` },
})
asking = false
diff --git a/example/demo.html b/example/demo.html
index 91e18de..da7d3f9 100644
--- a/example/demo.html
+++ b/example/demo.html
@@ -17,11 +17,13 @@
<div class="bar" id="bar"></div>
<p id="status" role="status"></p>
+<script>if (!/[?&]model=/.test(location.search)) document.head.append(Object.assign(document.createElement('link'), { rel: 'preload', as: 'fetch', crossOrigin: 'anonymous', href: '../characters/d03ft.livliv' }))</script>
<script type="module">
import { Player } from '../dist/index.js'
import { cameraButton } from './camera.js'
- import { loading } from './loading.js'
+ import { loading, needWebGL } from './loading.js'
+ needWebGL(Player)
const player = new Player(document.getElementById('stage'))
const bar = document.getElementById('bar'), status = document.getElementById('status')
window.player = player // to try things in the console
@@ -33,7 +35,8 @@
// a model: a featured character, ?model=<URL of a model.json or its folder>, or a folder picked below
async function show(source) {
status.textContent = ''
- const done = loading(document.body, typeof source === 'string' && source.endsWith('/') ? `${source}avatar.webp` : null)
+ // a featured character's small picture lies in its folder, beside its package
+ const done = loading(document.body, typeof source === 'string' && /(\/|\.livliv)$/.test(source) ? `${source.replace(/\.livliv$/, '/')}avatar.webp` : null)
try {
const m = await player.load(source)
status.textContent = `${m.name} · ${m.format} ${m.version} · mark: ${m.watermark ?? 'none'}${m.meta?.license ? ` · ${m.meta.license.id}` : ''}`
@@ -72,6 +75,6 @@
window.camera = cameraButton(button('camera'), player, status)
}
- show(new URLSearchParams(location.search).get('model') || '../characters/d03ft/')
+ show(new URLSearchParams(location.search).get('model') || '../characters/d03ft.livliv')
</script>
</html>
diff --git a/example/gallery.html b/example/gallery.html
index f62455b..955b1f9 100644
--- a/example/gallery.html
+++ b/example/gallery.html
@@ -25,9 +25,15 @@
</div>
<ul class="cast" id="cast"></ul>
</main>
+<script>
+ // the first character's package starts downloading before the script below
+ const first = new URLSearchParams(location.search).get('id')
+ document.head.append(Object.assign(document.createElement('link'), { rel: 'preload', as: 'fetch', crossOrigin: 'anonymous',
+ href: `../characters/${first && /^[\w-]+$/.test(first) ? first : 'd03ft'}.livliv` }))
+</script>
<script type="module">
import { Player } from '../dist/index.js'
- import { loading } from './loading.js'
+ import { loading, needWebGL } from './loading.js'
// All the featured characters: one plays on the stage (one player, one WebGL context), the rest wait as their
// pictures; a click on one puts it on the stage. ?id=<id> starts with that one (else Refined Nana); ?demo=1 plays expressions and
@@ -38,6 +44,7 @@
const rank = (id) => (FIRST.includes(id) ? FIRST.indexOf(id) : FIRST.length)
const cast = list.sort((a, b) => rank(a.id) - rank(b.id)) // the rest keep the list's order
+ needWebGL(Player)
const player = new Player(document.getElementById('stage'), { maxPixelRatio: 1.5 })
const bar = document.getElementById('bar'), name = document.getElementById('name'), ul = document.getElementById('cast')
const buttons = new Map()
@@ -55,10 +62,18 @@
async function show(id) {
if (id === current) return
+ const before = current
current = id
- for (const [k, b] of buttons) b.setAttribute('aria-pressed', String(k === id))
+ const press = (on) => { for (const [k, b] of buttons) b.setAttribute('aria-pressed', String(k === on)) }
+ press(id)
const done = loading(document.getElementById('view'), `../characters/${id}/avatar.webp`)
- const m = await player.load(`../characters/${id}/`).finally(done)
+ let m
+ try {
+ m = await player.load(`../characters/${id}.livliv`).finally(done)
+ } catch (e) { // the one on the stage stays; a click can try this one again
+ if (id === current) { current = before; press(before); name.textContent = e.message }
+ return
+ }
if (id !== current) return // another was picked meanwhile
name.textContent = m.name
exprs = m.expressions; next = -1
diff --git a/example/index.html b/example/index.html
index 645fd7f..09e5b5e 100644
--- a/example/index.html
+++ b/example/index.html
@@ -6,13 +6,16 @@
<style>html, body { margin: 0; height: 100%; background: #20222b } canvas { display: block; width: 100%; height: 100% }</style>
<canvas id="stage"></canvas>
<p id="wait" style="position:fixed;inset:45% 0 auto;margin:0;color:#eef;font:14px system-ui,sans-serif;text-align:center">Loading…</p>
+<link rel="preload" as="fetch" crossorigin href="../characters/d03ft.livliv">
<script type="module">
import { Player } from '../dist/index.js'
// a featured character; your own export works the same
+ const wait = document.getElementById('wait')
+ if (!Player.isSupported()) { wait.textContent = 'This browser has no WebGL2'; throw new Error('no WebGL2') }
const player = new Player(document.getElementById('stage'))
- const model = await player.load('../characters/d03ft/')
- document.getElementById('wait').remove()
+ const model = await player.load('../characters/d03ft.livliv').catch((e) => { wait.textContent = e.message; throw e })
+ wait.remove()
// each click shows the next expression, then neutral again
let i = -1
diff --git a/example/loading.js b/example/loading.js
index 7ffec32..0ff7104 100644
--- a/example/loading.js
+++ b/example/loading.js
@@ -15,3 +15,13 @@ export function loading(parent, picture) {
const later = setTimeout(() => parent.append(box), 150)
return () => { clearTimeout(later); box.remove() }
}
+
+/** Where WebGL2 is missing: says so on the page and stops the page's script here (new Player() would throw). */
+export function needWebGL(Player) {
+ if (Player.isSupported()) return
+ const note = document.createElement('p')
+ note.style.cssText = 'margin:32px 16px;text-align:center;font:15px/1.5 system-ui,sans-serif'
+ note.textContent = 'This browser cannot show the character: it needs WebGL2, which is missing or turned off here.'
+ document.body.append(note)
+ throw new Error('no WebGL2')
+}
diff --git a/package.json b/package.json
index 4753af4..4d4351e 100644
--- a/package.json
+++ b/package.json
@@ -39,6 +39,7 @@
"module": "./dist/index.js",
"types": "./dist/types/src/index.d.ts",
"exports": {
+ "./package.json": "./package.json",
".": {
"types": "./dist/types/src/index.d.ts",
"import": "./dist/index.js"
@@ -90,7 +91,8 @@
"vite": "^8.3.4"
},
"scripts": {
- "build": "rm -rf dist && vite build && tsc -p tsconfig.build.json",
+ "build": "rm -rf dist && vite build && tsc -p tsconfig.build.json && node scripts/dts-extensions.mjs && node scripts/pack-characters.mjs",
+ "test:types": "node test/types.mjs",
"check": "tsc -p tsconfig.json",
"test": "node --disable-warning=ExperimentalWarning --import ./test/ts.mjs --test \"test/*.test.ts\"",
"test:browser": "node test/smoke.mjs"
diff --git a/scripts/dts-extensions.mjs b/scripts/dts-extensions.mjs
new file mode 100644
index 0000000..e92b63a
--- /dev/null
+++ b/scripts/dts-extensions.mjs
@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+// The declarations as tsc writes them keep the source's relative imports, without extensions: fine for a bundler, but under
+// moduleResolution node16 / nodenext (a package with "type": "module") each is an error or, with skipLibCheck, silently
+// `any`. This gives them the extension the files have at run time, after `tsc -p tsconfig.build.json`.
+import fs from 'node:fs'
+import path from 'node:path'
+
+const root = path.resolve(process.argv[2] || 'dist/types')
+const IMPORT = /((?:from\s+|import\s*\(\s*|^\s*import\s+)['"])(\.\.?\/[^'"]+?)(['"])/gm
+let changed = 0
+const walk = (d) => fs.readdirSync(d, { withFileTypes: true }).flatMap((e) => (e.isDirectory() ? walk(path.join(d, e.name)) : [path.join(d, e.name)]))
+for (const f of walk(root).filter((f) => f.endsWith('.d.ts'))) {
+ const text = fs.readFileSync(f, 'utf8')
+ const out = text.replace(IMPORT, (m, a, spec, z) => {
+ if (!fs.existsSync(path.resolve(path.dirname(f), `${spec}.d.ts`))) throw new Error(`${path.relative(root, f)} imports ${spec}: no such declaration`)
+ changed++
+ return `${a}${spec}.js${z}`
+ })
+ if (out !== text) fs.writeFileSync(f, out)
+}
+console.log(`declarations: ${changed} relative imports now say .js`)
diff --git a/scripts/pack-characters.mjs b/scripts/pack-characters.mjs
new file mode 100644
index 0000000..40e8ff1
--- /dev/null
+++ b/scripts/pack-characters.mjs
@@ -0,0 +1,46 @@
+#!/usr/bin/env node
+// Each featured character as one file, characters/<id>.livliv, next to its folder (FORMAT.md §1: model.json first, images
+// and model.json stored, keyforms.json deflated). The examples load these: one request instead of one per image, which on
+// a slow path saves more than half the wait. Written by `npm run build`, not kept in git: they would only double what the
+// repository holds. The bytes depend on the files alone (no dates), so a build changes nothing it need not.
+import fs from 'node:fs'
+import path from 'node:path'
+import { fileURLToPath } from 'node:url'
+import { crc32, deflateRawSync } from 'node:zlib'
+
+const root = fileURLToPath(new URL('../characters/', import.meta.url))
+
+function zip(dir) {
+ const list = (at = '') => fs.readdirSync(path.join(dir, at), { withFileTypes: true }).flatMap((e) => {
+ const p = at ? `${at}/${e.name}` : e.name
+ return e.isDirectory() ? list(p) : [p]
+ }).sort()
+ const names = ['model.json', ...list().filter((f) => f !== 'model.json')]
+ const local = [], central = []
+ let at = 0
+ for (const name of names) {
+ const raw = fs.readFileSync(path.join(dir, name)), deflate = name === 'keyforms.json'
+ const data = deflate ? deflateRawSync(raw, { level: 9 }) : raw, file = Buffer.from(name), crc = crc32(raw)
+ const method = deflate ? 8 : 0
+ const h = Buffer.alloc(30)
+ h.writeUInt32LE(0x04034b50, 0); h.writeUInt16LE(20, 4); h.writeUInt16LE(0x0800, 6); h.writeUInt16LE(method, 8); h.writeUInt16LE(0x21, 12)
+ h.writeUInt32LE(crc, 14); h.writeUInt32LE(data.length, 18); h.writeUInt32LE(raw.length, 22); h.writeUInt16LE(file.length, 26)
+ const c = Buffer.alloc(46)
+ c.writeUInt32LE(0x02014b50, 0); c.writeUInt16LE(20, 4); c.writeUInt16LE(20, 6); c.writeUInt16LE(0x0800, 8); c.writeUInt16LE(method, 10); c.writeUInt16LE(0x21, 14)
+ c.writeUInt32LE(crc, 16); c.writeUInt32LE(data.length, 20); c.writeUInt32LE(raw.length, 24); c.writeUInt16LE(file.length, 28); c.writeUInt32LE(at, 42)
+ local.push(h, file, data); central.push(c, file)
+ at += 30 + file.length + data.length
+ }
+ const cd = Buffer.concat(central), end = Buffer.alloc(22)
+ end.writeUInt32LE(0x06054b50, 0); end.writeUInt16LE(names.length, 8); end.writeUInt16LE(names.length, 10); end.writeUInt32LE(cd.length, 12); end.writeUInt32LE(at, 16)
+ return Buffer.concat([...local, cd, end])
+}
+
+let n = 0, bytes = 0
+for (const e of fs.existsSync(root) ? fs.readdirSync(root, { withFileTypes: true }) : []) {
+ if (!e.isDirectory()) continue
+ const b = zip(path.join(root, e.name))
+ fs.writeFileSync(path.join(root, `${e.name}.livliv`), b)
+ n++; bytes += b.length
+}
+console.log(`characters: ${n} packages, ${(bytes / 1e6).toFixed(1)} MB`)
diff --git a/src/body.ts b/src/body.ts
index 0ec9383..7f45526 100644
--- a/src/body.ts
+++ b/src/body.ts
@@ -5,7 +5,7 @@
// the idle motion, the drivers, actions, expressions); the Body turns it into its own picture.
import { ActionPlayer, type ActionId } from './runtime/actions'
import { pickLabel, uiLang, type Labels } from './runtime/format'
-import { PARAM_RANGES, defaultParams, type Params } from './runtime/puppet'
+import { PARAM_RANGES, defaultParams, type Params } from './runtime/params'
import { Idle, type Seen } from './follow'
import type { Driver, Frame, Host, View } from './index'
@@ -19,7 +19,11 @@ export interface Pose { params: Params; move: { x: number; y: number; s: number
export interface Body {
/** what kind of character it draws ('png', 'vrm', ...): BodyInfo.format */
readonly format: string
- /** loads from its files (by file name); its name and the expressions it can show */
+ /** optional: called before load with the canvas it will draw on; throw if it cannot draw there (the canvas already has
+ * another kind of context), and load() rejects with that */
+ attach?(canvas: HTMLCanvasElement): void
+ /** loads from its files (by file name); its name and the expressions it can show. If it rejects, the character before
+ * must stay as it is */
load(files: Map<string, Blob>): Promise<{ name: string; expressions: BodyExpression[] }>
/** draws the pose into the canvas, w × h device pixels, dt seconds after the last frame */
render(canvas: HTMLCanvasElement, pose: Pose, w: number, h: number, dt: number): void
@@ -121,11 +125,18 @@ export class BodyPlayer implements Host {
get expressions() { return this.info?.expressions ?? [] }
get playing() { return this.raf !== 0 }
- /** Loads a character from its files (see BodySource) in place of the one before. Loads run one after another */
+ /** Loads a character from its files (see BodySource) in place of the one before (a failed load keeps it). Loads run one
+ * after another */
load(source: BodySource): Promise<BodyInfo> {
const run = this.queue.then(async () => {
- if (this.disposed) throw new Error('LivLiv: the player was disposed')
- const got = await this.body.load(await filesOf(source))
+ const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }
+ stillHere()
+ this.body.attach?.(this.canvas)
+ const files = await filesOf(source)
+ stillHere()
+ const got = await this.body.load(files)
+ if (this.disposed) this.body.dispose() // disposed while it loaded: nothing of it may stay
+ stillHere()
const ex = [{ id: 'smile', label: '', labels: SMILE }, ...got.expressions.filter((e) => e.id !== 'smile')
.map((e) => ({ id: e.id, label: e.label, labels: e.labels ?? { en: e.label } }))]
this.info = { name: got.name, format: this.body.format,
@@ -154,6 +165,7 @@ export class BodyPlayer implements Host {
blink() { this.idler.blink(this.time) }
use(driver: Driver): () => void {
+ if (this.disposed) { driver.dispose?.(); return () => {} } // as Player's: what started meanwhile does not stay on
this.drivers.push(driver)
this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))
return () => {
diff --git a/src/follow.ts b/src/follow.ts
index 4f2cb44..a3ba21a 100644
--- a/src/follow.ts
+++ b/src/follow.ts
@@ -3,7 +3,8 @@
// character looks back at the viewer when it leaves. What a player tells it each frame (talking or acting, a face tracker
// driving, an expression showing) and where the pointer is, in one place for Player and BodyPlayer.
import { LivelyIdle, pointAt } from './runtime/idle'
-import { defaultParams, type Params, type Puppet } from './runtime/puppet'
+import { defaultParams, type Params } from './runtime/params'
+import type { Puppet } from './runtime/puppet'
/** the keys the idle motion sets (with it on, your values for these are not used); its smile is added to yours */
export const IDLE_KEYS = ['angleX', 'angleY', 'angleZ', 'eyeLOpen', 'eyeROpen', 'eyeBallX', 'eyeBallY', 'browLY', 'browRY', 'bodyX', 'bodyZ', 'breath'] as const
diff --git a/src/index.ts b/src/index.ts
index cb11467..baa5204 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -96,6 +96,9 @@ export interface Host {
}
+/** what the runtime throws when a canvas gives it no WebGL2 context */
+const NO_WEBGL2 = 'this browser has no WebGL2'
+
/** Plays one LivLiv model on a canvas. Size the canvas with CSS; the drawing follows its size. */
export class Player implements Host {
readonly canvas: HTMLCanvasElement
@@ -128,13 +131,21 @@ export class Player implements Host {
private disposed = false
private frame: Frame & { move: { x: number; y: number; s: number } }
+ /** Whether this browser can play a LivLiv model (WebGL2 starts on a new canvas). Where it cannot, new Player() throws */
+ static isSupported(): boolean {
+ const gl = document.createElement('canvas').getContext('webgl2')
+ gl?.getExtension('WEBGL_lose_context')?.loseContext() // the try takes none of the contexts a page may have at once
+ return !!gl
+ }
+
constructor(canvas: HTMLCanvasElement, options: PlayerOptions = {}) {
this.canvas = canvas
try {
this.puppet = new Puppet(canvas)
} catch (e) {
throw new Error(typeof WebGL2RenderingContext === 'undefined' ? 'LivLiv: this browser has no WebGL2'
- : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`)
+ : (e as Error).message === NO_WEBGL2 ? 'LivLiv: could not start WebGL2 on this canvas: it is turned off in this browser, or the canvas already draws in 2D'
+ : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`)
}
this.puppet.maxDpr = options.maxPixelRatio ?? 2
this.mark = new Mark(this.puppet.gl)
@@ -172,13 +183,13 @@ export class Player implements Host {
}
private async loadNow(source: ModelSource): Promise<ModelInfo> {
- if (this.disposed) throw new Error('LivLiv: the player was disposed')
+ const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }
+ stillHere()
const p = this.puppet
let files: Exclude<ModelSource, string | URL> | null = null
if (typeof source === 'string' || source instanceof URL) {
const url = new URL(String(source), document.baseURI)
- if (/\.(livliv|zip)$/i.test(url.pathname)) files = await fetchPackage(url.href)
- else {
+ if (/\.(livliv|zip)$/i.test(url.pathname)) { files = await fetchPackage(url.href); stillHere() } else {
try {
await p.load(url.href, { pack: false })
} catch (e) {
@@ -188,8 +199,12 @@ export class Player implements Host {
} else files = source
if (files) {
const { model, image, keyforms } = await fromFiles(files)
+ stillHere()
await p.loadModel(model, image, { keyforms }).catch((e) => { throw explain(e, 'model.json') })
}
+ // disposed while the images came in: the runtime built its layers all the same; nothing of them may stay or be drawn
+ if (this.disposed) p.dispose()
+ stillHere()
const m = p.model
const ex = m.expressions || (m.layers.some((l) => l.id === 'expr_smile') ? [{ id: 'smile', label: '', labels: SMILE }] : [])
this.info = {
@@ -231,8 +246,10 @@ export class Player implements Host {
/** Blinks now (with the idle motion on) */
blink() { this.idler.blink(this.puppet.time) }
- /** Adds a driver; returns a function that removes it */
+ /** Adds a driver; returns a function that removes it. On a disposed player it is disposed at once: what started
+ * while the player went (a camera, a microphone) does not stay on */
use(driver: Driver): () => void {
+ if (this.disposed) { driver.dispose?.(); return () => {} }
this.drivers.push(driver)
this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))
return () => {
@@ -262,7 +279,8 @@ export class Player implements Host {
/** The current frame as an image (PNG data URL), the mark included */
snapshot(): string { return this.canvas.toDataURL('image/png') }
- /** Stops, removes the drivers and frees the GPU memory. The canvas can take a new Player afterwards */
+ /** Stops, removes the drivers and frees the GPU memory. A load still running rejects. The canvas can take a new Player
+ * afterwards */
dispose() {
if (this.disposed) return
this.pause()
@@ -279,7 +297,7 @@ export class Player implements Host {
private draw(dt: number) {
const p = this.puppet, P = p.params, f = this.frame
- if (!p.model) return
+ if (!p.model || this.disposed) return
// the pose from scratch: your values, the idle motion, the drivers
this.idler.shown(P)
Object.assign(P, this.params)
diff --git a/src/png.ts b/src/png.ts
index 5fd6aca..2dc9f8c 100644
--- a/src/png.ts
+++ b/src/png.ts
@@ -20,27 +20,39 @@ export class PngBody implements Body {
private pics: Partial<Record<PngSlot, ImageBitmap>> = {}
private g: CanvasRenderingContext2D | null = null
+ attach(canvas: HTMLCanvasElement) {
+ this.g = canvas.getContext('2d')
+ if (!this.g) throw new Error('LivLiv: this canvas already draws with WebGL (a Player or a VRM): a PNGTuber needs a canvas of its own')
+ }
+
async load(files: Map<string, Blob>) {
- this.dispose()
- let pixels = 0
- for (const [name, blob] of files) {
- const slot = PNG_SLOTS.find((s) => name.replace(/\.[^.]*$/, '').toLowerCase() === s.id)?.id
- if (!slot) continue
- const b = await createImageBitmap(blob).catch(() => { throw new Error(`LivLiv: ${name} is not an image this browser can read`) })
- if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) {
- b.close()
- throw new Error(`LivLiv: ${name} is too large (${b.width} x ${b.height})`)
+ // the new pictures apart from the ones shown: anything wrong with them leaves those on the canvas
+ const pics: typeof this.pics = {}
+ let name = 'PNGTuber'
+ try {
+ let pixels = 0
+ for (const [file, blob] of files) {
+ const slot = PNG_SLOTS.find((s) => file.replace(/\.[^.]*$/, '').toLowerCase() === s.id)?.id
+ if (!slot) continue
+ const b = await createImageBitmap(blob).catch(() => { throw new Error(`LivLiv: ${file} is not an image this browser can read`) })
+ pics[slot] = b
+ if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) throw new Error(`LivLiv: ${file} is too large (${b.width} x ${b.height})`)
}
- this.pics[slot] = b
+ if (!pics.closed || !pics.open) throw new Error('LivLiv: a PNGTuber needs at least the closed and the open picture')
+ const txt = files.get('name.txt')
+ if (txt) name = (await txt.slice(0, 200).text()).trim() || name
+ } catch (e) {
+ for (const b of Object.values(pics)) b?.close()
+ throw e
}
- if (!this.pics.closed || !this.pics.open) throw new Error('LivLiv: a PNGTuber needs at least the closed and the open picture')
- const name = files.get('name.txt')
- return { name: name ? (await name.slice(0, 200).text()).trim() || 'PNGTuber' : 'PNGTuber', expressions: [] }
+ this.dispose()
+ this.pics = pics
+ return { name, expressions: [] }
}
render(canvas: HTMLCanvasElement, { params: P, move }: Pose, w: number, h: number) {
if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h }
- const g = (this.g ??= canvas.getContext('2d'))
+ const g = (this.g ??= canvas.getContext('2d')) // attach() has told the page if it cannot have one
if (!g) return
g.clearRect(0, 0, w, h)
const x = this.pics
diff --git a/src/runtime/actions.ts b/src/runtime/actions.ts
index e038282..82c79f6 100644
--- a/src/runtime/actions.ts
+++ b/src/runtime/actions.ts
@@ -2,7 +2,7 @@
// one-shot animation over tracking. Each is a curve of parameter offsets that fades in and out (~0.2 s), so starting
// one mid-move or pressing it again never jumps. "away" is a state, not a move: eyes closed, head down, slow breath,
// until it is turned off (or, entered because no face was seen for a while, until a face is back for a second).
-import { smooth, type Params } from './puppet'
+import { smooth, type Params } from './params'
export type ActionId = 'nod' | 'shake' | 'tilt' | 'hop' | 'shy' | 'away'
diff --git a/src/runtime/params.ts b/src/runtime/params.ts
new file mode 100644
index 0000000..5d029aa
--- /dev/null
+++ b/src/runtime/params.ts
@@ -0,0 +1,26 @@
+// The pose and face a character takes, on their own: what the drivers, the idle motion and the bodies that are not a
+// puppet (a PNGTuber, a VRM) work with, so that none of them has to load the renderer for these. puppet.ts exports them too.
+
+export type Params = {
+ angleX: number; angleY: number; angleZ: number
+ eyeLOpen: number; eyeROpen: number; eyeBallX: number; eyeBallY: number
+ browLY: number; browRY: number
+ mouthOpen: number; smile: number
+ /** the voice's vowel (vowel.ts): -1 round (o, u) .. +1 wide (i, e), 0 neither */
+ vowel: number
+ bodyX: number; bodyZ: number; breath: number
+}
+export const defaultParams = (): Params => ({
+ angleX: 0, angleY: 0, angleZ: 0, eyeLOpen: 1, eyeROpen: 1, eyeBallX: 0, eyeBallY: 0,
+ browLY: 0, browRY: 0, mouthOpen: 0, smile: 0, vowel: 0, bodyX: 0, bodyZ: 0, breath: 0,
+})
+export const PARAM_RANGES: Record<keyof Params, [number, number]> = {
+ angleX: [-30, 30], angleY: [-30, 30], angleZ: [-30, 30], eyeLOpen: [0, 1], eyeROpen: [0, 1],
+ eyeBallX: [-1, 1], eyeBallY: [-1, 1], browLY: [-1, 1], browRY: [-1, 1],
+ mouthOpen: [0, 1], smile: [0, 1], vowel: [-1, 1], bodyX: [-10, 10], bodyZ: [-10, 10], breath: [0, 1],
+}
+
+export const smooth = (a: number, b: number, x: number) => {
+ const t = Math.min(1, Math.max(0, (x - a) / (b - a)))
+ return t * t * (3 - 2 * t)
+}
diff --git a/src/runtime/puppet.ts b/src/runtime/puppet.ts
index 030a2f1..ef1d580 100644
--- a/src/runtime/puppet.ts
+++ b/src/runtime/puppet.ts
@@ -17,6 +17,9 @@
// the bangs where it lies over them
// - body: lags the head turn / tilt, breathes (shoulders + chest), optional chest spring
import { LIMITS, readModel, type Labels, type ModelFiles, type ModelMeta } from './format'
+import { PARAM_RANGES, defaultParams, smooth, type Params } from './params'
+
+export { PARAM_RANGES, defaultParams, smooth, type Params }
export interface LayerDef {
id: string; name: string; texture: string
@@ -189,24 +192,6 @@ export const defaultRig = (): Rig => ({
* each take its place when set, so a model saved with hair alone moves both as before */
export type Tuning = { turn: number; nod: number; hair: number; blink: number; expr: number; mouth: number; breath: number; hairFront?: number; hairBack?: number }
export const defaultTuning = (): Tuning => ({ turn: 1, nod: 1, hair: 1, blink: 1, expr: 1, mouth: 1, breath: 1 })
-export type Params = {
- angleX: number; angleY: number; angleZ: number
- eyeLOpen: number; eyeROpen: number; eyeBallX: number; eyeBallY: number
- browLY: number; browRY: number
- mouthOpen: number; smile: number
- /** the voice's vowel (vowel.ts): -1 round (o, u) .. +1 wide (i, e), 0 neither */
- vowel: number
- bodyX: number; bodyZ: number; breath: number
-}
-export const defaultParams = (): Params => ({
- angleX: 0, angleY: 0, angleZ: 0, eyeLOpen: 1, eyeROpen: 1, eyeBallX: 0, eyeBallY: 0,
- browLY: 0, browRY: 0, mouthOpen: 0, smile: 0, vowel: 0, bodyX: 0, bodyZ: 0, breath: 0,
-})
-export const PARAM_RANGES: Record<keyof Params, [number, number]> = {
- angleX: [-30, 30], angleY: [-30, 30], angleZ: [-30, 30], eyeLOpen: [0, 1], eyeROpen: [0, 1],
- eyeBallX: [-1, 1], eyeBallY: [-1, 1], browLY: [-1, 1], browRY: [-1, 1],
- mouthOpen: [0, 1], smile: [0, 1], vowel: [-1, 1], bodyX: [-10, 10], bodyZ: [-10, 10], breath: [0, 1],
-}
interface GLLayer {
def: LayerDef
@@ -296,11 +281,6 @@ const VOWEL_IN = 0.65, VOWEL_OUT = 0.35
/** how long after the last open frame the character still counts as talking (s): the gaps between syllables */
const TALK_LATCH = 0.4
-export const smooth = (a: number, b: number, x: number) => {
- const t = Math.min(1, Math.max(0, (x - a) / (b - a)))
- return t * t * (3 - 2 * t)
-}
-
/** cap (optional): per sample the most the brow may move there (model px), where it lies under opaque bangs */
export type BrowLine = { x: number[]; bottom: number[]; lid: number[]; cap?: number[] }
const interp = (xs: number[], ys: number[], x: number) => {
diff --git a/src/runtime/talk.ts b/src/runtime/talk.ts
index 5d8bd33..817f278 100644
--- a/src/runtime/talk.ts
+++ b/src/runtime/talk.ts
@@ -5,7 +5,7 @@
// - loud onsets now and then lift the brows; the eyes come back to the viewer
// - a pause after talking brings a blink
// - bounce (optional): the whole character hops a little on syllables, as PNGTubers do
-import { smooth, type Params } from './puppet'
+import { smooth, type Params } from './params'
/** a syllable: the mouth opens by RISE above the last low point, to at least ONSET (within a phrase the voice rarely
* closes it between syllables, so a fixed threshold misses most); the next one counts once it has come down by FALL.
diff --git a/src/runtime/tracking.ts b/src/runtime/tracking.ts
index 96c074a..137fe14 100644
--- a/src/runtime/tracking.ts
+++ b/src/runtime/tracking.ts
@@ -1,5 +1,5 @@
// Webcam face tracking -> puppet params, via MediaPipe Face Landmarker (runs locally in the browser).
-import { smooth, type Params } from './puppet'
+import { smooth, type Params } from './params'
import { OneEuro } from './filters'
import { BS, LM, SynthStats, TraceRecorder, TraceSource, synthTrace, type Frame } from './trace'
diff --git a/src/tracking.ts b/src/tracking.ts
index e38dcac..fb20d57 100644
--- a/src/tracking.ts
+++ b/src/tracking.ts
@@ -3,7 +3,7 @@
// the @mediapipe/tasks-vision package and downloads ~8 MB of model the first time (the wasm runtime and the face model;
// cached by the browser after that).
import { FaceTracker, LOST_AFTER, V, loadFaceModel, setFaceModelFiles, type StartStep } from './runtime/tracking'
-import { defaultParams, type Params } from './runtime/puppet'
+import { defaultParams, type Params } from './runtime/params'
import type { Driver, Host } from './index'
export type { StartStep }
diff --git a/src/vrm.ts b/src/vrm.ts
index 8b2c7a8..bf8ba70 100644
--- a/src/vrm.ts
+++ b/src/vrm.ts
@@ -32,6 +32,13 @@ export class VrmBody implements Body {
private canvas: HTMLCanvasElement | null = null
private headAt = { x: 0.5, y: 0.25 }
+ attach(canvas: HTMLCanvasElement) {
+ // the attributes render() gives three.js, which then finds this same context
+ if (!canvas.getContext('webgl2', { alpha: true, antialias: true, premultipliedAlpha: true, preserveDrawingBuffer: true })) {
+ throw new Error('LivLiv: a VRM needs WebGL2: this browser has none, or this canvas already draws in 2D (a PNGTuber)')
+ }
+ }
+
async load(files: Map<string, Blob>) {
const entry = [...files].find(([n]) => /\.(vrm|glb)$/i.test(n))
if (!entry) throw new Error('LivLiv: no .vrm file among the files')
@@ -40,7 +47,6 @@ export class VrmBody implements Body {
throw new Error(`LivLiv: a VRM needs the three and @pixiv/three-vrm packages (${(e as Error).message})`)
})
this.THREE = THREE
- this.dispose()
const loader = new GLTFLoader()
loader.register((parser) => new VRMLoaderPlugin(parser))
const url = URL.createObjectURL(entry[1])
@@ -53,14 +59,18 @@ export class VrmBody implements Body {
URL.revokeObjectURL(url)
}
const vrm = gltf.userData.vrm as VRM | undefined
- if (!vrm) throw new Error(`LivLiv: ${entry[0]} is a glTF without VRM data`)
+ if (!vrm) {
+ VRMUtils.deepDispose(gltf.scene)
+ throw new Error(`LivLiv: ${entry[0]} is a glTF without VRM data`)
+ }
+ // the new character, apart from the one shown (a failed load leaves that on the canvas): it takes over at the end
VRMUtils.removeUnnecessaryVertices(gltf.scene)
VRMUtils.combineSkeletons(gltf.scene)
VRMUtils.rotateVRM0(vrm) // VRM 0.x faces the other way
vrm.scene.traverse((o) => { o.frustumCulled = false })
- this.vrm = vrm
const h = vrm.humanoid
- for (const b of ['head', 'neck', 'spine', 'chest'] as const) this.bones[b] = h.getNormalizedBoneNode(b) ?? undefined
+ const bones: typeof this.bones = {}
+ for (const b of ['head', 'neck', 'spine', 'chest'] as const) bones[b] = h.getNormalizedBoneNode(b) ?? undefined
// arms down from the T pose, a little forward
const lu = h.getNormalizedBoneNode('leftUpperArm'), ru = h.getNormalizedBoneNode('rightUpperArm')
if (lu) lu.rotation.set(0, 0, -1.2)
@@ -71,24 +81,31 @@ export class VrmBody implements Body {
if (vrm.lookAt) vrm.lookAt.autoUpdate = false
vrm.update(0)
- this.scene = new THREE.Scene()
- this.scene.add(vrm.scene)
+ const scene = new THREE.Scene()
+ scene.add(vrm.scene)
const light = new THREE.DirectionalLight(0xffffff, Math.PI)
light.position.set(1, 1, 1).normalize()
- this.scene.add(light, new THREE.AmbientLight(0xffffff, 0.6))
+ scene.add(light, new THREE.AmbientLight(0xffffff, 0.6))
// the head's top to the hips: what a LivLiv character's canvas shows of a person
const at = (b: 'head' | 'hips') => { const v = new THREE.Vector3(); h.getNormalizedBoneNode(b)?.getWorldPosition(v); return v }
const head = at('head'), hips = at('hips')
const top = head.y + 0.2, bottom = hips.y
+ const has = new Set((vrm.expressionManager?.expressions ?? []).map((e) => e.expressionName))
+ const meta = vrm.meta as { name?: string; title?: string } | undefined
+ const expressions: BodyExpression[] = FEELINGS.filter((f) => f !== 'happy' && has.has(f)).map((f) => ({ id: f, label: LABELS[f].en, labels: LABELS[f] }))
+ if (has.has('happy')) expressions.unshift({ id: 'happy', label: LABELS.happy.en, labels: LABELS.happy })
+ const name = (meta?.name || meta?.title || entry[0].replace(/\.(vrm|glb)$/i, '')).slice(0, 200)
+
+ const old = this.vrm
+ this.vrm = vrm
+ this.scene = scene
+ this.bones = bones
+ this.has = has
this.frame = { height: Math.max(0.1, top - bottom), y: (top + bottom) / 2 }
this.camera = new THREE.PerspectiveCamera(20, 2 / 3, 0.05, 30)
- this.place(2 / 3)
-
- this.has = new Set((vrm.expressionManager?.expressions ?? []).map((e) => e.expressionName))
- const meta = vrm.meta as { name?: string; title?: string } | undefined
- const expressions: BodyExpression[] = FEELINGS.filter((f) => f !== 'happy' && this.has.has(f)).map((f) => ({ id: f, label: LABELS[f].en, labels: LABELS[f] }))
- if (this.has.has('happy')) expressions.unshift({ id: 'happy', label: LABELS.happy.en, labels: LABELS.happy })
- return { name: (meta?.name || meta?.title || entry[0].replace(/\.(vrm|glb)$/i, '')).slice(0, 200), expressions }
+ this.place(2 / 3) // render() gives it the canvas's aspect
+ if (old) VRMUtils.deepDispose(old.scene)
+ return { name, expressions }
}
/** the camera where the framing fits the canvas's aspect */
@@ -111,7 +128,8 @@ export class VrmBody implements Body {
this.renderer.setClearColor(0x000000, 0)
this.canvas = canvas
}
- if (canvas.width !== w || canvas.height !== h) { this.renderer.setSize(w, h, false); this.place(w / h) }
+ if (canvas.width !== w || canvas.height !== h) this.renderer.setSize(w, h, false)
+ if (this.camera.aspect !== w / h) this.place(w / h) // a new character's camera, on a canvas already this size too
const b = this.bones
// the head turns most, the neck some (yaw: to the screen's right; pitch: up; roll: toward the screen's left)
b.neck?.rotation.set(-P.angleY * D * 0.35, P.angleX * D * 0.35, -P.angleZ * D * 0.35)
diff --git a/test/bodies.html b/test/bodies.html
index db09cdd..37e6bd2 100644
--- a/test/bodies.html
+++ b/test/bodies.html
@@ -66,6 +66,19 @@
tracking.stop(); speech.dispose()
out.png = { name: m.name, format: m.format, expressions: m.expressions.map((e) => e.id), shown, ...seen }
p.dispose()
+ // a failed load leaves the pictures shown, and the name; on a canvas with WebGL it says so
+ const canvas2 = document.createElement('canvas')
+ canvas2.style.cssText = 'width: 100px; height: 120px'
+ document.body.append(canvas2)
+ const q = new BodyPlayer(canvas2, new PngBody(), { autoplay: false, idle: false })
+ await q.load({ closed: await solid('#ff0000'), open: await solid('#00ff00'), 'name.txt': new Blob(['First']) })
+ const refused = await q.load({ closed: await solid('#0000ff') }).then(() => 'loaded', (e) => e.message)
+ q.resize()
+ out.pngFailed = { refused, kept: q.model?.name, pixel: pixel(canvas2, canvas2.width / 2, canvas2.height * 0.8) }
+ q.dispose(); canvas2.remove()
+ const gl = document.createElement('canvas')
+ gl.getContext('webgl2')
+ out.pngOnGl = await new BodyPlayer(gl, new PngBody(), { autoplay: false }).load({ closed: await solid('#ff0000'), open: await solid('#00ff00') }).then(() => 'loaded', (e) => e.message)
} catch (e) { out.png = { error: String(e.message || e) } }
try {
const canvas = document.getElementById('vrm')
@@ -80,6 +93,24 @@
const tracking = await startFaceTracking(p, { trace: 'synth' })
await wait(800)
tracking.stop()
+ // a second load, on a canvas that is not 2:3 and already has its size, and a failed one
+ const wide = document.createElement('canvas')
+ wide.style.cssText = 'width: 400px; height: 200px'
+ document.body.append(wide)
+ wide.width = 400; wide.height = 200
+ const w = new BodyPlayer(wide, new VrmBody(), { autoplay: false, idle: false })
+ await w.load('/__tiny.vrm'); w.resize()
+ const aspect1 = w.body.camera.aspect
+ await w.load('/__tiny.vrm'); w.resize()
+ const aspect2 = w.body.camera.aspect
+ const bad = new File([new Blob(['not a glb'])], 'x.vrm')
+ const refused = await w.load(bad).then(() => 'loaded', (e) => e.message)
+ w.resize()
+ out.vrmAgain = { aspect1, aspect2, refused, kept: w.model?.name, share: covered(wide) }
+ w.dispose(); wide.remove()
+ const flat = document.createElement('canvas')
+ flat.getContext('2d')
+ out.vrmOn2d = await new BodyPlayer(flat, new VrmBody(), { autoplay: false }).load('/__tiny.vrm').then(() => 'loaded', (e) => e.message)
const head = p.body.head()
out.vrm = { name: m.name, format: m.format, expressions: m.expressions.map((e) => e.label), share, happy, smile, unknown, tracked: seen.tracked,
head: [head.x, head.y].every(Number.isFinite) }
diff --git a/test/smoke.html b/test/smoke.html
index 00149d3..1a4c75a 100644
--- a/test/smoke.html
+++ b/test/smoke.html
@@ -8,6 +8,7 @@
// and action without a WebGL error, and shows its mark; one loads from files instead of a URL, and again as the v3
// package of the same model (it must draw the same pixels); a model of a newer format is refused.
import { FORMAT_VERSION, LEGACY_FORMAT, Player } from '../dist/index.js'
+ import { startLipSync } from '../dist/voice.js'
const canvas = document.getElementById('c')
const g = document.createElement('canvas').getContext('2d', { willReadFrequently: true })
@@ -59,9 +60,9 @@
const p = new Player(canvas, { autoplay: false, followPointer: false, idle: false })
const info = await p.load(source)
p.resize()
- const d = pixels(), drawn = JSON.parse(JSON.stringify(p.puppet.model))
+ const d = pixels(), drawn = JSON.parse(JSON.stringify(p.puppet.model)), keys = p.puppet.keys.length
p.dispose()
- return { d, info, drawn }
+ return { d, info, drawn, keys }
}
/** what the model's mark changes in a still frame (drawn as it comes, and with the mark off), how much of that lies
@@ -110,6 +111,13 @@
fromFiles = { error: String(e.message || e) }
}
+ // the same characters as one file from a URL (made by the build): the same frame, keyforms and all
+ const packaged = []
+ for (const p of ids.filter((i) => i === id || FILES[i].includes('keyforms.json'))) {
+ const a = await still(`../characters/${p}/`), b = await still(`../characters/${p}.livliv`)
+ packaged.push({ id: p, changed: changed(a.d, b.d), keys: [a.keys, b.keys] })
+ }
+
// a newer format: refused, and the model before stays
let newer
try {
@@ -134,7 +142,54 @@
} catch (e) {
legacy = { error: String(e.message || e) }
}
- return { out, fromFiles, newer, legacy }
+ return { out, fromFiles, newer, legacy, packaged }
+ }
+
+ /** disposing while something runs: a load rejects and leaves nothing on the GPU or the canvas; a driver added to a
+ * disposed player is disposed; a microphone that opens after it is let go */
+ window.runLifecycle = async (id) => {
+ const out = { supported: Player.isSupported() }
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 200px; height: 200px'
+ document.body.append(c)
+ {
+ const p = new Player(c, { followPointer: false }), gl = p.puppet.gl
+ let made = 0, freed = 0
+ const { createTexture, deleteTexture } = gl
+ gl.createTexture = function () { made++; return createTexture.call(gl) }
+ gl.deleteTexture = function (t) { if (t) freed++; return deleteTexture.call(gl, t) }
+ const load = p.load(`../characters/${id}/`).then(() => 'resolved', (e) => e.message)
+ await new Promise((r) => setTimeout(r, 5))
+ p.dispose()
+ out.load = await load
+ p.resize()
+ await new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)))
+ const px = new Uint8Array(4)
+ gl.readPixels(100, 100, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, px)
+ Object.assign(out, { model: p.model, drawn: px[3], made, freed, glError: gl.getError() })
+ }
+ {
+ const p = new Player(c, { followPointer: false })
+ p.dispose()
+ let disposed = 0
+ const remove = p.use({ update() {}, dispose() { disposed++ } })
+ remove()
+ out.driver = disposed
+ const again = new Player(c, { followPointer: false })
+ await again.load(`../characters/${id}/`)
+ again.resize()
+ out.again = again.model?.name ?? null
+ const real = navigator.mediaDevices.getUserMedia.bind(navigator.mediaDevices)
+ let stream
+ navigator.mediaDevices.getUserMedia = async (k) => { stream = await real(k); return stream }
+ const starting = startLipSync(again)
+ again.dispose()
+ await starting
+ navigator.mediaDevices.getUserMedia = real
+ out.mic = stream.getTracks()[0].readyState
+ }
+ c.remove()
+ return out
}
/** hostile packages: each must be refused (or, for broken keyforms, played on the rig alone), the model before kept */
diff --git a/test/smoke.mjs b/test/smoke.mjs
index f581d48..d23b7e6 100644
--- a/test/smoke.mjs
+++ b/test/smoke.mjs
@@ -46,7 +46,8 @@ const server = createServer((req, res) => {
})
await new Promise((r) => server.listen(0, '127.0.0.1', r))
-const browser = await chromium.launch({ args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--autoplay-policy=no-user-gesture-required'] })
+const browser = await chromium.launch({ args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--autoplay-policy=no-user-gesture-required',
+ '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] })
let failed = 0
try {
const page = await browser.newPage()
@@ -54,7 +55,7 @@ try {
page.on('pageerror', (e) => errors.push(e.message))
await page.goto(`http://127.0.0.1:${server.address().port}/test/smoke.html`)
await page.waitForFunction(() => typeof window.run === 'function')
- const { out, fromFiles, newer, legacy } = await page.evaluate((a) => window.run(a), { ids, files })
+ const { out, fromFiles, newer, legacy, packaged } = await page.evaluate((a) => window.run(a), { ids, files })
for (const r of out) {
// the featured characters carry the LivLiv mark in the corner only: the player must draw it there and nowhere else
const ok = !r.error && r.share > 0.05 && r.glError === 0 && r.watermark === 'LivLiv' && r.mark > 0.001 && r.outside < 0.0003 && r.noise < 0.0005
@@ -65,6 +66,11 @@ try {
const filesOk = !fromFiles.error && fromFiles.share > 0.05
if (!filesOk) failed++
console.log(`${filesOk ? 'ok ' : 'FAIL'} from files: ${fromFiles.error ?? `${fromFiles.name}, covers ${(fromFiles.share * 100).toFixed(1)} %`}`)
+ for (const k of packaged) {
+ const ok = k.changed < 0.0005 && k.keys[0] === k.keys[1]
+ if (!ok) failed++
+ console.log(`${ok ? 'ok ' : 'FAIL'} ${k.id}.livliv from a URL: ${(k.changed * 100).toFixed(2)} % of the pixels differ from its folder, keyforms ${k.keys.join(' / ')}`)
+ }
const legacyOk = !legacy.error && /^livemaker-puppet 3$/.test(legacy.format) && legacy.watermark === null && legacy.changed < 0.0005
if (!legacyOk) failed++
console.log(`${legacyOk ? 'ok ' : 'FAIL'} v3 package: ${legacy.error ?? `${legacy.format}, ${(legacy.changed * 100).toFixed(2)} % of the pixels differ from its v4`}`)
@@ -84,10 +90,17 @@ try {
if (!kfOk) failed++
console.log(`${kfOk ? 'ok ' : 'FAIL'} broken keyforms: ${hostile.keyforms.refused ?? `${hostile.keyforms.name} plays on the rig alone`}`)
+ // disposing while something runs (see runLifecycle)
+ const life = await page.evaluate((id) => window.runLifecycle(id), ids[0])
+ const lifeOk = /disposed/.test(life.load) && life.model === null && life.drawn === 0 && life.freed >= life.made && life.glError === 0
+ && life.driver === 1 && !!life.again && life.mic === 'ended' && life.supported === true
+ if (!lifeOk) failed++
+ console.log(`${lifeOk ? 'ok ' : 'FAIL'} dispose while running: load "${life.load}", ${life.made} textures made, ${life.freed} freed, drawn ${life.drawn}, GL error ${life.glError}, driver disposed ${life.driver}, a new player ${life.again}, microphone ${life.mic}`)
+
// bodies: a PNGTuber shows each picture and follows the drivers; a VRM loads, draws and takes its expressions
await page.goto(`http://127.0.0.1:${server.address().port}/test/bodies.html`)
await page.waitForFunction(() => typeof window.runBodies === 'function')
- const { png, vrm: v } = await page.evaluate(() => window.runBodies())
+ const { png, vrm: v, pngFailed, pngOnGl, vrmAgain, vrmOn2d } = await page.evaluate(() => window.runBodies())
const is = (rgb, [r, g, b]) => Math.abs(rgb[0] - r) < 40 && Math.abs(rgb[1] - g) < 40 && Math.abs(rgb[2] - b) < 40
const pngOk = !png.error && png.format === 'png' && is(png.shown.closed, [255, 0, 0]) && is(png.shown.open, [0, 255, 0])
&& is(png.shown.blink, [0, 0, 255]) && is(png.shown.blinkOpen, [255, 255, 0]) && png.tracked && png.angle > 0 && png.mouth > 0.3
@@ -97,7 +110,35 @@ try {
&& ['Smile', 'Happy', 'Sad'].every((l) => v.expressions.includes(l))
if (!vrmOk) failed++
console.log(`${vrmOk ? 'ok ' : 'FAIL'} VRM: ${v.error ?? `${v.name}, ${v.expressions.join(' / ')}, covers ${(v.share * 100).toFixed(1)} %, face tracked ${v.tracked}`}`)
+ const pfOk = /needs at least/.test(pngFailed?.refused) && pngFailed.kept === 'First' && is(pngFailed.pixel, [255, 0, 0]) && /own/.test(pngOnGl)
+ if (!pfOk) failed++
+ console.log(`${pfOk ? 'ok ' : 'FAIL'} PNGTuber, a failed load: ${pngFailed?.refused} (kept ${pngFailed?.kept}, pixel ${pngFailed?.pixel}); on a WebGL canvas: ${pngOnGl}`)
+ const vaOk = vrmAgain?.aspect1 === 2 && vrmAgain.aspect2 === 2 && /not a VRM/.test(vrmAgain.refused) && vrmAgain.kept === 'Tiny' && vrmAgain.share > 0.02 && /2D/.test(vrmOn2d)
+ if (!vaOk) failed++
+ console.log(`${vaOk ? 'ok ' : 'FAIL'} VRM, loaded again on a wide canvas: aspect ${vrmAgain?.aspect1}, ${vrmAgain?.aspect2}; a failed load kept ${vrmAgain?.kept}, covers ${(vrmAgain?.share * 100).toFixed(1)} %; on a 2D canvas: ${vrmOn2d}`)
if (errors.length) { failed++; console.log(`FAIL page errors:\n ${errors.join('\n ')}`) }
+ // no WebGL at all: the example pages say so, the check says no, new Player() says why (the pages fail on purpose: no
+ // error of theirs counts here)
+ const plain = await chromium.launch({ args: ['--disable-gpu', '--disable-software-rasterizer', '--disable-webgl', '--disable-webgl2', '--disable-3d-apis'] })
+ try {
+ const p = await plain.newPage()
+ for (const name of ['index', 'demo', 'gallery']) {
+ await p.goto(`http://127.0.0.1:${server.address().port}/example/${name}.html`)
+ await p.waitForFunction(() => /WebGL2/.test(document.body.innerText), null, { timeout: 8000 }).catch(() => {})
+ const said = /WebGL2/.test(await p.evaluate(() => document.body.innerText))
+ if (!said) failed++
+ console.log(`${said ? 'ok ' : 'FAIL'} no WebGL: example/${name}.html ${said ? 'says so' : 'says nothing'}`)
+ }
+ const r = await p.evaluate(async () => {
+ const { Player } = await import('/dist/index.js')
+ let thrown = ''
+ try { new Player(document.createElement('canvas')) } catch (e) { thrown = e.message }
+ return { supported: Player.isSupported(), thrown }
+ })
+ const ok = r.supported === false && /turned off/.test(r.thrown)
+ if (!ok) failed++
+ console.log(`${ok ? 'ok ' : 'FAIL'} no WebGL: isSupported ${r.supported}, new Player() says "${r.thrown}"`)
+ } finally { await plain.close() }
} finally {
await browser.close()
server.close()
diff --git a/test/types.mjs b/test/types.mjs
new file mode 100644
index 0000000..834d49a
--- /dev/null
+++ b/test/types.mjs
@@ -0,0 +1,56 @@
+// The package as a project that installed it sees it: its entries and types under the moduleResolution settings TypeScript
+// projects use, with the declarations checked (skipLibCheck off). A broken declaration fails here; with skipLibCheck on it
+// would pass and leave `any` behind, which the @ts-expect-error below catches.
+// npm run build && npm run test:types
+import { execFileSync } from 'node:child_process'
+import fs from 'node:fs'
+import { createRequire } from 'node:module'
+import path from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const root = fileURLToPath(new URL('..', import.meta.url))
+const { name } = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'))
+if (!fs.existsSync(path.join(root, 'dist/index.js'))) { console.error('types: build first (npm run build)'); process.exit(1) }
+const dir = path.join(root, 'node_modules/.tmp/consumer')
+fs.rmSync(dir, { recursive: true, force: true })
+fs.mkdirSync(path.join(dir, 'node_modules', path.dirname(name)), { recursive: true })
+fs.symlinkSync(root, path.join(dir, 'node_modules', name), 'dir')
+fs.writeFileSync(path.join(dir, 'use.ts'), `
+import { Player, VERSION, type Driver, type ModelInfo, type View } from '${name}'
+import { preloadFaceModel, startFaceTracking, type FaceTracking } from '${name}/tracking'
+import { createSpeech, startLipSync, type LipSync } from '${name}/voice'
+import { BODY_LIMIT, BodyPlayer, type Body } from '${name}/body'
+import { PngBody } from '${name}/png'
+import { VrmBody } from '${name}/vrm'
+declare const canvas: HTMLCanvasElement
+const player = new Player(canvas)
+const model: Promise<ModelInfo> = player.load('/avatars/sena.livliv')
+const driver: Driver = { update(frame) { frame.add({ angleX: 1 }) } }
+const view: View = player.view
+const tracking: Promise<FaceTracking> = startFaceTracking(player)
+const sync: Promise<LipSync> = startLipSync(player)
+const body: Body = new PngBody()
+player.use(driver)
+void [VERSION, model, view, tracking, sync, preloadFaceModel(), createSpeech(player), BODY_LIMIT, new BodyPlayer(canvas, body), new VrmBody()]
+// the types are real, not any: these must be errors
+// @ts-expect-error not an action
+player.act('nope')
+// @ts-expect-error a name is text
+const n: number = (await model).name
+void n
+`)
+const tsc = createRequire(path.join(root, 'package.json')).resolve('typescript/bin/tsc')
+let failed = 0
+for (const [moduleResolution, module] of [['node16', 'node16'], ['nodenext', 'nodenext'], ['bundler', 'esnext']]) {
+ fs.writeFileSync(path.join(dir, 'tsconfig.json'), JSON.stringify({ compilerOptions: { noEmit: true, strict: true, target: 'es2022', lib: ['es2023', 'dom', 'dom.iterable'],
+ module, moduleResolution, skipLibCheck: false, types: [] }, files: ['use.ts'] }))
+ try {
+ execFileSync(process.execPath, [tsc, '-p', dir], { stdio: 'pipe' })
+ console.log(`ok moduleResolution ${moduleResolution}`)
+ } catch (e) {
+ failed++
+ console.log(`FAIL moduleResolution ${moduleResolution}\n${String(e.stdout).split('\n').slice(0, 12).join('\n')}`)
+ }
+}
+fs.rmSync(dir, { recursive: true, force: true })
+process.exit(failed ? 1 : 0)
diff --git a/vite.config.ts b/vite.config.ts
index 75bd0d8..c5fa9ea 100644
--- a/vite.config.ts
+++ b/vite.config.ts
@@ -15,7 +15,9 @@ export default defineConfig({
},
rollupOptions: {
external: ['@mediapipe/tasks-vision', 'three', /^three\//, '@pixiv/three-vrm'],
- output: { chunkFileNames: 'chunks/[name]-[hash].js' },
+ // minify: true above leaves the whitespace of an ES library in place (and these comments with the file names): the
+ // package's readers download dist as it is from a CDN, so this one takes it out too (pure annotations stay)
+ output: { chunkFileNames: 'chunks/[name]-[hash].js', minify: true },
},
},
})