A lightweight 2.5D character player for the web, with lip sync and webcam face tracking.
A 30 KB core; types for node16 and nodenext; PNGTuber and VRM without the renderer; one file per character on the examples site; the face model served with the pages
LivLivd69d01d250e2
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 476ddeb..c6ec096 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -20,5 +20,6 @@ jobs:- run: npm run check- run: npm run build- run: npm test+ - run: npm run test:types- run: npx playwright-core install --with-deps chromium-headless-shell- run: npm run test:browserdiff --git a/.github/workflows/pages.yml b/.github/workflows/pages.ymlindex be2f994..f7ee8a8 100644--- a/.github/workflows/pages.yml+++ b/.github/workflows/pages.yml@@ -32,6 +32,15 @@ jobs:cp -r dist example characters media _site/cp example/pages-index.html _site/index.htmltouch _site/.nojekyll+ # the face model's files, served with the pages: the camera demo then needs neither jsDelivr nor Google's bucket+ # (some networks reach neither). The checksum is of the model the player was tested with+ mkdir -p _site/mediapipe/wasm+ cp node_modules/@mediapipe/tasks-vision/vision_bundle.mjs _site/mediapipe/+ cp node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_internal.* node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_nosimd_internal.* _site/mediapipe/wasm/+ curl -fsSL --retry 3 -o _site/mediapipe/face_landmarker.task https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task+ echo "64184e229b263107bc2b804c6625db1341ff2bb731874b0bcc2fe6544e0bc9ff _site/mediapipe/face_landmarker.task" | sha256sum -c -+ sed -i 's#https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@[^/]*/vision_bundle.mjs#../mediapipe/vision_bundle.mjs#' _site/example/demo.html _site/example/png.html _site/example/vrm.html+ grep -q '"../mediapipe/vision_bundle.mjs"' _site/example/demo.html- uses: actions/configure-pages@v5- uses: actions/upload-pages-artifact@v3with:diff --git a/.gitignore b/.gitignoreindex b711778..90e5b3a 100644--- a/.gitignore+++ b/.gitignore@@ -3,3 +3,5 @@ dist/.DS_Store# your own unzipped model package, to try the examples withexample/model/+# made by the build (scripts/pack-characters.mjs)+characters/*.livlivdiff --git a/API.md b/API.mdindex c43ba93..ed21a4c 100644--- a/API.md+++ b/API.md@@ -2,7 +2,7 @@A lightweight 2.5D character player for the web. `Player` animates a LivLiv model (a `.livliv` package, or its folder:`model.json` and `textures/`) on a `<canvas>` with WebGL2; VRM and PNGTuber characters play on the same API through-`@livliv/player/body`. The core has no dependencies (~98 KB minified, ~34 KB gzipped, the renderer and the idle motion included).+`@livliv/player/body`. The core has no dependencies (~75 KB minified, ~30 KB gzipped, the renderer and the idle motion included).Lip sync and face tracking are separate imports, so you only ship them if you use them.> Not on npm yet. Build it from this repository (see [Building](#building)) and import `dist/index.js`.@@ -40,6 +40,8 @@ load, as they are ([FORMAT-v3.md](FORMAT-v3.md)).- `load()` takes the URL of a `.livliv` file, of `model.json`, or of its folder. Image paths are read relative to`model.json`, so the folder can live anywhere: your server, a CDN, another origin.+- **Prefer the `.livliv` on a page people wait for.** It is one request, where the folder is one for each image (30-70):+ on a slow path that saved more than half the wait in our measurements, and a browser's cache has only one file to check.- **Another origin** (a CDN, `cdn.example.com` while the page is on `example.com`): the server must send`Access-Control-Allow-Origin` for `model.json` and every image (WebGL can only read images that allow it). Without it`load()` rejects with an error saying so.@@ -62,6 +64,9 @@ new Player(canvas: HTMLCanvasElement, options?: PlayerOptions)`idle` and `followPointer` can also be changed later on the player.+`new Player()` throws where WebGL2 cannot start: the browser has none or has it turned off, or the canvas already draws+in 2D. `Player.isSupported()` asks first, so a page can say so or show a picture instead.+### `load(source): Promise<ModelInfo>`Loads a model and shows it in place of the one before (loads run one after another; a failed load keeps the old one).@@ -133,7 +138,8 @@ only with `idle` off (or use a [driver](#drivers) to add to them). `mouthOpen`,- `play()`, `pause()`, `playing`. The loop also skips drawing while the canvas is scrolled out of view.- `resize()`: draws now at the canvas's current size. Only needed after resizing a paused player.- `snapshot()`: the current frame as a PNG data URL (with the mark).-- `dispose()`: stops, removes drivers, frees GPU memory. The canvas can take a new `Player` afterwards.+- `dispose()`: stops, removes drivers, frees GPU memory. A `load()` still running rejects, and a camera or microphone still+ starting (`startFaceTracking`, `startLipSync`) is let go as soon as it opens. The canvas can take a new `Player` afterwards.- `puppet`: the renderer underneath, for what this API does not cover. Its fields can change between versions.### The mark@@ -294,8 +300,10 @@ await vrm.load('/avatars/me.vrm') // a URL, a File, or files pickedcontrol. Without a bundler, map `three`, `three/examples/jsm/` and `@pixiv/three-vrm` in an import map, as`example/vrm.html` does.- **Your own**: anything with `load(files)`, `render(canvas, pose, w, h, dt)`, `head()` and `dispose()` (the `Body`- type). `pose` has the params after the idle motion and the drivers, the picture's `move` and each expression's weight.+ type), and if it cares which canvas it gets, `attach(canvas)`. `pose` has the params after the idle motion and the+ drivers, the picture's `move` and each expression's weight.+A failed `load()` keeps the character before, as on a Player. A canvas has one kind of context for good: a PNGTuber draws in 2D, a Player and a VRM in WebGL2, so a PNGTuber on a canvas that was a Player's (or the other way round) is refused with an error; give it its own.A BodyPlayer draws no mark and has no `puppet`; `view.head` is not used. Live2D models (`.moc3`) are not supported:reading them takes Live2D's proprietary Cubism Core, under its own licence.diff --git a/README.ja.md b/README.ja.mdindex 8fa23a5..8923b50 100644--- a/README.ja.md+++ b/README.ja.md@@ -14,7 +14,7 @@<p align="center"><img src="media/demo.gif" alt="厳選キャラクターが呼吸し、まばたきし、振り向き、話す様子" width="720"></p>-- WebGL2、依存なし、コアは gzip で約 34 KB+- WebGL2、依存なし、コアは gzip で約 30 KB- リップシンク(マイク、音声、動画)と Web カメラのフェイストラッキング(MediaPipe)はブラウザ内で動作し、必要なときだけ読み込めます- LivLiv・VRM・PNGTuber のキャラクターを同じ API で再生- [`characters/`](characters/) に厳選キャラクター 21 体。すぐに使えますdiff --git a/README.md b/README.mdindex 7d10040..a694aa8 100644--- a/README.md+++ b/README.md@@ -14,7 +14,7 @@<p align="center"><img src="media/demo.gif" alt="Featured characters breathing, blinking, turning their heads and talking" width="720"></p>-- WebGL2, no dependencies, about 34 KB gzipped for the core+- WebGL2, no dependencies, about 30 KB gzipped for the core- Lip sync (microphone, audio, video) and webcam face tracking (MediaPipe), both in the browser, imported only when you need them- Plays LivLiv, VRM and PNGTuber characters through the same API- 21 featured characters in [`characters/`](characters/), ready to usediff --git a/README.zh-CN.md b/README.zh-CN.mdindex 148f5ba..5bca6ca 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -16,7 +16,7 @@特点:-- WebGL2,无依赖,核心 gzip 后约 34 KB+- WebGL2,无依赖,核心 gzip 后约 30 KB- 口型同步(麦克风、音频、视频)和摄像头面捕(MediaPipe),都在浏览器里运行,按需引入- 支持 LivLiv、VRM 和 PNGTuber 三种角色,用同一套接口- [`characters/`](characters/) 里有 21 个精选角色,可以直接用diff --git a/example/camera.js b/example/camera.jsindex 5d29e4e..74e37d5 100644--- a/example/camera.js+++ b/example/camera.js@@ -7,6 +7,12 @@ const NOTES = {NotReadableError: 'The camera is busy in another app.',}+// The face model's files: on the examples' site they are served with the pages (../mediapipe/), else they come from+// MediaPipe's own hosts, the player's default. On your own site, host them and pass these two the same way.+const here = (p) => new URL(`../mediapipe/${p}`, location.href).href+const faceModelFiles = () => fetch(here('face_landmarker.task'), { method: 'HEAD' })+ .then((r) => (r.ok ? { wasmUrl: here('wasm'), modelUrl: here('face_landmarker.task') } : {}), () => ({}))+const style = document.createElement('style')style.textContent = `.camera-card { position: fixed; z-index: 2; box-sizing: border-box; width: min(320px, calc(100vw - 24px)); padding: 12px;@@ -56,6 +62,7 @@ export function cameraButton(button, player, status) {const { startFaceTracking } = await import('../dist/tracking.js')running = await startFaceTracking(player, {trace,+ ...(trace ? {} : await faceModelFiles()),onProgress: (s) => { if (asking) status.textContent = s.step === 'download' ? `Face model ${Math.round(s.progress * 100)}%` : `${s.step}…` },})asking = falsediff --git a/example/demo.html b/example/demo.htmlindex 91e18de..da7d3f9 100644--- a/example/demo.html+++ b/example/demo.html@@ -17,11 +17,13 @@<div class="bar" id="bar"></div><p id="status" role="status"></p>+<script>if (!/[?&]model=/.test(location.search)) document.head.append(Object.assign(document.createElement('link'), { rel: 'preload', as: 'fetch', crossOrigin: 'anonymous', href: '../characters/d03ft.livliv' }))</script><script type="module">import { Player } from '../dist/index.js'import { cameraButton } from './camera.js'- import { loading } from './loading.js'+ import { loading, needWebGL } from './loading.js'+ needWebGL(Player)const player = new Player(document.getElementById('stage'))const bar = document.getElementById('bar'), status = document.getElementById('status')window.player = player // to try things in the console@@ -33,7 +35,8 @@// a model: a featured character, ?model=<URL of a model.json or its folder>, or a folder picked belowasync function show(source) {status.textContent = ''- const done = loading(document.body, typeof source === 'string' && source.endsWith('/') ? `${source}avatar.webp` : null)+ // a featured character's small picture lies in its folder, beside its package+ const done = loading(document.body, typeof source === 'string' && /(\/|\.livliv)$/.test(source) ? `${source.replace(/\.livliv$/, '/')}avatar.webp` : null)try {const m = await player.load(source)status.textContent = `${m.name} · ${m.format} ${m.version} · mark: ${m.watermark ?? 'none'}${m.meta?.license ? ` · ${m.meta.license.id}` : ''}`@@ -72,6 +75,6 @@window.camera = cameraButton(button('camera'), player, status)}- show(new URLSearchParams(location.search).get('model') || '../characters/d03ft/')+ show(new URLSearchParams(location.search).get('model') || '../characters/d03ft.livliv')</script></html>diff --git a/example/gallery.html b/example/gallery.htmlindex f62455b..955b1f9 100644--- a/example/gallery.html+++ b/example/gallery.html@@ -25,9 +25,15 @@</div><ul class="cast" id="cast"></ul></main>+<script>+ // the first character's package starts downloading before the script below+ const first = new URLSearchParams(location.search).get('id')+ document.head.append(Object.assign(document.createElement('link'), { rel: 'preload', as: 'fetch', crossOrigin: 'anonymous',+ href: `../characters/${first && /^[\w-]+$/.test(first) ? first : 'd03ft'}.livliv` }))+</script><script type="module">import { Player } from '../dist/index.js'- import { loading } from './loading.js'+ import { loading, needWebGL } from './loading.js'// All the featured characters: one plays on the stage (one player, one WebGL context), the rest wait as their// pictures; a click on one puts it on the stage. ?id=<id> starts with that one (else Refined Nana); ?demo=1 plays expressions and@@ -38,6 +44,7 @@const rank = (id) => (FIRST.includes(id) ? FIRST.indexOf(id) : FIRST.length)const cast = list.sort((a, b) => rank(a.id) - rank(b.id)) // the rest keep the list's order+ needWebGL(Player)const player = new Player(document.getElementById('stage'), { maxPixelRatio: 1.5 })const bar = document.getElementById('bar'), name = document.getElementById('name'), ul = document.getElementById('cast')const buttons = new Map()@@ -55,10 +62,18 @@async function show(id) {if (id === current) return+ const before = currentcurrent = id- for (const [k, b] of buttons) b.setAttribute('aria-pressed', String(k === id))+ const press = (on) => { for (const [k, b] of buttons) b.setAttribute('aria-pressed', String(k === on)) }+ press(id)const done = loading(document.getElementById('view'), `../characters/${id}/avatar.webp`)- const m = await player.load(`../characters/${id}/`).finally(done)+ let m+ try {+ m = await player.load(`../characters/${id}.livliv`).finally(done)+ } catch (e) { // the one on the stage stays; a click can try this one again+ if (id === current) { current = before; press(before); name.textContent = e.message }+ return+ }if (id !== current) return // another was picked meanwhilename.textContent = m.nameexprs = m.expressions; next = -1diff --git a/example/index.html b/example/index.htmlindex 645fd7f..09e5b5e 100644--- a/example/index.html+++ b/example/index.html@@ -6,13 +6,16 @@<style>html, body { margin: 0; height: 100%; background: #20222b } canvas { display: block; width: 100%; height: 100% }</style><canvas id="stage"></canvas><p id="wait" style="position:fixed;inset:45% 0 auto;margin:0;color:#eef;font:14px system-ui,sans-serif;text-align:center">Loading…</p>+<link rel="preload" as="fetch" crossorigin href="../characters/d03ft.livliv"><script type="module">import { Player } from '../dist/index.js'// a featured character; your own export works the same+ const wait = document.getElementById('wait')+ if (!Player.isSupported()) { wait.textContent = 'This browser has no WebGL2'; throw new Error('no WebGL2') }const player = new Player(document.getElementById('stage'))- const model = await player.load('../characters/d03ft/')- document.getElementById('wait').remove()+ const model = await player.load('../characters/d03ft.livliv').catch((e) => { wait.textContent = e.message; throw e })+ wait.remove()// each click shows the next expression, then neutral againlet i = -1diff --git a/example/loading.js b/example/loading.jsindex 7ffec32..0ff7104 100644--- a/example/loading.js+++ b/example/loading.js@@ -15,3 +15,13 @@ export function loading(parent, picture) {const later = setTimeout(() => parent.append(box), 150)return () => { clearTimeout(later); box.remove() }}++/** Where WebGL2 is missing: says so on the page and stops the page's script here (new Player() would throw). */+export function needWebGL(Player) {+ if (Player.isSupported()) return+ const note = document.createElement('p')+ note.style.cssText = 'margin:32px 16px;text-align:center;font:15px/1.5 system-ui,sans-serif'+ note.textContent = 'This browser cannot show the character: it needs WebGL2, which is missing or turned off here.'+ document.body.append(note)+ throw new Error('no WebGL2')+}diff --git a/package.json b/package.jsonindex 4753af4..4d4351e 100644--- a/package.json+++ b/package.json@@ -39,6 +39,7 @@"module": "./dist/index.js","types": "./dist/types/src/index.d.ts","exports": {+ "./package.json": "./package.json",".": {"types": "./dist/types/src/index.d.ts","import": "./dist/index.js"@@ -90,7 +91,8 @@"vite": "^8.3.4"},"scripts": {- "build": "rm -rf dist && vite build && tsc -p tsconfig.build.json",+ "build": "rm -rf dist && vite build && tsc -p tsconfig.build.json && node scripts/dts-extensions.mjs && node scripts/pack-characters.mjs",+ "test:types": "node test/types.mjs","check": "tsc -p tsconfig.json","test": "node --disable-warning=ExperimentalWarning --import ./test/ts.mjs --test \"test/*.test.ts\"","test:browser": "node test/smoke.mjs"diff --git a/scripts/dts-extensions.mjs b/scripts/dts-extensions.mjsnew file mode 100644index 0000000..e92b63a--- /dev/null+++ b/scripts/dts-extensions.mjs@@ -0,0 +1,21 @@+#!/usr/bin/env node+// The declarations as tsc writes them keep the source's relative imports, without extensions: fine for a bundler, but under+// moduleResolution node16 / nodenext (a package with "type": "module") each is an error or, with skipLibCheck, silently+// `any`. This gives them the extension the files have at run time, after `tsc -p tsconfig.build.json`.+import fs from 'node:fs'+import path from 'node:path'++const root = path.resolve(process.argv[2] || 'dist/types')+const IMPORT = /((?:from\s+|import\s*\(\s*|^\s*import\s+)['"])(\.\.?\/[^'"]+?)(['"])/gm+let changed = 0+const walk = (d) => fs.readdirSync(d, { withFileTypes: true }).flatMap((e) => (e.isDirectory() ? walk(path.join(d, e.name)) : [path.join(d, e.name)]))+for (const f of walk(root).filter((f) => f.endsWith('.d.ts'))) {+ const text = fs.readFileSync(f, 'utf8')+ const out = text.replace(IMPORT, (m, a, spec, z) => {+ if (!fs.existsSync(path.resolve(path.dirname(f), `${spec}.d.ts`))) throw new Error(`${path.relative(root, f)} imports ${spec}: no such declaration`)+ changed+++ return `${a}${spec}.js${z}`+ })+ if (out !== text) fs.writeFileSync(f, out)+}+console.log(`declarations: ${changed} relative imports now say .js`)diff --git a/scripts/pack-characters.mjs b/scripts/pack-characters.mjsnew file mode 100644index 0000000..40e8ff1--- /dev/null+++ b/scripts/pack-characters.mjs@@ -0,0 +1,46 @@+#!/usr/bin/env node+// Each featured character as one file, characters/<id>.livliv, next to its folder (FORMAT.md §1: model.json first, images+// and model.json stored, keyforms.json deflated). The examples load these: one request instead of one per image, which on+// a slow path saves more than half the wait. Written by `npm run build`, not kept in git: they would only double what the+// repository holds. The bytes depend on the files alone (no dates), so a build changes nothing it need not.+import fs from 'node:fs'+import path from 'node:path'+import { fileURLToPath } from 'node:url'+import { crc32, deflateRawSync } from 'node:zlib'++const root = fileURLToPath(new URL('../characters/', import.meta.url))++function zip(dir) {+ const list = (at = '') => fs.readdirSync(path.join(dir, at), { withFileTypes: true }).flatMap((e) => {+ const p = at ? `${at}/${e.name}` : e.name+ return e.isDirectory() ? list(p) : [p]+ }).sort()+ const names = ['model.json', ...list().filter((f) => f !== 'model.json')]+ const local = [], central = []+ let at = 0+ for (const name of names) {+ const raw = fs.readFileSync(path.join(dir, name)), deflate = name === 'keyforms.json'+ const data = deflate ? deflateRawSync(raw, { level: 9 }) : raw, file = Buffer.from(name), crc = crc32(raw)+ const method = deflate ? 8 : 0+ const h = Buffer.alloc(30)+ h.writeUInt32LE(0x04034b50, 0); h.writeUInt16LE(20, 4); h.writeUInt16LE(0x0800, 6); h.writeUInt16LE(method, 8); h.writeUInt16LE(0x21, 12)+ h.writeUInt32LE(crc, 14); h.writeUInt32LE(data.length, 18); h.writeUInt32LE(raw.length, 22); h.writeUInt16LE(file.length, 26)+ const c = Buffer.alloc(46)+ c.writeUInt32LE(0x02014b50, 0); c.writeUInt16LE(20, 4); c.writeUInt16LE(20, 6); c.writeUInt16LE(0x0800, 8); c.writeUInt16LE(method, 10); c.writeUInt16LE(0x21, 14)+ c.writeUInt32LE(crc, 16); c.writeUInt32LE(data.length, 20); c.writeUInt32LE(raw.length, 24); c.writeUInt16LE(file.length, 28); c.writeUInt32LE(at, 42)+ local.push(h, file, data); central.push(c, file)+ at += 30 + file.length + data.length+ }+ const cd = Buffer.concat(central), end = Buffer.alloc(22)+ end.writeUInt32LE(0x06054b50, 0); end.writeUInt16LE(names.length, 8); end.writeUInt16LE(names.length, 10); end.writeUInt32LE(cd.length, 12); end.writeUInt32LE(at, 16)+ return Buffer.concat([...local, cd, end])+}++let n = 0, bytes = 0+for (const e of fs.existsSync(root) ? fs.readdirSync(root, { withFileTypes: true }) : []) {+ if (!e.isDirectory()) continue+ const b = zip(path.join(root, e.name))+ fs.writeFileSync(path.join(root, `${e.name}.livliv`), b)+ n++; bytes += b.length+}+console.log(`characters: ${n} packages, ${(bytes / 1e6).toFixed(1)} MB`)diff --git a/src/body.ts b/src/body.tsindex 0ec9383..7f45526 100644--- a/src/body.ts+++ b/src/body.ts@@ -5,7 +5,7 @@// the idle motion, the drivers, actions, expressions); the Body turns it into its own picture.import { ActionPlayer, type ActionId } from './runtime/actions'import { pickLabel, uiLang, type Labels } from './runtime/format'-import { PARAM_RANGES, defaultParams, type Params } from './runtime/puppet'+import { PARAM_RANGES, defaultParams, type Params } from './runtime/params'import { Idle, type Seen } from './follow'import type { Driver, Frame, Host, View } from './index'@@ -19,7 +19,11 @@ export interface Pose { params: Params; move: { x: number; y: number; s: numberexport interface Body {/** what kind of character it draws ('png', 'vrm', ...): BodyInfo.format */readonly format: string- /** loads from its files (by file name); its name and the expressions it can show */+ /** optional: called before load with the canvas it will draw on; throw if it cannot draw there (the canvas already has+ * another kind of context), and load() rejects with that */+ attach?(canvas: HTMLCanvasElement): void+ /** loads from its files (by file name); its name and the expressions it can show. If it rejects, the character before+ * must stay as it is */load(files: Map<string, Blob>): Promise<{ name: string; expressions: BodyExpression[] }>/** draws the pose into the canvas, w × h device pixels, dt seconds after the last frame */render(canvas: HTMLCanvasElement, pose: Pose, w: number, h: number, dt: number): void@@ -121,11 +125,18 @@ export class BodyPlayer implements Host {get expressions() { return this.info?.expressions ?? [] }get playing() { return this.raf !== 0 }- /** Loads a character from its files (see BodySource) in place of the one before. Loads run one after another */+ /** Loads a character from its files (see BodySource) in place of the one before (a failed load keeps it). Loads run one+ * after another */load(source: BodySource): Promise<BodyInfo> {const run = this.queue.then(async () => {- if (this.disposed) throw new Error('LivLiv: the player was disposed')- const got = await this.body.load(await filesOf(source))+ const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }+ stillHere()+ this.body.attach?.(this.canvas)+ const files = await filesOf(source)+ stillHere()+ const got = await this.body.load(files)+ if (this.disposed) this.body.dispose() // disposed while it loaded: nothing of it may stay+ stillHere()const ex = [{ id: 'smile', label: '', labels: SMILE }, ...got.expressions.filter((e) => e.id !== 'smile').map((e) => ({ id: e.id, label: e.label, labels: e.labels ?? { en: e.label } }))]this.info = { name: got.name, format: this.body.format,@@ -154,6 +165,7 @@ export class BodyPlayer implements Host {blink() { this.idler.blink(this.time) }use(driver: Driver): () => void {+ if (this.disposed) { driver.dispose?.(); return () => {} } // as Player's: what started meanwhile does not stay onthis.drivers.push(driver)this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))return () => {diff --git a/src/follow.ts b/src/follow.tsindex 4f2cb44..a3ba21a 100644--- a/src/follow.ts+++ b/src/follow.ts@@ -3,7 +3,8 @@// character looks back at the viewer when it leaves. What a player tells it each frame (talking or acting, a face tracker// driving, an expression showing) and where the pointer is, in one place for Player and BodyPlayer.import { LivelyIdle, pointAt } from './runtime/idle'-import { defaultParams, type Params, type Puppet } from './runtime/puppet'+import { defaultParams, type Params } from './runtime/params'+import type { Puppet } from './runtime/puppet'/** the keys the idle motion sets (with it on, your values for these are not used); its smile is added to yours */export const IDLE_KEYS = ['angleX', 'angleY', 'angleZ', 'eyeLOpen', 'eyeROpen', 'eyeBallX', 'eyeBallY', 'browLY', 'browRY', 'bodyX', 'bodyZ', 'breath'] as constdiff --git a/src/index.ts b/src/index.tsindex cb11467..baa5204 100644--- a/src/index.ts+++ b/src/index.ts@@ -96,6 +96,9 @@ export interface Host {}+/** what the runtime throws when a canvas gives it no WebGL2 context */+const NO_WEBGL2 = 'this browser has no WebGL2'+/** Plays one LivLiv model on a canvas. Size the canvas with CSS; the drawing follows its size. */export class Player implements Host {readonly canvas: HTMLCanvasElement@@ -128,13 +131,21 @@ export class Player implements Host {private disposed = falseprivate frame: Frame & { move: { x: number; y: number; s: number } }+ /** Whether this browser can play a LivLiv model (WebGL2 starts on a new canvas). Where it cannot, new Player() throws */+ static isSupported(): boolean {+ const gl = document.createElement('canvas').getContext('webgl2')+ gl?.getExtension('WEBGL_lose_context')?.loseContext() // the try takes none of the contexts a page may have at once+ return !!gl+ }+constructor(canvas: HTMLCanvasElement, options: PlayerOptions = {}) {this.canvas = canvastry {this.puppet = new Puppet(canvas)} catch (e) {throw new Error(typeof WebGL2RenderingContext === 'undefined' ? 'LivLiv: this browser has no WebGL2'- : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`)+ : (e as Error).message === NO_WEBGL2 ? 'LivLiv: could not start WebGL2 on this canvas: it is turned off in this browser, or the canvas already draws in 2D'+ : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`)}this.puppet.maxDpr = options.maxPixelRatio ?? 2this.mark = new Mark(this.puppet.gl)@@ -172,13 +183,13 @@ export class Player implements Host {}private async loadNow(source: ModelSource): Promise<ModelInfo> {- if (this.disposed) throw new Error('LivLiv: the player was disposed')+ const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }+ stillHere()const p = this.puppetlet files: Exclude<ModelSource, string | URL> | null = nullif (typeof source === 'string' || source instanceof URL) {const url = new URL(String(source), document.baseURI)- if (/\.(livliv|zip)$/i.test(url.pathname)) files = await fetchPackage(url.href)- else {+ if (/\.(livliv|zip)$/i.test(url.pathname)) { files = await fetchPackage(url.href); stillHere() } else {try {await p.load(url.href, { pack: false })} catch (e) {@@ -188,8 +199,12 @@ export class Player implements Host {} else files = sourceif (files) {const { model, image, keyforms } = await fromFiles(files)+ stillHere()await p.loadModel(model, image, { keyforms }).catch((e) => { throw explain(e, 'model.json') })}+ // disposed while the images came in: the runtime built its layers all the same; nothing of them may stay or be drawn+ if (this.disposed) p.dispose()+ stillHere()const m = p.modelconst ex = m.expressions || (m.layers.some((l) => l.id === 'expr_smile') ? [{ id: 'smile', label: '', labels: SMILE }] : [])this.info = {@@ -231,8 +246,10 @@ export class Player implements Host {/** Blinks now (with the idle motion on) */blink() { this.idler.blink(this.puppet.time) }- /** Adds a driver; returns a function that removes it */+ /** Adds a driver; returns a function that removes it. On a disposed player it is disposed at once: what started+ * while the player went (a camera, a microphone) does not stay on */use(driver: Driver): () => void {+ if (this.disposed) { driver.dispose?.(); return () => {} }this.drivers.push(driver)this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))return () => {@@ -262,7 +279,8 @@ export class Player implements Host {/** The current frame as an image (PNG data URL), the mark included */snapshot(): string { return this.canvas.toDataURL('image/png') }- /** Stops, removes the drivers and frees the GPU memory. The canvas can take a new Player afterwards */+ /** Stops, removes the drivers and frees the GPU memory. A load still running rejects. The canvas can take a new Player+ * afterwards */dispose() {if (this.disposed) returnthis.pause()@@ -279,7 +297,7 @@ export class Player implements Host {private draw(dt: number) {const p = this.puppet, P = p.params, f = this.frame- if (!p.model) return+ if (!p.model || this.disposed) return// the pose from scratch: your values, the idle motion, the driversthis.idler.shown(P)Object.assign(P, this.params)diff --git a/src/png.ts b/src/png.tsindex 5fd6aca..2dc9f8c 100644--- a/src/png.ts+++ b/src/png.ts@@ -20,27 +20,39 @@ export class PngBody implements Body {private pics: Partial<Record<PngSlot, ImageBitmap>> = {}private g: CanvasRenderingContext2D | null = null+ attach(canvas: HTMLCanvasElement) {+ this.g = canvas.getContext('2d')+ if (!this.g) throw new Error('LivLiv: this canvas already draws with WebGL (a Player or a VRM): a PNGTuber needs a canvas of its own')+ }+async load(files: Map<string, Blob>) {- this.dispose()- let pixels = 0- for (const [name, blob] of files) {- const slot = PNG_SLOTS.find((s) => name.replace(/\.[^.]*$/, '').toLowerCase() === s.id)?.id- if (!slot) continue- const b = await createImageBitmap(blob).catch(() => { throw new Error(`LivLiv: ${name} is not an image this browser can read`) })- if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) {- b.close()- throw new Error(`LivLiv: ${name} is too large (${b.width} x ${b.height})`)+ // the new pictures apart from the ones shown: anything wrong with them leaves those on the canvas+ const pics: typeof this.pics = {}+ let name = 'PNGTuber'+ try {+ let pixels = 0+ for (const [file, blob] of files) {+ const slot = PNG_SLOTS.find((s) => file.replace(/\.[^.]*$/, '').toLowerCase() === s.id)?.id+ if (!slot) continue+ const b = await createImageBitmap(blob).catch(() => { throw new Error(`LivLiv: ${file} is not an image this browser can read`) })+ pics[slot] = b+ if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) throw new Error(`LivLiv: ${file} is too large (${b.width} x ${b.height})`)}- this.pics[slot] = b+ if (!pics.closed || !pics.open) throw new Error('LivLiv: a PNGTuber needs at least the closed and the open picture')+ const txt = files.get('name.txt')+ if (txt) name = (await txt.slice(0, 200).text()).trim() || name+ } catch (e) {+ for (const b of Object.values(pics)) b?.close()+ throw e}- if (!this.pics.closed || !this.pics.open) throw new Error('LivLiv: a PNGTuber needs at least the closed and the open picture')- const name = files.get('name.txt')- return { name: name ? (await name.slice(0, 200).text()).trim() || 'PNGTuber' : 'PNGTuber', expressions: [] }+ this.dispose()+ this.pics = pics+ return { name, expressions: [] }}render(canvas: HTMLCanvasElement, { params: P, move }: Pose, w: number, h: number) {if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h }- const g = (this.g ??= canvas.getContext('2d'))+ const g = (this.g ??= canvas.getContext('2d')) // attach() has told the page if it cannot have oneif (!g) returng.clearRect(0, 0, w, h)const x = this.picsdiff --git a/src/runtime/actions.ts b/src/runtime/actions.tsindex e038282..82c79f6 100644--- a/src/runtime/actions.ts+++ b/src/runtime/actions.ts@@ -2,7 +2,7 @@// one-shot animation over tracking. Each is a curve of parameter offsets that fades in and out (~0.2 s), so starting// one mid-move or pressing it again never jumps. "away" is a state, not a move: eyes closed, head down, slow breath,// until it is turned off (or, entered because no face was seen for a while, until a face is back for a second).-import { smooth, type Params } from './puppet'+import { smooth, type Params } from './params'export type ActionId = 'nod' | 'shake' | 'tilt' | 'hop' | 'shy' | 'away'diff --git a/src/runtime/params.ts b/src/runtime/params.tsnew file mode 100644index 0000000..5d029aa--- /dev/null+++ b/src/runtime/params.ts@@ -0,0 +1,26 @@+// The pose and face a character takes, on their own: what the drivers, the idle motion and the bodies that are not a+// puppet (a PNGTuber, a VRM) work with, so that none of them has to load the renderer for these. puppet.ts exports them too.++export type Params = {+ angleX: number; angleY: number; angleZ: number+ eyeLOpen: number; eyeROpen: number; eyeBallX: number; eyeBallY: number+ browLY: number; browRY: number+ mouthOpen: number; smile: number+ /** the voice's vowel (vowel.ts): -1 round (o, u) .. +1 wide (i, e), 0 neither */+ vowel: number+ bodyX: number; bodyZ: number; breath: number+}+export const defaultParams = (): Params => ({+ angleX: 0, angleY: 0, angleZ: 0, eyeLOpen: 1, eyeROpen: 1, eyeBallX: 0, eyeBallY: 0,+ browLY: 0, browRY: 0, mouthOpen: 0, smile: 0, vowel: 0, bodyX: 0, bodyZ: 0, breath: 0,+})+export const PARAM_RANGES: Record<keyof Params, [number, number]> = {+ angleX: [-30, 30], angleY: [-30, 30], angleZ: [-30, 30], eyeLOpen: [0, 1], eyeROpen: [0, 1],+ eyeBallX: [-1, 1], eyeBallY: [-1, 1], browLY: [-1, 1], browRY: [-1, 1],+ mouthOpen: [0, 1], smile: [0, 1], vowel: [-1, 1], bodyX: [-10, 10], bodyZ: [-10, 10], breath: [0, 1],+}++export const smooth = (a: number, b: number, x: number) => {+ const t = Math.min(1, Math.max(0, (x - a) / (b - a)))+ return t * t * (3 - 2 * t)+}diff --git a/src/runtime/puppet.ts b/src/runtime/puppet.tsindex 030a2f1..ef1d580 100644--- a/src/runtime/puppet.ts+++ b/src/runtime/puppet.ts@@ -17,6 +17,9 @@// the bangs where it lies over them// - body: lags the head turn / tilt, breathes (shoulders + chest), optional chest springimport { LIMITS, readModel, type Labels, type ModelFiles, type ModelMeta } from './format'+import { PARAM_RANGES, defaultParams, smooth, type Params } from './params'++export { PARAM_RANGES, defaultParams, smooth, type Params }export interface LayerDef {id: string; name: string; texture: string@@ -189,24 +192,6 @@ export const defaultRig = (): Rig => ({* each take its place when set, so a model saved with hair alone moves both as before */export type Tuning = { turn: number; nod: number; hair: number; blink: number; expr: number; mouth: number; breath: number; hairFront?: number; hairBack?: number }export const defaultTuning = (): Tuning => ({ turn: 1, nod: 1, hair: 1, blink: 1, expr: 1, mouth: 1, breath: 1 })-export type Params = {- angleX: number; angleY: number; angleZ: number- eyeLOpen: number; eyeROpen: number; eyeBallX: number; eyeBallY: number- browLY: number; browRY: number- mouthOpen: number; smile: number- /** the voice's vowel (vowel.ts): -1 round (o, u) .. +1 wide (i, e), 0 neither */- vowel: number- bodyX: number; bodyZ: number; breath: number-}-export const defaultParams = (): Params => ({- angleX: 0, angleY: 0, angleZ: 0, eyeLOpen: 1, eyeROpen: 1, eyeBallX: 0, eyeBallY: 0,- browLY: 0, browRY: 0, mouthOpen: 0, smile: 0, vowel: 0, bodyX: 0, bodyZ: 0, breath: 0,-})-export const PARAM_RANGES: Record<keyof Params, [number, number]> = {- angleX: [-30, 30], angleY: [-30, 30], angleZ: [-30, 30], eyeLOpen: [0, 1], eyeROpen: [0, 1],- eyeBallX: [-1, 1], eyeBallY: [-1, 1], browLY: [-1, 1], browRY: [-1, 1],- mouthOpen: [0, 1], smile: [0, 1], vowel: [-1, 1], bodyX: [-10, 10], bodyZ: [-10, 10], breath: [0, 1],-}interface GLLayer {def: LayerDef@@ -296,11 +281,6 @@ const VOWEL_IN = 0.65, VOWEL_OUT = 0.35/** how long after the last open frame the character still counts as talking (s): the gaps between syllables */const TALK_LATCH = 0.4-export const smooth = (a: number, b: number, x: number) => {- const t = Math.min(1, Math.max(0, (x - a) / (b - a)))- return t * t * (3 - 2 * t)-}-/** cap (optional): per sample the most the brow may move there (model px), where it lies under opaque bangs */export type BrowLine = { x: number[]; bottom: number[]; lid: number[]; cap?: number[] }const interp = (xs: number[], ys: number[], x: number) => {diff --git a/src/runtime/talk.ts b/src/runtime/talk.tsindex 5d8bd33..817f278 100644--- a/src/runtime/talk.ts+++ b/src/runtime/talk.ts@@ -5,7 +5,7 @@// - loud onsets now and then lift the brows; the eyes come back to the viewer// - a pause after talking brings a blink// - bounce (optional): the whole character hops a little on syllables, as PNGTubers do-import { smooth, type Params } from './puppet'+import { smooth, type Params } from './params'/** a syllable: the mouth opens by RISE above the last low point, to at least ONSET (within a phrase the voice rarely* closes it between syllables, so a fixed threshold misses most); the next one counts once it has come down by FALL.diff --git a/src/runtime/tracking.ts b/src/runtime/tracking.tsindex 96c074a..137fe14 100644--- a/src/runtime/tracking.ts+++ b/src/runtime/tracking.ts@@ -1,5 +1,5 @@// Webcam face tracking -> puppet params, via MediaPipe Face Landmarker (runs locally in the browser).-import { smooth, type Params } from './puppet'+import { smooth, type Params } from './params'import { OneEuro } from './filters'import { BS, LM, SynthStats, TraceRecorder, TraceSource, synthTrace, type Frame } from './trace'diff --git a/src/tracking.ts b/src/tracking.tsindex e38dcac..fb20d57 100644--- a/src/tracking.ts+++ b/src/tracking.ts@@ -3,7 +3,7 @@// the @mediapipe/tasks-vision package and downloads ~8 MB of model the first time (the wasm runtime and the face model;// cached by the browser after that).import { FaceTracker, LOST_AFTER, V, loadFaceModel, setFaceModelFiles, type StartStep } from './runtime/tracking'-import { defaultParams, type Params } from './runtime/puppet'+import { defaultParams, type Params } from './runtime/params'import type { Driver, Host } from './index'export type { StartStep }diff --git a/src/vrm.ts b/src/vrm.tsindex 8b2c7a8..bf8ba70 100644--- a/src/vrm.ts+++ b/src/vrm.ts@@ -32,6 +32,13 @@ export class VrmBody implements Body {private canvas: HTMLCanvasElement | null = nullprivate headAt = { x: 0.5, y: 0.25 }+ attach(canvas: HTMLCanvasElement) {+ // the attributes render() gives three.js, which then finds this same context+ if (!canvas.getContext('webgl2', { alpha: true, antialias: true, premultipliedAlpha: true, preserveDrawingBuffer: true })) {+ throw new Error('LivLiv: a VRM needs WebGL2: this browser has none, or this canvas already draws in 2D (a PNGTuber)')+ }+ }+async load(files: Map<string, Blob>) {const entry = [...files].find(([n]) => /\.(vrm|glb)$/i.test(n))if (!entry) throw new Error('LivLiv: no .vrm file among the files')@@ -40,7 +47,6 @@ export class VrmBody implements Body {throw new Error(`LivLiv: a VRM needs the three and @pixiv/three-vrm packages (${(e as Error).message})`)})this.THREE = THREE- this.dispose()const loader = new GLTFLoader()loader.register((parser) => new VRMLoaderPlugin(parser))const url = URL.createObjectURL(entry[1])@@ -53,14 +59,18 @@ export class VrmBody implements Body {URL.revokeObjectURL(url)}const vrm = gltf.userData.vrm as VRM | undefined- if (!vrm) throw new Error(`LivLiv: ${entry[0]} is a glTF without VRM data`)+ if (!vrm) {+ VRMUtils.deepDispose(gltf.scene)+ throw new Error(`LivLiv: ${entry[0]} is a glTF without VRM data`)+ }+ // the new character, apart from the one shown (a failed load leaves that on the canvas): it takes over at the endVRMUtils.removeUnnecessaryVertices(gltf.scene)VRMUtils.combineSkeletons(gltf.scene)VRMUtils.rotateVRM0(vrm) // VRM 0.x faces the other wayvrm.scene.traverse((o) => { o.frustumCulled = false })- this.vrm = vrmconst h = vrm.humanoid- for (const b of ['head', 'neck', 'spine', 'chest'] as const) this.bones[b] = h.getNormalizedBoneNode(b) ?? undefined+ const bones: typeof this.bones = {}+ for (const b of ['head', 'neck', 'spine', 'chest'] as const) bones[b] = h.getNormalizedBoneNode(b) ?? undefined// arms down from the T pose, a little forwardconst lu = h.getNormalizedBoneNode('leftUpperArm'), ru = h.getNormalizedBoneNode('rightUpperArm')if (lu) lu.rotation.set(0, 0, -1.2)@@ -71,24 +81,31 @@ export class VrmBody implements Body {if (vrm.lookAt) vrm.lookAt.autoUpdate = falsevrm.update(0)- this.scene = new THREE.Scene()- this.scene.add(vrm.scene)+ const scene = new THREE.Scene()+ scene.add(vrm.scene)const light = new THREE.DirectionalLight(0xffffff, Math.PI)light.position.set(1, 1, 1).normalize()- this.scene.add(light, new THREE.AmbientLight(0xffffff, 0.6))+ scene.add(light, new THREE.AmbientLight(0xffffff, 0.6))// the head's top to the hips: what a LivLiv character's canvas shows of a personconst at = (b: 'head' | 'hips') => { const v = new THREE.Vector3(); h.getNormalizedBoneNode(b)?.getWorldPosition(v); return v }const head = at('head'), hips = at('hips')const top = head.y + 0.2, bottom = hips.y+ const has = new Set((vrm.expressionManager?.expressions ?? []).map((e) => e.expressionName))+ const meta = vrm.meta as { name?: string; title?: string } | undefined+ const expressions: BodyExpression[] = FEELINGS.filter((f) => f !== 'happy' && has.has(f)).map((f) => ({ id: f, label: LABELS[f].en, labels: LABELS[f] }))+ if (has.has('happy')) expressions.unshift({ id: 'happy', label: LABELS.happy.en, labels: LABELS.happy })+ const name = (meta?.name || meta?.title || entry[0].replace(/\.(vrm|glb)$/i, '')).slice(0, 200)++ const old = this.vrm+ this.vrm = vrm+ this.scene = scene+ this.bones = bones+ this.has = hasthis.frame = { height: Math.max(0.1, top - bottom), y: (top + bottom) / 2 }this.camera = new THREE.PerspectiveCamera(20, 2 / 3, 0.05, 30)- this.place(2 / 3)-- this.has = new Set((vrm.expressionManager?.expressions ?? []).map((e) => e.expressionName))- const meta = vrm.meta as { name?: string; title?: string } | undefined- const expressions: BodyExpression[] = FEELINGS.filter((f) => f !== 'happy' && this.has.has(f)).map((f) => ({ id: f, label: LABELS[f].en, labels: LABELS[f] }))- if (this.has.has('happy')) expressions.unshift({ id: 'happy', label: LABELS.happy.en, labels: LABELS.happy })- return { name: (meta?.name || meta?.title || entry[0].replace(/\.(vrm|glb)$/i, '')).slice(0, 200), expressions }+ this.place(2 / 3) // render() gives it the canvas's aspect+ if (old) VRMUtils.deepDispose(old.scene)+ return { name, expressions }}/** the camera where the framing fits the canvas's aspect */@@ -111,7 +128,8 @@ export class VrmBody implements Body {this.renderer.setClearColor(0x000000, 0)this.canvas = canvas}- if (canvas.width !== w || canvas.height !== h) { this.renderer.setSize(w, h, false); this.place(w / h) }+ if (canvas.width !== w || canvas.height !== h) this.renderer.setSize(w, h, false)+ if (this.camera.aspect !== w / h) this.place(w / h) // a new character's camera, on a canvas already this size tooconst b = this.bones// the head turns most, the neck some (yaw: to the screen's right; pitch: up; roll: toward the screen's left)b.neck?.rotation.set(-P.angleY * D * 0.35, P.angleX * D * 0.35, -P.angleZ * D * 0.35)diff --git a/test/bodies.html b/test/bodies.htmlindex db09cdd..37e6bd2 100644--- a/test/bodies.html+++ b/test/bodies.html@@ -66,6 +66,19 @@tracking.stop(); speech.dispose()out.png = { name: m.name, format: m.format, expressions: m.expressions.map((e) => e.id), shown, ...seen }p.dispose()+ // a failed load leaves the pictures shown, and the name; on a canvas with WebGL it says so+ const canvas2 = document.createElement('canvas')+ canvas2.style.cssText = 'width: 100px; height: 120px'+ document.body.append(canvas2)+ const q = new BodyPlayer(canvas2, new PngBody(), { autoplay: false, idle: false })+ await q.load({ closed: await solid('#ff0000'), open: await solid('#00ff00'), 'name.txt': new Blob(['First']) })+ const refused = await q.load({ closed: await solid('#0000ff') }).then(() => 'loaded', (e) => e.message)+ q.resize()+ out.pngFailed = { refused, kept: q.model?.name, pixel: pixel(canvas2, canvas2.width / 2, canvas2.height * 0.8) }+ q.dispose(); canvas2.remove()+ const gl = document.createElement('canvas')+ gl.getContext('webgl2')+ out.pngOnGl = await new BodyPlayer(gl, new PngBody(), { autoplay: false }).load({ closed: await solid('#ff0000'), open: await solid('#00ff00') }).then(() => 'loaded', (e) => e.message)} catch (e) { out.png = { error: String(e.message || e) } }try {const canvas = document.getElementById('vrm')@@ -80,6 +93,24 @@const tracking = await startFaceTracking(p, { trace: 'synth' })await wait(800)tracking.stop()+ // a second load, on a canvas that is not 2:3 and already has its size, and a failed one+ const wide = document.createElement('canvas')+ wide.style.cssText = 'width: 400px; height: 200px'+ document.body.append(wide)+ wide.width = 400; wide.height = 200+ const w = new BodyPlayer(wide, new VrmBody(), { autoplay: false, idle: false })+ await w.load('/__tiny.vrm'); w.resize()+ const aspect1 = w.body.camera.aspect+ await w.load('/__tiny.vrm'); w.resize()+ const aspect2 = w.body.camera.aspect+ const bad = new File([new Blob(['not a glb'])], 'x.vrm')+ const refused = await w.load(bad).then(() => 'loaded', (e) => e.message)+ w.resize()+ out.vrmAgain = { aspect1, aspect2, refused, kept: w.model?.name, share: covered(wide) }+ w.dispose(); wide.remove()+ const flat = document.createElement('canvas')+ flat.getContext('2d')+ out.vrmOn2d = await new BodyPlayer(flat, new VrmBody(), { autoplay: false }).load('/__tiny.vrm').then(() => 'loaded', (e) => e.message)const head = p.body.head()out.vrm = { name: m.name, format: m.format, expressions: m.expressions.map((e) => e.label), share, happy, smile, unknown, tracked: seen.tracked,head: [head.x, head.y].every(Number.isFinite) }diff --git a/test/smoke.html b/test/smoke.htmlindex 00149d3..1a4c75a 100644--- a/test/smoke.html+++ b/test/smoke.html@@ -8,6 +8,7 @@// and action without a WebGL error, and shows its mark; one loads from files instead of a URL, and again as the v3// package of the same model (it must draw the same pixels); a model of a newer format is refused.import { FORMAT_VERSION, LEGACY_FORMAT, Player } from '../dist/index.js'+ import { startLipSync } from '../dist/voice.js'const canvas = document.getElementById('c')const g = document.createElement('canvas').getContext('2d', { willReadFrequently: true })@@ -59,9 +60,9 @@const p = new Player(canvas, { autoplay: false, followPointer: false, idle: false })const info = await p.load(source)p.resize()- const d = pixels(), drawn = JSON.parse(JSON.stringify(p.puppet.model))+ const d = pixels(), drawn = JSON.parse(JSON.stringify(p.puppet.model)), keys = p.puppet.keys.lengthp.dispose()- return { d, info, drawn }+ return { d, info, drawn, keys }}/** what the model's mark changes in a still frame (drawn as it comes, and with the mark off), how much of that lies@@ -110,6 +111,13 @@fromFiles = { error: String(e.message || e) }}+ // the same characters as one file from a URL (made by the build): the same frame, keyforms and all+ const packaged = []+ for (const p of ids.filter((i) => i === id || FILES[i].includes('keyforms.json'))) {+ const a = await still(`../characters/${p}/`), b = await still(`../characters/${p}.livliv`)+ packaged.push({ id: p, changed: changed(a.d, b.d), keys: [a.keys, b.keys] })+ }+// a newer format: refused, and the model before stayslet newertry {@@ -134,7 +142,54 @@} catch (e) {legacy = { error: String(e.message || e) }}- return { out, fromFiles, newer, legacy }+ return { out, fromFiles, newer, legacy, packaged }+ }++ /** disposing while something runs: a load rejects and leaves nothing on the GPU or the canvas; a driver added to a+ * disposed player is disposed; a microphone that opens after it is let go */+ window.runLifecycle = async (id) => {+ const out = { supported: Player.isSupported() }+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 200px; height: 200px'+ document.body.append(c)+ {+ const p = new Player(c, { followPointer: false }), gl = p.puppet.gl+ let made = 0, freed = 0+ const { createTexture, deleteTexture } = gl+ gl.createTexture = function () { made++; return createTexture.call(gl) }+ gl.deleteTexture = function (t) { if (t) freed++; return deleteTexture.call(gl, t) }+ const load = p.load(`../characters/${id}/`).then(() => 'resolved', (e) => e.message)+ await new Promise((r) => setTimeout(r, 5))+ p.dispose()+ out.load = await load+ p.resize()+ await new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)))+ const px = new Uint8Array(4)+ gl.readPixels(100, 100, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, px)+ Object.assign(out, { model: p.model, drawn: px[3], made, freed, glError: gl.getError() })+ }+ {+ const p = new Player(c, { followPointer: false })+ p.dispose()+ let disposed = 0+ const remove = p.use({ update() {}, dispose() { disposed++ } })+ remove()+ out.driver = disposed+ const again = new Player(c, { followPointer: false })+ await again.load(`../characters/${id}/`)+ again.resize()+ out.again = again.model?.name ?? null+ const real = navigator.mediaDevices.getUserMedia.bind(navigator.mediaDevices)+ let stream+ navigator.mediaDevices.getUserMedia = async (k) => { stream = await real(k); return stream }+ const starting = startLipSync(again)+ again.dispose()+ await starting+ navigator.mediaDevices.getUserMedia = real+ out.mic = stream.getTracks()[0].readyState+ }+ c.remove()+ return out}/** hostile packages: each must be refused (or, for broken keyforms, played on the rig alone), the model before kept */diff --git a/test/smoke.mjs b/test/smoke.mjsindex f581d48..d23b7e6 100644--- a/test/smoke.mjs+++ b/test/smoke.mjs@@ -46,7 +46,8 @@ const server = createServer((req, res) => {})await new Promise((r) => server.listen(0, '127.0.0.1', r))-const browser = await chromium.launch({ args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--autoplay-policy=no-user-gesture-required'] })+const browser = await chromium.launch({ args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--autoplay-policy=no-user-gesture-required',+ '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] })let failed = 0try {const page = await browser.newPage()@@ -54,7 +55,7 @@ try {page.on('pageerror', (e) => errors.push(e.message))await page.goto(`http://127.0.0.1:${server.address().port}/test/smoke.html`)await page.waitForFunction(() => typeof window.run === 'function')- const { out, fromFiles, newer, legacy } = await page.evaluate((a) => window.run(a), { ids, files })+ const { out, fromFiles, newer, legacy, packaged } = await page.evaluate((a) => window.run(a), { ids, files })for (const r of out) {// the featured characters carry the LivLiv mark in the corner only: the player must draw it there and nowhere elseconst ok = !r.error && r.share > 0.05 && r.glError === 0 && r.watermark === 'LivLiv' && r.mark > 0.001 && r.outside < 0.0003 && r.noise < 0.0005@@ -65,6 +66,11 @@ try {const filesOk = !fromFiles.error && fromFiles.share > 0.05if (!filesOk) failed++console.log(`${filesOk ? 'ok ' : 'FAIL'} from files: ${fromFiles.error ?? `${fromFiles.name}, covers ${(fromFiles.share * 100).toFixed(1)} %`}`)+ for (const k of packaged) {+ const ok = k.changed < 0.0005 && k.keys[0] === k.keys[1]+ if (!ok) failed+++ console.log(`${ok ? 'ok ' : 'FAIL'} ${k.id}.livliv from a URL: ${(k.changed * 100).toFixed(2)} % of the pixels differ from its folder, keyforms ${k.keys.join(' / ')}`)+ }const legacyOk = !legacy.error && /^livemaker-puppet 3$/.test(legacy.format) && legacy.watermark === null && legacy.changed < 0.0005if (!legacyOk) failed++console.log(`${legacyOk ? 'ok ' : 'FAIL'} v3 package: ${legacy.error ?? `${legacy.format}, ${(legacy.changed * 100).toFixed(2)} % of the pixels differ from its v4`}`)@@ -84,10 +90,17 @@ try {if (!kfOk) failed++console.log(`${kfOk ? 'ok ' : 'FAIL'} broken keyforms: ${hostile.keyforms.refused ?? `${hostile.keyforms.name} plays on the rig alone`}`)+ // disposing while something runs (see runLifecycle)+ const life = await page.evaluate((id) => window.runLifecycle(id), ids[0])+ const lifeOk = /disposed/.test(life.load) && life.model === null && life.drawn === 0 && life.freed >= life.made && life.glError === 0+ && life.driver === 1 && !!life.again && life.mic === 'ended' && life.supported === true+ if (!lifeOk) failed+++ console.log(`${lifeOk ? 'ok ' : 'FAIL'} dispose while running: load "${life.load}", ${life.made} textures made, ${life.freed} freed, drawn ${life.drawn}, GL error ${life.glError}, driver disposed ${life.driver}, a new player ${life.again}, microphone ${life.mic}`)+// bodies: a PNGTuber shows each picture and follows the drivers; a VRM loads, draws and takes its expressionsawait page.goto(`http://127.0.0.1:${server.address().port}/test/bodies.html`)await page.waitForFunction(() => typeof window.runBodies === 'function')- const { png, vrm: v } = await page.evaluate(() => window.runBodies())+ const { png, vrm: v, pngFailed, pngOnGl, vrmAgain, vrmOn2d } = await page.evaluate(() => window.runBodies())const is = (rgb, [r, g, b]) => Math.abs(rgb[0] - r) < 40 && Math.abs(rgb[1] - g) < 40 && Math.abs(rgb[2] - b) < 40const pngOk = !png.error && png.format === 'png' && is(png.shown.closed, [255, 0, 0]) && is(png.shown.open, [0, 255, 0])&& is(png.shown.blink, [0, 0, 255]) && is(png.shown.blinkOpen, [255, 255, 0]) && png.tracked && png.angle > 0 && png.mouth > 0.3@@ -97,7 +110,35 @@ try {&& ['Smile', 'Happy', 'Sad'].every((l) => v.expressions.includes(l))if (!vrmOk) failed++console.log(`${vrmOk ? 'ok ' : 'FAIL'} VRM: ${v.error ?? `${v.name}, ${v.expressions.join(' / ')}, covers ${(v.share * 100).toFixed(1)} %, face tracked ${v.tracked}`}`)+ const pfOk = /needs at least/.test(pngFailed?.refused) && pngFailed.kept === 'First' && is(pngFailed.pixel, [255, 0, 0]) && /own/.test(pngOnGl)+ if (!pfOk) failed+++ console.log(`${pfOk ? 'ok ' : 'FAIL'} PNGTuber, a failed load: ${pngFailed?.refused} (kept ${pngFailed?.kept}, pixel ${pngFailed?.pixel}); on a WebGL canvas: ${pngOnGl}`)+ const vaOk = vrmAgain?.aspect1 === 2 && vrmAgain.aspect2 === 2 && /not a VRM/.test(vrmAgain.refused) && vrmAgain.kept === 'Tiny' && vrmAgain.share > 0.02 && /2D/.test(vrmOn2d)+ if (!vaOk) failed+++ console.log(`${vaOk ? 'ok ' : 'FAIL'} VRM, loaded again on a wide canvas: aspect ${vrmAgain?.aspect1}, ${vrmAgain?.aspect2}; a failed load kept ${vrmAgain?.kept}, covers ${(vrmAgain?.share * 100).toFixed(1)} %; on a 2D canvas: ${vrmOn2d}`)if (errors.length) { failed++; console.log(`FAIL page errors:\n ${errors.join('\n ')}`) }+ // no WebGL at all: the example pages say so, the check says no, new Player() says why (the pages fail on purpose: no+ // error of theirs counts here)+ const plain = await chromium.launch({ args: ['--disable-gpu', '--disable-software-rasterizer', '--disable-webgl', '--disable-webgl2', '--disable-3d-apis'] })+ try {+ const p = await plain.newPage()+ for (const name of ['index', 'demo', 'gallery']) {+ await p.goto(`http://127.0.0.1:${server.address().port}/example/${name}.html`)+ await p.waitForFunction(() => /WebGL2/.test(document.body.innerText), null, { timeout: 8000 }).catch(() => {})+ const said = /WebGL2/.test(await p.evaluate(() => document.body.innerText))+ if (!said) failed+++ console.log(`${said ? 'ok ' : 'FAIL'} no WebGL: example/${name}.html ${said ? 'says so' : 'says nothing'}`)+ }+ const r = await p.evaluate(async () => {+ const { Player } = await import('/dist/index.js')+ let thrown = ''+ try { new Player(document.createElement('canvas')) } catch (e) { thrown = e.message }+ return { supported: Player.isSupported(), thrown }+ })+ const ok = r.supported === false && /turned off/.test(r.thrown)+ if (!ok) failed+++ console.log(`${ok ? 'ok ' : 'FAIL'} no WebGL: isSupported ${r.supported}, new Player() says "${r.thrown}"`)+ } finally { await plain.close() }} finally {await browser.close()server.close()diff --git a/test/types.mjs b/test/types.mjsnew file mode 100644index 0000000..834d49a--- /dev/null+++ b/test/types.mjs@@ -0,0 +1,56 @@+// The package as a project that installed it sees it: its entries and types under the moduleResolution settings TypeScript+// projects use, with the declarations checked (skipLibCheck off). A broken declaration fails here; with skipLibCheck on it+// would pass and leave `any` behind, which the @ts-expect-error below catches.+// npm run build && npm run test:types+import { execFileSync } from 'node:child_process'+import fs from 'node:fs'+import { createRequire } from 'node:module'+import path from 'node:path'+import { fileURLToPath } from 'node:url'++const root = fileURLToPath(new URL('..', import.meta.url))+const { name } = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'))+if (!fs.existsSync(path.join(root, 'dist/index.js'))) { console.error('types: build first (npm run build)'); process.exit(1) }+const dir = path.join(root, 'node_modules/.tmp/consumer')+fs.rmSync(dir, { recursive: true, force: true })+fs.mkdirSync(path.join(dir, 'node_modules', path.dirname(name)), { recursive: true })+fs.symlinkSync(root, path.join(dir, 'node_modules', name), 'dir')+fs.writeFileSync(path.join(dir, 'use.ts'), `+import { Player, VERSION, type Driver, type ModelInfo, type View } from '${name}'+import { preloadFaceModel, startFaceTracking, type FaceTracking } from '${name}/tracking'+import { createSpeech, startLipSync, type LipSync } from '${name}/voice'+import { BODY_LIMIT, BodyPlayer, type Body } from '${name}/body'+import { PngBody } from '${name}/png'+import { VrmBody } from '${name}/vrm'+declare const canvas: HTMLCanvasElement+const player = new Player(canvas)+const model: Promise<ModelInfo> = player.load('/avatars/sena.livliv')+const driver: Driver = { update(frame) { frame.add({ angleX: 1 }) } }+const view: View = player.view+const tracking: Promise<FaceTracking> = startFaceTracking(player)+const sync: Promise<LipSync> = startLipSync(player)+const body: Body = new PngBody()+player.use(driver)+void [VERSION, model, view, tracking, sync, preloadFaceModel(), createSpeech(player), BODY_LIMIT, new BodyPlayer(canvas, body), new VrmBody()]+// the types are real, not any: these must be errors+// @ts-expect-error not an action+player.act('nope')+// @ts-expect-error a name is text+const n: number = (await model).name+void n+`)+const tsc = createRequire(path.join(root, 'package.json')).resolve('typescript/bin/tsc')+let failed = 0+for (const [moduleResolution, module] of [['node16', 'node16'], ['nodenext', 'nodenext'], ['bundler', 'esnext']]) {+ fs.writeFileSync(path.join(dir, 'tsconfig.json'), JSON.stringify({ compilerOptions: { noEmit: true, strict: true, target: 'es2022', lib: ['es2023', 'dom', 'dom.iterable'],+ module, moduleResolution, skipLibCheck: false, types: [] }, files: ['use.ts'] }))+ try {+ execFileSync(process.execPath, [tsc, '-p', dir], { stdio: 'pipe' })+ console.log(`ok moduleResolution ${moduleResolution}`)+ } catch (e) {+ failed+++ console.log(`FAIL moduleResolution ${moduleResolution}\n${String(e.stdout).split('\n').slice(0, 12).join('\n')}`)+ }+}+fs.rmSync(dir, { recursive: true, force: true })+process.exit(failed ? 1 : 0)diff --git a/vite.config.ts b/vite.config.tsindex 75bd0d8..c5fa9ea 100644--- a/vite.config.ts+++ b/vite.config.ts@@ -15,7 +15,9 @@ export default defineConfig({},rollupOptions: {external: ['@mediapipe/tasks-vision', 'three', /^three\//, '@pixiv/three-vrm'],- output: { chunkFileNames: 'chunks/[name]-[hash].js' },+ // minify: true above leaves the whitespace of an ES library in place (and these comments with the file names): the+ // package's readers download dist as it is from a CDN, so this one takes it out too (pure annotations stay)+ output: { chunkFileNames: 'chunks/[name]-[hash].js', minify: true },},},})