A lightweight 2.5D character player for the web, with lip sync and webcam face tracking.
Player 0.3.0: a new load replaces the one running; load options, errors with a code, context loss
LivLiv27265faa7eed
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex c6ec096..09d4635 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -23,3 +23,5 @@ jobs:- run: npm run test:types- run: npx playwright-core install --with-deps chromium-headless-shell- run: npm run test:browser+ - run: npx playwright-core install --with-deps firefox webkit+ - run: npm run test:enginesdiff --git a/API.md b/API.mdindex 60d0b4b..d687060 100644--- a/API.md+++ b/API.md@@ -67,10 +67,10 @@ new Player(canvas: HTMLCanvasElement, options?: PlayerOptions)`new Player()` throws where WebGL2 cannot start: the browser has none or has it turned off, or the canvas already drawsin 2D. `Player.isSupported()` asks first, so a page can say so or show a picture instead.-### `load(source): Promise<ModelInfo>`+### `load(source, options?): Promise<ModelInfo>`-Loads a model and shows it in place of the one before (loads run one after another; a failed load keeps the old one).-`source` is one of:+Loads a model and shows it in place of the one before. A load that fails keeps the old one, whatever goes wrong. `source`+is one of:- a URL (string or `URL`) of a `.livliv` (or `.zip`) package, of `model.json`, or of its folder; relative URLs resolveagainst the page. A query string (`?token=…`) is sent with every file.@@ -87,6 +87,49 @@ input.onchange = () => player.load(input.files)A package is checked before anything is decoded: a newer format than this player reads, a feature it does not have,paths outside the package, an encrypted or oversized ZIP are refused with an error saying so.+| option | |+|---|---|+| `signal` | an `AbortSignal` that calls the load off: it rejects with an `AbortError` (or the signal's own reason) and the character before stays |+| `onProgress` | called with `{ step: 'download', progress }` as the files come in (0..1, always rising; bytes for a package, images for a folder; where the size is not known, 0 and then 1), then once with `{ step: 'build' }` as the character goes in |+| `poster` | the URL of a picture to show until the first frame (a featured character's `cover.webp`). It is the canvas's CSS background, so it fits the canvas's box, adds nothing to the page, and the canvas's own background comes back after. Only while the canvas shows nothing yet: over a character, that one stays until the next is ready |++**A new `load()` replaces one still running**, so the last one asked for is the one that plays, and the download before it+stops (Safari may keep the connection open). The one replaced rejects with an `AbortError`. Nobody has to catch it: a+page that only calls `player.load(a); player.load(b)` sees nothing in the console, while a real failure is reported as+any uncaught rejection is. A signal that is already aborted rejects at once and leaves the running load alone. The last+moment, when the model goes in, cannot be interrupted and takes a few milliseconds to a few hundred: a load that has+reached it finishes.++```js+const stop = new AbortController()+try {+ await player.load(url, { signal: stop.signal, poster: 'cover.webp', onProgress: (p) => { if (p.step === 'download') bar.value = p.progress } })+} catch (e) {+ if (e.name === 'AbortError') return // replaced by a newer load, or called off+ if (e.code === 'unreachable') say('Check the connection')+ else say(e.message)+}+```++**Errors.** What a load rejects with for a reason of its own is a `LivLivError` (from `@livliv/player` and+`@livliv/player/body`): `code` says what went wrong and `status` the HTTP status. The message is for people and can change.+`new Player()` throws the same class. A camera or microphone keeps the browser's own errors.++| `code` | |+|---|---|+| `no-webgl` | this browser or canvas gives no WebGL2 |+| `canvas-in-use` | the canvas already draws in a way this character cannot (a PNGTuber on a WebGL canvas) |+| `unreachable` | the file did not come: no connection, a connection that dropped, or another origin without a CORS header |+| `http` | the server said no (`status`) |+| `not-a-model` | not a model at all, or not one for this player (a VRM handed to a `Player`) |+| `invalid` | a model, but something in it is wrong or damaged |+| `too-large` | over this player's limits, or the GPU's memory |+| `newer-format` | written for a newer player |+| `missing-feature` | needs something this page or player does not have |+| `missing-file` | a file it names is not among the files |+| `context-lost` | the canvas's WebGL context is gone, see [below](#when-the-context-is-lost) |+| `released` | the canvas gave its context up with `dispose({ release: true })`: use a new canvas |+`ModelInfo`:| field | |@@ -119,14 +162,15 @@ top; it never writes them back, so a value you set stays until you change it.With `idle` on, the idle motion sets the head, eyes, gaze, brows, body and breath, so your values for those countonly with `idle` off (or use a [driver](#drivers) to add to them). `mouthOpen`, `vowel` and `smile` are always yours.-`PARAM_RANGES` and `defaultParams()` are exported.+`PARAM_RANGES` and `defaultParams()` are exported. Whoever sets them, the pose stays in these ranges, and a value that is+not a number counts as the default.### Expressions and actions- `setExpression(id, weight = 1): boolean`: eases over to that expression (ids from `expressions`; `null` = neutral).Returns false for an id the model doesn't have.-- `act(action)`: a short move over whatever else drives the character: `'nod'`, `'shake'`, `'tilt'`, `'hop'`,- `'shy'`; `'away'` toggles a dozing pose.+- `act(action)`: a short move over whatever else drives the character: `'nod'`, `'shake'`, `'tilt'`, `'hop'`, `'shy'`.+- `setAway(on)`: dozes (eyes closed, head down, a slow breath, expressions hidden), easing in, until `setAway(false)`.- `lookAt(x, y)`: looks toward a point, each -1..1 from the canvas center (y up), and keeps looking there insteadof following the pointer; `lookAt(null)` goes back to the pointer. Needs `idle`.- `blink()`: blinks now (with `idle`).@@ -137,12 +181,63 @@ only with `idle` off (or use a [driver](#drivers) to add to them). `mouthOpen`,center, `head: true` frames the head instead of the whole drawing (avatars, chat bubbles).- `play()`, `pause()`, `playing`. The loop also skips drawing while the canvas is scrolled out of view.- `resize()`: draws now at the canvas's current size. Only needed after resizing a paused player.-- `snapshot()`: the current frame as a PNG data URL (with the mark).-- `dispose()`: stops, removes drivers, frees the character's GPU memory. A `load()` still running rejects, and a camera or- microphone still starting (`startFaceTracking`, `startLipSync`) is let go as soon as it opens. The canvas keeps its WebGL- context and can take a new `Player` afterwards. A browser allows about 16 contexts at once and drops the oldest past that,- so a page that shows many characters over time should reuse its canvases.-- `puppet`: the renderer underneath, for what this API does not cover. Its fields can change between versions.+- `maxPixelRatio`: as the option, and it can be changed at any time.+- `snapshot()`: the pose as it is now, as a PNG data URL (with the mark).+- `canvasToModel(x, y)`: a point on the canvas (CSS px from its top left) in the model's own px, as the last frame drew it.+- `poke(x, y, strength = 1)`: a flick at a point in the model's px: the swinging pieces near it are thrown (a tap on a twin+ tail). Returns how many it moved: none on a model without any.+- `dispose({ release })`: stops, removes drivers, frees the character's GPU memory. A `load()` still running rejects, and a+ camera or microphone still starting (`startFaceTracking`, `startLipSync`) is let go as soon as it opens.++### Canvases and contexts++A browser allows about 16 WebGL contexts on a page and drops the oldest past that, which can be a character still showing.+So a page where characters come and go has two ways to stay under it:++- **Reuse canvases.** After `dispose()` the canvas keeps its context and can take a new `Player`. This is the only way that+ works everywhere: Safari does not get the room back from a released context.+- **Release what leaves.** `dispose({ release: true })` gives the context up, for a canvas leaving the page. Measured here, 40+ players came and went beside one that stayed, and it stayed (Chromium, Firefox). The canvas is spent after that: a new+ player on it throws `released`, unless it is made at once, in the same task, as a framework's unmount and mount is.++In **React**, this is the whole of it, and it is right under StrictMode, which mounts everything twice ([example/react.html](example/react.html)):++```jsx+function Character({ source }) {+ const canvas = useRef(null), player = useRef(null)+ useEffect(() => {+ const p = player.current = new Player(canvas.current)+ return () => p.dispose({ release: true })+ }, [])+ useEffect(() => {+ player.current.load(source).catch((e) => { if (e.name !== 'AbortError') console.error(e) })+ }, [source])+ return <canvas ref={canvas} style={{ width: 320, height: 400 }} />+}+```++### When the context is lost++A browser can take a page's WebGL context away: a phone reclaiming memory while the page is in the background, a graphics+driver restarting, too many contexts on the page. The character is gone from that canvas and does not come back by itself.+`player.contextLost` says so, and `player.onContextLost` is called once. Meanwhile frames do nothing, and `load()` rejects with+`context-lost`. The way back is a new canvas and a new player:++```js+function show(parent, source) {+ const canvas = document.createElement('canvas')+ parent.replaceChildren(canvas)+ const player = new Player(canvas)+ player.onContextLost = () => {+ player.dispose()+ const again = () => show(parent, source)+ if (document.hidden) addEventListener('visibilitychange', again, { once: true }) // when someone is back to see it+ else again()+ }+ player.load(source).catch(console.error)+ return player+}+```### The mark@@ -311,11 +406,15 @@ await vrm.load('/avatars/me.vrm') // a URL, a File, or files pickedfile's own words in `raw`. The player does not enforce them. A page playing other people's avatars should show them andkeep to them, as you would with a LivLiv package's `meta.license`.- **Your own**: anything with `load(files)`, `render(canvas, pose, w, h, dt)`, `head()` and `dispose()` (the `Body`- type), and if it cares which canvas it gets, `attach(canvas)`. `pose` has the params after the idle motion and the+ type), and if it cares which canvas it gets, `attach(canvas)`. One that draws with WebGL can also have `context()`, returning+ its context, so that `contextLost` and `dispose({ release })` work for it. `pose` has the params after the idle motion and thedrivers, the picture's `move` and each expression's weight.A failed `load()` keeps the character before, as on a Player. A canvas has one kind of context for good: a PNGTuber draws in 2D, a Player and a VRM in WebGL2, so a PNGTuber on a canvas that was a Player's (or the other way round) is refused with an error; give it its own.-A BodyPlayer draws no mark and has no `puppet`; `view.head` is not used. Live2D models (`.moc3`) are not supported:+A BodyPlayer takes the same `load()` options and its errors are the same, with one difference: a Body swaps its character+inside its own `load`, which cannot be interrupted, so a character already going in finishes and the newer one follows.+Downloads and loads still waiting are replaced, as on a Player. It draws no mark and has no `poke` or `canvasToModel`;+`view.head` is not used. `onContextLost` and `dispose({ release })` are for a VRM; a PNGTuber has no context to lose. Live2D models (`.moc3`) are not supported:reading them takes Live2D's proprietary Cubism Core, under its own licence.## Untrusted filesdiff --git a/CHANGELOG.md b/CHANGELOG.mdnew file mode 100644index 0000000..e3f1534--- /dev/null+++ b/CHANGELOG.md@@ -0,0 +1,35 @@+# Changelog++## 0.3.0++The interface settled before the package goes on npm: these change what a page written for 0.2 does.++### What to change++| 0.2 | 0.3 |+|---|---|+| `player.act('away')` | `player.setAway(true)` and `setAway(false)`. `act('away')` now throws: it was a switch among one-off moves. |+| `player.puppet` | No longer in the types. `player.poke()` and `player.canvasToModel()` are what pages used it for. The field still exists and can change in any version. |+| `player.load(a); player.load(b)` played `a`, then `b` | The new load replaces the one running, which rejects with an `AbortError`. If you catch errors, let this one through: `if (e.name === 'AbortError') return`. |+| `dispose()` on a load running rejected with an `Error` | It rejects with an `AbortError`, as `load()` on a disposed player does. |+| Errors told apart by their message | `e.code`, on a `LivLivError`. The messages can change. |++### New++- `load(source, { signal, onProgress, poster })`: call a load off, follow how far it has come, show a picture until the first frame.+- `LivLivError`, with a `code` for each reason, from `@livliv/player` and `@livliv/player/body`.+- `player.contextLost` and `player.onContextLost`: the page hears when the WebGL context is gone. See [API.md](API.md#when-the-context-is-lost).+- `dispose({ release: true })`: gives the WebGL context up, for a canvas leaving the page.+- `player.setAway()`, `player.poke()`, `player.canvasToModel()`.+- `example/react.html`.++### Changed++- A model that breaks while it is built leaves the character before as it was, to the pixel. It used to leave an empty canvas.+- The pose stays in the ranges of `PARAM_RANGES` on `Player` too, as it did on `BodyPlayer`. A value that is not a number counts as the default.+- `maxPixelRatio` can be read and changed on a `Player`.+- `snapshot()` on a `Player` draws first, so it is the pose as it is now, on a paused player too.+- A `BodyPlayer`'s `Frame.time` counts the frames before this one, as a `Player`'s does. Its idle motion runs one frame later.+- The two classes now share their code. Nothing a `Player` draws changed: frame by frame, with the same random numbers, 1300 frames each of 4 characters came out identical.++See the [commits](https://github.com/livliv-dev/player/commits/main) for what came before 0.3.diff --git a/README.ja.md b/README.ja.mdindex 18be7ca..4690ef8 100644--- a/README.ja.md+++ b/README.ja.md@@ -75,7 +75,7 @@ API の全体は [API.md](API.md)(英語)にあります:オプションKujira、Kujira Q、Pen は他の方の作品に基づくもので、上記のライセンスとクレジットは適用されません。出典、クレジット、条件は [characters/CREDITS.md](characters/CREDITS.md) をご覧ください。-各キャラクターには `cover.webp`(512 x 768、約 50 KB)もあります。最初のフレームで、マークは入っていません。キャンバスの下に `object-fit: contain` で置き、準備ができたらキャラクターをフェードインさせます([example/loading.js](example/loading.js) がこの方法です)。ページにすぐ絵が出て、切り替わるときにずれません。+各キャラクターには `cover.webp`(512 x 768、約 50 KB)もあります。最初のフレームで、マークは入っていません。`player.load(source, { poster })` を使うと、最初のフレームが出るまでキャンバスに表示されます。スクリプトが動く前から絵を出したいときは、キャンバスの下に `object-fit: contain` で置き、準備ができたらキャラクターをフェードインさせます([example/loading.js](example/loading.js) がこの方法です)。キャラクターの一覧は[英語の README](README.md#featured-characters) にあります。diff --git a/README.md b/README.mdindex 1075c96..62ef696 100644--- a/README.md+++ b/README.md@@ -83,9 +83,9 @@ LivLiv's mascot. The terms are in [characters/LICENSE.md](characters/LICENSE.md)Kujira, Kujira Q and Pen come from other people's work: that licence and that credit don't cover them. Their sources,credits and terms are in [characters/CREDITS.md](characters/CREDITS.md).-Each also has a `cover.webp` (512 x 768, about 50 KB), its first frame without the mark. Show it under the canvas with-`object-fit: contain` and fade the character in over it, as [example/loading.js](example/loading.js) does: the page has a-picture at once, and nothing jumps.+Each also has a `cover.webp` (512 x 768, about 50 KB), its first frame without the mark. `player.load(source, { poster })`+shows it on the canvas until the first frame. To have it before any script has run, put it under the canvas with+`object-fit: contain` and fade the character in over it, as [example/loading.js](example/loading.js) does.| id | name | id | name | id | name ||---|---|---|---|---|---|diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 1ca952f..fd4a418 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -77,7 +77,7 @@ await png.load({ closed: 'me-closed.png', open: 'me-open.png' })Kujira、Kujira Q 和 Pen 来自别人的作品,不适用上面的许可和署名。它们的来源、署名和条款见 [characters/CREDITS.md](characters/CREDITS.md)。-每个角色还带一张 `cover.webp`(512 x 768,约 50 KB),是它的第一帧,不带标。把它放在画布下面(`object-fit: contain`),角色好了再淡入,[example/loading.js](example/loading.js) 就是这样做的:页面马上有画面,也不会跳一下。+每个角色还带一张 `cover.webp`(512 x 768,约 50 KB),是它的第一帧,不带标。`player.load(source, { poster })` 会在第一帧出来之前把它显示在画布上。如果想在脚本运行之前就有画面,就把它放在画布下面(`object-fit: contain`),角色好了再淡入,[example/loading.js](example/loading.js) 就是这样做的。角色列表见[英文 README](README.md#featured-characters)。diff --git a/example/demo.html b/example/demo.htmlindex 71f3217..e41d43d 100644--- a/example/demo.html+++ b/example/demo.html@@ -23,11 +23,12 @@<script type="module">import { Player } from '../dist/index.js'import { cameraButton } from './camera.js'- import { loadWithFallback, loading, needWebGL, sameSite } from './loading.js'+ import { loadWithFallback, loading, needWebGL, sameSite, watchContext } from './loading.js'needWebGL(Player)const canvas = document.getElementById('stage'), cover = document.getElementById('cover')const player = new Player(canvas)+ watchContext(player)const bar = document.getElementById('bar'), status = document.getElementById('status')window.player = player // to try things in the console@@ -40,16 +41,16 @@status.textContent = ''const done = loading(canvas, cover, source)try {- const m = await loadWithFallback(player, source)+ const m = await loadWithFallback(player, source, { onProgress: done.progress })status.textContent = `${m.name} · ${m.format} ${m.version} · mark: ${m.watermark ?? 'none'}${m.meta?.license ? ` · ${m.meta.license.id}` : ''}`controls(m)} catch (e) {+ if (e.name === 'AbortError') return // another was picked meanwhilestatus.textContent = e.message// a VRM has its own page hereif (/VRM/.test(e.message)) { const a = document.createElement('a'); a.href = 'vrm.html'; a.style.color = 'inherit'; a.textContent = ' Open the VRM page'; status.append(a) }controls(player.model)- }- done()+ } finally { done() }}function controls(m) {diff --git a/example/gallery.html b/example/gallery.htmlindex 781599e..b26323c 100644--- a/example/gallery.html+++ b/example/gallery.html@@ -34,7 +34,7 @@</main><script type="module">import { Player } from '../dist/index.js'- import { loadWithFallback, loading, needWebGL } from './loading.js'+ import { loadWithFallback, loading, needWebGL, watchContext } from './loading.js'// All the featured characters: one plays on the stage (one player, one WebGL context), the rest wait as their// pictures; a click on one puts it on the stage. ?id=<id> starts with that one (else Refined Nana); ?demo=1 plays expressions and@@ -48,6 +48,7 @@needWebGL(Player)const canvas = document.getElementById('stage'), cover = document.getElementById('cover')const player = new Player(canvas, { maxPixelRatio: 1.5 })+ watchContext(player)const bar = document.getElementById('bar'), name = document.getElementById('name'), credit = document.getElementById('credit'), ul = document.getElementById('cast')const buttons = new Map()let current = null, exprs = [], next = -1@@ -71,8 +72,9 @@const source = `../characters/${id}.livliv`, done = loading(canvas, cover, source)let mtry {- m = await loadWithFallback(player, source).finally(done)+ m = await loadWithFallback(player, source, { onProgress: done.progress }).finally(done)} catch (e) { // the one on the stage stays; a click can try this one again+ if (e.name === 'AbortError') return // another was picked meanwhileif (id === current) { current = before; press(before); name.textContent = e.message }return}diff --git a/example/index.html b/example/index.htmlindex 42b2694..4a0cd87 100644--- a/example/index.html+++ b/example/index.html@@ -10,14 +10,15 @@<p id="note" style="position:fixed;z-index:1;inset:45% 0 auto;margin:0;color:#eef;font:14px system-ui,sans-serif;text-align:center"></p><script type="module">import { Player } from '../dist/index.js'- import { loadWithFallback, loading } from './loading.js'+ import { loadWithFallback, loading, watchContext } from './loading.js'// a featured character; your own export works the sameconst note = document.getElementById('note'), canvas = document.getElementById('stage')if (!Player.isSupported()) { note.textContent = 'This browser has no WebGL2'; throw new Error('no WebGL2') }const player = new Player(canvas)+ watchContext(player)const done = loading(canvas, document.getElementById('cover'), '../characters/d03ft.livliv')- const model = await loadWithFallback(player, '../characters/d03ft.livliv').catch((e) => { note.textContent = e.message; throw e }).finally(done)+ const model = await loadWithFallback(player, '../characters/d03ft.livliv', { onProgress: done.progress }).catch((e) => { note.textContent = e.message; throw e }).finally(done)// each click shows the next expression, then neutral againlet i = -1diff --git a/example/loading.js b/example/loading.jsindex decbfd3..37acc23 100644--- a/example/loading.js+++ b/example/loading.js@@ -24,7 +24,7 @@ export function loading(canvas, cover, source) {note.textContent = 'Loading…'note.style.cssText = 'position:absolute;z-index:1;left:50%;bottom:12%;transform:translateX(-50%);margin:0;padding:4px 10px;border-radius:8px;background:#000a;color:#fff;font:13px system-ui,sans-serif;pointer-events:none'const later = setTimeout(() => canvas.parentElement.append(note), url ? 800 : 150)- return () => {+ const done = () => {waiting = falseclearTimeout(later)note.remove()@@ -34,6 +34,9 @@ export function loading(canvas, cover, source) {canvas.style.opacity = 1setTimeout(() => { if (me === latest) cover.hidden = true }, 300)}+ /** for load()'s onProgress: how far the note says it has come */+ done.progress = (p) => { note.textContent = p.step === 'download' && p.progress > 0 && p.progress < 1 ? `Loading… ${Math.round(p.progress * 100)} %` : 'Loading…' }+ return done}/** Is this URL on the page's own site? ?model= opens nothing else: a link must not put someone else's file on this@@ -52,12 +55,26 @@ export function needWebGL(Player) {throw new Error('no WebGL2')}+/** When the browser takes the page's graphics away (a phone reclaiming memory, say), the character is gone from this canvas+ * for good. These pages say so and offer to start again; a page of your own can make a new canvas and a new player at once:+ * see "When the context is lost" in API.md. */+export function watchContext(player) {+ player.onContextLost = () => {+ const note = document.createElement('button')+ note.id = 'lost'+ note.style.cssText = 'position:fixed;z-index:2;left:16px;right:16px;top:32px;padding:8px 12px;border:0;border-radius:8px;background:#000c;color:#fff;text-align:center;font:14px system-ui,sans-serif'+ note.textContent = 'The browser took this page\'s graphics away. Tap to start again.'+ note.onclick = () => location.reload()+ document.body.append(note)+ }+}+/** Loads a model, with a featured character's folder as the second chance when its package (one big file) does not come. */-export async function loadWithFallback(player, source) {+export async function loadWithFallback(player, source, options) {try {- return await player.load(source)+ return await player.load(source, options)} catch (e) {- if (typeof source !== 'string' || !source.endsWith('.livliv')) throw e- return player.load(source.replace(/\.livliv$/, '/')).catch(() => { throw e })+ if (e.name === 'AbortError' || typeof source !== 'string' || !source.endsWith('.livliv')) throw e // another was picked meanwhile+ return player.load(source.replace(/\.livliv$/, '/'), options).catch(() => { throw e })}}diff --git a/example/png.html b/example/png.htmlindex dcf0554..3db636d 100644--- a/example/png.html+++ b/example/png.html@@ -49,7 +49,7 @@try {const m = await player.load(files)status.textContent = `${m.name} · ${m.format}`- } catch (e) { status.textContent = e.message }+ } catch (e) { if (e.name !== 'AbortError') status.textContent = e.message } // (another was picked meanwhile)}document.getElementById('pick').onchange = (e) => e.target.files.length && show(e.target.files)show({ closed: await face(true, false), open: await face(true, true), blink: await face(false, false), 'blink-open': await face(false, true) })diff --git a/example/react.html b/example/react.htmlnew file mode 100644index 0000000..9eb4825--- /dev/null+++ b/example/react.html@@ -0,0 +1,54 @@+<!doctype html>+<html lang="en">+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1">+<title>LivLiv Player in React</title>+<style>+ body { margin: 0; font: 15px/1.5 system-ui, sans-serif; background: #f4f2ef; color: #222; padding: 16px; }+ canvas { display: block; width: 320px; height: 400px; }+ button { margin: 8px 8px 0 0; }+</style>+<script type="importmap">{ "imports": {+ "react": "https://esm.sh/[email protected]",+ "react-dom/client": "https://esm.sh/[email protected]/client"+} }</script>+<div id="root"></div>+<p id="status"></p>+<script type="module">+ // A player in a React component, under StrictMode, which mounts everything twice. One effect owns the player and gives its+ // canvas up on unmount; another loads what the props say. StrictMode's second mount comes in the same task, so the new+ // player keeps the context, and its load replaces the first, which rejects with an AbortError.+ import { StrictMode, createElement as h, useEffect, useRef, useState } from 'react'+ import { createRoot } from 'react-dom/client'+ import { Player } from '../dist/index.js'++ const status = document.getElementById('status')++ function Character({ source }) {+ const canvas = useRef(null), player = useRef(null)+ useEffect(() => {+ const p = player.current = new Player(canvas.current)+ return () => p.dispose({ release: true })+ }, [])+ useEffect(() => {+ player.current.load(source).then(+ (m) => { status.textContent = `${m.name} · ${m.format} ${m.version}` },+ (e) => { if (e.name !== 'AbortError') status.textContent = e.message })+ }, [source])+ return h('canvas', { ref: canvas })+ }++ const CAST = ['k01', 'd03', 'v05']++ function App() {+ const [who, setWho] = useState(0), [on, setOn] = useState(true)+ return h('div', null,+ on && h(Character, { source: `../characters/${CAST[who]}.livliv` }),+ h('button', { id: 'next', onClick: () => setWho((who + 1) % CAST.length) }, 'next character'),+ h('button', { id: 'toggle', onClick: () => setOn(!on) }, on ? 'unmount' : 'mount'))+ }++ createRoot(document.getElementById('root')).render(h(StrictMode, null, h(App)))+ window.__react = true+</script>+</html>diff --git a/example/vrm.html b/example/vrm.htmlindex 669ecc4..bf02557 100644--- a/example/vrm.html+++ b/example/vrm.html@@ -29,9 +29,10 @@import { BodyPlayer } from '../dist/body.js'import { VrmBody } from '../dist/vrm.js'import { cameraButton } from './camera.js'- import { sameSite } from './loading.js'+ import { sameSite, watchContext } from './loading.js'const player = new BodyPlayer(document.getElementById('stage'), new VrmBody())+ watchContext(player)const status = document.getElementById('status'), bar = document.getElementById('bar')window.player = player@@ -50,7 +51,7 @@for (const a of ['nod', 'shake', 'tilt']) {const b = document.createElement('button'); b.className = 'x'; b.textContent = a; b.onclick = () => player.act(a); bar.append(b)}- } catch (e) { status.textContent = e.message }+ } catch (e) { if (e.name !== 'AbortError') status.textContent = e.message } // (another was picked meanwhile)}document.getElementById('pick').onchange = (e) => e.target.files.length && show(e.target.files[0])const url = new URLSearchParams(location.search).get('model')diff --git a/package.json b/package.jsonindex 14b7dd4..ec0626b 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "@livliv/player",- "version": "0.2.1",+ "version": "0.3.0","description": "LivLiv Player: a lightweight 2.5D character player for the web, with lip sync and webcam face tracking. Plays LivLiv, VRM and PNGTuber characters; WebGL2, no dependencies in the core.","private": true,"license": "Apache-2.0",@@ -29,6 +29,7 @@"dist","README.md","API.md",+ "CHANGELOG.md","FORMAT.md","FORMAT-v3.md","schema",@@ -95,6 +96,7 @@"test:types": "node test/types.mjs","check": "tsc -p tsconfig.json","test": "node --disable-warning=ExperimentalWarning --import ./test/ts.mjs --test \"test/*.test.ts\"",- "test:browser": "node test/smoke.mjs"+ "test:browser": "node test/smoke.mjs",+ "test:engines": "node test/smoke.mjs firefox && node test/smoke.mjs webkit"}}diff --git a/src/base.ts b/src/base.tsnew file mode 100644index 0000000..5d03db2--- /dev/null+++ b/src/base.ts@@ -0,0 +1,413 @@+// What Player and BodyPlayer have in common, in one place: the frame loop, the pose each frame, expressions, actions,+// drivers, the pointer and the order loads run in. A subclass says what it draws and how it loads.+//+// Each frame the pose is put together from scratch: your `params`, then the idle motion (head, eyes, brows, body, breath;+// follow.ts: livliv.ai's, following the pointer with the eyes and the face), then the drivers in order, then actions and+// expressions, kept in the ranges of PARAM_RANGES. Nothing the player adds is written back to `params`, so a value you set+// stays until you change it, and turning a driver off leaves no trace.+import { ActionPlayer, type ActionId as AnyAction } from './runtime/actions'+import { uiLang } from './runtime/format'+import { PARAM_RANGES, defaultParams, type Params } from './runtime/params'+import { LivLivError } from './errors'+import { Idle, type Seen } from './follow'+import { releaseCanvas } from './release'++export { LivLivError, type LivLivErrorCode } from './errors'++/** a short move: see act() */+export type ActionId = Exclude<AnyAction, 'away'>++export interface PlayerOptions {+ /** move by itself: breathe, blink, sway, look around (default true). Off: only your params and drivers move it */+ idle?: boolean+ /** the eyes and head follow the pointer anywhere on the page ('page', the default), only over the canvas+ * ('canvas'), or not at all (false) */+ followPointer?: 'page' | 'canvas' | false+ /** drawing-buffer pixels per CSS pixel at most (default 2; 1 saves power on high-DPI screens) */+ maxPixelRatio?: number+ /** start the frame loop at once (default true) */+ autoplay?: boolean+ /** the language expression names are picked in (`expressions[].label`; a BCP 47 tag, default the browser's) */+ lang?: string+}++/** where the character sits in the canvas: x, y move it by shares of the canvas (y up), zoom scales it about the+ * center; head: frame the head instead of the whole drawing (avatars, chat bubbles; a BodyPlayer does not use it) */+export interface View { x: number; y: number; zoom: number; head: boolean }++/** one frame being put together, as a driver sees it */+export interface Frame {+ /** the pose so far: write to it (for this frame only) */+ readonly params: Params+ /** seconds since the last frame, and since the model loaded */+ readonly dt: number+ readonly time: number+ /** a face tracker drives the head this frame (lip sync then leaves the head alone) */+ tracked: boolean+ /** add to params, kept in their ranges */+ add(offsets: Partial<Params>): void+ /** move the picture: x, y shares of the canvas (y up), s a scale */+ readonly move: { x: number; y: number; s: number }+}++/** something that moves the character each frame: face tracking and lip sync are drivers, and your own code can be */+export interface Driver {+ /** lower runs first: face tracking -10, yours 0 (default), lip sync 10 */+ order?: number+ update(frame: Frame, player: Host): void+ /** called when it is removed, or the player disposed */+ dispose?(): void+}++/** what a driver, the lip sync and the face tracking need of a player: a Player, or a BodyPlayer (@livliv/player/body) */+export interface Host {+ /** adds a driver; returns a function that removes it */+ use(driver: Driver): () => void+ /** blinks now (with the idle motion on) */+ blink(): void+}++/** how far a load has come: the files coming in (0..1, in step order, 1 once they are all here; where the size is not+ * known it stays at 0 until then), then the character going in */+export type LoadProgress = { step: 'download'; progress: number } | { step: 'build' }++export interface LoadOptions {+ /** calls the load off: it rejects with an AbortError (or the signal's own reason) and the character before stays. An+ * already aborted signal rejects at once and leaves a load that is running alone */+ signal?: AbortSignal+ onProgress?: (progress: LoadProgress) => void+ /** the URL of a picture to show until the first frame (a character's cover.webp), as the canvas's CSS background: it+ * fits the canvas's box, adds nothing to the page, and the canvas's own background comes back after. Only while the+ * canvas shows nothing yet: over a character, that one stays until the next is ready */+ poster?: string+}++/** what a running load gives its steps+ * @internal */+export interface LoadJob {+ signal: AbortSignal+ /** the files have come this far (0..1) */+ progress(share: number): void+ /** they are all here, and the character goes in */+ build(): void+}++/** @internal */+export const clamp = (v: number, a: number, b: number) => Math.min(b, Math.max(a, v))++/** what a load rejects with when it is called off. A DOMException like a fetch's, so `e.name === 'AbortError'` tells+ * @internal */+export const aborted = (why: string) => new DOMException(`LivLiv: ${why}`, 'AbortError')++/** @internal */+export const contextGone = () => new LivLivError('context-lost', 'LivLiv: this canvas has lost its WebGL context: use a new canvas and a new player')++/** a response's bytes, saying how far they have come+ * @internal */+export async function readBody(r: Response, job: LoadJob, share = (p: number) => p): Promise<Blob> {+ const size = r.headers.get('content-encoding') ? 0 : Number(r.headers.get('content-length')) || 0+ if (!r.body) return r.blob()+ const reader = r.body.getReader(), parts: Uint8Array<ArrayBuffer>[] = []+ let got = 0+ for (;;) {+ const { done, value } = await reader.read()+ if (done) break+ parts.push(value)+ got += value.byteLength+ if (size) job.progress(share(Math.min(1, got / size)))+ }+ return new Blob(parts)+}++const DEFAULTS = defaultParams()+const KEYS = Object.keys(PARAM_RANGES) as (keyof Params)[]++export abstract class PlayerBase<Info extends { expressions: { id: string }[] }> implements Host {+ readonly canvas: HTMLCanvasElement+ /** your values for the pose and face (ranges in PARAM_RANGES). The player never changes them */+ readonly params: Params = defaultParams()+ /** see PlayerOptions.idle */+ idle: boolean+ /** see PlayerOptions.followPointer */+ followPointer: 'page' | 'canvas' | false+ readonly view: View = { x: 0, y: 0, zoom: 1, head: false }+ /** see PlayerOptions.lang */+ readonly lang: string+ /** Called when the canvas's WebGL context is lost (a phone reclaiming the page's GPU memory, a driver reset, too many+ * contexts on the page). The character is gone from this canvas and does not come back by itself: put a new canvas in+ * its place and make a new player. On a page with many players, a new context can push out another: release those+ * that have left (dispose({ release: true })) */+ onContextLost?: () => void++ /** @internal */+ protected info: Info | null = null+ /** @internal */+ protected disposed = false+ private idler = new Idle()+ /** last frame's: a face tracker drove it, the mouth moved (the idle starts no move while the character talks) */+ private was = { tracked: false, mouth: 0 }+ /** @internal */+ protected actions = new ActionPlayer()+ private drivers: Driver[] = []+ private expr: Record<string, number> = {}+ private smile = 0+ private raf = 0+ private last = 0+ private visible = true+ private io?: IntersectionObserver+ private queue: Promise<unknown> = Promise.resolve()+ private lost = false+ /** the load that a new one replaces */+ private running: AbortController | null = null+ /** the canvas's own background while a poster is showing */+ private posterOver: Record<string, string> | null = null+ /** @internal */+ protected frame: Frame & { move: { x: number; y: number; s: number } }++ /** pose: the Params the subclass draws from, written to each frame */+ constructor(canvas: HTMLCanvasElement, pose: Params, options: PlayerOptions) {+ this.canvas = canvas+ this.idle = options.idle ?? true+ this.followPointer = options.followPointer ?? 'page'+ this.lang = options.lang || uiLang()+ this.frame = {+ params: pose, dt: 0, time: 0, tracked: false, move: { x: 0, y: 0, s: 1 },+ add(o) {+ for (const [k, v] of Object.entries(o) as [keyof Params, number | undefined][]) {+ if (v) pose[k] = clamp(pose[k] + v, PARAM_RANGES[k][0], PARAM_RANGES[k][1])+ }+ },+ }+ addEventListener('pointermove', this.onMove)+ document.addEventListener('pointerleave', this.onLeave)+ canvas.addEventListener('webglcontextlost', this.onLost)+ if (typeof IntersectionObserver !== 'undefined') {+ this.io = new IntersectionObserver((e) => { this.visible = e[e.length - 1].isIntersecting })+ this.io.observe(canvas)+ }+ if (options.autoplay ?? true) this.play()+ }++ /** something is loaded to draw+ * @internal */+ protected abstract get drawable(): boolean+ /** seconds this character has played+ * @internal */+ protected abstract get clock(): number+ /** each expression's weight as shown, eased in draw()+ * @internal */+ protected abstract get shownExpr(): Record<string, number>+ /** where the head is drawn, for where the eyes look+ * @internal */+ protected abstract seen(): Seen+ /** draws the pose put together in frame+ * @internal */+ protected abstract show(dt: number): void+ /** frees what it drew with+ * @internal */+ protected abstract free(): void++ /** the context this draws with, where it has one and has started it+ * @internal */+ protected abstract get gl(): WebGLRenderingContext | WebGL2RenderingContext | null++ /** the canvas's WebGL context is gone: nothing draws, and load() rejects with code 'context-lost' */+ get contextLost(): boolean { return this.lost || !!this.gl?.isContextLost() }++ /** the loaded character (null before the first load) */+ get model(): Info | null { return this.info }+ get expressions() { return this.info?.expressions ?? [] as Info['expressions'] }+ get playing() { return this.raf !== 0 }++ /** runs a load: the one running is called off, and this one's steps are told how to be. A load called off rejects like+ * any other, but nobody has to catch it (player.load(a); player.load(b) reports nothing); a real error is still+ * reported when nobody catches it+ * @internal */+ protected start<T>({ signal, onProgress, poster }: LoadOptions, load: (job: LoadJob) => Promise<T>): Promise<T> {+ let give!: (v: T) => void, refuse!: (e: unknown) => void+ const out = new Promise<T>((a, b) => { give = a; refuse = b })+ const fail = (e: unknown) => {+ if ((e as Error | null)?.name === 'AbortError') out.catch(() => {})+ refuse(e)+ }+ if (signal?.aborted) { fail(signal.reason); return out }+ if (this.disposed) { fail(aborted('the player was disposed')); return out }+ if (this.contextLost) { fail(contextGone()); return out }+ this.running?.abort(aborted('a newer load took its place'))+ const ctl = this.running = new AbortController()+ const calledOff = () => ctl.abort(signal!.reason)+ signal?.addEventListener('abort', calledOff, { once: true })+ this.setPoster(poster)+ const tell = (p: LoadProgress) => { try { onProgress?.(p) } catch (e) { console.error(e) } }+ let shown = -1, built = false+ const job: LoadJob = {+ signal: ctl.signal,+ progress(share) { if (share > shown && !built) { shown = share; tell({ step: 'download', progress: share }) } },+ build() { if (!built) { if (shown >= 0 && shown < 1) tell({ step: 'download', progress: 1 }); built = true; tell({ step: 'build' }) } },+ }+ const end = () => {+ signal?.removeEventListener('abort', calledOff)+ if (this.running === ctl) this.running = null+ }+ load(job).then((v) => { end(); give(v) }, (e) => {+ const mine = this.running === ctl+ end()+ if (mine) this.setPoster() // the page can say why; a newer load has already set its own+ fail(ctl.signal.aborted ? ctl.signal.reason : e)+ })+ return out+ }++ /** the poster over an empty canvas, or none */+ private setPoster(url?: string) {+ const s = this.canvas.style+ if (this.posterOver) { for (const [k, v] of Object.entries(this.posterOver)) s.setProperty(k, v); this.posterOver = null }+ if (!url || this.info) return+ const set = { 'background-image': `url(${JSON.stringify(String(url))})`, 'background-position': 'center', 'background-size': 'contain', 'background-repeat': 'no-repeat' }+ this.posterOver = Object.fromEntries(Object.keys(set).map((k) => [k, s.getPropertyValue(k)]))+ for (const [k, v] of Object.entries(set)) s.setProperty(k, v)+ }++ /** runs `load` after the loads before it+ * @internal */+ protected inTurn<T>(load: () => Promise<T>): Promise<T> {+ const run = this.queue.then(load)+ this.queue = run.catch(() => {})+ return run+ }++ /** a new character starts on a neutral face+ * @internal */+ protected neutral() { this.expr = {}; this.smile = 0 }++ /** Shows an expression (an id from `expressions`), easing over from the one before; null goes back to neutral.+ * weight: how strongly, 0..1. False when the model has no such expression */+ setExpression(id: string | null, weight = 1): boolean {+ if (id !== null && !this.expressions.some((e) => e.id === id)) return false+ const w = clamp(weight, 0, 1)+ this.smile = id === 'smile' ? w : 0+ this.expr = id === null || id === 'smile' ? {} : { [id]: w }+ return true+ }++ /** Plays a short move over whatever drives the character: nod, shake, tilt, hop, shy */+ act(action: ActionId) {+ if ((action as string) === 'away') throw new TypeError("LivLiv: act('away') is setAway(true) now")+ this.actions.play(action)+ }++ /** Dozes (eyes closed, head down, a slow breath, expressions hidden), easing in, until setAway(false) */+ setAway(on: boolean) { this.actions.setAway(on) }++ /** Looks toward (x, y), each -1..1 from the canvas center (y up), instead of following the pointer; null goes back to+ * following it. Needs the idle motion on */+ lookAt(x: number | null, y = 0) {+ this.idler.lookAt(this.seen(), x === null ? null : clamp(x, -1, 1), clamp(y, -1, 1), this.canvas.clientWidth, this.canvas.clientHeight)+ }++ /** Blinks now (with the idle motion on) */+ blink() { this.idler.blink(this.clock) }++ /** Adds a driver; returns a function that removes it. On a disposed player it is disposed at once: what started+ * while the player went (a camera, a microphone) does not stay on */+ use(driver: Driver): () => void {+ if (this.disposed) { driver.dispose?.(); return () => {} }+ this.drivers.push(driver)+ this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))+ return () => {+ const i = this.drivers.indexOf(driver)+ if (i >= 0) { this.drivers.splice(i, 1); driver.dispose?.() }+ }+ }++ play() {+ if (this.raf || this.disposed) return+ this.last = performance.now()+ const loop = (now: number) => {+ this.raf = requestAnimationFrame(loop)+ const dt = Math.min(0.05, (now - this.last) / 1000)+ this.last = now+ if (this.visible) this.draw(dt)+ }+ this.raf = requestAnimationFrame(loop)+ }++ pause() { cancelAnimationFrame(this.raf); this.raf = 0 }++ /** Draws now at the canvas's current size. A playing player follows the size by itself; call this after resizing a+ * paused one */+ resize() { this.draw(0) }++ /** The current frame as an image (PNG data URL), as the pose is now, the mark included */+ snapshot(): string { this.draw(0); return this.canvas.toDataURL('image/png') }++ /** Stops, removes the drivers and frees the GPU memory. A load still running rejects. The canvas keeps its WebGL context+ * and can take a new player afterwards. release: also give the context up, when the canvas is leaving the page: a+ * browser allows about 16 at once and drops the oldest past that, which may be a character still showing. A canvas+ * released this way takes no new player, unless it is made at once, in the same task (a framework's unmount and mount).+ * Safari does not get the room back: reuse canvases there */+ dispose({ release = false }: { release?: boolean } = {}) {+ const gl = this.gl+ this.stop()+ if (release) releaseCanvas(this.canvas, gl)+ }++ private stop() {+ if (this.disposed) return+ this.pause()+ this.disposed = true+ this.running?.abort(aborted('the player was disposed'))+ this.setPoster()+ removeEventListener('pointermove', this.onMove)+ document.removeEventListener('pointerleave', this.onLeave)+ this.canvas.removeEventListener('webglcontextlost', this.onLost)+ this.io?.disconnect()+ for (const d of this.drivers.splice(0)) d.dispose?.()+ this.free()+ this.info = null+ }++ private draw(dt: number) {+ // (the context itself, not only the event, which comes a little later: three.js throws starting on a lost context)+ if (!this.drawable || this.disposed || this.contextLost) return+ const f = this.frame, P = f.params, t = this.clock, E = this.shownExpr, A = this.actions+ // the pose from scratch: your values, the idle motion, the drivers+ this.idler.shown(P)+ Object.assign(P, this.params)+ if (this.idle) {+ this.idler.apply(P, t, dt, { busy: A.busy || this.was.mouth > 0.05, tracked: this.was.tracked,+ faceFree: !this.smile && !Object.values(E).some((v) => v > 0.02) })+ }+ Object.assign(f, { dt, time: t, tracked: false })+ Object.assign(f.move, { x: 0, y: 0, s: 1 })+ for (const d of this.drivers) d.update(f, this)+ this.was = { tracked: f.tracked, mouth: P.mouthOpen }+ f.add(A.update(dt))+ if (this.smile) P.smile = Math.max(P.smile, this.smile)+ if (A.awayW) for (const [k, v] of Object.entries(A.awayPose(t)) as [keyof Params, number][]) P[k] += (v - P[k]) * A.awayW+ // in range whoever set it; a value that is no number (one NaN stays in the springs for good) counts as not set+ for (const k of KEYS) {+ const v = +P[k]+ P[k] = v === v ? clamp(v, PARAM_RANGES[k][0], PARAM_RANGES[k][1]) : DEFAULTS[k]+ }+ // expressions ease in and out (one no longer shown eases back to 0); dozing hides them, each is a face with open eyes+ for (const k of Object.keys(this.expr)) if (!(k in E)) E[k] = 0+ for (const k of Object.keys(E)) E[k] += ((this.expr[k] ?? 0) * (1 - A.awayW) - E[k]) * Math.min(1, dt * 8)+ this.show(dt)+ if (this.posterOver) this.setPoster()+ }++ private onMove = (e: PointerEvent) => {+ if (!this.followPointer) return+ const r = this.canvas.getBoundingClientRect()+ // 'canvas': off it the pointer has left the stage, and the character looks back at the viewer+ const inside = !(e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom)+ this.idler.pointer(this.seen(), e.clientX - r.left, e.clientY - r.top, r.width, r.height, this.followPointer === 'page' || inside)+ }+ private onLeave = () => this.idler.leave()+ private onLost = () => {+ this.lost = true+ this.running?.abort(contextGone())+ this.onContextLost?.()+ }+}diff --git a/src/body.ts b/src/body.tsindex abe6fe8..24acc69 100644--- a/src/body.ts+++ b/src/body.ts@@ -3,11 +3,14 @@// from Player: the idle motion, expressions, actions and drivers, so lip sync (@livliv/player/voice) and face tracking// (@livliv/player/tracking) work for it the same way. Each frame the pose is put together as Player does it (your params,// the idle motion, the drivers, actions, expressions); the Body turns it into its own picture.-import { ActionPlayer, type ActionId } from './runtime/actions'-import { pickLabel, uiLang, type Labels } from './runtime/format'-import { PARAM_RANGES, defaultParams, type Params } from './runtime/params'-import { Idle, type Seen } from './follow'-import type { Driver, Frame, Host, View } from './index'+import { pickLabel, type Labels } from './runtime/format'+import { defaultParams, type Params } from './runtime/params'+import { LivLivError, PlayerBase, aborted, contextGone, readBody, type LoadJob, type LoadOptions, type PlayerOptions } from './base'+import type { Seen } from './follow'+import { claimCanvas } from './release'++export { LivLivError }+export type { ActionId, Driver, Frame, Host, LivLivErrorCode, LoadOptions, LoadProgress, PlayerOptions, View } from './base'/** an expression a Body can show: `labels` its names by language (BCP 47 tag -> name), `label` the one to fall back on */export interface BodyExpression { id: string; label: string; labels?: Labels }@@ -25,6 +28,9 @@ export interface Body {/** loads from its files (by file name); its name and the expressions it can show. If it rejects, the character before* must stay as it is */load(files: Map<string, Blob>): Promise<{ name: string; expressions: BodyExpression[]; meta?: BodyMeta }>+ /** optional: the WebGL context it draws with, once it has one: what lets the player say when it is lost and give it up+ * (dispose({ release: true })). A Body without WebGL leaves this out */+ context?(): WebGLRenderingContext | WebGL2RenderingContext | null/** draws the pose into the canvas, w × h device pixels, dt seconds after the last frame */render(canvas: HTMLCanvasElement, pose: Pose, w: number, h: number, dt: number): void/** where the head is, in shares of the canvas (from the top left) and its radii */@@ -50,223 +56,110 @@ export interface BodyInfo {* or URL (`{ closed: 'me-closed.png', open: 'me-open.png' }`). Files are known by their name, without folders */export type BodySource = string | URL | Blob | FileList | Iterable<File> | Map<string, Blob | string> | Record<string, Blob | string>-export interface BodyPlayerOptions {- /** move by itself: breathe, blink, sway, look around (default true) */- idle?: boolean- /** the eyes and head follow the pointer anywhere on the page ('page', the default), over the canvas, or not */- followPointer?: 'page' | 'canvas' | false- /** drawing-buffer pixels per CSS pixel at most (default 2) */- maxPixelRatio?: number- /** start the frame loop at once (default true) */- autoplay?: boolean- /** the language expression names are picked in (default the browser's) */- lang?: string-}+export type BodyPlayerOptions = PlayerOptions/** the most a body's files may hold in all (a VRM is usually 5-50 MB) */export const BODY_LIMIT = 256 * 2 ** 20const SMILE: Labels = { en: 'Smile', zh: '笑', 'zh-Hant': '笑', ja: '笑顔' }-const clamp = (v: number, a: number, b: number) => Math.min(b, Math.max(a, v))/** Plays a character a Body draws, with the API of Player where it applies (no mark, no `puppet`, no `view.head`) */-export class BodyPlayer implements Host {- readonly canvas: HTMLCanvasElement+export class BodyPlayer extends PlayerBase<BodyInfo> {readonly body: Body- /** your values for the pose and face (ranges in PARAM_RANGES). The player never changes them */- readonly params: Params = defaultParams()- idle: boolean- followPointer: 'page' | 'canvas' | false- /** x, y move the character by shares of the canvas (y up), zoom scales it (`head` is not used) */- readonly view: View = { x: 0, y: 0, zoom: 1, head: false }+ /** see PlayerOptions.maxPixelRatio */maxPixelRatio: number- readonly lang: string-- private info: BodyInfo | null = null- private pose: Pose = { params: defaultParams(), move: { x: 0, y: 0, s: 1 }, expr: {} }- private idler = new Idle()- /** last frame's: a face tracker drove it, the mouth moved */- private was = { tracked: false, mouth: 0 }- private actions = new ActionPlayer()- private drivers: Driver[] = []- private expr: Record<string, number> = {}- private smile = 0+ private pose: Poseprivate time = 0- private raf = 0- private last = 0- private visible = true- private io?: IntersectionObserver- private queue: Promise<unknown> = Promise.resolve()- private disposed = false- private frame: Frame & { move: { x: number; y: number; s: number } }constructor(canvas: HTMLCanvasElement, body: Body, options: BodyPlayerOptions = {}) {- this.canvas = canvas+ claimCanvas(canvas)+ const pose: Pose = { params: defaultParams(), move: { x: 0, y: 0, s: 1 }, expr: {} }+ super(canvas, pose.params, options)+ this.pose = posethis.body = body- this.idle = options.idle ?? true- this.followPointer = options.followPointer ?? 'page'this.maxPixelRatio = options.maxPixelRatio ?? 2- this.lang = options.lang || uiLang()- const P = this.pose.params- this.frame = {- params: P, dt: 0, time: 0, tracked: false, move: { x: 0, y: 0, s: 1 },- add(o) {- for (const [k, v] of Object.entries(o) as [keyof Params, number | undefined][]) {- if (v) P[k] = clamp(P[k] + v, PARAM_RANGES[k][0], PARAM_RANGES[k][1])- }- },- }- addEventListener('pointermove', this.onMove)- document.addEventListener('pointerleave', this.onLeave)- if (typeof IntersectionObserver !== 'undefined') {- this.io = new IntersectionObserver((e) => { this.visible = e[e.length - 1].isIntersecting })- this.io.observe(canvas)- }- if (options.autoplay ?? true) this.play()}- /** the loaded character (null before the first load) */- get model(): BodyInfo | null { return this.info }- get expressions() { return this.info?.expressions ?? [] }- get playing() { return this.raf !== 0 }+ /** the Body's own word for it: asking the canvas would start a context on one that may be meant for 2D+ * @internal */+ protected get gl() { return this.body.context?.() ?? null }+ /** @internal */+ protected get drawable() { return !!this.info }+ /** @internal */+ protected get clock() { return this.time }+ /** @internal */+ protected get shownExpr() { return this.pose.expr }++ /** the body's head in the terms the idle's pointAt reads a puppet in (its canvas shares as clip space)+ * @internal */+ protected seen(): Seen {+ const hd = this.info ? this.body.head() : { x: 0.5, y: 0.3, rx: 0.15, ry: 0.12 }+ return { model: { anchors: { head: { cx: 2 * hd.x - 1, cy: 2 * hd.y - 1, rx: 2 * hd.rx, ry: 2 * hd.ry } } } as Seen['model'], lastView: [1, 1, 0, 0, 2, 2] }+ }- /** Loads a character from its files (see BodySource) in place of the one before (a failed load keeps it). Loads run one- * after another */- load(source: BodySource): Promise<BodyInfo> {- const run = this.queue.then(async () => {- const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }- stillHere()+ /** Loads a character from its files (see BodySource) in place of the one before (a failed load keeps it). A new load+ * replaces one still downloading or waiting, which rejects with an AbortError (see LoadOptions); one whose character is+ * already going in finishes first, since a Body swaps it inside, and the newer one then follows */+ load(source: BodySource, options: LoadOptions = {}): Promise<BodyInfo> {+ return this.start(options, (job) => this.inTurn(async () => {+ const { signal } = job+ if (signal.aborted) throw signal.reasonthis.body.attach?.(this.canvas)- const files = await filesOf(source)- stillHere()+ const files = await filesOf(source, job)+ if (signal.aborted) throw signal.reason+ job.build()const got = await this.body.load(files)- if (this.disposed) this.body.dispose() // disposed while it loaded: nothing of it may stay- stillHere()+ if (this.disposed) { this.body.dispose(); throw aborted('the player was disposed') } // disposed while it loaded: nothing of it may stay+ if (this.contextLost) throw contextGone()const ex = [{ id: 'smile', label: '', labels: SMILE }, ...got.expressions.filter((e) => e.id !== 'smile').map((e) => ({ id: e.id, label: e.label, labels: e.labels ?? { en: e.label } }))]this.info = { name: got.name, format: this.body.format, meta: got.meta ?? null,expressions: ex.map((e) => ({ id: e.id, label: pickLabel(e.labels, this.lang, e.label || e.id), labels: e.labels })) }- this.expr = {}; this.smile = 0; this.pose.expr = {}- if (!this.playing) this.draw(0)+ this.neutral(); this.pose.expr = {}+ if (!this.playing) this.resize()return this.info- })- this.queue = run.catch(() => {})- return run- }-- /** as Player's: an expression by id (smile is the smile param), easing over; null back to neutral */- setExpression(id: string | null, weight = 1): boolean {- if (id !== null && !this.expressions.some((e) => e.id === id)) return false- const w = clamp(weight, 0, 1)- this.smile = id === 'smile' ? w : 0- this.expr = id === null || id === 'smile' ? {} : { [id]: w }- return true- }-- act(action: ActionId) { this.actions.play(action) }- lookAt(x: number | null, y = 0) {- this.idler.lookAt(this.seen(), x === null ? null : clamp(x, -1, 1), clamp(y, -1, 1), this.canvas.clientWidth, this.canvas.clientHeight)- }- blink() { this.idler.blink(this.time) }-- use(driver: Driver): () => void {- if (this.disposed) { driver.dispose?.(); return () => {} } // as Player's: what started meanwhile does not stay on- this.drivers.push(driver)- this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))- return () => {- const i = this.drivers.indexOf(driver)- if (i >= 0) { this.drivers.splice(i, 1); driver.dispose?.() }- }+ }))}- play() {- if (this.raf || this.disposed) return- this.last = performance.now()- const loop = (now: number) => {- this.raf = requestAnimationFrame(loop)- const dt = Math.min(0.05, (now - this.last) / 1000)- this.last = now- if (this.visible) this.draw(dt)- }- this.raf = requestAnimationFrame(loop)- }-- pause() { cancelAnimationFrame(this.raf); this.raf = 0 }- /** draws now at the canvas's current size */- resize() { this.draw(0) }- /** the current frame as a PNG data URL */- snapshot(): string { this.draw(0); return this.canvas.toDataURL('image/png') }-- dispose() {- if (this.disposed) return- this.pause()- this.disposed = true- removeEventListener('pointermove', this.onMove)- document.removeEventListener('pointerleave', this.onLeave)- this.io?.disconnect()- for (const d of this.drivers.splice(0)) d.dispose?.()- this.body.dispose()- this.info = null- }-- private draw(dt: number) {- if (!this.info) return- this.time += dt- const P = this.pose.params, f = this.frame- this.idler.shown(P)- Object.assign(P, this.params)- if (this.idle) {- this.idler.apply(P, this.time, dt, { busy: this.actions.busy || this.was.mouth > 0.05, tracked: this.was.tracked,- faceFree: !this.smile && !Object.values(this.pose.expr).some((v) => v > 0.02) })- }- Object.assign(f, { dt, time: this.time, tracked: false })- Object.assign(f.move, { x: 0, y: 0, s: 1 })- for (const d of this.drivers) d.update(f, this)- this.was = { tracked: f.tracked, mouth: P.mouthOpen }- const A = this.actions- f.add(A.update(dt))- if (this.smile) P.smile = Math.max(P.smile, this.smile)- if (A.awayW) for (const [k, v] of Object.entries(A.awayPose(this.time)) as [keyof Params, number][]) P[k] += (v - P[k]) * A.awayW- for (const k of Object.keys(P) as (keyof Params)[]) P[k] = clamp(P[k], PARAM_RANGES[k][0], PARAM_RANGES[k][1])- // expressions ease in and out; dozing hides them- const E = this.pose.expr- for (const k of Object.keys(this.expr)) if (!(k in E)) E[k] = 0- for (const k of Object.keys(E)) E[k] += ((this.expr[k] ?? 0) * (1 - A.awayW) - E[k]) * Math.min(1, dt * 8)- const V = this.view- this.pose.move = { x: f.move.x + V.x, y: f.move.y + A.y + V.y, s: f.move.s * V.zoom }+ /** @internal */+ protected show(dt: number) {+ const f = this.frame, V = this.view+ this.pose.move = { x: f.move.x + V.x, y: f.move.y + this.actions.y + V.y, s: f.move.s * V.zoom }const dpr = Math.min(this.maxPixelRatio, window.devicePixelRatio || 1)const w = Math.round(this.canvas.clientWidth * dpr), h = Math.round(this.canvas.clientHeight * dpr)if (w >= 2 && h >= 2) this.body.render(this.canvas, this.pose, w, h, dt)+ this.time += dt}- /** the body's head in the terms the idle's pointAt reads a puppet in (its canvas shares as clip space) */- private seen(): Seen {- const hd = this.info ? this.body.head() : { x: 0.5, y: 0.3, rx: 0.15, ry: 0.12 }- return { model: { anchors: { head: { cx: 2 * hd.x - 1, cy: 2 * hd.y - 1, rx: 2 * hd.rx, ry: 2 * hd.ry } } } as Seen['model'], lastView: [1, 1, 0, 0, 2, 2] }- }- private onMove = (e: PointerEvent) => {- if (!this.followPointer) return- const r = this.canvas.getBoundingClientRect()- const inside = !(e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom)- this.idler.pointer(this.seen(), e.clientX - r.left, e.clientY - r.top, r.width, r.height, this.followPointer === 'page' || inside)- }- private onLeave = () => this.idler.leave()+ /** @internal */+ protected free() { this.body.dispose() }}/** a body's files by name, URLs fetched */-async function filesOf(source: BodySource): Promise<Map<string, Blob>> {+async function filesOf(source: BodySource, job: LoadJob): Promise<Map<string, Blob>> {const out = new Map<string, Blob>()let total = 0+ // files given by URL come one after another: each is its share of the download+ const given = source instanceof Map ? [...source.values()] : typeof source === 'object' && !(source instanceof URL) && !(Symbol.iterator in source) && !(source instanceof Blob) ? Object.values(source) : []+ const urls = typeof source === 'string' || source instanceof URL ? 1 : given.filter((v) => typeof v === 'string').length+ let came = 0const add = async (key: string, v: Blob | string) => {const name = key.replace(/\\/g, '/').split('/').filter(Boolean).pop() || 'file'let b: Blobif (typeof v === 'string') {const url = new URL(v, document.baseURI).href- const r = await fetch(url).catch(() => { throw new Error(`LivLiv: could not load ${url}: unreachable, or on another origin that sends no CORS header`) })- if (!r.ok) throw new Error(`LivLiv: could not load ${url} (HTTP ${r.status})`)- b = await r.blob()+ job.progress(came / urls)+ const unreachable = (e: unknown) => {+ if (job.signal.aborted) throw e+ throw new LivLivError('unreachable', `LivLiv: could not load ${url}: unreachable, or on another origin that sends no CORS header`, { cause: e })+ }+ const r = await fetch(url, { signal: job.signal }).catch(unreachable)+ if (!r.ok) throw new LivLivError('http', `LivLiv: could not load ${url} (HTTP ${r.status})`, { status: r.status })+ b = await readBody(r, job, (p) => (came + p) / urls).catch(unreachable)+ came+++ job.progress(came / urls)} else b = v- if ((total += b.size) > BODY_LIMIT) throw new Error(`LivLiv: the files are larger than ${BODY_LIMIT} bytes`)+ if ((total += b.size) > BODY_LIMIT) throw new LivLivError('too-large', `LivLiv: the files are larger than ${BODY_LIMIT} bytes`)out.set(name, b)}if (typeof source === 'string' || source instanceof URL) {diff --git a/src/errors.ts b/src/errors.tsnew file mode 100644index 0000000..6a727f2--- /dev/null+++ b/src/errors.ts@@ -0,0 +1,29 @@+// What a load rejects with, in a file of its own: a PNGTuber or a VRM page has it without the renderer.++/** what went wrong, to tell apart in code (the message is for people, and can change):+ * - no-webgl: this browser or canvas gives no WebGL2 (new Player(), a VRM)+ * - canvas-in-use: the canvas already draws in a way this character cannot (a PNGTuber on a WebGL canvas)+ * - unreachable: the file did not come: no connection, the connection dropped, or another origin without a CORS header+ * - http: the server said no (`status`)+ * - not-a-model: this is not a model at all, or not one for this player (a VRM handed to a Player)+ * - invalid: a model, but something in it is wrong or damaged+ * - too-large: over this player's limits, or the GPU's memory+ * - newer-format: written for a newer player+ * - missing-feature: needs something this page or player does not have+ * - missing-file: a file it names is not among the files+ * - context-lost: the canvas's WebGL context is gone (see `onContextLost`)+ * - released: the canvas's context was given up by `dispose({ release: true })`: use a new canvas */+export type LivLivErrorCode = 'no-webgl' | 'canvas-in-use' | 'unreachable' | 'http' | 'not-a-model' | 'invalid' | 'too-large'+ | 'newer-format' | 'missing-feature' | 'missing-file' | 'context-lost' | 'released'++export class LivLivError extends Error {+ readonly code: LivLivErrorCode+ /** with code 'http': the HTTP status */+ readonly status?: number+ constructor(code: LivLivErrorCode, message: string, options: { status?: number; cause?: unknown } = {}) {+ super(message, options.cause === undefined ? undefined : { cause: options.cause })+ this.name = 'LivLivError'+ this.code = code+ if (options.status !== undefined) this.status = options.status+ }+}diff --git a/src/index.ts b/src/index.tsindex adace25..63476b8 100644--- a/src/index.ts+++ b/src/index.ts@@ -2,43 +2,26 @@// livliv.ai. The renderer is livliv.ai's own (src/runtime/puppet.ts); this wraps it in a small API: load, a frame// loop with the idle motion, params, expressions, actions, the corner mark the package carries, and drivers. Face// tracking and lip sync are separate imports ('@livliv/player/tracking', '@livliv/player/voice'), so the core stays small.-//-// Each frame the pose is put together from scratch: your `params`, then the idle motion (head, eyes, brows, body,-// breath; follow.ts: livliv.ai's, following the pointer with the eyes and the face), then the drivers in order, then-// actions and expressions. Nothing the player adds is written back to-// `params`, so a value you set stays until you change it, and turning a driver off leaves no trace.-import { ActionPlayer, type ActionId } from './runtime/actions'-import { langOf, pickLabel, readModel, uiLang, type Labels, type ModelMeta } from './runtime/format'+import { langOf, pickLabel, readModel, type Labels, type ModelMeta } from './runtime/format'import { FORMAT, FORMAT_VERSION, LEGACY_FORMAT, PARAM_RANGES, Puppet, defaultParams, loadBlob, type KeyformFile, type ModelDef, type Params } from './runtime/puppet'-import { Idle } from './follow'+import { LivLivError, PlayerBase, aborted, contextGone, readBody, type LivLivErrorCode, type LoadJob, type LoadOptions, type PlayerOptions } from './base'import { Mark } from './mark'+import { claimCanvas } from './release'import { ZIP_LIMITS, isZip, unzip } from './zip'export { FORMAT, FORMAT_VERSION, LEGACY_FORMAT, PARAM_RANGES, defaultParams }-export type { ActionId, Labels, ModelDef, ModelMeta, Params }+export type { Labels, ModelDef, ModelMeta, Params }+export { LivLivError }+export type { ActionId, Driver, Frame, Host, LivLivErrorCode, LoadOptions, LoadProgress, PlayerOptions, View } from './base'/** this package's version */-export const VERSION = '0.2.1'+export const VERSION = '0.3.0'/** where a model comes from: the URL of its model.json, of the folder holding it, or of a .livliv package (relative URLs* resolve against the page); a .livliv package itself (a File or Blob: a ZIP of the folder); or its files: a FileList /* File[] from an <input type="file"> (a folder with webkitdirectory, or one picked .livliv), or a map of path -> Blob */export type ModelSource = string | URL | Blob | FileList | Iterable<File> | Map<string, Blob> | Record<string, Blob>-export interface PlayerOptions {- /** move by itself: breathe, blink, sway, look around (default true). Off: only your params and drivers move it */- idle?: boolean- /** the eyes and head follow the pointer anywhere on the page ('page', the default), only over the canvas- * ('canvas'), or not at all (false) */- followPointer?: 'page' | 'canvas' | false- /** drawing-buffer pixels per CSS pixel at most (default 2; 1 saves power on high-DPI screens) */- maxPixelRatio?: number- /** start the frame loop at once (default true) */- autoplay?: boolean- /** the language expression names are picked in (`expressions[].label`; a BCP 47 tag, default the browser's) */- lang?: string-}-export interface ModelInfo {name: string/** model.json's format: FORMAT ("livliv-model"), or LEGACY_FORMAT ("livemaker-puppet") for packages exported before v4 */@@ -59,77 +42,15 @@ export interface ModelInfo {/** the build's smile on a model whose list was never written (packages from before 2026-10) */const SMILE: Labels = { zh: '笑', 'zh-Hant': '笑', en: 'Smile', ja: '笑顔' }-/** where the character sits in the canvas: x, y move it by shares of the canvas (y up), zoom scales it about the- * center; head: frame the head instead of the whole drawing (avatars, chat bubbles) */-export interface View { x: number; y: number; zoom: number; head: boolean }--/** one frame being put together, as a driver sees it */-export interface Frame {- /** the pose so far: write to it (for this frame only) */- readonly params: Params- /** seconds since the last frame, and since the model loaded */- readonly dt: number- readonly time: number- /** a face tracker drives the head this frame (lip sync then leaves the head alone) */- tracked: boolean- /** add to params, kept in their ranges */- add(offsets: Partial<Params>): void- /** move the picture: x, y shares of the canvas (y up), s a scale */- readonly move: { x: number; y: number; s: number }-}--/** something that moves the character each frame: face tracking and lip sync are drivers, and your own code can be */-export interface Driver {- /** lower runs first: face tracking -10, yours 0 (default), lip sync 10 */- order?: number- update(frame: Frame, player: Host): void- /** called when it is removed, or the player disposed */- dispose?(): void-}--/** what a driver, the lip sync and the face tracking need of a player: a Player, or a BodyPlayer (@livliv/player/body) */-export interface Host {- /** adds a driver; returns a function that removes it */- use(driver: Driver): () => void- /** blinks now (with the idle motion on) */- blink(): void-}--/** what the runtime throws when a canvas gives it no WebGL2 context */const NO_WEBGL2 = 'this browser has no WebGL2'/** Plays one LivLiv model on a canvas. Size the canvas with CSS; the drawing follows its size. */-export class Player implements Host {- readonly canvas: HTMLCanvasElement- /** the renderer underneath, for what this API does not cover. Its fields can change between versions */+export class Player extends PlayerBase<ModelInfo> {+ /** the renderer underneath: not part of the API, and not in the types+ * @internal */readonly puppet: Puppet- /** your values for the pose and face (ranges in PARAM_RANGES). The player never changes them */- readonly params: Params = defaultParams()- /** see PlayerOptions.idle */- idle: boolean- /** see PlayerOptions.followPointer */- followPointer: 'page' | 'canvas' | false- readonly view: View = { x: 0, y: 0, zoom: 1, head: false }- /** see PlayerOptions.lang */- readonly lang: string-- private info: ModelInfo | null = nullprivate mark: Mark- private idler = new Idle()- /** last frame's: a face tracker drove it, the mouth moved (the idle starts no move while the character talks) */- private was = { tracked: false, mouth: 0 }- private actions = new ActionPlayer()- private drivers: Driver[] = []- private expr: Record<string, number> = {}- private smile = 0- private raf = 0- private last = 0- private visible = true- private io?: IntersectionObserver- private queue: Promise<unknown> = Promise.resolve()- private disposed = false- private frame: Frame & { move: { x: number; y: number; s: number } }/** Whether this browser can play a LivLiv model (WebGL2 starts on a new canvas). Where it cannot, new Player() throws */static isSupported(): boolean {@@ -139,72 +60,74 @@ export class Player implements Host {}constructor(canvas: HTMLCanvasElement, options: PlayerOptions = {}) {- this.canvas = canvas+ claimCanvas(canvas)+ let puppet: Puppettry {- this.puppet = new Puppet(canvas)+ puppet = new Puppet(canvas)} catch (e) {- throw new Error(typeof WebGL2RenderingContext === 'undefined' ? 'LivLiv: this browser has no WebGL2'+ if (canvas.getContext('webgl2')?.isContextLost()) throw new LivLivError('context-lost', 'LivLiv: this canvas has lost its WebGL context: use a new canvas', { cause: e })+ throw new LivLivError('no-webgl', typeof WebGL2RenderingContext === 'undefined' ? 'LivLiv: this browser has no WebGL2': (e as Error).message === NO_WEBGL2 ? 'LivLiv: could not start WebGL2 on this canvas: it is turned off in this browser, or the canvas already draws in 2D'- : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`)+ : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`, { cause: e })}+ super(canvas, puppet.params, options)+ this.puppet = puppetthis.puppet.maxDpr = options.maxPixelRatio ?? 2this.mark = new Mark(this.puppet.gl)- this.idle = options.idle ?? true- this.followPointer = options.followPointer ?? 'page'- this.lang = options.lang || uiLang()- const P = this.puppet.params- this.frame = {- params: P, dt: 0, time: 0, tracked: false, move: { x: 0, y: 0, s: 1 },- add(o) {- for (const [k, v] of Object.entries(o) as [keyof Params, number | undefined][]) {- if (v) P[k] = Math.max(PARAM_RANGES[k][0], Math.min(PARAM_RANGES[k][1], P[k] + v))- }- },- }- addEventListener('pointermove', this.onMove)- document.addEventListener('pointerleave', this.onLeave)- if (typeof IntersectionObserver !== 'undefined') {- this.io = new IntersectionObserver((e) => { this.visible = e[e.length - 1].isIntersecting })- this.io.observe(canvas)- }- if (options.autoplay ?? true) this.play()}- /** the loaded model (null before the first load) */- get model(): ModelInfo | null { return this.info }- get expressions() { return this.info?.expressions ?? [] }- get playing() { return this.raf !== 0 }+ /** see PlayerOptions.maxPixelRatio */+ get maxPixelRatio() { return this.puppet.maxDpr }+ set maxPixelRatio(v: number) { this.puppet.maxDpr = v }++ /** @internal */+ protected get gl() { return this.puppet.gl }+ /** @internal */+ protected get drawable() { return !!this.puppet.model }+ /** @internal */+ protected get clock() { return this.puppet.time }+ /** @internal */+ protected get shownExpr() { return this.puppet.expr }+ /** @internal */+ protected seen() { return this.puppet }++ /** A point on the canvas (CSS px from its top left) in the model's own px, as the last frame drew it */+ canvasToModel(x: number, y: number): [number, number] { return this.puppet.canvasToModel(x, y) }++ /** A flick at a point in the model's px (canvasToModel): the swinging pieces passing near it are thrown, the nearest+ * joint hardest (a tap on a twin tail). Returns how many it moved: none on a model without any */+ poke(x: number, y: number, strength = 1): number { return this.puppet.poke(x, y, strength) }- /** Loads a model (see ModelSource) and shows it in place of the one before. Loads run one after another */- load(source: ModelSource): Promise<ModelInfo> {- const run = this.queue.then(() => this.loadNow(source))- this.queue = run.catch(() => {})- return run+ /** Loads a model (see ModelSource) and shows it in place of the one before. A load that fails keeps it; a new load+ * replaces one still running, which rejects with an AbortError (see LoadOptions) */+ load(source: ModelSource, options: LoadOptions = {}): Promise<ModelInfo> {+ return this.start(options, (job) => this.loadNow(source, job))}- private async loadNow(source: ModelSource): Promise<ModelInfo> {- const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }- stillHere()- const p = this.puppet+ private async loadNow(source: ModelSource, job: LoadJob): Promise<ModelInfo> {+ const p = this.puppet, { signal } = joblet files: Exclude<ModelSource, string | URL> | null = nullif (typeof source === 'string' || source instanceof URL) {const url = new URL(String(source), document.baseURI)- if (/\.(livliv|zip)$/i.test(url.pathname)) { files = await fetchPackage(url.href); stillHere() } else {- try {- await p.load(url.href, { pack: false })- } catch (e) {- throw explain(e, url.href)- }+ if (/\.(livliv|zip)$/i.test(url.pathname)) {+ job.progress(0)+ files = await fetchPackage(url.href, job)+ if (signal.aborted) throw signal.reason+ } else {+ job.progress(0)+ await p.load(url.href, { pack: false, signal, onImage: (came, of) => { job.progress(came / of); if (came === of) job.build() } }).catch((e) => { throw explain(e, url.href) })}} else files = sourceif (files) {+ job.build()const { model, image, keyforms } = await fromFiles(files)- stillHere()- await p.loadModel(model, image, { keyforms }).catch((e) => { throw explain(e, 'model.json') })+ if (signal.aborted) throw signal.reason+ await p.loadModel(model, image, { keyforms, signal }).catch((e) => { throw explain(e, 'model.json') })}- // disposed while the images came in: the runtime built its layers all the same; nothing of them may stay or be drawn- if (this.disposed) p.dispose()- stillHere()+ // the model is in: from here it is this load's, whatever came after it+ // (disposed in the moment since: nothing of it may stay or be drawn)+ if (this.disposed) { p.dispose(); throw aborted('the player was disposed') }+ if (p.gl.isContextLost()) throw contextGone()const m = p.modelconst ex = m.expressions || (m.layers.some((l) => l.id === 'expr_smile') ? [{ id: 'smile', label: '', labels: SMILE }] : [])this.info = {@@ -219,156 +142,64 @@ export class Player implements Host {// v4 says which mark it is; before it the text "LivLiv" meant the logoconst kind = m.meta ? m.meta.watermark?.kind ?? 'none' : m.watermark === 'LivLiv' ? 'livliv' : m.watermark ? 'text' : 'none'this.mark.set(kind, m.watermark ?? null, m.meta?.watermark?.center !== false)- this.expr = {}; this.smile = 0- if (!this.playing) this.draw(0)+ this.neutral()+ if (!this.playing) this.resize()return this.info}- /** Shows an expression (an id from `expressions`), easing over from the one before; null goes back to neutral.- * weight: how strongly, 0..1. False when the model has no such expression */- setExpression(id: string | null, weight = 1): boolean {- if (id !== null && !this.expressions.some((e) => e.id === id)) return false- const w = Math.max(0, Math.min(1, weight))- this.smile = id === 'smile' ? w : 0- this.expr = id === null || id === 'smile' ? {} : { [id]: w }- return true- }-- /** Plays a short move over whatever drives the character: nod, shake, tilt, hop, shy. 'away' toggles a dozing pose */- act(action: ActionId) { this.actions.play(action) }-- /** Looks toward (x, y), each -1..1 from the canvas center (y up), instead of following the pointer; null goes back to- * following it. Needs the idle motion on */- lookAt(x: number | null, y = 0) {- this.idler.lookAt(this.puppet, x === null ? null : clamp(x, -1, 1), clamp(y, -1, 1), this.canvas.clientWidth, this.canvas.clientHeight)- }-- /** Blinks now (with the idle motion on) */- blink() { this.idler.blink(this.puppet.time) }-- /** Adds a driver; returns a function that removes it. On a disposed player it is disposed at once: what started- * while the player went (a camera, a microphone) does not stay on */- use(driver: Driver): () => void {- if (this.disposed) { driver.dispose?.(); return () => {} }- this.drivers.push(driver)- this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))- return () => {- const i = this.drivers.indexOf(driver)- if (i >= 0) { this.drivers.splice(i, 1); driver.dispose?.() }- }- }-- play() {- if (this.raf || this.disposed) return- this.last = performance.now()- const loop = (now: number) => {- this.raf = requestAnimationFrame(loop)- const dt = Math.min(0.05, (now - this.last) / 1000)- this.last = now- if (this.visible) this.draw(dt)- }- this.raf = requestAnimationFrame(loop)- }-- pause() { cancelAnimationFrame(this.raf); this.raf = 0 }-- /** Draws now at the canvas's current size. A playing player follows the size by itself; call this after resizing a- * paused one */- resize() { this.draw(0) }-- /** The current frame as an image (PNG data URL), the mark included */- snapshot(): string { return this.canvas.toDataURL('image/png') }-- /** Stops, removes the drivers and frees the GPU memory. A load still running rejects. The canvas can take a new Player- * afterwards */- dispose() {- if (this.disposed) return- this.pause()- this.disposed = true- removeEventListener('pointermove', this.onMove)- document.removeEventListener('pointerleave', this.onLeave)- this.io?.disconnect()- for (const d of this.drivers.splice(0)) d.dispose?.()- const gl = this.puppet.gl- this.puppet.dispose(); this.mark.dispose(); gl.deleteProgram(this.puppet.prog)- gl.clearColor(0, 0, 0, 0); gl.clear(gl.COLOR_BUFFER_BIT)- this.info = null- }-- private draw(dt: number) {- const p = this.puppet, P = p.params, f = this.frame- if (!p.model || this.disposed) return- // the pose from scratch: your values, the idle motion, the drivers- this.idler.shown(P)- Object.assign(P, this.params)- if (this.idle) {- this.idler.apply(P, p.time, dt, { busy: this.actions.busy || this.was.mouth > 0.05, tracked: this.was.tracked,- faceFree: !this.smile && !Object.values(p.expr).some((v) => v > 0.02) })- }- Object.assign(f, { dt, time: p.time, tracked: false })- Object.assign(f.move, { x: 0, y: 0, s: 1 })- for (const d of this.drivers) d.update(f, this)- this.was = { tracked: f.tracked, mouth: P.mouthOpen }- const A = this.actions- f.add(A.update(dt))- if (this.smile) P.smile = Math.max(P.smile, this.smile)- if (A.awayW) for (const [k, v] of Object.entries(A.awayPose(p.time)) as [keyof Params, number][]) P[k] += (v - P[k]) * A.awayW- // expressions ease in and out (one no longer shown eases back to 0); dozing hides them, each is a face with open eyes- for (const k of Object.keys(this.expr)) if (!(k in p.expr)) p.expr[k] = 0- for (const k of Object.keys(p.expr)) p.expr[k] += ((this.expr[k] ?? 0) * (1 - A.awayW) - p.expr[k]) * Math.min(1, dt * 8)- const V = this.view+ /** @internal */+ protected show(dt: number) {+ const p = this.puppet, f = this.frame, V = this.viewp.view = { x: V.x, y: V.y, s: V.zoom }p.focusHead = V.head- p.move = { x: f.move.x, y: f.move.y + A.y, s: f.move.s }+ p.move = { x: f.move.x, y: f.move.y + this.actions.y, s: f.move.s }p.update(dt)p.render()this.mark.draw(performance.now(), p.gl.drawingBufferWidth / Math.max(1, this.canvas.clientWidth))}- private onMove = (e: PointerEvent) => {- if (!this.followPointer) return- const r = this.canvas.getBoundingClientRect()- // 'canvas': off it the pointer has left the stage, and the character looks back at the viewer- const inside = !(e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom)- this.idler.pointer(this.puppet, e.clientX - r.left, e.clientY - r.top, r.width, r.height, this.followPointer === 'page' || inside)+ /** @internal */+ protected free() {+ const gl = this.puppet.gl+ this.puppet.dispose(); this.mark.dispose(); gl.deleteProgram(this.puppet.prog)+ gl.clearColor(0, 0, 0, 0); gl.clear(gl.COLOR_BUFFER_BIT)}- private onLeave = () => this.idler.leave()}-const clamp = (v: number, a: number, b: number) => Math.min(b, Math.max(a, v))-/** errors in the words of someone embedding the player (the runtime's own are written for livliv.ai) */-function explain(e: unknown, where: string): Error {- const err = e as Error & { status?: number }- const msg = String(err?.message || e)- if (err?.status) return new Error(`LivLiv: could not load ${where} (HTTP ${err.status})`)+function explain(e: unknown, where: string): unknown {+ const err = e as Error & { status?: number; code?: LivLivErrorCode }+ if (err instanceof LivLivError || err?.name === 'AbortError') return e+ const msg = String(err?.message || e), cause = e+ if (err?.status) return new LivLivError('http', `LivLiv: could not load ${where} (HTTP ${err.status})`, { status: err.status, cause })if (msg.startsWith('image ')) {- return new Error(`LivLiv: could not load the image ${msg.slice(6)}: it is missing, or on another origin that sends no CORS header`)+ return new LivLivError('missing-file', `LivLiv: could not load the image ${msg.slice(6)}: it is missing, or on another origin that sends no CORS header`, { cause })}- if (err instanceof SyntaxError) return new Error(`LivLiv: ${where} is not a model.json (${msg})`)- if (err instanceof TypeError && /fetch/i.test(msg)) {- return new Error(`LivLiv: could not load ${where}: unreachable, or on another origin that sends no CORS header`)+ if (err instanceof SyntaxError) return new LivLivError('not-a-model', `LivLiv: ${where} is not a model.json (${msg})`, { cause })+ if (err instanceof TypeError && /fetch|load failed|network/i.test(msg)) {+ return new LivLivError('unreachable', `LivLiv: could not load ${where}: unreachable, or on another origin that sends no CORS header`, { cause })}- return new Error(msg.startsWith('LivLiv') ? msg : `LivLiv: ${msg}`)+ return new LivLivError(err?.code ?? 'invalid', msg.startsWith('LivLiv') ? msg : `LivLiv: ${msg}`, { cause })}/** a file a Player cannot take but a BodyPlayer can: the words that say so */-const VRM_HERE = 'LivLiv: this is a VRM or glTF file, and a Player plays LivLiv models: use a BodyPlayer with VrmBody (@livliv/player/vrm)'+const vrmHere = () => new LivLivError('not-a-model', 'LivLiv: this is a VRM or glTF file, and a Player plays LivLiv models: use a BodyPlayer with VrmBody (@livliv/player/vrm)')const isGlb = async (b: Blob) => { const d = new DataView(await b.slice(0, 4).arrayBuffer()); return d.byteLength === 4 && d.getUint32(0, true) === 0x46546c67 }/** a .livliv package from a URL, as a Blob. One file is one connection to lose where a folder has dozens, and a dropped one* breaks in the body as often as at the start: tried once more, unless the server said no (4xx). What the page gets is* our words, never the browser's ("Failed to fetch", "Load failed") */-async function fetchPackage(url: string): Promise<Blob> {+async function fetchPackage(url: string, job: LoadJob): Promise<Blob> {for (let again = true; ; again = false) {try {- const r = await fetch(url)+ const r = await fetch(url, { signal: job.signal })if (!r.ok) throw Object.assign(new Error(`HTTP ${r.status}`), { status: r.status })- return await r.blob()+ return await readBody(r, job)} catch (e) {+ if (job.signal.aborted) throw econst status = (e as { status?: number }).statusif (status && status < 500) throw explain(e, url)- if (!again) throw status ? explain(e, url) : new Error(`LivLiv: could not load ${url}: unreachable or the connection dropped, or on another origin that sends no CORS header`)+ if (!again) throw status ? explain(e, url) : new LivLivError('unreachable', `LivLiv: could not load ${url}: unreachable or the connection dropped, or on another origin that sends no CORS header`, { cause: e })}}}@@ -380,8 +211,8 @@ async function fromFiles(source: Exclude<ModelSource, string | URL>) {const files = new Map<string, Blob>()const norm = (k: string) => k.replace(/\\/g, '/').replace(/^(\.\/|\/)+/, '')const unpack = async (b: Blob) => {- if (await isGlb(b)) throw new Error(VRM_HERE)- if (!(await isZip(b))) throw new Error('LivLiv: this file is not a .livliv package (not a ZIP)')+ if (await isGlb(b)) throw vrmHere()+ if (!(await isZip(b))) throw new LivLivError('not-a-model', 'LivLiv: this file is not a .livliv package (not a ZIP)')return unzip(b).catch((e) => { throw explain(e, 'the package') })}if (source instanceof Blob) (await unpack(source)).forEach((b, k) => files.set(k, b))@@ -395,10 +226,10 @@ async function fromFiles(source: Exclude<ModelSource, string | URL>) {}}const json = [...files.keys()].filter(isJson).sort((a, b) => a.length - b.length)[0]- if (!json && [...files.keys()].some((k) => /\.(vrm|glb)$/i.test(k))) throw new Error(VRM_HERE)- if (!json) throw new Error('LivLiv: no model.json among the files')+ if (!json && [...files.keys()].some((k) => /\.(vrm|glb)$/i.test(k))) throw vrmHere()+ if (!json) throw new LivLivError('missing-file', 'LivLiv: no model.json among the files')const dir = json.slice(0, json.length - 'model.json'.length)- if (files.get(json)!.size > ZIP_LIMITS.modelJson) throw new Error(`LivLiv: ${json} is larger than ${ZIP_LIMITS.modelJson} bytes`)+ if (files.get(json)!.size > ZIP_LIMITS.modelJson) throw new LivLivError('too-large', `LivLiv: ${json} is larger than ${ZIP_LIMITS.modelJson} bytes`)let model: ModelDeftry {model = readModel(JSON.parse(await files.get(json)!.text()))@@ -411,7 +242,7 @@ async function fromFiles(source: Exclude<ModelSource, string | URL>) {const find = (path: string) => files.get(dir + norm(path)) ?? byName.get(path.slice(path.lastIndexOf('/') + 1))const image = (path: string) => {const b = find(path)- return b ? loadBlob(b) : Promise.reject(new Error(`LivLiv: ${path} is not among the files`))+ return b ? loadBlob(b) : Promise.reject(new LivLivError('missing-file', `LivLiv: ${path} is not among the files`))}let keyforms: KeyformFile | undefinedconst kf = model.keyforms ? find(model.keyforms) : undefined // read only within its limit (format.ts LIMITS)diff --git a/src/png.ts b/src/png.tsindex 2dc9f8c..7982877 100644--- a/src/png.ts+++ b/src/png.ts@@ -3,6 +3,7 @@// mouth open too). The picture sways with the head, tilts, breathes and hops with the voice, standing on the bottom of// the canvas. Files by name (any image the browser decodes): closed, open, blink, blink-open, and name.txt.import type { Body, Pose } from './body'+import { LivLivError } from './errors'export const PNG_SLOTS = [{ id: 'closed', label: 'Mouth closed', need: true },@@ -22,7 +23,7 @@ export class PngBody implements Body {attach(canvas: HTMLCanvasElement) {this.g = canvas.getContext('2d')- if (!this.g) throw new Error('LivLiv: this canvas already draws with WebGL (a Player or a VRM): a PNGTuber needs a canvas of its own')+ if (!this.g) throw new LivLivError('canvas-in-use', 'LivLiv: this canvas already draws with WebGL (a Player or a VRM): a PNGTuber needs a canvas of its own')}async load(files: Map<string, Blob>) {@@ -34,11 +35,11 @@ export class PngBody implements Body {for (const [file, blob] of files) {const slot = PNG_SLOTS.find((s) => file.replace(/\.[^.]*$/, '').toLowerCase() === s.id)?.idif (!slot) continue- const b = await createImageBitmap(blob).catch(() => { throw new Error(`LivLiv: ${file} is not an image this browser can read`) })+ const b = await createImageBitmap(blob).catch(() => { throw new LivLivError('invalid', `LivLiv: ${file} is not an image this browser can read`) })pics[slot] = b- if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) throw new Error(`LivLiv: ${file} is too large (${b.width} x ${b.height})`)+ if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) throw new LivLivError('too-large', `LivLiv: ${file} is too large (${b.width} x ${b.height})`)}- if (!pics.closed || !pics.open) throw new Error('LivLiv: a PNGTuber needs at least the closed and the open picture')+ if (!pics.closed || !pics.open) throw new LivLivError('missing-file', 'LivLiv: a PNGTuber needs at least the closed and the open picture')const txt = files.get('name.txt')if (txt) name = (await txt.slice(0, 200).text()).trim() || name} catch (e) {diff --git a/src/release.ts b/src/release.tsnew file mode 100644index 0000000..7d5f729--- /dev/null+++ b/src/release.ts@@ -0,0 +1,26 @@+// A canvas whose WebGL context a player gave up (dispose({ release: true })): the browser's room for contexts back, and the+// canvas spent: a lost context does not come back by itself, and a new renderer on it fails in ways that say nothing.+import { LivLivError } from './errors'++const spent = new WeakSet<HTMLCanvasElement>()+const going = new WeakMap<HTMLCanvasElement, () => void>()++/** gives up the canvas's context, one task from now: losing and restoring within one task does nothing in Chromium and+ * WebKit, and that is just what a framework's unmount and mount of the same canvas is (React's StrictMode). A player made+ * on the canvas before then keeps it (see claimCanvas). A canvas without a WebGL context has nothing to give up */+export function releaseCanvas(canvas: HTMLCanvasElement, gl: WebGLRenderingContext | WebGL2RenderingContext | null) {+ if (!gl || spent.has(canvas) || going.has(canvas)) return+ const timer = setTimeout(() => {+ going.delete(canvas)+ spent.add(canvas)+ gl.getExtension('WEBGL_lose_context')?.loseContext()+ })+ going.set(canvas, () => clearTimeout(timer))+}++/** a new player on this canvas: a release still to come is off; one that has happened cannot be undone */+export function claimCanvas(canvas: HTMLCanvasElement) {+ going.get(canvas)?.()+ going.delete(canvas)+ if (spent.has(canvas)) throw new LivLivError('released', 'LivLiv: this canvas gave up its WebGL context with dispose({ release: true }): use a new canvas')+}diff --git a/src/runtime/format.ts b/src/runtime/format.tsindex 0f16483..3705233 100644--- a/src/runtime/format.ts+++ b/src/runtime/format.ts@@ -41,6 +41,9 @@ export const LIMITS = {* the layers bent by chains times how many bend them. The heaviest of 111 models here: 46 000 and 40 000 */vertices: 2 ** 21, skinned: 2 ** 20,}+/** an error a player can tell apart by its `code` (the SDK's LivLivError); livliv.ai reads only the words */+export const coded = (code: string, message: string) => Object.assign(new Error(message), { code })+/** no number in a model comes near this; past it sums overflow to Infinity and then NaN, and the character is gone */const BIG = 1e6const GROUPS = ['head', 'body', 'hair_back', 'hair_side', 'hair_front', 'hair_inner']@@ -56,23 +59,23 @@ const read = new WeakSet<object>()* expressions' `label` from their names by language (all of them stay in `labels`). Reading a result again returns* it as it is */export function readModel(raw: unknown, lang = uiLang()): ModelDef {- if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw new Error('not a LivLiv model (model.json is not an object)')+ if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw coded('not-a-model', 'not a LivLiv model (model.json is not an object)')if (read.has(raw)) return raw as ModelDefconst m = raw as Record<string, unknown>if (m.format === undefined || m.format === LEGACY_FORMAT) {const v = (m.version as number | undefined) ?? LEGACY_VERSION- if (v > LEGACY_VERSION) throw new Error(`this model says format "${LEGACY_FORMAT}" version ${v}, which never existed`)+ if (v > LEGACY_VERSION) throw coded('not-a-model', `this model says format "${LEGACY_FORMAT}" version ${v}, which never existed`)check(m as unknown as ModelDef)read.add(m)return m as unknown as ModelDef}- if (m.format !== FORMAT) throw new Error(`not a LivLiv model (format "${String(m.format)}")`)+ if (m.format !== FORMAT) throw coded('not-a-model', `not a LivLiv model (format "${String(m.format)}")`)const v = m.version as numberif (typeof v !== 'number' || v > FORMAT_VERSION) {- throw new Error(`this model is format version ${v}; this player reads up to ${FORMAT_VERSION}: update the player`)+ throw coded('newer-format', `this model is format version ${v}; this player reads up to ${FORMAT_VERSION}: update the player`)}const missing = ((m.requires as string[] | undefined) || []).filter((f) => !FEATURES.includes(f))- if (missing.length) throw new Error(`this model needs ${missing.join(', ')}, which this player does not have: update the player`)+ if (missing.length) throw coded('missing-feature', `this model needs ${missing.join(', ')}, which this player does not have: update the player`)const out = lower(m as unknown as V4, lang)check(out)read.add(out)@@ -82,7 +85,7 @@ export function readModel(raw: unknown, lang = uiLang()): ModelDef {/** what any model must be before it is drawn, whatever wrote it: sizes and counts in range (LIMITS), numbers finite,* every image and file a path inside the package. Throws on the first thing wrong */function check(m: ModelDef) {- const bad = (why: string): never => { throw new Error(`this model is not valid: ${why}`) }+ const bad = (why: string, code = 'invalid'): never => { throw coded(code, `this model is not valid: ${why}`) }const num = (v: unknown): v is number => typeof v === 'number' && Number.isFinite(v)const list = (v: unknown, most: number, what: string) => {if (v === undefined) return []@@ -90,9 +93,9 @@ function check(m: ModelDef) {return v as unknown[]}const [w = NaN, h = NaN] = Array.isArray(m.canvas) ? m.canvas : []- if (!(num(w) && num(h) && w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) bad(`its canvas ${w} x ${h} is out of range`)+ if (!(num(w) && num(h) && w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) bad(`its canvas ${w} x ${h} is out of range`, w > LIMITS.canvas || h > LIMITS.canvas ? 'too-large' : 'invalid')if (!Array.isArray(m.layers) || !m.layers.length) bad('it has no layers')- if (m.layers.length > LIMITS.layers) bad(`it has ${m.layers.length} layers, more than ${LIMITS.layers}`)+ if (m.layers.length > LIMITS.layers) bad(`it has ${m.layers.length} layers, more than ${LIMITS.layers}`, 'too-large')const A = m.anchors, hd = A?.head, inRange = (v: unknown) => num(v) && Math.abs(v) <= BIGif (!hd || ![hd.cx, hd.cy, hd.rx, hd.ry, A.neck_y, A.top_y, A.bottom_y, A.hair_root_y].every(inRange) || !(hd.rx > 1e-3 && hd.ry > 1e-3)|| !Array.isArray(A.face) || A.face.length !== 4 || !A.face.every(inRange)) bad('its anchors are missing or not numbers')@@ -174,9 +177,9 @@ function check(m: ModelDef) {skinned += verts * (bones.length + (overBangs && Array.isArray(bangs) ? bangs.length : 0))if (overBangs) read += L.w * L.h}- if (vertices > LIMITS.vertices) bad(`its layers' meshes would have ${vertices} vertices, more than ${LIMITS.vertices}`)- if (skinned > LIMITS.skinned) bad(`its chains would bend ${skinned} vertices, more than ${LIMITS.skinned}`)- if (read > LIMITS.texels) bad(`its brows layers cover ${read} pixels, more than ${LIMITS.texels}`)+ if (vertices > LIMITS.vertices) bad(`its layers' meshes would have ${vertices} vertices, more than ${LIMITS.vertices}`, 'too-large')+ if (skinned > LIMITS.skinned) bad(`its chains would bend ${skinned} vertices, more than ${LIMITS.skinned}`, 'too-large')+ if (read > LIMITS.texels) bad(`its brows layers cover ${read} pixels, more than ${LIMITS.texels}`, 'too-large')for (const e of list(m.expressions, LIMITS.expressions, 'expressions') as ({ id?: unknown } | null)[]) if (!e || typeof e.id !== 'string') bad('an expression has no id')if (m.keyforms !== undefined && (typeof m.keyforms !== 'string' || !FILE.test(m.keyforms))) bad('keyforms is not a path inside the package')if (m.watermark != null && typeof m.watermark !== 'string') bad('its watermark is not text')@@ -191,17 +194,17 @@ interface V4 {}function lower(m: V4, lang: string): ModelDef {- if (!Array.isArray(m.layers) || !m.layers.length) throw new Error('this model has no layers')- if (m.layers.length > LIMITS.layers) throw new Error(`this model has ${m.layers.length} layers, more than ${LIMITS.layers}`)+ if (!Array.isArray(m.layers) || !m.layers.length) throw coded('invalid', 'this model has no layers')+ if (m.layers.length > LIMITS.layers) throw coded('too-large', `this model has ${m.layers.length} layers, more than ${LIMITS.layers}`)const [w, h] = m.canvas || []- if (!(w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) throw new Error(`this model's canvas ${w} x ${h} is out of range`)+ if (!(w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) throw coded(w > LIMITS.canvas || h > LIMITS.canvas ? 'too-large' : 'invalid', `this model's canvas ${w} x ${h} is out of range`)for (const L of m.layers) {for (const p of [L.texture, ...(L.frames || []), L.mask, L.talk, L.brow_tex]) {- if (p !== undefined && !IMAGE.test(p)) throw new Error(`layer ${L.id}: "${p}" is not an image path of the package`)+ if (p !== undefined && !IMAGE.test(p)) throw coded('invalid', `layer ${L.id}: "${p}" is not an image path of the package`)}}for (const [k, p] of Object.entries(m.files || {})) {- if (!k.startsWith('x-') && (typeof p !== 'string' || !FILE.test(p))) throw new Error(`files.${k}: "${p}" is not a path in the package`)+ if (!k.startsWith('x-') && (typeof p !== 'string' || !FILE.test(p))) throw coded('invalid', `files.${k}: "${p}" is not a path in the package`)}const layers = m.layers.map(({ label, expression, ...L }) =>({ ...L, depth: L.depth ?? 0, name: label ? pickLabel(label, 'en', L.id) : L.id, ...(expression ? { param: expression } : {}) }) as LayerDef)diff --git a/src/runtime/puppet.ts b/src/runtime/puppet.tsindex 428b785..d7b22de 100644--- a/src/runtime/puppet.ts+++ b/src/runtime/puppet.ts@@ -16,7 +16,7 @@// the `brows` layer alone where the build gave the brows one (drawn over the bangs); that layer moves with// the bangs where it lies over them// - body: lags the head turn / tilt, breathes (shoulders + chest), optional chest spring-import { LIMITS, latticeCell, readModel, type Labels, type ModelFiles, type ModelMeta } from './format'+import { LIMITS, coded, latticeCell, readModel, type Labels, type ModelFiles, type ModelMeta } from './format'import { PARAM_RANGES, defaultParams, smooth, type Params } from './params'export { PARAM_RANGES, defaultParams, smooth, type Params }@@ -189,9 +189,11 @@ export const defaultRig = (): Rig => ({})/** user-facing "feel" settings (livliv.ai's feel sliders); 1 = default. nod scales up / down on top of turn. hair is how* freely the hair swings; hairFront (the bangs' springs) and hairBack (long hair's, and the swinging pieces': chains)- * each take its place when set, so a model saved with hair alone moves both as before */-export type Tuning = { turn: number; nod: number; hair: number; blink: number; expr: number; mouth: number; breath: number; hairFront?: number; hairBack?: number }-export const defaultTuning = (): Tuning => ({ turn: 1, nod: 1, hair: 1, blink: 1, expr: 1, mouth: 1, breath: 1 })+ * each take its place when set, so a model saved with hair alone moves both as before.+ * gaze: the share of the iris's travel it uses, 0-1. Less shows less of what a far look uncovers in the whites and fixes+ * none of it; above 1 only shows more, so it stops at 1 */+export type Tuning = { turn: number; nod: number; hair: number; blink: number; expr: number; mouth: number; breath: number; hairFront?: number; hairBack?: number; gaze: number }+export const defaultTuning = (): Tuning => ({ turn: 1, nod: 1, hair: 1, blink: 1, expr: 1, mouth: 1, breath: 1, gaze: 1 })interface GLLayer {def: LayerDef@@ -314,6 +316,12 @@ function browLimits(A: ModelDef['anchors'], amp: number): [number, number] {return any ? [up, down] : [0, 0]}+/** what a caller may give a load: see Puppet.load */+interface LoadExtras { signal?: AbortSignal; onImage?: (came: number, of: number) => void }++/** a layer's images, as they came in */+interface LayerImages { main: HTMLImageElement; frames: HTMLImageElement[]; mask?: HTMLImageElement; talk?: HTMLImageElement; brows?: HTMLImageElement }+interface Spring { x: number; y: number; vx: number; vy: number }const spring = (): Spring => ({ x: 0, y: 0, vx: 0, vy: 0 })type Group = 'front' | 'back' | 'side'@@ -490,13 +498,15 @@ export class Puppet {/** baseUrl: the model's folder, or its model.json (the SDK: any name ending in .json); the images are read relative to* it. png: load the lossless layer PNGs even where a WebP copy exists (eval: rest-pose PSNR is scored against the* source, and quality-92 WebP alone costs 1.3–6.4 dB on line art). pack: try the server's textures.bin first (an- * exported package has none: the SDK turns it off rather than ask for it and get a 404) */- async load(baseUrl: string, { png = false, pack: usePack = true } = {}) {+ * exported package has none: the SDK turns it off rather than ask for it and get a 404). signal: calls the load off+ * until the model starts going in, which then happens at once and whole; it rejects with the signal's reason and the+ * model before stays. onImage: each image that has come in, of how many */+ async load(baseUrl: string, { png = false, pack: usePack = true, signal, onImage }: { png?: boolean; pack?: boolean } & LoadExtras = {}) {const { file, base, query } = locate(baseUrl)- const r = await fetch(file + (query ? `?${query}` : ''), { cache: 'no-store' })+ const r = await fetch(file + (query ? `?${query}` : ''), { cache: 'no-store', signal })if (!r.ok) throw Object.assign(new Error(r.status === 404 ? 'model not found' : `model.json ${r.status}`), { status: r.status })const text = await r.text()- if (text.length > LIMITS.modelJson) throw new Error(`model.json is larger than ${LIMITS.modelJson} bytes`)+ if (text.length > LIMITS.modelJson) throw coded('too-large', `model.json is larger than ${LIMITS.modelJson} bytes`)const model = readModel(JSON.parse(text))// layer files keep their names when a character is rebuilt and are cached for hours, while model.json never is:// tag them with the version of their bytes the server sends with it (tex), so a save of the feel settings, which rewrites@@ -512,26 +522,31 @@ export class Puppet {}const v = `?v=${tag}${query ? `&${query}` : ''}`// alongside the images, not before them. Without it the character still plays, on the rig alone- const keyforms = model.keyforms ? fetch(base + model.keyforms + v).then(async (k) => {+ const keyforms = model.keyforms ? fetch(base + model.keyforms + v, { signal }).then(async (k) => {const t = k.ok ? await k.text() : ''return t && t.length <= LIMITS.keyforms ? JSON.parse(t) as KeyformFile : undefined}).catch(() => undefined) : undefined// all of them in one file when the server has it: one request instead of ~30- const pack = png || !usePack ? null : await loadPack(packUrl(baseUrl, tag))- await this.loadModel(model, (f) => { const b = pack?.get(f); return b ? loadBlob(b) : loadImage(base + f + v) }, { png, keyforms })+ const pack = png || !usePack ? null : await loadPack(packUrl(baseUrl, tag), signal)+ // (an <img> cannot be called off: setting src to '' is an error, removing it never ends)+ const fetched = (f: string) => (signal ? fetchImage(base + f + v, signal) : loadImage(base + f + v))+ await this.loadModel(model, (f) => { const b = pack?.get(f); return b ? loadBlob(b) : fetched(f) }, { png, keyforms, signal, onImage })}/** a model from anywhere (the SDK: files the user picked): its model.json, and each image it names by its path there */async loadModel(raw: ModelDef, image: (path: string) => Promise<HTMLImageElement>,- { png = false, keyforms: keyformsComing }: { png?: boolean; keyforms?: KeyformFile | Promise<KeyformFile | undefined> } = {}) {+ { png = false, keyforms: keyformsComing, signal, onImage }: { png?: boolean; keyforms?: KeyformFile | Promise<KeyformFile | undefined> } & LoadExtras = {}) {const model = readModel(raw)+ let asked = 0, came = 0+ const arrived = (i: HTMLImageElement) => { onImage?.(++came, asked); return i }// every image at once: a layer's second image waiting for its first cost a round trip each on a slow pathconst [imgs, keyforms] = await Promise.all([Promise.all(model.layers.map(async (L) => {- const load = (p: string) => image(L.webp && !png ? p.replace(/\.png$/, '.webp') : p)+ const load = (p: string) => { asked++; return image(L.webp && !png ? p.replace(/\.png$/, '.webp') : p).then(arrived) }const [main, frames, mask, talk, brows] = await Promise.all([load(L.texture), Promise.all((L.frames || []).map(load)),L.mask ? load(L.mask) : undefined, L.talk ? load(L.talk) : undefined, L.brow_tex ? load(L.brow_tex) : undefined])return { main, frames, mask, talk, brows }})), keyformsComing])+ if (signal?.aborted) throw signal.reason// before anything reaches the GPU (an image's size is known once it loads, before it is decoded in full)const most = Math.min(LIMITS.texture, (this.gl?.getParameter(this.gl.MAX_TEXTURE_SIZE) as number) || LIMITS.texture)let texels = 0@@ -539,10 +554,37 @@ export class Puppet {if (!i) continueconst w = i.naturalWidth || i.width, h = i.naturalHeight || i.heightif (w > most || h > most || (texels += w * h) > LIMITS.texels) {- throw new Error(`this model's images are too large (${w} x ${h}: ${most} px a side and ${LIMITS.texels} pixels in all at most)`)+ throw coded('too-large', `this model's images are too large (${w} x ${h}: ${most} px a side and ${LIMITS.texels} pixels in all at most)`)+ }+ }+ this.install(model, imgs, keyforms)+ }++ /** the model in place of the one before, or, if anything throws on the way (its layers, its keyforms, the GPU out of+ * memory), the one before exactly as it was: its layers on the GPU, every field, every spring. All in one go: no frame+ * runs in between. The GPU holds both for that moment */+ private install(model: ModelDef, imgs: LayerImages[], keyforms: KeyformFile | undefined) {+ const gl = this.gl+ // every field; copies of the two this changes in place rather than replaces+ const before = { ...this, drive: { ...this.drive },+ chains: Object.fromEntries(Object.entries(this.chains).map(([g, c]) => [g, c.map((q) => ({ ...q }))])) as Puppet['chains'] }+ const built: GLLayer[] = []+ for (let i = 0; i < 8 && gl.getError(); i++); // an error left by anything earlier is not this model's+ try {+ this.build(model, imgs, keyforms, built)+ for (let i = 0, e; i < 8 && (e = gl.getError()); i++) { // (one flag at a time, in no set order)+ if (e === gl.OUT_OF_MEMORY) throw coded('too-large', 'the GPU ran out of memory for this model')}+ } catch (e) {+ this.freeLayers(built)+ Object.assign(this, before)+ throw e}- this.dispose()+ this.freeLayers(before.layers)+ this.freeMask()+ }++ private build(model: ModelDef, imgs: LayerImages[], keyforms: KeyformFile | undefined, built: GLLayer[]) {this.model = modelthis.chainRuns = (model.rig?.chains || []).map(chainRun)const mouth = model.layers.find((L) => L.role === 'mouth')@@ -560,8 +602,9 @@ export class Puppet {// the brows layers lie over the bangs where the build put them there: they follow what is under themconst fi = model.layers.findIndex((L) => L.group === 'hair_front')const bangs = fi >= 0 ? { img: imgs[fi].main, def: model.layers[fi] } : undefined- this.layers = model.layers.map((L, i) => this.buildLayer(L, imgs[i].main, imgs[i].frames, imgs[i].mask, imgs[i].talk, imgs[i].brows,- L.id === 'brows' || L.role === 'expr_brows' || L.brow_tex ? bangs : undefined))+ model.layers.forEach((L, i) => built.push(this.buildLayer(L, imgs[i].main, imgs[i].frames, imgs[i].mask, imgs[i].talk, imgs[i].brows,+ L.id === 'brows' || L.role === 'expr_brows' || L.brow_tex ? bangs : undefined)))+ this.layers = builtthis.byId = new Map(this.layers.map((l) => [l.def.id, l]))// each iris's room toward the nose: the nose lies toward the eyes' midpoint (a view with one eye has no limit)this.nasal = {}@@ -665,14 +708,23 @@ export class Puppet {}dispose() {+ this.freeLayers(this.layers)+ this.freeMask()+ this.layers = []+ }++ private freeLayers(layers: GLLayer[]) {const gl = this.gl- for (const L of this.layers) {+ for (const L of layers) {gl.deleteTexture(L.tex); L.frames.forEach((t) => gl.deleteTexture(t)); if (L.mask) gl.deleteTexture(L.mask); if (L.talk) gl.deleteTexture(L.talk)if (L.browTex) gl.deleteTexture(L.browTex)gl.deleteVertexArray(L.vao); gl.deleteBuffer(L.pos); gl.deleteBuffer(L.uvs); gl.deleteBuffer(L.tris)}+ }++ private freeMask() {+ const gl = this.glif (this.maskFb) { gl.deleteFramebuffer(this.maskFb); gl.deleteTexture(this.maskTex); this.maskFb = this.maskTex = null; this.maskSize = [0, 0] }- this.layers = []}private texture(img: HTMLImageElement) {@@ -680,7 +732,12 @@ export class Puppet {const t = gl.createTexture()!gl.bindTexture(gl.TEXTURE_2D, t)gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true)- gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img)+ try {+ gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img)+ } catch (e) {+ gl.deleteTexture(t)+ throw e+ }gl.generateMipmap(gl.TEXTURE_2D)gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_LINEAR)gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)@@ -858,9 +915,19 @@ export class Puppet {const tri = new Uint32Array(idx)gl.bufferData(gl.ELEMENT_ARRAY_BUFFER, tri, gl.STATIC_DRAW)gl.bindVertexArray(null)+ // an upload can throw (a canvas's image it may not read): nothing of this layer stays on the GPU+ const made: WebGLTexture[] = []+ const texture = (i: HTMLImageElement) => { const t = this.texture(i); made.push(t); return t }+ let tex: WebGLTexture, frameTex: WebGLTexture[], maskTex: WebGLTexture | undefined, talkTex: WebGLTexture | undefined, browTex: WebGLTexture | undefined+ try {+ tex = texture(img); frameTex = frames.map(texture); maskTex = mask && texture(mask); talkTex = talk && texture(talk); browTex = brows && texture(brows)+ } catch (e) {+ made.forEach((t) => gl.deleteTexture(t))+ gl.deleteVertexArray(vao); gl.deleteBuffer(pb); gl.deleteBuffer(ub); gl.deleteBuffer(ib)+ throw e+ }return {- def: L, tex: this.texture(img), frames: frames.map((f) => this.texture(f)), mask: mask ? this.texture(mask) : undefined,- talk: talk ? this.texture(talk) : undefined, browTex: brows ? this.texture(brows) : undefined,+ def: L, tex, frames: frameTex, mask: maskTex, talk: talkTex, browTex,vao, pos: pb, uvs: ub, tris: ib, count: idx.length, rest, out: new Float32Array(rest.length), tri, tw: img.naturalWidth || img.width, th: img.naturalHeight || img.height,wHead, wPhys1, wPhys2, dep, wBrow, browSide, browCap, wFront, wJaw, wNeck, wTwist, fol, folB, alt,outB: folB ? new Float32Array(rest.length) : undefined, stencilRef: 0, soft: 0, maskCh: -1, skin, altSkin,@@ -1301,6 +1368,7 @@ export class Puppet {}}const roll = (P.angleZ * Math.PI) / 180 * this.rig.roll * this.tuning.turn+ const g = this.tuning.gaze, gz = typeof g === 'number' && g >= 0 && g < 1 ? g : 1 // the feel settings' eye movement; anything else is the full travel// rig.neck: how far the chin goes sideways from where the torso alone puts it (model px)let chinDx = 0if (this.rig.neck) {@@ -1322,9 +1390,9 @@ export class Puppet {const squash = 0.12 + 0.88 * openconst iris = role.startsWith('iris_')let ix = 0, iy = 0- if (iris && d.range) { ix = P.eyeBallX * d.range[0]; iy = -P.eyeBallY * d.range[1] }+ if (iris && d.range) { ix = P.eyeBallX * d.range[0] * gz; iy = -P.eyeBallY * d.range[1] * gz }// (a model with swinging chains: its highlights follow the gaze too)- if (this.chained && role.startsWith('shine_') && d.range) { ix = P.eyeBallX * d.range[0]; iy = -P.eyeBallY * d.range[1] }+ if (this.chained && role.startsWith('shine_') && d.range) { ix = P.eyeBallX * d.range[0] * gz; iy = -P.eyeBallY * d.range[1] * gz }// irisNasal: past its room toward the nose the inner edge stays put and the far edge travels on (at a// vertex the shift is nx + nk * its rest x); the eye's highlights go with itlet nx = ix, nk = 0@@ -1737,16 +1805,17 @@ export function packUrl(baseUrl: string, tag: string) {/** textures.bin: the header's length (4 bytes, little-endian), the header (JSON {path: [offset,* length]}, and tex: the images' version), the files. null when it is not there or unreadable: the files are then* fetched one by one */-async function loadPack(url: string): Promise<Map<string, Blob> | null> {+async function loadPack(url: string, signal?: AbortSignal): Promise<Map<string, Blob> | null> {try {- const r = await fetch(url)+ const r = await fetch(url, { signal })if (!r.ok) return nullconst buf = await r.arrayBuffer()const n = new DataView(buf).getUint32(0, true)const head: Record<string, [number, number] | string> = JSON.parse(new TextDecoder().decode(new Uint8Array(buf, 4, n)))return new Map(Object.entries(head).filter((e): e is [string, [number, number]] => Array.isArray(e[1])).map(([k, [o, len]]) =>[k, new Blob([new Uint8Array(buf, 4 + n + o, len)], { type: k.endsWith('.png') ? 'image/png' : 'image/webp' })]))- } catch {+ } catch (e) {+ if (signal?.aborted) throw ereturn null}}@@ -1758,6 +1827,20 @@ export function loadBlob(b: Blob): Promise<HTMLImageElement> {return loadImage(u).finally(() => URL.revokeObjectURL(u))}+/** an image that can be called off: fetched, then decoded like any other */+async function fetchImage(src: string, signal: AbortSignal): Promise<HTMLImageElement> {+ let b: Blob+ try {+ const r = await fetch(src, { signal })+ if (!r.ok) throw new Error()+ b = await r.blob()+ } catch (e) {+ if (signal.aborted) throw e+ throw new Error('image ' + src)+ }+ return loadBlob(b)+}+export function loadImage(src: string): Promise<HTMLImageElement> {return new Promise((res, rej) => {const i = new Image(); i.crossOrigin = 'anonymous'diff --git a/src/tracking.ts b/src/tracking.tsindex 61d4b5a..3a20b15 100644--- a/src/tracking.ts+++ b/src/tracking.ts@@ -4,7 +4,7 @@// cached by the browser after that).import { FaceTracker, LOST_AFTER, V, loadFaceModel, setFaceModelFiles, type StartStep } from './runtime/tracking'import { defaultParams, type Params } from './runtime/params'-import type { Driver, Host } from './index'+import type { Driver, Host } from './base'export type { StartStep }diff --git a/src/voice.ts b/src/voice.tsindex 06da359..19d4888 100644--- a/src/voice.ts+++ b/src/voice.ts@@ -7,7 +7,7 @@import { VoiceMeter } from './runtime/voice'import { SpeechPlayer } from './runtime/speech'import { TalkMotion } from './runtime/talk'-import type { Driver, Host } from './index'+import type { Driver, Host } from './base'export interface LipSyncOptions {/** how much the voice also moves the head, brows and eyes: 0 (off) .. 2, default 1. Not while a face is tracked */diff --git a/src/vrm.ts b/src/vrm.tsindex 54db1d8..e42f878 100644--- a/src/vrm.ts+++ b/src/vrm.ts@@ -9,6 +9,7 @@ import type * as T from 'three'import type { VRM } from '@pixiv/three-vrm'import type { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader.js'import type { Body, BodyExpression, BodyMeta, Pose } from './body'+import { LivLivError, type LivLivErrorCode } from './errors'const FEELINGS = ['happy', 'angry', 'sad', 'relaxed', 'surprised'] as constconst LABELS: Record<(typeof FEELINGS)[number], Record<string, string>> = {@@ -60,6 +61,7 @@ export class VrmBody implements Body {private scene!: T.Sceneprivate camera!: T.PerspectiveCameraprivate renderer: T.WebGLRenderer | null = null+ private gl: WebGL2RenderingContext | null = nullprivate bones: Partial<Record<'head' | 'neck' | 'spine' | 'chest', T.Object3D>> = {}/** the framing: the world height shown (m), and its center */private frame = { height: 1, y: 1 }@@ -72,18 +74,21 @@ export class VrmBody implements Body {attach(canvas: HTMLCanvasElement) {// the attributes render() gives three.js, which then finds this same context- if (!canvas.getContext('webgl2', { alpha: true, antialias: true, premultipliedAlpha: true, preserveDrawingBuffer: true })) {- throw new Error('LivLiv: a VRM needs WebGL2: this browser has none, or this canvas already draws in 2D (a PNGTuber)')- }+ const gl = canvas.getContext('webgl2', { alpha: true, antialias: true, premultipliedAlpha: true, preserveDrawingBuffer: true })+ if (!gl) throw new LivLivError('no-webgl', 'LivLiv: a VRM needs WebGL2: this browser has none, or this canvas already draws in 2D (a PNGTuber)')+ this.gl = gl+ if (gl.isContextLost()) throw new LivLivError('context-lost', 'LivLiv: this canvas has lost its WebGL context: use a new canvas')}+ context() { return this.gl }+async load(files: Map<string, Blob>) {const entry = [...files].find(([n]) => /\.(vrm|glb)$/i.test(n))- if (!entry) throw new Error('LivLiv: no .vrm file among the files')+ if (!entry) throw new LivLivError('missing-file', 'LivLiv: no .vrm file among the files')await checkGlb(entry[1], entry[0], !this.options.configureLoader)const [THREE, { GLTFLoader }, { VRMLoaderPlugin, VRMUtils }] = await Promise.all([import('three'), import('three/examples/jsm/loaders/GLTFLoader.js'), import('@pixiv/three-vrm')]).catch((e) => {- throw new Error(`LivLiv: could not load three or @pixiv/three-vrm, which a VRM needs: are they installed, and can this page reach them? (${(e as Error).message})`)+ throw new LivLivError('unreachable', `LivLiv: could not load three or @pixiv/three-vrm, which a VRM needs: are they installed, and can this page reach them? (${(e as Error).message})`)})this.THREE = THREEconst loader = new GLTFLoader()@@ -94,20 +99,20 @@ export class VrmBody implements Body {try {gltf = await loader.loadAsync(url)} catch (e) {- throw new Error(`LivLiv: ${entry[0]} is not a VRM this player can read (${(e as Error).message})`)+ throw new LivLivError('not-a-model', `LivLiv: ${entry[0]} is not a VRM this player can read (${(e as Error).message})`, { cause: e })} finally {URL.revokeObjectURL(url)}const vrm = gltf.userData.vrm as VRM | undefinedif (!vrm) {VRMUtils.deepDispose(gltf.scene)- throw new Error(`LivLiv: ${entry[0]} is a glTF without VRM data (exported as plain glTF? VRM exporters write a .vrm)`)+ throw new LivLivError('not-a-model', `LivLiv: ${entry[0]} is a glTF without VRM data (exported as plain glTF? VRM exporters write a .vrm)`)}let meshes = 0vrm.scene.traverse((o) => { if ((o as T.Mesh).isMesh) meshes++ })if (!meshes) {VRMUtils.deepDispose(gltf.scene)- throw new Error(`LivLiv: ${entry[0]} has a skeleton but no mesh, so there is nothing to draw (export the armature together with its meshes)`)+ throw new LivLivError('invalid', `LivLiv: ${entry[0]} has a skeleton but no mesh, so there is nothing to draw (export the armature together with its meshes)`)}// the new character, apart from the one shown (a failed load leaves that on the canvas): it takes over at the endVRMUtils.removeUnnecessaryVertices(gltf.scene)@@ -272,13 +277,13 @@ const COMPRESSED: Record<string, string> = { KHR_draco_mesh_compression: 'Draco'* fetch them), its compression one the page can decode, and images and meshes within VRM_LIMITS, read from the file's* headers before anything is decoded */async function checkGlb(file: Blob, name: string, noDecoders: boolean) {- const bad = (why: string): never => { throw new Error(`LivLiv: ${name} ${why}`) }+ const bad = (why: string, code: LivLivErrorCode = 'invalid'): never => { throw new LivLivError(code, `LivLiv: ${name} ${why}`) }const head = new DataView(await file.slice(0, 20).arrayBuffer())- if (head.byteLength < 20 || head.getUint32(0, true) !== 0x46546c67) bad('is not a VRM: it should begin as a binary glTF does')- if (head.getUint32(4, true) !== 2) bad(`is a glTF of version ${head.getUint32(4, true)}, and VRM uses 2`)+ if (head.byteLength < 20 || head.getUint32(0, true) !== 0x46546c67) bad('is not a VRM: it should begin as a binary glTF does', 'not-a-model')+ if (head.getUint32(4, true) !== 2) bad(`is a glTF of version ${head.getUint32(4, true)}, and VRM uses 2`, 'not-a-model')const jsonLen = head.getUint32(12, true)if (head.getUint32(16, true) !== 0x4e4f534a) bad('is damaged (no JSON where a binary glTF has it)')- if (jsonLen > VRM_LIMITS.json) bad(`is not one this player reads: its JSON has ${jsonLen} bytes, more than ${VRM_LIMITS.json}`)+ if (jsonLen > VRM_LIMITS.json) bad(`is not one this player reads: its JSON has ${jsonLen} bytes, more than ${VRM_LIMITS.json}`, 'too-large')if (20 + jsonLen > file.size) bad('is damaged or cut short')let gltf: { buffers?: { uri?: string }[]; images?: { uri?: string; bufferView?: number }[]; bufferViews?: { byteOffset?: number; byteLength?: number }[]accessors?: { count?: number }[]; meshes?: { primitives?: { attributes?: { POSITION?: number } }[] }[]; extensionsUsed?: string[] }@@ -288,14 +293,14 @@ async function checkGlb(file: Blob, name: string, noDecoders: boolean) {}if (noDecoders) {for (const [ext, what] of Object.entries(COMPRESSED)) {- if (gltf.extensionsUsed?.includes(ext)) bad(`is compressed with ${what}, and this page has no decoder for that. Export it uncompressed, or pass VrmBody a configureLoader`)+ if (gltf.extensionsUsed?.includes(ext)) bad(`is compressed with ${what}, and this page has no decoder for that. Export it uncompressed, or pass VrmBody a configureLoader`, 'missing-feature')}}const positions = new Set<number>()for (const m of gltf.meshes ?? []) for (const p of m?.primitives ?? []) if (typeof p?.attributes?.POSITION === 'number') positions.add(p.attributes.POSITION)let vertices = 0for (const i of positions) vertices += gltf.accessors?.[i]?.count ?? 0- if (vertices > VRM_LIMITS.vertices) bad(`has ${vertices} vertices, more than ${VRM_LIMITS.vertices}`)+ if (vertices > VRM_LIMITS.vertices) bad(`has ${vertices} vertices, more than ${VRM_LIMITS.vertices}`, 'too-large')// where the binary chunk starts, and each picture's size from its first bytesconst binary = 20 + jsonLen + 8let texels = 0@@ -305,7 +310,7 @@ async function checkGlb(file: Blob, name: string, noDecoders: boolean) {const size = await pictureSize(file.slice(binary + (bv.byteOffset ?? 0), binary + (bv.byteOffset ?? 0) + (bv.byteLength ?? 0)))if (!size) continueif (size[0] > VRM_LIMITS.textureSide || size[1] > VRM_LIMITS.textureSide || (texels += size[0] * size[1]) > VRM_LIMITS.texels) {- bad(`has pictures too large for this player (${size[0]} x ${size[1]}; ${VRM_LIMITS.textureSide} px a side and ${VRM_LIMITS.texels} pixels in all at most)`)+ bad(`has pictures too large for this player (${size[0]} x ${size[1]}; ${VRM_LIMITS.textureSide} px a side and ${VRM_LIMITS.texels} pixels in all at most)`, 'too-large')}}}diff --git a/src/zip.ts b/src/zip.tsindex 300c73e..66e27e8 100644--- a/src/zip.ts+++ b/src/zip.ts@@ -4,10 +4,13 @@// what a package can hold: no encryption, no ZIP64, no paths outside the package.import { LIMITS } from './runtime/format'+import { LivLivError, type LivLivErrorCode } from './errors'/** what a package may hold before anything is decoded; model.json and keyforms.json as format.ts says */export const ZIP_LIMITS = { entries: 2048, bytes: 256 * 2 ** 20, modelJson: LIMITS.modelJson, keyforms: LIMITS.keyforms }+const fail = (why: string, code: LivLivErrorCode = 'invalid') => new LivLivError(code, `LivLiv: ${why}`)+const TYPES: Record<string, string> = { webp: 'image/webp', png: 'image/png', jpg: 'image/jpeg', jpeg: 'image/jpeg', json: 'application/json' }/** whether these bytes begin a ZIP (a local file header, or the end record of an empty one) */@@ -23,16 +26,16 @@ export async function unzip(zip: Blob): Promise<Map<string, Blob>> {const tail = new DataView(await zip.slice(zip.size - tailLen).arrayBuffer())let e = -1for (let i = tail.byteLength - 22; i >= 0; i--) if (tail.getUint32(i, true) === 0x06054b50) { e = i; break }- if (e < 0) throw new Error('this is not a ZIP (no end record)')+ if (e < 0) throw fail('this is not a ZIP (no end record)', 'not-a-model')const count = tail.getUint16(e + 10, true), cdSize = tail.getUint32(e + 12, true), cdOff = tail.getUint32(e + 16, true)- if (count === 0xffff || cdSize === 0xffffffff || cdOff === 0xffffffff) throw new Error('ZIP64 packages are not supported')- if (count > ZIP_LIMITS.entries) throw new Error(`the package has ${count} entries, more than ${ZIP_LIMITS.entries}`)+ if (count === 0xffff || cdSize === 0xffffffff || cdOff === 0xffffffff) throw fail('ZIP64 packages are not supported', 'too-large')+ if (count > ZIP_LIMITS.entries) throw fail(`the package has ${count} entries, more than ${ZIP_LIMITS.entries}`, 'too-large')const cd = new DataView(await zip.slice(cdOff, cdOff + cdSize).arrayBuffer())const utf8 = new TextDecoder()const out = new Map<string, Blob>()let total = 0for (let o = 0, n = 0; n < count; n++) {- if (o + 46 > cd.byteLength || cd.getUint32(o, true) !== 0x02014b50) throw new Error('the package is damaged (central directory)')+ if (o + 46 > cd.byteLength || cd.getUint32(o, true) !== 0x02014b50) throw fail('the package is damaged (central directory)')const flags = cd.getUint16(o + 8, true), method = cd.getUint16(o + 10, true)const csize = cd.getUint32(o + 20, true), usize = cd.getUint32(o + 24, true)const nlen = cd.getUint16(o + 28, true), elen = cd.getUint16(o + 30, true), clen = cd.getUint16(o + 32, true)@@ -43,30 +46,30 @@ export async function unzip(zip: Blob): Promise<Map<string, Blob>> {const base = name.slice(name.lastIndexOf('/') + 1)if (name.endsWith('/') || name.startsWith('__MACOSX/') || base.startsWith('._') || base === '.DS_Store' || base === 'Thumbs.db') continueif (name.startsWith('/') || /^[A-Za-z]:/.test(name) || name.split('/').some((s) => s === '..' || s === '.')) {- throw new Error(`the package holds a path outside itself: ${raw}`)+ throw fail(`the package holds a path outside itself: ${raw}`)}- if (flags & 1) throw new Error(`${name} is encrypted`)- if (out.has(name)) throw new Error(`${name} is in the package twice`)+ if (flags & 1) throw fail(`${name} is encrypted`)+ if (out.has(name)) throw fail(`${name} is in the package twice`)const isKeyforms = base === 'keyforms.json'const cap = base === 'model.json' ? ZIP_LIMITS.modelJson : isKeyforms ? ZIP_LIMITS.keyforms : ZIP_LIMITS.bytes- if (usize > cap || (total += usize) > ZIP_LIMITS.bytes) throw new Error(`${name} is too large (${usize} bytes)`)+ if (usize > cap || (total += usize) > ZIP_LIMITS.bytes) throw fail(`${name} is too large (${usize} bytes)`, 'too-large')const h = new DataView(await zip.slice(local, local + 30).arrayBuffer())- if (h.byteLength < 30 || h.getUint32(0, true) !== 0x04034b50) throw new Error(`the package is damaged (${name})`)+ if (h.byteLength < 30 || h.getUint32(0, true) !== 0x04034b50) throw fail(`the package is damaged (${name})`)const start = local + 30 + h.getUint16(26, true) + h.getUint16(28, true)const data = zip.slice(start, start + csize)const type = TYPES[base.slice(base.lastIndexOf('.') + 1).toLowerCase()] || ''- if (start + csize > zip.size) throw new Error(`the package is damaged (${name} runs past its end)`)+ if (start + csize > zip.size) throw fail(`the package is damaged (${name} runs past its end)`)if (method === 0) {- if (csize !== usize) throw new Error(`the package is damaged (${name})`)+ if (csize !== usize) throw fail(`the package is damaged (${name})`)out.set(name, data.slice(0, csize, type))} else if (method === 8) {if (!canInflate()) {if (isKeyforms) { console.warn(`LivLiv: this browser cannot inflate ${name}; the character plays without it`); continue }- throw new Error(`this browser cannot open the compressed ${name} (no DecompressionStream): store it uncompressed`)+ throw fail(`this browser cannot open the compressed ${name} (no DecompressionStream): store it uncompressed`, 'missing-feature')}out.set(name, await inflate(data, usize, name, type))} else {- throw new Error(`${name} uses ZIP method ${method}; a package stores its files or deflates them`)+ throw fail(`${name} uses ZIP method ${method}; a package stores its files or deflates them`)}}return out@@ -90,12 +93,12 @@ async function inflate(data: Blob, size: number, name: string, type: string): Prconst { done, value } = await reader.read()if (done) breakn += value.length- if (n > size) { void reader.cancel(); throw new Error(`${name} inflates past its ${size} bytes`) }+ if (n > size) { void reader.cancel(); throw fail(`${name} inflates past its ${size} bytes`) }parts.push(value)}} catch (e) {- throw (e as Error).message?.startsWith(name) ? e : new Error(`the package is damaged (${name}: ${(e as Error).message})`)+ throw (e as Error).message?.startsWith(name) ? e : fail(`the package is damaged (${name}: ${(e as Error).message})`)}- if (n !== size) throw new Error(`the package is damaged (${name}: ${n} bytes, ${size} expected)`)+ if (n !== size) throw fail(`the package is damaged (${name}: ${n} bytes, ${size} expected)`)return new Blob(parts as BlobPart[], { type })}diff --git a/test/bodies.html b/test/bodies.htmlindex d2f4fd7..b37a9dc 100644--- a/test/bodies.html+++ b/test/bodies.html@@ -14,6 +14,7 @@// Run by smoke.mjs: a PNGTuber shows the picture for each mouth / eye state, and follows the face tracker (its// synthetic trace) and a speech clip; a VRM (tiny-vrm.mjs, served by smoke.mjs) loads, draws, takes its expressions// and follows the face tracker.+ import { Player } from '../dist/index.js'import { BodyPlayer } from '../dist/body.js'import { PngBody } from '../dist/png.js'import { VrmBody } from '../dist/vrm.js'@@ -30,6 +31,8 @@const covered = (canvas) => { const d = read(canvas).data; let n = 0; for (let i = 3; i < d.length; i += 4) if (d[i] > 0) n++; return n / (d.length / 4) }const solid = (color) => { const c = new OffscreenCanvas(100, 120), x = c.getContext('2d'); x.fillStyle = color; x.fillRect(0, 0, 100, 120); return c.convertToBlob() }const wait = (ms) => new Promise((r) => setTimeout(r, ms))+ /** until the condition holds, or some seconds: what a browser does next may take a while on a busy machine */+ const until = async (cond, ms = 5000) => { for (const t0 = performance.now(); !cond() && performance.now() - t0 < ms;) await wait(10) }/** what the drivers did, seen by a driver that runs after them */const watch = (player) => {const seen = { tracked: false, mouth: 0, angle: 0 }@@ -46,6 +49,197 @@return new Blob([b.buffer], { type: 'audio/wav' })}+ /** what Player and BodyPlayer promise alike: one test, run on both (see PlayerBase) */+ window.runContract = async (kind, id) => {+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 200px; height: 240px'+ document.body.append(c)+ const options = { autoplay: false, followPointer: false, idle: false, maxPixelRatio: 2 }+ const p = kind === 'player' ? new Player(c, options) : new BodyPlayer(c, new PngBody(), options)+ const good = kind === 'player' ? `../characters/${id}.livliv` : new Map([['closed.png', await solid('#f00')], ['open.png', await solid('#0f0')]])+ const bad = kind === 'player' ? new Map([['model.json', new Blob(['{'])]]) : new Map([['closed.png', await solid('#f00')]])+ const out = { before: p.model, playing: p.playing }+ const info = await p.load(good)+ out.name = info.name === p.model?.name+ out.expressions = p.expressions === info.expressions+ // the pixel ratio can be read and changed+ out.ratio = [p.maxPixelRatio]+ p.maxPixelRatio = 0.5; p.resize(); out.ratio.push(p.maxPixelRatio, c.width)+ p.maxPixelRatio = 2; p.resize(); out.ratio.push(c.width)+ // a snapshot is the pose as it is now, on a paused player too+ const a = p.snapshot()+ p.params.mouthOpen = 1+ const b = p.snapshot()+ out.snapshot = a !== b && b.startsWith('data:image/png')+ p.params.mouthOpen = 0+ // what a driver gets; the time counts the frames before this one+ const times = []+ let frame = null+ const remove = p.use({ update(f) { times.push(f.time); frame = f } })+ p.draw(0.01); p.draw(0.02); p.draw(0.03)+ out.times = [times[1] - times[0], times[2] - times[1]]+ out.frame = !!frame.params && typeof frame.add === 'function' && frame.dt === 0.03 && 'x' in frame.move+ remove()+ p.draw(0.01)+ out.removed = times.length === 3+ // the pose stays in range, and what is no number is the default, whoever set it+ const pose = () => p.frame.params+ const P = p.params+ P.angleX = 1e9; P.eyeLOpen = -5; p.draw(0.01); out.clamped = [pose().angleX, pose().eyeLOpen]+ P.angleX = NaN; P.eyeLOpen = NaN; P.bodyX = 'x'; p.draw(0.01); out.nan = [pose().angleX, pose().eyeLOpen, pose().bodyX]+ const undo = p.use({ order: 100, update(f) { f.params.angleY = NaN; f.params.mouthOpen = 7 } })+ p.draw(0.01); out.driver = [pose().angleY, pose().mouthOpen]+ undo()+ Object.assign(P, { angleX: 0, eyeLOpen: 1, bodyX: 0 })+ p.draw(0.01)+ out.alive = kind === 'player' ? !p.puppet.layers.some((L) => Number.isNaN(L.out[0])) : true+ // away is a switch, on both+ p.setAway(true)+ for (let i = 0; i < 90; i++) p.draw(1 / 30)+ out.away = pose().eyeLOpen+ p.setAway(false)+ for (let i = 0; i < 90; i++) p.draw(1 / 30)+ out.back = pose().eyeLOpen+ out.oldAway = (() => { try { p.act('away'); return 'took it' } catch (e) { return e.message } })()+ // the rest of the surface+ out.unknown = p.setExpression('no-such-expression')+ out.neutral = p.setExpression(null)+ for (const a2 of ['nod', 'shake', 'tilt', 'hop', 'shy']) p.act(a2)+ p.lookAt(0.5, 0.5); p.lookAt(null); p.blink(); p.draw(0.01)+ p.play(); out.plays = p.playing; p.pause(); out.paused = p.playing+ // Player alone: from the canvas to the model, and a flick+ if (kind === 'player') {+ const [mx, my] = p.canvasToModel(c.clientWidth / 2, c.clientHeight / 2), [mw, mh] = p.model.size+ out.toModel = mx > 0 && my > 0 && mx < mw && my < mh+ const R = p.puppet.chainRuns[0]?.R+ out.poke = [p.poke(mx, my), R ? p.poke(R[R.length - 2], R[R.length - 1]) : -1]+ }+ // a load that fails keeps the character; a disposed player says so and takes nothing more+ const name = p.model.name+ out.failed = await p.load(bad).then(() => 'loaded', () => 'refused')+ out.kept = p.model?.name === name+ p.dispose(); p.dispose()+ out.after = p.model+ out.disposedLoad = await p.load(good).then(() => 'loaded', (e) => e.message)+ let gone = 0+ p.use({ update() {}, dispose() { gone++ } })+ out.gone = gone+ c.remove()+ return out+ }++ /** BodyPlayer: several loads at once. Those still downloading or waiting stop; one already going in (inside the Body,+ * which cannot be stopped) finishes, and the newer one follows */+ window.runBodyLoads = async () => {+ const out = {}+ const slow = (color) => ({ closed: `/__slow-png/${color}.png`, open: `/__slow-png/${color}.png` })+ {+ const c = document.createElement('canvas'), p = new BodyPlayer(c, new PngBody(), { autoplay: false, followPointer: false, idle: false })+ c.style.cssText = 'width: 40px; height: 40px'+ document.body.append(c)+ const say = (e) => e.name+ const a = p.load(slow('ff0000')).then(() => 'loaded', say)+ await wait(60)+ const b = p.load(slow('00ff00')).then(() => 'loaded', say)+ await wait(60)+ const steps = []+ const last = await p.load(slow('0000ff'), { onProgress: (s) => steps.push(s) }).then(() => 'loaded', say)+ out.row = { a: await a, b: await b, last, pixel: pixel(c, 20, 20), steps }+ p.dispose(); c.remove()+ }+ {+ // a Body holding its load: the page decides when it ends+ const held = [], calls = []+ const body = {+ format: 'test',+ load(files) { calls.push([...files.keys()][0]); return new Promise((res) => held.push(() => res({ name: [...files.keys()][0], expressions: [] }))) },+ render() {}, head: () => ({ x: 0.5, y: 0.3, rx: 0.1, ry: 0.1 }), dispose() {},+ }+ const c = document.createElement('canvas'), p = new BodyPlayer(c, body, { autoplay: false, followPointer: false })+ const file = (n) => new Map([[n, new Blob(['x'])]])+ const a = p.load(file('a')).then((m) => m.name, (e) => e.name)+ await wait(20)+ const b = p.load(file('b')).then((m) => m.name, (e) => e.name)+ const d = p.load(file('d')).then((m) => m.name, (e) => e.name)+ await wait(20)+ held.shift()()+ out.held = { a: await a, b: await b }+ await wait(20)+ held.shift()()+ out.held.d = await d+ out.held.calls = calls+ out.held.shown = p.model?.name+ p.dispose()+ }+ {+ // a Body that takes its 2D context only at its first frame: asking about the context must not start a WebGL one+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 40px; height: 40px'+ document.body.append(c)+ let got = null+ const body = {+ format: 'late', async load() { return { name: 'late', expressions: [] } },+ render(cv) { got = cv.getContext('2d') }, head: () => ({ x: 0.5, y: 0.3, rx: 0.1, ry: 0.1 }), dispose() {},+ }+ const p = new BodyPlayer(c, body, { autoplay: false, followPointer: false })+ out.lateLost = p.contextLost+ const going = p.load(new Map([['a', new Blob(['x'])]]))+ out.lateLost2 = p.contextLost+ await going+ out.late2d = !!got+ p.dispose({ release: true })+ await wait(20)+ out.lateAgain = (() => { try { new BodyPlayer(c, body, { autoplay: false }).dispose(); return 'made' } catch (e) { return e.code } })()+ c.remove()+ }+ return out+ }++ /** the context on a BodyPlayer: a VRM is told and released like a Player; a PNGTuber has none to lose */+ window.runBodyContext = async () => {+ const out = {}+ const code = (e) => e.code ?? e.name+ {+ const c = document.createElement('canvas'), p = new BodyPlayer(c, new VrmBody(), { followPointer: false })+ c.style.cssText = 'width: 60px; height: 80px'+ document.body.append(c)+ let told = 0+ p.onContextLost = () => told+++ await p.load('/__tiny.vrm')+ const gl = c.getContext('webgl2')+ gl.getExtension('WEBGL_lose_context').loseContext()+ await until(() => told)+ out.vrm = { told, lost: p.contextLost, load: await p.load('/__tiny.vrm').then(() => 'loaded', code) }+ p.dispose(); c.remove()+ }+ {+ const c = document.createElement('canvas'), p = new BodyPlayer(c, new VrmBody(), { followPointer: false })+ c.style.cssText = 'width: 60px; height: 80px'+ document.body.append(c)+ await p.load('/__tiny.vrm')+ const gl = c.getContext('webgl2')+ p.dispose({ release: true })+ await until(() => gl.isContextLost())+ out.vrmReleased = { lost: gl.isContextLost(), reuse: (() => { try { new BodyPlayer(c, new VrmBody()); return 'made' } catch (e) { return code(e) } })() }+ c.remove()+ }+ {+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 60px; height: 80px'+ document.body.append(c)+ const files = new Map([['closed.png', await solid('#f00')], ['open.png', await solid('#0f0')]])+ const p = new BodyPlayer(c, new PngBody(), { autoplay: false, followPointer: false, idle: false })+ await p.load(files)+ out.png = { lost: p.contextLost }+ p.dispose({ release: true })+ await wait(30)+ const q = new BodyPlayer(c, new PngBody(), { autoplay: false, followPointer: false, idle: false })+ out.png.again = await q.load(files).then((m) => m.name, code)+ q.dispose(); c.remove()+ }+ return out+ }+/** the camera refused while the face model comes in (the slow route of smoke.mjs); ahead: it was asked for before */window.runRefused = async (ahead = false) => {const files = { wasmUrl: '/node_modules/@mediapipe/tasks-vision/wasm', modelUrl: '/__model.task' }diff --git a/test/smoke.html b/test/smoke.htmlindex be2ffef..f8e532f 100644--- a/test/smoke.html+++ b/test/smoke.html@@ -42,6 +42,7 @@}let FILES = {}+ window.setFiles = (files) => { FILES = files }/** a model as files (path -> Blob) under a folder, as a folder picked in the browser gives them: its model.json as* given, everything else as the folder has it */async function filesOf(id, model, swap = {}) {@@ -122,6 +123,20 @@const vrmHere = await player.load(new File([await (await fetch('/__tiny.vrm')).blob()], 'x.vrm')).then(() => 'loaded', (e) => e.message)const vrmFolder = await player.load(new Map([['a/x.vrm', new Blob(['x'])]])).then(() => 'loaded', (e) => e.message)+ // what a load rejects with says what went wrong, in code+ const codeOf = (source) => player.load(source).then(() => 'loaded', (e) => (e.name === 'LivLivError' ? e.code : `not a LivLivError: ${e.name} ${e.message}`))+ const codes = {+ 'http': await codeOf('/__nothing-here.livliv'), 'not-a-model': await codeOf(new File([new Blob(['just text'])], 'x.livliv')),+ 'not-a-model json': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ format: 'something else' })])]])),+ 'not-a-model vrm': await codeOf(new File([await (await fetch('/__tiny.vrm')).blob()], 'x.vrm')),+ 'newer-format': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, version: FORMAT_VERSION + 1 })])]])),+ 'missing-feature': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, requires: ['from-the-future'] })])]])),+ 'missing-file': await codeOf(new Map([['model.json', new Blob([JSON.stringify(model)])]])),+ 'missing-file none': await codeOf(new Map([['a.txt', new Blob(['x'])]])),+ 'invalid': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, layers: [] })])]])),+ 'too-large': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, canvas: [100000, 100] })])]])),+ }+// a newer format: refused, and the model before stayslet newertry {@@ -146,7 +161,7 @@} catch (e) {legacy = { error: String(e.message || e) }}- return { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder }+ return { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder, codes }}/** disposing while something runs: a load rejects and leaves nothing on the GPU or the canvas; a driver added to a@@ -165,7 +180,7 @@// disposed just as the images start coming in (a timer could lose the race to a quick load)const loadModel = p.puppet.loadModel.bind(p.puppet)p.puppet.loadModel = (...args) => { p.dispose(); return loadModel(...args) }- out.load = await p.load(`../characters/${id}/`).then(() => 'resolved', (e) => e.message)+ out.load = await p.load(`../characters/${id}/`).then(() => 'resolved', (e) => `${e.name}: ${e.message}`)p.resize()await new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)))const px = new Uint8Array(4)@@ -210,6 +225,273 @@return out}+ /** a model that breaks while it is built, past the checks and the download: wherever it breaks, the character before+ * stays as it was (every pixel, every field, what is on the GPU) and the next good model plays. k: the character+ * standing there, in motion so that a spring put back wrong would show; b: the one that breaks, with chains and keyforms */+ window.runKeep = async (k, b) => {+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 200px; height: 240px'+ document.body.append(c)+ const p = new Player(c, { autoplay: false, followPointer: false, idle: false }), pup = p.puppet, gl = pup.gl+ const live = { texture: 0, buffer: 0, vertexArray: 0 }+ for (const [kind, make, free] of [['texture', 'createTexture', 'deleteTexture'], ['buffer', 'createBuffer', 'deleteBuffer'], ['vertexArray', 'createVertexArray', 'deleteVertexArray']]) {+ const m = gl[make], f = gl[free]+ gl[make] = function () { live[kind]++; return m.call(gl) }+ gl[free] = function (t) { if (t) live[kind]--; return f.call(gl, t) }+ }+ const kept = await filesOf(k, bare(await json(k))), broken = await filesOf(b, bare(await json(b)))+ const grab = () => {+ const o = document.createElement('canvas'); o.width = c.width; o.height = c.height+ const x = o.getContext('2d', { willReadFrequently: true })+ x.drawImage(c, 0, 0)+ return x.getImageData(0, 0, o.width, o.height).data+ }+ const fields = () => JSON.stringify({ name: pup.model?.name, layers: pup.layers.length, chains: pup.chains, drive: pup.drive, body: pup.body, bust: pup.bust, lastHead: pup.lastHead,+ smile: pup.smileShown, mouth: [pup.mouthFrame, pup.mouthW, pup.mouthHeld, pup.talkLeft, pup.frameKind, pup.frameJaw], keys: [pup.keys, pup.kfOff], rig: pup.rig, tuning: pup.tuning,+ time: pup.time, runs: pup.chainRuns.map((C) => [...C.p]), params: pup.params, player: p.model?.name })+ const stand = async () => {+ await p.load(kept)+ Object.assign(p.params, { angleX: 25, mouthOpen: 0.8, smile: 1 })+ for (let i = 0; i < 25; i++) p.draw(1 / 60)+ p.draw(0)+ }+ /** each arms one failure and returns how to take it away, which says whether the load got there */+ const places = {+ // nothing breaks: what the comparison below does with no failure at all (the good model simply plays)+ none: null,+ // the chains, the first thing built+ chain() { const f = Array.prototype.flat; let n = 0; Array.prototype.flat = function () { n++; throw new Error('injected') }; return () => { Array.prototype.flat = f; return n > 0 } },+ // a layer, part of the way through the list: those before it are on the GPU+ layer() { const f = pup.buildLayer; let n = 0; pup.buildLayer = function (...a) { if (++n === 4) throw new Error('injected'); return f.apply(this, a) }; return () => { delete pup.buildLayer; return n >= 4 } },+ // an upload, inside a layer: its mesh and some of its textures are on the GPU+ upload() { const f = gl.texImage2D; let n = 0; gl.texImage2D = function (...a) { if (++n === 3) throw new Error('injected'); return f.apply(gl, a) }; return () => { delete gl.texImage2D; return n >= 3 } },+ // the springs: the layers are done and most fields are already the new model's+ physics() { let n = 0; pup.resetPhysics = function () { n++; throw new Error('injected') }; return () => { delete pup.resetPhysics; return n > 0 } },+ // the keyforms, last and heaviest, in the middle of swapping the pose for its own+ keyforms() { const f = pup.prepareKeyforms; let n = 0; pup.prepareKeyforms = function (K) { n++; return f.call(this, { ...K, keys: [K.keys[0], null] }) }; return () => { delete pup.prepareKeyforms; return n > 0 } },+ // the GPU out of memory: nothing throws, it says so afterwards+ memory() { let n = 0; gl.getError = function () { return n++ ? gl.OUT_OF_MEMORY : gl.NO_ERROR }; return () => { delete gl.getError; return n > 1 } },+ }+ const out = {}+ for (const [name, arm] of Object.entries(places)) {+ await stand()+ const px = grab(), state = fields(), gpu = JSON.stringify(live)+ if (!arm) { p.draw(0); out[name] = { differ: differs(px, grab()) }; continue }+ const done = arm()+ const said = await p.load(broken).then(() => null, (e) => String(e.message || e))+ const reached = done(), after = fields(), gpuAfter = JSON.stringify(live)+ p.draw(0)+ const differ = differs(px, grab())+ const next = await p.load(broken).then((m) => m.name, (e) => `refused: ${e.message}`)+ p.draw(1 / 60)+ out[name] = { said, reached, differ, sameFields: state === after, sameGpu: gpu === gpuAfter, next, glError: gl.getError() }+ }+ p.dispose()+ out.left = { ...live }+ c.remove()+ return out+ }+ const differs = (a, b) => { let n = 0; for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) n++; return n }++ const wait = (ms) => new Promise((r) => setTimeout(r, ms))+ /** until the condition holds, or some seconds: what a browser does next may take a while on a busy machine */+ const until = async (cond, ms = 5000) => { for (const t0 = performance.now(); !cond() && performance.now() - t0 < ms;) await wait(10) }+ const nameOf = (m) => m.name, errName = (e) => e.name++ /** several loads at once, and calling a load off: the last one plays, the others stop and say so. Slow: a route of+ * smoke.mjs that takes about a second */+ window.runLoads = async ([a, b, d]) => {+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 160px; height: 200px'+ document.body.append(c)+ const p = new Player(c, { autoplay: false, followPointer: false }), out = {}+ const slow = (id) => `/__slow/${id}.livliv`+ // three in a row: only the last goes in+ let went = 0+ const install = p.puppet.install+ p.puppet.install = function (...args) { went++; return install.apply(this, args) }+ const first = p.load(slow(a)).then(nameOf, errName)+ await wait(120)+ const second = p.load(slow(b)).then(nameOf, errName)+ await wait(120)+ const third = await p.load(`../characters/${d}.livliv`).then(nameOf, errName)+ out.row = { first: await first, second: await second, third, shown: p.model?.name, went }+ delete p.puppet.install+ // nobody catching: a replaced load reports nothing, a real error is reported as ever+ const unhandled = []+ const note = (e) => { unhandled.push(e.reason?.code ?? e.reason?.name); e.preventDefault() }+ addEventListener('unhandledrejection', note)+ p.load(slow(a))+ await p.load(`../characters/${b}.livliv`)+ await wait(80)+ out.quiet = unhandled.length+ p.load('/__nothing-here.livliv')+ await until(() => unhandled.length)+ await wait(50)+ out.loud = [...unhandled]+ removeEventListener('unhandledrejection', note)+ // a signal already aborted: refused at once, and it leaves the running load alone+ const running = p.load(`../characters/${a}.livliv`).then(nameOf, errName)+ const off = new AbortController()+ off.abort()+ out.already = { said: await p.load(`../characters/${d}.livliv`, { signal: off.signal }).then(nameOf, errName), running: await running }+ // one's own signal: the load stops, the character stays, the reason is its own+ const before = p.model.name+ const own = new AbortController()+ const mine = p.load(slow(b), { signal: own.signal }).then(nameOf, errName)+ await wait(120)+ own.abort()+ out.own = { said: await mine, kept: p.model.name === before }+ const why = new AbortController()+ const reason = p.load(slow(b), { signal: why.signal }).then(nameOf, (e) => e.message)+ await wait(50)+ why.abort(new Error('my reason'))+ out.reason = await reason+ // progress, for each way a model comes+ const steps = async (source) => { const seen = []; await p.load(source, { onProgress: (s) => seen.push(s) }); return seen }+ out.progress = { package: await steps(slow(a)), unknown: await steps(`/__slow-nosize/${a}.livliv`), folder: await steps(`../characters/${a}/`), files: await steps(await filesOf(a, await json(a))) }+ // a callback that throws does not break the load+ const log = console.error+ console.error = () => {}+ out.badCallback = await p.load(`../characters/${a}.livliv`, { onProgress() { throw new Error('mine') } }).then(nameOf, (e) => e.message)+ console.error = log+ // disposed while it runs+ const going = p.load(slow(b)).then(nameOf, (e) => [e.name, e.message])+ await wait(80)+ p.dispose()+ out.disposed = await going+ out.after = await p.load(slow(b)).then(nameOf, errName)+ c.remove()+ return out+ }++ /** the WebGL context lost, and given up */+ window.runContext = async (id, big = true) => {+ const out = {}+ const box = () => { const c = document.createElement('canvas'); c.style.cssText = 'width: 60px; height: 80px'; document.body.append(c); return c }+ const pk = `../characters/${id}.livliv`+ const lose = (p) => p.puppet.gl.getExtension('WEBGL_lose_context').loseContext()+ const code = (e) => e.code ?? e.name+ {+ // lost: the page is told once, nothing draws or throws, a load says so at once, and one running is called off+ const c = box(), p = new Player(c, { followPointer: false })+ await p.load(pk)+ let told = 0+ p.onContextLost = () => told+++ out.before = p.contextLost+ const going = p.load(`/__slow/${id}.livliv`).then(() => 'loaded', code)+ await wait(60)+ lose(p)+ await until(() => told)+ out.running = await going+ out.told = told+ out.lost = p.contextLost+ out.load = await p.load(pk).then(() => 'loaded', code)+ p.play(); await wait(120) // frames on a lost context+ p.resize(); out.snapshot = typeof p.snapshot()+ p.dispose()+ out.newOnLost = (() => { try { new Player(c); return 'made' } catch (e) { return code(e) } })()+ c.remove()+ }+ {+ // released: gone one task later, and the canvas says so; the page is not told of its own doing+ const c = box(), p = new Player(c, { followPointer: false }), gl = p.puppet.gl+ await p.load(pk)+ let told = 0+ p.onContextLost = () => told+++ p.dispose({ release: true })+ out.sameTask = gl.isContextLost()+ await until(() => gl.isContextLost())+ await wait(30)+ out.nextTask = gl.isContextLost()+ out.releasedTold = told+ out.reuse = (() => { try { new Player(c); return 'made' } catch (e) { return code(e) } })()+ c.remove()+ }+ {+ // a framework's unmount and mount of one canvas, in one task: the new player keeps the context+ const c = box()+ const a = new Player(c, { followPointer: false })+ await a.load(pk)+ let told = 0+ a.onContextLost = () => told+++ a.dispose({ release: true })+ const b = new Player(c, { followPointer: false })+ await wait(30)+ out.remount = { lost: b.contextLost, loaded: await b.load(pk).then((m) => !!m.name, code), told }+ b.resize()+ out.remount.share = (() => { const x = document.createElement('canvas'); x.width = c.width; x.height = c.height; const g2 = x.getContext('2d', { willReadFrequently: true }); g2.drawImage(c, 0, 0); const d = g2.getImageData(0, 0, x.width, x.height).data; let n = 0; for (let i = 3; i < d.length; i += 4) if (d[i]) n++; return n / (d.length / 4) })()+ b.dispose({ release: true })+ c.remove()+ }+ if (big) {+ // characters coming and going beside one that stays: with release it stays, without it is pushed out+ const stay = box(), keep = new Player(stay, { followPointer: false, autoplay: false })+ await keep.load(pk)+ let told = 0+ keep.onContextLost = () => told+++ for (let i = 0; i < 40; i++) {+ const c = box(), p = new Player(c, { followPointer: false, autoplay: false })+ p.dispose({ release: true })+ c.remove()+ await wait(0)+ }+ await wait(50)+ out.withRelease = { lost: keep.contextLost, told }+ const others = []+ for (let i = 0; i < 20; i++) { const c = box(); others.push(new Player(c, { followPointer: false, autoplay: false })); await wait(0) }+ await until(() => told, 2000)+ out.withoutRelease = { lost: keep.contextLost, told }+ out.overLimitLoad = await keep.load(pk).then(() => 'loaded', code)+ keep.dispose()+ for (const p of others) { p.dispose({ release: true }); p.canvas.remove() }+ stay.remove()+ await wait(30)+ }+ return out+ }++ /** the poster: on the canvas until the first frame, the canvas's own background back after */+ window.runPoster = async (id) => {+ const c = document.createElement('canvas')+ c.style.cssText = 'width: 160px; height: 200px; background-color: rgb(1, 2, 3)'+ document.body.append(c)+ const look = () => { const s = getComputedStyle(c); return { image: s.backgroundImage, size: s.backgroundSize, repeat: s.backgroundRepeat, color: s.backgroundColor } }+ const poster = `../characters/${id}/cover.webp`, out = {}+ const clean = look()+ let p = new Player(c, { autoplay: false, followPointer: false })+ const going = p.load(`/__slow/${id}.livliv`, { poster })+ out.during = look()+ await going+ out.after = look()+ out.same = JSON.stringify(out.after) === JSON.stringify(clean)+ // over a character: none, that one stays until the next is ready+ const next = p.load(`/__slow/${id}.livliv`, { poster })+ out.over = look().image+ await next+ p.dispose()+ // a load that fails takes it away, so that the page can say why+ p = new Player(c, { autoplay: false, followPointer: false })+ const refused = p.load('/__nothing-here.livliv', { poster }).catch(() => {})+ out.failing = look().image+ await refused+ out.failed = look().image+ // replaced by a load without one: gone at once; with one: that one+ const a = p.load(`/__slow/${id}.livliv`, { poster }).catch(() => {})+ const b = p.load(`/__slow/${id}.livliv`)+ out.replaced = look().image+ await a; await b+ p.dispose()+ // disposed while it shows+ p = new Player(c, { autoplay: false, followPointer: false })+ const d = p.load(`/__slow/${id}.livliv`, { poster }).catch(() => {})+ p.dispose()+ await d+ out.disposed = look().image+ c.remove()+ return out+ }+/** random changes to a model.json, in the real player: whatever happens, the player goes on working. Refused: the model* before is still drawn. Loaded: the frames do not throw. And a good model after any of them draws (nothing is poisoned) */window.runFuzz = async (id, keep, from, to) => {@@ -272,8 +554,9 @@* gets our words, not the browser's; a 404 is not tried again */window.runDropped = async (id) => {const p = new Player(canvas, { autoplay: false, followPointer: false })- const tell = (route) => p.load(route).then((m) => `played ${m.name}`, (e) => e.message)- const out = { once: await tell(`/__dropped/once/${id}.livliv`), always: await tell(`/__dropped/always/${id}.livliv`), missing: await tell('/__dropped/missing/x.livliv') }+ const tell = (route) => p.load(route).then((m) => `played ${m.name}`, (e) => { said.push([e.code, e.status]); return e.message })+ const said = []+ const out = { said, once: await tell(`/__dropped/once/${id}.livliv`), always: await tell(`/__dropped/always/${id}.livliv`), missing: await tell('/__dropped/missing/x.livliv') }p.dispose()return out}@@ -298,7 +581,7 @@}const out = {}for (const [k, files] of Object.entries(cases)) {- try { await player.load(await files); out[k] = { loaded: true } } catch (e) { out[k] = { refused: String(e.message || e), kept: player.model?.name } }+ try { await player.load(await files); out[k] = { loaded: true } } catch (e) { out[k] = { refused: String(e.message || e), code: e.code, kept: player.model?.name } }}// keyforms that are not right: the character still plays, on the rig aloneconst kf = await json(kfId)diff --git a/test/smoke.mjs b/test/smoke.mjsindex a93b47e..af526b4 100644--- a/test/smoke.mjs+++ b/test/smoke.mjs@@ -4,11 +4,13 @@// (bodies.html).// npm run build && npm run test:browser// Needs a Chromium for playwright-core once: npx playwright-core install chromium-headless-shell+// With an engine's name (firefox, webkit: npm run test:engines), a small sample on that browser's engine: what the three+// engines might do differently (loads replaced, the context lost and given up, the poster, the contract of the two classes).import { createServer } from 'node:http'import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'import { extname, join, normalize } from 'node:path'import { fileURLToPath } from 'node:url'-import { chromium } from 'playwright-core'+import { chromium, firefox, webkit } from 'playwright-core'import { crc32, deflateSync } from 'node:zlib'import { tinyVrm } from './tiny-vrm.mjs'@@ -40,6 +42,30 @@ const bomb = bombPng()const slowModel = { asked: 0, finished: 0 }/** requests per route of runDropped */const dropped = {}+/** the slow routes (runLoads): how many were asked for, ran to the end, were closed before it */+const slow = { asked: 0, finished: 0, cut: 0 }+const slowly = (res, b, headers, ms = 40, parts = 25) => {+ slow.asked+++ res.writeHead(200, { 'content-type': 'application/octet-stream', ...headers })+ let i = 0+ const tick = setInterval(() => {+ res.write(b.subarray(Math.floor((i * b.length) / parts), Math.floor(((i + 1) * b.length) / parts)))+ if (++i === parts) { clearInterval(tick); slow.finished++; res.end() }+ }, ms)+ res.on('close', () => { if (i < parts) { slow.cut++; clearInterval(tick) } })+}+/** a PNG of one colour */+function solidPng(hex, side = 8) {+ const chunk = (type, data) => {+ const b = Buffer.alloc(12 + data.length), t = Buffer.concat([Buffer.from(type), data])+ b.writeUInt32BE(data.length, 0); t.copy(b, 4); b.writeUInt32BE(crc32(t), 8 + data.length)+ return b+ }+ const ihdr = Buffer.alloc(13); ihdr.writeUInt32BE(side, 0); ihdr.writeUInt32BE(side, 4); ihdr[8] = 8; ihdr[9] = 2+ const line = Buffer.concat([Buffer.from([0]), ...Array.from({ length: side }, () => Buffer.from(hex, 'hex'))])+ return Buffer.concat([Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), chunk('IHDR', ihdr),+ chunk('IDAT', deflateSync(Buffer.concat(Array.from({ length: side }, () => line)))), chunk('IEND', Buffer.alloc(0))])+}const server = createServer((req, res) => {if (req.url.startsWith('/__model.task')) {slowModel.asked++@@ -53,6 +79,14 @@ const server = createServer((req, res) => {res.on('close', () => clearInterval(tick))return}+ const taking = req.url.match(/^\/__slow(-nosize)?\/(.+)\.livliv$/)+ if (taking) {+ const b = readFileSync(join(chars, `${taking[2]}.livliv`))+ slowly(res, b, taking[1] ? {} : { 'content-length': b.length })+ return+ }+ const picture = req.url.match(/^\/__slow-png\/(\w+)\.png$/)+ if (picture) { const b = solidPng(picture[1]); slowly(res, b, { 'content-type': 'image/png', 'content-length': b.length }, 60, 5); return }const drop = req.url.match(/^\/__dropped\/(once|always|missing)\/(.+)\.livliv$/)if (drop) {const [, how, id] = drop, n = (dropped[how] = (dropped[how] || 0) + 1)@@ -77,6 +111,53 @@ const server = createServer((req, res) => {})await new Promise((r) => server.listen(0, '127.0.0.1', r))+const engine = process.argv[2]+if (engine) {+ const browser = await ({ firefox, webkit }[engine]).launch(engine === 'firefox' ? { firefoxUserPrefs: { 'webgl.force-enabled': true, 'webgl.disabled': false } } : {})+ let failedHere = 0+ const check = (ok, what) => { if (!ok) failedHere++; console.log(`${ok ? 'ok ' : 'FAIL'} ${engine}: ${what}`) }+ try {+ const ctx = await browser.newContext({ permissions: [] }), page = await ctx.newPage()+ const errors = []+ page.on('pageerror', (e) => errors.push(e.message))+ await page.goto(`http://127.0.0.1:${server.address().port}/test/smoke.html`)+ await page.waitForFunction(() => typeof window.run === 'function')+ const chosen = ids.slice(0, 3), nameIn = (id) => JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')).name+ await page.evaluate((f) => window.setFiles(f), files)+ const r = await page.evaluate((a) => window.runLoads(a), chosen)+ check(r.row.first === 'AbortError' && r.row.second === 'AbortError' && r.row.third === nameIn(chosen[2]) && r.row.shown === nameIn(chosen[2]) && r.row.went === 1, `the last of three loads plays, once: ${JSON.stringify(r.row)}`)+ // (WebKit may keep the connection of a download called off)+ if (engine !== 'webkit') check(slow.cut >= 2, `the replaced downloads stop: ${slow.cut} cut`)+ check(r.quiet === 0 && r.loud.join() === 'http', `nobody catching: replaced ${r.quiet}, a real error ${r.loud}`)+ check(r.already.said === 'AbortError' && r.already.running === nameIn(chosen[0]), 'a signal already aborted leaves the running load alone')+ check(r.own.said === 'AbortError' && r.own.kept && r.reason === 'my reason', "one's own signal")+ const rising = (steps) => { const d = steps.filter((s) => s.step === 'download').map((s) => s.progress); return d.every((v, i) => !i || v > d[i - 1]) && d.at(-1) === 1 && steps.at(-1).step === 'build' }+ check(rising(r.progress.package) && rising(r.progress.unknown) && rising(r.progress.folder), `progress rises to 1: ${r.progress.package.length} steps`)+ check(r.disposed[0] === 'AbortError', 'disposed while it runs')+ const q = await page.evaluate((id) => window.runPoster(id), chosen[0])+ check(/cover\.webp/.test(q.during.image) && q.same && q.over === 'none' && q.failed === 'none' && q.replaced === 'none' && q.disposed === 'none', 'the poster, and the background back after')+ // (only Chromium and Firefox get room back from a released context; Firefox has not been seen to drop one by count)+ const x = await page.evaluate(([id, big]) => window.runContext(id, big), [chosen[0], engine === 'firefox'])+ check(x.before === false && x.lost === true && x.told === 1 && x.running === 'context-lost' && x.load === 'context-lost' && x.snapshot === 'string', `the context lost: told ${x.told}, a load ${x.load}`)+ check(x.nextTask === true && x.releasedTold === 0 && x.reuse === 'released', `released: gone next task ${x.nextTask}, the canvas says ${x.reuse}`)+ check(x.remount.lost === false && x.remount.loaded === true && x.remount.share > 0.05, `unmount and mount in one task: ${JSON.stringify(x.remount)}`)+ if (engine === 'firefox') check(x.withRelease.lost === false, '40 players came and went with release: the one that stays stays')+ await page.goto(`http://127.0.0.1:${server.address().port}/test/bodies.html`)+ await page.waitForFunction(() => typeof window.runBodies === 'function')+ for (const kind of ['player', 'png']) {+ const c = await page.evaluate(([k, id]) => window.runContract(k, id), [kind, chosen[0]])+ check(c.snapshot && c.clamped.join() === '30,0' && c.nan.join() === '0,1,0' && c.failed === 'refused' && c.kept && c.after === null && c.away < 0.05 && c.back > 0.9, `the contract, ${kind}: ${c.name}`)+ }+ const b = await page.evaluate(() => window.runBodyLoads())+ check(b.row.a === 'AbortError' && b.row.b === 'AbortError' && b.row.last === 'loaded' && b.held.d === 'd' && b.held.calls.join() === 'a,d', 'a BodyPlayer, several loads')+ const v = await page.evaluate(() => window.runBodyContext())+ check(v.vrm.told === 1 && v.vrm.load === 'context-lost' && v.vrmReleased.reuse === 'released' && !!v.png.again, 'a BodyPlayer, the context')+ check(!errors.length, `no page errors${errors.length ? `: ${errors.join(' | ')}` : ''}`)+ } finally { await browser.close(); server.close() }+ console.log(failedHere ? `smoke ${engine}: ${failedHere} failed` : `smoke ${engine}: ok`)+ process.exit(failedHere ? 1 : 0)+}+const browser = await chromium.launch({ args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--autoplay-policy=no-user-gesture-required','--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] })let failed = 0@@ -86,7 +167,10 @@ try {page.on('pageerror', (e) => errors.push(e.message))await page.goto(`http://127.0.0.1:${server.address().port}/test/smoke.html`)await page.waitForFunction(() => typeof window.run === 'function')- const { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder } = await page.evaluate((a) => window.run(a), { ids, files })+ const { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder, codes } = await page.evaluate((a) => window.run(a), { ids, files })+ const codesOk = Object.entries(codes).every(([want, got]) => want.split(' ')[0] === got)+ if (!codesOk) failed+++ console.log(`${codesOk ? 'ok ' : 'FAIL'} what a load rejects with says what went wrong: ${Object.entries(codes).map(([k, v]) => (k.split(' ')[0] === v ? v : `${k}: ${v}`)).join(', ')}`)const vrmHereOk = /BodyPlayer/.test(vrmHere) && /BodyPlayer/.test(vrmFolder)if (!vrmHereOk) failed++console.log(`${vrmHereOk ? 'ok ' : 'FAIL'} a VRM handed to a Player: ${vrmHere}`)@@ -115,8 +199,9 @@ try {const kfId = ids.find((id) => JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')).files?.keyforms) ?? ids[0]const hostile = await page.evaluate(([a, b]) => window.runHostile(a, b), [ids[0], kfId])const why = { bomb: /too large/, path: /not an image path|not a path/, canvas: /canvas/, box: /box/, json: /not a model\.json/, layers: /layers/, mesh: /vertices/, chain: /no id/, browLine: /brow line/, rig: /rig\.roll/ }+ const code = { bomb: 'too-large', path: 'invalid', canvas: 'too-large', box: 'invalid', json: 'not-a-model', layers: 'too-large', mesh: 'too-large', chain: 'invalid', browLine: 'invalid', rig: 'invalid' }for (const [k, re] of Object.entries(why)) {- const r = hostile[k], ok = !r.loaded && re.test(r.refused) && !!r.kept+ const r = hostile[k], ok = !r.loaded && re.test(r.refused) && !!r.kept && r.code === code[k]if (!ok) failed++console.log(`${ok ? 'ok ' : 'FAIL'} refuses ${k}: ${r.loaded ? 'it loaded' : r.refused.slice(0, 110)} (kept: ${r.kept})`)}@@ -124,6 +209,16 @@ try {if (!kfOk) failed++console.log(`${kfOk ? 'ok ' : 'FAIL'} broken keyforms: ${hostile.keyforms.refused ?? `${hostile.keyforms.name} plays on the rig alone`}`)+ // a model that breaks while it is built (see runKeep): the character before stays, to the pixel+ const has = (test) => ids.find((id) => test(JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')), id))+ const rigged = has((m) => m.rig?.chains?.length && m.files?.keyforms) ?? kfId+ const chained = has((m, id) => m.rig?.chains?.length && id !== rigged) ?? ids[0]+ const keep = await page.evaluate(([a, b]) => window.runKeep(a, b), [chained, rigged])+ const { none: noFail, left: keepLeft, ...failures } = keep+ const keepOk = noFail.differ === 0 && Object.values(keepLeft).every((n) => n <= 0) && Object.values(failures).every((r) => r.said && r.reached && r.differ === 0 && r.sameFields && r.sameGpu && !/^refused/.test(r.next) && r.glError === 0)+ if (!keepOk) failed+++ console.log(`${keepOk ? 'ok ' : 'FAIL'} a model breaks while it is built (${rigged} over ${chained}): ${Object.entries(failures).map(([k, r]) => `${k}: ${r.reached ? '' : 'NOT REACHED '}${r.differ} values differ${r.sameFields ? '' : ', FIELDS changed'}${r.sameGpu ? '' : ', GPU changed'}${/^refused/.test(r.next) ? `, then ${r.next}` : ''}${r.glError ? `, GL error ${r.glError}` : ''}`).join('; ')}${keepOk ? '' : `; with no failure ${noFail.differ}; left ${JSON.stringify(keepLeft)}`}`)+// random changes to a model.json in the real player (see runFuzz): the seeds of a run are the same every timeconst fuzzed = await page.evaluate(([a, b]) => window.runFuzz(a, b, 1, 150), [ids.find((i) => i === 'k02') ?? ids[0], ids[0]])const fuzzOk = !fuzzed.wrong.length@@ -132,13 +227,58 @@ try {// a package whose connection drops (see runDropped)const drops = await page.evaluate((id) => window.runDropped(id), ids[0])const dropsOk = /^played/.test(drops.once) && dropped.once === 2 && /could not load/.test(drops.always) && !/Failed to fetch/.test(drops.always) && dropped.always === 2- && /HTTP 404/.test(drops.missing) && dropped.missing === 1+ && /HTTP 404/.test(drops.missing) && dropped.missing === 1 && JSON.stringify(drops.said) === JSON.stringify([['unreachable', null], ['http', 404]])if (!dropsOk) failed++console.log(`${dropsOk ? 'ok ' : 'FAIL'} a package's connection drops: once "${drops.once}" (${dropped.once} requests), always "${drops.always.slice(0, 60)}…" (${dropped.always}), a 404 "${drops.missing.slice(0, 50)}…" (${dropped.missing})`)+ // several loads at once, called off, how far they have come (see runLoads)+ {+ const chosen = ids.slice(0, 3)+ const r = await page.evaluate((a) => window.runLoads(a), chosen)+ const nameIn = (id) => JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')).name+ const rising = (steps) => {+ const d = steps.filter((s) => s.step === 'download').map((s) => s.progress)+ return d.every((v, i) => !i || v > d[i - 1]) && d.at(-1) === 1 && steps.at(-1).step === 'build' && steps.filter((s) => s.step === 'build').length === 1+ }+ const checks = {+ 'only the last plays': r.row.first === 'AbortError' && r.row.second === 'AbortError' && r.row.third === nameIn(chosen[2]) && r.row.shown === nameIn(chosen[2]) && r.row.went === 1,+ 'the replaced downloads stop': slow.cut >= 2,+ 'nobody catching': r.quiet === 0 && r.loud.join() === 'http',+ 'a signal already aborted': r.already.said === 'AbortError' && r.already.running === nameIn(chosen[0]),+ "one's own signal": r.own.said === 'AbortError' && r.own.kept && r.reason === 'my reason',+ 'progress': rising(r.progress.package) && r.progress.package.filter((s) => s.step === 'download').length > 5 && rising(r.progress.unknown) && rising(r.progress.folder) && r.progress.files.length === 1 && r.progress.files[0].step === 'build',+ 'a callback that throws': r.badCallback === nameIn(chosen[0]),+ 'dispose': r.disposed[0] === 'AbortError' && /disposed/.test(r.disposed[1]) && r.after === 'AbortError',+ }+ const wrong = Object.entries(checks).filter(([, ok]) => !ok).map(([k]) => k)+ if (wrong.length) failed+++ console.log(`${wrong.length ? 'FAIL' : 'ok '} several loads, called off: ${wrong.length ? `${wrong.join(', ')}: ${JSON.stringify(r).slice(0, 900)}` : `${slow.cut} downloads stopped of ${slow.asked}; ${r.progress.package.length} steps from a package`}`)+ const q = await page.evaluate((id) => window.runPoster(id), chosen[0])+ const has = (v) => /cover\.webp/.test(v), posterOk = has(q.during.image) && q.during.size === 'contain' && q.during.repeat === 'no-repeat' && q.during.color === 'rgb(1, 2, 3)' && q.same+ && q.over === 'none' && has(q.failing) && q.failed === 'none' && q.replaced === 'none' && q.disposed === 'none'+ if (!posterOk) failed+++ console.log(`${posterOk ? 'ok ' : 'FAIL'} poster: shown until the first frame, the canvas's own background after${posterOk ? '' : `: ${JSON.stringify(q)}`}`)+ }++ // the context lost, and given up (see runContext)+ {+ const r = await page.evaluate((id) => window.runContext(id), ids[0])+ const checks = {+ lost: r.before === false && r.lost === true && r.told === 1,+ 'a load running': r.running === 'context-lost', 'a load after': r.load === 'context-lost', 'frames, a snapshot': r.snapshot === 'string', 'a new player on it': r.newOnLost === 'context-lost',+ release: r.sameTask === false && r.nextTask === true && r.releasedTold === 0 && r.reuse === 'released',+ 'unmount and mount': r.remount.lost === false && r.remount.loaded === true && r.remount.told === 0 && r.remount.share > 0.05,+ 'with release, the one that stays stays': r.withRelease.lost === false && r.withRelease.told === 0,+ 'without, it is pushed out and told': r.withoutRelease.lost === true && r.withoutRelease.told === 1 && r.overLimitLoad === 'context-lost',+ }+ const wrong = Object.entries(checks).filter(([, ok]) => !ok).map(([k]) => k)+ if (wrong.length) failed+++ console.log(`${wrong.length ? 'FAIL' : 'ok '} the context lost and given up${wrong.length ? `: ${wrong.join(', ')}: ${JSON.stringify(r)}` : ': told once, frames idle, loads say so; 40 players came and went beside one that stayed'}`)+ }+// disposing while something runs (see runLifecycle)const life = await page.evaluate((id) => window.runLifecycle(id), ids[0])- const lifeOk = /disposed/.test(life.load) && life.model === null && life.drawn === 0 && life.freed >= life.made && life.glError === 0+ const lifeOk = /AbortError: .*disposed/.test(life.load) && life.model === null && life.drawn === 0 && life.freed >= life.made && life.glError === 0&& life.driver === 1 && !!life.again && life.mic === 'ended' && life.supported === true&& Object.values(life.left).every((n) => n <= 0)if (!lifeOk) failed++@@ -190,6 +330,41 @@ try {if (!vaOk) failed++console.log(`${vaOk ? 'ok ' : 'FAIL'} VRM, loaded again on a wide canvas: aspect ${vrmAgain?.aspect1}, ${vrmAgain?.aspect2}; a failed load kept ${vrmAgain?.kept}, covers ${(vrmAgain?.share * 100).toFixed(1)} %; on a 2D canvas: ${vrmOn2d}`)if (errors.length) { failed++; console.log(`FAIL page errors:\n ${errors.join('\n ')}`) }+ // a BodyPlayer's loads (see runBodyLoads)+ {+ const r = await page.evaluate(() => window.runBodyLoads())+ const blue = r.row.pixel[2] > 200 && r.row.pixel[0] < 50+ const stepsOk = r.row.steps.at(-1).step === 'build' && r.row.steps.filter((s) => s.step === 'download').at(-1).progress === 1+ const ok = r.row.a === 'AbortError' && r.row.b === 'AbortError' && r.row.last === 'loaded' && blue && stepsOk+ && r.held.a === 'a' && r.held.b === 'AbortError' && r.held.d === 'd' && r.held.calls.join() === 'a,d' && r.held.shown === 'd'+ && r.lateLost === false && r.lateLost2 === false && r.late2d === true && r.lateAgain === 'made'+ if (!ok) failed+++ console.log(`${ok ? 'ok ' : 'FAIL'} BodyPlayer, several loads: ${ok ? 'the last plays; one already inside the Body finishes first' : JSON.stringify(r)}`)+ }++ // the context on a BodyPlayer (see runBodyContext)+ {+ const r = await page.evaluate(() => window.runBodyContext())+ const ok = r.vrm.told === 1 && r.vrm.lost === true && r.vrm.load === 'context-lost' && r.vrmReleased.lost === true && r.vrmReleased.reuse === 'released'+ && r.png.lost === false && !!r.png.again && r.png.again !== 'released'+ if (!ok) failed+++ console.log(`${ok ? 'ok ' : 'FAIL'} the context on a BodyPlayer${ok ? ': a VRM is told and released like a Player; a PNGTuber has none to lose' : `: ${JSON.stringify(r)}`}`)+ }++ // what both players promise alike (see runContract)+ for (const kind of ['player', 'png']) {+ const r = await page.evaluate(([k, id]) => window.runContract(k, id), [kind, has((m) => m.rig?.chains?.length) ?? ids[0]])+ const near = (a, b) => Math.abs(a - b) < 1e-6+ const problems = Object.entries({+ 'before a load': r.before === null && r.playing === false, 'model': r.name && r.expressions, 'pixel ratio': r.ratio.join() === '2,0.5,100,200', snapshot: r.snapshot,+ 'frame time': near(r.times[0], 0.01) && near(r.times[1], 0.02), frame: r.frame && r.removed, range: r.clamped.join() === '30,0', 'not a number': r.nan.join() === '0,1,0',+ 'from a driver': r.driver.join() === '0,1', alive: r.alive, away: r.away < 0.05 && r.back > 0.9 && /setAway/.test(r.oldAway),+ ...(kind === 'player' ? { 'canvas to model': r.toModel, poke: r.poke[1] > 0 } : {}), expressions2: r.unknown === false && r.neutral === true, 'play and pause': r.plays === true && r.paused === false,+ 'failed load': r.failed === 'refused' && r.kept, disposed: r.after === null && /disposed/.test(r.disposedLoad) && r.gone === 1,+ }).filter(([, ok]) => !ok).map(([k]) => k)+ if (problems.length) failed+++ console.log(`${problems.length ? 'FAIL' : 'ok '} contract, ${kind}: ${problems.length ? `${problems.join(', ')}: ${JSON.stringify(r)}` : 'the same on both'}`)+ }// a camera that is refused: the face model's download stops with it, unless someone asked for the model ahead of timeawait page.goto(`http://127.0.0.1:${server.address().port}/test/bodies.html`)await page.waitForFunction(() => typeof window.runBodies === 'function')@@ -221,6 +396,41 @@ try {console.log(`${waited && faded ? 'ok ' : 'FAIL'} example/${name}.html: cover first ${waited}, then the character ${faded}`)await p.close()}+ // the example pages say so when the context is lost, on the first screen+ {+ const p = await browser.newPage()+ await p.goto(`http://127.0.0.1:${server.address().port}/example/demo.html`)+ const up = await p.waitForFunction(() => window.player?.model, null, { timeout: 20000 }).then(() => true, () => false)+ await p.evaluate(() => window.player.canvas.getContext('webgl2').getExtension('WEBGL_lose_context').loseContext())+ const said = up && await p.waitForFunction(() => { const r = document.getElementById('lost')?.getBoundingClientRect(); return r && r.top >= 0 && r.bottom <= innerHeight }, null, { timeout: 5000 }).then(() => true, () => false)+ if (!said) failed+++ console.log(`${said ? 'ok ' : 'FAIL'} example/demo.html: the context lost, the page says so`)+ await p.close()+ }+ // React under StrictMode, which mounts everything twice (its files come from a CDN: without a connection, not tested)+ {+ const p = await browser.newPage(), said = []+ p.on('pageerror', (e) => said.push(e.message))+ await p.goto(`http://127.0.0.1:${server.address().port}/example/react.html`)+ const up = await p.waitForFunction(() => document.getElementById('status')?.textContent && window.__react, null, { timeout: 20000 }).then(() => true, () => false)+ if (!up) console.log('skip example/react.html: React did not come from the CDN')+ else {+ await p.waitForFunction(() => /·/.test(document.getElementById('status').textContent), null, { timeout: 20000 })+ const first = await p.textContent('#status')+ await p.click('#next')+ await p.waitForFunction((before) => document.getElementById('status').textContent !== before, first, { timeout: 20000 })+ const second = await p.textContent('#status'), kept = await p.evaluate(() => !document.querySelector('canvas').getContext('webgl2').isContextLost())+ const gl = await p.evaluateHandle(() => document.querySelector('canvas').getContext('webgl2'))+ await p.click('#toggle')+ await p.waitForFunction(() => !document.querySelector('canvas'), null, { timeout: 5000 })+ await new Promise((r) => setTimeout(r, 100))+ const gone = await gl.evaluate((c) => c.isContextLost())+ const ok = first !== second && kept && gone && !said.length && !/Error|LivLiv:/.test(second)+ if (!ok) failed+++ console.log(`${ok ? 'ok ' : 'FAIL'} example/react.html under StrictMode: "${first}", then "${second}"; the context kept while mounted ${kept}, given up on unmount ${gone}${said.length ? `; errors ${said.join(' | ')}` : ''}`)+ }+ await p.close()+ }// ?model=: never a request to another site, however it is written; the page's own files still openfor (const name of ['demo', 'vrm']) {const p = await browser.newPage(), asked = []diff --git a/test/types.mjs b/test/types.mjsindex 834d49a..76fd042 100644--- a/test/types.mjs+++ b/test/types.mjs@@ -16,10 +16,10 @@ fs.rmSync(dir, { recursive: true, force: true })fs.mkdirSync(path.join(dir, 'node_modules', path.dirname(name)), { recursive: true })fs.symlinkSync(root, path.join(dir, 'node_modules', name), 'dir')fs.writeFileSync(path.join(dir, 'use.ts'), `-import { Player, VERSION, type Driver, type ModelInfo, type View } from '${name}'+import { LivLivError, Player, VERSION, type ActionId, type Driver, type LivLivErrorCode, type LoadOptions, type LoadProgress, type ModelInfo, type View } from '${name}'import { preloadFaceModel, startFaceTracking, type FaceTracking } from '${name}/tracking'import { createSpeech, startLipSync, type LipSync } from '${name}/voice'-import { BODY_LIMIT, BodyPlayer, type Body } from '${name}/body'+import { BODY_LIMIT, BodyPlayer, LivLivError as FromBody, type Body } from '${name}/body'import { PngBody } from '${name}/png'import { VrmBody } from '${name}/vrm'declare const canvas: HTMLCanvasElement@@ -32,7 +32,26 @@ const sync: Promise<LipSync> = startLipSync(player)const body: Body = new PngBody()player.use(driver)void [VERSION, model, view, tracking, sync, preloadFaceModel(), createSpeech(player), BODY_LIMIT, new BodyPlayer(canvas, body), new VrmBody()]+const options: LoadOptions = { signal: new AbortController().signal, poster: '/cover.webp', onProgress: (p: LoadProgress) => void (p.step === 'download' ? p.progress : 0) }+const again: Promise<ModelInfo> = player.load('/avatars/sena.livliv', options)+const point: [number, number] = player.canvasToModel(1, 2)+const moved: number = player.poke(point[0], point[1])+const code: LivLivErrorCode | null = again instanceof LivLivError ? again.code : null+const move: ActionId = 'nod'+player.act(move)+player.setAway(true)+player.maxPixelRatio = 1+void [moved, code, new FromBody('http', 'x', { status: 404 }).status]// the types are real, not any: these must be errors+// @ts-expect-error a switch now: setAway+player.act('away')+// @ts-expect-error not part of the API+void player.puppet+// @ts-expect-error nor are the insides+void player.frame+// @ts-expect-error the codes are a list+const nope: LivLivErrorCode = 'nope'+void nope// @ts-expect-error not an actionplayer.act('nope')// @ts-expect-error a name is textdiff --git a/tsconfig.build.json b/tsconfig.build.jsonindex cff411d..c5864b5 100644--- a/tsconfig.build.json+++ b/tsconfig.build.json@@ -4,6 +4,7 @@"noEmit": false,"declaration": true,"emitDeclarationOnly": true,+ "stripInternal": true,"declarationDir": "dist/types","rootDir": ".","tsBuildInfoFile": null