Skip to content

A lightweight 2.5D character player for the web, with lip sync and webcam face tracking.

Player 0.3.0: a new load replaces the one running; load options, errors with a code, context loss

LivLiv27265faa7eed
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c6ec096..09d4635 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -23,3 +23,5 @@ jobs:
- run: npm run test:types
- run: npx playwright-core install --with-deps chromium-headless-shell
- run: npm run test:browser
+ - run: npx playwright-core install --with-deps firefox webkit
+ - run: npm run test:engines
diff --git a/API.md b/API.md
index 60d0b4b..d687060 100644
--- a/API.md
+++ b/API.md
@@ -67,10 +67,10 @@ new Player(canvas: HTMLCanvasElement, options?: PlayerOptions)
`new Player()` throws where WebGL2 cannot start: the browser has none or has it turned off, or the canvas already draws
in 2D. `Player.isSupported()` asks first, so a page can say so or show a picture instead.
-### `load(source): Promise<ModelInfo>`
+### `load(source, options?): Promise<ModelInfo>`
-Loads a model and shows it in place of the one before (loads run one after another; a failed load keeps the old one).
-`source` is one of:
+Loads a model and shows it in place of the one before. A load that fails keeps the old one, whatever goes wrong. `source`
+is one of:
- a URL (string or `URL`) of a `.livliv` (or `.zip`) package, of `model.json`, or of its folder; relative URLs resolve
against the page. A query string (`?token=…`) is sent with every file.
@@ -87,6 +87,49 @@ input.onchange = () => player.load(input.files)
A package is checked before anything is decoded: a newer format than this player reads, a feature it does not have,
paths outside the package, an encrypted or oversized ZIP are refused with an error saying so.
+| option | |
+|---|---|
+| `signal` | an `AbortSignal` that calls the load off: it rejects with an `AbortError` (or the signal's own reason) and the character before stays |
+| `onProgress` | called with `{ step: 'download', progress }` as the files come in (0..1, always rising; bytes for a package, images for a folder; where the size is not known, 0 and then 1), then once with `{ step: 'build' }` as the character goes in |
+| `poster` | the URL of a picture to show until the first frame (a featured character's `cover.webp`). It is the canvas's CSS background, so it fits the canvas's box, adds nothing to the page, and the canvas's own background comes back after. Only while the canvas shows nothing yet: over a character, that one stays until the next is ready |
+
+**A new `load()` replaces one still running**, so the last one asked for is the one that plays, and the download before it
+stops (Safari may keep the connection open). The one replaced rejects with an `AbortError`. Nobody has to catch it: a
+page that only calls `player.load(a); player.load(b)` sees nothing in the console, while a real failure is reported as
+any uncaught rejection is. A signal that is already aborted rejects at once and leaves the running load alone. The last
+moment, when the model goes in, cannot be interrupted and takes a few milliseconds to a few hundred: a load that has
+reached it finishes.
+
+```js
+const stop = new AbortController()
+try {
+ await player.load(url, { signal: stop.signal, poster: 'cover.webp', onProgress: (p) => { if (p.step === 'download') bar.value = p.progress } })
+} catch (e) {
+ if (e.name === 'AbortError') return // replaced by a newer load, or called off
+ if (e.code === 'unreachable') say('Check the connection')
+ else say(e.message)
+}
+```
+
+**Errors.** What a load rejects with for a reason of its own is a `LivLivError` (from `@livliv/player` and
+`@livliv/player/body`): `code` says what went wrong and `status` the HTTP status. The message is for people and can change.
+`new Player()` throws the same class. A camera or microphone keeps the browser's own errors.
+
+| `code` | |
+|---|---|
+| `no-webgl` | this browser or canvas gives no WebGL2 |
+| `canvas-in-use` | the canvas already draws in a way this character cannot (a PNGTuber on a WebGL canvas) |
+| `unreachable` | the file did not come: no connection, a connection that dropped, or another origin without a CORS header |
+| `http` | the server said no (`status`) |
+| `not-a-model` | not a model at all, or not one for this player (a VRM handed to a `Player`) |
+| `invalid` | a model, but something in it is wrong or damaged |
+| `too-large` | over this player's limits, or the GPU's memory |
+| `newer-format` | written for a newer player |
+| `missing-feature` | needs something this page or player does not have |
+| `missing-file` | a file it names is not among the files |
+| `context-lost` | the canvas's WebGL context is gone, see [below](#when-the-context-is-lost) |
+| `released` | the canvas gave its context up with `dispose({ release: true })`: use a new canvas |
+
`ModelInfo`:
| field | |
@@ -119,14 +162,15 @@ top; it never writes them back, so a value you set stays until you change it.
With `idle` on, the idle motion sets the head, eyes, gaze, brows, body and breath, so your values for those count
only with `idle` off (or use a [driver](#drivers) to add to them). `mouthOpen`, `vowel` and `smile` are always yours.
-`PARAM_RANGES` and `defaultParams()` are exported.
+`PARAM_RANGES` and `defaultParams()` are exported. Whoever sets them, the pose stays in these ranges, and a value that is
+not a number counts as the default.
### Expressions and actions
- `setExpression(id, weight = 1): boolean`: eases over to that expression (ids from `expressions`; `null` = neutral).
Returns false for an id the model doesn't have.
-- `act(action)`: a short move over whatever else drives the character: `'nod'`, `'shake'`, `'tilt'`, `'hop'`,
- `'shy'`; `'away'` toggles a dozing pose.
+- `act(action)`: a short move over whatever else drives the character: `'nod'`, `'shake'`, `'tilt'`, `'hop'`, `'shy'`.
+- `setAway(on)`: dozes (eyes closed, head down, a slow breath, expressions hidden), easing in, until `setAway(false)`.
- `lookAt(x, y)`: looks toward a point, each -1..1 from the canvas center (y up), and keeps looking there instead
of following the pointer; `lookAt(null)` goes back to the pointer. Needs `idle`.
- `blink()`: blinks now (with `idle`).
@@ -137,12 +181,63 @@ only with `idle` off (or use a [driver](#drivers) to add to them). `mouthOpen`,
center, `head: true` frames the head instead of the whole drawing (avatars, chat bubbles).
- `play()`, `pause()`, `playing`. The loop also skips drawing while the canvas is scrolled out of view.
- `resize()`: draws now at the canvas's current size. Only needed after resizing a paused player.
-- `snapshot()`: the current frame as a PNG data URL (with the mark).
-- `dispose()`: stops, removes drivers, frees the character's GPU memory. A `load()` still running rejects, and a camera or
- microphone still starting (`startFaceTracking`, `startLipSync`) is let go as soon as it opens. The canvas keeps its WebGL
- context and can take a new `Player` afterwards. A browser allows about 16 contexts at once and drops the oldest past that,
- so a page that shows many characters over time should reuse its canvases.
-- `puppet`: the renderer underneath, for what this API does not cover. Its fields can change between versions.
+- `maxPixelRatio`: as the option, and it can be changed at any time.
+- `snapshot()`: the pose as it is now, as a PNG data URL (with the mark).
+- `canvasToModel(x, y)`: a point on the canvas (CSS px from its top left) in the model's own px, as the last frame drew it.
+- `poke(x, y, strength = 1)`: a flick at a point in the model's px: the swinging pieces near it are thrown (a tap on a twin
+ tail). Returns how many it moved: none on a model without any.
+- `dispose({ release })`: stops, removes drivers, frees the character's GPU memory. A `load()` still running rejects, and a
+ camera or microphone still starting (`startFaceTracking`, `startLipSync`) is let go as soon as it opens.
+
+### Canvases and contexts
+
+A browser allows about 16 WebGL contexts on a page and drops the oldest past that, which can be a character still showing.
+So a page where characters come and go has two ways to stay under it:
+
+- **Reuse canvases.** After `dispose()` the canvas keeps its context and can take a new `Player`. This is the only way that
+ works everywhere: Safari does not get the room back from a released context.
+- **Release what leaves.** `dispose({ release: true })` gives the context up, for a canvas leaving the page. Measured here, 40
+ players came and went beside one that stayed, and it stayed (Chromium, Firefox). The canvas is spent after that: a new
+ player on it throws `released`, unless it is made at once, in the same task, as a framework's unmount and mount is.
+
+In **React**, this is the whole of it, and it is right under StrictMode, which mounts everything twice ([example/react.html](example/react.html)):
+
+```jsx
+function Character({ source }) {
+ const canvas = useRef(null), player = useRef(null)
+ useEffect(() => {
+ const p = player.current = new Player(canvas.current)
+ return () => p.dispose({ release: true })
+ }, [])
+ useEffect(() => {
+ player.current.load(source).catch((e) => { if (e.name !== 'AbortError') console.error(e) })
+ }, [source])
+ return <canvas ref={canvas} style={{ width: 320, height: 400 }} />
+}
+```
+
+### When the context is lost
+
+A browser can take a page's WebGL context away: a phone reclaiming memory while the page is in the background, a graphics
+driver restarting, too many contexts on the page. The character is gone from that canvas and does not come back by itself.
+`player.contextLost` says so, and `player.onContextLost` is called once. Meanwhile frames do nothing, and `load()` rejects with
+`context-lost`. The way back is a new canvas and a new player:
+
+```js
+function show(parent, source) {
+ const canvas = document.createElement('canvas')
+ parent.replaceChildren(canvas)
+ const player = new Player(canvas)
+ player.onContextLost = () => {
+ player.dispose()
+ const again = () => show(parent, source)
+ if (document.hidden) addEventListener('visibilitychange', again, { once: true }) // when someone is back to see it
+ else again()
+ }
+ player.load(source).catch(console.error)
+ return player
+}
+```
### The mark
@@ -311,11 +406,15 @@ await vrm.load('/avatars/me.vrm') // a URL, a File, or files picked
file's own words in `raw`. The player does not enforce them. A page playing other people's avatars should show them and
keep to them, as you would with a LivLiv package's `meta.license`.
- **Your own**: anything with `load(files)`, `render(canvas, pose, w, h, dt)`, `head()` and `dispose()` (the `Body`
- type), and if it cares which canvas it gets, `attach(canvas)`. `pose` has the params after the idle motion and the
+ type), and if it cares which canvas it gets, `attach(canvas)`. One that draws with WebGL can also have `context()`, returning
+ its context, so that `contextLost` and `dispose({ release })` work for it. `pose` has the params after the idle motion and the
drivers, the picture's `move` and each expression's weight.
A failed `load()` keeps the character before, as on a Player. A canvas has one kind of context for good: a PNGTuber draws in 2D, a Player and a VRM in WebGL2, so a PNGTuber on a canvas that was a Player's (or the other way round) is refused with an error; give it its own.
-A BodyPlayer draws no mark and has no `puppet`; `view.head` is not used. Live2D models (`.moc3`) are not supported:
+A BodyPlayer takes the same `load()` options and its errors are the same, with one difference: a Body swaps its character
+inside its own `load`, which cannot be interrupted, so a character already going in finishes and the newer one follows.
+Downloads and loads still waiting are replaced, as on a Player. It draws no mark and has no `poke` or `canvasToModel`;
+`view.head` is not used. `onContextLost` and `dispose({ release })` are for a VRM; a PNGTuber has no context to lose. Live2D models (`.moc3`) are not supported:
reading them takes Live2D's proprietary Cubism Core, under its own licence.
## Untrusted files
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..e3f1534
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,35 @@
+# Changelog
+
+## 0.3.0
+
+The interface settled before the package goes on npm: these change what a page written for 0.2 does.
+
+### What to change
+
+| 0.2 | 0.3 |
+|---|---|
+| `player.act('away')` | `player.setAway(true)` and `setAway(false)`. `act('away')` now throws: it was a switch among one-off moves. |
+| `player.puppet` | No longer in the types. `player.poke()` and `player.canvasToModel()` are what pages used it for. The field still exists and can change in any version. |
+| `player.load(a); player.load(b)` played `a`, then `b` | The new load replaces the one running, which rejects with an `AbortError`. If you catch errors, let this one through: `if (e.name === 'AbortError') return`. |
+| `dispose()` on a load running rejected with an `Error` | It rejects with an `AbortError`, as `load()` on a disposed player does. |
+| Errors told apart by their message | `e.code`, on a `LivLivError`. The messages can change. |
+
+### New
+
+- `load(source, { signal, onProgress, poster })`: call a load off, follow how far it has come, show a picture until the first frame.
+- `LivLivError`, with a `code` for each reason, from `@livliv/player` and `@livliv/player/body`.
+- `player.contextLost` and `player.onContextLost`: the page hears when the WebGL context is gone. See [API.md](API.md#when-the-context-is-lost).
+- `dispose({ release: true })`: gives the WebGL context up, for a canvas leaving the page.
+- `player.setAway()`, `player.poke()`, `player.canvasToModel()`.
+- `example/react.html`.
+
+### Changed
+
+- A model that breaks while it is built leaves the character before as it was, to the pixel. It used to leave an empty canvas.
+- The pose stays in the ranges of `PARAM_RANGES` on `Player` too, as it did on `BodyPlayer`. A value that is not a number counts as the default.
+- `maxPixelRatio` can be read and changed on a `Player`.
+- `snapshot()` on a `Player` draws first, so it is the pose as it is now, on a paused player too.
+- A `BodyPlayer`'s `Frame.time` counts the frames before this one, as a `Player`'s does. Its idle motion runs one frame later.
+- The two classes now share their code. Nothing a `Player` draws changed: frame by frame, with the same random numbers, 1300 frames each of 4 characters came out identical.
+
+See the [commits](https://github.com/livliv-dev/player/commits/main) for what came before 0.3.
diff --git a/README.ja.md b/README.ja.md
index 18be7ca..4690ef8 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -75,7 +75,7 @@ API の全体は [API.md](API.md)(英語)にあります:オプション
Kujira、Kujira Q、Pen は他の方の作品に基づくもので、上記のライセンスとクレジットは適用されません。出典、クレジット、条件は [characters/CREDITS.md](characters/CREDITS.md) をご覧ください。
-各キャラクターには `cover.webp`(512 x 768、約 50 KB)もあります。最初のフレームで、マークは入っていません。キャンバスの下に `object-fit: contain` で置き、準備ができたらキャラクターをフェードインさせます([example/loading.js](example/loading.js) がこの方法です)。ページにすぐ絵が出て、切り替わるときにずれません。
+各キャラクターには `cover.webp`(512 x 768、約 50 KB)もあります。最初のフレームで、マークは入っていません。`player.load(source, { poster })` を使うと、最初のフレームが出るまでキャンバスに表示されます。スクリプトが動く前から絵を出したいときは、キャンバスの下に `object-fit: contain` で置き、準備ができたらキャラクターをフェードインさせます([example/loading.js](example/loading.js) がこの方法です)。
キャラクターの一覧は[英語の README](README.md#featured-characters) にあります。
diff --git a/README.md b/README.md
index 1075c96..62ef696 100644
--- a/README.md
+++ b/README.md
@@ -83,9 +83,9 @@ LivLiv's mascot. The terms are in [characters/LICENSE.md](characters/LICENSE.md)
Kujira, Kujira Q and Pen come from other people's work: that licence and that credit don't cover them. Their sources,
credits and terms are in [characters/CREDITS.md](characters/CREDITS.md).
-Each also has a `cover.webp` (512 x 768, about 50 KB), its first frame without the mark. Show it under the canvas with
-`object-fit: contain` and fade the character in over it, as [example/loading.js](example/loading.js) does: the page has a
-picture at once, and nothing jumps.
+Each also has a `cover.webp` (512 x 768, about 50 KB), its first frame without the mark. `player.load(source, { poster })`
+shows it on the canvas until the first frame. To have it before any script has run, put it under the canvas with
+`object-fit: contain` and fade the character in over it, as [example/loading.js](example/loading.js) does.
| id | name | id | name | id | name |
|---|---|---|---|---|---|
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 1ca952f..fd4a418 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -77,7 +77,7 @@ await png.load({ closed: 'me-closed.png', open: 'me-open.png' })
Kujira、Kujira Q 和 Pen 来自别人的作品,不适用上面的许可和署名。它们的来源、署名和条款见 [characters/CREDITS.md](characters/CREDITS.md)。
-每个角色还带一张 `cover.webp`(512 x 768,约 50 KB),是它的第一帧,不带标。把它放在画布下面(`object-fit: contain`),角色好了再淡入,[example/loading.js](example/loading.js) 就是这样做的:页面马上有画面,也不会跳一下。
+每个角色还带一张 `cover.webp`(512 x 768,约 50 KB),是它的第一帧,不带标。`player.load(source, { poster })` 会在第一帧出来之前把它显示在画布上。如果想在脚本运行之前就有画面,就把它放在画布下面(`object-fit: contain`),角色好了再淡入,[example/loading.js](example/loading.js) 就是这样做的。
角色列表见[英文 README](README.md#featured-characters)。
diff --git a/example/demo.html b/example/demo.html
index 71f3217..e41d43d 100644
--- a/example/demo.html
+++ b/example/demo.html
@@ -23,11 +23,12 @@
<script type="module">
import { Player } from '../dist/index.js'
import { cameraButton } from './camera.js'
- import { loadWithFallback, loading, needWebGL, sameSite } from './loading.js'
+ import { loadWithFallback, loading, needWebGL, sameSite, watchContext } from './loading.js'
needWebGL(Player)
const canvas = document.getElementById('stage'), cover = document.getElementById('cover')
const player = new Player(canvas)
+ watchContext(player)
const bar = document.getElementById('bar'), status = document.getElementById('status')
window.player = player // to try things in the console
@@ -40,16 +41,16 @@
status.textContent = ''
const done = loading(canvas, cover, source)
try {
- const m = await loadWithFallback(player, source)
+ const m = await loadWithFallback(player, source, { onProgress: done.progress })
status.textContent = `${m.name} · ${m.format} ${m.version} · mark: ${m.watermark ?? 'none'}${m.meta?.license ? ` · ${m.meta.license.id}` : ''}`
controls(m)
} catch (e) {
+ if (e.name === 'AbortError') return // another was picked meanwhile
status.textContent = e.message
// a VRM has its own page here
if (/VRM/.test(e.message)) { const a = document.createElement('a'); a.href = 'vrm.html'; a.style.color = 'inherit'; a.textContent = ' Open the VRM page'; status.append(a) }
controls(player.model)
- }
- done()
+ } finally { done() }
}
function controls(m) {
diff --git a/example/gallery.html b/example/gallery.html
index 781599e..b26323c 100644
--- a/example/gallery.html
+++ b/example/gallery.html
@@ -34,7 +34,7 @@
</main>
<script type="module">
import { Player } from '../dist/index.js'
- import { loadWithFallback, loading, needWebGL } from './loading.js'
+ import { loadWithFallback, loading, needWebGL, watchContext } from './loading.js'
// All the featured characters: one plays on the stage (one player, one WebGL context), the rest wait as their
// pictures; a click on one puts it on the stage. ?id=<id> starts with that one (else Refined Nana); ?demo=1 plays expressions and
@@ -48,6 +48,7 @@
needWebGL(Player)
const canvas = document.getElementById('stage'), cover = document.getElementById('cover')
const player = new Player(canvas, { maxPixelRatio: 1.5 })
+ watchContext(player)
const bar = document.getElementById('bar'), name = document.getElementById('name'), credit = document.getElementById('credit'), ul = document.getElementById('cast')
const buttons = new Map()
let current = null, exprs = [], next = -1
@@ -71,8 +72,9 @@
const source = `../characters/${id}.livliv`, done = loading(canvas, cover, source)
let m
try {
- m = await loadWithFallback(player, source).finally(done)
+ m = await loadWithFallback(player, source, { onProgress: done.progress }).finally(done)
} catch (e) { // the one on the stage stays; a click can try this one again
+ if (e.name === 'AbortError') return // another was picked meanwhile
if (id === current) { current = before; press(before); name.textContent = e.message }
return
}
diff --git a/example/index.html b/example/index.html
index 42b2694..4a0cd87 100644
--- a/example/index.html
+++ b/example/index.html
@@ -10,14 +10,15 @@
<p id="note" style="position:fixed;z-index:1;inset:45% 0 auto;margin:0;color:#eef;font:14px system-ui,sans-serif;text-align:center"></p>
<script type="module">
import { Player } from '../dist/index.js'
- import { loadWithFallback, loading } from './loading.js'
+ import { loadWithFallback, loading, watchContext } from './loading.js'
// a featured character; your own export works the same
const note = document.getElementById('note'), canvas = document.getElementById('stage')
if (!Player.isSupported()) { note.textContent = 'This browser has no WebGL2'; throw new Error('no WebGL2') }
const player = new Player(canvas)
+ watchContext(player)
const done = loading(canvas, document.getElementById('cover'), '../characters/d03ft.livliv')
- const model = await loadWithFallback(player, '../characters/d03ft.livliv').catch((e) => { note.textContent = e.message; throw e }).finally(done)
+ const model = await loadWithFallback(player, '../characters/d03ft.livliv', { onProgress: done.progress }).catch((e) => { note.textContent = e.message; throw e }).finally(done)
// each click shows the next expression, then neutral again
let i = -1
diff --git a/example/loading.js b/example/loading.js
index decbfd3..37acc23 100644
--- a/example/loading.js
+++ b/example/loading.js
@@ -24,7 +24,7 @@ export function loading(canvas, cover, source) {
note.textContent = 'Loading…'
note.style.cssText = 'position:absolute;z-index:1;left:50%;bottom:12%;transform:translateX(-50%);margin:0;padding:4px 10px;border-radius:8px;background:#000a;color:#fff;font:13px system-ui,sans-serif;pointer-events:none'
const later = setTimeout(() => canvas.parentElement.append(note), url ? 800 : 150)
- return () => {
+ const done = () => {
waiting = false
clearTimeout(later)
note.remove()
@@ -34,6 +34,9 @@ export function loading(canvas, cover, source) {
canvas.style.opacity = 1
setTimeout(() => { if (me === latest) cover.hidden = true }, 300)
}
+ /** for load()'s onProgress: how far the note says it has come */
+ done.progress = (p) => { note.textContent = p.step === 'download' && p.progress > 0 && p.progress < 1 ? `Loading… ${Math.round(p.progress * 100)} %` : 'Loading…' }
+ return done
}
/** Is this URL on the page's own site? ?model= opens nothing else: a link must not put someone else's file on this
@@ -52,12 +55,26 @@ export function needWebGL(Player) {
throw new Error('no WebGL2')
}
+/** When the browser takes the page's graphics away (a phone reclaiming memory, say), the character is gone from this canvas
+ * for good. These pages say so and offer to start again; a page of your own can make a new canvas and a new player at once:
+ * see "When the context is lost" in API.md. */
+export function watchContext(player) {
+ player.onContextLost = () => {
+ const note = document.createElement('button')
+ note.id = 'lost'
+ note.style.cssText = 'position:fixed;z-index:2;left:16px;right:16px;top:32px;padding:8px 12px;border:0;border-radius:8px;background:#000c;color:#fff;text-align:center;font:14px system-ui,sans-serif'
+ note.textContent = 'The browser took this page\'s graphics away. Tap to start again.'
+ note.onclick = () => location.reload()
+ document.body.append(note)
+ }
+}
+
/** Loads a model, with a featured character's folder as the second chance when its package (one big file) does not come. */
-export async function loadWithFallback(player, source) {
+export async function loadWithFallback(player, source, options) {
try {
- return await player.load(source)
+ return await player.load(source, options)
} catch (e) {
- if (typeof source !== 'string' || !source.endsWith('.livliv')) throw e
- return player.load(source.replace(/\.livliv$/, '/')).catch(() => { throw e })
+ if (e.name === 'AbortError' || typeof source !== 'string' || !source.endsWith('.livliv')) throw e // another was picked meanwhile
+ return player.load(source.replace(/\.livliv$/, '/'), options).catch(() => { throw e })
}
}
diff --git a/example/png.html b/example/png.html
index dcf0554..3db636d 100644
--- a/example/png.html
+++ b/example/png.html
@@ -49,7 +49,7 @@
try {
const m = await player.load(files)
status.textContent = `${m.name} · ${m.format}`
- } catch (e) { status.textContent = e.message }
+ } catch (e) { if (e.name !== 'AbortError') status.textContent = e.message } // (another was picked meanwhile)
}
document.getElementById('pick').onchange = (e) => e.target.files.length && show(e.target.files)
show({ closed: await face(true, false), open: await face(true, true), blink: await face(false, false), 'blink-open': await face(false, true) })
diff --git a/example/react.html b/example/react.html
new file mode 100644
index 0000000..9eb4825
--- /dev/null
+++ b/example/react.html
@@ -0,0 +1,54 @@
+<!doctype html>
+<html lang="en">
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>LivLiv Player in React</title>
+<style>
+ body { margin: 0; font: 15px/1.5 system-ui, sans-serif; background: #f4f2ef; color: #222; padding: 16px; }
+ canvas { display: block; width: 320px; height: 400px; }
+ button { margin: 8px 8px 0 0; }
+</style>
+<script type="importmap">{ "imports": {
+ "react": "https://esm.sh/[email protected]",
+ "react-dom/client": "https://esm.sh/[email protected]/client"
+} }</script>
+<div id="root"></div>
+<p id="status"></p>
+<script type="module">
+ // A player in a React component, under StrictMode, which mounts everything twice. One effect owns the player and gives its
+ // canvas up on unmount; another loads what the props say. StrictMode's second mount comes in the same task, so the new
+ // player keeps the context, and its load replaces the first, which rejects with an AbortError.
+ import { StrictMode, createElement as h, useEffect, useRef, useState } from 'react'
+ import { createRoot } from 'react-dom/client'
+ import { Player } from '../dist/index.js'
+
+ const status = document.getElementById('status')
+
+ function Character({ source }) {
+ const canvas = useRef(null), player = useRef(null)
+ useEffect(() => {
+ const p = player.current = new Player(canvas.current)
+ return () => p.dispose({ release: true })
+ }, [])
+ useEffect(() => {
+ player.current.load(source).then(
+ (m) => { status.textContent = `${m.name} · ${m.format} ${m.version}` },
+ (e) => { if (e.name !== 'AbortError') status.textContent = e.message })
+ }, [source])
+ return h('canvas', { ref: canvas })
+ }
+
+ const CAST = ['k01', 'd03', 'v05']
+
+ function App() {
+ const [who, setWho] = useState(0), [on, setOn] = useState(true)
+ return h('div', null,
+ on && h(Character, { source: `../characters/${CAST[who]}.livliv` }),
+ h('button', { id: 'next', onClick: () => setWho((who + 1) % CAST.length) }, 'next character'),
+ h('button', { id: 'toggle', onClick: () => setOn(!on) }, on ? 'unmount' : 'mount'))
+ }
+
+ createRoot(document.getElementById('root')).render(h(StrictMode, null, h(App)))
+ window.__react = true
+</script>
+</html>
diff --git a/example/vrm.html b/example/vrm.html
index 669ecc4..bf02557 100644
--- a/example/vrm.html
+++ b/example/vrm.html
@@ -29,9 +29,10 @@
import { BodyPlayer } from '../dist/body.js'
import { VrmBody } from '../dist/vrm.js'
import { cameraButton } from './camera.js'
- import { sameSite } from './loading.js'
+ import { sameSite, watchContext } from './loading.js'
const player = new BodyPlayer(document.getElementById('stage'), new VrmBody())
+ watchContext(player)
const status = document.getElementById('status'), bar = document.getElementById('bar')
window.player = player
@@ -50,7 +51,7 @@
for (const a of ['nod', 'shake', 'tilt']) {
const b = document.createElement('button'); b.className = 'x'; b.textContent = a; b.onclick = () => player.act(a); bar.append(b)
}
- } catch (e) { status.textContent = e.message }
+ } catch (e) { if (e.name !== 'AbortError') status.textContent = e.message } // (another was picked meanwhile)
}
document.getElementById('pick').onchange = (e) => e.target.files.length && show(e.target.files[0])
const url = new URLSearchParams(location.search).get('model')
diff --git a/package.json b/package.json
index 14b7dd4..ec0626b 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "@livliv/player",
- "version": "0.2.1",
+ "version": "0.3.0",
"description": "LivLiv Player: a lightweight 2.5D character player for the web, with lip sync and webcam face tracking. Plays LivLiv, VRM and PNGTuber characters; WebGL2, no dependencies in the core.",
"private": true,
"license": "Apache-2.0",
@@ -29,6 +29,7 @@
"dist",
"README.md",
"API.md",
+ "CHANGELOG.md",
"FORMAT.md",
"FORMAT-v3.md",
"schema",
@@ -95,6 +96,7 @@
"test:types": "node test/types.mjs",
"check": "tsc -p tsconfig.json",
"test": "node --disable-warning=ExperimentalWarning --import ./test/ts.mjs --test \"test/*.test.ts\"",
- "test:browser": "node test/smoke.mjs"
+ "test:browser": "node test/smoke.mjs",
+ "test:engines": "node test/smoke.mjs firefox && node test/smoke.mjs webkit"
}
}
diff --git a/src/base.ts b/src/base.ts
new file mode 100644
index 0000000..5d03db2
--- /dev/null
+++ b/src/base.ts
@@ -0,0 +1,413 @@
+// What Player and BodyPlayer have in common, in one place: the frame loop, the pose each frame, expressions, actions,
+// drivers, the pointer and the order loads run in. A subclass says what it draws and how it loads.
+//
+// Each frame the pose is put together from scratch: your `params`, then the idle motion (head, eyes, brows, body, breath;
+// follow.ts: livliv.ai's, following the pointer with the eyes and the face), then the drivers in order, then actions and
+// expressions, kept in the ranges of PARAM_RANGES. Nothing the player adds is written back to `params`, so a value you set
+// stays until you change it, and turning a driver off leaves no trace.
+import { ActionPlayer, type ActionId as AnyAction } from './runtime/actions'
+import { uiLang } from './runtime/format'
+import { PARAM_RANGES, defaultParams, type Params } from './runtime/params'
+import { LivLivError } from './errors'
+import { Idle, type Seen } from './follow'
+import { releaseCanvas } from './release'
+
+export { LivLivError, type LivLivErrorCode } from './errors'
+
+/** a short move: see act() */
+export type ActionId = Exclude<AnyAction, 'away'>
+
+export interface PlayerOptions {
+ /** move by itself: breathe, blink, sway, look around (default true). Off: only your params and drivers move it */
+ idle?: boolean
+ /** the eyes and head follow the pointer anywhere on the page ('page', the default), only over the canvas
+ * ('canvas'), or not at all (false) */
+ followPointer?: 'page' | 'canvas' | false
+ /** drawing-buffer pixels per CSS pixel at most (default 2; 1 saves power on high-DPI screens) */
+ maxPixelRatio?: number
+ /** start the frame loop at once (default true) */
+ autoplay?: boolean
+ /** the language expression names are picked in (`expressions[].label`; a BCP 47 tag, default the browser's) */
+ lang?: string
+}
+
+/** where the character sits in the canvas: x, y move it by shares of the canvas (y up), zoom scales it about the
+ * center; head: frame the head instead of the whole drawing (avatars, chat bubbles; a BodyPlayer does not use it) */
+export interface View { x: number; y: number; zoom: number; head: boolean }
+
+/** one frame being put together, as a driver sees it */
+export interface Frame {
+ /** the pose so far: write to it (for this frame only) */
+ readonly params: Params
+ /** seconds since the last frame, and since the model loaded */
+ readonly dt: number
+ readonly time: number
+ /** a face tracker drives the head this frame (lip sync then leaves the head alone) */
+ tracked: boolean
+ /** add to params, kept in their ranges */
+ add(offsets: Partial<Params>): void
+ /** move the picture: x, y shares of the canvas (y up), s a scale */
+ readonly move: { x: number; y: number; s: number }
+}
+
+/** something that moves the character each frame: face tracking and lip sync are drivers, and your own code can be */
+export interface Driver {
+ /** lower runs first: face tracking -10, yours 0 (default), lip sync 10 */
+ order?: number
+ update(frame: Frame, player: Host): void
+ /** called when it is removed, or the player disposed */
+ dispose?(): void
+}
+
+/** what a driver, the lip sync and the face tracking need of a player: a Player, or a BodyPlayer (@livliv/player/body) */
+export interface Host {
+ /** adds a driver; returns a function that removes it */
+ use(driver: Driver): () => void
+ /** blinks now (with the idle motion on) */
+ blink(): void
+}
+
+/** how far a load has come: the files coming in (0..1, in step order, 1 once they are all here; where the size is not
+ * known it stays at 0 until then), then the character going in */
+export type LoadProgress = { step: 'download'; progress: number } | { step: 'build' }
+
+export interface LoadOptions {
+ /** calls the load off: it rejects with an AbortError (or the signal's own reason) and the character before stays. An
+ * already aborted signal rejects at once and leaves a load that is running alone */
+ signal?: AbortSignal
+ onProgress?: (progress: LoadProgress) => void
+ /** the URL of a picture to show until the first frame (a character's cover.webp), as the canvas's CSS background: it
+ * fits the canvas's box, adds nothing to the page, and the canvas's own background comes back after. Only while the
+ * canvas shows nothing yet: over a character, that one stays until the next is ready */
+ poster?: string
+}
+
+/** what a running load gives its steps
+ * @internal */
+export interface LoadJob {
+ signal: AbortSignal
+ /** the files have come this far (0..1) */
+ progress(share: number): void
+ /** they are all here, and the character goes in */
+ build(): void
+}
+
+/** @internal */
+export const clamp = (v: number, a: number, b: number) => Math.min(b, Math.max(a, v))
+
+/** what a load rejects with when it is called off. A DOMException like a fetch's, so `e.name === 'AbortError'` tells
+ * @internal */
+export const aborted = (why: string) => new DOMException(`LivLiv: ${why}`, 'AbortError')
+
+/** @internal */
+export const contextGone = () => new LivLivError('context-lost', 'LivLiv: this canvas has lost its WebGL context: use a new canvas and a new player')
+
+/** a response's bytes, saying how far they have come
+ * @internal */
+export async function readBody(r: Response, job: LoadJob, share = (p: number) => p): Promise<Blob> {
+ const size = r.headers.get('content-encoding') ? 0 : Number(r.headers.get('content-length')) || 0
+ if (!r.body) return r.blob()
+ const reader = r.body.getReader(), parts: Uint8Array<ArrayBuffer>[] = []
+ let got = 0
+ for (;;) {
+ const { done, value } = await reader.read()
+ if (done) break
+ parts.push(value)
+ got += value.byteLength
+ if (size) job.progress(share(Math.min(1, got / size)))
+ }
+ return new Blob(parts)
+}
+
+const DEFAULTS = defaultParams()
+const KEYS = Object.keys(PARAM_RANGES) as (keyof Params)[]
+
+export abstract class PlayerBase<Info extends { expressions: { id: string }[] }> implements Host {
+ readonly canvas: HTMLCanvasElement
+ /** your values for the pose and face (ranges in PARAM_RANGES). The player never changes them */
+ readonly params: Params = defaultParams()
+ /** see PlayerOptions.idle */
+ idle: boolean
+ /** see PlayerOptions.followPointer */
+ followPointer: 'page' | 'canvas' | false
+ readonly view: View = { x: 0, y: 0, zoom: 1, head: false }
+ /** see PlayerOptions.lang */
+ readonly lang: string
+ /** Called when the canvas's WebGL context is lost (a phone reclaiming the page's GPU memory, a driver reset, too many
+ * contexts on the page). The character is gone from this canvas and does not come back by itself: put a new canvas in
+ * its place and make a new player. On a page with many players, a new context can push out another: release those
+ * that have left (dispose({ release: true })) */
+ onContextLost?: () => void
+
+ /** @internal */
+ protected info: Info | null = null
+ /** @internal */
+ protected disposed = false
+ private idler = new Idle()
+ /** last frame's: a face tracker drove it, the mouth moved (the idle starts no move while the character talks) */
+ private was = { tracked: false, mouth: 0 }
+ /** @internal */
+ protected actions = new ActionPlayer()
+ private drivers: Driver[] = []
+ private expr: Record<string, number> = {}
+ private smile = 0
+ private raf = 0
+ private last = 0
+ private visible = true
+ private io?: IntersectionObserver
+ private queue: Promise<unknown> = Promise.resolve()
+ private lost = false
+ /** the load that a new one replaces */
+ private running: AbortController | null = null
+ /** the canvas's own background while a poster is showing */
+ private posterOver: Record<string, string> | null = null
+ /** @internal */
+ protected frame: Frame & { move: { x: number; y: number; s: number } }
+
+ /** pose: the Params the subclass draws from, written to each frame */
+ constructor(canvas: HTMLCanvasElement, pose: Params, options: PlayerOptions) {
+ this.canvas = canvas
+ this.idle = options.idle ?? true
+ this.followPointer = options.followPointer ?? 'page'
+ this.lang = options.lang || uiLang()
+ this.frame = {
+ params: pose, dt: 0, time: 0, tracked: false, move: { x: 0, y: 0, s: 1 },
+ add(o) {
+ for (const [k, v] of Object.entries(o) as [keyof Params, number | undefined][]) {
+ if (v) pose[k] = clamp(pose[k] + v, PARAM_RANGES[k][0], PARAM_RANGES[k][1])
+ }
+ },
+ }
+ addEventListener('pointermove', this.onMove)
+ document.addEventListener('pointerleave', this.onLeave)
+ canvas.addEventListener('webglcontextlost', this.onLost)
+ if (typeof IntersectionObserver !== 'undefined') {
+ this.io = new IntersectionObserver((e) => { this.visible = e[e.length - 1].isIntersecting })
+ this.io.observe(canvas)
+ }
+ if (options.autoplay ?? true) this.play()
+ }
+
+ /** something is loaded to draw
+ * @internal */
+ protected abstract get drawable(): boolean
+ /** seconds this character has played
+ * @internal */
+ protected abstract get clock(): number
+ /** each expression's weight as shown, eased in draw()
+ * @internal */
+ protected abstract get shownExpr(): Record<string, number>
+ /** where the head is drawn, for where the eyes look
+ * @internal */
+ protected abstract seen(): Seen
+ /** draws the pose put together in frame
+ * @internal */
+ protected abstract show(dt: number): void
+ /** frees what it drew with
+ * @internal */
+ protected abstract free(): void
+
+ /** the context this draws with, where it has one and has started it
+ * @internal */
+ protected abstract get gl(): WebGLRenderingContext | WebGL2RenderingContext | null
+
+ /** the canvas's WebGL context is gone: nothing draws, and load() rejects with code 'context-lost' */
+ get contextLost(): boolean { return this.lost || !!this.gl?.isContextLost() }
+
+ /** the loaded character (null before the first load) */
+ get model(): Info | null { return this.info }
+ get expressions() { return this.info?.expressions ?? [] as Info['expressions'] }
+ get playing() { return this.raf !== 0 }
+
+ /** runs a load: the one running is called off, and this one's steps are told how to be. A load called off rejects like
+ * any other, but nobody has to catch it (player.load(a); player.load(b) reports nothing); a real error is still
+ * reported when nobody catches it
+ * @internal */
+ protected start<T>({ signal, onProgress, poster }: LoadOptions, load: (job: LoadJob) => Promise<T>): Promise<T> {
+ let give!: (v: T) => void, refuse!: (e: unknown) => void
+ const out = new Promise<T>((a, b) => { give = a; refuse = b })
+ const fail = (e: unknown) => {
+ if ((e as Error | null)?.name === 'AbortError') out.catch(() => {})
+ refuse(e)
+ }
+ if (signal?.aborted) { fail(signal.reason); return out }
+ if (this.disposed) { fail(aborted('the player was disposed')); return out }
+ if (this.contextLost) { fail(contextGone()); return out }
+ this.running?.abort(aborted('a newer load took its place'))
+ const ctl = this.running = new AbortController()
+ const calledOff = () => ctl.abort(signal!.reason)
+ signal?.addEventListener('abort', calledOff, { once: true })
+ this.setPoster(poster)
+ const tell = (p: LoadProgress) => { try { onProgress?.(p) } catch (e) { console.error(e) } }
+ let shown = -1, built = false
+ const job: LoadJob = {
+ signal: ctl.signal,
+ progress(share) { if (share > shown && !built) { shown = share; tell({ step: 'download', progress: share }) } },
+ build() { if (!built) { if (shown >= 0 && shown < 1) tell({ step: 'download', progress: 1 }); built = true; tell({ step: 'build' }) } },
+ }
+ const end = () => {
+ signal?.removeEventListener('abort', calledOff)
+ if (this.running === ctl) this.running = null
+ }
+ load(job).then((v) => { end(); give(v) }, (e) => {
+ const mine = this.running === ctl
+ end()
+ if (mine) this.setPoster() // the page can say why; a newer load has already set its own
+ fail(ctl.signal.aborted ? ctl.signal.reason : e)
+ })
+ return out
+ }
+
+ /** the poster over an empty canvas, or none */
+ private setPoster(url?: string) {
+ const s = this.canvas.style
+ if (this.posterOver) { for (const [k, v] of Object.entries(this.posterOver)) s.setProperty(k, v); this.posterOver = null }
+ if (!url || this.info) return
+ const set = { 'background-image': `url(${JSON.stringify(String(url))})`, 'background-position': 'center', 'background-size': 'contain', 'background-repeat': 'no-repeat' }
+ this.posterOver = Object.fromEntries(Object.keys(set).map((k) => [k, s.getPropertyValue(k)]))
+ for (const [k, v] of Object.entries(set)) s.setProperty(k, v)
+ }
+
+ /** runs `load` after the loads before it
+ * @internal */
+ protected inTurn<T>(load: () => Promise<T>): Promise<T> {
+ const run = this.queue.then(load)
+ this.queue = run.catch(() => {})
+ return run
+ }
+
+ /** a new character starts on a neutral face
+ * @internal */
+ protected neutral() { this.expr = {}; this.smile = 0 }
+
+ /** Shows an expression (an id from `expressions`), easing over from the one before; null goes back to neutral.
+ * weight: how strongly, 0..1. False when the model has no such expression */
+ setExpression(id: string | null, weight = 1): boolean {
+ if (id !== null && !this.expressions.some((e) => e.id === id)) return false
+ const w = clamp(weight, 0, 1)
+ this.smile = id === 'smile' ? w : 0
+ this.expr = id === null || id === 'smile' ? {} : { [id]: w }
+ return true
+ }
+
+ /** Plays a short move over whatever drives the character: nod, shake, tilt, hop, shy */
+ act(action: ActionId) {
+ if ((action as string) === 'away') throw new TypeError("LivLiv: act('away') is setAway(true) now")
+ this.actions.play(action)
+ }
+
+ /** Dozes (eyes closed, head down, a slow breath, expressions hidden), easing in, until setAway(false) */
+ setAway(on: boolean) { this.actions.setAway(on) }
+
+ /** Looks toward (x, y), each -1..1 from the canvas center (y up), instead of following the pointer; null goes back to
+ * following it. Needs the idle motion on */
+ lookAt(x: number | null, y = 0) {
+ this.idler.lookAt(this.seen(), x === null ? null : clamp(x, -1, 1), clamp(y, -1, 1), this.canvas.clientWidth, this.canvas.clientHeight)
+ }
+
+ /** Blinks now (with the idle motion on) */
+ blink() { this.idler.blink(this.clock) }
+
+ /** Adds a driver; returns a function that removes it. On a disposed player it is disposed at once: what started
+ * while the player went (a camera, a microphone) does not stay on */
+ use(driver: Driver): () => void {
+ if (this.disposed) { driver.dispose?.(); return () => {} }
+ this.drivers.push(driver)
+ this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))
+ return () => {
+ const i = this.drivers.indexOf(driver)
+ if (i >= 0) { this.drivers.splice(i, 1); driver.dispose?.() }
+ }
+ }
+
+ play() {
+ if (this.raf || this.disposed) return
+ this.last = performance.now()
+ const loop = (now: number) => {
+ this.raf = requestAnimationFrame(loop)
+ const dt = Math.min(0.05, (now - this.last) / 1000)
+ this.last = now
+ if (this.visible) this.draw(dt)
+ }
+ this.raf = requestAnimationFrame(loop)
+ }
+
+ pause() { cancelAnimationFrame(this.raf); this.raf = 0 }
+
+ /** Draws now at the canvas's current size. A playing player follows the size by itself; call this after resizing a
+ * paused one */
+ resize() { this.draw(0) }
+
+ /** The current frame as an image (PNG data URL), as the pose is now, the mark included */
+ snapshot(): string { this.draw(0); return this.canvas.toDataURL('image/png') }
+
+ /** Stops, removes the drivers and frees the GPU memory. A load still running rejects. The canvas keeps its WebGL context
+ * and can take a new player afterwards. release: also give the context up, when the canvas is leaving the page: a
+ * browser allows about 16 at once and drops the oldest past that, which may be a character still showing. A canvas
+ * released this way takes no new player, unless it is made at once, in the same task (a framework's unmount and mount).
+ * Safari does not get the room back: reuse canvases there */
+ dispose({ release = false }: { release?: boolean } = {}) {
+ const gl = this.gl
+ this.stop()
+ if (release) releaseCanvas(this.canvas, gl)
+ }
+
+ private stop() {
+ if (this.disposed) return
+ this.pause()
+ this.disposed = true
+ this.running?.abort(aborted('the player was disposed'))
+ this.setPoster()
+ removeEventListener('pointermove', this.onMove)
+ document.removeEventListener('pointerleave', this.onLeave)
+ this.canvas.removeEventListener('webglcontextlost', this.onLost)
+ this.io?.disconnect()
+ for (const d of this.drivers.splice(0)) d.dispose?.()
+ this.free()
+ this.info = null
+ }
+
+ private draw(dt: number) {
+ // (the context itself, not only the event, which comes a little later: three.js throws starting on a lost context)
+ if (!this.drawable || this.disposed || this.contextLost) return
+ const f = this.frame, P = f.params, t = this.clock, E = this.shownExpr, A = this.actions
+ // the pose from scratch: your values, the idle motion, the drivers
+ this.idler.shown(P)
+ Object.assign(P, this.params)
+ if (this.idle) {
+ this.idler.apply(P, t, dt, { busy: A.busy || this.was.mouth > 0.05, tracked: this.was.tracked,
+ faceFree: !this.smile && !Object.values(E).some((v) => v > 0.02) })
+ }
+ Object.assign(f, { dt, time: t, tracked: false })
+ Object.assign(f.move, { x: 0, y: 0, s: 1 })
+ for (const d of this.drivers) d.update(f, this)
+ this.was = { tracked: f.tracked, mouth: P.mouthOpen }
+ f.add(A.update(dt))
+ if (this.smile) P.smile = Math.max(P.smile, this.smile)
+ if (A.awayW) for (const [k, v] of Object.entries(A.awayPose(t)) as [keyof Params, number][]) P[k] += (v - P[k]) * A.awayW
+ // in range whoever set it; a value that is no number (one NaN stays in the springs for good) counts as not set
+ for (const k of KEYS) {
+ const v = +P[k]
+ P[k] = v === v ? clamp(v, PARAM_RANGES[k][0], PARAM_RANGES[k][1]) : DEFAULTS[k]
+ }
+ // expressions ease in and out (one no longer shown eases back to 0); dozing hides them, each is a face with open eyes
+ for (const k of Object.keys(this.expr)) if (!(k in E)) E[k] = 0
+ for (const k of Object.keys(E)) E[k] += ((this.expr[k] ?? 0) * (1 - A.awayW) - E[k]) * Math.min(1, dt * 8)
+ this.show(dt)
+ if (this.posterOver) this.setPoster()
+ }
+
+ private onMove = (e: PointerEvent) => {
+ if (!this.followPointer) return
+ const r = this.canvas.getBoundingClientRect()
+ // 'canvas': off it the pointer has left the stage, and the character looks back at the viewer
+ const inside = !(e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom)
+ this.idler.pointer(this.seen(), e.clientX - r.left, e.clientY - r.top, r.width, r.height, this.followPointer === 'page' || inside)
+ }
+ private onLeave = () => this.idler.leave()
+ private onLost = () => {
+ this.lost = true
+ this.running?.abort(contextGone())
+ this.onContextLost?.()
+ }
+}
diff --git a/src/body.ts b/src/body.ts
index abe6fe8..24acc69 100644
--- a/src/body.ts
+++ b/src/body.ts
@@ -3,11 +3,14 @@
// from Player: the idle motion, expressions, actions and drivers, so lip sync (@livliv/player/voice) and face tracking
// (@livliv/player/tracking) work for it the same way. Each frame the pose is put together as Player does it (your params,
// the idle motion, the drivers, actions, expressions); the Body turns it into its own picture.
-import { ActionPlayer, type ActionId } from './runtime/actions'
-import { pickLabel, uiLang, type Labels } from './runtime/format'
-import { PARAM_RANGES, defaultParams, type Params } from './runtime/params'
-import { Idle, type Seen } from './follow'
-import type { Driver, Frame, Host, View } from './index'
+import { pickLabel, type Labels } from './runtime/format'
+import { defaultParams, type Params } from './runtime/params'
+import { LivLivError, PlayerBase, aborted, contextGone, readBody, type LoadJob, type LoadOptions, type PlayerOptions } from './base'
+import type { Seen } from './follow'
+import { claimCanvas } from './release'
+
+export { LivLivError }
+export type { ActionId, Driver, Frame, Host, LivLivErrorCode, LoadOptions, LoadProgress, PlayerOptions, View } from './base'
/** an expression a Body can show: `labels` its names by language (BCP 47 tag -> name), `label` the one to fall back on */
export interface BodyExpression { id: string; label: string; labels?: Labels }
@@ -25,6 +28,9 @@ export interface Body {
/** loads from its files (by file name); its name and the expressions it can show. If it rejects, the character before
* must stay as it is */
load(files: Map<string, Blob>): Promise<{ name: string; expressions: BodyExpression[]; meta?: BodyMeta }>
+ /** optional: the WebGL context it draws with, once it has one: what lets the player say when it is lost and give it up
+ * (dispose({ release: true })). A Body without WebGL leaves this out */
+ context?(): WebGLRenderingContext | WebGL2RenderingContext | null
/** draws the pose into the canvas, w × h device pixels, dt seconds after the last frame */
render(canvas: HTMLCanvasElement, pose: Pose, w: number, h: number, dt: number): void
/** where the head is, in shares of the canvas (from the top left) and its radii */
@@ -50,223 +56,110 @@ export interface BodyInfo {
* or URL (`{ closed: 'me-closed.png', open: 'me-open.png' }`). Files are known by their name, without folders */
export type BodySource = string | URL | Blob | FileList | Iterable<File> | Map<string, Blob | string> | Record<string, Blob | string>
-export interface BodyPlayerOptions {
- /** move by itself: breathe, blink, sway, look around (default true) */
- idle?: boolean
- /** the eyes and head follow the pointer anywhere on the page ('page', the default), over the canvas, or not */
- followPointer?: 'page' | 'canvas' | false
- /** drawing-buffer pixels per CSS pixel at most (default 2) */
- maxPixelRatio?: number
- /** start the frame loop at once (default true) */
- autoplay?: boolean
- /** the language expression names are picked in (default the browser's) */
- lang?: string
-}
+export type BodyPlayerOptions = PlayerOptions
/** the most a body's files may hold in all (a VRM is usually 5-50 MB) */
export const BODY_LIMIT = 256 * 2 ** 20
const SMILE: Labels = { en: 'Smile', zh: '笑', 'zh-Hant': '笑', ja: '笑顔' }
-const clamp = (v: number, a: number, b: number) => Math.min(b, Math.max(a, v))
/** Plays a character a Body draws, with the API of Player where it applies (no mark, no `puppet`, no `view.head`) */
-export class BodyPlayer implements Host {
- readonly canvas: HTMLCanvasElement
+export class BodyPlayer extends PlayerBase<BodyInfo> {
readonly body: Body
- /** your values for the pose and face (ranges in PARAM_RANGES). The player never changes them */
- readonly params: Params = defaultParams()
- idle: boolean
- followPointer: 'page' | 'canvas' | false
- /** x, y move the character by shares of the canvas (y up), zoom scales it (`head` is not used) */
- readonly view: View = { x: 0, y: 0, zoom: 1, head: false }
+ /** see PlayerOptions.maxPixelRatio */
maxPixelRatio: number
- readonly lang: string
-
- private info: BodyInfo | null = null
- private pose: Pose = { params: defaultParams(), move: { x: 0, y: 0, s: 1 }, expr: {} }
- private idler = new Idle()
- /** last frame's: a face tracker drove it, the mouth moved */
- private was = { tracked: false, mouth: 0 }
- private actions = new ActionPlayer()
- private drivers: Driver[] = []
- private expr: Record<string, number> = {}
- private smile = 0
+ private pose: Pose
private time = 0
- private raf = 0
- private last = 0
- private visible = true
- private io?: IntersectionObserver
- private queue: Promise<unknown> = Promise.resolve()
- private disposed = false
- private frame: Frame & { move: { x: number; y: number; s: number } }
constructor(canvas: HTMLCanvasElement, body: Body, options: BodyPlayerOptions = {}) {
- this.canvas = canvas
+ claimCanvas(canvas)
+ const pose: Pose = { params: defaultParams(), move: { x: 0, y: 0, s: 1 }, expr: {} }
+ super(canvas, pose.params, options)
+ this.pose = pose
this.body = body
- this.idle = options.idle ?? true
- this.followPointer = options.followPointer ?? 'page'
this.maxPixelRatio = options.maxPixelRatio ?? 2
- this.lang = options.lang || uiLang()
- const P = this.pose.params
- this.frame = {
- params: P, dt: 0, time: 0, tracked: false, move: { x: 0, y: 0, s: 1 },
- add(o) {
- for (const [k, v] of Object.entries(o) as [keyof Params, number | undefined][]) {
- if (v) P[k] = clamp(P[k] + v, PARAM_RANGES[k][0], PARAM_RANGES[k][1])
- }
- },
- }
- addEventListener('pointermove', this.onMove)
- document.addEventListener('pointerleave', this.onLeave)
- if (typeof IntersectionObserver !== 'undefined') {
- this.io = new IntersectionObserver((e) => { this.visible = e[e.length - 1].isIntersecting })
- this.io.observe(canvas)
- }
- if (options.autoplay ?? true) this.play()
}
- /** the loaded character (null before the first load) */
- get model(): BodyInfo | null { return this.info }
- get expressions() { return this.info?.expressions ?? [] }
- get playing() { return this.raf !== 0 }
+ /** the Body's own word for it: asking the canvas would start a context on one that may be meant for 2D
+ * @internal */
+ protected get gl() { return this.body.context?.() ?? null }
+ /** @internal */
+ protected get drawable() { return !!this.info }
+ /** @internal */
+ protected get clock() { return this.time }
+ /** @internal */
+ protected get shownExpr() { return this.pose.expr }
+
+ /** the body's head in the terms the idle's pointAt reads a puppet in (its canvas shares as clip space)
+ * @internal */
+ protected seen(): Seen {
+ const hd = this.info ? this.body.head() : { x: 0.5, y: 0.3, rx: 0.15, ry: 0.12 }
+ return { model: { anchors: { head: { cx: 2 * hd.x - 1, cy: 2 * hd.y - 1, rx: 2 * hd.rx, ry: 2 * hd.ry } } } as Seen['model'], lastView: [1, 1, 0, 0, 2, 2] }
+ }
- /** Loads a character from its files (see BodySource) in place of the one before (a failed load keeps it). Loads run one
- * after another */
- load(source: BodySource): Promise<BodyInfo> {
- const run = this.queue.then(async () => {
- const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }
- stillHere()
+ /** Loads a character from its files (see BodySource) in place of the one before (a failed load keeps it). A new load
+ * replaces one still downloading or waiting, which rejects with an AbortError (see LoadOptions); one whose character is
+ * already going in finishes first, since a Body swaps it inside, and the newer one then follows */
+ load(source: BodySource, options: LoadOptions = {}): Promise<BodyInfo> {
+ return this.start(options, (job) => this.inTurn(async () => {
+ const { signal } = job
+ if (signal.aborted) throw signal.reason
this.body.attach?.(this.canvas)
- const files = await filesOf(source)
- stillHere()
+ const files = await filesOf(source, job)
+ if (signal.aborted) throw signal.reason
+ job.build()
const got = await this.body.load(files)
- if (this.disposed) this.body.dispose() // disposed while it loaded: nothing of it may stay
- stillHere()
+ if (this.disposed) { this.body.dispose(); throw aborted('the player was disposed') } // disposed while it loaded: nothing of it may stay
+ if (this.contextLost) throw contextGone()
const ex = [{ id: 'smile', label: '', labels: SMILE }, ...got.expressions.filter((e) => e.id !== 'smile')
.map((e) => ({ id: e.id, label: e.label, labels: e.labels ?? { en: e.label } }))]
this.info = { name: got.name, format: this.body.format, meta: got.meta ?? null,
expressions: ex.map((e) => ({ id: e.id, label: pickLabel(e.labels, this.lang, e.label || e.id), labels: e.labels })) }
- this.expr = {}; this.smile = 0; this.pose.expr = {}
- if (!this.playing) this.draw(0)
+ this.neutral(); this.pose.expr = {}
+ if (!this.playing) this.resize()
return this.info
- })
- this.queue = run.catch(() => {})
- return run
- }
-
- /** as Player's: an expression by id (smile is the smile param), easing over; null back to neutral */
- setExpression(id: string | null, weight = 1): boolean {
- if (id !== null && !this.expressions.some((e) => e.id === id)) return false
- const w = clamp(weight, 0, 1)
- this.smile = id === 'smile' ? w : 0
- this.expr = id === null || id === 'smile' ? {} : { [id]: w }
- return true
- }
-
- act(action: ActionId) { this.actions.play(action) }
- lookAt(x: number | null, y = 0) {
- this.idler.lookAt(this.seen(), x === null ? null : clamp(x, -1, 1), clamp(y, -1, 1), this.canvas.clientWidth, this.canvas.clientHeight)
- }
- blink() { this.idler.blink(this.time) }
-
- use(driver: Driver): () => void {
- if (this.disposed) { driver.dispose?.(); return () => {} } // as Player's: what started meanwhile does not stay on
- this.drivers.push(driver)
- this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))
- return () => {
- const i = this.drivers.indexOf(driver)
- if (i >= 0) { this.drivers.splice(i, 1); driver.dispose?.() }
- }
+ }))
}
- play() {
- if (this.raf || this.disposed) return
- this.last = performance.now()
- const loop = (now: number) => {
- this.raf = requestAnimationFrame(loop)
- const dt = Math.min(0.05, (now - this.last) / 1000)
- this.last = now
- if (this.visible) this.draw(dt)
- }
- this.raf = requestAnimationFrame(loop)
- }
-
- pause() { cancelAnimationFrame(this.raf); this.raf = 0 }
- /** draws now at the canvas's current size */
- resize() { this.draw(0) }
- /** the current frame as a PNG data URL */
- snapshot(): string { this.draw(0); return this.canvas.toDataURL('image/png') }
-
- dispose() {
- if (this.disposed) return
- this.pause()
- this.disposed = true
- removeEventListener('pointermove', this.onMove)
- document.removeEventListener('pointerleave', this.onLeave)
- this.io?.disconnect()
- for (const d of this.drivers.splice(0)) d.dispose?.()
- this.body.dispose()
- this.info = null
- }
-
- private draw(dt: number) {
- if (!this.info) return
- this.time += dt
- const P = this.pose.params, f = this.frame
- this.idler.shown(P)
- Object.assign(P, this.params)
- if (this.idle) {
- this.idler.apply(P, this.time, dt, { busy: this.actions.busy || this.was.mouth > 0.05, tracked: this.was.tracked,
- faceFree: !this.smile && !Object.values(this.pose.expr).some((v) => v > 0.02) })
- }
- Object.assign(f, { dt, time: this.time, tracked: false })
- Object.assign(f.move, { x: 0, y: 0, s: 1 })
- for (const d of this.drivers) d.update(f, this)
- this.was = { tracked: f.tracked, mouth: P.mouthOpen }
- const A = this.actions
- f.add(A.update(dt))
- if (this.smile) P.smile = Math.max(P.smile, this.smile)
- if (A.awayW) for (const [k, v] of Object.entries(A.awayPose(this.time)) as [keyof Params, number][]) P[k] += (v - P[k]) * A.awayW
- for (const k of Object.keys(P) as (keyof Params)[]) P[k] = clamp(P[k], PARAM_RANGES[k][0], PARAM_RANGES[k][1])
- // expressions ease in and out; dozing hides them
- const E = this.pose.expr
- for (const k of Object.keys(this.expr)) if (!(k in E)) E[k] = 0
- for (const k of Object.keys(E)) E[k] += ((this.expr[k] ?? 0) * (1 - A.awayW) - E[k]) * Math.min(1, dt * 8)
- const V = this.view
- this.pose.move = { x: f.move.x + V.x, y: f.move.y + A.y + V.y, s: f.move.s * V.zoom }
+ /** @internal */
+ protected show(dt: number) {
+ const f = this.frame, V = this.view
+ this.pose.move = { x: f.move.x + V.x, y: f.move.y + this.actions.y + V.y, s: f.move.s * V.zoom }
const dpr = Math.min(this.maxPixelRatio, window.devicePixelRatio || 1)
const w = Math.round(this.canvas.clientWidth * dpr), h = Math.round(this.canvas.clientHeight * dpr)
if (w >= 2 && h >= 2) this.body.render(this.canvas, this.pose, w, h, dt)
+ this.time += dt
}
- /** the body's head in the terms the idle's pointAt reads a puppet in (its canvas shares as clip space) */
- private seen(): Seen {
- const hd = this.info ? this.body.head() : { x: 0.5, y: 0.3, rx: 0.15, ry: 0.12 }
- return { model: { anchors: { head: { cx: 2 * hd.x - 1, cy: 2 * hd.y - 1, rx: 2 * hd.rx, ry: 2 * hd.ry } } } as Seen['model'], lastView: [1, 1, 0, 0, 2, 2] }
- }
- private onMove = (e: PointerEvent) => {
- if (!this.followPointer) return
- const r = this.canvas.getBoundingClientRect()
- const inside = !(e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom)
- this.idler.pointer(this.seen(), e.clientX - r.left, e.clientY - r.top, r.width, r.height, this.followPointer === 'page' || inside)
- }
- private onLeave = () => this.idler.leave()
+ /** @internal */
+ protected free() { this.body.dispose() }
}
/** a body's files by name, URLs fetched */
-async function filesOf(source: BodySource): Promise<Map<string, Blob>> {
+async function filesOf(source: BodySource, job: LoadJob): Promise<Map<string, Blob>> {
const out = new Map<string, Blob>()
let total = 0
+ // files given by URL come one after another: each is its share of the download
+ const given = source instanceof Map ? [...source.values()] : typeof source === 'object' && !(source instanceof URL) && !(Symbol.iterator in source) && !(source instanceof Blob) ? Object.values(source) : []
+ const urls = typeof source === 'string' || source instanceof URL ? 1 : given.filter((v) => typeof v === 'string').length
+ let came = 0
const add = async (key: string, v: Blob | string) => {
const name = key.replace(/\\/g, '/').split('/').filter(Boolean).pop() || 'file'
let b: Blob
if (typeof v === 'string') {
const url = new URL(v, document.baseURI).href
- const r = await fetch(url).catch(() => { throw new Error(`LivLiv: could not load ${url}: unreachable, or on another origin that sends no CORS header`) })
- if (!r.ok) throw new Error(`LivLiv: could not load ${url} (HTTP ${r.status})`)
- b = await r.blob()
+ job.progress(came / urls)
+ const unreachable = (e: unknown) => {
+ if (job.signal.aborted) throw e
+ throw new LivLivError('unreachable', `LivLiv: could not load ${url}: unreachable, or on another origin that sends no CORS header`, { cause: e })
+ }
+ const r = await fetch(url, { signal: job.signal }).catch(unreachable)
+ if (!r.ok) throw new LivLivError('http', `LivLiv: could not load ${url} (HTTP ${r.status})`, { status: r.status })
+ b = await readBody(r, job, (p) => (came + p) / urls).catch(unreachable)
+ came++
+ job.progress(came / urls)
} else b = v
- if ((total += b.size) > BODY_LIMIT) throw new Error(`LivLiv: the files are larger than ${BODY_LIMIT} bytes`)
+ if ((total += b.size) > BODY_LIMIT) throw new LivLivError('too-large', `LivLiv: the files are larger than ${BODY_LIMIT} bytes`)
out.set(name, b)
}
if (typeof source === 'string' || source instanceof URL) {
diff --git a/src/errors.ts b/src/errors.ts
new file mode 100644
index 0000000..6a727f2
--- /dev/null
+++ b/src/errors.ts
@@ -0,0 +1,29 @@
+// What a load rejects with, in a file of its own: a PNGTuber or a VRM page has it without the renderer.
+
+/** what went wrong, to tell apart in code (the message is for people, and can change):
+ * - no-webgl: this browser or canvas gives no WebGL2 (new Player(), a VRM)
+ * - canvas-in-use: the canvas already draws in a way this character cannot (a PNGTuber on a WebGL canvas)
+ * - unreachable: the file did not come: no connection, the connection dropped, or another origin without a CORS header
+ * - http: the server said no (`status`)
+ * - not-a-model: this is not a model at all, or not one for this player (a VRM handed to a Player)
+ * - invalid: a model, but something in it is wrong or damaged
+ * - too-large: over this player's limits, or the GPU's memory
+ * - newer-format: written for a newer player
+ * - missing-feature: needs something this page or player does not have
+ * - missing-file: a file it names is not among the files
+ * - context-lost: the canvas's WebGL context is gone (see `onContextLost`)
+ * - released: the canvas's context was given up by `dispose({ release: true })`: use a new canvas */
+export type LivLivErrorCode = 'no-webgl' | 'canvas-in-use' | 'unreachable' | 'http' | 'not-a-model' | 'invalid' | 'too-large'
+ | 'newer-format' | 'missing-feature' | 'missing-file' | 'context-lost' | 'released'
+
+export class LivLivError extends Error {
+ readonly code: LivLivErrorCode
+ /** with code 'http': the HTTP status */
+ readonly status?: number
+ constructor(code: LivLivErrorCode, message: string, options: { status?: number; cause?: unknown } = {}) {
+ super(message, options.cause === undefined ? undefined : { cause: options.cause })
+ this.name = 'LivLivError'
+ this.code = code
+ if (options.status !== undefined) this.status = options.status
+ }
+}
diff --git a/src/index.ts b/src/index.ts
index adace25..63476b8 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -2,43 +2,26 @@
// livliv.ai. The renderer is livliv.ai's own (src/runtime/puppet.ts); this wraps it in a small API: load, a frame
// loop with the idle motion, params, expressions, actions, the corner mark the package carries, and drivers. Face
// tracking and lip sync are separate imports ('@livliv/player/tracking', '@livliv/player/voice'), so the core stays small.
-//
-// Each frame the pose is put together from scratch: your `params`, then the idle motion (head, eyes, brows, body,
-// breath; follow.ts: livliv.ai's, following the pointer with the eyes and the face), then the drivers in order, then
-// actions and expressions. Nothing the player adds is written back to
-// `params`, so a value you set stays until you change it, and turning a driver off leaves no trace.
-import { ActionPlayer, type ActionId } from './runtime/actions'
-import { langOf, pickLabel, readModel, uiLang, type Labels, type ModelMeta } from './runtime/format'
+import { langOf, pickLabel, readModel, type Labels, type ModelMeta } from './runtime/format'
import { FORMAT, FORMAT_VERSION, LEGACY_FORMAT, PARAM_RANGES, Puppet, defaultParams, loadBlob, type KeyformFile, type ModelDef, type Params } from './runtime/puppet'
-import { Idle } from './follow'
+import { LivLivError, PlayerBase, aborted, contextGone, readBody, type LivLivErrorCode, type LoadJob, type LoadOptions, type PlayerOptions } from './base'
import { Mark } from './mark'
+import { claimCanvas } from './release'
import { ZIP_LIMITS, isZip, unzip } from './zip'
export { FORMAT, FORMAT_VERSION, LEGACY_FORMAT, PARAM_RANGES, defaultParams }
-export type { ActionId, Labels, ModelDef, ModelMeta, Params }
+export type { Labels, ModelDef, ModelMeta, Params }
+export { LivLivError }
+export type { ActionId, Driver, Frame, Host, LivLivErrorCode, LoadOptions, LoadProgress, PlayerOptions, View } from './base'
/** this package's version */
-export const VERSION = '0.2.1'
+export const VERSION = '0.3.0'
/** where a model comes from: the URL of its model.json, of the folder holding it, or of a .livliv package (relative URLs
* resolve against the page); a .livliv package itself (a File or Blob: a ZIP of the folder); or its files: a FileList /
* File[] from an <input type="file"> (a folder with webkitdirectory, or one picked .livliv), or a map of path -> Blob */
export type ModelSource = string | URL | Blob | FileList | Iterable<File> | Map<string, Blob> | Record<string, Blob>
-export interface PlayerOptions {
- /** move by itself: breathe, blink, sway, look around (default true). Off: only your params and drivers move it */
- idle?: boolean
- /** the eyes and head follow the pointer anywhere on the page ('page', the default), only over the canvas
- * ('canvas'), or not at all (false) */
- followPointer?: 'page' | 'canvas' | false
- /** drawing-buffer pixels per CSS pixel at most (default 2; 1 saves power on high-DPI screens) */
- maxPixelRatio?: number
- /** start the frame loop at once (default true) */
- autoplay?: boolean
- /** the language expression names are picked in (`expressions[].label`; a BCP 47 tag, default the browser's) */
- lang?: string
-}
-
export interface ModelInfo {
name: string
/** model.json's format: FORMAT ("livliv-model"), or LEGACY_FORMAT ("livemaker-puppet") for packages exported before v4 */
@@ -59,77 +42,15 @@ export interface ModelInfo {
/** the build's smile on a model whose list was never written (packages from before 2026-10) */
const SMILE: Labels = { zh: '笑', 'zh-Hant': '笑', en: 'Smile', ja: '笑顔' }
-/** where the character sits in the canvas: x, y move it by shares of the canvas (y up), zoom scales it about the
- * center; head: frame the head instead of the whole drawing (avatars, chat bubbles) */
-export interface View { x: number; y: number; zoom: number; head: boolean }
-
-/** one frame being put together, as a driver sees it */
-export interface Frame {
- /** the pose so far: write to it (for this frame only) */
- readonly params: Params
- /** seconds since the last frame, and since the model loaded */
- readonly dt: number
- readonly time: number
- /** a face tracker drives the head this frame (lip sync then leaves the head alone) */
- tracked: boolean
- /** add to params, kept in their ranges */
- add(offsets: Partial<Params>): void
- /** move the picture: x, y shares of the canvas (y up), s a scale */
- readonly move: { x: number; y: number; s: number }
-}
-
-/** something that moves the character each frame: face tracking and lip sync are drivers, and your own code can be */
-export interface Driver {
- /** lower runs first: face tracking -10, yours 0 (default), lip sync 10 */
- order?: number
- update(frame: Frame, player: Host): void
- /** called when it is removed, or the player disposed */
- dispose?(): void
-}
-
-/** what a driver, the lip sync and the face tracking need of a player: a Player, or a BodyPlayer (@livliv/player/body) */
-export interface Host {
- /** adds a driver; returns a function that removes it */
- use(driver: Driver): () => void
- /** blinks now (with the idle motion on) */
- blink(): void
-}
-
-
/** what the runtime throws when a canvas gives it no WebGL2 context */
const NO_WEBGL2 = 'this browser has no WebGL2'
/** Plays one LivLiv model on a canvas. Size the canvas with CSS; the drawing follows its size. */
-export class Player implements Host {
- readonly canvas: HTMLCanvasElement
- /** the renderer underneath, for what this API does not cover. Its fields can change between versions */
+export class Player extends PlayerBase<ModelInfo> {
+ /** the renderer underneath: not part of the API, and not in the types
+ * @internal */
readonly puppet: Puppet
- /** your values for the pose and face (ranges in PARAM_RANGES). The player never changes them */
- readonly params: Params = defaultParams()
- /** see PlayerOptions.idle */
- idle: boolean
- /** see PlayerOptions.followPointer */
- followPointer: 'page' | 'canvas' | false
- readonly view: View = { x: 0, y: 0, zoom: 1, head: false }
- /** see PlayerOptions.lang */
- readonly lang: string
-
- private info: ModelInfo | null = null
private mark: Mark
- private idler = new Idle()
- /** last frame's: a face tracker drove it, the mouth moved (the idle starts no move while the character talks) */
- private was = { tracked: false, mouth: 0 }
- private actions = new ActionPlayer()
- private drivers: Driver[] = []
- private expr: Record<string, number> = {}
- private smile = 0
- private raf = 0
- private last = 0
- private visible = true
- private io?: IntersectionObserver
- private queue: Promise<unknown> = Promise.resolve()
- private disposed = false
- private frame: Frame & { move: { x: number; y: number; s: number } }
/** Whether this browser can play a LivLiv model (WebGL2 starts on a new canvas). Where it cannot, new Player() throws */
static isSupported(): boolean {
@@ -139,72 +60,74 @@ export class Player implements Host {
}
constructor(canvas: HTMLCanvasElement, options: PlayerOptions = {}) {
- this.canvas = canvas
+ claimCanvas(canvas)
+ let puppet: Puppet
try {
- this.puppet = new Puppet(canvas)
+ puppet = new Puppet(canvas)
} catch (e) {
- throw new Error(typeof WebGL2RenderingContext === 'undefined' ? 'LivLiv: this browser has no WebGL2'
+ if (canvas.getContext('webgl2')?.isContextLost()) throw new LivLivError('context-lost', 'LivLiv: this canvas has lost its WebGL context: use a new canvas', { cause: e })
+ throw new LivLivError('no-webgl', typeof WebGL2RenderingContext === 'undefined' ? 'LivLiv: this browser has no WebGL2'
: (e as Error).message === NO_WEBGL2 ? 'LivLiv: could not start WebGL2 on this canvas: it is turned off in this browser, or the canvas already draws in 2D'
- : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`)
+ : `LivLiv: could not start WebGL2 on this canvas (${(e as Error).message})`, { cause: e })
}
+ super(canvas, puppet.params, options)
+ this.puppet = puppet
this.puppet.maxDpr = options.maxPixelRatio ?? 2
this.mark = new Mark(this.puppet.gl)
- this.idle = options.idle ?? true
- this.followPointer = options.followPointer ?? 'page'
- this.lang = options.lang || uiLang()
- const P = this.puppet.params
- this.frame = {
- params: P, dt: 0, time: 0, tracked: false, move: { x: 0, y: 0, s: 1 },
- add(o) {
- for (const [k, v] of Object.entries(o) as [keyof Params, number | undefined][]) {
- if (v) P[k] = Math.max(PARAM_RANGES[k][0], Math.min(PARAM_RANGES[k][1], P[k] + v))
- }
- },
- }
- addEventListener('pointermove', this.onMove)
- document.addEventListener('pointerleave', this.onLeave)
- if (typeof IntersectionObserver !== 'undefined') {
- this.io = new IntersectionObserver((e) => { this.visible = e[e.length - 1].isIntersecting })
- this.io.observe(canvas)
- }
- if (options.autoplay ?? true) this.play()
}
- /** the loaded model (null before the first load) */
- get model(): ModelInfo | null { return this.info }
- get expressions() { return this.info?.expressions ?? [] }
- get playing() { return this.raf !== 0 }
+ /** see PlayerOptions.maxPixelRatio */
+ get maxPixelRatio() { return this.puppet.maxDpr }
+ set maxPixelRatio(v: number) { this.puppet.maxDpr = v }
+
+ /** @internal */
+ protected get gl() { return this.puppet.gl }
+ /** @internal */
+ protected get drawable() { return !!this.puppet.model }
+ /** @internal */
+ protected get clock() { return this.puppet.time }
+ /** @internal */
+ protected get shownExpr() { return this.puppet.expr }
+ /** @internal */
+ protected seen() { return this.puppet }
+
+ /** A point on the canvas (CSS px from its top left) in the model's own px, as the last frame drew it */
+ canvasToModel(x: number, y: number): [number, number] { return this.puppet.canvasToModel(x, y) }
+
+ /** A flick at a point in the model's px (canvasToModel): the swinging pieces passing near it are thrown, the nearest
+ * joint hardest (a tap on a twin tail). Returns how many it moved: none on a model without any */
+ poke(x: number, y: number, strength = 1): number { return this.puppet.poke(x, y, strength) }
- /** Loads a model (see ModelSource) and shows it in place of the one before. Loads run one after another */
- load(source: ModelSource): Promise<ModelInfo> {
- const run = this.queue.then(() => this.loadNow(source))
- this.queue = run.catch(() => {})
- return run
+ /** Loads a model (see ModelSource) and shows it in place of the one before. A load that fails keeps it; a new load
+ * replaces one still running, which rejects with an AbortError (see LoadOptions) */
+ load(source: ModelSource, options: LoadOptions = {}): Promise<ModelInfo> {
+ return this.start(options, (job) => this.loadNow(source, job))
}
- private async loadNow(source: ModelSource): Promise<ModelInfo> {
- const stillHere = () => { if (this.disposed) throw new Error('LivLiv: the player was disposed') }
- stillHere()
- const p = this.puppet
+ private async loadNow(source: ModelSource, job: LoadJob): Promise<ModelInfo> {
+ const p = this.puppet, { signal } = job
let files: Exclude<ModelSource, string | URL> | null = null
if (typeof source === 'string' || source instanceof URL) {
const url = new URL(String(source), document.baseURI)
- if (/\.(livliv|zip)$/i.test(url.pathname)) { files = await fetchPackage(url.href); stillHere() } else {
- try {
- await p.load(url.href, { pack: false })
- } catch (e) {
- throw explain(e, url.href)
- }
+ if (/\.(livliv|zip)$/i.test(url.pathname)) {
+ job.progress(0)
+ files = await fetchPackage(url.href, job)
+ if (signal.aborted) throw signal.reason
+ } else {
+ job.progress(0)
+ await p.load(url.href, { pack: false, signal, onImage: (came, of) => { job.progress(came / of); if (came === of) job.build() } }).catch((e) => { throw explain(e, url.href) })
}
} else files = source
if (files) {
+ job.build()
const { model, image, keyforms } = await fromFiles(files)
- stillHere()
- await p.loadModel(model, image, { keyforms }).catch((e) => { throw explain(e, 'model.json') })
+ if (signal.aborted) throw signal.reason
+ await p.loadModel(model, image, { keyforms, signal }).catch((e) => { throw explain(e, 'model.json') })
}
- // disposed while the images came in: the runtime built its layers all the same; nothing of them may stay or be drawn
- if (this.disposed) p.dispose()
- stillHere()
+ // the model is in: from here it is this load's, whatever came after it
+ // (disposed in the moment since: nothing of it may stay or be drawn)
+ if (this.disposed) { p.dispose(); throw aborted('the player was disposed') }
+ if (p.gl.isContextLost()) throw contextGone()
const m = p.model
const ex = m.expressions || (m.layers.some((l) => l.id === 'expr_smile') ? [{ id: 'smile', label: '', labels: SMILE }] : [])
this.info = {
@@ -219,156 +142,64 @@ export class Player implements Host {
// v4 says which mark it is; before it the text "LivLiv" meant the logo
const kind = m.meta ? m.meta.watermark?.kind ?? 'none' : m.watermark === 'LivLiv' ? 'livliv' : m.watermark ? 'text' : 'none'
this.mark.set(kind, m.watermark ?? null, m.meta?.watermark?.center !== false)
- this.expr = {}; this.smile = 0
- if (!this.playing) this.draw(0)
+ this.neutral()
+ if (!this.playing) this.resize()
return this.info
}
- /** Shows an expression (an id from `expressions`), easing over from the one before; null goes back to neutral.
- * weight: how strongly, 0..1. False when the model has no such expression */
- setExpression(id: string | null, weight = 1): boolean {
- if (id !== null && !this.expressions.some((e) => e.id === id)) return false
- const w = Math.max(0, Math.min(1, weight))
- this.smile = id === 'smile' ? w : 0
- this.expr = id === null || id === 'smile' ? {} : { [id]: w }
- return true
- }
-
- /** Plays a short move over whatever drives the character: nod, shake, tilt, hop, shy. 'away' toggles a dozing pose */
- act(action: ActionId) { this.actions.play(action) }
-
- /** Looks toward (x, y), each -1..1 from the canvas center (y up), instead of following the pointer; null goes back to
- * following it. Needs the idle motion on */
- lookAt(x: number | null, y = 0) {
- this.idler.lookAt(this.puppet, x === null ? null : clamp(x, -1, 1), clamp(y, -1, 1), this.canvas.clientWidth, this.canvas.clientHeight)
- }
-
- /** Blinks now (with the idle motion on) */
- blink() { this.idler.blink(this.puppet.time) }
-
- /** Adds a driver; returns a function that removes it. On a disposed player it is disposed at once: what started
- * while the player went (a camera, a microphone) does not stay on */
- use(driver: Driver): () => void {
- if (this.disposed) { driver.dispose?.(); return () => {} }
- this.drivers.push(driver)
- this.drivers.sort((a, b) => (a.order ?? 0) - (b.order ?? 0))
- return () => {
- const i = this.drivers.indexOf(driver)
- if (i >= 0) { this.drivers.splice(i, 1); driver.dispose?.() }
- }
- }
-
- play() {
- if (this.raf || this.disposed) return
- this.last = performance.now()
- const loop = (now: number) => {
- this.raf = requestAnimationFrame(loop)
- const dt = Math.min(0.05, (now - this.last) / 1000)
- this.last = now
- if (this.visible) this.draw(dt)
- }
- this.raf = requestAnimationFrame(loop)
- }
-
- pause() { cancelAnimationFrame(this.raf); this.raf = 0 }
-
- /** Draws now at the canvas's current size. A playing player follows the size by itself; call this after resizing a
- * paused one */
- resize() { this.draw(0) }
-
- /** The current frame as an image (PNG data URL), the mark included */
- snapshot(): string { return this.canvas.toDataURL('image/png') }
-
- /** Stops, removes the drivers and frees the GPU memory. A load still running rejects. The canvas can take a new Player
- * afterwards */
- dispose() {
- if (this.disposed) return
- this.pause()
- this.disposed = true
- removeEventListener('pointermove', this.onMove)
- document.removeEventListener('pointerleave', this.onLeave)
- this.io?.disconnect()
- for (const d of this.drivers.splice(0)) d.dispose?.()
- const gl = this.puppet.gl
- this.puppet.dispose(); this.mark.dispose(); gl.deleteProgram(this.puppet.prog)
- gl.clearColor(0, 0, 0, 0); gl.clear(gl.COLOR_BUFFER_BIT)
- this.info = null
- }
-
- private draw(dt: number) {
- const p = this.puppet, P = p.params, f = this.frame
- if (!p.model || this.disposed) return
- // the pose from scratch: your values, the idle motion, the drivers
- this.idler.shown(P)
- Object.assign(P, this.params)
- if (this.idle) {
- this.idler.apply(P, p.time, dt, { busy: this.actions.busy || this.was.mouth > 0.05, tracked: this.was.tracked,
- faceFree: !this.smile && !Object.values(p.expr).some((v) => v > 0.02) })
- }
- Object.assign(f, { dt, time: p.time, tracked: false })
- Object.assign(f.move, { x: 0, y: 0, s: 1 })
- for (const d of this.drivers) d.update(f, this)
- this.was = { tracked: f.tracked, mouth: P.mouthOpen }
- const A = this.actions
- f.add(A.update(dt))
- if (this.smile) P.smile = Math.max(P.smile, this.smile)
- if (A.awayW) for (const [k, v] of Object.entries(A.awayPose(p.time)) as [keyof Params, number][]) P[k] += (v - P[k]) * A.awayW
- // expressions ease in and out (one no longer shown eases back to 0); dozing hides them, each is a face with open eyes
- for (const k of Object.keys(this.expr)) if (!(k in p.expr)) p.expr[k] = 0
- for (const k of Object.keys(p.expr)) p.expr[k] += ((this.expr[k] ?? 0) * (1 - A.awayW) - p.expr[k]) * Math.min(1, dt * 8)
- const V = this.view
+ /** @internal */
+ protected show(dt: number) {
+ const p = this.puppet, f = this.frame, V = this.view
p.view = { x: V.x, y: V.y, s: V.zoom }
p.focusHead = V.head
- p.move = { x: f.move.x, y: f.move.y + A.y, s: f.move.s }
+ p.move = { x: f.move.x, y: f.move.y + this.actions.y, s: f.move.s }
p.update(dt)
p.render()
this.mark.draw(performance.now(), p.gl.drawingBufferWidth / Math.max(1, this.canvas.clientWidth))
}
- private onMove = (e: PointerEvent) => {
- if (!this.followPointer) return
- const r = this.canvas.getBoundingClientRect()
- // 'canvas': off it the pointer has left the stage, and the character looks back at the viewer
- const inside = !(e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom)
- this.idler.pointer(this.puppet, e.clientX - r.left, e.clientY - r.top, r.width, r.height, this.followPointer === 'page' || inside)
+ /** @internal */
+ protected free() {
+ const gl = this.puppet.gl
+ this.puppet.dispose(); this.mark.dispose(); gl.deleteProgram(this.puppet.prog)
+ gl.clearColor(0, 0, 0, 0); gl.clear(gl.COLOR_BUFFER_BIT)
}
- private onLeave = () => this.idler.leave()
}
-const clamp = (v: number, a: number, b: number) => Math.min(b, Math.max(a, v))
-
/** errors in the words of someone embedding the player (the runtime's own are written for livliv.ai) */
-function explain(e: unknown, where: string): Error {
- const err = e as Error & { status?: number }
- const msg = String(err?.message || e)
- if (err?.status) return new Error(`LivLiv: could not load ${where} (HTTP ${err.status})`)
+function explain(e: unknown, where: string): unknown {
+ const err = e as Error & { status?: number; code?: LivLivErrorCode }
+ if (err instanceof LivLivError || err?.name === 'AbortError') return e
+ const msg = String(err?.message || e), cause = e
+ if (err?.status) return new LivLivError('http', `LivLiv: could not load ${where} (HTTP ${err.status})`, { status: err.status, cause })
if (msg.startsWith('image ')) {
- return new Error(`LivLiv: could not load the image ${msg.slice(6)}: it is missing, or on another origin that sends no CORS header`)
+ return new LivLivError('missing-file', `LivLiv: could not load the image ${msg.slice(6)}: it is missing, or on another origin that sends no CORS header`, { cause })
}
- if (err instanceof SyntaxError) return new Error(`LivLiv: ${where} is not a model.json (${msg})`)
- if (err instanceof TypeError && /fetch/i.test(msg)) {
- return new Error(`LivLiv: could not load ${where}: unreachable, or on another origin that sends no CORS header`)
+ if (err instanceof SyntaxError) return new LivLivError('not-a-model', `LivLiv: ${where} is not a model.json (${msg})`, { cause })
+ if (err instanceof TypeError && /fetch|load failed|network/i.test(msg)) {
+ return new LivLivError('unreachable', `LivLiv: could not load ${where}: unreachable, or on another origin that sends no CORS header`, { cause })
}
- return new Error(msg.startsWith('LivLiv') ? msg : `LivLiv: ${msg}`)
+ return new LivLivError(err?.code ?? 'invalid', msg.startsWith('LivLiv') ? msg : `LivLiv: ${msg}`, { cause })
}
/** a file a Player cannot take but a BodyPlayer can: the words that say so */
-const VRM_HERE = 'LivLiv: this is a VRM or glTF file, and a Player plays LivLiv models: use a BodyPlayer with VrmBody (@livliv/player/vrm)'
+const vrmHere = () => new LivLivError('not-a-model', 'LivLiv: this is a VRM or glTF file, and a Player plays LivLiv models: use a BodyPlayer with VrmBody (@livliv/player/vrm)')
const isGlb = async (b: Blob) => { const d = new DataView(await b.slice(0, 4).arrayBuffer()); return d.byteLength === 4 && d.getUint32(0, true) === 0x46546c67 }
/** a .livliv package from a URL, as a Blob. One file is one connection to lose where a folder has dozens, and a dropped one
* breaks in the body as often as at the start: tried once more, unless the server said no (4xx). What the page gets is
* our words, never the browser's ("Failed to fetch", "Load failed") */
-async function fetchPackage(url: string): Promise<Blob> {
+async function fetchPackage(url: string, job: LoadJob): Promise<Blob> {
for (let again = true; ; again = false) {
try {
- const r = await fetch(url)
+ const r = await fetch(url, { signal: job.signal })
if (!r.ok) throw Object.assign(new Error(`HTTP ${r.status}`), { status: r.status })
- return await r.blob()
+ return await readBody(r, job)
} catch (e) {
+ if (job.signal.aborted) throw e
const status = (e as { status?: number }).status
if (status && status < 500) throw explain(e, url)
- if (!again) throw status ? explain(e, url) : new Error(`LivLiv: could not load ${url}: unreachable or the connection dropped, or on another origin that sends no CORS header`)
+ if (!again) throw status ? explain(e, url) : new LivLivError('unreachable', `LivLiv: could not load ${url}: unreachable or the connection dropped, or on another origin that sends no CORS header`, { cause: e })
}
}
}
@@ -380,8 +211,8 @@ async function fromFiles(source: Exclude<ModelSource, string | URL>) {
const files = new Map<string, Blob>()
const norm = (k: string) => k.replace(/\\/g, '/').replace(/^(\.\/|\/)+/, '')
const unpack = async (b: Blob) => {
- if (await isGlb(b)) throw new Error(VRM_HERE)
- if (!(await isZip(b))) throw new Error('LivLiv: this file is not a .livliv package (not a ZIP)')
+ if (await isGlb(b)) throw vrmHere()
+ if (!(await isZip(b))) throw new LivLivError('not-a-model', 'LivLiv: this file is not a .livliv package (not a ZIP)')
return unzip(b).catch((e) => { throw explain(e, 'the package') })
}
if (source instanceof Blob) (await unpack(source)).forEach((b, k) => files.set(k, b))
@@ -395,10 +226,10 @@ async function fromFiles(source: Exclude<ModelSource, string | URL>) {
}
}
const json = [...files.keys()].filter(isJson).sort((a, b) => a.length - b.length)[0]
- if (!json && [...files.keys()].some((k) => /\.(vrm|glb)$/i.test(k))) throw new Error(VRM_HERE)
- if (!json) throw new Error('LivLiv: no model.json among the files')
+ if (!json && [...files.keys()].some((k) => /\.(vrm|glb)$/i.test(k))) throw vrmHere()
+ if (!json) throw new LivLivError('missing-file', 'LivLiv: no model.json among the files')
const dir = json.slice(0, json.length - 'model.json'.length)
- if (files.get(json)!.size > ZIP_LIMITS.modelJson) throw new Error(`LivLiv: ${json} is larger than ${ZIP_LIMITS.modelJson} bytes`)
+ if (files.get(json)!.size > ZIP_LIMITS.modelJson) throw new LivLivError('too-large', `LivLiv: ${json} is larger than ${ZIP_LIMITS.modelJson} bytes`)
let model: ModelDef
try {
model = readModel(JSON.parse(await files.get(json)!.text()))
@@ -411,7 +242,7 @@ async function fromFiles(source: Exclude<ModelSource, string | URL>) {
const find = (path: string) => files.get(dir + norm(path)) ?? byName.get(path.slice(path.lastIndexOf('/') + 1))
const image = (path: string) => {
const b = find(path)
- return b ? loadBlob(b) : Promise.reject(new Error(`LivLiv: ${path} is not among the files`))
+ return b ? loadBlob(b) : Promise.reject(new LivLivError('missing-file', `LivLiv: ${path} is not among the files`))
}
let keyforms: KeyformFile | undefined
const kf = model.keyforms ? find(model.keyforms) : undefined // read only within its limit (format.ts LIMITS)
diff --git a/src/png.ts b/src/png.ts
index 2dc9f8c..7982877 100644
--- a/src/png.ts
+++ b/src/png.ts
@@ -3,6 +3,7 @@
// mouth open too). The picture sways with the head, tilts, breathes and hops with the voice, standing on the bottom of
// the canvas. Files by name (any image the browser decodes): closed, open, blink, blink-open, and name.txt.
import type { Body, Pose } from './body'
+import { LivLivError } from './errors'
export const PNG_SLOTS = [
{ id: 'closed', label: 'Mouth closed', need: true },
@@ -22,7 +23,7 @@ export class PngBody implements Body {
attach(canvas: HTMLCanvasElement) {
this.g = canvas.getContext('2d')
- if (!this.g) throw new Error('LivLiv: this canvas already draws with WebGL (a Player or a VRM): a PNGTuber needs a canvas of its own')
+ if (!this.g) throw new LivLivError('canvas-in-use', 'LivLiv: this canvas already draws with WebGL (a Player or a VRM): a PNGTuber needs a canvas of its own')
}
async load(files: Map<string, Blob>) {
@@ -34,11 +35,11 @@ export class PngBody implements Body {
for (const [file, blob] of files) {
const slot = PNG_SLOTS.find((s) => file.replace(/\.[^.]*$/, '').toLowerCase() === s.id)?.id
if (!slot) continue
- const b = await createImageBitmap(blob).catch(() => { throw new Error(`LivLiv: ${file} is not an image this browser can read`) })
+ const b = await createImageBitmap(blob).catch(() => { throw new LivLivError('invalid', `LivLiv: ${file} is not an image this browser can read`) })
pics[slot] = b
- if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) throw new Error(`LivLiv: ${file} is too large (${b.width} x ${b.height})`)
+ if (b.width > MAX_SIDE || b.height > MAX_SIDE || (pixels += b.width * b.height) > MAX_PIXELS) throw new LivLivError('too-large', `LivLiv: ${file} is too large (${b.width} x ${b.height})`)
}
- if (!pics.closed || !pics.open) throw new Error('LivLiv: a PNGTuber needs at least the closed and the open picture')
+ if (!pics.closed || !pics.open) throw new LivLivError('missing-file', 'LivLiv: a PNGTuber needs at least the closed and the open picture')
const txt = files.get('name.txt')
if (txt) name = (await txt.slice(0, 200).text()).trim() || name
} catch (e) {
diff --git a/src/release.ts b/src/release.ts
new file mode 100644
index 0000000..7d5f729
--- /dev/null
+++ b/src/release.ts
@@ -0,0 +1,26 @@
+// A canvas whose WebGL context a player gave up (dispose({ release: true })): the browser's room for contexts back, and the
+// canvas spent: a lost context does not come back by itself, and a new renderer on it fails in ways that say nothing.
+import { LivLivError } from './errors'
+
+const spent = new WeakSet<HTMLCanvasElement>()
+const going = new WeakMap<HTMLCanvasElement, () => void>()
+
+/** gives up the canvas's context, one task from now: losing and restoring within one task does nothing in Chromium and
+ * WebKit, and that is just what a framework's unmount and mount of the same canvas is (React's StrictMode). A player made
+ * on the canvas before then keeps it (see claimCanvas). A canvas without a WebGL context has nothing to give up */
+export function releaseCanvas(canvas: HTMLCanvasElement, gl: WebGLRenderingContext | WebGL2RenderingContext | null) {
+ if (!gl || spent.has(canvas) || going.has(canvas)) return
+ const timer = setTimeout(() => {
+ going.delete(canvas)
+ spent.add(canvas)
+ gl.getExtension('WEBGL_lose_context')?.loseContext()
+ })
+ going.set(canvas, () => clearTimeout(timer))
+}
+
+/** a new player on this canvas: a release still to come is off; one that has happened cannot be undone */
+export function claimCanvas(canvas: HTMLCanvasElement) {
+ going.get(canvas)?.()
+ going.delete(canvas)
+ if (spent.has(canvas)) throw new LivLivError('released', 'LivLiv: this canvas gave up its WebGL context with dispose({ release: true }): use a new canvas')
+}
diff --git a/src/runtime/format.ts b/src/runtime/format.ts
index 0f16483..3705233 100644
--- a/src/runtime/format.ts
+++ b/src/runtime/format.ts
@@ -41,6 +41,9 @@ export const LIMITS = {
* the layers bent by chains times how many bend them. The heaviest of 111 models here: 46 000 and 40 000 */
vertices: 2 ** 21, skinned: 2 ** 20,
}
+/** an error a player can tell apart by its `code` (the SDK's LivLivError); livliv.ai reads only the words */
+export const coded = (code: string, message: string) => Object.assign(new Error(message), { code })
+
/** no number in a model comes near this; past it sums overflow to Infinity and then NaN, and the character is gone */
const BIG = 1e6
const GROUPS = ['head', 'body', 'hair_back', 'hair_side', 'hair_front', 'hair_inner']
@@ -56,23 +59,23 @@ const read = new WeakSet<object>()
* expressions' `label` from their names by language (all of them stay in `labels`). Reading a result again returns
* it as it is */
export function readModel(raw: unknown, lang = uiLang()): ModelDef {
- if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw new Error('not a LivLiv model (model.json is not an object)')
+ if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw coded('not-a-model', 'not a LivLiv model (model.json is not an object)')
if (read.has(raw)) return raw as ModelDef
const m = raw as Record<string, unknown>
if (m.format === undefined || m.format === LEGACY_FORMAT) {
const v = (m.version as number | undefined) ?? LEGACY_VERSION
- if (v > LEGACY_VERSION) throw new Error(`this model says format "${LEGACY_FORMAT}" version ${v}, which never existed`)
+ if (v > LEGACY_VERSION) throw coded('not-a-model', `this model says format "${LEGACY_FORMAT}" version ${v}, which never existed`)
check(m as unknown as ModelDef)
read.add(m)
return m as unknown as ModelDef
}
- if (m.format !== FORMAT) throw new Error(`not a LivLiv model (format "${String(m.format)}")`)
+ if (m.format !== FORMAT) throw coded('not-a-model', `not a LivLiv model (format "${String(m.format)}")`)
const v = m.version as number
if (typeof v !== 'number' || v > FORMAT_VERSION) {
- throw new Error(`this model is format version ${v}; this player reads up to ${FORMAT_VERSION}: update the player`)
+ throw coded('newer-format', `this model is format version ${v}; this player reads up to ${FORMAT_VERSION}: update the player`)
}
const missing = ((m.requires as string[] | undefined) || []).filter((f) => !FEATURES.includes(f))
- if (missing.length) throw new Error(`this model needs ${missing.join(', ')}, which this player does not have: update the player`)
+ if (missing.length) throw coded('missing-feature', `this model needs ${missing.join(', ')}, which this player does not have: update the player`)
const out = lower(m as unknown as V4, lang)
check(out)
read.add(out)
@@ -82,7 +85,7 @@ export function readModel(raw: unknown, lang = uiLang()): ModelDef {
/** what any model must be before it is drawn, whatever wrote it: sizes and counts in range (LIMITS), numbers finite,
* every image and file a path inside the package. Throws on the first thing wrong */
function check(m: ModelDef) {
- const bad = (why: string): never => { throw new Error(`this model is not valid: ${why}`) }
+ const bad = (why: string, code = 'invalid'): never => { throw coded(code, `this model is not valid: ${why}`) }
const num = (v: unknown): v is number => typeof v === 'number' && Number.isFinite(v)
const list = (v: unknown, most: number, what: string) => {
if (v === undefined) return []
@@ -90,9 +93,9 @@ function check(m: ModelDef) {
return v as unknown[]
}
const [w = NaN, h = NaN] = Array.isArray(m.canvas) ? m.canvas : []
- if (!(num(w) && num(h) && w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) bad(`its canvas ${w} x ${h} is out of range`)
+ if (!(num(w) && num(h) && w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) bad(`its canvas ${w} x ${h} is out of range`, w > LIMITS.canvas || h > LIMITS.canvas ? 'too-large' : 'invalid')
if (!Array.isArray(m.layers) || !m.layers.length) bad('it has no layers')
- if (m.layers.length > LIMITS.layers) bad(`it has ${m.layers.length} layers, more than ${LIMITS.layers}`)
+ if (m.layers.length > LIMITS.layers) bad(`it has ${m.layers.length} layers, more than ${LIMITS.layers}`, 'too-large')
const A = m.anchors, hd = A?.head, inRange = (v: unknown) => num(v) && Math.abs(v) <= BIG
if (!hd || ![hd.cx, hd.cy, hd.rx, hd.ry, A.neck_y, A.top_y, A.bottom_y, A.hair_root_y].every(inRange) || !(hd.rx > 1e-3 && hd.ry > 1e-3)
|| !Array.isArray(A.face) || A.face.length !== 4 || !A.face.every(inRange)) bad('its anchors are missing or not numbers')
@@ -174,9 +177,9 @@ function check(m: ModelDef) {
skinned += verts * (bones.length + (overBangs && Array.isArray(bangs) ? bangs.length : 0))
if (overBangs) read += L.w * L.h
}
- if (vertices > LIMITS.vertices) bad(`its layers' meshes would have ${vertices} vertices, more than ${LIMITS.vertices}`)
- if (skinned > LIMITS.skinned) bad(`its chains would bend ${skinned} vertices, more than ${LIMITS.skinned}`)
- if (read > LIMITS.texels) bad(`its brows layers cover ${read} pixels, more than ${LIMITS.texels}`)
+ if (vertices > LIMITS.vertices) bad(`its layers' meshes would have ${vertices} vertices, more than ${LIMITS.vertices}`, 'too-large')
+ if (skinned > LIMITS.skinned) bad(`its chains would bend ${skinned} vertices, more than ${LIMITS.skinned}`, 'too-large')
+ if (read > LIMITS.texels) bad(`its brows layers cover ${read} pixels, more than ${LIMITS.texels}`, 'too-large')
for (const e of list(m.expressions, LIMITS.expressions, 'expressions') as ({ id?: unknown } | null)[]) if (!e || typeof e.id !== 'string') bad('an expression has no id')
if (m.keyforms !== undefined && (typeof m.keyforms !== 'string' || !FILE.test(m.keyforms))) bad('keyforms is not a path inside the package')
if (m.watermark != null && typeof m.watermark !== 'string') bad('its watermark is not text')
@@ -191,17 +194,17 @@ interface V4 {
}
function lower(m: V4, lang: string): ModelDef {
- if (!Array.isArray(m.layers) || !m.layers.length) throw new Error('this model has no layers')
- if (m.layers.length > LIMITS.layers) throw new Error(`this model has ${m.layers.length} layers, more than ${LIMITS.layers}`)
+ if (!Array.isArray(m.layers) || !m.layers.length) throw coded('invalid', 'this model has no layers')
+ if (m.layers.length > LIMITS.layers) throw coded('too-large', `this model has ${m.layers.length} layers, more than ${LIMITS.layers}`)
const [w, h] = m.canvas || []
- if (!(w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) throw new Error(`this model's canvas ${w} x ${h} is out of range`)
+ if (!(w > 0 && h > 0 && w <= LIMITS.canvas && h <= LIMITS.canvas)) throw coded(w > LIMITS.canvas || h > LIMITS.canvas ? 'too-large' : 'invalid', `this model's canvas ${w} x ${h} is out of range`)
for (const L of m.layers) {
for (const p of [L.texture, ...(L.frames || []), L.mask, L.talk, L.brow_tex]) {
- if (p !== undefined && !IMAGE.test(p)) throw new Error(`layer ${L.id}: "${p}" is not an image path of the package`)
+ if (p !== undefined && !IMAGE.test(p)) throw coded('invalid', `layer ${L.id}: "${p}" is not an image path of the package`)
}
}
for (const [k, p] of Object.entries(m.files || {})) {
- if (!k.startsWith('x-') && (typeof p !== 'string' || !FILE.test(p))) throw new Error(`files.${k}: "${p}" is not a path in the package`)
+ if (!k.startsWith('x-') && (typeof p !== 'string' || !FILE.test(p))) throw coded('invalid', `files.${k}: "${p}" is not a path in the package`)
}
const layers = m.layers.map(({ label, expression, ...L }) =>
({ ...L, depth: L.depth ?? 0, name: label ? pickLabel(label, 'en', L.id) : L.id, ...(expression ? { param: expression } : {}) }) as LayerDef)
diff --git a/src/runtime/puppet.ts b/src/runtime/puppet.ts
index 428b785..d7b22de 100644
--- a/src/runtime/puppet.ts
+++ b/src/runtime/puppet.ts
@@ -16,7 +16,7 @@
// the `brows` layer alone where the build gave the brows one (drawn over the bangs); that layer moves with
// the bangs where it lies over them
// - body: lags the head turn / tilt, breathes (shoulders + chest), optional chest spring
-import { LIMITS, latticeCell, readModel, type Labels, type ModelFiles, type ModelMeta } from './format'
+import { LIMITS, coded, latticeCell, readModel, type Labels, type ModelFiles, type ModelMeta } from './format'
import { PARAM_RANGES, defaultParams, smooth, type Params } from './params'
export { PARAM_RANGES, defaultParams, smooth, type Params }
@@ -189,9 +189,11 @@ export const defaultRig = (): Rig => ({
})
/** user-facing "feel" settings (livliv.ai's feel sliders); 1 = default. nod scales up / down on top of turn. hair is how
* freely the hair swings; hairFront (the bangs' springs) and hairBack (long hair's, and the swinging pieces': chains)
- * each take its place when set, so a model saved with hair alone moves both as before */
-export type Tuning = { turn: number; nod: number; hair: number; blink: number; expr: number; mouth: number; breath: number; hairFront?: number; hairBack?: number }
-export const defaultTuning = (): Tuning => ({ turn: 1, nod: 1, hair: 1, blink: 1, expr: 1, mouth: 1, breath: 1 })
+ * each take its place when set, so a model saved with hair alone moves both as before.
+ * gaze: the share of the iris's travel it uses, 0-1. Less shows less of what a far look uncovers in the whites and fixes
+ * none of it; above 1 only shows more, so it stops at 1 */
+export type Tuning = { turn: number; nod: number; hair: number; blink: number; expr: number; mouth: number; breath: number; hairFront?: number; hairBack?: number; gaze: number }
+export const defaultTuning = (): Tuning => ({ turn: 1, nod: 1, hair: 1, blink: 1, expr: 1, mouth: 1, breath: 1, gaze: 1 })
interface GLLayer {
def: LayerDef
@@ -314,6 +316,12 @@ function browLimits(A: ModelDef['anchors'], amp: number): [number, number] {
return any ? [up, down] : [0, 0]
}
+/** what a caller may give a load: see Puppet.load */
+interface LoadExtras { signal?: AbortSignal; onImage?: (came: number, of: number) => void }
+
+/** a layer's images, as they came in */
+interface LayerImages { main: HTMLImageElement; frames: HTMLImageElement[]; mask?: HTMLImageElement; talk?: HTMLImageElement; brows?: HTMLImageElement }
+
interface Spring { x: number; y: number; vx: number; vy: number }
const spring = (): Spring => ({ x: 0, y: 0, vx: 0, vy: 0 })
type Group = 'front' | 'back' | 'side'
@@ -490,13 +498,15 @@ export class Puppet {
/** baseUrl: the model's folder, or its model.json (the SDK: any name ending in .json); the images are read relative to
* it. png: load the lossless layer PNGs even where a WebP copy exists (eval: rest-pose PSNR is scored against the
* source, and quality-92 WebP alone costs 1.3–6.4 dB on line art). pack: try the server's textures.bin first (an
- * exported package has none: the SDK turns it off rather than ask for it and get a 404) */
- async load(baseUrl: string, { png = false, pack: usePack = true } = {}) {
+ * exported package has none: the SDK turns it off rather than ask for it and get a 404). signal: calls the load off
+ * until the model starts going in, which then happens at once and whole; it rejects with the signal's reason and the
+ * model before stays. onImage: each image that has come in, of how many */
+ async load(baseUrl: string, { png = false, pack: usePack = true, signal, onImage }: { png?: boolean; pack?: boolean } & LoadExtras = {}) {
const { file, base, query } = locate(baseUrl)
- const r = await fetch(file + (query ? `?${query}` : ''), { cache: 'no-store' })
+ const r = await fetch(file + (query ? `?${query}` : ''), { cache: 'no-store', signal })
if (!r.ok) throw Object.assign(new Error(r.status === 404 ? 'model not found' : `model.json ${r.status}`), { status: r.status })
const text = await r.text()
- if (text.length > LIMITS.modelJson) throw new Error(`model.json is larger than ${LIMITS.modelJson} bytes`)
+ if (text.length > LIMITS.modelJson) throw coded('too-large', `model.json is larger than ${LIMITS.modelJson} bytes`)
const model = readModel(JSON.parse(text))
// layer files keep their names when a character is rebuilt and are cached for hours, while model.json never is:
// tag them with the version of their bytes the server sends with it (tex), so a save of the feel settings, which rewrites
@@ -512,26 +522,31 @@ export class Puppet {
}
const v = `?v=${tag}${query ? `&${query}` : ''}`
// alongside the images, not before them. Without it the character still plays, on the rig alone
- const keyforms = model.keyforms ? fetch(base + model.keyforms + v).then(async (k) => {
+ const keyforms = model.keyforms ? fetch(base + model.keyforms + v, { signal }).then(async (k) => {
const t = k.ok ? await k.text() : ''
return t && t.length <= LIMITS.keyforms ? JSON.parse(t) as KeyformFile : undefined
}).catch(() => undefined) : undefined
// all of them in one file when the server has it: one request instead of ~30
- const pack = png || !usePack ? null : await loadPack(packUrl(baseUrl, tag))
- await this.loadModel(model, (f) => { const b = pack?.get(f); return b ? loadBlob(b) : loadImage(base + f + v) }, { png, keyforms })
+ const pack = png || !usePack ? null : await loadPack(packUrl(baseUrl, tag), signal)
+ // (an <img> cannot be called off: setting src to '' is an error, removing it never ends)
+ const fetched = (f: string) => (signal ? fetchImage(base + f + v, signal) : loadImage(base + f + v))
+ await this.loadModel(model, (f) => { const b = pack?.get(f); return b ? loadBlob(b) : fetched(f) }, { png, keyforms, signal, onImage })
}
/** a model from anywhere (the SDK: files the user picked): its model.json, and each image it names by its path there */
async loadModel(raw: ModelDef, image: (path: string) => Promise<HTMLImageElement>,
- { png = false, keyforms: keyformsComing }: { png?: boolean; keyforms?: KeyformFile | Promise<KeyformFile | undefined> } = {}) {
+ { png = false, keyforms: keyformsComing, signal, onImage }: { png?: boolean; keyforms?: KeyformFile | Promise<KeyformFile | undefined> } & LoadExtras = {}) {
const model = readModel(raw)
+ let asked = 0, came = 0
+ const arrived = (i: HTMLImageElement) => { onImage?.(++came, asked); return i }
// every image at once: a layer's second image waiting for its first cost a round trip each on a slow path
const [imgs, keyforms] = await Promise.all([Promise.all(model.layers.map(async (L) => {
- const load = (p: string) => image(L.webp && !png ? p.replace(/\.png$/, '.webp') : p)
+ const load = (p: string) => { asked++; return image(L.webp && !png ? p.replace(/\.png$/, '.webp') : p).then(arrived) }
const [main, frames, mask, talk, brows] = await Promise.all([load(L.texture), Promise.all((L.frames || []).map(load)),
L.mask ? load(L.mask) : undefined, L.talk ? load(L.talk) : undefined, L.brow_tex ? load(L.brow_tex) : undefined])
return { main, frames, mask, talk, brows }
})), keyformsComing])
+ if (signal?.aborted) throw signal.reason
// before anything reaches the GPU (an image's size is known once it loads, before it is decoded in full)
const most = Math.min(LIMITS.texture, (this.gl?.getParameter(this.gl.MAX_TEXTURE_SIZE) as number) || LIMITS.texture)
let texels = 0
@@ -539,10 +554,37 @@ export class Puppet {
if (!i) continue
const w = i.naturalWidth || i.width, h = i.naturalHeight || i.height
if (w > most || h > most || (texels += w * h) > LIMITS.texels) {
- throw new Error(`this model's images are too large (${w} x ${h}: ${most} px a side and ${LIMITS.texels} pixels in all at most)`)
+ throw coded('too-large', `this model's images are too large (${w} x ${h}: ${most} px a side and ${LIMITS.texels} pixels in all at most)`)
+ }
+ }
+ this.install(model, imgs, keyforms)
+ }
+
+ /** the model in place of the one before, or, if anything throws on the way (its layers, its keyforms, the GPU out of
+ * memory), the one before exactly as it was: its layers on the GPU, every field, every spring. All in one go: no frame
+ * runs in between. The GPU holds both for that moment */
+ private install(model: ModelDef, imgs: LayerImages[], keyforms: KeyformFile | undefined) {
+ const gl = this.gl
+ // every field; copies of the two this changes in place rather than replaces
+ const before = { ...this, drive: { ...this.drive },
+ chains: Object.fromEntries(Object.entries(this.chains).map(([g, c]) => [g, c.map((q) => ({ ...q }))])) as Puppet['chains'] }
+ const built: GLLayer[] = []
+ for (let i = 0; i < 8 && gl.getError(); i++); // an error left by anything earlier is not this model's
+ try {
+ this.build(model, imgs, keyforms, built)
+ for (let i = 0, e; i < 8 && (e = gl.getError()); i++) { // (one flag at a time, in no set order)
+ if (e === gl.OUT_OF_MEMORY) throw coded('too-large', 'the GPU ran out of memory for this model')
}
+ } catch (e) {
+ this.freeLayers(built)
+ Object.assign(this, before)
+ throw e
}
- this.dispose()
+ this.freeLayers(before.layers)
+ this.freeMask()
+ }
+
+ private build(model: ModelDef, imgs: LayerImages[], keyforms: KeyformFile | undefined, built: GLLayer[]) {
this.model = model
this.chainRuns = (model.rig?.chains || []).map(chainRun)
const mouth = model.layers.find((L) => L.role === 'mouth')
@@ -560,8 +602,9 @@ export class Puppet {
// the brows layers lie over the bangs where the build put them there: they follow what is under them
const fi = model.layers.findIndex((L) => L.group === 'hair_front')
const bangs = fi >= 0 ? { img: imgs[fi].main, def: model.layers[fi] } : undefined
- this.layers = model.layers.map((L, i) => this.buildLayer(L, imgs[i].main, imgs[i].frames, imgs[i].mask, imgs[i].talk, imgs[i].brows,
- L.id === 'brows' || L.role === 'expr_brows' || L.brow_tex ? bangs : undefined))
+ model.layers.forEach((L, i) => built.push(this.buildLayer(L, imgs[i].main, imgs[i].frames, imgs[i].mask, imgs[i].talk, imgs[i].brows,
+ L.id === 'brows' || L.role === 'expr_brows' || L.brow_tex ? bangs : undefined)))
+ this.layers = built
this.byId = new Map(this.layers.map((l) => [l.def.id, l]))
// each iris's room toward the nose: the nose lies toward the eyes' midpoint (a view with one eye has no limit)
this.nasal = {}
@@ -665,14 +708,23 @@ export class Puppet {
}
dispose() {
+ this.freeLayers(this.layers)
+ this.freeMask()
+ this.layers = []
+ }
+
+ private freeLayers(layers: GLLayer[]) {
const gl = this.gl
- for (const L of this.layers) {
+ for (const L of layers) {
gl.deleteTexture(L.tex); L.frames.forEach((t) => gl.deleteTexture(t)); if (L.mask) gl.deleteTexture(L.mask); if (L.talk) gl.deleteTexture(L.talk)
if (L.browTex) gl.deleteTexture(L.browTex)
gl.deleteVertexArray(L.vao); gl.deleteBuffer(L.pos); gl.deleteBuffer(L.uvs); gl.deleteBuffer(L.tris)
}
+ }
+
+ private freeMask() {
+ const gl = this.gl
if (this.maskFb) { gl.deleteFramebuffer(this.maskFb); gl.deleteTexture(this.maskTex); this.maskFb = this.maskTex = null; this.maskSize = [0, 0] }
- this.layers = []
}
private texture(img: HTMLImageElement) {
@@ -680,7 +732,12 @@ export class Puppet {
const t = gl.createTexture()!
gl.bindTexture(gl.TEXTURE_2D, t)
gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, true)
- gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img)
+ try {
+ gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img)
+ } catch (e) {
+ gl.deleteTexture(t)
+ throw e
+ }
gl.generateMipmap(gl.TEXTURE_2D)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR_MIPMAP_LINEAR)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
@@ -858,9 +915,19 @@ export class Puppet {
const tri = new Uint32Array(idx)
gl.bufferData(gl.ELEMENT_ARRAY_BUFFER, tri, gl.STATIC_DRAW)
gl.bindVertexArray(null)
+ // an upload can throw (a canvas's image it may not read): nothing of this layer stays on the GPU
+ const made: WebGLTexture[] = []
+ const texture = (i: HTMLImageElement) => { const t = this.texture(i); made.push(t); return t }
+ let tex: WebGLTexture, frameTex: WebGLTexture[], maskTex: WebGLTexture | undefined, talkTex: WebGLTexture | undefined, browTex: WebGLTexture | undefined
+ try {
+ tex = texture(img); frameTex = frames.map(texture); maskTex = mask && texture(mask); talkTex = talk && texture(talk); browTex = brows && texture(brows)
+ } catch (e) {
+ made.forEach((t) => gl.deleteTexture(t))
+ gl.deleteVertexArray(vao); gl.deleteBuffer(pb); gl.deleteBuffer(ub); gl.deleteBuffer(ib)
+ throw e
+ }
return {
- def: L, tex: this.texture(img), frames: frames.map((f) => this.texture(f)), mask: mask ? this.texture(mask) : undefined,
- talk: talk ? this.texture(talk) : undefined, browTex: brows ? this.texture(brows) : undefined,
+ def: L, tex, frames: frameTex, mask: maskTex, talk: talkTex, browTex,
vao, pos: pb, uvs: ub, tris: ib, count: idx.length, rest, out: new Float32Array(rest.length), tri, tw: img.naturalWidth || img.width, th: img.naturalHeight || img.height,
wHead, wPhys1, wPhys2, dep, wBrow, browSide, browCap, wFront, wJaw, wNeck, wTwist, fol, folB, alt,
outB: folB ? new Float32Array(rest.length) : undefined, stencilRef: 0, soft: 0, maskCh: -1, skin, altSkin,
@@ -1301,6 +1368,7 @@ export class Puppet {
}
}
const roll = (P.angleZ * Math.PI) / 180 * this.rig.roll * this.tuning.turn
+ const g = this.tuning.gaze, gz = typeof g === 'number' && g >= 0 && g < 1 ? g : 1 // the feel settings' eye movement; anything else is the full travel
// rig.neck: how far the chin goes sideways from where the torso alone puts it (model px)
let chinDx = 0
if (this.rig.neck) {
@@ -1322,9 +1390,9 @@ export class Puppet {
const squash = 0.12 + 0.88 * open
const iris = role.startsWith('iris_')
let ix = 0, iy = 0
- if (iris && d.range) { ix = P.eyeBallX * d.range[0]; iy = -P.eyeBallY * d.range[1] }
+ if (iris && d.range) { ix = P.eyeBallX * d.range[0] * gz; iy = -P.eyeBallY * d.range[1] * gz }
// (a model with swinging chains: its highlights follow the gaze too)
- if (this.chained && role.startsWith('shine_') && d.range) { ix = P.eyeBallX * d.range[0]; iy = -P.eyeBallY * d.range[1] }
+ if (this.chained && role.startsWith('shine_') && d.range) { ix = P.eyeBallX * d.range[0] * gz; iy = -P.eyeBallY * d.range[1] * gz }
// irisNasal: past its room toward the nose the inner edge stays put and the far edge travels on (at a
// vertex the shift is nx + nk * its rest x); the eye's highlights go with it
let nx = ix, nk = 0
@@ -1737,16 +1805,17 @@ export function packUrl(baseUrl: string, tag: string) {
/** textures.bin: the header's length (4 bytes, little-endian), the header (JSON {path: [offset,
* length]}, and tex: the images' version), the files. null when it is not there or unreadable: the files are then
* fetched one by one */
-async function loadPack(url: string): Promise<Map<string, Blob> | null> {
+async function loadPack(url: string, signal?: AbortSignal): Promise<Map<string, Blob> | null> {
try {
- const r = await fetch(url)
+ const r = await fetch(url, { signal })
if (!r.ok) return null
const buf = await r.arrayBuffer()
const n = new DataView(buf).getUint32(0, true)
const head: Record<string, [number, number] | string> = JSON.parse(new TextDecoder().decode(new Uint8Array(buf, 4, n)))
return new Map(Object.entries(head).filter((e): e is [string, [number, number]] => Array.isArray(e[1])).map(([k, [o, len]]) =>
[k, new Blob([new Uint8Array(buf, 4 + n + o, len)], { type: k.endsWith('.png') ? 'image/png' : 'image/webp' })]))
- } catch {
+ } catch (e) {
+ if (signal?.aborted) throw e
return null
}
}
@@ -1758,6 +1827,20 @@ export function loadBlob(b: Blob): Promise<HTMLImageElement> {
return loadImage(u).finally(() => URL.revokeObjectURL(u))
}
+/** an image that can be called off: fetched, then decoded like any other */
+async function fetchImage(src: string, signal: AbortSignal): Promise<HTMLImageElement> {
+ let b: Blob
+ try {
+ const r = await fetch(src, { signal })
+ if (!r.ok) throw new Error()
+ b = await r.blob()
+ } catch (e) {
+ if (signal.aborted) throw e
+ throw new Error('image ' + src)
+ }
+ return loadBlob(b)
+}
+
export function loadImage(src: string): Promise<HTMLImageElement> {
return new Promise((res, rej) => {
const i = new Image(); i.crossOrigin = 'anonymous'
diff --git a/src/tracking.ts b/src/tracking.ts
index 61d4b5a..3a20b15 100644
--- a/src/tracking.ts
+++ b/src/tracking.ts
@@ -4,7 +4,7 @@
// cached by the browser after that).
import { FaceTracker, LOST_AFTER, V, loadFaceModel, setFaceModelFiles, type StartStep } from './runtime/tracking'
import { defaultParams, type Params } from './runtime/params'
-import type { Driver, Host } from './index'
+import type { Driver, Host } from './base'
export type { StartStep }
diff --git a/src/voice.ts b/src/voice.ts
index 06da359..19d4888 100644
--- a/src/voice.ts
+++ b/src/voice.ts
@@ -7,7 +7,7 @@
import { VoiceMeter } from './runtime/voice'
import { SpeechPlayer } from './runtime/speech'
import { TalkMotion } from './runtime/talk'
-import type { Driver, Host } from './index'
+import type { Driver, Host } from './base'
export interface LipSyncOptions {
/** how much the voice also moves the head, brows and eyes: 0 (off) .. 2, default 1. Not while a face is tracked */
diff --git a/src/vrm.ts b/src/vrm.ts
index 54db1d8..e42f878 100644
--- a/src/vrm.ts
+++ b/src/vrm.ts
@@ -9,6 +9,7 @@ import type * as T from 'three'
import type { VRM } from '@pixiv/three-vrm'
import type { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader.js'
import type { Body, BodyExpression, BodyMeta, Pose } from './body'
+import { LivLivError, type LivLivErrorCode } from './errors'
const FEELINGS = ['happy', 'angry', 'sad', 'relaxed', 'surprised'] as const
const LABELS: Record<(typeof FEELINGS)[number], Record<string, string>> = {
@@ -60,6 +61,7 @@ export class VrmBody implements Body {
private scene!: T.Scene
private camera!: T.PerspectiveCamera
private renderer: T.WebGLRenderer | null = null
+ private gl: WebGL2RenderingContext | null = null
private bones: Partial<Record<'head' | 'neck' | 'spine' | 'chest', T.Object3D>> = {}
/** the framing: the world height shown (m), and its center */
private frame = { height: 1, y: 1 }
@@ -72,18 +74,21 @@ export class VrmBody implements Body {
attach(canvas: HTMLCanvasElement) {
// the attributes render() gives three.js, which then finds this same context
- if (!canvas.getContext('webgl2', { alpha: true, antialias: true, premultipliedAlpha: true, preserveDrawingBuffer: true })) {
- throw new Error('LivLiv: a VRM needs WebGL2: this browser has none, or this canvas already draws in 2D (a PNGTuber)')
- }
+ const gl = canvas.getContext('webgl2', { alpha: true, antialias: true, premultipliedAlpha: true, preserveDrawingBuffer: true })
+ if (!gl) throw new LivLivError('no-webgl', 'LivLiv: a VRM needs WebGL2: this browser has none, or this canvas already draws in 2D (a PNGTuber)')
+ this.gl = gl
+ if (gl.isContextLost()) throw new LivLivError('context-lost', 'LivLiv: this canvas has lost its WebGL context: use a new canvas')
}
+ context() { return this.gl }
+
async load(files: Map<string, Blob>) {
const entry = [...files].find(([n]) => /\.(vrm|glb)$/i.test(n))
- if (!entry) throw new Error('LivLiv: no .vrm file among the files')
+ if (!entry) throw new LivLivError('missing-file', 'LivLiv: no .vrm file among the files')
await checkGlb(entry[1], entry[0], !this.options.configureLoader)
const [THREE, { GLTFLoader }, { VRMLoaderPlugin, VRMUtils }] = await Promise.all([
import('three'), import('three/examples/jsm/loaders/GLTFLoader.js'), import('@pixiv/three-vrm')]).catch((e) => {
- throw new Error(`LivLiv: could not load three or @pixiv/three-vrm, which a VRM needs: are they installed, and can this page reach them? (${(e as Error).message})`)
+ throw new LivLivError('unreachable', `LivLiv: could not load three or @pixiv/three-vrm, which a VRM needs: are they installed, and can this page reach them? (${(e as Error).message})`)
})
this.THREE = THREE
const loader = new GLTFLoader()
@@ -94,20 +99,20 @@ export class VrmBody implements Body {
try {
gltf = await loader.loadAsync(url)
} catch (e) {
- throw new Error(`LivLiv: ${entry[0]} is not a VRM this player can read (${(e as Error).message})`)
+ throw new LivLivError('not-a-model', `LivLiv: ${entry[0]} is not a VRM this player can read (${(e as Error).message})`, { cause: e })
} finally {
URL.revokeObjectURL(url)
}
const vrm = gltf.userData.vrm as VRM | undefined
if (!vrm) {
VRMUtils.deepDispose(gltf.scene)
- throw new Error(`LivLiv: ${entry[0]} is a glTF without VRM data (exported as plain glTF? VRM exporters write a .vrm)`)
+ throw new LivLivError('not-a-model', `LivLiv: ${entry[0]} is a glTF without VRM data (exported as plain glTF? VRM exporters write a .vrm)`)
}
let meshes = 0
vrm.scene.traverse((o) => { if ((o as T.Mesh).isMesh) meshes++ })
if (!meshes) {
VRMUtils.deepDispose(gltf.scene)
- throw new Error(`LivLiv: ${entry[0]} has a skeleton but no mesh, so there is nothing to draw (export the armature together with its meshes)`)
+ throw new LivLivError('invalid', `LivLiv: ${entry[0]} has a skeleton but no mesh, so there is nothing to draw (export the armature together with its meshes)`)
}
// the new character, apart from the one shown (a failed load leaves that on the canvas): it takes over at the end
VRMUtils.removeUnnecessaryVertices(gltf.scene)
@@ -272,13 +277,13 @@ const COMPRESSED: Record<string, string> = { KHR_draco_mesh_compression: 'Draco'
* fetch them), its compression one the page can decode, and images and meshes within VRM_LIMITS, read from the file's
* headers before anything is decoded */
async function checkGlb(file: Blob, name: string, noDecoders: boolean) {
- const bad = (why: string): never => { throw new Error(`LivLiv: ${name} ${why}`) }
+ const bad = (why: string, code: LivLivErrorCode = 'invalid'): never => { throw new LivLivError(code, `LivLiv: ${name} ${why}`) }
const head = new DataView(await file.slice(0, 20).arrayBuffer())
- if (head.byteLength < 20 || head.getUint32(0, true) !== 0x46546c67) bad('is not a VRM: it should begin as a binary glTF does')
- if (head.getUint32(4, true) !== 2) bad(`is a glTF of version ${head.getUint32(4, true)}, and VRM uses 2`)
+ if (head.byteLength < 20 || head.getUint32(0, true) !== 0x46546c67) bad('is not a VRM: it should begin as a binary glTF does', 'not-a-model')
+ if (head.getUint32(4, true) !== 2) bad(`is a glTF of version ${head.getUint32(4, true)}, and VRM uses 2`, 'not-a-model')
const jsonLen = head.getUint32(12, true)
if (head.getUint32(16, true) !== 0x4e4f534a) bad('is damaged (no JSON where a binary glTF has it)')
- if (jsonLen > VRM_LIMITS.json) bad(`is not one this player reads: its JSON has ${jsonLen} bytes, more than ${VRM_LIMITS.json}`)
+ if (jsonLen > VRM_LIMITS.json) bad(`is not one this player reads: its JSON has ${jsonLen} bytes, more than ${VRM_LIMITS.json}`, 'too-large')
if (20 + jsonLen > file.size) bad('is damaged or cut short')
let gltf: { buffers?: { uri?: string }[]; images?: { uri?: string; bufferView?: number }[]; bufferViews?: { byteOffset?: number; byteLength?: number }[]
accessors?: { count?: number }[]; meshes?: { primitives?: { attributes?: { POSITION?: number } }[] }[]; extensionsUsed?: string[] }
@@ -288,14 +293,14 @@ async function checkGlb(file: Blob, name: string, noDecoders: boolean) {
}
if (noDecoders) {
for (const [ext, what] of Object.entries(COMPRESSED)) {
- if (gltf.extensionsUsed?.includes(ext)) bad(`is compressed with ${what}, and this page has no decoder for that. Export it uncompressed, or pass VrmBody a configureLoader`)
+ if (gltf.extensionsUsed?.includes(ext)) bad(`is compressed with ${what}, and this page has no decoder for that. Export it uncompressed, or pass VrmBody a configureLoader`, 'missing-feature')
}
}
const positions = new Set<number>()
for (const m of gltf.meshes ?? []) for (const p of m?.primitives ?? []) if (typeof p?.attributes?.POSITION === 'number') positions.add(p.attributes.POSITION)
let vertices = 0
for (const i of positions) vertices += gltf.accessors?.[i]?.count ?? 0
- if (vertices > VRM_LIMITS.vertices) bad(`has ${vertices} vertices, more than ${VRM_LIMITS.vertices}`)
+ if (vertices > VRM_LIMITS.vertices) bad(`has ${vertices} vertices, more than ${VRM_LIMITS.vertices}`, 'too-large')
// where the binary chunk starts, and each picture's size from its first bytes
const binary = 20 + jsonLen + 8
let texels = 0
@@ -305,7 +310,7 @@ async function checkGlb(file: Blob, name: string, noDecoders: boolean) {
const size = await pictureSize(file.slice(binary + (bv.byteOffset ?? 0), binary + (bv.byteOffset ?? 0) + (bv.byteLength ?? 0)))
if (!size) continue
if (size[0] > VRM_LIMITS.textureSide || size[1] > VRM_LIMITS.textureSide || (texels += size[0] * size[1]) > VRM_LIMITS.texels) {
- bad(`has pictures too large for this player (${size[0]} x ${size[1]}; ${VRM_LIMITS.textureSide} px a side and ${VRM_LIMITS.texels} pixels in all at most)`)
+ bad(`has pictures too large for this player (${size[0]} x ${size[1]}; ${VRM_LIMITS.textureSide} px a side and ${VRM_LIMITS.texels} pixels in all at most)`, 'too-large')
}
}
}
diff --git a/src/zip.ts b/src/zip.ts
index 300c73e..66e27e8 100644
--- a/src/zip.ts
+++ b/src/zip.ts
@@ -4,10 +4,13 @@
// what a package can hold: no encryption, no ZIP64, no paths outside the package.
import { LIMITS } from './runtime/format'
+import { LivLivError, type LivLivErrorCode } from './errors'
/** what a package may hold before anything is decoded; model.json and keyforms.json as format.ts says */
export const ZIP_LIMITS = { entries: 2048, bytes: 256 * 2 ** 20, modelJson: LIMITS.modelJson, keyforms: LIMITS.keyforms }
+const fail = (why: string, code: LivLivErrorCode = 'invalid') => new LivLivError(code, `LivLiv: ${why}`)
+
const TYPES: Record<string, string> = { webp: 'image/webp', png: 'image/png', jpg: 'image/jpeg', jpeg: 'image/jpeg', json: 'application/json' }
/** whether these bytes begin a ZIP (a local file header, or the end record of an empty one) */
@@ -23,16 +26,16 @@ export async function unzip(zip: Blob): Promise<Map<string, Blob>> {
const tail = new DataView(await zip.slice(zip.size - tailLen).arrayBuffer())
let e = -1
for (let i = tail.byteLength - 22; i >= 0; i--) if (tail.getUint32(i, true) === 0x06054b50) { e = i; break }
- if (e < 0) throw new Error('this is not a ZIP (no end record)')
+ if (e < 0) throw fail('this is not a ZIP (no end record)', 'not-a-model')
const count = tail.getUint16(e + 10, true), cdSize = tail.getUint32(e + 12, true), cdOff = tail.getUint32(e + 16, true)
- if (count === 0xffff || cdSize === 0xffffffff || cdOff === 0xffffffff) throw new Error('ZIP64 packages are not supported')
- if (count > ZIP_LIMITS.entries) throw new Error(`the package has ${count} entries, more than ${ZIP_LIMITS.entries}`)
+ if (count === 0xffff || cdSize === 0xffffffff || cdOff === 0xffffffff) throw fail('ZIP64 packages are not supported', 'too-large')
+ if (count > ZIP_LIMITS.entries) throw fail(`the package has ${count} entries, more than ${ZIP_LIMITS.entries}`, 'too-large')
const cd = new DataView(await zip.slice(cdOff, cdOff + cdSize).arrayBuffer())
const utf8 = new TextDecoder()
const out = new Map<string, Blob>()
let total = 0
for (let o = 0, n = 0; n < count; n++) {
- if (o + 46 > cd.byteLength || cd.getUint32(o, true) !== 0x02014b50) throw new Error('the package is damaged (central directory)')
+ if (o + 46 > cd.byteLength || cd.getUint32(o, true) !== 0x02014b50) throw fail('the package is damaged (central directory)')
const flags = cd.getUint16(o + 8, true), method = cd.getUint16(o + 10, true)
const csize = cd.getUint32(o + 20, true), usize = cd.getUint32(o + 24, true)
const nlen = cd.getUint16(o + 28, true), elen = cd.getUint16(o + 30, true), clen = cd.getUint16(o + 32, true)
@@ -43,30 +46,30 @@ export async function unzip(zip: Blob): Promise<Map<string, Blob>> {
const base = name.slice(name.lastIndexOf('/') + 1)
if (name.endsWith('/') || name.startsWith('__MACOSX/') || base.startsWith('._') || base === '.DS_Store' || base === 'Thumbs.db') continue
if (name.startsWith('/') || /^[A-Za-z]:/.test(name) || name.split('/').some((s) => s === '..' || s === '.')) {
- throw new Error(`the package holds a path outside itself: ${raw}`)
+ throw fail(`the package holds a path outside itself: ${raw}`)
}
- if (flags & 1) throw new Error(`${name} is encrypted`)
- if (out.has(name)) throw new Error(`${name} is in the package twice`)
+ if (flags & 1) throw fail(`${name} is encrypted`)
+ if (out.has(name)) throw fail(`${name} is in the package twice`)
const isKeyforms = base === 'keyforms.json'
const cap = base === 'model.json' ? ZIP_LIMITS.modelJson : isKeyforms ? ZIP_LIMITS.keyforms : ZIP_LIMITS.bytes
- if (usize > cap || (total += usize) > ZIP_LIMITS.bytes) throw new Error(`${name} is too large (${usize} bytes)`)
+ if (usize > cap || (total += usize) > ZIP_LIMITS.bytes) throw fail(`${name} is too large (${usize} bytes)`, 'too-large')
const h = new DataView(await zip.slice(local, local + 30).arrayBuffer())
- if (h.byteLength < 30 || h.getUint32(0, true) !== 0x04034b50) throw new Error(`the package is damaged (${name})`)
+ if (h.byteLength < 30 || h.getUint32(0, true) !== 0x04034b50) throw fail(`the package is damaged (${name})`)
const start = local + 30 + h.getUint16(26, true) + h.getUint16(28, true)
const data = zip.slice(start, start + csize)
const type = TYPES[base.slice(base.lastIndexOf('.') + 1).toLowerCase()] || ''
- if (start + csize > zip.size) throw new Error(`the package is damaged (${name} runs past its end)`)
+ if (start + csize > zip.size) throw fail(`the package is damaged (${name} runs past its end)`)
if (method === 0) {
- if (csize !== usize) throw new Error(`the package is damaged (${name})`)
+ if (csize !== usize) throw fail(`the package is damaged (${name})`)
out.set(name, data.slice(0, csize, type))
} else if (method === 8) {
if (!canInflate()) {
if (isKeyforms) { console.warn(`LivLiv: this browser cannot inflate ${name}; the character plays without it`); continue }
- throw new Error(`this browser cannot open the compressed ${name} (no DecompressionStream): store it uncompressed`)
+ throw fail(`this browser cannot open the compressed ${name} (no DecompressionStream): store it uncompressed`, 'missing-feature')
}
out.set(name, await inflate(data, usize, name, type))
} else {
- throw new Error(`${name} uses ZIP method ${method}; a package stores its files or deflates them`)
+ throw fail(`${name} uses ZIP method ${method}; a package stores its files or deflates them`)
}
}
return out
@@ -90,12 +93,12 @@ async function inflate(data: Blob, size: number, name: string, type: string): Pr
const { done, value } = await reader.read()
if (done) break
n += value.length
- if (n > size) { void reader.cancel(); throw new Error(`${name} inflates past its ${size} bytes`) }
+ if (n > size) { void reader.cancel(); throw fail(`${name} inflates past its ${size} bytes`) }
parts.push(value)
}
} catch (e) {
- throw (e as Error).message?.startsWith(name) ? e : new Error(`the package is damaged (${name}: ${(e as Error).message})`)
+ throw (e as Error).message?.startsWith(name) ? e : fail(`the package is damaged (${name}: ${(e as Error).message})`)
}
- if (n !== size) throw new Error(`the package is damaged (${name}: ${n} bytes, ${size} expected)`)
+ if (n !== size) throw fail(`the package is damaged (${name}: ${n} bytes, ${size} expected)`)
return new Blob(parts as BlobPart[], { type })
}
diff --git a/test/bodies.html b/test/bodies.html
index d2f4fd7..b37a9dc 100644
--- a/test/bodies.html
+++ b/test/bodies.html
@@ -14,6 +14,7 @@
// Run by smoke.mjs: a PNGTuber shows the picture for each mouth / eye state, and follows the face tracker (its
// synthetic trace) and a speech clip; a VRM (tiny-vrm.mjs, served by smoke.mjs) loads, draws, takes its expressions
// and follows the face tracker.
+ import { Player } from '../dist/index.js'
import { BodyPlayer } from '../dist/body.js'
import { PngBody } from '../dist/png.js'
import { VrmBody } from '../dist/vrm.js'
@@ -30,6 +31,8 @@
const covered = (canvas) => { const d = read(canvas).data; let n = 0; for (let i = 3; i < d.length; i += 4) if (d[i] > 0) n++; return n / (d.length / 4) }
const solid = (color) => { const c = new OffscreenCanvas(100, 120), x = c.getContext('2d'); x.fillStyle = color; x.fillRect(0, 0, 100, 120); return c.convertToBlob() }
const wait = (ms) => new Promise((r) => setTimeout(r, ms))
+ /** until the condition holds, or some seconds: what a browser does next may take a while on a busy machine */
+ const until = async (cond, ms = 5000) => { for (const t0 = performance.now(); !cond() && performance.now() - t0 < ms;) await wait(10) }
/** what the drivers did, seen by a driver that runs after them */
const watch = (player) => {
const seen = { tracked: false, mouth: 0, angle: 0 }
@@ -46,6 +49,197 @@
return new Blob([b.buffer], { type: 'audio/wav' })
}
+ /** what Player and BodyPlayer promise alike: one test, run on both (see PlayerBase) */
+ window.runContract = async (kind, id) => {
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 200px; height: 240px'
+ document.body.append(c)
+ const options = { autoplay: false, followPointer: false, idle: false, maxPixelRatio: 2 }
+ const p = kind === 'player' ? new Player(c, options) : new BodyPlayer(c, new PngBody(), options)
+ const good = kind === 'player' ? `../characters/${id}.livliv` : new Map([['closed.png', await solid('#f00')], ['open.png', await solid('#0f0')]])
+ const bad = kind === 'player' ? new Map([['model.json', new Blob(['{'])]]) : new Map([['closed.png', await solid('#f00')]])
+ const out = { before: p.model, playing: p.playing }
+ const info = await p.load(good)
+ out.name = info.name === p.model?.name
+ out.expressions = p.expressions === info.expressions
+ // the pixel ratio can be read and changed
+ out.ratio = [p.maxPixelRatio]
+ p.maxPixelRatio = 0.5; p.resize(); out.ratio.push(p.maxPixelRatio, c.width)
+ p.maxPixelRatio = 2; p.resize(); out.ratio.push(c.width)
+ // a snapshot is the pose as it is now, on a paused player too
+ const a = p.snapshot()
+ p.params.mouthOpen = 1
+ const b = p.snapshot()
+ out.snapshot = a !== b && b.startsWith('data:image/png')
+ p.params.mouthOpen = 0
+ // what a driver gets; the time counts the frames before this one
+ const times = []
+ let frame = null
+ const remove = p.use({ update(f) { times.push(f.time); frame = f } })
+ p.draw(0.01); p.draw(0.02); p.draw(0.03)
+ out.times = [times[1] - times[0], times[2] - times[1]]
+ out.frame = !!frame.params && typeof frame.add === 'function' && frame.dt === 0.03 && 'x' in frame.move
+ remove()
+ p.draw(0.01)
+ out.removed = times.length === 3
+ // the pose stays in range, and what is no number is the default, whoever set it
+ const pose = () => p.frame.params
+ const P = p.params
+ P.angleX = 1e9; P.eyeLOpen = -5; p.draw(0.01); out.clamped = [pose().angleX, pose().eyeLOpen]
+ P.angleX = NaN; P.eyeLOpen = NaN; P.bodyX = 'x'; p.draw(0.01); out.nan = [pose().angleX, pose().eyeLOpen, pose().bodyX]
+ const undo = p.use({ order: 100, update(f) { f.params.angleY = NaN; f.params.mouthOpen = 7 } })
+ p.draw(0.01); out.driver = [pose().angleY, pose().mouthOpen]
+ undo()
+ Object.assign(P, { angleX: 0, eyeLOpen: 1, bodyX: 0 })
+ p.draw(0.01)
+ out.alive = kind === 'player' ? !p.puppet.layers.some((L) => Number.isNaN(L.out[0])) : true
+ // away is a switch, on both
+ p.setAway(true)
+ for (let i = 0; i < 90; i++) p.draw(1 / 30)
+ out.away = pose().eyeLOpen
+ p.setAway(false)
+ for (let i = 0; i < 90; i++) p.draw(1 / 30)
+ out.back = pose().eyeLOpen
+ out.oldAway = (() => { try { p.act('away'); return 'took it' } catch (e) { return e.message } })()
+ // the rest of the surface
+ out.unknown = p.setExpression('no-such-expression')
+ out.neutral = p.setExpression(null)
+ for (const a2 of ['nod', 'shake', 'tilt', 'hop', 'shy']) p.act(a2)
+ p.lookAt(0.5, 0.5); p.lookAt(null); p.blink(); p.draw(0.01)
+ p.play(); out.plays = p.playing; p.pause(); out.paused = p.playing
+ // Player alone: from the canvas to the model, and a flick
+ if (kind === 'player') {
+ const [mx, my] = p.canvasToModel(c.clientWidth / 2, c.clientHeight / 2), [mw, mh] = p.model.size
+ out.toModel = mx > 0 && my > 0 && mx < mw && my < mh
+ const R = p.puppet.chainRuns[0]?.R
+ out.poke = [p.poke(mx, my), R ? p.poke(R[R.length - 2], R[R.length - 1]) : -1]
+ }
+ // a load that fails keeps the character; a disposed player says so and takes nothing more
+ const name = p.model.name
+ out.failed = await p.load(bad).then(() => 'loaded', () => 'refused')
+ out.kept = p.model?.name === name
+ p.dispose(); p.dispose()
+ out.after = p.model
+ out.disposedLoad = await p.load(good).then(() => 'loaded', (e) => e.message)
+ let gone = 0
+ p.use({ update() {}, dispose() { gone++ } })
+ out.gone = gone
+ c.remove()
+ return out
+ }
+
+ /** BodyPlayer: several loads at once. Those still downloading or waiting stop; one already going in (inside the Body,
+ * which cannot be stopped) finishes, and the newer one follows */
+ window.runBodyLoads = async () => {
+ const out = {}
+ const slow = (color) => ({ closed: `/__slow-png/${color}.png`, open: `/__slow-png/${color}.png` })
+ {
+ const c = document.createElement('canvas'), p = new BodyPlayer(c, new PngBody(), { autoplay: false, followPointer: false, idle: false })
+ c.style.cssText = 'width: 40px; height: 40px'
+ document.body.append(c)
+ const say = (e) => e.name
+ const a = p.load(slow('ff0000')).then(() => 'loaded', say)
+ await wait(60)
+ const b = p.load(slow('00ff00')).then(() => 'loaded', say)
+ await wait(60)
+ const steps = []
+ const last = await p.load(slow('0000ff'), { onProgress: (s) => steps.push(s) }).then(() => 'loaded', say)
+ out.row = { a: await a, b: await b, last, pixel: pixel(c, 20, 20), steps }
+ p.dispose(); c.remove()
+ }
+ {
+ // a Body holding its load: the page decides when it ends
+ const held = [], calls = []
+ const body = {
+ format: 'test',
+ load(files) { calls.push([...files.keys()][0]); return new Promise((res) => held.push(() => res({ name: [...files.keys()][0], expressions: [] }))) },
+ render() {}, head: () => ({ x: 0.5, y: 0.3, rx: 0.1, ry: 0.1 }), dispose() {},
+ }
+ const c = document.createElement('canvas'), p = new BodyPlayer(c, body, { autoplay: false, followPointer: false })
+ const file = (n) => new Map([[n, new Blob(['x'])]])
+ const a = p.load(file('a')).then((m) => m.name, (e) => e.name)
+ await wait(20)
+ const b = p.load(file('b')).then((m) => m.name, (e) => e.name)
+ const d = p.load(file('d')).then((m) => m.name, (e) => e.name)
+ await wait(20)
+ held.shift()()
+ out.held = { a: await a, b: await b }
+ await wait(20)
+ held.shift()()
+ out.held.d = await d
+ out.held.calls = calls
+ out.held.shown = p.model?.name
+ p.dispose()
+ }
+ {
+ // a Body that takes its 2D context only at its first frame: asking about the context must not start a WebGL one
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 40px; height: 40px'
+ document.body.append(c)
+ let got = null
+ const body = {
+ format: 'late', async load() { return { name: 'late', expressions: [] } },
+ render(cv) { got = cv.getContext('2d') }, head: () => ({ x: 0.5, y: 0.3, rx: 0.1, ry: 0.1 }), dispose() {},
+ }
+ const p = new BodyPlayer(c, body, { autoplay: false, followPointer: false })
+ out.lateLost = p.contextLost
+ const going = p.load(new Map([['a', new Blob(['x'])]]))
+ out.lateLost2 = p.contextLost
+ await going
+ out.late2d = !!got
+ p.dispose({ release: true })
+ await wait(20)
+ out.lateAgain = (() => { try { new BodyPlayer(c, body, { autoplay: false }).dispose(); return 'made' } catch (e) { return e.code } })()
+ c.remove()
+ }
+ return out
+ }
+
+ /** the context on a BodyPlayer: a VRM is told and released like a Player; a PNGTuber has none to lose */
+ window.runBodyContext = async () => {
+ const out = {}
+ const code = (e) => e.code ?? e.name
+ {
+ const c = document.createElement('canvas'), p = new BodyPlayer(c, new VrmBody(), { followPointer: false })
+ c.style.cssText = 'width: 60px; height: 80px'
+ document.body.append(c)
+ let told = 0
+ p.onContextLost = () => told++
+ await p.load('/__tiny.vrm')
+ const gl = c.getContext('webgl2')
+ gl.getExtension('WEBGL_lose_context').loseContext()
+ await until(() => told)
+ out.vrm = { told, lost: p.contextLost, load: await p.load('/__tiny.vrm').then(() => 'loaded', code) }
+ p.dispose(); c.remove()
+ }
+ {
+ const c = document.createElement('canvas'), p = new BodyPlayer(c, new VrmBody(), { followPointer: false })
+ c.style.cssText = 'width: 60px; height: 80px'
+ document.body.append(c)
+ await p.load('/__tiny.vrm')
+ const gl = c.getContext('webgl2')
+ p.dispose({ release: true })
+ await until(() => gl.isContextLost())
+ out.vrmReleased = { lost: gl.isContextLost(), reuse: (() => { try { new BodyPlayer(c, new VrmBody()); return 'made' } catch (e) { return code(e) } })() }
+ c.remove()
+ }
+ {
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 60px; height: 80px'
+ document.body.append(c)
+ const files = new Map([['closed.png', await solid('#f00')], ['open.png', await solid('#0f0')]])
+ const p = new BodyPlayer(c, new PngBody(), { autoplay: false, followPointer: false, idle: false })
+ await p.load(files)
+ out.png = { lost: p.contextLost }
+ p.dispose({ release: true })
+ await wait(30)
+ const q = new BodyPlayer(c, new PngBody(), { autoplay: false, followPointer: false, idle: false })
+ out.png.again = await q.load(files).then((m) => m.name, code)
+ q.dispose(); c.remove()
+ }
+ return out
+ }
+
/** the camera refused while the face model comes in (the slow route of smoke.mjs); ahead: it was asked for before */
window.runRefused = async (ahead = false) => {
const files = { wasmUrl: '/node_modules/@mediapipe/tasks-vision/wasm', modelUrl: '/__model.task' }
diff --git a/test/smoke.html b/test/smoke.html
index be2ffef..f8e532f 100644
--- a/test/smoke.html
+++ b/test/smoke.html
@@ -42,6 +42,7 @@
}
let FILES = {}
+ window.setFiles = (files) => { FILES = files }
/** a model as files (path -> Blob) under a folder, as a folder picked in the browser gives them: its model.json as
* given, everything else as the folder has it */
async function filesOf(id, model, swap = {}) {
@@ -122,6 +123,20 @@
const vrmHere = await player.load(new File([await (await fetch('/__tiny.vrm')).blob()], 'x.vrm')).then(() => 'loaded', (e) => e.message)
const vrmFolder = await player.load(new Map([['a/x.vrm', new Blob(['x'])]])).then(() => 'loaded', (e) => e.message)
+ // what a load rejects with says what went wrong, in code
+ const codeOf = (source) => player.load(source).then(() => 'loaded', (e) => (e.name === 'LivLivError' ? e.code : `not a LivLivError: ${e.name} ${e.message}`))
+ const codes = {
+ 'http': await codeOf('/__nothing-here.livliv'), 'not-a-model': await codeOf(new File([new Blob(['just text'])], 'x.livliv')),
+ 'not-a-model json': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ format: 'something else' })])]])),
+ 'not-a-model vrm': await codeOf(new File([await (await fetch('/__tiny.vrm')).blob()], 'x.vrm')),
+ 'newer-format': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, version: FORMAT_VERSION + 1 })])]])),
+ 'missing-feature': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, requires: ['from-the-future'] })])]])),
+ 'missing-file': await codeOf(new Map([['model.json', new Blob([JSON.stringify(model)])]])),
+ 'missing-file none': await codeOf(new Map([['a.txt', new Blob(['x'])]])),
+ 'invalid': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, layers: [] })])]])),
+ 'too-large': await codeOf(new Map([['model.json', new Blob([JSON.stringify({ ...model, canvas: [100000, 100] })])]])),
+ }
+
// a newer format: refused, and the model before stays
let newer
try {
@@ -146,7 +161,7 @@
} catch (e) {
legacy = { error: String(e.message || e) }
}
- return { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder }
+ return { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder, codes }
}
/** disposing while something runs: a load rejects and leaves nothing on the GPU or the canvas; a driver added to a
@@ -165,7 +180,7 @@
// disposed just as the images start coming in (a timer could lose the race to a quick load)
const loadModel = p.puppet.loadModel.bind(p.puppet)
p.puppet.loadModel = (...args) => { p.dispose(); return loadModel(...args) }
- out.load = await p.load(`../characters/${id}/`).then(() => 'resolved', (e) => e.message)
+ out.load = await p.load(`../characters/${id}/`).then(() => 'resolved', (e) => `${e.name}: ${e.message}`)
p.resize()
await new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)))
const px = new Uint8Array(4)
@@ -210,6 +225,273 @@
return out
}
+ /** a model that breaks while it is built, past the checks and the download: wherever it breaks, the character before
+ * stays as it was (every pixel, every field, what is on the GPU) and the next good model plays. k: the character
+ * standing there, in motion so that a spring put back wrong would show; b: the one that breaks, with chains and keyforms */
+ window.runKeep = async (k, b) => {
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 200px; height: 240px'
+ document.body.append(c)
+ const p = new Player(c, { autoplay: false, followPointer: false, idle: false }), pup = p.puppet, gl = pup.gl
+ const live = { texture: 0, buffer: 0, vertexArray: 0 }
+ for (const [kind, make, free] of [['texture', 'createTexture', 'deleteTexture'], ['buffer', 'createBuffer', 'deleteBuffer'], ['vertexArray', 'createVertexArray', 'deleteVertexArray']]) {
+ const m = gl[make], f = gl[free]
+ gl[make] = function () { live[kind]++; return m.call(gl) }
+ gl[free] = function (t) { if (t) live[kind]--; return f.call(gl, t) }
+ }
+ const kept = await filesOf(k, bare(await json(k))), broken = await filesOf(b, bare(await json(b)))
+ const grab = () => {
+ const o = document.createElement('canvas'); o.width = c.width; o.height = c.height
+ const x = o.getContext('2d', { willReadFrequently: true })
+ x.drawImage(c, 0, 0)
+ return x.getImageData(0, 0, o.width, o.height).data
+ }
+ const fields = () => JSON.stringify({ name: pup.model?.name, layers: pup.layers.length, chains: pup.chains, drive: pup.drive, body: pup.body, bust: pup.bust, lastHead: pup.lastHead,
+ smile: pup.smileShown, mouth: [pup.mouthFrame, pup.mouthW, pup.mouthHeld, pup.talkLeft, pup.frameKind, pup.frameJaw], keys: [pup.keys, pup.kfOff], rig: pup.rig, tuning: pup.tuning,
+ time: pup.time, runs: pup.chainRuns.map((C) => [...C.p]), params: pup.params, player: p.model?.name })
+ const stand = async () => {
+ await p.load(kept)
+ Object.assign(p.params, { angleX: 25, mouthOpen: 0.8, smile: 1 })
+ for (let i = 0; i < 25; i++) p.draw(1 / 60)
+ p.draw(0)
+ }
+ /** each arms one failure and returns how to take it away, which says whether the load got there */
+ const places = {
+ // nothing breaks: what the comparison below does with no failure at all (the good model simply plays)
+ none: null,
+ // the chains, the first thing built
+ chain() { const f = Array.prototype.flat; let n = 0; Array.prototype.flat = function () { n++; throw new Error('injected') }; return () => { Array.prototype.flat = f; return n > 0 } },
+ // a layer, part of the way through the list: those before it are on the GPU
+ layer() { const f = pup.buildLayer; let n = 0; pup.buildLayer = function (...a) { if (++n === 4) throw new Error('injected'); return f.apply(this, a) }; return () => { delete pup.buildLayer; return n >= 4 } },
+ // an upload, inside a layer: its mesh and some of its textures are on the GPU
+ upload() { const f = gl.texImage2D; let n = 0; gl.texImage2D = function (...a) { if (++n === 3) throw new Error('injected'); return f.apply(gl, a) }; return () => { delete gl.texImage2D; return n >= 3 } },
+ // the springs: the layers are done and most fields are already the new model's
+ physics() { let n = 0; pup.resetPhysics = function () { n++; throw new Error('injected') }; return () => { delete pup.resetPhysics; return n > 0 } },
+ // the keyforms, last and heaviest, in the middle of swapping the pose for its own
+ keyforms() { const f = pup.prepareKeyforms; let n = 0; pup.prepareKeyforms = function (K) { n++; return f.call(this, { ...K, keys: [K.keys[0], null] }) }; return () => { delete pup.prepareKeyforms; return n > 0 } },
+ // the GPU out of memory: nothing throws, it says so afterwards
+ memory() { let n = 0; gl.getError = function () { return n++ ? gl.OUT_OF_MEMORY : gl.NO_ERROR }; return () => { delete gl.getError; return n > 1 } },
+ }
+ const out = {}
+ for (const [name, arm] of Object.entries(places)) {
+ await stand()
+ const px = grab(), state = fields(), gpu = JSON.stringify(live)
+ if (!arm) { p.draw(0); out[name] = { differ: differs(px, grab()) }; continue }
+ const done = arm()
+ const said = await p.load(broken).then(() => null, (e) => String(e.message || e))
+ const reached = done(), after = fields(), gpuAfter = JSON.stringify(live)
+ p.draw(0)
+ const differ = differs(px, grab())
+ const next = await p.load(broken).then((m) => m.name, (e) => `refused: ${e.message}`)
+ p.draw(1 / 60)
+ out[name] = { said, reached, differ, sameFields: state === after, sameGpu: gpu === gpuAfter, next, glError: gl.getError() }
+ }
+ p.dispose()
+ out.left = { ...live }
+ c.remove()
+ return out
+ }
+ const differs = (a, b) => { let n = 0; for (let i = 0; i < a.length; i++) if (a[i] !== b[i]) n++; return n }
+
+ const wait = (ms) => new Promise((r) => setTimeout(r, ms))
+ /** until the condition holds, or some seconds: what a browser does next may take a while on a busy machine */
+ const until = async (cond, ms = 5000) => { for (const t0 = performance.now(); !cond() && performance.now() - t0 < ms;) await wait(10) }
+ const nameOf = (m) => m.name, errName = (e) => e.name
+
+ /** several loads at once, and calling a load off: the last one plays, the others stop and say so. Slow: a route of
+ * smoke.mjs that takes about a second */
+ window.runLoads = async ([a, b, d]) => {
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 160px; height: 200px'
+ document.body.append(c)
+ const p = new Player(c, { autoplay: false, followPointer: false }), out = {}
+ const slow = (id) => `/__slow/${id}.livliv`
+ // three in a row: only the last goes in
+ let went = 0
+ const install = p.puppet.install
+ p.puppet.install = function (...args) { went++; return install.apply(this, args) }
+ const first = p.load(slow(a)).then(nameOf, errName)
+ await wait(120)
+ const second = p.load(slow(b)).then(nameOf, errName)
+ await wait(120)
+ const third = await p.load(`../characters/${d}.livliv`).then(nameOf, errName)
+ out.row = { first: await first, second: await second, third, shown: p.model?.name, went }
+ delete p.puppet.install
+ // nobody catching: a replaced load reports nothing, a real error is reported as ever
+ const unhandled = []
+ const note = (e) => { unhandled.push(e.reason?.code ?? e.reason?.name); e.preventDefault() }
+ addEventListener('unhandledrejection', note)
+ p.load(slow(a))
+ await p.load(`../characters/${b}.livliv`)
+ await wait(80)
+ out.quiet = unhandled.length
+ p.load('/__nothing-here.livliv')
+ await until(() => unhandled.length)
+ await wait(50)
+ out.loud = [...unhandled]
+ removeEventListener('unhandledrejection', note)
+ // a signal already aborted: refused at once, and it leaves the running load alone
+ const running = p.load(`../characters/${a}.livliv`).then(nameOf, errName)
+ const off = new AbortController()
+ off.abort()
+ out.already = { said: await p.load(`../characters/${d}.livliv`, { signal: off.signal }).then(nameOf, errName), running: await running }
+ // one's own signal: the load stops, the character stays, the reason is its own
+ const before = p.model.name
+ const own = new AbortController()
+ const mine = p.load(slow(b), { signal: own.signal }).then(nameOf, errName)
+ await wait(120)
+ own.abort()
+ out.own = { said: await mine, kept: p.model.name === before }
+ const why = new AbortController()
+ const reason = p.load(slow(b), { signal: why.signal }).then(nameOf, (e) => e.message)
+ await wait(50)
+ why.abort(new Error('my reason'))
+ out.reason = await reason
+ // progress, for each way a model comes
+ const steps = async (source) => { const seen = []; await p.load(source, { onProgress: (s) => seen.push(s) }); return seen }
+ out.progress = { package: await steps(slow(a)), unknown: await steps(`/__slow-nosize/${a}.livliv`), folder: await steps(`../characters/${a}/`), files: await steps(await filesOf(a, await json(a))) }
+ // a callback that throws does not break the load
+ const log = console.error
+ console.error = () => {}
+ out.badCallback = await p.load(`../characters/${a}.livliv`, { onProgress() { throw new Error('mine') } }).then(nameOf, (e) => e.message)
+ console.error = log
+ // disposed while it runs
+ const going = p.load(slow(b)).then(nameOf, (e) => [e.name, e.message])
+ await wait(80)
+ p.dispose()
+ out.disposed = await going
+ out.after = await p.load(slow(b)).then(nameOf, errName)
+ c.remove()
+ return out
+ }
+
+ /** the WebGL context lost, and given up */
+ window.runContext = async (id, big = true) => {
+ const out = {}
+ const box = () => { const c = document.createElement('canvas'); c.style.cssText = 'width: 60px; height: 80px'; document.body.append(c); return c }
+ const pk = `../characters/${id}.livliv`
+ const lose = (p) => p.puppet.gl.getExtension('WEBGL_lose_context').loseContext()
+ const code = (e) => e.code ?? e.name
+ {
+ // lost: the page is told once, nothing draws or throws, a load says so at once, and one running is called off
+ const c = box(), p = new Player(c, { followPointer: false })
+ await p.load(pk)
+ let told = 0
+ p.onContextLost = () => told++
+ out.before = p.contextLost
+ const going = p.load(`/__slow/${id}.livliv`).then(() => 'loaded', code)
+ await wait(60)
+ lose(p)
+ await until(() => told)
+ out.running = await going
+ out.told = told
+ out.lost = p.contextLost
+ out.load = await p.load(pk).then(() => 'loaded', code)
+ p.play(); await wait(120) // frames on a lost context
+ p.resize(); out.snapshot = typeof p.snapshot()
+ p.dispose()
+ out.newOnLost = (() => { try { new Player(c); return 'made' } catch (e) { return code(e) } })()
+ c.remove()
+ }
+ {
+ // released: gone one task later, and the canvas says so; the page is not told of its own doing
+ const c = box(), p = new Player(c, { followPointer: false }), gl = p.puppet.gl
+ await p.load(pk)
+ let told = 0
+ p.onContextLost = () => told++
+ p.dispose({ release: true })
+ out.sameTask = gl.isContextLost()
+ await until(() => gl.isContextLost())
+ await wait(30)
+ out.nextTask = gl.isContextLost()
+ out.releasedTold = told
+ out.reuse = (() => { try { new Player(c); return 'made' } catch (e) { return code(e) } })()
+ c.remove()
+ }
+ {
+ // a framework's unmount and mount of one canvas, in one task: the new player keeps the context
+ const c = box()
+ const a = new Player(c, { followPointer: false })
+ await a.load(pk)
+ let told = 0
+ a.onContextLost = () => told++
+ a.dispose({ release: true })
+ const b = new Player(c, { followPointer: false })
+ await wait(30)
+ out.remount = { lost: b.contextLost, loaded: await b.load(pk).then((m) => !!m.name, code), told }
+ b.resize()
+ out.remount.share = (() => { const x = document.createElement('canvas'); x.width = c.width; x.height = c.height; const g2 = x.getContext('2d', { willReadFrequently: true }); g2.drawImage(c, 0, 0); const d = g2.getImageData(0, 0, x.width, x.height).data; let n = 0; for (let i = 3; i < d.length; i += 4) if (d[i]) n++; return n / (d.length / 4) })()
+ b.dispose({ release: true })
+ c.remove()
+ }
+ if (big) {
+ // characters coming and going beside one that stays: with release it stays, without it is pushed out
+ const stay = box(), keep = new Player(stay, { followPointer: false, autoplay: false })
+ await keep.load(pk)
+ let told = 0
+ keep.onContextLost = () => told++
+ for (let i = 0; i < 40; i++) {
+ const c = box(), p = new Player(c, { followPointer: false, autoplay: false })
+ p.dispose({ release: true })
+ c.remove()
+ await wait(0)
+ }
+ await wait(50)
+ out.withRelease = { lost: keep.contextLost, told }
+ const others = []
+ for (let i = 0; i < 20; i++) { const c = box(); others.push(new Player(c, { followPointer: false, autoplay: false })); await wait(0) }
+ await until(() => told, 2000)
+ out.withoutRelease = { lost: keep.contextLost, told }
+ out.overLimitLoad = await keep.load(pk).then(() => 'loaded', code)
+ keep.dispose()
+ for (const p of others) { p.dispose({ release: true }); p.canvas.remove() }
+ stay.remove()
+ await wait(30)
+ }
+ return out
+ }
+
+ /** the poster: on the canvas until the first frame, the canvas's own background back after */
+ window.runPoster = async (id) => {
+ const c = document.createElement('canvas')
+ c.style.cssText = 'width: 160px; height: 200px; background-color: rgb(1, 2, 3)'
+ document.body.append(c)
+ const look = () => { const s = getComputedStyle(c); return { image: s.backgroundImage, size: s.backgroundSize, repeat: s.backgroundRepeat, color: s.backgroundColor } }
+ const poster = `../characters/${id}/cover.webp`, out = {}
+ const clean = look()
+ let p = new Player(c, { autoplay: false, followPointer: false })
+ const going = p.load(`/__slow/${id}.livliv`, { poster })
+ out.during = look()
+ await going
+ out.after = look()
+ out.same = JSON.stringify(out.after) === JSON.stringify(clean)
+ // over a character: none, that one stays until the next is ready
+ const next = p.load(`/__slow/${id}.livliv`, { poster })
+ out.over = look().image
+ await next
+ p.dispose()
+ // a load that fails takes it away, so that the page can say why
+ p = new Player(c, { autoplay: false, followPointer: false })
+ const refused = p.load('/__nothing-here.livliv', { poster }).catch(() => {})
+ out.failing = look().image
+ await refused
+ out.failed = look().image
+ // replaced by a load without one: gone at once; with one: that one
+ const a = p.load(`/__slow/${id}.livliv`, { poster }).catch(() => {})
+ const b = p.load(`/__slow/${id}.livliv`)
+ out.replaced = look().image
+ await a; await b
+ p.dispose()
+ // disposed while it shows
+ p = new Player(c, { autoplay: false, followPointer: false })
+ const d = p.load(`/__slow/${id}.livliv`, { poster }).catch(() => {})
+ p.dispose()
+ await d
+ out.disposed = look().image
+ c.remove()
+ return out
+ }
+
/** random changes to a model.json, in the real player: whatever happens, the player goes on working. Refused: the model
* before is still drawn. Loaded: the frames do not throw. And a good model after any of them draws (nothing is poisoned) */
window.runFuzz = async (id, keep, from, to) => {
@@ -272,8 +554,9 @@
* gets our words, not the browser's; a 404 is not tried again */
window.runDropped = async (id) => {
const p = new Player(canvas, { autoplay: false, followPointer: false })
- const tell = (route) => p.load(route).then((m) => `played ${m.name}`, (e) => e.message)
- const out = { once: await tell(`/__dropped/once/${id}.livliv`), always: await tell(`/__dropped/always/${id}.livliv`), missing: await tell('/__dropped/missing/x.livliv') }
+ const tell = (route) => p.load(route).then((m) => `played ${m.name}`, (e) => { said.push([e.code, e.status]); return e.message })
+ const said = []
+ const out = { said, once: await tell(`/__dropped/once/${id}.livliv`), always: await tell(`/__dropped/always/${id}.livliv`), missing: await tell('/__dropped/missing/x.livliv') }
p.dispose()
return out
}
@@ -298,7 +581,7 @@
}
const out = {}
for (const [k, files] of Object.entries(cases)) {
- try { await player.load(await files); out[k] = { loaded: true } } catch (e) { out[k] = { refused: String(e.message || e), kept: player.model?.name } }
+ try { await player.load(await files); out[k] = { loaded: true } } catch (e) { out[k] = { refused: String(e.message || e), code: e.code, kept: player.model?.name } }
}
// keyforms that are not right: the character still plays, on the rig alone
const kf = await json(kfId)
diff --git a/test/smoke.mjs b/test/smoke.mjs
index a93b47e..af526b4 100644
--- a/test/smoke.mjs
+++ b/test/smoke.mjs
@@ -4,11 +4,13 @@
// (bodies.html).
// npm run build && npm run test:browser
// Needs a Chromium for playwright-core once: npx playwright-core install chromium-headless-shell
+// With an engine's name (firefox, webkit: npm run test:engines), a small sample on that browser's engine: what the three
+// engines might do differently (loads replaced, the context lost and given up, the poster, the contract of the two classes).
import { createServer } from 'node:http'
import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'
import { extname, join, normalize } from 'node:path'
import { fileURLToPath } from 'node:url'
-import { chromium } from 'playwright-core'
+import { chromium, firefox, webkit } from 'playwright-core'
import { crc32, deflateSync } from 'node:zlib'
import { tinyVrm } from './tiny-vrm.mjs'
@@ -40,6 +42,30 @@ const bomb = bombPng()
const slowModel = { asked: 0, finished: 0 }
/** requests per route of runDropped */
const dropped = {}
+/** the slow routes (runLoads): how many were asked for, ran to the end, were closed before it */
+const slow = { asked: 0, finished: 0, cut: 0 }
+const slowly = (res, b, headers, ms = 40, parts = 25) => {
+ slow.asked++
+ res.writeHead(200, { 'content-type': 'application/octet-stream', ...headers })
+ let i = 0
+ const tick = setInterval(() => {
+ res.write(b.subarray(Math.floor((i * b.length) / parts), Math.floor(((i + 1) * b.length) / parts)))
+ if (++i === parts) { clearInterval(tick); slow.finished++; res.end() }
+ }, ms)
+ res.on('close', () => { if (i < parts) { slow.cut++; clearInterval(tick) } })
+}
+/** a PNG of one colour */
+function solidPng(hex, side = 8) {
+ const chunk = (type, data) => {
+ const b = Buffer.alloc(12 + data.length), t = Buffer.concat([Buffer.from(type), data])
+ b.writeUInt32BE(data.length, 0); t.copy(b, 4); b.writeUInt32BE(crc32(t), 8 + data.length)
+ return b
+ }
+ const ihdr = Buffer.alloc(13); ihdr.writeUInt32BE(side, 0); ihdr.writeUInt32BE(side, 4); ihdr[8] = 8; ihdr[9] = 2
+ const line = Buffer.concat([Buffer.from([0]), ...Array.from({ length: side }, () => Buffer.from(hex, 'hex'))])
+ return Buffer.concat([Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), chunk('IHDR', ihdr),
+ chunk('IDAT', deflateSync(Buffer.concat(Array.from({ length: side }, () => line)))), chunk('IEND', Buffer.alloc(0))])
+}
const server = createServer((req, res) => {
if (req.url.startsWith('/__model.task')) {
slowModel.asked++
@@ -53,6 +79,14 @@ const server = createServer((req, res) => {
res.on('close', () => clearInterval(tick))
return
}
+ const taking = req.url.match(/^\/__slow(-nosize)?\/(.+)\.livliv$/)
+ if (taking) {
+ const b = readFileSync(join(chars, `${taking[2]}.livliv`))
+ slowly(res, b, taking[1] ? {} : { 'content-length': b.length })
+ return
+ }
+ const picture = req.url.match(/^\/__slow-png\/(\w+)\.png$/)
+ if (picture) { const b = solidPng(picture[1]); slowly(res, b, { 'content-type': 'image/png', 'content-length': b.length }, 60, 5); return }
const drop = req.url.match(/^\/__dropped\/(once|always|missing)\/(.+)\.livliv$/)
if (drop) {
const [, how, id] = drop, n = (dropped[how] = (dropped[how] || 0) + 1)
@@ -77,6 +111,53 @@ const server = createServer((req, res) => {
})
await new Promise((r) => server.listen(0, '127.0.0.1', r))
+const engine = process.argv[2]
+if (engine) {
+ const browser = await ({ firefox, webkit }[engine]).launch(engine === 'firefox' ? { firefoxUserPrefs: { 'webgl.force-enabled': true, 'webgl.disabled': false } } : {})
+ let failedHere = 0
+ const check = (ok, what) => { if (!ok) failedHere++; console.log(`${ok ? 'ok ' : 'FAIL'} ${engine}: ${what}`) }
+ try {
+ const ctx = await browser.newContext({ permissions: [] }), page = await ctx.newPage()
+ const errors = []
+ page.on('pageerror', (e) => errors.push(e.message))
+ await page.goto(`http://127.0.0.1:${server.address().port}/test/smoke.html`)
+ await page.waitForFunction(() => typeof window.run === 'function')
+ const chosen = ids.slice(0, 3), nameIn = (id) => JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')).name
+ await page.evaluate((f) => window.setFiles(f), files)
+ const r = await page.evaluate((a) => window.runLoads(a), chosen)
+ check(r.row.first === 'AbortError' && r.row.second === 'AbortError' && r.row.third === nameIn(chosen[2]) && r.row.shown === nameIn(chosen[2]) && r.row.went === 1, `the last of three loads plays, once: ${JSON.stringify(r.row)}`)
+ // (WebKit may keep the connection of a download called off)
+ if (engine !== 'webkit') check(slow.cut >= 2, `the replaced downloads stop: ${slow.cut} cut`)
+ check(r.quiet === 0 && r.loud.join() === 'http', `nobody catching: replaced ${r.quiet}, a real error ${r.loud}`)
+ check(r.already.said === 'AbortError' && r.already.running === nameIn(chosen[0]), 'a signal already aborted leaves the running load alone')
+ check(r.own.said === 'AbortError' && r.own.kept && r.reason === 'my reason', "one's own signal")
+ const rising = (steps) => { const d = steps.filter((s) => s.step === 'download').map((s) => s.progress); return d.every((v, i) => !i || v > d[i - 1]) && d.at(-1) === 1 && steps.at(-1).step === 'build' }
+ check(rising(r.progress.package) && rising(r.progress.unknown) && rising(r.progress.folder), `progress rises to 1: ${r.progress.package.length} steps`)
+ check(r.disposed[0] === 'AbortError', 'disposed while it runs')
+ const q = await page.evaluate((id) => window.runPoster(id), chosen[0])
+ check(/cover\.webp/.test(q.during.image) && q.same && q.over === 'none' && q.failed === 'none' && q.replaced === 'none' && q.disposed === 'none', 'the poster, and the background back after')
+ // (only Chromium and Firefox get room back from a released context; Firefox has not been seen to drop one by count)
+ const x = await page.evaluate(([id, big]) => window.runContext(id, big), [chosen[0], engine === 'firefox'])
+ check(x.before === false && x.lost === true && x.told === 1 && x.running === 'context-lost' && x.load === 'context-lost' && x.snapshot === 'string', `the context lost: told ${x.told}, a load ${x.load}`)
+ check(x.nextTask === true && x.releasedTold === 0 && x.reuse === 'released', `released: gone next task ${x.nextTask}, the canvas says ${x.reuse}`)
+ check(x.remount.lost === false && x.remount.loaded === true && x.remount.share > 0.05, `unmount and mount in one task: ${JSON.stringify(x.remount)}`)
+ if (engine === 'firefox') check(x.withRelease.lost === false, '40 players came and went with release: the one that stays stays')
+ await page.goto(`http://127.0.0.1:${server.address().port}/test/bodies.html`)
+ await page.waitForFunction(() => typeof window.runBodies === 'function')
+ for (const kind of ['player', 'png']) {
+ const c = await page.evaluate(([k, id]) => window.runContract(k, id), [kind, chosen[0]])
+ check(c.snapshot && c.clamped.join() === '30,0' && c.nan.join() === '0,1,0' && c.failed === 'refused' && c.kept && c.after === null && c.away < 0.05 && c.back > 0.9, `the contract, ${kind}: ${c.name}`)
+ }
+ const b = await page.evaluate(() => window.runBodyLoads())
+ check(b.row.a === 'AbortError' && b.row.b === 'AbortError' && b.row.last === 'loaded' && b.held.d === 'd' && b.held.calls.join() === 'a,d', 'a BodyPlayer, several loads')
+ const v = await page.evaluate(() => window.runBodyContext())
+ check(v.vrm.told === 1 && v.vrm.load === 'context-lost' && v.vrmReleased.reuse === 'released' && !!v.png.again, 'a BodyPlayer, the context')
+ check(!errors.length, `no page errors${errors.length ? `: ${errors.join(' | ')}` : ''}`)
+ } finally { await browser.close(); server.close() }
+ console.log(failedHere ? `smoke ${engine}: ${failedHere} failed` : `smoke ${engine}: ok`)
+ process.exit(failedHere ? 1 : 0)
+}
+
const browser = await chromium.launch({ args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist', '--autoplay-policy=no-user-gesture-required',
'--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream'] })
let failed = 0
@@ -86,7 +167,10 @@ try {
page.on('pageerror', (e) => errors.push(e.message))
await page.goto(`http://127.0.0.1:${server.address().port}/test/smoke.html`)
await page.waitForFunction(() => typeof window.run === 'function')
- const { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder } = await page.evaluate((a) => window.run(a), { ids, files })
+ const { out, fromFiles, newer, legacy, packaged, vrmHere, vrmFolder, codes } = await page.evaluate((a) => window.run(a), { ids, files })
+ const codesOk = Object.entries(codes).every(([want, got]) => want.split(' ')[0] === got)
+ if (!codesOk) failed++
+ console.log(`${codesOk ? 'ok ' : 'FAIL'} what a load rejects with says what went wrong: ${Object.entries(codes).map(([k, v]) => (k.split(' ')[0] === v ? v : `${k}: ${v}`)).join(', ')}`)
const vrmHereOk = /BodyPlayer/.test(vrmHere) && /BodyPlayer/.test(vrmFolder)
if (!vrmHereOk) failed++
console.log(`${vrmHereOk ? 'ok ' : 'FAIL'} a VRM handed to a Player: ${vrmHere}`)
@@ -115,8 +199,9 @@ try {
const kfId = ids.find((id) => JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')).files?.keyforms) ?? ids[0]
const hostile = await page.evaluate(([a, b]) => window.runHostile(a, b), [ids[0], kfId])
const why = { bomb: /too large/, path: /not an image path|not a path/, canvas: /canvas/, box: /box/, json: /not a model\.json/, layers: /layers/, mesh: /vertices/, chain: /no id/, browLine: /brow line/, rig: /rig\.roll/ }
+ const code = { bomb: 'too-large', path: 'invalid', canvas: 'too-large', box: 'invalid', json: 'not-a-model', layers: 'too-large', mesh: 'too-large', chain: 'invalid', browLine: 'invalid', rig: 'invalid' }
for (const [k, re] of Object.entries(why)) {
- const r = hostile[k], ok = !r.loaded && re.test(r.refused) && !!r.kept
+ const r = hostile[k], ok = !r.loaded && re.test(r.refused) && !!r.kept && r.code === code[k]
if (!ok) failed++
console.log(`${ok ? 'ok ' : 'FAIL'} refuses ${k}: ${r.loaded ? 'it loaded' : r.refused.slice(0, 110)} (kept: ${r.kept})`)
}
@@ -124,6 +209,16 @@ try {
if (!kfOk) failed++
console.log(`${kfOk ? 'ok ' : 'FAIL'} broken keyforms: ${hostile.keyforms.refused ?? `${hostile.keyforms.name} plays on the rig alone`}`)
+ // a model that breaks while it is built (see runKeep): the character before stays, to the pixel
+ const has = (test) => ids.find((id) => test(JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')), id))
+ const rigged = has((m) => m.rig?.chains?.length && m.files?.keyforms) ?? kfId
+ const chained = has((m, id) => m.rig?.chains?.length && id !== rigged) ?? ids[0]
+ const keep = await page.evaluate(([a, b]) => window.runKeep(a, b), [chained, rigged])
+ const { none: noFail, left: keepLeft, ...failures } = keep
+ const keepOk = noFail.differ === 0 && Object.values(keepLeft).every((n) => n <= 0) && Object.values(failures).every((r) => r.said && r.reached && r.differ === 0 && r.sameFields && r.sameGpu && !/^refused/.test(r.next) && r.glError === 0)
+ if (!keepOk) failed++
+ console.log(`${keepOk ? 'ok ' : 'FAIL'} a model breaks while it is built (${rigged} over ${chained}): ${Object.entries(failures).map(([k, r]) => `${k}: ${r.reached ? '' : 'NOT REACHED '}${r.differ} values differ${r.sameFields ? '' : ', FIELDS changed'}${r.sameGpu ? '' : ', GPU changed'}${/^refused/.test(r.next) ? `, then ${r.next}` : ''}${r.glError ? `, GL error ${r.glError}` : ''}`).join('; ')}${keepOk ? '' : `; with no failure ${noFail.differ}; left ${JSON.stringify(keepLeft)}`}`)
+
// random changes to a model.json in the real player (see runFuzz): the seeds of a run are the same every time
const fuzzed = await page.evaluate(([a, b]) => window.runFuzz(a, b, 1, 150), [ids.find((i) => i === 'k02') ?? ids[0], ids[0]])
const fuzzOk = !fuzzed.wrong.length
@@ -132,13 +227,58 @@ try {
// a package whose connection drops (see runDropped)
const drops = await page.evaluate((id) => window.runDropped(id), ids[0])
const dropsOk = /^played/.test(drops.once) && dropped.once === 2 && /could not load/.test(drops.always) && !/Failed to fetch/.test(drops.always) && dropped.always === 2
- && /HTTP 404/.test(drops.missing) && dropped.missing === 1
+ && /HTTP 404/.test(drops.missing) && dropped.missing === 1 && JSON.stringify(drops.said) === JSON.stringify([['unreachable', null], ['http', 404]])
if (!dropsOk) failed++
console.log(`${dropsOk ? 'ok ' : 'FAIL'} a package's connection drops: once "${drops.once}" (${dropped.once} requests), always "${drops.always.slice(0, 60)}…" (${dropped.always}), a 404 "${drops.missing.slice(0, 50)}…" (${dropped.missing})`)
+ // several loads at once, called off, how far they have come (see runLoads)
+ {
+ const chosen = ids.slice(0, 3)
+ const r = await page.evaluate((a) => window.runLoads(a), chosen)
+ const nameIn = (id) => JSON.parse(readFileSync(join(chars, id, 'model.json'), 'utf8')).name
+ const rising = (steps) => {
+ const d = steps.filter((s) => s.step === 'download').map((s) => s.progress)
+ return d.every((v, i) => !i || v > d[i - 1]) && d.at(-1) === 1 && steps.at(-1).step === 'build' && steps.filter((s) => s.step === 'build').length === 1
+ }
+ const checks = {
+ 'only the last plays': r.row.first === 'AbortError' && r.row.second === 'AbortError' && r.row.third === nameIn(chosen[2]) && r.row.shown === nameIn(chosen[2]) && r.row.went === 1,
+ 'the replaced downloads stop': slow.cut >= 2,
+ 'nobody catching': r.quiet === 0 && r.loud.join() === 'http',
+ 'a signal already aborted': r.already.said === 'AbortError' && r.already.running === nameIn(chosen[0]),
+ "one's own signal": r.own.said === 'AbortError' && r.own.kept && r.reason === 'my reason',
+ 'progress': rising(r.progress.package) && r.progress.package.filter((s) => s.step === 'download').length > 5 && rising(r.progress.unknown) && rising(r.progress.folder) && r.progress.files.length === 1 && r.progress.files[0].step === 'build',
+ 'a callback that throws': r.badCallback === nameIn(chosen[0]),
+ 'dispose': r.disposed[0] === 'AbortError' && /disposed/.test(r.disposed[1]) && r.after === 'AbortError',
+ }
+ const wrong = Object.entries(checks).filter(([, ok]) => !ok).map(([k]) => k)
+ if (wrong.length) failed++
+ console.log(`${wrong.length ? 'FAIL' : 'ok '} several loads, called off: ${wrong.length ? `${wrong.join(', ')}: ${JSON.stringify(r).slice(0, 900)}` : `${slow.cut} downloads stopped of ${slow.asked}; ${r.progress.package.length} steps from a package`}`)
+ const q = await page.evaluate((id) => window.runPoster(id), chosen[0])
+ const has = (v) => /cover\.webp/.test(v), posterOk = has(q.during.image) && q.during.size === 'contain' && q.during.repeat === 'no-repeat' && q.during.color === 'rgb(1, 2, 3)' && q.same
+ && q.over === 'none' && has(q.failing) && q.failed === 'none' && q.replaced === 'none' && q.disposed === 'none'
+ if (!posterOk) failed++
+ console.log(`${posterOk ? 'ok ' : 'FAIL'} poster: shown until the first frame, the canvas's own background after${posterOk ? '' : `: ${JSON.stringify(q)}`}`)
+ }
+
+ // the context lost, and given up (see runContext)
+ {
+ const r = await page.evaluate((id) => window.runContext(id), ids[0])
+ const checks = {
+ lost: r.before === false && r.lost === true && r.told === 1,
+ 'a load running': r.running === 'context-lost', 'a load after': r.load === 'context-lost', 'frames, a snapshot': r.snapshot === 'string', 'a new player on it': r.newOnLost === 'context-lost',
+ release: r.sameTask === false && r.nextTask === true && r.releasedTold === 0 && r.reuse === 'released',
+ 'unmount and mount': r.remount.lost === false && r.remount.loaded === true && r.remount.told === 0 && r.remount.share > 0.05,
+ 'with release, the one that stays stays': r.withRelease.lost === false && r.withRelease.told === 0,
+ 'without, it is pushed out and told': r.withoutRelease.lost === true && r.withoutRelease.told === 1 && r.overLimitLoad === 'context-lost',
+ }
+ const wrong = Object.entries(checks).filter(([, ok]) => !ok).map(([k]) => k)
+ if (wrong.length) failed++
+ console.log(`${wrong.length ? 'FAIL' : 'ok '} the context lost and given up${wrong.length ? `: ${wrong.join(', ')}: ${JSON.stringify(r)}` : ': told once, frames idle, loads say so; 40 players came and went beside one that stayed'}`)
+ }
+
// disposing while something runs (see runLifecycle)
const life = await page.evaluate((id) => window.runLifecycle(id), ids[0])
- const lifeOk = /disposed/.test(life.load) && life.model === null && life.drawn === 0 && life.freed >= life.made && life.glError === 0
+ const lifeOk = /AbortError: .*disposed/.test(life.load) && life.model === null && life.drawn === 0 && life.freed >= life.made && life.glError === 0
&& life.driver === 1 && !!life.again && life.mic === 'ended' && life.supported === true
&& Object.values(life.left).every((n) => n <= 0)
if (!lifeOk) failed++
@@ -190,6 +330,41 @@ try {
if (!vaOk) failed++
console.log(`${vaOk ? 'ok ' : 'FAIL'} VRM, loaded again on a wide canvas: aspect ${vrmAgain?.aspect1}, ${vrmAgain?.aspect2}; a failed load kept ${vrmAgain?.kept}, covers ${(vrmAgain?.share * 100).toFixed(1)} %; on a 2D canvas: ${vrmOn2d}`)
if (errors.length) { failed++; console.log(`FAIL page errors:\n ${errors.join('\n ')}`) }
+ // a BodyPlayer's loads (see runBodyLoads)
+ {
+ const r = await page.evaluate(() => window.runBodyLoads())
+ const blue = r.row.pixel[2] > 200 && r.row.pixel[0] < 50
+ const stepsOk = r.row.steps.at(-1).step === 'build' && r.row.steps.filter((s) => s.step === 'download').at(-1).progress === 1
+ const ok = r.row.a === 'AbortError' && r.row.b === 'AbortError' && r.row.last === 'loaded' && blue && stepsOk
+ && r.held.a === 'a' && r.held.b === 'AbortError' && r.held.d === 'd' && r.held.calls.join() === 'a,d' && r.held.shown === 'd'
+ && r.lateLost === false && r.lateLost2 === false && r.late2d === true && r.lateAgain === 'made'
+ if (!ok) failed++
+ console.log(`${ok ? 'ok ' : 'FAIL'} BodyPlayer, several loads: ${ok ? 'the last plays; one already inside the Body finishes first' : JSON.stringify(r)}`)
+ }
+
+ // the context on a BodyPlayer (see runBodyContext)
+ {
+ const r = await page.evaluate(() => window.runBodyContext())
+ const ok = r.vrm.told === 1 && r.vrm.lost === true && r.vrm.load === 'context-lost' && r.vrmReleased.lost === true && r.vrmReleased.reuse === 'released'
+ && r.png.lost === false && !!r.png.again && r.png.again !== 'released'
+ if (!ok) failed++
+ console.log(`${ok ? 'ok ' : 'FAIL'} the context on a BodyPlayer${ok ? ': a VRM is told and released like a Player; a PNGTuber has none to lose' : `: ${JSON.stringify(r)}`}`)
+ }
+
+ // what both players promise alike (see runContract)
+ for (const kind of ['player', 'png']) {
+ const r = await page.evaluate(([k, id]) => window.runContract(k, id), [kind, has((m) => m.rig?.chains?.length) ?? ids[0]])
+ const near = (a, b) => Math.abs(a - b) < 1e-6
+ const problems = Object.entries({
+ 'before a load': r.before === null && r.playing === false, 'model': r.name && r.expressions, 'pixel ratio': r.ratio.join() === '2,0.5,100,200', snapshot: r.snapshot,
+ 'frame time': near(r.times[0], 0.01) && near(r.times[1], 0.02), frame: r.frame && r.removed, range: r.clamped.join() === '30,0', 'not a number': r.nan.join() === '0,1,0',
+ 'from a driver': r.driver.join() === '0,1', alive: r.alive, away: r.away < 0.05 && r.back > 0.9 && /setAway/.test(r.oldAway),
+ ...(kind === 'player' ? { 'canvas to model': r.toModel, poke: r.poke[1] > 0 } : {}), expressions2: r.unknown === false && r.neutral === true, 'play and pause': r.plays === true && r.paused === false,
+ 'failed load': r.failed === 'refused' && r.kept, disposed: r.after === null && /disposed/.test(r.disposedLoad) && r.gone === 1,
+ }).filter(([, ok]) => !ok).map(([k]) => k)
+ if (problems.length) failed++
+ console.log(`${problems.length ? 'FAIL' : 'ok '} contract, ${kind}: ${problems.length ? `${problems.join(', ')}: ${JSON.stringify(r)}` : 'the same on both'}`)
+ }
// a camera that is refused: the face model's download stops with it, unless someone asked for the model ahead of time
await page.goto(`http://127.0.0.1:${server.address().port}/test/bodies.html`)
await page.waitForFunction(() => typeof window.runBodies === 'function')
@@ -221,6 +396,41 @@ try {
console.log(`${waited && faded ? 'ok ' : 'FAIL'} example/${name}.html: cover first ${waited}, then the character ${faded}`)
await p.close()
}
+ // the example pages say so when the context is lost, on the first screen
+ {
+ const p = await browser.newPage()
+ await p.goto(`http://127.0.0.1:${server.address().port}/example/demo.html`)
+ const up = await p.waitForFunction(() => window.player?.model, null, { timeout: 20000 }).then(() => true, () => false)
+ await p.evaluate(() => window.player.canvas.getContext('webgl2').getExtension('WEBGL_lose_context').loseContext())
+ const said = up && await p.waitForFunction(() => { const r = document.getElementById('lost')?.getBoundingClientRect(); return r && r.top >= 0 && r.bottom <= innerHeight }, null, { timeout: 5000 }).then(() => true, () => false)
+ if (!said) failed++
+ console.log(`${said ? 'ok ' : 'FAIL'} example/demo.html: the context lost, the page says so`)
+ await p.close()
+ }
+ // React under StrictMode, which mounts everything twice (its files come from a CDN: without a connection, not tested)
+ {
+ const p = await browser.newPage(), said = []
+ p.on('pageerror', (e) => said.push(e.message))
+ await p.goto(`http://127.0.0.1:${server.address().port}/example/react.html`)
+ const up = await p.waitForFunction(() => document.getElementById('status')?.textContent && window.__react, null, { timeout: 20000 }).then(() => true, () => false)
+ if (!up) console.log('skip example/react.html: React did not come from the CDN')
+ else {
+ await p.waitForFunction(() => /·/.test(document.getElementById('status').textContent), null, { timeout: 20000 })
+ const first = await p.textContent('#status')
+ await p.click('#next')
+ await p.waitForFunction((before) => document.getElementById('status').textContent !== before, first, { timeout: 20000 })
+ const second = await p.textContent('#status'), kept = await p.evaluate(() => !document.querySelector('canvas').getContext('webgl2').isContextLost())
+ const gl = await p.evaluateHandle(() => document.querySelector('canvas').getContext('webgl2'))
+ await p.click('#toggle')
+ await p.waitForFunction(() => !document.querySelector('canvas'), null, { timeout: 5000 })
+ await new Promise((r) => setTimeout(r, 100))
+ const gone = await gl.evaluate((c) => c.isContextLost())
+ const ok = first !== second && kept && gone && !said.length && !/Error|LivLiv:/.test(second)
+ if (!ok) failed++
+ console.log(`${ok ? 'ok ' : 'FAIL'} example/react.html under StrictMode: "${first}", then "${second}"; the context kept while mounted ${kept}, given up on unmount ${gone}${said.length ? `; errors ${said.join(' | ')}` : ''}`)
+ }
+ await p.close()
+ }
// ?model=: never a request to another site, however it is written; the page's own files still open
for (const name of ['demo', 'vrm']) {
const p = await browser.newPage(), asked = []
diff --git a/test/types.mjs b/test/types.mjs
index 834d49a..76fd042 100644
--- a/test/types.mjs
+++ b/test/types.mjs
@@ -16,10 +16,10 @@ fs.rmSync(dir, { recursive: true, force: true })
fs.mkdirSync(path.join(dir, 'node_modules', path.dirname(name)), { recursive: true })
fs.symlinkSync(root, path.join(dir, 'node_modules', name), 'dir')
fs.writeFileSync(path.join(dir, 'use.ts'), `
-import { Player, VERSION, type Driver, type ModelInfo, type View } from '${name}'
+import { LivLivError, Player, VERSION, type ActionId, type Driver, type LivLivErrorCode, type LoadOptions, type LoadProgress, type ModelInfo, type View } from '${name}'
import { preloadFaceModel, startFaceTracking, type FaceTracking } from '${name}/tracking'
import { createSpeech, startLipSync, type LipSync } from '${name}/voice'
-import { BODY_LIMIT, BodyPlayer, type Body } from '${name}/body'
+import { BODY_LIMIT, BodyPlayer, LivLivError as FromBody, type Body } from '${name}/body'
import { PngBody } from '${name}/png'
import { VrmBody } from '${name}/vrm'
declare const canvas: HTMLCanvasElement
@@ -32,7 +32,26 @@ const sync: Promise<LipSync> = startLipSync(player)
const body: Body = new PngBody()
player.use(driver)
void [VERSION, model, view, tracking, sync, preloadFaceModel(), createSpeech(player), BODY_LIMIT, new BodyPlayer(canvas, body), new VrmBody()]
+const options: LoadOptions = { signal: new AbortController().signal, poster: '/cover.webp', onProgress: (p: LoadProgress) => void (p.step === 'download' ? p.progress : 0) }
+const again: Promise<ModelInfo> = player.load('/avatars/sena.livliv', options)
+const point: [number, number] = player.canvasToModel(1, 2)
+const moved: number = player.poke(point[0], point[1])
+const code: LivLivErrorCode | null = again instanceof LivLivError ? again.code : null
+const move: ActionId = 'nod'
+player.act(move)
+player.setAway(true)
+player.maxPixelRatio = 1
+void [moved, code, new FromBody('http', 'x', { status: 404 }).status]
// the types are real, not any: these must be errors
+// @ts-expect-error a switch now: setAway
+player.act('away')
+// @ts-expect-error not part of the API
+void player.puppet
+// @ts-expect-error nor are the insides
+void player.frame
+// @ts-expect-error the codes are a list
+const nope: LivLivErrorCode = 'nope'
+void nope
// @ts-expect-error not an action
player.act('nope')
// @ts-expect-error a name is text
diff --git a/tsconfig.build.json b/tsconfig.build.json
index cff411d..c5864b5 100644
--- a/tsconfig.build.json
+++ b/tsconfig.build.json
@@ -4,6 +4,7 @@
"noEmit": false,
"declaration": true,
"emitDeclarationOnly": true,
+ "stripInternal": true,
"declarationDir": "dist/types",
"rootDir": ".",
"tsBuildInfoFile": null