Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

docs(video): add bilingual Video guide, wire sidebar + WeChat cross-links

yhhc97166b190b7
diff --git a/docs/astro/astro.config.mjs b/docs/astro/astro.config.mjs
index 68ae81fb4..f1f4df7b3 100644
--- a/docs/astro/astro.config.mjs
+++ b/docs/astro/astro.config.mjs
@@ -86,6 +86,7 @@ export default defineConfig({
translations: { 'zh-CN': '图形与 UI' },
items: [
{ label: 'Sprites & Rendering', translations: { 'zh-CN': '精灵与渲染' }, slug: 'guides/sprites' },
+ { label: 'Video', translations: { 'zh-CN': '视频' }, slug: 'guides/video' },
{ label: 'Tilemaps', translations: { 'zh-CN': '瓦片地图' }, slug: 'guides/tilemap' },
{ label: 'UI', translations: { 'zh-CN': 'UI' }, slug: 'guides/ui' },
{ label: 'UI Components', translations: { 'zh-CN': 'UI 控件' }, slug: 'guides/ui-components' },
diff --git a/docs/astro/src/content/docs/guides/video.mdx b/docs/astro/src/content/docs/guides/video.mdx
new file mode 100644
index 000000000..3e372e9b2
--- /dev/null
+++ b/docs/astro/src/content/docs/guides/video.mdx
@@ -0,0 +1,167 @@
+---
+title: Video
+description: The complete guide to video — the declarative Video component, playing on Sprites, UI and meshes, the imperative VideoPlayer, sharing one stream across many surfaces, import settings, and the WeChat wasm decode path.
+---
+
+import { Aside } from '@astrojs/starlight/components';
+
+Estella plays video onto anything you can draw through the declarative `Video`
+component — *a Sprite whose texture is alive*. One decoded stream drives a live
+GPU texture that refreshes every frame. On web and desktop an `HTMLVideoElement`
+decodes (zero-copy on WebGL2); on WeChat an engine-owned MPEG-1 wasm decoder runs
+the **identical** texture pump on every device — phone, PC, and headless alike.
+
+## How it works
+
+Add a `Video` component to an entity that also has a renderable — a `Sprite`,
+`UIVisual`, or `Mesh2D` — and the video system decodes `source` into a texture and
+writes that texture handle onto the renderable every frame. The frame texture is
+**pathless** (never evicted) and stable, so a handle written once stays live for
+the life of the stream.
+
+Video decoding runs in **play mode only** — nothing decodes while you edit, so the
+editor stays responsive. The plugin is registered by default, like audio: there's
+no plugin to add.
+
+<Aside type="note">
+ Muted clips autoplay everywhere. Unmuted autoplay is blocked until a user
+ gesture (browser policy), so start sound-on playback from a tap or key. A
+ `Video` needs a `Sprite` / `UIVisual` / `Mesh2D` sibling to show on — on its own
+ it decodes but has nothing to draw to.
+</Aside>
+
+## The `Video` component
+
+In the editor, add a **Video** component to an entity with a **Sprite** and set its
+**Source**. In code, insert it alongside the renderable:
+
+```typescript
+import { defineSystem, Commands, Transform, Sprite, Video } from 'esengine';
+
+const spawnScreen = defineSystem([Commands()], (cmds) => {
+ cmds.spawn()
+ .insert(Transform, { position: { x: 0, y: 0, z: 0 } })
+ .insert(Sprite, { size: { x: 360, y: 360 } })
+ .insert(Video, { source: 'assets/clip.mp4', autoplay: true, loop: true, muted: true });
+});
+```
+
+| Property | Type | Default | Description |
+|---|---|---|---|
+| `source` | asset | `''` | The video clip to play. |
+| `autoplay` | boolean | `true` | Start as soon as the first frame is ready. |
+| `loop` | boolean | `true` | Restart from the beginning at the end. |
+| `muted` | boolean | `true` | Mute audio. Muted clips autoplay; unmuted needs a user gesture. |
+| `volume` | number | `1.0` | Audio volume, 0..1 (ignored while `muted`). |
+| `playbackRate` | number | `1.0` | Playback speed / pitch (`1` = normal). |
+| `fitSize` | boolean | `true` | On the first frame, resize the Sprite to the video's native pixels. |
+| `enabled` | boolean | `true` | Disable to stop and release the stream without removing the component. |
+
+The same component drives whichever renderable it finds, in order **`Sprite` →
+`UIVisual` → `Mesh2D`**: put a `Video` on a `UIVisual` for a video panel in your
+HUD, or on a `Mesh2D` to map a clip onto custom geometry.
+
+## One stream, many surfaces
+
+The frame texture is a plain handle, so any number of sprites can share it — read
+the playing sprite's `texture` and assign it to others. Each surface then picks a
+**region** of the frame with the Sprite's `uvOffset` / `uvScale` (the same
+convention atlas frames use, where `v = 0` is the image's bottom row). One decode
+feeds them all — no per-surface decoding, no extra API.
+
+```typescript
+// Mirror the live video texture onto every tile, then each samples a 1/N region.
+const shareVideoTexture = defineSystem(
+ [Query(Sprite, Video), Query(Mut(Sprite), PuzzlePiece)],
+ (preview, pieces) => {
+ let texture = 0;
+ for (const [, sprite] of preview) { texture = sprite.texture; break; }
+ if (!texture) return; // first frame not decoded yet
+ for (const [, sprite] of pieces) {
+ if (sprite.texture !== texture) sprite.texture = texture; // one write per piece
+ }
+ },
+);
+```
+
+This is exactly how the **video-puzzle** example turns a single clip into a board
+of live, swappable tiles.
+
+## Imperative playback — `VideoPlayer`
+
+For code-driven video with no component, the `VideoPlayer` resource plays a source
+and hands back a **stream handle** whose `textureHandle` you can put on any sprite
+or material:
+
+```typescript
+import { defineSystem, Res, VideoPlayer } from 'esengine';
+
+const playIntro = defineSystem([Res(VideoPlayer)], (video) => {
+ const stream = video.play('assets/intro.mp4', { loop: false, muted: true });
+ stream.onEnded = () => { /* advance past the intro */ };
+ // assign stream.textureHandle onto a Sprite/Mesh2D once stream.isReady
+});
+```
+
+| Member | Kind | Description |
+|---|---|---|
+| `play()` / `pause()` / `stop()` | method | Transport control. |
+| `seek(seconds)` | method | Jump to a time. |
+| `setVolume(v)` / `setMuted(m)` | method | Live audio adjust. |
+| `setLoop(l)` / `setPlaybackRate(r)` | method | Live loop / speed adjust. |
+| `textureHandle` | readonly | The live frame texture — `0` until the first frame, then pathless (never evicted). |
+| `width` / `height` | readonly | Native pixel size. |
+| `isReady` | readonly | The first frame has decoded. |
+| `isPlaying` | readonly | Currently playing. |
+| `currentTime` / `duration` | readonly | Seconds. |
+| `onReady` / `onEnded` / `onError` | callback | Lifecycle hooks. |
+
+`video.play(source, options)` takes the same options as the component
+(`autoplay`, `loop`, `muted`, `volume`, `playbackRate`); `video.stop(handle)` ends
+one stream and `video.stopAll()` ends every stream.
+
+## Import settings & cooked video
+
+Selecting a video asset shows a preview plus **Import Settings**. `Autoplay`,
+`Loop`, and `Muted` are the suggested states applied when the clip is dropped onto
+a `Video` component. Two **advanced** settings control the WeChat transcode:
+
+| Setting | Range | Default | Description |
+|---|---|---|---|
+| **Cook Quality** | `2`–`31` | `4` | MPEG-1 quantizer for the wasm decode path — lower is higher quality and a larger file. |
+| **Cook Audio Bitrate** | `96` / `128` / `192` kbps | `128` | AAC bitrate of the demuxed audio track. |
+
+On export the cook transcodes each shipped video into a codec-tagged **`.esv`**
+(MPEG-1 Program Stream) plus an **`.m4a`** audio-track sibling — the format the wasm
+decoder reads, with the audio track played through the audio pipeline and slaved as
+the video's clock. On web and desktop the original file plays untouched.
+
+## Platform backends
+
+The texture-handle contract is identical on every backend — the same scene, the
+same component, the same handle sharing — only the decoder differs:
+
+| Platform | Decoder | Notes |
+|---|---|---|
+| Web / Desktop | `HTMLVideoElement` | Any browser-supported codec (H.264 `.mp4`, WebM…). WebGL2 uploads each frame zero-copy; WebGPU via a canvas readback. |
+| WeChat | Engine-owned MPEG-1 wasm (`videodec`, pl_mpeg) | Deterministic on every device — `wx.createVideoDecoder` is absent on PC and unreliable on phones. Ships automatically when a scene uses `Video`; plays the cooked `.esv` + `.m4a`. |
+
+## Best practices
+
+- **Keep `muted: true` for autoplay** — start sound-on playback from a user
+ gesture, or the browser will block it.
+- **Share one stream** across many surfaces via the texture handle +
+ `uvOffset` / `uvScale`, rather than many `Video` components decoding the same clip.
+- **Give the Video a renderable** — a `Sprite` (world), `UIVisual` (HUD), or
+ `Mesh2D` (custom geometry) — to actually show a frame.
+- **Mind `fitSize`** — it snaps the Sprite to the clip's native pixels on the first
+ frame; turn it off when you set an explicit size (e.g. a fixed screen).
+- **Target WeChat early** — the cook handles the MPEG-1 transcode for you, but keep
+ clips short and modest in resolution: software decode and download both scale with
+ size.
+
+## See also
+
+- [Sprites & Rendering](/docs/guides/sprites/) — the texture a Video writes to and its `uvOffset` / `uvScale`.
+- [Assets](/docs/guides/assets/) — loading and bundling video clips.
+- [WeChat MiniGame](/docs/guides/wechat/) — the wasm decode path and export.
diff --git a/docs/astro/src/content/docs/guides/wechat.mdx b/docs/astro/src/content/docs/guides/wechat.mdx
index 9f1c0ce35..d71ba6805 100644
--- a/docs/astro/src/content/docs/guides/wechat.mdx
+++ b/docs/astro/src/content/docs/guides/wechat.mdx
@@ -19,6 +19,7 @@ node build-tools/cli.js build -t wechat
node build-tools/cli.js build -t physics-wechat # physics
node build-tools/cli.js build -t spine-wechat # Spine (builds 3.8 + 4.1 + 4.2)
node build-tools/cli.js build -t basis-wechat # KTX2 / compressed textures
+node build-tools/cli.js build -t videodec-wechat # video (MPEG-1 wasm decoder)
```
<Aside type="note">
@@ -104,4 +105,5 @@ levels download on demand rather than bloating the initial package:
- [Assets](/docs/guides/assets/) — addressable groups and cooking map onto subpackages.
- [Building & Exporting](/docs/guides/build-export/) — the Package Project dialog.
+- [Video](/docs/guides/video/) — the MPEG-1 wasm decode path shipped for WeChat.
- [Networking](/docs/guides/networking/) — sockets use `wx.connectSocket` here.
diff --git a/docs/astro/src/content/docs/zh-cn/guides/video.mdx b/docs/astro/src/content/docs/zh-cn/guides/video.mdx
new file mode 100644
index 000000000..6fa3e258c
--- /dev/null
+++ b/docs/astro/src/content/docs/zh-cn/guides/video.mdx
@@ -0,0 +1,151 @@
+---
+title: 视频
+description: 视频完整指南——声明式 Video 组件、在精灵/UI/网格上播放、命令式 VideoPlayer、一条流喂多个表面、导入设置,以及微信的 wasm 解码路径。
+---
+
+import { Aside } from '@astrojs/starlight/components';
+
+Estella 通过声明式的 `Video` 组件把视频播放到任何可绘制的东西上——*一个纹理活着的精灵*。
+一条解码流驱动一张每帧刷新的活 GPU 纹理。在 Web 和桌面端由 `HTMLVideoElement` 解码
+(WebGL2 上零拷贝);在微信上则由引擎自带的 MPEG-1 wasm 解码器跑**完全相同**的纹理泵——
+手机、PC、无头环境一视同仁。
+
+## 工作原理
+
+给一个同时带有可绘制组件(`Sprite`、`UIVisual` 或 `Mesh2D`)的实体加上 `Video` 组件,
+视频系统就会把 `source` 解码成一张纹理,并每帧把该纹理句柄写到那个可绘制组件上。这张帧纹理
+是**无路径的**(永不淘汰)且稳定的,所以写一次的句柄在整条流的生命周期内一直有效。
+
+视频解码**只在运行模式下进行**——编辑时不解码,编辑器因此保持流畅。这个插件和音频一样默认
+已注册:无需加插件。
+
+<Aside type="note">
+ 静音的片段在任何地方都能自动播放。非静音的自动播放会被拦截,直到有用户手势(浏览器策略),
+ 所以带声音的播放要从点击或按键开始。`Video` 需要一个 `Sprite` / `UIVisual` / `Mesh2D`
+ 兄弟组件来显示——单独存在时它会解码,但没有可绘制的目标。
+</Aside>
+
+## `Video` 组件
+
+在编辑器里,给一个带 **Sprite** 的实体加上 **Video** 组件并设置它的 **Source**。在代码里,
+把它和可绘制组件一起插入:
+
+```typescript
+import { defineSystem, Commands, Transform, Sprite, Video } from 'esengine';
+
+const spawnScreen = defineSystem([Commands()], (cmds) => {
+ cmds.spawn()
+ .insert(Transform, { position: { x: 0, y: 0, z: 0 } })
+ .insert(Sprite, { size: { x: 360, y: 360 } })
+ .insert(Video, { source: 'assets/clip.mp4', autoplay: true, loop: true, muted: true });
+});
+```
+
+| 属性 | 类型 | 默认值 | 说明 |
+|---|---|---|---|
+| `source` | 资产 | `''` | 要播放的视频片段。 |
+| `autoplay` | boolean | `true` | 首帧就绪后立即开始。 |
+| `loop` | boolean | `true` | 播完后从头重新开始。 |
+| `muted` | boolean | `true` | 静音。静音片段可自动播放;非静音需要用户手势。 |
+| `volume` | number | `1.0` | 音量,0..1(`muted` 时忽略)。 |
+| `playbackRate` | number | `1.0` | 播放速度 / 音高(`1` = 正常)。 |
+| `fitSize` | boolean | `true` | 首帧时把 Sprite 调整为视频的原生像素尺寸。 |
+| `enabled` | boolean | `true` | 关闭即停止并释放流,而不移除组件。 |
+
+同一个组件会驱动它找到的第一个可绘制组件,顺序为 **`Sprite` → `UIVisual` → `Mesh2D`**:
+把 `Video` 放到 `UIVisual` 上做 HUD 里的视频面板,或放到 `Mesh2D` 上把片段映射到自定义几何。
+
+## 一条流,多个表面
+
+帧纹理只是一个普通句柄,所以任意多个精灵都能共享它——读取正在播放的精灵的 `texture` 再赋给
+其他精灵即可。每个表面再用 Sprite 的 `uvOffset` / `uvScale` 挑选帧的一个**区域**(与图集帧
+相同的约定,`v = 0` 是图像的底行)。一次解码喂饱所有表面——没有逐表面解码,也没有额外 API。
+
+```typescript
+// 把活视频纹理镜像到每个瓦片上,然后每个采样 1/N 的区域。
+const shareVideoTexture = defineSystem(
+ [Query(Sprite, Video), Query(Mut(Sprite), PuzzlePiece)],
+ (preview, pieces) => {
+ let texture = 0;
+ for (const [, sprite] of preview) { texture = sprite.texture; break; }
+ if (!texture) return; // 首帧还没解码
+ for (const [, sprite] of pieces) {
+ if (sprite.texture !== texture) sprite.texture = texture; // 每片写一次
+ }
+ },
+);
+```
+
+**video-puzzle** 示例正是这样把单个片段变成一整块由活瓦片组成、可交换的拼图板。
+
+## 命令式播放 —— `VideoPlayer`
+
+对于不用组件、由代码驱动的视频,`VideoPlayer` 资源播放一个源并交回一个**流句柄**,你可以把它的
+`textureHandle` 放到任何精灵或材质上:
+
+```typescript
+import { defineSystem, Res, VideoPlayer } from 'esengine';
+
+const playIntro = defineSystem([Res(VideoPlayer)], (video) => {
+ const stream = video.play('assets/intro.mp4', { loop: false, muted: true });
+ stream.onEnded = () => { /* 跳过片头 */ };
+ // stream.isReady 后把 stream.textureHandle 赋到某个 Sprite/Mesh2D 上
+});
+```
+
+| 成员 | 种类 | 说明 |
+|---|---|---|
+| `play()` / `pause()` / `stop()` | 方法 | 传输控制。 |
+| `seek(seconds)` | 方法 | 跳到某个时间。 |
+| `setVolume(v)` / `setMuted(m)` | 方法 | 实时调整音频。 |
+| `setLoop(l)` / `setPlaybackRate(r)` | 方法 | 实时调整循环 / 速度。 |
+| `textureHandle` | 只读 | 活帧纹理——首帧前为 `0`,之后无路径(永不淘汰)。 |
+| `width` / `height` | 只读 | 原生像素尺寸。 |
+| `isReady` | 只读 | 首帧已解码。 |
+| `isPlaying` | 只读 | 正在播放。 |
+| `currentTime` / `duration` | 只读 | 秒。 |
+| `onReady` / `onEnded` / `onError` | 回调 | 生命周期钩子。 |
+
+`video.play(source, options)` 接受与组件相同的选项(`autoplay`、`loop`、`muted`、
+`volume`、`playbackRate`);`video.stop(handle)` 结束一条流,`video.stopAll()` 结束所有流。
+
+## 导入设置与烹饪后的视频
+
+选中一个视频资产会显示预览以及**导入设置**。`Autoplay`、`Loop`、`Muted` 是把片段拖到
+`Video` 组件上时套用的建议状态。两个**高级**设置控制微信的转码:
+
+| 设置 | 范围 | 默认值 | 说明 |
+|---|---|---|---|
+| **Cook Quality** | `2`–`31` | `4` | wasm 解码路径的 MPEG-1 量化器——越低画质越高、文件越大。 |
+| **Cook Audio Bitrate** | `96` / `128` / `192` kbps | `128` | 解复用出的音轨的 AAC 码率。 |
+
+导出时,烹饪会把每个随包发布的视频转码成一个带编解码标记的 **`.esv`**(MPEG-1 程序流)加上一个
+**`.m4a`** 音轨兄弟文件——这正是 wasm 解码器读取的格式,音轨通过音频管线播放并作为视频的时钟。
+在 Web 和桌面端,原始文件原样播放。
+
+## 平台后端
+
+纹理句柄契约在每个后端上都完全一致——同一个场景、同一个组件、同一套句柄共享——只有解码器不同:
+
+| 平台 | 解码器 | 说明 |
+|---|---|---|
+| Web / 桌面 | `HTMLVideoElement` | 任何浏览器支持的编解码(H.264 `.mp4`、WebM……)。WebGL2 逐帧零拷贝上传;WebGPU 走 canvas 回读。 |
+| 微信 | 引擎自带 MPEG-1 wasm(`videodec`,pl_mpeg) | 每个设备上都确定——`wx.createVideoDecoder` 在 PC 上缺失、在手机上不可靠。场景一旦用到 `Video` 就自动随包;播放烹饪后的 `.esv` + `.m4a`。 |
+
+## 最佳实践
+
+- **自动播放就保持 `muted: true`**——带声音的播放要从用户手势开始,否则浏览器会拦截。
+- **共享一条流**给多个表面(用纹理句柄 + `uvOffset` / `uvScale`),而不是用多个 `Video` 组件
+ 重复解码同一个片段。
+- **给 Video 一个可绘制组件**——`Sprite`(世界)、`UIVisual`(HUD)或 `Mesh2D`(自定义几何)
+ ——才能真正显示帧。
+- **注意 `fitSize`**——它会在首帧把 Sprite 贴到片段的原生像素;当你设了明确尺寸(比如固定的
+ 屏幕)时把它关掉。
+- **尽早面向微信**——烹饪会替你完成 MPEG-1 转码,但把片段控制得短小、分辨率适中:软解码和下载
+ 都随尺寸增长。
+
+## 参见
+
+- [精灵与渲染](/docs/zh-cn/guides/sprites/) —— Video 写入的纹理及其 `uvOffset` / `uvScale`。
+- [资源](/docs/zh-cn/guides/assets/) —— 加载与打包视频片段。
+- [微信小游戏](/docs/zh-cn/guides/wechat/) —— wasm 解码路径与导出。
diff --git a/docs/astro/src/content/docs/zh-cn/guides/wechat.mdx b/docs/astro/src/content/docs/zh-cn/guides/wechat.mdx
index b30e5f267..36cdb10f1 100644
--- a/docs/astro/src/content/docs/zh-cn/guides/wechat.mdx
+++ b/docs/astro/src/content/docs/zh-cn/guides/wechat.mdx
@@ -18,6 +18,7 @@ node build-tools/cli.js build -t wechat
node build-tools/cli.js build -t physics-wechat # 物理
node build-tools/cli.js build -t spine-wechat # Spine(一次构建 3.8 + 4.1 + 4.2)
node build-tools/cli.js build -t basis-wechat # KTX2 / 压缩纹理
+node build-tools/cli.js build -t videodec-wechat # 视频(MPEG-1 wasm 解码器)
```
<Aside type="note">
@@ -92,4 +93,5 @@ node build-tools/cli.js build -t basis-wechat # KTX2 / 压缩纹理
- [资源](/docs/zh-cn/guides/assets/) —— 可寻址分组与 cook 映射到分包。
- [构建与导出](/docs/zh-cn/guides/build-export/) —— Package Project 对话框。
+- [视频](/docs/zh-cn/guides/video/) —— 为微信随包发布的 MPEG-1 wasm 解码路径。
- [联网](/docs/zh-cn/guides/networking/) —— 那里套接字用 `wx.connectSocket`。