Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(assets): a texture a mesh draws starts small while texture streaming is on, and grows to what its views need

yhha0b6bad08a93
diff --git a/desktop b/desktop
index 5710a5d2d..3e715db86 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 5710a5d2d939137b0e77107b5b23715bff604911
+Subproject commit 3e715db868520fee34f0bc8a40b9c49895e07ce7
diff --git a/docs/astro/src/content/docs/performance/quality-levels.mdx b/docs/astro/src/content/docs/performance/quality-levels.mdx
index 4afff5549..6f46a1ff5 100644
--- a/docs/astro/src/content/docs/performance/quality-levels.mdx
+++ b/docs/astro/src/content/docs/performance/quality-levels.mdx
@@ -62,26 +62,28 @@ it. A draw costs the CPU and the driver more than the GPU, and on a phone or in
first. To bring it down, see [Static batching](/docs/performance/static-batching/) for many different meshes and fewer shadow
cascades for the shadow pass.
-**Texture streaming** holds each texture that only meshes draw at the size its views need, and grows it back as the camera
-comes close. For every mesh a view draws, the renderer works out how many texels across its textures need for a texel to be
-about a pixel there, from the mesh's own UV density, its size and its distance; four times a second the textures that are the
-wrong size are decoded again from their files at the next power of two above that, and put behind the same handle. A texture
-grows as soon as it is needed bigger and shrinks only after it has been needed smaller for three seconds; one no view draws for
-five seconds goes down to 64 texels. It is on at Low and Medium and off at High; **Texture streaming budget** caps what the
-streamed textures take together (128 MB at Low, 256 MB at Medium, 0 is no cap), and past it they are held a level softer.
-
-Measured in the editor's Play at Low, in the `texture-streaming` editor check:
-
-| Example | GPU memory, streaming off | On, ten seconds in | The two pictures |
-|---|---|---|---|
-| `dusk-fort` | 285.6 MB | 31.0 MB (12 textures, 2048 held at 64 to 256) | mean difference 0.28 of 255 |
-| `bistro` | 1293.5 MB | 980.7 MB (228 textures, 62 moved so far) | mean difference 1.8 of 255; 0.05% of pixels differ visibly |
-
-What it does not do: it saves GPU memory, not the download or the decode, since a file is read and decoded whole the first time
-and again for each size it is moved to; a move costs a decode and an upload on the frame it lands, which shows as a frame or two
-of tens of milliseconds while a scene settles. A texture a sprite, a UI element, a tilemap, a particle or a trail also draws is left
-at its size. A KTX2 file is cut at a level without decoding; where the host transcodes KTX2 itself (the native builds) it cannot
-be, and the texture is left at its size.
+**Texture streaming** holds each texture that only meshes draw at the size its views need. A texture a MeshRenderer draws, as
+its texture, its normal map or through its material, is first loaded no larger than 256 texels on a side, so a scene comes up in a
+fraction of the memory; then, for every mesh a view draws, the renderer works out how many texels across its textures need for a
+texel to be about a pixel there, from the mesh's own UV density, its size and its distance, and four times a second the textures
+that are the wrong size are decoded again from their files at the next power of two above that and put behind the same handle. A
+texture grows as soon as it is needed bigger and shrinks only after it has been needed smaller for three seconds; one no view
+draws for five seconds goes down to 64 texels. It is on at Low and Medium and off at High; **Texture streaming budget** caps what
+the streamed textures take together (128 MB at Low, 256 MB at Medium, 0 is no cap), and past it they are held a level softer.
+
+Measured in the editor's Play at Low, in the `texture-streaming` editor check (the engine's own estimate of GPU memory):
+
+| Example | Streaming off | On: first frame | On: ten seconds in | The two pictures, settled |
+|---|---|---|---|---|
+| `dusk-fort` | 285.6 MB | 33.6 MB | 31.0 MB | mean difference 0.28 of 255 |
+| `bistro` | 1293.5 MB | 321.1 MB | 449.6 MB (228 textures, 149 moves) | mean difference 1.1 of 255; 0.04% of pixels differ visibly |
+
+What it does not do: it saves GPU memory, not the download, since a file is read whole the first time and again for each size it
+is moved to. Until a view has asked for more, a texture is drawn at the size it started at: the first second or two of a scene can
+be softer than the rest, and a move costs a decode and an upload on the frame it lands, which shows as a frame or two of tens of
+milliseconds while a scene settles. A texture a sprite, a UI element, a tilemap, a particle or a trail also draws is put back at its
+size and left there. A KTX2 file is cut at a level without decoding; where the host transcodes KTX2 itself (the native builds) it
+cannot be, and the texture is left at its size.
## Leaving effects out
diff --git a/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx b/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx
index db6e3d8ae..5d1793043 100644
--- a/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx
+++ b/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx
@@ -46,16 +46,16 @@ import { Aside } from '@astrojs/starlight/components';
**Draw call 预算**是这一档每帧最多应向 GPU 发出的 draw 数:低档 150、中档 400、高档 1500。所有 pass 都算,不只是场景自己的绘制列表:每个阴影级联都要把投影物再画一次,深度预通道要把不透明场景再画一次,后处理每个 pass 也要一次 draw。不会为了达标而少画什么。Profiler 的**渲染**模块显示这一帧的 GPU draw 数和预算,并按 pass 拆开;统计浮层在超出时把数字标红。一次 draw 花在 CPU 和驱动上的比花在 GPU 上的多,在手机和小游戏里,3D 场景最先用完的就是它。要降下来,不同网格很多时看[静态合并](/docs/zh-cn/performance/static-batching/),阴影那部分则减少级联数。
-**贴图流式加载**把只被网格使用的贴图按视图实际需要的尺寸常驻,相机靠近时再变大。渲染器对视图里画的每个网格,根据网格自身的 UV 密度、尺寸和距离,算出它的贴图需要多少纹素宽才能做到一个纹素约等于一个像素;每秒四次,尺寸不对的贴图会从文件按刚好够用的 2 的幂重新解码,放回同一个句柄后面。需要变大时立刻变大;需要变小时,要持续三秒才会缩;五秒没有被任何视图画到的贴图缩到 64 纹素。低档和中档默认开、高档默认关;**贴图流式预算**限制被流式管理的贴图合计占用(低档 128 MB,中档 256 MB,0 表示不设上限),超出时整体降一级清晰度。
+**贴图流式加载**把只被网格使用的贴图按视图实际需要的尺寸常驻。被 MeshRenderer 用到的贴图(它的贴图、法线贴图,或者经过它的材质)第一次加载时边长最多 256 纹素,所以场景刚出来时只占一小部分显存;之后,渲染器对视图里画的每个网格,根据网格自身的 UV 密度、尺寸和距离,算出它的贴图需要多少纹素宽才能做到一个纹素约等于一个像素,每秒四次把尺寸不对的贴图从文件按刚好够用的 2 的幂重新解码,放回同一个句柄后面。需要变大时立刻变大;需要变小时,要持续三秒才会缩;五秒没有被任何视图画到的贴图缩到 64 纹素。低档和中档默认开、高档默认关;**贴图流式预算**限制被流式管理的贴图合计占用(低档 128 MB,中档 256 MB,0 表示不设上限),超出时整体降一级清晰度。
-在编辑器 Play 里用低档实测(`texture-streaming` 编辑器检查):
+在编辑器 Play 里用低档实测(`texture-streaming` 编辑器检查,数字是引擎自己的显存估计):
-| 示例 | 关流式时的显存 | 开流式十秒后 | 两次画面 |
-|---|---|---|---|
-| `dusk-fort` | 285.6 MB | 31.0 MB(12 张贴图,2048 的缩到 64 到 256) | 平均差 0.28/255 |
-| `bistro` | 1293.5 MB | 980.7 MB(228 张贴图,十秒内完成 62 次缩放) | 平均差 1.8/255;0.05% 的像素有明显变化 |
+| 示例 | 关流式 | 开:首帧 | 开:十秒后 | 稳定后两次画面 |
+|---|---|---|---|---|
+| `dusk-fort` | 285.6 MB | 33.6 MB | 31.0 MB | 平均差 0.28/255 |
+| `bistro` | 1293.5 MB | 321.1 MB | 449.6 MB(228 张贴图,149 次缩放) | 平均差 1.1/255;0.04% 的像素有明显变化 |
-它做不到的:省的是显存,不是下载和解码。文件第一次仍然完整读取和解码,每换一次尺寸还要再解码一次;一次缩放要在它落地的那一帧付出解码和上传,场景稳定下来之前会有一两帧几十毫秒。同时被精灵、UI、瓦片地图、粒子或拖尾使用的贴图保持原尺寸。KTX2 文件直接从某一级截取,不用重新解码;宿主自己转码 KTX2 的平台(原生打包)做不到这一点,这类贴图保持原尺寸。
+它做不到的:省的是显存,不是下载。文件第一次仍然完整读取,每换一次尺寸还要再读一次。在视图要求更大之前,贴图按起始尺寸绘制:场景开头的一两秒可能比之后模糊;一次缩放要在它落地的那一帧付出解码和上传,场景稳定之前会有一两帧几十毫秒。同时被精灵、UI、瓦片地图、粒子或拖尾使用的贴图会恢复原尺寸并保持不变。KTX2 文件直接从某一级截取,不用重新解码;宿主自己转码 KTX2 的平台(原生打包)做不到这一点,这类贴图保持原尺寸。
## 去掉一些效果
diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.md
index b168086f5..12b041dbf 100644
--- a/sdk/etc/index.api.md
+++ b/sdk/etc/index.api.md
@@ -1088,7 +1088,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => void
setRefCounter: (counter: AssetRefCounter) => void
setRemoteRoot: (url: string | undefined) => void
setTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void
+setTextureStreamingStart: (edge: number | (() => number)) => void
textureFormatReport: () => TextureFormatReport
+textureFullSize: (handle: number) => { width: number; height: number; } | null
textureResizable: (handle: number) => boolean
texturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]
trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void
@@ -3235,6 +3237,8 @@ positions: Float32Array<ArrayBufferLike>
```
height: number
pixels: Uint8Array<ArrayBufferLike>
+sourceHeight: number | undefined
+sourceWidth: number | undefined
width: number
```
@@ -8990,6 +8994,8 @@ skipped: AmbientSkip | null
```
byType: Map<string, Set<string>>
dragonBones: SpineAssetRef[]
+meshMaterials: Set<string>
+meshTextures: Set<string>
rawByType: Map<string, Set<string>>
rawFor: Map<string, string>
spines: SpineAssetRef[]
@@ -11350,6 +11356,8 @@ number
## TextureInfo — interface @experimental
```
+fullHeight: number | undefined
+fullWidth: number | undefined
handle: number
height: number
width: number
@@ -11370,6 +11378,7 @@ held: Set<number>
inFlight: number
moves: number
refused: Set<number>
+restoring: Set<number>
textures: Map<number, StreamedTexture>
static new (): TextureStreaming
static prototype: TextureStreaming
diff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.md
index 4a13b3383..05df402b0 100644
--- a/sdk/etc/index.native.api.md
+++ b/sdk/etc/index.native.api.md
@@ -1088,7 +1088,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => void
setRefCounter: (counter: AssetRefCounter) => void
setRemoteRoot: (url: string | undefined) => void
setTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void
+setTextureStreamingStart: (edge: number | (() => number)) => void
textureFormatReport: () => TextureFormatReport
+textureFullSize: (handle: number) => { width: number; height: number; } | null
textureResizable: (handle: number) => boolean
texturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]
trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void
@@ -3243,6 +3245,8 @@ positions: Float32Array<ArrayBufferLike>
```
height: number
pixels: Uint8Array<ArrayBufferLike>
+sourceHeight: number | undefined
+sourceWidth: number | undefined
width: number
```
@@ -9203,6 +9207,8 @@ skipped: AmbientSkip | null
```
byType: Map<string, Set<string>>
dragonBones: SpineAssetRef[]
+meshMaterials: Set<string>
+meshTextures: Set<string>
rawByType: Map<string, Set<string>>
rawFor: Map<string, string>
spines: SpineAssetRef[]
@@ -11563,6 +11569,8 @@ number
## TextureInfo — interface @experimental
```
+fullHeight: number | undefined
+fullWidth: number | undefined
handle: number
height: number
width: number
@@ -11583,6 +11591,7 @@ held: Set<number>
inFlight: number
moves: number
refused: Set<number>
+restoring: Set<number>
textures: Map<number, StreamedTexture>
static new (): TextureStreaming
static prototype: TextureStreaming
diff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.md
index 24d1995ac..0156c4d28 100644
--- a/sdk/etc/index.node.api.md
+++ b/sdk/etc/index.node.api.md
@@ -1083,7 +1083,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => void
setRefCounter: (counter: AssetRefCounter) => void
setRemoteRoot: (url: string | undefined) => void
setTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void
+setTextureStreamingStart: (edge: number | (() => number)) => void
textureFormatReport: () => TextureFormatReport
+textureFullSize: (handle: number) => { width: number; height: number; } | null
textureResizable: (handle: number) => boolean
texturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]
trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void
@@ -3230,6 +3232,8 @@ positions: Float32Array<ArrayBufferLike>
```
height: number
pixels: Uint8Array<ArrayBufferLike>
+sourceHeight: number | undefined
+sourceWidth: number | undefined
width: number
```
@@ -8985,6 +8989,8 @@ skipped: AmbientSkip | null
```
byType: Map<string, Set<string>>
dragonBones: SpineAssetRef[]
+meshMaterials: Set<string>
+meshTextures: Set<string>
rawByType: Map<string, Set<string>>
rawFor: Map<string, string>
spines: SpineAssetRef[]
@@ -11345,6 +11351,8 @@ number
## TextureInfo — interface @experimental
```
+fullHeight: number | undefined
+fullWidth: number | undefined
handle: number
height: number
width: number
@@ -11365,6 +11373,7 @@ held: Set<number>
inFlight: number
moves: number
refused: Set<number>
+restoring: Set<number>
textures: Map<number, StreamedTexture>
static new (): TextureStreaming
static prototype: TextureStreaming
diff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.md
index 645c382d3..e9971dac6 100644
--- a/sdk/etc/minigame.api.md
+++ b/sdk/etc/minigame.api.md
@@ -1083,7 +1083,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => void
setRefCounter: (counter: AssetRefCounter) => void
setRemoteRoot: (url: string | undefined) => void
setTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void
+setTextureStreamingStart: (edge: number | (() => number)) => void
textureFormatReport: () => TextureFormatReport
+textureFullSize: (handle: number) => { width: number; height: number; } | null
textureResizable: (handle: number) => boolean
texturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]
trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void
@@ -3230,6 +3232,8 @@ positions: Float32Array<ArrayBufferLike>
```
height: number
pixels: Uint8Array<ArrayBufferLike>
+sourceHeight: number | undefined
+sourceWidth: number | undefined
width: number
```
@@ -9280,6 +9284,8 @@ skipped: AmbientSkip | null
```
byType: Map<string, Set<string>>
dragonBones: SpineAssetRef[]
+meshMaterials: Set<string>
+meshTextures: Set<string>
rawByType: Map<string, Set<string>>
rawFor: Map<string, string>
spines: SpineAssetRef[]
@@ -11640,6 +11646,8 @@ number
## TextureInfo — interface @experimental
```
+fullHeight: number | undefined
+fullWidth: number | undefined
handle: number
height: number
width: number
@@ -11660,6 +11668,7 @@ held: Set<number>
inFlight: number
moves: number
refused: Set<number>
+restoring: Set<number>
textures: Map<number, StreamedTexture>
static new (): TextureStreaming
static prototype: TextureStreaming
diff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.md
index 7bfc66078..e53bbf6ea 100644
--- a/sdk/etc/wechat.api.md
+++ b/sdk/etc/wechat.api.md
@@ -1083,7 +1083,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => void
setRefCounter: (counter: AssetRefCounter) => void
setRemoteRoot: (url: string | undefined) => void
setTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void
+setTextureStreamingStart: (edge: number | (() => number)) => void
textureFormatReport: () => TextureFormatReport
+textureFullSize: (handle: number) => { width: number; height: number; } | null
textureResizable: (handle: number) => boolean
texturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]
trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void
@@ -3230,6 +3232,8 @@ positions: Float32Array<ArrayBufferLike>
```
height: number
pixels: Uint8Array<ArrayBufferLike>
+sourceHeight: number | undefined
+sourceWidth: number | undefined
width: number
```
@@ -8992,6 +8996,8 @@ skipped: AmbientSkip | null
```
byType: Map<string, Set<string>>
dragonBones: SpineAssetRef[]
+meshMaterials: Set<string>
+meshTextures: Set<string>
rawByType: Map<string, Set<string>>
rawFor: Map<string, string>
spines: SpineAssetRef[]
@@ -11352,6 +11358,8 @@ number
## TextureInfo — interface @experimental
```
+fullHeight: number | undefined
+fullWidth: number | undefined
handle: number
height: number
width: number
@@ -11372,6 +11380,7 @@ held: Set<number>
inFlight: number
moves: number
refused: Set<number>
+restoring: Set<number>
textures: Map<number, StreamedTexture>
static new (): TextureStreaming
static prototype: TextureStreaming
diff --git a/sdk/src/asset/AssetLoader.ts b/sdk/src/asset/AssetLoader.ts
index 1cd028d96..54f52c2e2 100644
--- a/sdk/src/asset/AssetLoader.ts
+++ b/sdk/src/asset/AssetLoader.ts
@@ -14,6 +14,9 @@ export interface TextureResult {
handle: TextureHandle;
width: number;
height: number;
+ /** The file's own size, where it was made smaller than that (texture streaming); absent when it is the file's size. */
+ fullWidth?: number;
+ fullHeight?: number;
}
/**
@@ -122,6 +125,8 @@ export interface LoadContext {
* this loader was handed.
*/
acquireTexture(path: string, flipY?: boolean): Promise<AssetLease<TextureResult>>;
+ /** Say that material @p material binds texture @p texture, before it is acquired: a mesh material's textures may start small. */
+ noteMaterialTexture?(material: string, texture: string): void;
/**
* The same for an asset of ANY type — a material instance acquiring the
* parent it diffs against. The general door; {@link acquireTexture} is the
diff --git a/sdk/src/asset/Assets.ts b/sdk/src/asset/Assets.ts
index f9f81bcd1..db344c65d 100644
--- a/sdk/src/asset/Assets.ts
+++ b/sdk/src/asset/Assets.ts
@@ -387,6 +387,14 @@ function warnAmbiguousRelease_(key: string): void {
+ 'which may not be the one the caller holds. Release it by its lease instead.');
}
+/**
+ * What edge a mesh texture is first loaded at, asked as each one loads (texture streaming says, by the quality level): set once per
+ * realm, before any Assets exists, since a scene can load before the first frame. 0 loads it whole.
+ */
+let textureStreamingStart: () => number = () => 0;
+/** @internal */
+export function setTextureStreamingStartProvider(provider: () => number): void { textureStreamingStart = provider; }
+
export class Assets {
readonly backend: Backend;
readonly catalog: Catalog;
@@ -479,6 +487,12 @@ export class Assets {
* load. The REVERSE of resolution — inspectors and tooling that only hold a
* live World handle use it to name the asset (see {@link pathForHandle}). */
private handleToPath_ = new Map<string, string>();
+ /** Textures a mesh draws, by resolved path, directly or through a material: what texture streaming may load small to begin with. */
+ private readonly meshTextures_ = new Set<string>();
+ /** Materials a mesh draws with, by resolved path: their textures join {@link meshTextures_} as the material loads them. */
+ private readonly meshMaterials_ = new Set<string>();
+ /** The edge a mesh texture is first loaded at while texture streaming is on; 0 loads it whole. */
+ private streamingStartEdge_: number | (() => number) = 0;
/**
* Geometry built at run time rather than loaded (a static batch), by handle. It has no file to be replayed
@@ -557,6 +571,19 @@ export class Assets {
return size;
}
+ /** The size of the file behind @p handle, whatever size it is held at; null for a texture no file of this realm's stands behind. */
+ textureFullSize(handle: number): { width: number; height: number } | null {
+ return this.textureLoader_.fullSize(handle);
+ }
+
+ /**
+ * Load the textures meshes draw no larger than @p edge on a side to begin with (0 loads them whole): texture streaming grows each
+ * to what its views need. A texture something else also draws is put back at its size by the streamer.
+ */
+ setTextureStreamingStart(edge: number | (() => number)): void {
+ this.streamingStartEdge_ = edge;
+ }
+
/** Whether {@link resizeTexture} can move @p handle between sizes. */
textureResizable(handle: number): boolean {
return this.textureLoader_.resizable(handle);
@@ -729,6 +756,10 @@ export class Assets {
const revived = this.reviveResidentTexture_(cacheKey);
if (revived) return Promise.resolve(revived);
this.textureLoader_.setPendingSettings(settings);
+ // Asked as each texture loads: what decides it (the quality level) may arrive after whoever said how to ask.
+ const own = typeof this.streamingStartEdge_ === 'function' ? this.streamingStartEdge_() : this.streamingStartEdge_;
+ const start = own > 0 ? own : textureStreamingStart();
+ if (start > 0 && this.meshTextures_.has(path)) this.textureLoader_.setPendingStartEdge(start);
return flip
? this.textureLoader_.load(path, this.getLoadContext_())
: this.textureLoader_.loadRaw(path, this.getLoadContext_());
@@ -778,6 +809,7 @@ export class Assets {
rm.releaseTexture(handle);
return null;
}
+ this.textureLoader_.adoptRevived(handle, cacheKey);
return { handle, width: dims.width, height: dims.height };
}
@@ -1599,6 +1631,8 @@ export class Assets {
): Promise<SceneAssetResult> {
const missing: MissingAsset[] = [];
const discovered = discoverSceneAssets(sceneData, (ref) => this.resolveRef(ref));
+ for (const p of discovered.meshTextures) this.meshTextures_.add(p);
+ for (const p of discovered.meshMaterials) this.meshMaterials_.add(p);
for (const ref of discovered.unresolved) {
missing.push({ ref, reason: 'unresolved' });
}
@@ -2998,6 +3032,11 @@ export class Assets {
const { result, lease } = await self.acquireTextureVariant_(path, flipY !== false);
return self.textureLease_(lease, result);
},
+ noteMaterialTexture(material: string, texture: string): void {
+ if (self.meshMaterials_.has(self.resolveLoadPath_(material)) || self.meshMaterials_.has(material)) {
+ self.meshTextures_.add(self.resolveLoadPath_(texture));
+ }
+ },
async acquireAsset<T>(type: string, ref: string): Promise<AssetLease<T>> {
return self.acquireTyped<T>(type, ref);
},
diff --git a/sdk/src/asset/discoverAssets.ts b/sdk/src/asset/discoverAssets.ts
index 4fc3cd377..f35c29a21 100644
--- a/sdk/src/asset/discoverAssets.ts
+++ b/sdk/src/asset/discoverAssets.ts
@@ -66,6 +66,10 @@ export interface SceneAssetRefs {
* assets don't silently turn into texture handle 0.
*/
unresolved: string[];
+ /** Resolved paths of the textures a MeshRenderer draws directly (its texture and normal map): what texture streaming may start small. */
+ meshTextures: Set<string>;
+ /** Resolved paths of the materials a MeshRenderer draws with: their textures may start small too. */
+ meshMaterials: Set<string>;
}
/** Turns a raw serialized ref into a concrete path (or null if unknown). */
@@ -84,6 +88,8 @@ export function discoverSceneAssets(
// belong to two runtimes.
const skeletalKeys = new Set<string>();
const unresolved: string[] = [];
+ const meshTextures = new Set<string>();
+ const meshMaterials = new Set<string>();
const resolve = (raw: string): string | null => {
const r = refResolver ? refResolver(raw) : raw;
@@ -126,6 +132,14 @@ export function discoverSceneAssets(
const data = compData.data as Record<string, unknown>;
+ if (compData.type === 'MeshRenderer') {
+ for (const [field, into] of [['texture', meshTextures], ['normalMap', meshTextures], ['material', meshMaterials]] as const) {
+ const raw = data[field];
+ const at = typeof raw === 'string' && raw ? (refResolver ? refResolver(raw) : raw) : null;
+ if (at) into.add(at);
+ }
+ }
+
if (comp.discoverAssets) {
for (const ref of comp.discoverAssets(data)) {
if (typeof ref.path === 'string' && ref.path) {
@@ -163,7 +177,7 @@ export function discoverSceneAssets(
}
}
- return { byType, rawByType, rawFor, spines, dragonBones, unresolved };
+ return { byType, rawByType, rawFor, spines, dragonBones, unresolved, meshTextures, meshMaterials };
}
export function getAssetPathsByType(refs: SceneAssetRefs, type: AssetFieldType): Set<string> {
diff --git a/sdk/src/asset/imageDecode.ts b/sdk/src/asset/imageDecode.ts
index b4d797760..6feb7c2ce 100644
--- a/sdk/src/asset/imageDecode.ts
+++ b/sdk/src/asset/imageDecode.ts
@@ -47,6 +47,9 @@ export interface DecodedPixels {
width: number;
height: number;
pixels: Uint8Array;
+ /** The image's own size, where it was decoded smaller than that; absent when it is the image's size. */
+ sourceWidth?: number;
+ sourceHeight?: number;
}
/**
@@ -56,9 +59,13 @@ export interface DecodedPixels {
* v=1=top for general textures, v=0=top for spine / bitmap fonts.
*/
export async function decodeImagePixels(src: ImageBitmapSource, maxEdge = 0): Promise<DecodedPixels> {
- const bitmap = await shrinkBitmap(await decodeImageBitmap(src, false), maxEdge);
+ const whole = await decodeImageBitmap(src, false);
+ const sourceWidth = whole.width, sourceHeight = whole.height;
+ const bitmap = await shrinkBitmap(whole, maxEdge);
try {
- return readImagePixels(bitmap, maxEdge);
+ const pixels = readImagePixels(bitmap, maxEdge);
+ // The bitmap was already made smaller, so the read cannot see what it was made from.
+ return pixels.width < sourceWidth || pixels.height < sourceHeight ? { ...pixels, sourceWidth, sourceHeight } : pixels;
} finally {
bitmap.close?.();
}
@@ -107,7 +114,8 @@ export function readImagePixels(image: PlatformImage | ImageBitmap, maxEdge = 0)
if (!ctx) throw new Error('imageDecode: 2D context unavailable for pixel decode');
ctx.drawImage(image, 0, 0, width, height);
const data = ctx.getImageData(0, 0, width, height);
- return { width, height, pixels: new Uint8Array(data.data.buffer) };
+ const pixels = new Uint8Array(data.data.buffer);
+ return scale < 1 ? { width, height, pixels, sourceWidth: image.width, sourceHeight: image.height } : { width, height, pixels };
}
/**
diff --git a/sdk/src/asset/loaders/MaterialAssetLoader.ts b/sdk/src/asset/loaders/MaterialAssetLoader.ts
index 4f3d0ca32..4c06e4f27 100644
--- a/sdk/src/asset/loaders/MaterialAssetLoader.ts
+++ b/sdk/src/asset/loaders/MaterialAssetLoader.ts
@@ -55,6 +55,7 @@ export class MaterialAssetLoader implements AssetLoader<MaterialResult> {
for (const [name, value] of Object.entries(data.properties)) {
if (typeof value !== 'string') continue;
const texPath = resolveDocumentRef(matPath, value);
+ ctx.noteMaterialTexture?.(matPath, texPath);
try {
const tex = await ctx.acquireTexture(texPath);
Material.setUniform(handle, name, Material.tex(tex.value.handle));
diff --git a/sdk/src/asset/loaders/TextureLoader.ts b/sdk/src/asset/loaders/TextureLoader.ts
index 869d5feab..b7370befc 100644
--- a/sdk/src/asset/loaders/TextureLoader.ts
+++ b/sdk/src/asset/loaders/TextureLoader.ts
@@ -32,7 +32,7 @@ export type TexturePixelDecoder = (
path: string,
flip: boolean,
maxEdge?: number,
-) => Promise<{ width: number; height: number; pixels: Uint8Array }>;
+) => Promise<{ width: number; height: number; pixels: Uint8Array; sourceWidth?: number; sourceHeight?: number }>;
/**
* Where a host keeps its textures GPU-compressed: the KTX2 for `path`, `'pending'`
@@ -132,7 +132,13 @@ export class TextureLoader implements AssetLoader<TextureResult> {
*/
private readonly formats_ = new TextureFormatLog();
/** Where each texture this loader decoded from a file came from: what a resize decodes again. */
- private readonly sources_ = new Map<number, { path: string; flip: boolean; settings?: TextureImportSettings }>();
+ private readonly sources_ = new Map<number, { path: string; flip: boolean; settings?: TextureImportSettings; fullWidth: number; fullHeight: number }>();
+ private pendingStartEdge_ = 0;
+ /** How many loads began smaller than their file (texture streaming), and how many were asked to: a report, and what a check reads. */
+ startedSmall = 0;
+ askedSmall = 0;
+ /** The same sources by residency key, kept past a release: a texture revived from the warm cache is the same file at the size it was left at. */
+ private readonly sourcesByKey_ = new Map<string, { path: string; flip: boolean; settings?: TextureImportSettings; fullWidth: number; fullHeight: number }>();
/** The decision the in-flight upload came to. Set by the one entry point that
* chooses a path (decodeAndUpload_) and read by the one that records. */
private lastDecision_: TextureUploadDecision = RAW_PAYLOAD_UPLOAD;
@@ -237,11 +243,32 @@ export class TextureLoader implements AssetLoader<TextureResult> {
/** Whether @p handle was decoded from a file by this loader, so it can be decoded again at another size. */
resizable(handle: number): boolean { return this.sources_.has(handle); }
+ /** The size of the file behind @p handle, whatever size it is held at; null for one this loader did not decode from a file. */
+ fullSize(handle: number): { width: number; height: number } | null {
+ const s = this.sources_.get(handle);
+ return s ? { width: s.fullWidth, height: s.fullHeight } : null;
+ }
+
+ /** Decode the next load no larger than @p edge on a side (texture streaming's start), where the file's own size can still be known. */
+ setPendingStartEdge(edge: number): void { this.pendingStartEdge_ = Math.max(0, edge); }
+
+ /** A texture the warm cache gave back under @p key is the one this loader decoded under it: resizable again. */
+ adoptRevived(handle: number, key: string): void {
+ const source = this.sourcesByKey_.get(key);
+ if (source) this.sources_.set(handle, source);
+ }
+
private async loadWithFlip(
path: string, ctx: LoadContext, flip: boolean, settings?: TextureImportSettings,
): Promise<TextureResult> {
- const result = await this.decodeAndUpload_(path, ctx, flip, settings);
- this.sources_.set(result.handle, { path, flip, settings });
+ const start = this.pendingStartEdge_;
+ this.pendingStartEdge_ = 0;
+ if (start > 0) this.askedSmall++;
+ const result = await this.decodeAndUpload_(path, ctx, flip, settings, start);
+ const source = { path, flip, settings, fullWidth: result.fullWidth ?? result.width, fullHeight: result.fullHeight ?? result.height };
+ this.sources_.set(result.handle, source);
+ if (result.fullWidth !== undefined) this.startedSmall++;
+ this.sourcesByKey_.set(textureResidencyKey(path, flip), source);
// Recorded HERE, once, for every path into the loader — the compressed
// paths, the pixel decoder and the <img> upload alike. A record kept only
// where a fallback happens cannot say that everything else was fine.
@@ -277,11 +304,17 @@ export class TextureLoader implements AssetLoader<TextureResult> {
if (this.pixelDecoder_) {
const result = await this.pixelDecoder_(path, flip, edge);
const handle = createTextureFromPixels(this.module_, result, TextureContent.Asset, flip, pixelParams(settings));
- return { handle, width: result.width, height: result.height };
+ // A decoder that made it smaller says what it was made from; one that could not (a host decoding whole) answers with the
+ // file's own size, and the texture is simply whole.
+ return result.sourceWidth !== undefined && result.sourceHeight !== undefined
+ ? { handle, width: result.width, height: result.height, fullWidth: result.sourceWidth, fullHeight: result.sourceHeight }
+ : { handle, width: result.width, height: result.height };
}
const url = ctx.backend.resolveUrl(ctx.catalog.getBuildPath(path));
- const img = await this.loadImage(url, flip, edge, this.imageRevisions_.get(path));
- return this.createTextureFromImage(img, flip, settings);
+ const full = { width: 0, height: 0 };
+ const img = await this.loadImage(url, flip, edge, this.imageRevisions_.get(path), full);
+ const made = this.createTextureFromImage(img, flip, settings);
+ return full.width > made.width || full.height > made.height ? { ...made, fullWidth: full.width, fullHeight: full.height } : made;
}
/**
@@ -291,6 +324,19 @@ export class TextureLoader implements AssetLoader<TextureResult> {
*/
private async uploadKtx2_(
path: string, bytes: Uint8Array, settings?: TextureImportSettings, maxEdge = 0,
+ ): Promise<TextureResult> {
+ const made = await this.uploadKtx2Levels_(path, bytes, settings, maxEdge);
+ // The header's pixelWidth and pixelHeight (bytes 20 and 24): the file's size, whatever level the upload started at.
+ if (bytes.byteLength >= 28) {
+ const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+ const fullWidth = view.getUint32(20, true), fullHeight = view.getUint32(24, true);
+ if (fullWidth > made.width || fullHeight > made.height) return { ...made, fullWidth, fullHeight };
+ }
+ return made;
+ }
+
+ private async uploadKtx2Levels_(
+ path: string, bytes: Uint8Array, settings?: TextureImportSettings, maxEdge = 0,
): Promise<TextureResult> {
if (!isKtx2(bytes)) throw new Error(`TextureLoader: ${path} is not a KTX2 file`);
// Native (embedded Dawn): the ResourceManager transcodes the KTX2 with the
@@ -377,11 +423,12 @@ export class TextureLoader implements AssetLoader<TextureResult> {
* on it uploads every texture upside-down. The raw `<img>` fallback keeps the
* flag (it works for element/pixel sources) — see {@link createTextureFromImage}.
*/
- private loadImage(src: string, flip: boolean, maxEdge = 0, revision?: number): Promise<PlatformImage | ImageBitmap> {
+ private loadImage(src: string, flip: boolean, maxEdge = 0, revision?: number, full?: { width: number; height: number }): Promise<PlatformImage | ImageBitmap> {
return new Promise((resolve, reject) => {
const img = platformCreateImage();
img.crossOrigin = 'anonymous';
img.onload = async () => {
+ if (full) { full.width = img.width; full.height = img.height; }
if (typeof createImageBitmap !== 'undefined') {
try {
resolve(await shrinkBitmap(await decodeImageBitmap(img, flip), maxEdge));
diff --git a/sdk/src/render/textureStreaming.ts b/sdk/src/render/textureStreaming.ts
index 38cd70705..0f34952cc 100644
--- a/sdk/src/render/textureStreaming.ts
+++ b/sdk/src/render/textureStreaming.ts
@@ -22,6 +22,7 @@ import { defineResource } from '../ecs/resource';
import { getComponent } from '../ecs/component';
import { playModeOnly } from '../ecs/env';
import { Assets } from '../asset/AssetPlugin';
+import { setTextureStreamingStartProvider } from '../asset/Assets';
import { engineApi } from '../ecs/bridge/engineApi';
import { getTextureDimensions } from '../wasm/resourceManager';
import { platformNow } from '../platform';
@@ -134,6 +135,8 @@ export class TextureStreaming {
readonly held = new Set<number>();
/** Textures that came back larger than asked (a host that cannot cut a file at a level): left alone from then on. */
readonly refused = new Set<number>();
+ /** Textures being put back at their size because something other than a mesh draws them. */
+ readonly restoring = new Set<number>();
moves = 0;
failures = 0;
inFlight = 0;
@@ -165,7 +168,16 @@ export class TextureStreamingPlugin implements Plugin {
app.insertResource(TextureStreamingRes, state);
const world = app.world;
let last = 0, lastHeld = -Infinity;
- const MAX_IN_FLIGHT = 2;
+ const MAX_IN_FLIGHT = 4;
+ /** The edge a mesh texture starts at while streaming is on. */
+ const START_EDGE = 256;
+ // Asked as each texture loads, not first in the frame loop: a scene may start loading before the first frame does, and the
+ // quality level may be settled after this plugin is built.
+ const startEdge = (): number => {
+ const quality = app.hasResource(Quality) ? app.getResource(Quality) : null;
+ return quality?.profile.textureStreaming && engineApi(app)?.renderer_takeTextureWants ? START_EDGE : 0;
+ };
+ setTextureStreamingStartProvider(startEdge);
const parse = (raw: string): Map<number, number> => {
const out = new Map<number, number>();
@@ -207,15 +219,26 @@ export class TextureStreamingPlugin implements Plugin {
let entry = state.textures.get(handle);
if (!entry) {
if (!assets.textureResizable(handle)) continue;
- const size = getTextureDimensions(handle);
+ const size = assets.textureFullSize(handle) ?? getTextureDimensions(handle);
if (!size || size.width <= 0 || size.height <= 0) continue;
- entry = { handle, fullWidth: size.width, fullHeight: size.height, edge: Math.max(size.width, size.height), wanted: 0, seenAt: now, smallerSince: -1 };
+ const held = getTextureDimensions(handle);
+ entry = { handle, fullWidth: size.width, fullHeight: size.height, edge: held ? Math.max(held.width, held.height) : Math.max(size.width, size.height), wanted: 0, seenAt: now, smallerSince: -1 };
state.textures.set(handle, entry);
}
entry.wanted = entry.seenAt === now ? Math.max(entry.wanted, side) : side;
entry.seenAt = now;
}
- for (const handle of state.held) state.textures.delete(handle);
+ for (const handle of state.held) {
+ state.textures.delete(handle);
+ // Loaded small for a mesh and drawn by something no view measures: put back at its size.
+ const full = assets.textureFullSize(handle), now_ = getTextureDimensions(handle);
+ if (full && now_ && (now_.width < full.width || now_.height < full.height) && state.inFlight < MAX_IN_FLIGHT && !state.restoring.has(handle)) {
+ state.restoring.add(handle);
+ state.inFlight++;
+ void assets.resizeTexture(handle, 0).catch(() => { state.failures++; })
+ .finally(() => { state.inFlight--; state.restoring.delete(handle); });
+ }
+ }
const options: TextureStreamingOptions = {
...TEXTURE_STREAMING_DEFAULTS,
diff --git a/sdk/tests/texture-streaming-start.test.ts b/sdk/tests/texture-streaming-start.test.ts
new file mode 100644
index 000000000..85539537a
--- /dev/null
+++ b/sdk/tests/texture-streaming-start.test.ts
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+import { describe, expect, it } from 'vitest';
+import { levelsFrom } from '../src/asset/compressed';
+import { discoverSceneAssets } from '../src/asset/discoverAssets';
+
+const level = (side: number) => ({ width: side, height: side / 2, data: new Uint8Array(side) });
+
+describe('a KTX2 chain cut at a level', () => {
+ const chain = { width: 1024, height: 512, data: new Uint8Array(1024), levels: [level(1024), level(512), level(256), level(128), level(64)] };
+
+ it('starts at the first level no larger than asked, and keeps every level below it', () => {
+ const cut = levelsFrom(chain, 300);
+ expect(cut.width).toBe(256);
+ expect(cut.height).toBe(128);
+ expect(cut.levels!.map((l) => l.width)).toEqual([256, 128, 64]);
+ expect(cut.data).toBe(chain.levels[2]!.data);
+ });
+
+ it('leaves a chain whole when nothing is asked, or when it is small enough already', () => {
+ expect(levelsFrom(chain, 0)).toBe(chain);
+ expect(levelsFrom(chain, 4096)).toBe(chain);
+ });
+
+ it('keeps the smallest level when every level is larger than asked', () => {
+ expect(levelsFrom(chain, 16).width).toBe(64);
+ });
+});
+
+describe('which textures a scene\'s meshes draw', () => {
+ it('are a MeshRenderer\'s texture and normal map, and the materials it draws with, and not a sprite\'s', () => {
+ const scene = {
+ version: '1.0', name: 'S', entities: [
+ { id: 1, name: 'Wall', parent: null, children: [], visible: true, components: [
+ { type: 'MeshRenderer', data: { mesh: 'builtin:cube', texture: 'wall.png', normalMap: 'wall_n.png', material: 'wall.esmaterial' } }] },
+ { id: 2, name: 'Sign', parent: null, children: [], visible: true, components: [
+ { type: 'Sprite', data: { texture: 'sign.png' } }] },
+ ],
+ };
+ const found = discoverSceneAssets(scene as never, (ref) => `assets/${ref}`);
+ expect([...found.meshTextures].sort()).toEqual(['assets/wall.png', 'assets/wall_n.png']);
+ expect([...found.meshMaterials]).toEqual(['assets/wall.esmaterial']);
+ });
+});