A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(assets): a texture a mesh draws starts small while texture streaming is on, and grows to what its views need
yhha0b6bad08a93
diff --git a/desktop b/desktopindex 5710a5d2d..3e715db86 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit 5710a5d2d939137b0e77107b5b23715bff604911+Subproject commit 3e715db868520fee34f0bc8a40b9c49895e07ce7diff --git a/docs/astro/src/content/docs/performance/quality-levels.mdx b/docs/astro/src/content/docs/performance/quality-levels.mdxindex 4afff5549..6f46a1ff5 100644--- a/docs/astro/src/content/docs/performance/quality-levels.mdx+++ b/docs/astro/src/content/docs/performance/quality-levels.mdx@@ -62,26 +62,28 @@ it. A draw costs the CPU and the driver more than the GPU, and on a phone or infirst. To bring it down, see [Static batching](/docs/performance/static-batching/) for many different meshes and fewer shadowcascades for the shadow pass.-**Texture streaming** holds each texture that only meshes draw at the size its views need, and grows it back as the camera-comes close. For every mesh a view draws, the renderer works out how many texels across its textures need for a texel to be-about a pixel there, from the mesh's own UV density, its size and its distance; four times a second the textures that are the-wrong size are decoded again from their files at the next power of two above that, and put behind the same handle. A texture-grows as soon as it is needed bigger and shrinks only after it has been needed smaller for three seconds; one no view draws for-five seconds goes down to 64 texels. It is on at Low and Medium and off at High; **Texture streaming budget** caps what the-streamed textures take together (128 MB at Low, 256 MB at Medium, 0 is no cap), and past it they are held a level softer.--Measured in the editor's Play at Low, in the `texture-streaming` editor check:--| Example | GPU memory, streaming off | On, ten seconds in | The two pictures |-|---|---|---|---|-| `dusk-fort` | 285.6 MB | 31.0 MB (12 textures, 2048 held at 64 to 256) | mean difference 0.28 of 255 |-| `bistro` | 1293.5 MB | 980.7 MB (228 textures, 62 moved so far) | mean difference 1.8 of 255; 0.05% of pixels differ visibly |--What it does not do: it saves GPU memory, not the download or the decode, since a file is read and decoded whole the first time-and again for each size it is moved to; a move costs a decode and an upload on the frame it lands, which shows as a frame or two-of tens of milliseconds while a scene settles. A texture a sprite, a UI element, a tilemap, a particle or a trail also draws is left-at its size. A KTX2 file is cut at a level without decoding; where the host transcodes KTX2 itself (the native builds) it cannot-be, and the texture is left at its size.+**Texture streaming** holds each texture that only meshes draw at the size its views need. A texture a MeshRenderer draws, as+its texture, its normal map or through its material, is first loaded no larger than 256 texels on a side, so a scene comes up in a+fraction of the memory; then, for every mesh a view draws, the renderer works out how many texels across its textures need for a+texel to be about a pixel there, from the mesh's own UV density, its size and its distance, and four times a second the textures+that are the wrong size are decoded again from their files at the next power of two above that and put behind the same handle. A+texture grows as soon as it is needed bigger and shrinks only after it has been needed smaller for three seconds; one no view+draws for five seconds goes down to 64 texels. It is on at Low and Medium and off at High; **Texture streaming budget** caps what+the streamed textures take together (128 MB at Low, 256 MB at Medium, 0 is no cap), and past it they are held a level softer.++Measured in the editor's Play at Low, in the `texture-streaming` editor check (the engine's own estimate of GPU memory):++| Example | Streaming off | On: first frame | On: ten seconds in | The two pictures, settled |+|---|---|---|---|---|+| `dusk-fort` | 285.6 MB | 33.6 MB | 31.0 MB | mean difference 0.28 of 255 |+| `bistro` | 1293.5 MB | 321.1 MB | 449.6 MB (228 textures, 149 moves) | mean difference 1.1 of 255; 0.04% of pixels differ visibly |++What it does not do: it saves GPU memory, not the download, since a file is read whole the first time and again for each size it+is moved to. Until a view has asked for more, a texture is drawn at the size it started at: the first second or two of a scene can+be softer than the rest, and a move costs a decode and an upload on the frame it lands, which shows as a frame or two of tens of+milliseconds while a scene settles. A texture a sprite, a UI element, a tilemap, a particle or a trail also draws is put back at its+size and left there. A KTX2 file is cut at a level without decoding; where the host transcodes KTX2 itself (the native builds) it+cannot be, and the texture is left at its size.## Leaving effects outdiff --git a/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx b/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdxindex db6e3d8ae..5d1793043 100644--- a/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx+++ b/docs/astro/src/content/docs/zh-cn/performance/quality-levels.mdx@@ -46,16 +46,16 @@ import { Aside } from '@astrojs/starlight/components';**Draw call 预算**是这一档每帧最多应向 GPU 发出的 draw 数:低档 150、中档 400、高档 1500。所有 pass 都算,不只是场景自己的绘制列表:每个阴影级联都要把投影物再画一次,深度预通道要把不透明场景再画一次,后处理每个 pass 也要一次 draw。不会为了达标而少画什么。Profiler 的**渲染**模块显示这一帧的 GPU draw 数和预算,并按 pass 拆开;统计浮层在超出时把数字标红。一次 draw 花在 CPU 和驱动上的比花在 GPU 上的多,在手机和小游戏里,3D 场景最先用完的就是它。要降下来,不同网格很多时看[静态合并](/docs/zh-cn/performance/static-batching/),阴影那部分则减少级联数。-**贴图流式加载**把只被网格使用的贴图按视图实际需要的尺寸常驻,相机靠近时再变大。渲染器对视图里画的每个网格,根据网格自身的 UV 密度、尺寸和距离,算出它的贴图需要多少纹素宽才能做到一个纹素约等于一个像素;每秒四次,尺寸不对的贴图会从文件按刚好够用的 2 的幂重新解码,放回同一个句柄后面。需要变大时立刻变大;需要变小时,要持续三秒才会缩;五秒没有被任何视图画到的贴图缩到 64 纹素。低档和中档默认开、高档默认关;**贴图流式预算**限制被流式管理的贴图合计占用(低档 128 MB,中档 256 MB,0 表示不设上限),超出时整体降一级清晰度。+**贴图流式加载**把只被网格使用的贴图按视图实际需要的尺寸常驻。被 MeshRenderer 用到的贴图(它的贴图、法线贴图,或者经过它的材质)第一次加载时边长最多 256 纹素,所以场景刚出来时只占一小部分显存;之后,渲染器对视图里画的每个网格,根据网格自身的 UV 密度、尺寸和距离,算出它的贴图需要多少纹素宽才能做到一个纹素约等于一个像素,每秒四次把尺寸不对的贴图从文件按刚好够用的 2 的幂重新解码,放回同一个句柄后面。需要变大时立刻变大;需要变小时,要持续三秒才会缩;五秒没有被任何视图画到的贴图缩到 64 纹素。低档和中档默认开、高档默认关;**贴图流式预算**限制被流式管理的贴图合计占用(低档 128 MB,中档 256 MB,0 表示不设上限),超出时整体降一级清晰度。-在编辑器 Play 里用低档实测(`texture-streaming` 编辑器检查):+在编辑器 Play 里用低档实测(`texture-streaming` 编辑器检查,数字是引擎自己的显存估计):-| 示例 | 关流式时的显存 | 开流式十秒后 | 两次画面 |-|---|---|---|---|-| `dusk-fort` | 285.6 MB | 31.0 MB(12 张贴图,2048 的缩到 64 到 256) | 平均差 0.28/255 |-| `bistro` | 1293.5 MB | 980.7 MB(228 张贴图,十秒内完成 62 次缩放) | 平均差 1.8/255;0.05% 的像素有明显变化 |+| 示例 | 关流式 | 开:首帧 | 开:十秒后 | 稳定后两次画面 |+|---|---|---|---|---|+| `dusk-fort` | 285.6 MB | 33.6 MB | 31.0 MB | 平均差 0.28/255 |+| `bistro` | 1293.5 MB | 321.1 MB | 449.6 MB(228 张贴图,149 次缩放) | 平均差 1.1/255;0.04% 的像素有明显变化 |-它做不到的:省的是显存,不是下载和解码。文件第一次仍然完整读取和解码,每换一次尺寸还要再解码一次;一次缩放要在它落地的那一帧付出解码和上传,场景稳定下来之前会有一两帧几十毫秒。同时被精灵、UI、瓦片地图、粒子或拖尾使用的贴图保持原尺寸。KTX2 文件直接从某一级截取,不用重新解码;宿主自己转码 KTX2 的平台(原生打包)做不到这一点,这类贴图保持原尺寸。+它做不到的:省的是显存,不是下载。文件第一次仍然完整读取,每换一次尺寸还要再读一次。在视图要求更大之前,贴图按起始尺寸绘制:场景开头的一两秒可能比之后模糊;一次缩放要在它落地的那一帧付出解码和上传,场景稳定之前会有一两帧几十毫秒。同时被精灵、UI、瓦片地图、粒子或拖尾使用的贴图会恢复原尺寸并保持不变。KTX2 文件直接从某一级截取,不用重新解码;宿主自己转码 KTX2 的平台(原生打包)做不到这一点,这类贴图保持原尺寸。## 去掉一些效果diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.mdindex b168086f5..12b041dbf 100644--- a/sdk/etc/index.api.md+++ b/sdk/etc/index.api.md@@ -1088,7 +1088,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => voidsetRefCounter: (counter: AssetRefCounter) => voidsetRemoteRoot: (url: string | undefined) => voidsetTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void+setTextureStreamingStart: (edge: number | (() => number)) => voidtextureFormatReport: () => TextureFormatReport+textureFullSize: (handle: number) => { width: number; height: number; } | nulltextureResizable: (handle: number) => booleantexturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void@@ -3235,6 +3237,8 @@ positions: Float32Array<ArrayBufferLike>```height: numberpixels: Uint8Array<ArrayBufferLike>+sourceHeight: number | undefined+sourceWidth: number | undefinedwidth: number```@@ -8990,6 +8994,8 @@ skipped: AmbientSkip | null```byType: Map<string, Set<string>>dragonBones: SpineAssetRef[]+meshMaterials: Set<string>+meshTextures: Set<string>rawByType: Map<string, Set<string>>rawFor: Map<string, string>spines: SpineAssetRef[]@@ -11350,6 +11356,8 @@ number## TextureInfo — interface @experimental```+fullHeight: number | undefined+fullWidth: number | undefinedhandle: numberheight: numberwidth: number@@ -11370,6 +11378,7 @@ held: Set<number>inFlight: numbermoves: numberrefused: Set<number>+restoring: Set<number>textures: Map<number, StreamedTexture>static new (): TextureStreamingstatic prototype: TextureStreamingdiff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.mdindex 4a13b3383..05df402b0 100644--- a/sdk/etc/index.native.api.md+++ b/sdk/etc/index.native.api.md@@ -1088,7 +1088,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => voidsetRefCounter: (counter: AssetRefCounter) => voidsetRemoteRoot: (url: string | undefined) => voidsetTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void+setTextureStreamingStart: (edge: number | (() => number)) => voidtextureFormatReport: () => TextureFormatReport+textureFullSize: (handle: number) => { width: number; height: number; } | nulltextureResizable: (handle: number) => booleantexturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void@@ -3243,6 +3245,8 @@ positions: Float32Array<ArrayBufferLike>```height: numberpixels: Uint8Array<ArrayBufferLike>+sourceHeight: number | undefined+sourceWidth: number | undefinedwidth: number```@@ -9203,6 +9207,8 @@ skipped: AmbientSkip | null```byType: Map<string, Set<string>>dragonBones: SpineAssetRef[]+meshMaterials: Set<string>+meshTextures: Set<string>rawByType: Map<string, Set<string>>rawFor: Map<string, string>spines: SpineAssetRef[]@@ -11563,6 +11569,8 @@ number## TextureInfo — interface @experimental```+fullHeight: number | undefined+fullWidth: number | undefinedhandle: numberheight: numberwidth: number@@ -11583,6 +11591,7 @@ held: Set<number>inFlight: numbermoves: numberrefused: Set<number>+restoring: Set<number>textures: Map<number, StreamedTexture>static new (): TextureStreamingstatic prototype: TextureStreamingdiff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.mdindex 24d1995ac..0156c4d28 100644--- a/sdk/etc/index.node.api.md+++ b/sdk/etc/index.node.api.md@@ -1083,7 +1083,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => voidsetRefCounter: (counter: AssetRefCounter) => voidsetRemoteRoot: (url: string | undefined) => voidsetTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void+setTextureStreamingStart: (edge: number | (() => number)) => voidtextureFormatReport: () => TextureFormatReport+textureFullSize: (handle: number) => { width: number; height: number; } | nulltextureResizable: (handle: number) => booleantexturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void@@ -3230,6 +3232,8 @@ positions: Float32Array<ArrayBufferLike>```height: numberpixels: Uint8Array<ArrayBufferLike>+sourceHeight: number | undefined+sourceWidth: number | undefinedwidth: number```@@ -8985,6 +8989,8 @@ skipped: AmbientSkip | null```byType: Map<string, Set<string>>dragonBones: SpineAssetRef[]+meshMaterials: Set<string>+meshTextures: Set<string>rawByType: Map<string, Set<string>>rawFor: Map<string, string>spines: SpineAssetRef[]@@ -11345,6 +11351,8 @@ number## TextureInfo — interface @experimental```+fullHeight: number | undefined+fullWidth: number | undefinedhandle: numberheight: numberwidth: number@@ -11365,6 +11373,7 @@ held: Set<number>inFlight: numbermoves: numberrefused: Set<number>+restoring: Set<number>textures: Map<number, StreamedTexture>static new (): TextureStreamingstatic prototype: TextureStreamingdiff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.mdindex 645c382d3..e9971dac6 100644--- a/sdk/etc/minigame.api.md+++ b/sdk/etc/minigame.api.md@@ -1083,7 +1083,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => voidsetRefCounter: (counter: AssetRefCounter) => voidsetRemoteRoot: (url: string | undefined) => voidsetTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void+setTextureStreamingStart: (edge: number | (() => number)) => voidtextureFormatReport: () => TextureFormatReport+textureFullSize: (handle: number) => { width: number; height: number; } | nulltextureResizable: (handle: number) => booleantexturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void@@ -3230,6 +3232,8 @@ positions: Float32Array<ArrayBufferLike>```height: numberpixels: Uint8Array<ArrayBufferLike>+sourceHeight: number | undefined+sourceWidth: number | undefinedwidth: number```@@ -9280,6 +9284,8 @@ skipped: AmbientSkip | null```byType: Map<string, Set<string>>dragonBones: SpineAssetRef[]+meshMaterials: Set<string>+meshTextures: Set<string>rawByType: Map<string, Set<string>>rawFor: Map<string, string>spines: SpineAssetRef[]@@ -11640,6 +11646,8 @@ number## TextureInfo — interface @experimental```+fullHeight: number | undefined+fullWidth: number | undefinedhandle: numberheight: numberwidth: number@@ -11660,6 +11668,7 @@ held: Set<number>inFlight: numbermoves: numberrefused: Set<number>+restoring: Set<number>textures: Map<number, StreamedTexture>static new (): TextureStreamingstatic prototype: TextureStreamingdiff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.mdindex 7bfc66078..e53bbf6ea 100644--- a/sdk/etc/wechat.api.md+++ b/sdk/etc/wechat.api.md@@ -1083,7 +1083,9 @@ setManifest: (manifest: AddressableManifest | ManifestModel | null) => voidsetRefCounter: (counter: AssetRefCounter) => voidsetRemoteRoot: (url: string | undefined) => voidsetTextureImportSettingsResolver: (resolver: TextureImportSettingsResolver | null) => void+setTextureStreamingStart: (edge: number | (() => number)) => voidtextureFormatReport: () => TextureFormatReport+textureFullSize: (handle: number) => { width: number; height: number; } | nulltextureResizable: (handle: number) => booleantexturesAwaitingReupload: () => { handle: number; content: TextureContent; path: string; }[]trackSceneAssetReferences: (scene: SceneData, entities: ReadonlyMap<number, number>) => void@@ -3230,6 +3232,8 @@ positions: Float32Array<ArrayBufferLike>```height: numberpixels: Uint8Array<ArrayBufferLike>+sourceHeight: number | undefined+sourceWidth: number | undefinedwidth: number```@@ -8992,6 +8996,8 @@ skipped: AmbientSkip | null```byType: Map<string, Set<string>>dragonBones: SpineAssetRef[]+meshMaterials: Set<string>+meshTextures: Set<string>rawByType: Map<string, Set<string>>rawFor: Map<string, string>spines: SpineAssetRef[]@@ -11352,6 +11358,8 @@ number## TextureInfo — interface @experimental```+fullHeight: number | undefined+fullWidth: number | undefinedhandle: numberheight: numberwidth: number@@ -11372,6 +11380,7 @@ held: Set<number>inFlight: numbermoves: numberrefused: Set<number>+restoring: Set<number>textures: Map<number, StreamedTexture>static new (): TextureStreamingstatic prototype: TextureStreamingdiff --git a/sdk/src/asset/AssetLoader.ts b/sdk/src/asset/AssetLoader.tsindex 1cd028d96..54f52c2e2 100644--- a/sdk/src/asset/AssetLoader.ts+++ b/sdk/src/asset/AssetLoader.ts@@ -14,6 +14,9 @@ export interface TextureResult {handle: TextureHandle;width: number;height: number;+ /** The file's own size, where it was made smaller than that (texture streaming); absent when it is the file's size. */+ fullWidth?: number;+ fullHeight?: number;}/**@@ -122,6 +125,8 @@ export interface LoadContext {* this loader was handed.*/acquireTexture(path: string, flipY?: boolean): Promise<AssetLease<TextureResult>>;+ /** Say that material @p material binds texture @p texture, before it is acquired: a mesh material's textures may start small. */+ noteMaterialTexture?(material: string, texture: string): void;/*** The same for an asset of ANY type — a material instance acquiring the* parent it diffs against. The general door; {@link acquireTexture} is thediff --git a/sdk/src/asset/Assets.ts b/sdk/src/asset/Assets.tsindex f9f81bcd1..db344c65d 100644--- a/sdk/src/asset/Assets.ts+++ b/sdk/src/asset/Assets.ts@@ -387,6 +387,14 @@ function warnAmbiguousRelease_(key: string): void {+ 'which may not be the one the caller holds. Release it by its lease instead.');}+/**+ * What edge a mesh texture is first loaded at, asked as each one loads (texture streaming says, by the quality level): set once per+ * realm, before any Assets exists, since a scene can load before the first frame. 0 loads it whole.+ */+let textureStreamingStart: () => number = () => 0;+/** @internal */+export function setTextureStreamingStartProvider(provider: () => number): void { textureStreamingStart = provider; }+export class Assets {readonly backend: Backend;readonly catalog: Catalog;@@ -479,6 +487,12 @@ export class Assets {* load. The REVERSE of resolution — inspectors and tooling that only hold a* live World handle use it to name the asset (see {@link pathForHandle}). */private handleToPath_ = new Map<string, string>();+ /** Textures a mesh draws, by resolved path, directly or through a material: what texture streaming may load small to begin with. */+ private readonly meshTextures_ = new Set<string>();+ /** Materials a mesh draws with, by resolved path: their textures join {@link meshTextures_} as the material loads them. */+ private readonly meshMaterials_ = new Set<string>();+ /** The edge a mesh texture is first loaded at while texture streaming is on; 0 loads it whole. */+ private streamingStartEdge_: number | (() => number) = 0;/*** Geometry built at run time rather than loaded (a static batch), by handle. It has no file to be replayed@@ -557,6 +571,19 @@ export class Assets {return size;}+ /** The size of the file behind @p handle, whatever size it is held at; null for a texture no file of this realm's stands behind. */+ textureFullSize(handle: number): { width: number; height: number } | null {+ return this.textureLoader_.fullSize(handle);+ }++ /**+ * Load the textures meshes draw no larger than @p edge on a side to begin with (0 loads them whole): texture streaming grows each+ * to what its views need. A texture something else also draws is put back at its size by the streamer.+ */+ setTextureStreamingStart(edge: number | (() => number)): void {+ this.streamingStartEdge_ = edge;+ }+/** Whether {@link resizeTexture} can move @p handle between sizes. */textureResizable(handle: number): boolean {return this.textureLoader_.resizable(handle);@@ -729,6 +756,10 @@ export class Assets {const revived = this.reviveResidentTexture_(cacheKey);if (revived) return Promise.resolve(revived);this.textureLoader_.setPendingSettings(settings);+ // Asked as each texture loads: what decides it (the quality level) may arrive after whoever said how to ask.+ const own = typeof this.streamingStartEdge_ === 'function' ? this.streamingStartEdge_() : this.streamingStartEdge_;+ const start = own > 0 ? own : textureStreamingStart();+ if (start > 0 && this.meshTextures_.has(path)) this.textureLoader_.setPendingStartEdge(start);return flip? this.textureLoader_.load(path, this.getLoadContext_()): this.textureLoader_.loadRaw(path, this.getLoadContext_());@@ -778,6 +809,7 @@ export class Assets {rm.releaseTexture(handle);return null;}+ this.textureLoader_.adoptRevived(handle, cacheKey);return { handle, width: dims.width, height: dims.height };}@@ -1599,6 +1631,8 @@ export class Assets {): Promise<SceneAssetResult> {const missing: MissingAsset[] = [];const discovered = discoverSceneAssets(sceneData, (ref) => this.resolveRef(ref));+ for (const p of discovered.meshTextures) this.meshTextures_.add(p);+ for (const p of discovered.meshMaterials) this.meshMaterials_.add(p);for (const ref of discovered.unresolved) {missing.push({ ref, reason: 'unresolved' });}@@ -2998,6 +3032,11 @@ export class Assets {const { result, lease } = await self.acquireTextureVariant_(path, flipY !== false);return self.textureLease_(lease, result);},+ noteMaterialTexture(material: string, texture: string): void {+ if (self.meshMaterials_.has(self.resolveLoadPath_(material)) || self.meshMaterials_.has(material)) {+ self.meshTextures_.add(self.resolveLoadPath_(texture));+ }+ },async acquireAsset<T>(type: string, ref: string): Promise<AssetLease<T>> {return self.acquireTyped<T>(type, ref);},diff --git a/sdk/src/asset/discoverAssets.ts b/sdk/src/asset/discoverAssets.tsindex 4fc3cd377..f35c29a21 100644--- a/sdk/src/asset/discoverAssets.ts+++ b/sdk/src/asset/discoverAssets.ts@@ -66,6 +66,10 @@ export interface SceneAssetRefs {* assets don't silently turn into texture handle 0.*/unresolved: string[];+ /** Resolved paths of the textures a MeshRenderer draws directly (its texture and normal map): what texture streaming may start small. */+ meshTextures: Set<string>;+ /** Resolved paths of the materials a MeshRenderer draws with: their textures may start small too. */+ meshMaterials: Set<string>;}/** Turns a raw serialized ref into a concrete path (or null if unknown). */@@ -84,6 +88,8 @@ export function discoverSceneAssets(// belong to two runtimes.const skeletalKeys = new Set<string>();const unresolved: string[] = [];+ const meshTextures = new Set<string>();+ const meshMaterials = new Set<string>();const resolve = (raw: string): string | null => {const r = refResolver ? refResolver(raw) : raw;@@ -126,6 +132,14 @@ export function discoverSceneAssets(const data = compData.data as Record<string, unknown>;+ if (compData.type === 'MeshRenderer') {+ for (const [field, into] of [['texture', meshTextures], ['normalMap', meshTextures], ['material', meshMaterials]] as const) {+ const raw = data[field];+ const at = typeof raw === 'string' && raw ? (refResolver ? refResolver(raw) : raw) : null;+ if (at) into.add(at);+ }+ }+if (comp.discoverAssets) {for (const ref of comp.discoverAssets(data)) {if (typeof ref.path === 'string' && ref.path) {@@ -163,7 +177,7 @@ export function discoverSceneAssets(}}- return { byType, rawByType, rawFor, spines, dragonBones, unresolved };+ return { byType, rawByType, rawFor, spines, dragonBones, unresolved, meshTextures, meshMaterials };}export function getAssetPathsByType(refs: SceneAssetRefs, type: AssetFieldType): Set<string> {diff --git a/sdk/src/asset/imageDecode.ts b/sdk/src/asset/imageDecode.tsindex b4d797760..6feb7c2ce 100644--- a/sdk/src/asset/imageDecode.ts+++ b/sdk/src/asset/imageDecode.ts@@ -47,6 +47,9 @@ export interface DecodedPixels {width: number;height: number;pixels: Uint8Array;+ /** The image's own size, where it was decoded smaller than that; absent when it is the image's size. */+ sourceWidth?: number;+ sourceHeight?: number;}/**@@ -56,9 +59,13 @@ export interface DecodedPixels {* v=1=top for general textures, v=0=top for spine / bitmap fonts.*/export async function decodeImagePixels(src: ImageBitmapSource, maxEdge = 0): Promise<DecodedPixels> {- const bitmap = await shrinkBitmap(await decodeImageBitmap(src, false), maxEdge);+ const whole = await decodeImageBitmap(src, false);+ const sourceWidth = whole.width, sourceHeight = whole.height;+ const bitmap = await shrinkBitmap(whole, maxEdge);try {- return readImagePixels(bitmap, maxEdge);+ const pixels = readImagePixels(bitmap, maxEdge);+ // The bitmap was already made smaller, so the read cannot see what it was made from.+ return pixels.width < sourceWidth || pixels.height < sourceHeight ? { ...pixels, sourceWidth, sourceHeight } : pixels;} finally {bitmap.close?.();}@@ -107,7 +114,8 @@ export function readImagePixels(image: PlatformImage | ImageBitmap, maxEdge = 0)if (!ctx) throw new Error('imageDecode: 2D context unavailable for pixel decode');ctx.drawImage(image, 0, 0, width, height);const data = ctx.getImageData(0, 0, width, height);- return { width, height, pixels: new Uint8Array(data.data.buffer) };+ const pixels = new Uint8Array(data.data.buffer);+ return scale < 1 ? { width, height, pixels, sourceWidth: image.width, sourceHeight: image.height } : { width, height, pixels };}/**diff --git a/sdk/src/asset/loaders/MaterialAssetLoader.ts b/sdk/src/asset/loaders/MaterialAssetLoader.tsindex 4f3d0ca32..4c06e4f27 100644--- a/sdk/src/asset/loaders/MaterialAssetLoader.ts+++ b/sdk/src/asset/loaders/MaterialAssetLoader.ts@@ -55,6 +55,7 @@ export class MaterialAssetLoader implements AssetLoader<MaterialResult> {for (const [name, value] of Object.entries(data.properties)) {if (typeof value !== 'string') continue;const texPath = resolveDocumentRef(matPath, value);+ ctx.noteMaterialTexture?.(matPath, texPath);try {const tex = await ctx.acquireTexture(texPath);Material.setUniform(handle, name, Material.tex(tex.value.handle));diff --git a/sdk/src/asset/loaders/TextureLoader.ts b/sdk/src/asset/loaders/TextureLoader.tsindex 869d5feab..b7370befc 100644--- a/sdk/src/asset/loaders/TextureLoader.ts+++ b/sdk/src/asset/loaders/TextureLoader.ts@@ -32,7 +32,7 @@ export type TexturePixelDecoder = (path: string,flip: boolean,maxEdge?: number,-) => Promise<{ width: number; height: number; pixels: Uint8Array }>;+) => Promise<{ width: number; height: number; pixels: Uint8Array; sourceWidth?: number; sourceHeight?: number }>;/*** Where a host keeps its textures GPU-compressed: the KTX2 for `path`, `'pending'`@@ -132,7 +132,13 @@ export class TextureLoader implements AssetLoader<TextureResult> {*/private readonly formats_ = new TextureFormatLog();/** Where each texture this loader decoded from a file came from: what a resize decodes again. */- private readonly sources_ = new Map<number, { path: string; flip: boolean; settings?: TextureImportSettings }>();+ private readonly sources_ = new Map<number, { path: string; flip: boolean; settings?: TextureImportSettings; fullWidth: number; fullHeight: number }>();+ private pendingStartEdge_ = 0;+ /** How many loads began smaller than their file (texture streaming), and how many were asked to: a report, and what a check reads. */+ startedSmall = 0;+ askedSmall = 0;+ /** The same sources by residency key, kept past a release: a texture revived from the warm cache is the same file at the size it was left at. */+ private readonly sourcesByKey_ = new Map<string, { path: string; flip: boolean; settings?: TextureImportSettings; fullWidth: number; fullHeight: number }>();/** The decision the in-flight upload came to. Set by the one entry point that* chooses a path (decodeAndUpload_) and read by the one that records. */private lastDecision_: TextureUploadDecision = RAW_PAYLOAD_UPLOAD;@@ -237,11 +243,32 @@ export class TextureLoader implements AssetLoader<TextureResult> {/** Whether @p handle was decoded from a file by this loader, so it can be decoded again at another size. */resizable(handle: number): boolean { return this.sources_.has(handle); }+ /** The size of the file behind @p handle, whatever size it is held at; null for one this loader did not decode from a file. */+ fullSize(handle: number): { width: number; height: number } | null {+ const s = this.sources_.get(handle);+ return s ? { width: s.fullWidth, height: s.fullHeight } : null;+ }++ /** Decode the next load no larger than @p edge on a side (texture streaming's start), where the file's own size can still be known. */+ setPendingStartEdge(edge: number): void { this.pendingStartEdge_ = Math.max(0, edge); }++ /** A texture the warm cache gave back under @p key is the one this loader decoded under it: resizable again. */+ adoptRevived(handle: number, key: string): void {+ const source = this.sourcesByKey_.get(key);+ if (source) this.sources_.set(handle, source);+ }+private async loadWithFlip(path: string, ctx: LoadContext, flip: boolean, settings?: TextureImportSettings,): Promise<TextureResult> {- const result = await this.decodeAndUpload_(path, ctx, flip, settings);- this.sources_.set(result.handle, { path, flip, settings });+ const start = this.pendingStartEdge_;+ this.pendingStartEdge_ = 0;+ if (start > 0) this.askedSmall++;+ const result = await this.decodeAndUpload_(path, ctx, flip, settings, start);+ const source = { path, flip, settings, fullWidth: result.fullWidth ?? result.width, fullHeight: result.fullHeight ?? result.height };+ this.sources_.set(result.handle, source);+ if (result.fullWidth !== undefined) this.startedSmall++;+ this.sourcesByKey_.set(textureResidencyKey(path, flip), source);// Recorded HERE, once, for every path into the loader — the compressed// paths, the pixel decoder and the <img> upload alike. A record kept only// where a fallback happens cannot say that everything else was fine.@@ -277,11 +304,17 @@ export class TextureLoader implements AssetLoader<TextureResult> {if (this.pixelDecoder_) {const result = await this.pixelDecoder_(path, flip, edge);const handle = createTextureFromPixels(this.module_, result, TextureContent.Asset, flip, pixelParams(settings));- return { handle, width: result.width, height: result.height };+ // A decoder that made it smaller says what it was made from; one that could not (a host decoding whole) answers with the+ // file's own size, and the texture is simply whole.+ return result.sourceWidth !== undefined && result.sourceHeight !== undefined+ ? { handle, width: result.width, height: result.height, fullWidth: result.sourceWidth, fullHeight: result.sourceHeight }+ : { handle, width: result.width, height: result.height };}const url = ctx.backend.resolveUrl(ctx.catalog.getBuildPath(path));- const img = await this.loadImage(url, flip, edge, this.imageRevisions_.get(path));- return this.createTextureFromImage(img, flip, settings);+ const full = { width: 0, height: 0 };+ const img = await this.loadImage(url, flip, edge, this.imageRevisions_.get(path), full);+ const made = this.createTextureFromImage(img, flip, settings);+ return full.width > made.width || full.height > made.height ? { ...made, fullWidth: full.width, fullHeight: full.height } : made;}/**@@ -291,6 +324,19 @@ export class TextureLoader implements AssetLoader<TextureResult> {*/private async uploadKtx2_(path: string, bytes: Uint8Array, settings?: TextureImportSettings, maxEdge = 0,+ ): Promise<TextureResult> {+ const made = await this.uploadKtx2Levels_(path, bytes, settings, maxEdge);+ // The header's pixelWidth and pixelHeight (bytes 20 and 24): the file's size, whatever level the upload started at.+ if (bytes.byteLength >= 28) {+ const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);+ const fullWidth = view.getUint32(20, true), fullHeight = view.getUint32(24, true);+ if (fullWidth > made.width || fullHeight > made.height) return { ...made, fullWidth, fullHeight };+ }+ return made;+ }++ private async uploadKtx2Levels_(+ path: string, bytes: Uint8Array, settings?: TextureImportSettings, maxEdge = 0,): Promise<TextureResult> {if (!isKtx2(bytes)) throw new Error(`TextureLoader: ${path} is not a KTX2 file`);// Native (embedded Dawn): the ResourceManager transcodes the KTX2 with the@@ -377,11 +423,12 @@ export class TextureLoader implements AssetLoader<TextureResult> {* on it uploads every texture upside-down. The raw `<img>` fallback keeps the* flag (it works for element/pixel sources) — see {@link createTextureFromImage}.*/- private loadImage(src: string, flip: boolean, maxEdge = 0, revision?: number): Promise<PlatformImage | ImageBitmap> {+ private loadImage(src: string, flip: boolean, maxEdge = 0, revision?: number, full?: { width: number; height: number }): Promise<PlatformImage | ImageBitmap> {return new Promise((resolve, reject) => {const img = platformCreateImage();img.crossOrigin = 'anonymous';img.onload = async () => {+ if (full) { full.width = img.width; full.height = img.height; }if (typeof createImageBitmap !== 'undefined') {try {resolve(await shrinkBitmap(await decodeImageBitmap(img, flip), maxEdge));diff --git a/sdk/src/render/textureStreaming.ts b/sdk/src/render/textureStreaming.tsindex 38cd70705..0f34952cc 100644--- a/sdk/src/render/textureStreaming.ts+++ b/sdk/src/render/textureStreaming.ts@@ -22,6 +22,7 @@ import { defineResource } from '../ecs/resource';import { getComponent } from '../ecs/component';import { playModeOnly } from '../ecs/env';import { Assets } from '../asset/AssetPlugin';+import { setTextureStreamingStartProvider } from '../asset/Assets';import { engineApi } from '../ecs/bridge/engineApi';import { getTextureDimensions } from '../wasm/resourceManager';import { platformNow } from '../platform';@@ -134,6 +135,8 @@ export class TextureStreaming {readonly held = new Set<number>();/** Textures that came back larger than asked (a host that cannot cut a file at a level): left alone from then on. */readonly refused = new Set<number>();+ /** Textures being put back at their size because something other than a mesh draws them. */+ readonly restoring = new Set<number>();moves = 0;failures = 0;inFlight = 0;@@ -165,7 +168,16 @@ export class TextureStreamingPlugin implements Plugin {app.insertResource(TextureStreamingRes, state);const world = app.world;let last = 0, lastHeld = -Infinity;- const MAX_IN_FLIGHT = 2;+ const MAX_IN_FLIGHT = 4;+ /** The edge a mesh texture starts at while streaming is on. */+ const START_EDGE = 256;+ // Asked as each texture loads, not first in the frame loop: a scene may start loading before the first frame does, and the+ // quality level may be settled after this plugin is built.+ const startEdge = (): number => {+ const quality = app.hasResource(Quality) ? app.getResource(Quality) : null;+ return quality?.profile.textureStreaming && engineApi(app)?.renderer_takeTextureWants ? START_EDGE : 0;+ };+ setTextureStreamingStartProvider(startEdge);const parse = (raw: string): Map<number, number> => {const out = new Map<number, number>();@@ -207,15 +219,26 @@ export class TextureStreamingPlugin implements Plugin {let entry = state.textures.get(handle);if (!entry) {if (!assets.textureResizable(handle)) continue;- const size = getTextureDimensions(handle);+ const size = assets.textureFullSize(handle) ?? getTextureDimensions(handle);if (!size || size.width <= 0 || size.height <= 0) continue;- entry = { handle, fullWidth: size.width, fullHeight: size.height, edge: Math.max(size.width, size.height), wanted: 0, seenAt: now, smallerSince: -1 };+ const held = getTextureDimensions(handle);+ entry = { handle, fullWidth: size.width, fullHeight: size.height, edge: held ? Math.max(held.width, held.height) : Math.max(size.width, size.height), wanted: 0, seenAt: now, smallerSince: -1 };state.textures.set(handle, entry);}entry.wanted = entry.seenAt === now ? Math.max(entry.wanted, side) : side;entry.seenAt = now;}- for (const handle of state.held) state.textures.delete(handle);+ for (const handle of state.held) {+ state.textures.delete(handle);+ // Loaded small for a mesh and drawn by something no view measures: put back at its size.+ const full = assets.textureFullSize(handle), now_ = getTextureDimensions(handle);+ if (full && now_ && (now_.width < full.width || now_.height < full.height) && state.inFlight < MAX_IN_FLIGHT && !state.restoring.has(handle)) {+ state.restoring.add(handle);+ state.inFlight++;+ void assets.resizeTexture(handle, 0).catch(() => { state.failures++; })+ .finally(() => { state.inFlight--; state.restoring.delete(handle); });+ }+ }const options: TextureStreamingOptions = {...TEXTURE_STREAMING_DEFAULTS,diff --git a/sdk/tests/texture-streaming-start.test.ts b/sdk/tests/texture-streaming-start.test.tsnew file mode 100644index 000000000..85539537a--- /dev/null+++ b/sdk/tests/texture-streaming-start.test.ts@@ -0,0 +1,44 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+import { describe, expect, it } from 'vitest';+import { levelsFrom } from '../src/asset/compressed';+import { discoverSceneAssets } from '../src/asset/discoverAssets';++const level = (side: number) => ({ width: side, height: side / 2, data: new Uint8Array(side) });++describe('a KTX2 chain cut at a level', () => {+ const chain = { width: 1024, height: 512, data: new Uint8Array(1024), levels: [level(1024), level(512), level(256), level(128), level(64)] };++ it('starts at the first level no larger than asked, and keeps every level below it', () => {+ const cut = levelsFrom(chain, 300);+ expect(cut.width).toBe(256);+ expect(cut.height).toBe(128);+ expect(cut.levels!.map((l) => l.width)).toEqual([256, 128, 64]);+ expect(cut.data).toBe(chain.levels[2]!.data);+ });++ it('leaves a chain whole when nothing is asked, or when it is small enough already', () => {+ expect(levelsFrom(chain, 0)).toBe(chain);+ expect(levelsFrom(chain, 4096)).toBe(chain);+ });++ it('keeps the smallest level when every level is larger than asked', () => {+ expect(levelsFrom(chain, 16).width).toBe(64);+ });+});++describe('which textures a scene\'s meshes draw', () => {+ it('are a MeshRenderer\'s texture and normal map, and the materials it draws with, and not a sprite\'s', () => {+ const scene = {+ version: '1.0', name: 'S', entities: [+ { id: 1, name: 'Wall', parent: null, children: [], visible: true, components: [+ { type: 'MeshRenderer', data: { mesh: 'builtin:cube', texture: 'wall.png', normalMap: 'wall_n.png', material: 'wall.esmaterial' } }] },+ { id: 2, name: 'Sign', parent: null, children: [], visible: true, components: [+ { type: 'Sprite', data: { texture: 'sign.png' } }] },+ ],+ };+ const found = discoverSceneAssets(scene as never, (ref) => `assets/${ref}`);+ expect([...found.meshTextures].sort()).toEqual(['assets/wall.png', 'assets/wall_n.png']);+ expect([...found.meshMaterials]).toEqual(['assets/wall.esmaterial']);+ });+});