3D engine that AI agents drive
Fall back to the full world when the first view fails, and skip the prepared draw before a world has drawn (Part of #884)
Nguyen Xuan Anfae79a619cc7
diff --git a/docs/hero-loading.md b/docs/hero-loading.mdindex 8d783d38..dc7dba67 100644--- a/docs/hero-loading.md+++ b/docs/hero-loading.md@@ -8,10 +8,12 @@ The hero on the home page plays a flight over the Port showcase world. The world2. When the poster has settled (or after 5 s), `index.html` starts the first view: `scene.json`, `scene.first.glb` (about 2 MB) and `sky.preview.hdr`.3. When those have settled, it starts the full world: `scene.glb`, `scene.rest.glb` and `sky.hdr`.4. `fetchWorld(url, { firstView: true })` in `src/hero-lab/loop/world.js` resolves as soon as the first view is in. It holds a `detail` promise for the full world.-5. The world is built from the first view and draws. `completeDetail` (`src/hero-lab/loop/detail.js`) then waits for `detail`, parses it and calls `world.upgrade(detail)`. The upgrade uploads the textures and the sky while the first view keeps drawing. Only the swap itself (the sky prefilter, one `replaceParts` and a prepared draw) holds the draws.+5. The world is built from the first view and draws. `completeDetail` (`src/hero-lab/loop/detail.js`) then waits for `detail`, parses it and calls `world.upgrade(detail)`. The upgrade uploads the textures and the sky while the first view keeps drawing. Only the swap itself (the sky prefilter, one `replaceParts` and a prepared draw) holds the draws: about 140 ms on a desktop and a phone viewport (`swapHeldMs` of `scripts/measure-hero-load.mjs`, GPU shared with other jobs). Feeding `replaceParts` in slices was tried and doubled the hold (swap 0.8 s to 1.5 s) without shortening the longest gap between frames (about 110 ms either way).A link faster than 20 Mbit/s (measured from the resource timing of the first view's own download) skips the first view and builds the full world straight away.+A first view that is missing, corrupt or fails to build is replaced by the full world (`fetchFirstViewFiles` in `world.js`, `buildWithFallback` in `detail.js`). A world that was never drawn, such as a page opened in a background tab, swaps without a prepared draw; its first real frame does that work.+A full world that fails to arrive or to build is asked for again after 4, 10 and 20 s; after that the first view stays. A world that is freed in the middle of a swap ends the swap quietly.## The first view@@ -22,7 +24,7 @@ A full world that fails to arrive or to build is asked for again after 4, 10 and- It splits the rest into six distance bands (edges at 100, 170, 260, 400 and 650 m) and simplifies each band to a screen error of 3 px up to 400 m and 6 px beyond, locking the borders so the bands do not crack apart.- Textures are 512 px for materials seen within 400 m and 128 px for the rest. Materials that are never seen are dropped.-The options (`--near-pixels`, `--far-pixels` and others) are read by `parseArguments` in the script. The geometry the first view lacks lies outside the first 5 s of the flight, so `src/hero-lab/loop/hold.js` slows the flight from 2.5 s while the full world is pending and stops it at 5 s, until the full world is in (or has been given up on).+The options (`--near-pixels`, `--far-pixels` and others) are read by `parseArguments` in the script. The geometry the first view lacks lies outside the first 5 s of the flight, so `src/hero-lab/loop/hold.js` slows the flight from 2.5 s while the full world is pending and stops it at 5 s, until the full world is in (or has been given up on). The cost is that on a very slow link the frame is almost still for most of the wait (at 1.6 Mbit/s the first view appears at about 14 s and the swap lands at about 64 s); the alternative is holes at the edges of the picture, and the page before this change showed nothing at all for that time.## Measuringdiff --git a/scripts/measure-hero-load.mjs b/scripts/measure-hero-load.mjsindex fd8660f1..5e9e15e2 100644--- a/scripts/measure-hero-load.mjs+++ b/scripts/measure-hero-load.mjs@@ -9,6 +9,7 @@// up, with brotli for text, as a CDN would. Network time comes from the DevTools throttle, which GPU load does not change.// posterEndMs responseEnd of the poster image// firstFrameMs the first draw on a world canvas (the first live 3D frame)+// swapMs, swapHeldMs, swapLongestGapMs how long the swap to the full world took, how long its draws were held (sky prefilter, replaceParts, prepared draw), and the longest gap between animation frames around it// detailMs when the full world is drawn (the 'detailed' loop event); without staged loading it equals firstFrameMs// bytesBeforeFirstFrame, bytesTotal transfer bytes (compressed size on the wire) up to the first live frame, and up to detailMsimport { createServer } from 'node:http';@@ -87,6 +88,15 @@ async function startServer(root) {// first live 3D frame a visitor sees. The poster fades out only after that.function installFrameTimer(marker) {window.__firstWorldDraw = null;+ // Gaps between animation frames over 50 ms: the main thread was blocked. The swap to the full world should leave none that is long.+ window.__frameGaps = [];+ let previousFrameAt = performance.now();+ const watchFrameGaps = now => {+ if (now - previousFrameAt > 50) window.__frameGaps.push([Math.round(previousFrameAt), Math.round(now - previousFrameAt)]);+ previousFrameAt = now;+ requestAnimationFrame(watchFrameGaps);+ };+ requestAnimationFrame(watchFrameGaps);new MutationObserver(() => {if (window.__firstWorldDraw === null && document.querySelector('#stage')?.dataset.live === 'true') {window.__firstWorldDraw = performance.now();@@ -143,7 +153,9 @@ async function measureOnce(browser, options, viewportName, conditionName) {const events = window.__loop?.events ?? [];const detailed = events.find(event => event.kind === 'detailed');const failed = events.find(event => event.kind === 'failed');- return { first: window.__firstWorldDraw, detailed: detailed?.at ?? null, failed: failed?.message ?? null, state: window.__loop?.detail?.() ?? null, mode: window.__heroMode };+ const swapEnd = detailed?.at ?? Infinity;+ const swapGaps = (window.__frameGaps ?? []).filter(([at, gap]) => at + gap > swapEnd - (detailed?.ms ?? 0) - 100 && at < swapEnd + 100);+ return { first: window.__firstWorldDraw, detailed: detailed?.at ?? null, swapMs: detailed?.ms ?? null, swapHeldMs: (window.__phases ?? []).filter(phase => phase[0] === "swap").reduce((total, phase) => total + phase[3] - phase[2], 0), swapLongestGapMs: swapGaps.length ? Math.max(...swapGaps.map(([, gap]) => gap)) : 0, failed: failed?.message ?? null, state: window.__loop?.detail?.() ?? null, mode: window.__heroMode };});const deadline = Date.now() + options.timeoutSeconds * 1000;let status = await finished();@@ -164,6 +176,9 @@ async function measureOnce(browser, options, viewportName, conditionName) {posterEndMs: poster,firstFrameMs,detailMs,+ swapMs: status.swapMs,+ swapLongestGapMs: status.swapLongestGapMs,+ swapHeldMs: status.swapHeldMs,bytesBeforeFirstFrame: bytes.atFirstFrame,bytesAtDetail: detailMs === null ? null : bytesAt(bytes, detailMs),errors: [...errors, ...(status.failed ? [status.failed] : [])],@@ -193,6 +208,9 @@ async function runMatrix(browser, options) {firstFrameMs: median(runs.map(run => run.firstFrameMs)),firstFrameRangeMs: [Math.min(...runs.map(run => run.firstFrameMs ?? Infinity)), Math.max(...runs.map(run => run.firstFrameMs ?? 0))],detailMs: median(runs.map(run => run.detailMs)),+ swapMs: median(runs.map(run => run.swapMs)),+ swapHeldMs: median(runs.map(run => run.swapHeldMs)),+ swapLongestGapMs: median(runs.map(run => run.swapLongestGapMs)),detailRangeMs: [Math.min(...runs.map(run => run.detailMs ?? Infinity)), Math.max(...runs.map(run => run.detailMs ?? 0))],bytesBeforeFirstFrame: median(runs.map(run => run.bytesBeforeFirstFrame)),bytesAtDetail: median(runs.map(run => run.bytesAtDetail)),diff --git a/src/hero-lab/hero-loop.js b/src/hero-lab/hero-loop.jsindex 364ff49f..996871e8 100644--- a/src/hero-lab/hero-loop.js+++ b/src/hero-lab/hero-loop.js@@ -10,7 +10,7 @@ import { createFlight } from './loop/camera.js';import { buildWorld, fetchDetail, fetchWorld } from './loop/world.js';import { createPlayOnDemand } from './loop/play-on-demand.js';import { advanceHold } from './loop/hold.js';-import { completeDetail } from './loop/detail.js';+import { buildWithFallback, completeDetail } from './loop/detail.js';const parameters = new URLSearchParams(location.search);const captureMode = parameters.has('capture');@@ -97,7 +97,7 @@ function prepare(index, { immediately = false, firstView = false } = {}) {await entry.permission;if (entry.skipped) return;const started = performance.now();- const world = await buildFrom(download);+ const { world, builtFrom } = await buildWithFallback(download, buildFrom);await new Promise(resolve => setTimeout(resolve, 0));// The world joins the loop only after its first draw has linked every program and allocated every target, so the fade never waits on the driver.const warmStarted = performance.now();@@ -108,8 +108,8 @@ function prepare(index, { immediately = false, firstView = false } = {}) {return;}entry.world = world;- if (download.isFirstView) entry.detail = followDetail(entry, download);- note('built', { scene: SCENES[index].id, ms: Math.round(performance.now() - started), firstView: Boolean(download.isFirstView) });+ if (builtFrom.isFirstView) entry.detail = followDetail(entry, builtFrom);+ note('built', { scene: SCENES[index].id, ms: Math.round(performance.now() - started), firstView: Boolean(builtFrom.isFirstView) });loopState.peakWorlds = Math.max(loopState.peakWorlds, loopState.worldsAlive());}).catch(error => {diff --git a/src/hero-lab/loop/detail.js b/src/hero-lab/loop/detail.jsindex 1658772c..ae2dae1f 100644--- a/src/hero-lab/loop/detail.js+++ b/src/hero-lab/loop/detail.js@@ -41,3 +41,15 @@ export async function completeDetail({}}}++// Builds a downloaded world. A first view that does not parse or upload is not the end of the hero: the full world, which is already+// on its way, is built instead. Resolves with the world and the download it was built from.+export async function buildWithFallback(download, build) {+ try {+ return { world: await build(download), builtFrom: download };+ } catch (error) {+ if (!download.isFirstView) throw error;+ const fullDownload = await download.detail;+ return { world: await build(fullDownload), builtFrom: fullDownload };+ }+}diff --git a/src/hero-lab/loop/world.js b/src/hero-lab/loop/world.jsindex 9f2c1d44..5f4d94b8 100644--- a/src/hero-lab/loop/world.js+++ b/src/hero-lab/loop/world.js@@ -137,17 +137,58 @@ export function measureLinkMegabits(url) {return (entry.encodedBodySize * 8) / seconds / 1e6;}+async function fetchFirstViewFiles(description) {+ const paths = [description.glbFirstView];+ try {+ const [models, sky] = await Promise.all([Promise.all(paths.map(fetchBlob)), fetchBlob(description.hdriPreview)]);+ return { paths, models, sky };+ } catch {+ return null;+ }+}+export async function fetchWorld(descriptionUrl, { firstView = false, measureLink = measureLinkMegabits } = {}) {const description = await readDescription(descriptionUrl);if (!firstView || !hasFirstView(description)) return fetchDetail(description);- const paths = [description.glbFirstView];- const [models, sky] = await Promise.all([Promise.all(paths.map(fetchBlob)), fetchBlob(description.hdriPreview)]);+ const firstViewFiles = await fetchFirstViewFiles(description);+ // A first view that is missing or corrupt must not cost the visitor the hero: the full world, which the page serves anyway, takes its place.+ if (!firstViewFiles) return fetchDetail(description);+ const { paths, models, sky } = firstViewFiles;if ((measureLink(description.glbFirstView) ?? 0) > FAST_LINK_MEGABITS) return fetchDetail(description);const detail = fetchDetail(description);detail.catch(() => {});return { description, paths, models, sky, bytes: bytesOf(models, sky), isFirstView: true, detail };}+// The part of a swap that holds the draws: the sky prefilter, then the parts. Resolves false when the world was freed on the way.+// One replaceParts call, not slices: the draws are held either way, and measured on the port the slices made the hold twice as long.+export async function replaceRendererWorld(renderer, parts, environmentMap, isFreed) {+ const prefilterStart = performance.now();+ await renderer.prepareEnvironmentInSlices(environmentMap, createSlicer(SLICE_MILLISECONDS));+ recordSwapPhase('prefilter', prefilterStart);+ if (isFreed()) return false;+ const replaceStart = performance.now();+ renderer.replaceParts(parts);+ recordSwapPhase('replaceParts', replaceStart);+ return !isFreed();+}++// Leaves the length of one phase of the swap in window.__phases, next to the laps of buildWorld, for the load measurement.+function recordSwapPhase(name, start) {+ const phases = globalThis.window?.__phases;+ if (phases) phases.push(['swap', name, Math.round(start), Math.round(performance.now())]);+}++// The first draw of a swapped world at the pose of its last frame, so the next frame does not link or allocate. A world that was+// never drawn (a page opened in a background tab) has no pose, and its first real frame does that work.+export async function prepareFrameAtLastPose(renderer, lastDraw, frameFor) {+ if (!lastDraw) return false;+ const prepareStart = performance.now();+ await renderer.prepareFrame(frameFor(lastDraw.camera, lastDraw.seconds));+ recordSwapPhase('prepareFrame', prepareStart);+ return true;+}+async function parseFiles(download, sharedTextures) {const { description, paths, models } = download;const files = [];@@ -272,9 +313,8 @@ export async function buildWorld(download, canvas, overrides = {}) {if (disposed) return false;drawsHeld = true;try {- await renderer.prepareEnvironmentInSlices(fullMap, createSlicer(SLICE_MILLISECONDS));- if (disposed) return false;- renderer.replaceParts(fullWhole.parts);+ const replaced = await replaceRendererWorld(renderer, fullWhole.parts, fullMap, () => disposed);+ if (!replaced) return false;detachReflection();detachReflection = attachWaterReflection(renderer, fullWhole.surface);effects = fullEffects;@@ -282,8 +322,7 @@ export async function buildWorld(download, canvas, overrides = {}) {animationOrigin = null;upgradedMap = fullMap;isFirstView = false;- const pose = lastDraw?.camera ?? { eye: frame.eye, target: frame.target, fov: frame.fov };- await renderer.prepareFrame(frameFor(pose, lastDraw?.seconds ?? 0));+ await prepareFrameAtLastPose(renderer, lastDraw, frameFor);return !disposed;} catch (error) {// A world freed in the middle of its swap has no renderer left to finish on: that ends the swap, it does not fail it.diff --git a/tests/hero-progressive.test.js b/tests/hero-progressive.test.jsindex 9cfc9c80..5745c3d9 100644--- a/tests/hero-progressive.test.js+++ b/tests/hero-progressive.test.js@@ -1,9 +1,9 @@import assert from 'node:assert/strict';import { existsSync, readFileSync, statSync } from 'node:fs';import { afterEach, beforeEach, test } from 'node:test';-import { completeDetail } from '../src/hero-lab/loop/detail.js';+import { buildWithFallback, completeDetail } from '../src/hero-lab/loop/detail.js';import { advanceHold, FIRST_VIEW_COVERS_SECONDS, HOLD_FROM_SECONDS, HOLD_RATE, holdTargetRate } from '../src/hero-lab/loop/hold.js';-import { FAST_LINK_MEGABITS, fetchWorld, hasFirstView } from '../src/hero-lab/loop/world.js';+import { FAST_LINK_MEGABITS, fetchWorld, hasFirstView, prepareFrameAtLastPose, replaceRendererWorld } from '../src/hero-lab/loop/world.js';const DESCRIPTION_URL = '/showcase/port/scene.json';const description = {@@ -248,3 +248,65 @@ test('the first view the port description names exists and stays within its byteassert.ok(firstViewBytes <= 2_100_000, `first view is ${firstViewBytes} bytes`);assert.ok(existsSync(new URL(`../public${published.hdriPreview}`, import.meta.url)));});++test('fetchWorld falls back to the full world when the first view model or sky is missing', async () => {+ for (const missing of [description.glbFirstView, description.hdriPreview]) {+ installFetch();+ const original = globalThis.fetch;+ globalThis.fetch = url => (url === missing ? Promise.resolve({ ok: false, status: 404 }) : original(url));+ const download = await fetchWorld(DESCRIPTION_URL, { firstView: true, measureLink: slowLink });+ assert.equal(download.isFirstView, undefined, missing);+ assert.deepEqual(download.paths, [description.glb, ...description.glbRest], missing);+ }+});++test('a first view that fails to build is replaced by the full world, and a full world that fails to build is not hidden', async () => {+ const firstViewDownload = { isFirstView: true, detail: Promise.resolve({ id: 'full' }) };+ const built = [];+ const build = async download => {+ built.push(download);+ if (download.isFirstView) throw new Error('first view does not parse');+ return { world: download.id };+ };+ const result = await buildWithFallback(firstViewDownload, build);+ assert.deepEqual(result.world, { world: 'full' });+ assert.equal(result.builtFrom.isFirstView, undefined);+ assert.equal(built.length, 2);+ await assert.rejects(buildWithFallback({ id: 'full' }, async () => { throw new Error('full does not parse'); }), /full does not parse/);+});++function fakeRenderer(calls) {+ return {+ prepareEnvironmentInSlices: async () => { calls.push('environment'); },+ replaceParts: () => { calls.push('replaceParts'); },+ prepareFrame: async () => { calls.push('prepareFrame'); },+ };+}++const fakeParts = [{ geometry: { indices: [0, 1, 2] } }, { geometry: { indices: [0, 1, 2] } }];++test('the swap prefilters the sky before it replaces the parts, and stops when the world is freed on the way', async () => {+ const calls = [];+ assert.equal(await replaceRendererWorld(fakeRenderer(calls), fakeParts, {}, () => false), true);+ assert.equal(calls[0], 'environment');+ assert.ok(calls.length > 1 && calls.slice(1).every(name => name === 'replaceParts'));+ const freedCalls = [];+ assert.equal(await replaceRendererWorld(fakeRenderer(freedCalls), fakeParts, {}, () => true), false);+ assert.deepEqual(freedCalls, ['environment']);+});++test('the swap prepares a draw at the last pose, and none when the world was never drawn', async () => {+ const calls = [];+ const frames = [];+ const frameFor = (camera, seconds) => {+ frames.push([camera, seconds]);+ return { camera };+ };+ assert.equal(await prepareFrameAtLastPose(fakeRenderer(calls), null, frameFor), false);+ assert.deepEqual(calls, []);+ assert.equal(frames.length, 0, 'no pose, so no frame is built from a missing eye');+ const camera = { eye: [1, 2, 3], target: [0, 0, 0], fov: 40 };+ assert.equal(await prepareFrameAtLastPose(fakeRenderer(calls), { camera, seconds: 7 }, frameFor), true);+ assert.deepEqual(calls, ['prepareFrame']);+ assert.deepEqual(frames, [[camera, 7]]);+});