Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Add recipe crowds, a far pose rate, and maps and morph targets on crowd parts (part of #323)

Nguyen Xuan An05e0066274d7
diff --git a/changes/323.md b/changes/323.md
index 7d30eedb..138fd712 100644
--- a/changes/323.md
+++ b/changes/323.md
@@ -1 +1,2 @@
-- Add `67js/addons/crowd-skinning` (issue #323): `createCrowdRig`, `bakeCrowdPalette` and `createCrowdParts` bake a character's clip into a joint-matrix texture once and draw every instance of it as one instanced batch per material, each instance sampling its own clip time, so draw calls stop growing with the number of people. Instances are culled by a bound that covers the whole clip, and `createCrowdLods` gives coarser levels skinned with the same joints. The renderer gains small hooks for it (a `crowd` part field, `crowdJoints` and `crowdWeights` geometry attributes, a `crowdEnabled` shader uniform); skinned parts without `crowd` behave as before.
+- Add `67js/addons/crowd-skinning` (issue #323): `createCrowdRig`, `bakeCrowdPalette` and `createCrowdParts` bake a character's clip into a joint-matrix texture once and draw every instance of it as one instanced batch per material, each instance sampling its own clip time, so draw calls stop growing with the number of people. Instances are culled by a bound that covers the whole clip, and `createCrowdLods` gives coarser levels skinned with the same joints. The renderer gains small hooks for it (a `crowd` part field, a `crowdPacked` geometry attribute, a `crowdEnabled` shader uniform); skinned parts without `crowd` behave as before.
+- Add recipe crowds, a lower pose rate for far instances, and maps and morph targets on crowd parts (part of #323). An asset placement with `crowd: { clip, count, seed, layout, spacing, ... }` fills a group node with up to 1000 walking copies of one catalog character in one batch per material, and `scene_inspect` lists it under `assets` with `kind: "crowd"`. `farDistance` and `farRate` (recipe) or `farPose` and `setTimes(times, eye)` (addon) make far instances hold each pose for `1 / rate` seconds without breaking seek equals play. `createCrowdRig` keeps UVs so materials with maps draw on a crowd, and `{ morphs: true }` keeps morph targets on their own part. The crowd shader now reads its joints and weights packed in the colour attribute, which frees the UV attribute and makes the core bundle slightly smaller. `docs/retarget.md` has the measured RTX 3060 numbers: 500 `RobotExpressive` instances at a median 5.75 ms (8.27 ms in a later run on a shared card) with 7 draw calls.
diff --git a/changes/323.size.md b/changes/323.size.md
new file mode 100644
index 00000000..f42b6209
--- /dev/null
+++ b/changes/323.size.md
@@ -0,0 +1,5 @@
+- entry:crowd-skinning: the shader unpacks joints and weights from one attribute, and the rig keeps UVs, morph parts and packs the influences (about 0.4 KB gzip), which is what lets crowds carry maps and morph targets.
+- entry:agent-scene+lazy: the crowd recipe module and the crowd addon it uses (retarget.js, crowd-skinning.js) load on demand with the "crowd" marker, about 7 KB gzip, and only for a recipe that asks for a crowd.
+- entry:agent-asset+lazy: the same on-demand crowd module, reached from the asset placement code.
+- scene:agent-scene+lazy: the same on-demand crowd module, which every agent scene with all lazy modules loaded carries.
+- result:scene_capabilities: the crowd note (about 0.9 KB) is how an agent learns the asset.crowd field, which no schema carries; it follows the vegetation note.
diff --git a/docs/agent-scene.md b/docs/agent-scene.md
index dc8fb008..4f90c6d1 100644
--- a/docs/agent-scene.md
+++ b/docs/agent-scene.md
@@ -287,6 +287,17 @@ A node with a `vegetation` field is a group that grows trees from the addon in [
- Materials with tag `foliage` or `grass` sway. `environment.wind` is `{ "strength": 0.3, "direction": 0 }` (strength 0 to 4; direction in degrees, 0 toward +x and 90 toward +z, turning the sway and the grass gusts). The wind clock is the render time, so a render job or `scene_review` `rendererTime` gives the same frame at the same time.
- At most 1000 instances (a scatter whose area times density is above 4000 is refused first), and `maxNodes` for the generated parts. `scene_inspect` adds `vegetation`: per node the instances, culled count, triangles, and per species the instances and the count at each level.
+### Crowds
+
+An asset placement with a `crowd` field fills its group node with `count` (1 to 1000) copies of one catalog character playing one clip, each at its own time offset and rate, drawn as one batch per material. Options and measurements are in [Crowds](https://67js.codepawl.com/retarget.md).
+
+```json
+{ "id": "people", "asset": { "id": "<catalog asset id>", "crowd": { "clip": 3, "count": 200, "seed": 7, "spacing": 2, "farDistance": 25, "farRate": 6 } } }
+```
+
+- `clip` is a clip index, `layout` is `grid` or `scatter`, and the pose is a pure function of scene time, `seed` and the recipe camera, so a render at a time equals the frame reached by playing. People farther than `farDistance` from the recipe camera hold each pose for `1 / farRate` seconds.
+- The crowd takes one decoded copy of the asset, the generated part nodes (count times materials) count against `maxNodes`, and `scene_inspect` lists it under `assets` with `kind: "crowd"`, `instances`, `batches`, `frames`, `paletteBytes` and `boundsRadius`.
+
### Relations between parts
A mesh node can list `relations` that place it against other mesh nodes, so a resized body keeps its handle where it
diff --git a/docs/retarget.md b/docs/retarget.md
index c5b500bb..9e6cfa56 100644
--- a/docs/retarget.md
+++ b/docs/retarget.md
@@ -117,9 +117,28 @@ any seek time equals the one reached by playing. `crowdVertexAt(palette, geometr
outside the frustum are not drawn, and an instance far enough away uses a coarser level.
- **Levels** come from `createCrowdLods(geometry, { cells })` (on by default in `createCrowdParts`, `lods: false` turns them off): vertices that
share a grid cell and a dominant joint merge, and keep the joints and weights of one of them, so every level skins with the same palette.
-- Limits: morph targets are not applied; materials with maps are not supported (the geometry's UV and colour attributes carry joints and weights);
- an `assemblyAmount` does not move a crowd; a clip is baked once, so another clip is another palette and another batch. Lowering the pose rate
- for far instances is not done yet.
+- **Maps and morph targets.** `createCrowdRig` keeps UVs, so materials with maps draw on a crowd (the joints and weights travel packed in the colour
+ attribute, 12 bits of joint and weight each). `createCrowdRig(parts, { morphs: true })` keeps a part with morph targets as its own rig part:
+ it draws with the weights it has when the rig is made, the same for every instance, at full detail (no levels, and no culling of that part).
+ Without the option morph targets are not applied.
+- **Far pose rate.** `createCrowdParts({ ..., farPose: { distance, rate } })` and `setTimes(times, eye)`: an instance more than `distance` from
+ `eye` holds each pose for `1 / rate` seconds. The pose is still a pure function of the clip time and the eye, so a seek equals playing.
+- Limits: at most 4096 joints in a rig; an `assemblyAmount` does not move a crowd; a clip is baked once, so another clip is another palette and
+ another batch.
+
+### Recipe crowds
+
+An agent asks for a crowd in a recipe (`scene_create`, `scene_apply`) with the `crowd` field of an asset placement:
+
+```json
+{ "id": "people", "asset": { "id": "<catalog asset id>", "crowd": { "clip": 3, "count": 200, "seed": 7, "spacing": 2, "farDistance": 25, "farRate": 6 } } }
+```
+
+`clip` is the clip index (`scene_inspect` lists them), `count` is 1 to 1000, and `layout`, `spacing`, `columns`, `size`, `rateJitter`, `randomYaw`, `speed`,
+`frameRate`, `lods`, `morphs`, `farDistance` and `farRate` are the options above. The crowd is a group node of generated part nodes (count times the
+materials, which count against `maxNodes`), placed by the node transform. The pose depends on the scene time, the seed and the recipe camera, so
+a frame rendered at a time equals the one reached by playing to it. `scene_inspect` lists the crowd under `assets` with `kind: "crowd"`, its instances, batches, clip, baked frames,
+palette bytes and bound. It loads on demand with the `"crowd":{` marker, so recipes without one pay nothing.
### Measured cost
@@ -127,8 +146,21 @@ any seek time equals the one reached by playing. `crowdVertexAt(palette, geometr
(19 parts, 3 materials, 187 joints, 29 baked frames, 7,214 vertices) walking, 1280 by 720, no multisampling, a ground and one shadow map.
It is a real humanoid made of rigid parts on bones and four skinned hands, merged into the rig.
-SwiftShader (the path a headless agent has), the median frame of 5 after warm-up, update and render included. The machine was running other jobs,
-so two runs a few minutes apart are given as a range (in the last run, 500 crowd instances and 50 plain ones both took 208 ms):
+RTX 3060 (measured, one run each, default benchmark settings, 10 frames, update and render included, 7 draw calls and 1,564 instance draws after culling for 500 instances):
+
+| Shader | Instances | Median ms | Best ms |
+| --- | --- | --- | --- |
+| Joints and weights in two attributes (first GPU crowds) | 500 | 5.75 | 4.51 |
+| Joints and weights packed in one attribute (maps and morph targets) | 500 | 8.27 | 6.44 |
+
+The second run shared the card with other jobs, so read the two as a range of about 5 to 8 ms rather than a cost of the packing.
+
+That is under the 16.7 ms of 60 fps with room for a scene around it. The plain path (a skin pose and a batch per instance) cost about 0.26 ms of
+JavaScript per instance on a synthetic 3,000-vertex chain, so about 50 instances fit 60 fps; the crowd path does not grow with a draw call and a
+palette upload per instance.
+
+SwiftShader (the path a headless agent has), the median frame of 5 after warm-up. The machine was running other jobs, so two runs a few minutes
+apart are given as a range (in the last run, 500 crowd instances and 50 plain ones both took 208 ms):
| Path | Instances | Draw calls per frame | Frame ms |
| --- | --- | --- | --- |
@@ -137,16 +169,12 @@ so two runs a few minutes apart are given as a range (in the last run, 500 crowd
| One asset per instance | 20 | 632 | 102 to 156 |
| One asset per instance | 50 | 1,407 | 139 to 208 |
-On SwiftShader most of a crowd frame is fill and vertex work that a CPU rasteriser does slowly, so these numbers say that the cost no longer grows with
-a draw call and a palette upload per instance, not how fast a GPU is. The 7 calls are 3 materials in the colour pass, 3 in the shadow pass and the
-ground, for 20, 100 and 500 instances alike. In a 400-instance field seen from one corner, 340 instance draws were issued across the 7 calls, where
-every instance would give 1,200 per pass.
+On SwiftShader most of a crowd frame is fill and vertex work that a CPU rasteriser does slowly, so these numbers say that the cost no longer grows
+with a draw call and a palette upload per instance, not how fast a GPU is. The 7 calls are 3 materials in the colour pass, 3 in the shadow pass and
+the ground, for 20, 100 and 500 instances alike. In a 400-instance field seen from one corner, 340 instance draws were issued across the 7 calls,
+where every instance would give 1,200 per pass.
Pictures (SwiftShader): one crowd instance against the same robot drawn through the renderer's own rigid and skinned paths at the same pose differs by a
mean of 0.07 of 255, with 0.2 percent of pixels more than 8 levels apart. A crowd of 24 seeked straight to a time and the same crowd played up to it give
-identical pixels. The unit tests check the CPU skinning against per-instance joint matrices at frame times and between frames, and that every
-vertex stays inside `palette.bounds`.
-
-RTX 3060: not measured yet. Run `bash ~/.agents/tools/gpu-lock.sh npm run benchmark:crowd -- --counts 100,500` (or add the scene to
-`npm run benchmark:runtime`) and record the 500-instance frame here; the target is under 16.7 ms at 1280 by 720. The old table (0.26 ms per plain
-instance, 26.5 ms at 100) was measured on a synthetic 3,000-vertex chain and stays the reference for the plain path.
+identical pixels. The unit tests check the CPU skinning against per-instance joint matrices at frame times and between frames, that every vertex stays
+inside `palette.bounds`, and that the frame offsets of a crowd (with a far pose rate) reached by seeking equal those reached by playing.
diff --git a/scripts/agent-runtime.mjs b/scripts/agent-runtime.mjs
index f37546ab..79676fb8 100644
--- a/scripts/agent-runtime.mjs
+++ b/scripts/agent-runtime.mjs
@@ -7,6 +7,7 @@ import { createRequire } from 'node:module';
import { fileURLToPath } from 'node:url';
import packageMetadata from '../package.json' with { type: 'json' };
import { VEGETATION_CAPABILITY_NOTE } from '../src/agent-vegetation-notes.js';
+import { CROWD_CAPABILITY_NOTE } from '../src/agent-crowd-notes.js';
import { AGENT_SCENE_CAPABILITIES, AGENT_SCENE_DEFAULT_LIMITS, RENDER_PASS_NAMES, RENDER_PASS_DEPTH_RANGE, VIDEO_ENCODERS, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS, MAXIMUM_MEMORY_MIB, BAKE_WORKER_MAXIMUM_SECONDS } from '../src/agent-scene-capabilities.js';
import { PATH_TRACER_CAPABILITIES } from '../src/path-tracer-capabilities.js';
import { chooseDenoiser, finishWithOidn } from './agent-path-trace-denoise.mjs';
@@ -572,7 +573,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
return Object.freeze({
async capabilities() {
return {
- scene: { ...AGENT_SCENE_CAPABILITIES, vegetation: VEGETATION_CAPABILITY_NOTE, textureSequence: TEXTURE_SEQUENCE_CAPABILITIES },
+ scene: { ...AGENT_SCENE_CAPABILITIES, vegetation: VEGETATION_CAPABILITY_NOTE, crowd: CROWD_CAPABILITY_NOTE, textureSequence: TEXTURE_SEQUENCE_CAPABILITIES },
sceneLimits: SCENE_LIMITS,
preview: { backend, memoryBudgetMiB: memory, maximumWidth: 1024, maximumHeight: 1024, maximumImageBytes: MAXIMUM_IMAGE_BYTES, maximumActiveScenes: 1, maximumQueuedCalls: MAXIMUM_QUEUED_CALLS, evaluationTimeoutMs: EVALUATION_TIMEOUT_MS },
review: { ...REVIEW_LIMITS, backend, memoryBudgetMiB: memory, evaluationTimeoutMs: EVALUATION_TIMEOUT_MS,
diff --git a/scripts/measure-tool-cost.mjs b/scripts/measure-tool-cost.mjs
index dfad11dd..8f488802 100644
--- a/scripts/measure-tool-cost.mjs
+++ b/scripts/measure-tool-cost.mjs
@@ -4,6 +4,7 @@ import { Client } from '@modelcontextprotocol/client';
import { InMemoryTransport } from '@modelcontextprotocol/server';
import { createAgentMcpServer } from './mcp-server.mjs';
import { VEGETATION_CAPABILITY_NOTE } from '../src/agent-vegetation-notes.js';
+import { CROWD_CAPABILITY_NOTE } from '../src/agent-crowd-notes.js';
import { createAgentScene, AGENT_SCENE_CAPABILITIES } from '../src/agent-scene.js';
import { TEXTURE_SEQUENCE_CAPABILITIES } from '../src/texture-sequence-limits.js';
import { loadAgentSceneModules } from '../src/agent-scene-modules.js';
@@ -70,7 +71,7 @@ async function typicalResults(recipe) {
const meshReport = scene.inspectMeshes({ geometries: ['seat'] });
const { inspection, ...statusWithoutInspection } = status;
return {
- scene_capabilities: { ...AGENT_SCENE_CAPABILITIES, vegetation: VEGETATION_CAPABILITY_NOTE, textureSequence: TEXTURE_SEQUENCE_CAPABILITIES },
+ scene_capabilities: { ...AGENT_SCENE_CAPABILITIES, vegetation: VEGETATION_CAPABILITY_NOTE, crowd: CROWD_CAPABILITY_NOTE, textureSequence: TEXTURE_SEQUENCE_CAPABILITIES },
scene_create: status,
scene_inspect: { sessionId: SAMPLE_SESSION, ...scene.inspect() },
mesh_inspect: { sessionId: SAMPLE_SESSION, ...meshReport },
diff --git a/src/agent-crowd-notes.js b/src/agent-crowd-notes.js
new file mode 100644
index 00000000..f188ed93
--- /dev/null
+++ b/src/agent-crowd-notes.js
@@ -0,0 +1,2 @@
+// What scene_capabilities says about recipe crowds. It lives apart from agent-scene-capabilities.js, which every agent scene bundle carries; the MCP host adds it to the capabilities it returns.
+export const CROWD_CAPABILITY_NOTE = "Node field asset.crowd fills a group node with walking people: {id: catalog asset with a skinned clip, crowd:{clip=0 index, count 1-1000, seed=1, layout: grid|scatter, spacing=1.5, columns?, size?, rateJitter=0.15, speed=1, randomYaw, frameRate=30, lods=true, morphs=false, farDistance?+farRate?}}. All people share one batch per material (src/crowd-skinning.js), each at its own time offset; the pose is a pure function of scene time, seed and recipe camera, so a seek equals playing. farDistance m with farRate poses/s: people farther than that from the recipe camera eye hold each pose 1/farRate s. morphs:true keeps morph targets on their own part, at the weights the asset has when loaded, drawn at full detail. Part nodes count against maxNodes (count x materials). scene_inspect assets lists kind crowd with instances, batches, clip, frames, paletteBytes and boundsRadius.";
diff --git a/src/agent-crowd.js b/src/agent-crowd.js
new file mode 100644
index 00000000..8f0fa133
--- /dev/null
+++ b/src/agent-crowd.js
@@ -0,0 +1,152 @@
+import { createCrowd } from './retarget.js';
+import { bakeCrowdPalette, createCrowdParts, createCrowdRig } from './crowd-skinning.js';
+import { jsonMaterial, materialOutput, registerCrowdModule } from './agent-scene-assets.js';
+
+// Recipe crowd field: an asset placement with a `crowd` field turns one catalog character and one of its clips into
+// `count` walking instances that share a few instanced batches (src/crowd-skinning.js). The result travels the
+// placed-asset channel of agent-scene.js: generated geometries (type "asset"), materials (assetSource) and part nodes
+// (assetPart), so exports and edits already skip them. The pose of every instance is a pure function of the scene
+// time, the seed and the recipe camera, so a seek equals playing.
+
+export const CROWD_LIMITS = Object.freeze({ maxInstances: 1000, maxCachedCrowds: 4 });
+const CROWD_FIELDS = ['clip', 'count', 'seed', 'layout', 'spacing', 'columns', 'size', 'rateJitter', 'randomYaw', 'speed', 'frameRate', 'morphs', 'lods', 'farDistance', 'farRate'];
+
+function numberWithin(value, minimum, maximum, label) {
+ if (!Number.isFinite(value) || value < minimum || value > maximum) {
+ throw new RangeError(`crowd ${label} must be a number from ${minimum} to ${maximum}.`);
+ }
+ return value;
+}
+
+/** Validate the `crowd` field of an asset placement. Returns the canonical form kept in the recipe. */
+export function normalizeCrowd(value) {
+ if (!value || typeof value !== 'object' || Array.isArray(value)) throw new TypeError('crowd must be an object.');
+ for (const key of Object.keys(value)) {
+ if (!CROWD_FIELDS.includes(key)) throw new TypeError(`Unknown crowd field: ${key}. Allowed fields: ${CROWD_FIELDS.join(', ')}.`);
+ }
+ if (!Number.isInteger(value.count) || value.count < 1 || value.count > CROWD_LIMITS.maxInstances) {
+ throw new RangeError(`crowd count must be an integer from 1 to ${CROWD_LIMITS.maxInstances}.`);
+ }
+ const layout = value.layout ?? 'grid';
+ if (layout !== 'grid' && layout !== 'scatter') throw new RangeError('crowd layout must be grid or scatter.');
+ const clip = value.clip ?? 0;
+ if (!Number.isInteger(clip) || clip < 0) throw new RangeError('crowd clip must be a non-negative clip index.');
+ const result = {
+ clip,
+ count: value.count,
+ seed: numberWithin(value.seed ?? 1, -2147483648, 2147483647, 'seed'),
+ layout,
+ spacing: numberWithin(value.spacing ?? 1.5, 0.05, 1000, 'spacing'),
+ rateJitter: numberWithin(value.rateJitter ?? 0.15, 0, 0.9, 'rateJitter'),
+ speed: numberWithin(value.speed ?? 1, 0.05, 16, 'speed'),
+ frameRate: numberWithin(value.frameRate ?? 30, 4, 60, 'frameRate'),
+ };
+ if (value.columns !== undefined) result.columns = Math.round(numberWithin(value.columns, 1, CROWD_LIMITS.maxInstances, 'columns'));
+ if (value.size !== undefined) result.size = numberWithin(value.size, 0.1, 100_000, 'size');
+ for (const flag of ['randomYaw', 'morphs', 'lods']) {
+ if (value[flag] !== undefined && typeof value[flag] !== 'boolean') throw new TypeError(`crowd ${flag} must be boolean.`);
+ }
+ result.randomYaw = value.randomYaw ?? false;
+ result.morphs = value.morphs ?? false;
+ result.lods = value.lods ?? true;
+ if ((value.farDistance === undefined) !== (value.farRate === undefined)) throw new TypeError('crowd farDistance and farRate go together.');
+ if (value.farDistance !== undefined) {
+ result.farDistance = numberWithin(value.farDistance, 0.1, 100_000, 'farDistance');
+ result.farRate = numberWithin(value.farRate, 1, 60, 'farRate');
+ }
+ return result;
+}
+
+function buildCrowd(asset, assetId, settings) {
+ const source = asset.animations[settings.clip];
+ const rig = createCrowdRig(asset.parts, { morphs: settings.morphs });
+ const poseAt = seconds => asset.stagePose({ animation: { clip: settings.clip, time: seconds }, nodeOverrides: [] }).commit();
+ const palette = bakeCrowdPalette(rig, poseAt, { duration: source.duration, frameRate: settings.frameRate });
+ const crowd = createCrowd({
+ count: settings.count, clip: { duration: source.duration, tracks: [] }, seed: settings.seed, layout: settings.layout,
+ spacing: settings.spacing, columns: settings.columns, size: settings.size, rateJitter: settings.rateJitter, randomYaw: settings.randomYaw,
+ });
+ const farPose = settings.farDistance ? { distance: settings.farDistance, rate: settings.farRate } : null;
+ const set = createCrowdParts({ rig, palette, instances: crowd.instances, lods: settings.lods, farPose, name: assetId.slice(0, 12) });
+ const materialOutputs = rig.parts.map(rigPart => materialOutput(rigPart.material, {}));
+ return { asset, rig, palette, crowd, set, materialOutputs, clipName: source.name ?? `clip ${settings.clip}`, duration: source.duration };
+}
+
+/**
+ * Generate the geometries, materials and part nodes of one crowd placement into a normalized recipe, in place.
+ * `cache` (the scene's asset output map) keeps the baked palette and the instance list across edits, so a camera or
+ * unrelated edit does not bake again and the parts of the last stage stay reusable. `asset` is the decoded copy
+ * reserved for this placement, which the bake poses. Returns the nodes to add and the placement record.
+ */
+export function expandCrowd(recipe, node, asset, copy, cache, runtime, limits) {
+ const { id: assetId, crowd: settings } = node.asset;
+ if (!asset.animations[settings.clip]) {
+ throw new RangeError(`Node ${node.id}: asset ${assetId.slice(0, 12)} has ${asset.animations.length} clips, so crowd clip ${settings.clip} does not exist.`);
+ }
+ const key = `crowd:${node.id}:${assetId}:${copy}:${JSON.stringify(settings)}`;
+ if (cache.get(key)?.asset !== asset) {
+ const stale = [...cache.keys()].filter(candidate => candidate.startsWith('crowd:'));
+ if (stale.length >= CROWD_LIMITS.maxCachedCrowds) cache.delete(stale[0]);
+ try {
+ cache.set(key, buildCrowd(asset, assetId, settings));
+ } catch (error) {
+ error.message = `Node ${node.id} crowd: ${error.message}`;
+ throw error;
+ }
+ }
+ const built = cache.get(key);
+ const rigPartCount = built.rig.parts.length;
+ if (recipe.nodes.length + settings.count * rigPartCount > limits.maxNodes) {
+ throw new RangeError(`Node ${node.id}: a crowd of ${settings.count} makes ${settings.count * rigPartCount} part nodes (${rigPartCount} per instance), which with the ${recipe.nodes.length} authored nodes exceeds maxNodes ${limits.maxNodes}. Lower count or raise maxNodes.`);
+ }
+ const prefix = `crowd:${assetId.slice(0, 12)}:${copy}`;
+ const materialIds = built.rig.parts.map((rigPart, index) => {
+ const id = `${prefix}:${index}`;
+ runtime.materialOutputs.set(id, built.materialOutputs[index]);
+ recipe.materials.push({ id, ...jsonMaterial(rigPart.material), assetSource: true });
+ runtime.geometryParts.set(id, rigPart.geometry);
+ recipe.geometries.push({ id, type: 'asset', assetId, copy, part: index });
+ return id;
+ });
+ const nodes = [];
+ const partNodeIds = [];
+ built.set.parts.forEach((part, index) => {
+ const instance = Math.floor(index / rigPartCount);
+ const rigIndex = index % rigPartCount;
+ const id = `${node.id}/${instance}/${rigIndex}`;
+ partNodeIds.push(id);
+ nodes.push({
+ id, parent: node.id, label: `${built.clipName} ${instance}`,
+ translation: [0, 0, 0], rotation: [0, 0, 0, 1], scale: [1, 1, 1],
+ geometry: `${prefix}:${rigIndex}`, material: materialIds[rigIndex],
+ assetPart: { placement: node.id, index: instance, matrix: Array.from(part.matrix) },
+ });
+ runtime.partExtras.set(id, { crowd: part.crowd, assembly: part.assembly, ...(part.lods ? { lods: part.lods } : {}), ...(part.morph ? { morph: part.morph } : {}) });
+ });
+ const eye = recipe.camera?.eye ?? null;
+ const record = {
+ node: node.id, assetId, copy, asset, partNodeIds, clip: null, drive: null, drivePose: null,
+ crowd: {
+ /** The pose at a scene time, applied on commit: only frame offsets change, so nothing here can fail. */
+ stage(time) {
+ const seconds = time * settings.speed;
+ return {
+ commit: () => built.set.setTimes(index => built.crowd.clipTime(index, seconds), eye),
+ discard() {},
+ };
+ },
+ describe() {
+ return {
+ node: node.id, assetId, kind: 'crowd', instances: settings.count, parts: partNodeIds.length,
+ batches: rigPartCount, clip: { index: settings.clip, name: built.clipName, duration: built.duration },
+ frames: built.palette.frameCount, paletteBytes: built.palette.data.byteLength, boundsRadius: built.palette.bounds.radius,
+ lods: settings.lods, morphs: built.rig.parts.filter(part => part.morph).length,
+ ...(settings.farDistance ? { farPose: { distance: settings.farDistance, rate: settings.farRate } } : {}),
+ };
+ },
+ },
+ };
+ return { nodes, record };
+}
+
+registerCrowdModule({ normalizeCrowd, expandCrowd });
diff --git a/src/agent-scene-assets.js b/src/agent-scene-assets.js
index 4284eaca..d27b48c0 100644
--- a/src/agent-scene-assets.js
+++ b/src/agent-scene-assets.js
@@ -12,8 +12,20 @@ export * from './agent-texture-sequence.js';
export const ASSET_PLACEMENT_LIMITS = Object.freeze({ maxPlacements: 512, maxAssets: 16, maxAnimatedPlacements: 16, maxOverrides: 32 });
+// The crowd code registers itself here when it loads, so this file (which the MCP host imports) never pulls it in.
+let crowdModule = null;
+
+export function registerCrowdModule(module) {
+ crowdModule = module;
+}
+
+function crowdCode() {
+ if (!crowdModule) throw new Error('This recipe places a crowd, which loads on demand: await loadAgentSceneModules(recipe) before createAgentScene or apply.');
+ return crowdModule;
+}
+
const ASSET_ID = /^[a-f0-9]{64}$/;
-const PLACEMENT_FIELDS = ['id', 'materials', 'clip', 'drive'];
+const PLACEMENT_FIELDS = ['id', 'materials', 'clip', 'drive', 'crowd'];
const CLIP_FIELDS = ['index', 'start', 'speed', 'loop'];
const COLOR_FIELDS = ['color', 'emissiveColor'];
const UNIT_FIELDS = ['metallic', 'roughness', 'opacity'];
@@ -95,6 +107,10 @@ export function normalizeAssetPlacement(value) {
if (typeof loop !== 'boolean') throw new TypeError('asset clip loop must be boolean.');
result.clip = { index, start, speed, loop };
}
+ if (value.crowd !== undefined) {
+ if (value.clip !== undefined || value.drive !== undefined) throw new TypeError('An asset placement plays a clip, drives along a path or fills a crowd, only one of them.');
+ result.crowd = crowdCode().normalizeCrowd(value.crowd);
+ }
if (value.drive !== undefined) {
if (value.clip !== undefined) throw new TypeError('An asset placement plays a clip or drives along a path, not both.');
result.drive = normalizeDrive(value.drive);
@@ -131,7 +147,7 @@ export function recipeAssetNeeds(value) {
for (const placement of placements) {
const id = placement.id.toLowerCase();
const need = needs.get(id) ?? { id, copies: 1 };
- if (placement.clip || placement.drive) need.copies++;
+ if (placement.clip || placement.drive || placement.crowd) need.copies++;
needs.set(id, need);
}
return [...needs.values()];
@@ -141,7 +157,7 @@ export function assetCopyKey(id, copy) {
return `${id}:${copy}`;
}
-function jsonMaterial(source) {
+export function jsonMaterial(source) {
const result = {};
for (const [field, value] of Object.entries(source)) {
if (typeof value === 'number' || typeof value === 'string' || typeof value === 'boolean') {
@@ -162,7 +178,7 @@ function overrideFields(override) {
return fields;
}
-function materialOutput(source, fields) {
+export function materialOutput(source, fields) {
const output = { ...source };
for (const [field, value] of Object.entries(fields)) {
output[field] = Array.isArray(value) ? Object.freeze([...value]) : value;
@@ -185,7 +201,7 @@ export function expandAssetPlacements(recipe, assets, limits, outputCache = new
throw new Error('This recipe places catalog assets, which a host loads and decodes first. Pass them as the assets option of createAgentScene; MCP does this for scene_create.');
}
const distinct = new Set(placed.map(node => node.asset.id));
- const animated = placed.filter(node => node.asset.clip || node.asset.drive);
+ const animated = placed.filter(node => node.asset.clip || node.asset.drive || node.asset.crowd);
if (placed.length > ASSET_PLACEMENT_LIMITS.maxPlacements || distinct.size > ASSET_PLACEMENT_LIMITS.maxAssets ||
animated.length > ASSET_PLACEMENT_LIMITS.maxAnimatedPlacements) {
throw new RangeError(`A scene places at most ${ASSET_PLACEMENT_LIMITS.maxPlacements} assets, ${ASSET_PLACEMENT_LIMITS.maxAssets} different ones, ${ASSET_PLACEMENT_LIMITS.maxAnimatedPlacements} of them playing a clip; this recipe places ${placed.length}, ${distinct.size} and ${animated.length}.`);
@@ -195,9 +211,9 @@ export function expandAssetPlacements(recipe, assets, limits, outputCache = new
const materialIds = new Map();
const generatedNodes = [];
for (const node of placed) {
- const { id: assetId, materials: overrides = [], clip, drive } = node.asset;
+ const { id: assetId, materials: overrides = [], clip, drive, crowd } = node.asset;
let copy = 0;
- if (clip || drive) {
+ if (clip || drive || crowd) {
copy = (copyCounter.get(assetId) ?? 0) + 1;
copyCounter.set(assetId, copy);
}
@@ -208,6 +224,13 @@ export function expandAssetPlacements(recipe, assets, limits, outputCache = new
if (clip && !asset.animations[clip.index]) {
throw new RangeError(`Node ${node.id}: asset ${assetId.slice(0, 12)} has ${asset.animations.length} clips, so clip index ${clip.index} does not exist. scene_inspect lists them under assets.`);
}
+ if (crowd) {
+ const expanded = crowdCode().expandCrowd(recipe, node, asset, copy, outputCache, runtime, limits);
+ generatedNodes.push(...expanded.nodes);
+ runtime.placements.push(expanded.record);
+ runtime.animators.push(expanded.record);
+ continue;
+ }
const materialNames = new Set(asset.parts.map(part => part.material.name).filter(Boolean));
for (const override of overrides) {
if (!materialNames.has(override.name)) {
@@ -286,6 +309,10 @@ export function stageAssetClips(runtime, graph, time) {
try {
for (const record of runtime.animators) {
let transaction;
+ if (record.crowd) {
+ transactions.push(record.crowd.stage(time ?? 0));
+ continue;
+ }
if (record.drive) {
const driven = record.drive.sample(time ?? 0);
record.drivePose = driven.pose;
@@ -315,6 +342,7 @@ export function stageAssetClips(runtime, graph, time) {
/** What scene_inspect reports for each placement: the asset id, its world bounds, its clips and material names. */
export function describeAssetPlacements(runtime, graph, geometryBounds) {
return runtime.placements.map(record => {
+ if (record.crowd) return record.crowd.describe();
const minimum = [Infinity, Infinity, Infinity];
const maximum = [-Infinity, -Infinity, -Infinity];
for (const id of record.partNodeIds) {
diff --git a/src/agent-scene-modules.js b/src/agent-scene-modules.js
index 905217d0..ae57d1a7 100644
--- a/src/agent-scene-modules.js
+++ b/src/agent-scene-modules.js
@@ -27,6 +27,7 @@ const MODULE_LOADERS = {
cascades: () => import('./cascaded-shadows.js'),
assets: () => import('./agent-scene-assets.js'),
vegetation: () => import('./agent-vegetation.js'),
+ crowd: () => import('./agent-crowd.js'),
lineArt: () => import('./line-art.js'),
viewTransform: () => import('./view-transform-shader.js'),
coplanar: () => import('./coplanar-overlap.js'),
@@ -64,6 +65,7 @@ const MODULE_MARKERS = {
cascades: ['"shadowCascades"'],
assets: ['"asset":{', '"sequence":{'],
vegetation: ['"vegetation":{', '"wind":{'],
+ crowd: ['"crowd":{'],
lineArt: ['"lines":{', '"toon":'],
viewTransform: ['"viewTransform":"agx"', '"viewTransform":"neutral"', '"look":"'],
// The coplanar-overlap warning in scene_inspect appears only once this is loaded, which loadAllAgentSceneModules does; a recipe with resolveCoplanar needs it.
diff --git a/src/crowd-skinning.js b/src/crowd-skinning.js
index 8b7b5129..f722266c 100644
--- a/src/crowd-skinning.js
+++ b/src/crowd-skinning.js
@@ -15,8 +15,10 @@ const FLOATS_PER_MATRIX = 16;
const TEXELS_PER_MATRIX = 4;
const DEFAULT_FRAME_RATE = 30;
const DEFAULT_LEVEL_CELLS = Object.freeze([0.02, 0.05]);
+const JOINT_STRIDE = 4096;
+const WEIGHT_STEPS = 4095;
-// Joint ids sit in the geometry's crowdJoints attribute (location 13) and weights in crowdWeights (location 14).
+// Each vertex carries four packed influences in the colour attribute (location 14): joint * 4096 + weight * 4095, exact in a float.
// The frame blend is in assemblyOffset: x and y are the texel offsets of the two frames, z the blend between them.
installCrowdShader(`
uniform float crowdEnabled;
@@ -25,23 +27,26 @@ mat4 crowdJoint(int frameTexel, int joint) {
return mat4(skinTexel(jointPalette, texel), skinTexel(jointPalette, texel + 1),
skinTexel(jointPalette, texel + 2), skinTexel(jointPalette, texel + 3));
}
-mat4 crowdInfluence(float joint, float weight) {
+mat4 crowdInfluence(float packedInfluence) {
+ float joint = floor(packedInfluence / 4096.0);
+ float weight = (packedInfluence - joint * 4096.0) / 4095.0;
int firstFrame = int(assemblyOffset.x + 0.5);
int secondFrame = int(assemblyOffset.y + 0.5);
int jointIndex = int(joint + 0.5);
return (crowdJoint(firstFrame, jointIndex) * (1.0 - assemblyOffset.z) + crowdJoint(secondFrame, jointIndex) * assemblyOffset.z) * weight;
}
mat4 crowdTransform() {
- return crowdInfluence(uvSets.x, vertexColor.x) + crowdInfluence(uvSets.y, vertexColor.y)
- + crowdInfluence(uvSets.z, vertexColor.z) + crowdInfluence(uvSets.w, vertexColor.w);
+ return crowdInfluence(vertexColor.x) + crowdInfluence(vertexColor.y) + crowdInfluence(vertexColor.z) + crowdInfluence(vertexColor.w);
}`);
/**
* Merge the parts of a character into crowd geometries, one per material. A rigid part becomes one joint that
- * follows its matrix; a skinned part keeps its joints, each premultiplied by the part's matrix. Morph targets are not
- * applied. Each part is { name, geometry, matrix, material, skin? } as an imported asset lists them.
+ * follows its matrix; a skinned part keeps its joints, each premultiplied by the part's matrix. UVs are kept, so
+ * materials with maps work. Parts with morph targets are merged with their weights ignored, unless `morphs` is true:
+ * then each keeps its own geometry and draws with the weights it has when the rig is made (one set for the whole crowd,
+ * and no levels or culling for that part). Each part is { name, geometry, matrix, material, skin?, morph? } as an imported asset lists them.
*/
-export function createCrowdRig(sourceParts) {
+export function createCrowdRig(sourceParts, { morphs = false } = {}) {
if (!Array.isArray(sourceParts) || sourceParts.length === 0) throw new TypeError('A crowd rig needs a list of parts.');
const jointOffsets = [];
let jointCount = 0;
@@ -51,17 +56,30 @@ export function createCrowdRig(sourceParts) {
jointOffsets.push(jointCount);
jointCount += part.skin ? part.skin.jointMatrices.length / FLOATS_PER_MATRIX : 1;
}
- const partsByMaterial = new Map();
+ if (jointCount > JOINT_STRIDE) throw new RangeError(`A crowd rig holds at most ${JOINT_STRIDE} joints; this one has ${jointCount}.`);
+ const groups = new Map();
+ const morphedParts = [];
sourceParts.forEach((part, index) => {
- if (!partsByMaterial.has(part.material)) partsByMaterial.set(part.material, []);
- partsByMaterial.get(part.material).push(index);
+ if (morphs && part.morph && part.geometry.morphTargets) {
+ morphedParts.push(index);
+ return;
+ }
+ if (!groups.has(part.material)) groups.set(part.material, []);
+ groups.get(part.material).push(index);
});
const scratch = mat4.create();
- const parts = [...partsByMaterial].map(([material, indices]) => ({
+ const entryOf = index => ({ part: sourceParts[index], jointOffset: jointOffsets[index] });
+ const parts = [...groups].map(([material, indices]) => ({
name: sourceParts[indices[0]].name,
material,
- geometry: mergeCrowdGeometry(indices.map(index => ({ part: sourceParts[index], jointOffset: jointOffsets[index] }))),
+ geometry: mergeCrowdGeometry(indices.map(entryOf)),
}));
+ for (const index of morphedParts) {
+ const part = sourceParts[index];
+ const geometry = mergeCrowdGeometry([entryOf(index)]);
+ geometry.morphTargets = part.geometry.morphTargets;
+ parts.push({ name: part.name, material: part.material, geometry, morph: { weights: new Float32Array(part.morph.weights) } });
+ }
return {
jointCount,
parts,
@@ -93,6 +111,7 @@ function mergeCrowdGeometry(entries) {
crowdJoints: new Float32Array(vertexCount * 4),
crowdWeights: new Float32Array(vertexCount * 4),
};
+ if (entries.some(({ part }) => part.geometry.uvs)) merged.uvs = new Float32Array(vertexCount * 2);
let vertexStart = 0;
let indexStart = 0;
for (const { part, jointOffset } of entries) {
@@ -100,6 +119,7 @@ function mergeCrowdGeometry(entries) {
const count = geometry.positions.length / 3;
merged.positions.set(geometry.positions, vertexStart * 3);
merged.normals.set(geometry.normals, vertexStart * 3);
+ if (geometry.uvs) merged.uvs.set(geometry.uvs, vertexStart * 2);
for (let index = 0; index < geometry.indices.length; index++) merged.indices[indexStart + index] = geometry.indices[index] + vertexStart;
for (let vertex = 0; vertex < count; vertex++) {
const target = (vertexStart + vertex) * 4;
@@ -112,15 +132,24 @@ function mergeCrowdGeometry(entries) {
for (let influence = 0; influence < 4; influence++) total += geometry.skinWeights[vertex * 4 + influence];
for (let influence = 0; influence < 4; influence++) {
merged.crowdJoints[target + influence] = geometry.skinIndices[vertex * 4 + influence] + jointOffset;
- merged.crowdWeights[target + influence] = geometry.skinWeights[vertex * 4 + influence] / total;
+ merged.crowdWeights[target + influence] = Math.round(geometry.skinWeights[vertex * 4 + influence] / total * WEIGHT_STEPS) / WEIGHT_STEPS;
}
}
vertexStart += count;
indexStart += geometry.indices.length;
}
+ packCrowdInfluences(merged);
return merged;
}
+/** The attribute the renderer uploads: joint * 4096 + round(weight * 4095) for each of four influences. Weights are already whole steps. */
+function packCrowdInfluences(geometry) {
+ geometry.crowdPacked = new Float32Array(geometry.crowdJoints.length);
+ for (let index = 0; index < geometry.crowdJoints.length; index++) {
+ geometry.crowdPacked[index] = geometry.crowdJoints[index] * JOINT_STRIDE + Math.round(geometry.crowdWeights[index] * WEIGHT_STEPS);
+ }
+}
+
/** The position of one crowd vertex under a table of joint matrices at `frameOffset` (in matrices), the maths the shader runs. */
function skinCrowdVertex(geometry, vertex, matrices, frameOffset, output = [0, 0, 0]) {
const x = geometry.positions[vertex * 3];
@@ -262,7 +291,7 @@ function clusterCrowdGeometry(geometry, cellSize) {
const key = [0, 1, 2].map(axis => Math.floor(geometry.positions[vertex * 3 + axis] / cellSize)).join(',') + `:${geometry.crowdJoints[vertex * 4 + dominant]}`;
if (!clusterByKey.has(key)) {
clusterByKey.set(key, representatives.length);
- representatives.push({ vertex, count: 0, position: [0, 0, 0], normal: [0, 0, 0] });
+ representatives.push({ vertex, count: 0, position: [0, 0, 0], normal: [0, 0, 0], uv: [0, 0] });
}
const clusterIndex = clusterByKey.get(key);
const cluster = representatives[clusterIndex];
@@ -272,10 +301,15 @@ function clusterCrowdGeometry(geometry, cellSize) {
cluster.position[axis] += geometry.positions[vertex * 3 + axis];
cluster.normal[axis] += geometry.normals[vertex * 3 + axis];
}
+ if (geometry.uvs) {
+ cluster.uv[0] += geometry.uvs[vertex * 2];
+ cluster.uv[1] += geometry.uvs[vertex * 2 + 1];
+ }
}
const coarse = {
positions: new Float32Array(representatives.length * 3),
normals: new Float32Array(representatives.length * 3),
+ ...(geometry.uvs ? { uvs: new Float32Array(representatives.length * 2) } : {}),
crowdJoints: new Float32Array(representatives.length * 4),
crowdWeights: new Float32Array(representatives.length * 4),
};
@@ -285,6 +319,10 @@ function clusterCrowdGeometry(geometry, cellSize) {
coarse.positions[index * 3 + axis] = cluster.position[axis] / cluster.count;
coarse.normals[index * 3 + axis] = cluster.normal[axis] / length;
}
+ if (geometry.uvs) {
+ coarse.uvs[index * 2] = cluster.uv[0] / cluster.count;
+ coarse.uvs[index * 2 + 1] = cluster.uv[1] / cluster.count;
+ }
coarse.crowdJoints.set(geometry.crowdJoints.subarray(cluster.vertex * 4, cluster.vertex * 4 + 4), index * 4);
coarse.crowdWeights.set(geometry.crowdWeights.subarray(cluster.vertex * 4, cluster.vertex * 4 + 4), index * 4);
});
@@ -296,6 +334,7 @@ function clusterCrowdGeometry(geometry, cellSize) {
if (first !== second && second !== third && first !== third) indices.push(first, second, third);
}
coarse.indices = Uint32Array.from(indices);
+ packCrowdInfluences(coarse);
return coarse;
}
@@ -303,11 +342,14 @@ function clusterCrowdGeometry(geometry, cellSize) {
* The renderer parts of a crowd: one part per instance and material, all instances of a material sharing a geometry,
* a palette and `lods`, so they draw as one batch per material and level. `instances` come from createCrowd (or any
* list with a column-major `matrix`). Call `setTimes` with each instance's clip time, then `renderer.updateParts(parts)`.
- * `lods: false` turns the coarser levels off; `cells` sets their sizes.
+ * `lods: false` turns the coarser levels off; `cells` sets their sizes. `farPose: { distance, rate }` makes an instance
+ * more than `distance` from the eye passed to `setTimes` hold each pose for 1 / `rate` seconds. That is still a pure
+ * function of the clip time and the eye, so a seek equals playing.
*/
-export function createCrowdParts({ rig, palette, instances, lods = true, cells, name = 'crowd' }) {
+export function createCrowdParts({ rig, palette, instances, lods = true, cells, farPose = null, name = 'crowd' }) {
if (!instances?.length) throw new TypeError('A crowd needs a list of instances.');
- const rigParts = rig.parts.map(part => ({ ...part, levels: lods ? createCrowdLods(part.geometry, { cells }) : null }));
+ if (farPose && !(farPose.distance > 0 && farPose.rate > 0)) throw new RangeError('farPose needs a positive distance and rate.');
+ const rigParts = rig.parts.map(part => ({ ...part, levels: lods && !part.morph ? createCrowdLods(part.geometry, { cells }) : null }));
const blends = [];
const parts = [];
instances.forEach((instance, index) => {
@@ -321,15 +363,25 @@ export function createCrowdParts({ rig, palette, instances, lods = true, cells,
material: rigPart.material,
crowd: palette,
assembly: { offset: blend, detailOffset: [0, 0, 0] },
+ ...(rigPart.morph ? { morph: rigPart.morph } : {}),
...(rigPart.levels ? { lods: rigPart.levels } : {}),
});
}
});
return {
parts,
- /** Set every instance's clip time: an array of seconds or `(index) => seconds`. */
- setTimes(times) {
- blends.forEach((blend, index) => crowdFrameBlend(palette, typeof times === 'function' ? times(index) : times[index], blend));
+ /** Set every instance's clip time: an array of seconds or `(index) => seconds`. `eye` is the camera position, used with `farPose`. */
+ setTimes(times, eye = null) {
+ blends.forEach((blend, index) => {
+ let seconds = typeof times === 'function' ? times(index) : times[index];
+ if (farPose && eye) {
+ const matrix = instances[index].matrix;
+ if (Math.hypot(matrix[12] - eye[0], matrix[13] - eye[1], matrix[14] - eye[2]) > farPose.distance) {
+ seconds = Math.floor(seconds * farPose.rate) / farPose.rate;
+ }
+ }
+ crowdFrameBlend(palette, seconds, blend);
+ });
},
};
}
diff --git a/src/renderer.js b/src/renderer.js
index 33bde592..bedc71cf 100644
--- a/src/renderer.js
+++ b/src/renderer.js
@@ -833,7 +833,7 @@ export class Renderer67 {
let geometryBuffers = this.geometryBuffers.get(geometry);
if (!geometryBuffers) {
geometryBuffers = new Map();
- for (const field of ['positions', 'normals', 'uvs', 'colors', 'indices', 'crowdJoints', 'crowdWeights']) {
+ for (const field of ['positions', 'normals', 'uvs', 'colors', 'indices', 'crowdPacked']) {
if (geometry[field]) {
const target = field === 'indices' ? context.ELEMENT_ARRAY_BUFFER : context.ARRAY_BUFFER;
const values = field === 'uvs' && geometry.lightUvs ? interleaveUvSets(geometry) : geometry[field];
@@ -871,13 +871,11 @@ export class Renderer67 {
context.disableVertexAttribArray(14);
context.vertexAttrib3f(14, 1, 1, 1);
}
- // A crowd geometry keeps four joint ids and four weights per vertex where textures and tints would sit.
- if (geometry.crowdJoints) {
- for (const [location, field] of [[13, 'crowdJoints'], [14, 'crowdWeights']]) {
- context.bindBuffer(context.ARRAY_BUFFER, geometryBuffers.get(field));
- context.enableVertexAttribArray(location);
- context.vertexAttribPointer(location, 4, context.FLOAT, false, 0, 0);
- }
+ // A crowd geometry packs four joint ids and weights into the colour attribute; its UVs stay free for maps.
+ if (geometry.crowdPacked) {
+ context.bindBuffer(context.ARRAY_BUFFER, geometryBuffers.get('crowdPacked'));
+ context.enableVertexAttribArray(14);
+ context.vertexAttribPointer(14, 4, context.FLOAT, false, 0, 0);
}
context.bindBuffer(context.ELEMENT_ARRAY_BUFFER, geometryBuffers.get('indices'));
}
diff --git a/src/shaders.js b/src/shaders.js
index a01ce4cb..d0e0b36e 100644
--- a/src/shaders.js
+++ b/src/shaders.js
@@ -88,8 +88,8 @@ ${skinningShader(features)}
${morphingShader(features)}
${windShader(features)}
void main() {
- // A crowd batch reads its joints and weights from the UV and colour attributes, which then carry no texture or tint.
- vec2 uv = USES_CROWD ? vec2(-1.0) : uvSets.xy;
+ // A crowd batch reads its joints and weights from the colour attribute, which then carries no tint.
+ vec2 uv = uvSets.xy;
mat4 deformation = USES_CROWD ? crowdTransform() : (USES_SKIN ? skinTransform() : mat4(1.0));
vec4 world = instanceMatrix * deformation * vec4(USES_MORPH ? morphedPosition(position) : position, 1.0);
world.xyz += USES_CROWD ? vec3(0.0) : assemblyOffset * assemblyAmount + assemblyDetailOffset * smoothstep(0.3, 1.0, assemblyAmount);
@@ -142,7 +142,7 @@ ${skinningShader(features)}
${morphingShader(features)}
${windShader(features)}
void main() {
- vec2 uv = USES_CROWD ? vec2(-1.0) : uvSets.xy;
+ vec2 uv = uvSets.xy;
vec4 world = instanceMatrix * (USES_CROWD ? crowdTransform() : skinTransform()) * vec4(morphedPosition(position), 1.0);
world.xyz += USES_CROWD ? vec3(0.0) : assemblyOffset * assemblyAmount + assemblyDetailOffset * smoothstep(0.3, 1.0, assemblyAmount);
// Same displacement as the colour pass, so shadows follow the swaying leaves.
diff --git a/tests/agent-runtime.test.js b/tests/agent-runtime.test.js
index 4b6e714e..018e0a5e 100644
--- a/tests/agent-runtime.test.js
+++ b/tests/agent-runtime.test.js
@@ -7,6 +7,7 @@ import { createServer } from 'node:http';
import { randomUUID } from 'node:crypto';
import { createContext, Script } from 'node:vm';
import { VEGETATION_CAPABILITY_NOTE } from '../src/agent-vegetation-notes.js';
+import { CROWD_CAPABILITY_NOTE } from '../src/agent-crowd-notes.js';
import { AGENT_SCENE_CAPABILITIES, AGENT_SCENE_DEFAULT_LIMITS, RENDER_PASS_NAMES, RENDER_PASS_DEPTH_RANGE, VIDEO_ENCODERS, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS, MAXIMUM_MEMORY_MIB, BAKE_WORKER_MAXIMUM_SECONDS } from '../src/agent-scene-capabilities.js';
import { recipeAssetNeeds } from '../src/agent-scene-assets.js';
import { sequenceFrameIds } from '../src/agent-texture-sequence.js';
@@ -105,7 +106,7 @@ async function reviewRuntimeHarness({ failShot, blockShot = false, oversize = fa
const context = createContext({ createServer, randomUUID, existsSync: () => true,
readFile: async () => '', packageMetadata: { version: 'test' },
createRequire: () => ({ resolve: () => { throw new Error('no peer in tests'); } }),
- VEGETATION_CAPABILITY_NOTE, AGENT_SCENE_CAPABILITIES, AGENT_SCENE_DEFAULT_LIMITS, RENDER_PASS_NAMES, RENDER_PASS_DEPTH_RANGE, VIDEO_ENCODERS, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS, MAXIMUM_MEMORY_MIB, BAKE_WORKER_MAXIMUM_SECONDS, PATH_TRACER_CAPABILITIES,
+ VEGETATION_CAPABILITY_NOTE, CROWD_CAPABILITY_NOTE, AGENT_SCENE_CAPABILITIES, AGENT_SCENE_DEFAULT_LIMITS, RENDER_PASS_NAMES, RENDER_PASS_DEPTH_RANGE, VIDEO_ENCODERS, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS, MAXIMUM_MEMORY_MIB, BAKE_WORKER_MAXIMUM_SECONDS, PATH_TRACER_CAPABILITIES,
recipeAssetNeeds, sequenceFrameIds, TEXTURE_SEQUENCE_CAPABILITIES,
audioInputArguments, createAudioLibrary, measureStreamSeconds, filmTimelineDuration, filmTimelineShotStarts, sampleFilmTimeline, normalizeReview, REVIEW_LIMITS, isPreviewHost, BRANCH_LIMITS, validateBranchName,
chromium: { launch: async () => browser }, URL, Buffer, process, performance, clearTimeout,
diff --git a/tests/agent-scene-crowd.test.js b/tests/agent-scene-crowd.test.js
new file mode 100644
index 00000000..8f605e04
--- /dev/null
+++ b/tests/agent-scene-crowd.test.js
@@ -0,0 +1,112 @@
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import test from 'node:test';
+import { parseGltf } from '../src/gltf.js';
+import { createAgentScene, loadAllAgentSceneModules, agentSceneModulesNeeded } from '../src/agent-scene.js';
+import { assetCopyKey, recipeAssetNeeds } from '../src/agent-scene-assets.js';
+import { groupParts } from '../src/renderer.js';
+
+await loadAllAgentSceneModules();
+
+const ASSET_ID = 'ef'.repeat(32);
+const bytes = readFileSync(new URL('./fixtures/RobotExpressive.glb', import.meta.url));
+
+async function loadRobot() {
+ return parseGltf(bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength), { maxVertices: 500000 });
+}
+
+function recipe(crowd, nodes = []) {
+ return {
+ format: '67js.scene', schemaVersion: 1,
+ geometries: [{ id: 'ground', type: 'box' }],
+ materials: [{ id: 'grass', color: [0.2, 0.5, 0.2] }],
+ camera: { eye: [0, 3, 14], target: [0, 1, 0] },
+ nodes: [
+ { id: 'floor', geometry: 'ground', material: 'grass', scale: [40, 0.1, 40] },
+ { id: 'people', asset: { id: ASSET_ID, crowd } },
+ ...nodes,
+ ],
+ };
+}
+
+async function crowdScene(crowd, nodes) {
+ const robot = await loadRobot();
+ const walking = robot.animations.findIndex(animation => animation.name === 'Walking');
+ const settings = { clip: walking, count: 40, seed: 5, spacing: 2, ...crowd };
+ const assets = new Map([[assetCopyKey(ASSET_ID, 0), await loadRobot()], [assetCopyKey(ASSET_ID, 1), robot]]);
+ return createAgentScene(recipe(settings, nodes), {}, { assets });
+}
+
+const crowdBlends = scene => scene.parts.filter(part => part.crowd).map(part => Array.from(part.assembly.offset));
+
+test('a crowd field loads on the crowd marker and reserves one decoded copy', () => {
+ const placed = recipe({ clip: 0, count: 3 });
+ assert.ok(agentSceneModulesNeeded(placed).includes('crowd'));
+ assert.deepEqual(recipeAssetNeeds(placed), [{ id: ASSET_ID, copies: 2 }]);
+});
+
+test('a crowd draws as one batch per material however many people it has, and scene_inspect reports it', async () => {
+ const scene = await crowdScene({ count: 40 });
+ const report = scene.inspect();
+ const crowd = report.assets.find(entry => entry.kind === 'crowd');
+ assert.equal(crowd.instances, 40);
+ assert.equal(crowd.parts, 40 * crowd.batches);
+ assert.equal(groupParts(scene.parts).length, crowd.batches + 1, 'the crowd batches and the ground');
+ assert.ok(crowd.paletteBytes > 0 && crowd.boundsRadius > 0 && crowd.frames > 2);
+ assert.equal(report.nodes.length, 2, 'generated part nodes stay out of the node list');
+ assert.equal(scene.exportRecipe().nodes.length, 2);
+ assert.equal(scene.exportRecipe().nodes[1].asset.crowd.count, 40);
+});
+
+test('a crowd frame reached by seeking equals the one reached by playing, near and far', async () => {
+ const scene = await crowdScene({ count: 30, farDistance: 6, farRate: 4 });
+ scene.sampleMotion(1.7);
+ const sought = crowdBlends(scene);
+ scene.sampleMotion(0);
+ for (let step = 1; step <= 17; step++) scene.sampleMotion(step * 0.1);
+ assert.deepEqual(crowdBlends(scene), sought);
+ scene.sampleMotion(4);
+ scene.sampleMotion(1.7);
+ assert.deepEqual(crowdBlends(scene), sought, 'seeking backwards gives the same frame');
+});
+
+test('far instances change pose only farRate times a second while near ones change every frame', async () => {
+ const scene = await crowdScene({ count: 60, spacing: 2, farDistance: 8, farRate: 2, rateJitter: 0 });
+ const crowdParts = scene.parts.filter(part => part.crowd);
+ const perInstance = crowdParts.length / 60;
+ const distinctPoses = Array.from({ length: 60 }, () => new Set());
+ for (let step = 0; step <= 20; step++) {
+ scene.sampleMotion(1 + step * 0.05);
+ for (let index = 0; index < 60; index++) distinctPoses[index].add(crowdParts[index * perInstance].assembly.offset.join());
+ }
+ let farCount = 0;
+ let nearCount = 0;
+ for (let index = 0; index < 60; index++) {
+ const matrix = crowdParts[index * perInstance].matrix;
+ const far = Math.hypot(matrix[12], matrix[13] - 3, matrix[14] - 14) > 8;
+ if (far) {
+ farCount++;
+ assert.ok(distinctPoses[index].size <= 3, `a far person shows at most 3 poses in one second, not ${distinctPoses[index].size}`);
+ } else {
+ nearCount++;
+ assert.ok(distinctPoses[index].size >= 15, 'a near person changes pose nearly every step');
+ }
+ }
+ assert.ok(farCount > 0 && nearCount > 0, 'the field has near and far people');
+});
+
+test('crowd settings are validated and the clip must exist', async () => {
+ await assert.rejects(crowdScene({ count: 0 }), /count/);
+ await assert.rejects(crowdScene({ bogus: 1 }), /Unknown crowd field/);
+ await assert.rejects(crowdScene({ farDistance: 5 }), /go together/);
+ await assert.rejects(crowdScene({ clip: 99 }), /does not exist/);
+});
+
+test('a crowd keeps its baked parts across an edit that does not touch it', async () => {
+ const scene = await crowdScene({ count: 10 });
+ const before = scene.parts.filter(part => part.crowd);
+ scene.apply([{ type: 'add-node', node: { id: 'marker', translation: [0, 1, 0] } }], { expectedRevision: 0 });
+ const after = scene.parts.filter(part => part.crowd);
+ assert.equal(after.length, before.length);
+ assert.ok(after.every((part, index) => part === before[index]));
+});
diff --git a/tests/crowd-skinning.test.js b/tests/crowd-skinning.test.js
index 8fab76b3..32b98ea6 100644
--- a/tests/crowd-skinning.test.js
+++ b/tests/crowd-skinning.test.js
@@ -6,6 +6,7 @@ import { parseGltf } from '../src/gltf.js';
import { clipFromGltfAnimation, createAssetPose } from '../src/skeleton-pose.js';
import { createCrowd } from '../src/retarget.js';
import { groupParts, Renderer67 } from '../src/renderer.js';
+import { validateMorphGeometry } from '../src/morph.js';
import { InstanceSelection, extractFrustumPlanes, selectInstances } from '../src/render-culling.js';
import {
bakeCrowdPalette, createCrowdLods, createCrowdParts, createCrowdRig, crowdFrameBlend, crowdVertexAt,
@@ -154,3 +155,64 @@ test('the renderer culls crowd instances with the bound of the whole clip, and r
});
assert.equal(selection.visibleCount, 1, 'only the instance in front of the camera draws');
});
+
+test('a crowd reached by seeking equals the same crowd reached by playing, with a lower pose rate for far instances', async () => {
+ const { pose, rig, duration } = await loadWalker();
+ const palette = bakeCrowdPalette(rig, seconds => pose.apply(seconds), { duration });
+ const crowd = createCrowd({ count: 80, clip: { duration, tracks: [] }, seed: 11, spacing: 2 });
+ const eye = [0, 2, 8];
+ const farPose = { distance: 8, rate: 3 };
+ const played = createCrowdParts({ rig, palette, instances: crowd.instances, farPose });
+ const sought = createCrowdParts({ rig, palette, instances: crowd.instances, farPose });
+ const blends = set => set.parts.map(part => part.assembly.offset.slice());
+ for (let step = 0; step <= 40; step++) played.setTimes(index => crowd.clipTime(index, step * 0.05), eye);
+ sought.setTimes(index => crowd.clipTime(index, 2), eye);
+ assert.deepEqual(blends(played), blends(sought));
+ const everyFrame = createCrowdParts({ rig, palette, instances: crowd.instances });
+ everyFrame.setTimes(index => crowd.clipTime(index, 2), eye);
+ const farInstances = crowd.instances.filter(instance => Math.hypot(instance.matrix[12] - eye[0], instance.matrix[13] - eye[1], instance.matrix[14] - eye[2]) > farPose.distance);
+ assert.ok(farInstances.length > 0 && farInstances.length < crowd.instances.length, 'the field has near and far instances');
+ const differing = sought.parts.filter((part, index) => part.assembly.offset.join() !== everyFrame.parts[index].assembly.offset.join());
+ assert.ok(differing.length > 0, 'far instances use the held pose');
+ assert.ok(differing.length <= farInstances.length * rig.parts.length, 'near instances are untouched');
+});
+
+test('a crowd rig keeps UVs through levels and packs joints and weights exactly', () => {
+ const material = { color: [1, 1, 1], map: { name: 'checker' } };
+ const geometry = {
+ positions: new Float32Array([0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0.5, 0.5, 0]),
+ normals: new Float32Array([0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1]),
+ uvs: new Float32Array([0, 0, 1, 0, 1, 1, 0, 1, 0.5, 0.5]),
+ indices: new Uint32Array([0, 1, 4, 1, 2, 4, 2, 3, 4, 3, 0, 4]),
+ };
+ const identity = mat4.create();
+ const rig = createCrowdRig([{ name: 'quad', geometry, matrix: identity, material }, { name: 'other', geometry, matrix: identity, material }]);
+ const merged = rig.parts[0].geometry;
+ assert.equal(merged.uvs.length, 20);
+ assert.deepEqual(Array.from(merged.uvs.slice(10)), Array.from(geometry.uvs));
+ merged.crowdJoints.forEach((joint, index) => {
+ const packed = merged.crowdPacked[index];
+ assert.equal(Math.floor(packed / 4096), joint);
+ assert.equal((packed - joint * 4096) / 4095, merged.crowdWeights[index]);
+ });
+ const levels = createCrowdLods(merged, { cells: [0.5] });
+ assert.equal(levels[1].geometry.uvs.length / 2, levels[1].geometry.positions.length / 3);
+ assert.equal(levels[1].geometry.crowdPacked.length, levels[1].geometry.crowdJoints.length);
+});
+
+test('morph targets stay on their own crowd part when the rig asks for them', async () => {
+ const { asset, pose, duration } = await loadWalker();
+ const morphed = asset.parts.filter(part => part.morph && part.geometry.morphTargets);
+ assert.ok(morphed.length > 0, 'the robot has morph targets');
+ assert.ok(createCrowdRig(asset.parts).parts.every(part => !part.morph), 'morphs are off by default');
+ const withMorphs = createCrowdRig(asset.parts, { morphs: true });
+ const morphParts = withMorphs.parts.filter(part => part.morph);
+ assert.equal(morphParts.length, morphed.length);
+ for (const part of morphParts) {
+ const metadata = validateMorphGeometry(part.geometry, part.morph);
+ assert.equal(metadata.vertexCount, part.geometry.positions.length / 3);
+ }
+ const palette = bakeCrowdPalette(withMorphs, seconds => pose.apply(seconds), { duration });
+ const set = createCrowdParts({ rig: withMorphs, palette, instances: [{ matrix: mat4.create() }] });
+ assert.ok(set.parts.filter(part => part.morph).every(part => !part.lods), 'a morphed part draws at full detail');
+});