Skip to content

The VS Code of video editing, built for agents.

TypeScript

Add `<canvas>` element and related functionality for custom drawing

konstantin-paulusaaafd86eaa55
diff --git a/apps/cli/reference/jsx/README.md b/apps/cli/reference/jsx/README.md
index c2c5b3df..9b149046 100644
--- a/apps/cli/reference/jsx/README.md
+++ b/apps/cli/reference/jsx/README.md
@@ -18,6 +18,7 @@ The pipeline is driven by two commands: [`dapi mount`](../mount.md) renders the
| [scene.md](./scene.md), [group.md](./group.md), [rect.md](./rect.md), [text.md](./text.md), [video.md](./video.md), [image.md](./image.md), [audio.md](./audio.md) | Per-element props |
| [paints.md](./paints.md) | `<solidPaint>`, gradients, `<colorStop>` |
| [html-paint.md](./html-paint.md) | `<html>`: reactive HTML children drawn into the box |
+| [canvas-paint.md](./canvas-paint.md) | `<canvas>`: a ref-provided canvas you draw into, sampled every frame |
| [media.md](./media.md) | `src` resolution (paths, URLs, asset ids, `AssetRef`) |
| [timing.md](./timing.md) | `start` / `end` / `sourceIn` / `sourceOut`, time formats |
| [keyframes.md](./keyframes.md) | Keyframe animation and easing |
diff --git a/apps/cli/reference/jsx/canvas-paint.md b/apps/cli/reference/jsx/canvas-paint.md
new file mode 100644
index 00000000..af32a180
--- /dev/null
+++ b/apps/cli/reference/jsx/canvas-paint.md
@@ -0,0 +1,86 @@
+# `<canvas>`
+
+A paint backed by a **canvas you draw yourself** (long form: `<canvasPaint>`). The `ref` callback receives a detached `HTMLCanvasElement`; draw into it with any context type — 2d, webgl, webgpu — and the engine samples the bitmap every frame, stretching it into the parent geometry's box. Use it for procedural graphics and for external renderers (three.js, p5, chart libraries) that want to own a canvas.
+
+```tsx
+<canvas x={40} y={40} width={640} height={360} cornerRadius={24}
+ ref={(el) => {
+ const ctx = el.getContext("2d")!;
+ ctx.fillStyle = "#111";
+ ctx.fillRect(0, 0, el.width, el.height);
+ ctx.strokeStyle = "#7c9cff";
+ ctx.lineWidth = 6;
+ ctx.strokeRect(40, 40, el.width - 80, el.height - 80);
+ }} />
+```
+
+`<canvasPaint ref={...}>` is the paint child form, valid inside any filled visual element; `<canvas>` is a rectangle carrying one, with all [common props](./elements.md#common-props).
+
+## The ref and the bitmap
+
+- The ref runs **once**, when the paint materializes, inside the mount's reactive owner — `createEffect`, `onCleanup`, and [`useTicker`](./lifecycle.md) all work inside it.
+- **Callback form only** (`ref={(el) => ...}`). Variable refs (`let el; <canvas ref={el} />`) receive a renderer-internal node, not the canvas.
+- The bitmap is allocated at the element's box size (in composition pixels) **before the ref runs**; after that the engine never touches it — the bitmap belongs to your code. Resize it yourself (`el.width = ...`, or an external renderer's own API) for higher resolution; the bitmap is stretched into the box every frame either way, so an animated box scales pixels rather than re-rasterizing.
+- Unlike [`<html>`](./html-paint.md) no flagged browser API is needed, and the sampled pixels render in exports.
+
+## Reactivity
+
+The engine samples the canvas every frame, so anything you draw shows up on the next frame. Under [`dapi mount --live`](../mount.md) the reactive graph stays alive: create effects in the ref to redraw from signals, or drive frame-accurate motion from the ticker's composition time:
+
+```tsx
+<canvas width={400} height={400}
+ ref={(el) => {
+ const ctx = el.getContext("2d")!;
+ const { time } = useTicker();
+ createEffect(() => {
+ ctx.clearRect(0, 0, el.width, el.height);
+ ctx.beginPath();
+ ctx.arc(200, 200, 60 + 40 * Math.sin(time() * Math.PI), 0, Math.PI * 2);
+ ctx.fillStyle = "#44dd88";
+ ctx.fill();
+ });
+ }} />
+```
+
+Because the ticker follows the playhead, ticker-driven drawing stays frame-accurate in exports too; wall-clock timers (`setInterval`, `requestAnimationFrame`) render live but ignore the playhead.
+
+## External renderers
+
+Anything that accepts an existing canvas plugs in directly; a detached canvas is fine for WebGL:
+
+```tsx
+<canvas width={1280} height={720}
+ ref={(el) => {
+ const renderer = new THREE.WebGLRenderer({
+ canvas: el,
+ preserveDrawingBuffer: true,
+ alpha: true,
+ });
+ renderer.setSize(el.width, el.height, false);
+ const { time } = useTicker();
+ createEffect(() => {
+ mesh.rotation.y = time() * 0.5;
+ renderer.render(scene, camera);
+ });
+ onCleanup(() => renderer.dispose());
+ }} />
+```
+
+- **`preserveDrawingBuffer: true` is effectively required for WebGL** — by default the drawing buffer may be cleared after presentation, so the engine's per-frame sample can read back blank.
+- `renderer.setSize(w, h, false)` resizes the bitmap through three.js; the third argument skips CSS sizing, which is meaningless on a detached canvas.
+- Browsers cap live WebGL contexts per page (typically ~16, oldest evicted). One renderer is fine; don't give each of many paints its own GL context — share one renderer and copy frames out via `ImageBitmap` if you need many.
+
+## Props
+
+| Prop | Type | Default | Meaning |
+| ---- | ---- | ------- | ------- |
+| `ref` | `(canvas: HTMLCanvasElement) => void` | **required** | Receives the backing canvas at materialization. |
+| `opacity` | `Animatable<number>` | `1` | Paint opacity, `0`-`1`. |
+
+Like all paints it stacks with siblings in document order and clips to the parent's box (including `cornerRadius`). `<canvasPaint>` takes no children.
+
+## Requirements and limitations
+
+- The drawing is code, so the content lives only as long as its mount: it is not persisted with the document. Exports and captures started while the mount is alive sample the live canvas.
+- Only the `ref` attribute form on the element itself is routed; refs inside spread props are not.
+- `<canvas>` is not available as a DOM tag inside [`<html>`](./html-paint.md) content — in a composition it is always this element.
diff --git a/apps/cli/reference/jsx/elements.md b/apps/cli/reference/jsx/elements.md
index 792708a3..355be01f 100644
--- a/apps/cli/reference/jsx/elements.md
+++ b/apps/cli/reference/jsx/elements.md
@@ -17,6 +17,7 @@ Lowercase intrinsic elements map 1:1 onto internal node types:
| [`<linearGradientPaint>` / `<radialGradientPaint>`](./paints.md) (aliases `<linearGradient>` / `<radialGradient>`) | **Gradient paint** | Paint child; takes `<colorStop>` children. |
| [`<colorStop>`](./paints.md) (alias `<stop>`) | **Gradient color stop** | Valid only inside gradient paints. |
| [`<html>`](./html-paint.md) (long form `<htmlPaint>`) | **Html paint** | Paint child whose children are real, reactive HTML drawn by the browser (html-in-canvas, flagged Chromium API). |
+| [`<canvas>`](./canvas-paint.md) (long form `<canvasPaint>`) | **Canvas paint** | Paint child whose `ref` hands you a canvas to draw with any context type (2d, webgl, webgpu); sampled every frame. |
User-defined components are ordinary Solid components; they compose intrinsics and carry no runtime cost. Only intrinsic elements produce entities.
diff --git a/apps/cli/reference/jsx/paints.md b/apps/cli/reference/jsx/paints.md
index 5e4327a8..ffe7e970 100644
--- a/apps/cli/reference/jsx/paints.md
+++ b/apps/cli/reference/jsx/paints.md
@@ -20,8 +20,9 @@ Paint elements are valid inside any filled visual element (`<rect>`, `<scene>`,
| `<radialGradientPaint>` | `<radialGradient>` | `rotation`, `opacity` | Radial gradient centered in the parent's box. |
| `<colorStop>` | `<stop>` | `offset` (**required**, `0`-`1`), `color` (**required**), `opacity` | Gradient color stop. Valid only inside gradient paints, which take no other children. |
| [`<html>`](./html-paint.md) | `<htmlPaint>` | `opacity`, HTML children | Reactive HTML laid out and drawn into the parent's box (flagged Chromium API). |
+| [`<canvas>`](./canvas-paint.md) | `<canvasPaint>` | `opacity`, `ref` (**required**) | A canvas your `ref` callback draws into (any context type), sampled into the parent's box every frame. |
-Both names of a pair work everywhere; names shared with SVG (`<linearGradient>`, `<stop>`) are unambiguous because tags resolve by environment: inside an [`<html>`](./html-paint.md) paint they are the real SVG elements, anywhere else the paints.
+Both names of a pair work everywhere; names shared with SVG (`<linearGradient>`, `<stop>`) are unambiguous because tags resolve by environment: inside an [`<html>`](./html-paint.md) paint they are the real SVG elements, anywhere else the paints. `<html>` and `<canvas>` themselves are never DOM tags — in a composition they always mean these elements.
Colors accept any CSS color; alpha is ignored (use `opacity`). `color`, `opacity`, and `offset` are animatable (see [keyframes.md](./keyframes.md)), so gradients can animate. Paints have no spatial or timing props and cannot be document roots.
diff --git a/apps/web/src/components/engine/api/observers.ts b/apps/web/src/components/engine/api/observers.ts
index 90a1353f..cd436922 100644
--- a/apps/web/src/components/engine/api/observers.ts
+++ b/apps/web/src/components/engine/api/observers.ts
@@ -482,6 +482,7 @@ export function observeWorld(world: EngineWorld) {
utils.reactToChildDetached(world, eid);
utils.disposeDecoders(world, eid);
utils.disposeHtmlHosts(world, eid);
+ utils.disposeCanvasHosts(world, eid);
utils.disconnectAudioBus(world, eid);
utils.unpersistEntity(world, eid);
world.rebuildTimelineIndex();
diff --git a/apps/web/src/components/engine/api/query.ts b/apps/web/src/components/engine/api/query.ts
index 559b9b55..85e401de 100644
--- a/apps/web/src/components/engine/api/query.ts
+++ b/apps/web/src/components/engine/api/query.ts
@@ -75,6 +75,11 @@ export function hasHtmlPaint(world: EngineWorld, eid: number): boolean {
return (c.Cache.fills[eid] ?? []).some(fid => c.Paint[fid] === PaintType.HTML);
}
+export function hasCanvasPaint(world: EngineWorld, eid: number): boolean {
+ const c = world.components;
+ return (c.Cache.fills[eid] ?? []).some(fid => c.Paint[fid] === PaintType.CANVAS);
+}
+
export function isMask(world: EngineWorld, eid: number): boolean {
return hasComponent(world, eid, world.components.IsMask);
}
diff --git a/apps/web/src/components/engine/api/utils.ts b/apps/web/src/components/engine/api/utils.ts
index 6bd1a06b..bb3e8367 100644
--- a/apps/web/src/components/engine/api/utils.ts
+++ b/apps/web/src/components/engine/api/utils.ts
@@ -160,6 +160,17 @@ export function disposeHtmlHosts(world: EngineWorld, eid: number): void {
}
}
+export function disposeCanvasHosts(world: EngineWorld, eid: number): void {
+ const CanvasHost = world.components.CanvasHost;
+
+ CanvasHost[eid]?.dispose();
+ CanvasHost[eid] = null;
+
+ for (const child of query(world, [ChildOf(eid)])) {
+ disposeCanvasHosts(world, child);
+ }
+}
+
export function disconnectAudioBus(world: EngineWorld, eid: number): void {
const AudioBus = world.components.AudioBus;
AudioBus[eid]?.disconnect();
diff --git a/apps/web/src/components/engine/components/components.ts b/apps/web/src/components/engine/components/components.ts
index 5f15e479..f934507f 100644
--- a/apps/web/src/components/engine/components/components.ts
+++ b/apps/web/src/components/engine/components/components.ts
@@ -7,6 +7,7 @@ import type { ContainerFormat } from "../encode/types";
import type { Token } from "../utils/text";
import type { AudioBus as AudioBusInstance } from "../services/audio-bus";
import type { HtmlHost as HtmlHostInstance } from "../decoders/html";
+import type { CanvasHost as CanvasHostInstance } from "../decoders/canvas";
import type { ImageDecoder as ImageDecoderInstance } from "../decoders/image";
import type { VideoDecoderInstance } from "../decoders/video";
import type { SequenceDecoder as SequenceDecoderInstance } from "../decoders/sequence";
@@ -362,6 +363,8 @@ export const Cache = {
export const HtmlHost = [] as (HtmlHostInstance | null)[];
+export const CanvasHost = [] as (CanvasHostInstance | null)[];
+
export const ImageDecoder = [] as (ImageDecoderInstance | null)[];
export const VideoDecoder = [] as (VideoDecoderInstance | null)[];
diff --git a/apps/web/src/components/engine/components/constants.ts b/apps/web/src/components/engine/components/constants.ts
index b8551719..d5ac573f 100644
--- a/apps/web/src/components/engine/components/constants.ts
+++ b/apps/web/src/components/engine/components/constants.ts
@@ -66,6 +66,7 @@ export enum PaintType {
SEQUENCE,
WAVEFORM,
HTML,
+ CANVAS,
}
export enum EffectType {
diff --git a/apps/web/src/components/engine/decoders/canvas.ts b/apps/web/src/components/engine/decoders/canvas.ts
new file mode 100644
index 00000000..cca5b62e
--- /dev/null
+++ b/apps/web/src/components/engine/decoders/canvas.ts
@@ -0,0 +1,63 @@
+/* This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
+
+import { addComponent } from '../api/events';
+
+import type { EngineWorld } from '../api/world';
+
+type Ctx2D = CanvasRenderingContext2D | OffscreenCanvasRenderingContext2D;
+
+/**
+ * Backing surface for one canvas paint: a detached canvas handed to the
+ * mount's `ref` callback, which draws into it with any context type (2d,
+ * webgl, webgpu). The engine samples the bitmap every frame and stretches it
+ * into the parent geometry's box. After the initial box-sized allocation the
+ * bitmap belongs to user code — external renderers (e.g. three.js via
+ * `renderer.setSize`) resize it through their own APIs, so the engine never
+ * touches it again.
+ */
+export class CanvasHost {
+ public readonly canvas = document.createElement('canvas');
+ private disposed = false;
+
+ /** Initial bitmap allocation — called once at materialization, before the ref runs. */
+ public setSize(width: number, height: number): void {
+ this.canvas.width = Math.max(1, Math.round(width));
+ this.canvas.height = Math.max(1, Math.round(height));
+ }
+
+ public draw(ctx: Ctx2D, width: number, height: number): void {
+ if (this.disposed || this.canvas.width === 0 || this.canvas.height === 0) return;
+ ctx.drawImage(this.canvas, 0, 0, width, height);
+ }
+
+ public dispose(): void {
+ if (this.disposed) return;
+ this.disposed = true;
+ // Release the backing store; context teardown (e.g. a WebGL renderer)
+ // is the mount's own onCleanup concern.
+ this.canvas.width = 0;
+ this.canvas.height = 0;
+ }
+}
+
+/**
+ * Shares <canvasPaint> hosts with a cloned (offline) world. The backing
+ * canvas is runtime state serializeEntity cannot carry, so export/capture
+ * worlds reference the live hosts directly and sample whatever the mount's
+ * ref last drew while encoding. The source world keeps ownership; offline
+ * worlds must not dispose them.
+ */
+export function shareCanvasHosts(
+ sourceWorld: EngineWorld,
+ world: EngineWorld,
+ eidMap: Map<number, number>,
+) {
+ for (const [sourceEid, targetEid] of eidMap) {
+ const host = sourceWorld.components.CanvasHost[sourceEid];
+ if (!host) continue;
+ addComponent(world, targetEid, world.components.CanvasHost, false);
+ world.components.CanvasHost[targetEid] = host;
+ }
+}
diff --git a/apps/web/src/components/engine/encode/encoder.ts b/apps/web/src/components/engine/encode/encoder.ts
index d803faec..0ca75544 100644
--- a/apps/web/src/components/engine/encode/encoder.ts
+++ b/apps/web/src/components/engine/encode/encoder.ts
@@ -25,6 +25,7 @@ import { motionSystem } from '../systems/motion';
import { transformSystem } from '../systems/transform';
import { renderSystem } from '../systems/render';
import { shareHtmlHosts } from '../decoders/html';
+import { shareCanvasHosts } from '../decoders/canvas';
import { cloneFromRecords, serializeEntity } from '../api/serialize';
import { getEntityTree } from '../api/query';
import { AudioBus } from '../services/audio-bus';
@@ -134,6 +135,7 @@ export async function createEncoder(sourceWorld: EngineWorld, config: EncoderCon
if (videoEnabled) {
shareHtmlHosts(sourceWorld, world, eidMap);
+ shareCanvasHosts(sourceWorld, world, eidMap);
}
// Set up mediabunny output
diff --git a/apps/web/src/components/engine/encode/image-encoder.ts b/apps/web/src/components/engine/encode/image-encoder.ts
index 54ce2bd6..7710dd1c 100644
--- a/apps/web/src/components/engine/encode/image-encoder.ts
+++ b/apps/web/src/components/engine/encode/image-encoder.ts
@@ -16,6 +16,7 @@ import { transformSystem } from '../systems/transform';
import { renderSystem } from '../systems/render';
import { cloneFromRecords, serializeEntity } from '../api/serialize';
import { shareHtmlHosts } from '../decoders/html';
+import { shareCanvasHosts } from '../decoders/canvas';
import { getEntityTree } from '../api/query';
import { framesToSeconds, formatTimestamp, stampTimestampLabel } from '../utils';
import { assert } from '@/utils';
@@ -84,6 +85,7 @@ export async function createImageEncoder(sourceWorld: EngineWorld, config: Image
assert(rootEid !== undefined, 'Failed to clone entity subtree');
shareHtmlHosts(sourceWorld, world, eidMap);
+ shareCanvasHosts(sourceWorld, world, eidMap);
const clonedEids = [...eidMap.values()];
const c = world.components;
diff --git a/apps/web/src/components/engine/systems/render.ts b/apps/web/src/components/engine/systems/render.ts
index 1bac052e..cc76f498 100644
--- a/apps/web/src/components/engine/systems/render.ts
+++ b/apps/web/src/components/engine/systems/render.ts
@@ -360,6 +360,15 @@ export function renderFills(world: EngineWorld, eid: number): void {
} else if (c.Paint[fid] === PaintType.HTML) {
const host = c.HtmlHost[fid];
+ if (host) {
+ ctx.save();
+ ctx.clip();
+ host.draw(ctx, c.Computed.width[eid], c.Computed.height[eid]);
+ ctx.restore();
+ }
+ } else if (c.Paint[fid] === PaintType.CANVAS) {
+ const host = c.CanvasHost[fid];
+
if (host) {
ctx.save();
ctx.clip();
diff --git a/apps/web/src/components/engine/timeline/render/clip.ts b/apps/web/src/components/engine/timeline/render/clip.ts
index 002c4c50..ea0ec218 100644
--- a/apps/web/src/components/engine/timeline/render/clip.ts
+++ b/apps/web/src/components/engine/timeline/render/clip.ts
@@ -9,7 +9,7 @@ import { CLIP_BREAKPOINTS, CLIP_CORNER_RADIUS, CLIP_FONT, CLIP_LABEL_HEIGHT, CLI
import { renderImageThumbnails } from "./image";
import { renderAudioWaveform } from "./audio";
import { renderVideoThumbnails } from "./video";
-import { getParentEntity, hasHtmlPaint, isAdjustmentLayer, isCaption, isGroup, isMask, isScene, isText } from "../../api";
+import { getParentEntity, hasCanvasPaint, hasHtmlPaint, isAdjustmentLayer, isCaption, isGroup, isMask, isScene, isText } from "../../api";
import { renderCaption } from "./caption";
import { hasComponent, Not, Or, query } from "bitecs";
import { addComponent, clearComponent, removeComponent, setComponent } from "../../api/events";
@@ -330,7 +330,7 @@ export function getClipStyle(
return timeline.colors.clip.adjustment;
}
- if (hasHtmlPaint(world, eid)) {
+ if (hasHtmlPaint(world, eid) || hasCanvasPaint(world, eid)) {
return timeline.colors.clip.html;
}
diff --git a/apps/web/src/components/sidebar-right/inspector/fill-row.tsx b/apps/web/src/components/sidebar-right/inspector/fill-row.tsx
index fe3901e9..f609d789 100644
--- a/apps/web/src/components/sidebar-right/inspector/fill-row.tsx
+++ b/apps/web/src/components/sidebar-right/inspector/fill-row.tsx
@@ -34,6 +34,7 @@ const FILL_TYPE_LABELS: Record<PaintType, string> = {
[PaintType.RADIAL_GRADIENT]: 'Gradient',
[PaintType.WAVEFORM]: 'Waveform',
[PaintType.HTML]: 'Html',
+ [PaintType.CANVAS]: 'Canvas',
};
export function FillRow(props: FillRowProps) {
diff --git a/apps/web/src/context/editor-api/selection.ts b/apps/web/src/context/editor-api/selection.ts
index ca021fe7..cbd97d06 100644
--- a/apps/web/src/context/editor-api/selection.ts
+++ b/apps/web/src/context/editor-api/selection.ts
@@ -55,6 +55,7 @@ export function entityType(world: EngineWorld, eid: number): string {
case PaintType.IMAGE: return "imagePaint";
case PaintType.VIDEO: return "videoPaint";
case PaintType.HTML: return "htmlPaint";
+ case PaintType.CANVAS: return "canvasPaint";
default: return "paint";
}
}
diff --git a/apps/web/src/utils/jsx.ts b/apps/web/src/utils/jsx.ts
index a245d955..7051d799 100644
--- a/apps/web/src/utils/jsx.ts
+++ b/apps/web/src/utils/jsx.ts
@@ -38,6 +38,7 @@ import {
transcriptTrim,
} from "@/components/engine";
import { HtmlHost, isHtmlInCanvasSupported } from "@/components/engine/decoders/html";
+import { CanvasHost } from "@/components/engine/decoders/canvas";
import { resolveTranscript } from "@/components/engine/decoders/caption/utils";
import { ChildOf } from "@/components/engine/components";
import { ASPECT_RATIO_DIMENSIONS, findEmptyPlacement } from "@/utils/genai";
@@ -65,6 +66,8 @@ type DocumentNode = {
eid?: number;
dom?: Element | Text;
host?: HtmlHost;
+ canvasHost?: CanvasHost;
+ ref?(target: unknown): void;
};
let nextNodeId = 1;
@@ -583,6 +586,26 @@ export class WorldDocument implements ProjectDocument<DocumentNode> {
appendChild(world, fid, eid);
node.host = host;
break;
+ } case "canvasPaint": {
+ setComponent(world, eid, c.Paint, PaintType.CANVAS);
+ const host = new CanvasHost();
+ addComponent(world, eid, c.CanvasHost, false);
+ c.CanvasHost[eid] = host;
+ node.canvasHost = host;
+ break;
+ } case "canvas": {
+ // A rect carrying a canvas paint; the node's ref receives the
+ // paint's backing canvas, so `node.canvasHost` points at it.
+ setComponent(world, eid, c.Geometry, GeometryType.RECT);
+ setComponent(world, eid, c.Name, getNextName(world, "Canvas"));
+ const fid = createEntity(world);
+ setComponent(world, fid, c.Paint, PaintType.CANVAS);
+ const host = new CanvasHost();
+ addComponent(world, fid, c.CanvasHost, false);
+ c.CanvasHost[fid] = host;
+ appendChild(world, fid, eid);
+ node.canvasHost = host;
+ break;
} default: {
assert(false, `<${node.tag}> is only valid inside <html> content`);
}
@@ -1155,6 +1178,54 @@ export class WorldDocument implements ProjectDocument<DocumentNode> {
}
}
+ /**
+ * Records a `ref` callback (the renderer's `use`). Refs run at
+ * materialization, when the backing object exists — for canvas elements
+ * the paint's canvas, for DOM nodes under an <htmlPaint> the element.
+ */
+ public applyRef(node: DocumentNode, ref: (target: unknown) => void): void {
+ node.ref = ref;
+ }
+
+ /** Invokes and consumes a node's recorded ref, outside the current tracking scope. */
+ private invokeRef(node: DocumentNode, target: unknown): void {
+ const ref = node.ref;
+ if (ref === undefined) return;
+ node.ref = undefined;
+ solid.untrack(() => ref(target));
+ }
+
+ /**
+ * The box a canvas paint's bitmap is initially allocated at: the nearest
+ * explicit size up the shadow tree (every element's box defaults to its
+ * parent's), falling back to a live ancestor's computed size for mounts
+ * into existing entities (`dapi node insert`).
+ */
+ private resolveBoxSize(node: DocumentNode): { width: number; height: number } {
+ const c = this.world.components;
+ let width: number | undefined;
+ let height: number | undefined;
+
+ // Recorded props first — entities in the materializing subtree are fresh,
+ // so their Computed sizes are defaults, not layout. Live ancestors'
+ // Computed only backstops mounts into existing entities (node insert).
+ for (let cursor: DocumentNode | undefined = node; cursor !== undefined; cursor = cursor.parent) {
+ width ??= sizeProp(cursor.props.width);
+ height ??= sizeProp(cursor.props.height);
+ if (width !== undefined && height !== undefined) return { width, height };
+ }
+ for (let cursor: DocumentNode | undefined = node; cursor !== undefined; cursor = cursor.parent) {
+ // Freshly created entities have an eid but no Size yet; only already
+ // live ones (e.g. the target of a `node insert`) carry a real box.
+ if (cursor.eid === undefined || !hasComponent(this.world, cursor.eid, c.Size)) continue;
+ width ??= c.Computed.width[cursor.eid] || undefined;
+ height ??= c.Computed.height[cursor.eid] || undefined;
+ if (width !== undefined && height !== undefined) break;
+ }
+
+ return { width: width ?? 300, height: height ?? 150 };
+ }
+
/**
* Creates the entity for an element and replays its shadow state: tag
* components, children (attached while the subtree is still detached from
@@ -1164,6 +1235,13 @@ export class WorldDocument implements ProjectDocument<DocumentNode> {
node.eid = this.createEntityForTag(node);
for (const child of node.children) this.attach(node, child);
for (const [name, value] of Object.entries(node.props)) this.applyEcsProperty(node, name, value);
+
+ if (node.canvasHost !== undefined) {
+ const box = this.resolveBoxSize(node.tag === "canvasPaint" ? node.parent ?? node : node);
+ node.canvasHost.setSize(box.width, box.height);
+ this.invokeRef(node, node.canvasHost.canvas);
+ }
+ assert(node.ref === undefined, `\`ref\` is not supported on <${node.tag}>`);
}
/** Builds the DOM subtree for a node under an <htmlPaint>, props applied. */
@@ -1181,6 +1259,7 @@ export class WorldDocument implements ProjectDocument<DocumentNode> {
el.appendChild(child.dom!);
}
for (const [name, value] of Object.entries(node.props)) setDomProperty(el, name, value);
+ this.invokeRef(node, el);
}
public removeNode(parent: DocumentNode, node: DocumentNode) {
@@ -1306,6 +1385,17 @@ function parseFrames(value: string | null | undefined, fps: number): number | un
return seconds === undefined ? undefined : Math.round(seconds * fps);
}
+/**
+ * Reads a recorded width/height prop as a number; keyframed sizes read as their first keyframe.
+ */
+function sizeProp(value: unknown): number | undefined {
+ if (typeof value === "number") return value;
+ if (Array.isArray(value) && typeof (value[0] as { value?: unknown })?.value === "number") {
+ return (value[0] as { value: number }).value;
+ }
+ return undefined;
+}
+
/**
* Evaluates a compiled project module. The CLI ships a
* single-file ESM bundle whose host modules are left external under the
diff --git a/docs/composition.md b/docs/composition.md
index f3b38fb2..e5348040 100644
--- a/docs/composition.md
+++ b/docs/composition.md
@@ -39,6 +39,7 @@ The first rendered root becomes the active scene. New roots are auto-placed on t
| `<sequence>` | Track container for back-to-back clips (positions are explicit); hosts transitions. |
| `<captions>` | Styled captions: transcribes the scene's audio, or mounts a transcript given via `src`. |
| `<html>` | A rectangle whose children are HTML/SVG, laid out by the browser at the node's box size and drawn into it. Equivalent to a `<rect>` with an `<htmlPaint>`. |
+| `<canvas>` | A rectangle whose `ref` hands you a canvas to draw yourself (2d, webgl, webgpu); the bitmap is sampled into the box every frame. Equivalent to a `<rect>` with a `<canvasPaint>`. |
| `<solidPaint>`, `<linearGradientPaint>`, `<radialGradientPaint>`, `<colorStop>` (aliases `<solid>`, `<linearGradient>`, `<radialGradient>`, `<stop>`) | Fills, declared as children; see [Paints](#paints). |
User-defined components are ordinary Solid components; only intrinsic elements produce nodes.
@@ -74,6 +75,7 @@ Animatable props also take a keyframe list — see [Animation](#animation).
- **`<audio>`**: `src` (required), `volume` (dB; `0` = unity, negative attenuates), `muted`, `syncTo`, timing props.
- **`<text>`**: string children (required), `fontFamily` (see `dapi fonts`), `fontSize`, `fontWeight`, `fontStyle`, `fill`, `textAlign`, `textBaseline`.
- **`<html>`**: HTML/SVG children. Fully reactive: signals in attributes, styles, and text update the drawn content. Event handlers are dropped (the content is painted, not interactive).
+- **`<canvas>`**: `ref` (required, callback form) receives the backing `HTMLCanvasElement` once at materialization; draw with any context type, create effects inside the ref to redraw from signals or the ticker.
These same props are what `dapi node patch` assigns on existing nodes.
@@ -182,7 +184,7 @@ A node's fill is a **paint child**; the `fill` prop is shorthand for a solid pai
</rect>
```
-`<solidPaint>` takes `color` (required) and `opacity`; gradient paints take `rotation` and `opacity` and only `<colorStop>` children (`offset` and `color` required). Colors are any CSS color; alpha is ignored, use `opacity`. Each paint tag has a short alias (`<solid>`, `<linearGradient>`, `<radialGradient>`, `<stop>`); the SVG name overlap is unambiguous because tags resolve by environment. `<htmlPaint>` draws browser-laid-out HTML/SVG children into the node's box; the `<html>` element is shorthand for a `<rect>` carrying one.
+`<solidPaint>` takes `color` (required) and `opacity`; gradient paints take `rotation` and `opacity` and only `<colorStop>` children (`offset` and `color` required). Colors are any CSS color; alpha is ignored, use `opacity`. Each paint tag has a short alias (`<solid>`, `<linearGradient>`, `<radialGradient>`, `<stop>`); the SVG name overlap is unambiguous because tags resolve by environment. `<htmlPaint>` draws browser-laid-out HTML/SVG children into the node's box; the `<html>` element is shorthand for a `<rect>` carrying one. `<canvasPaint>` draws a canvas your `ref` callback owns into the node's box; the `<canvas>` element is likewise shorthand for a `<rect>` carrying one.
## Generated assets
diff --git a/packages/jsx/src/document.ts b/packages/jsx/src/document.ts
index 6768ca52..f0244a81 100644
--- a/packages/jsx/src/document.ts
+++ b/packages/jsx/src/document.ts
@@ -40,4 +40,5 @@ export interface ProjectDocument<N = unknown> {
getFirstChild(node: N): N | undefined;
getNextSibling(node: N): N | undefined;
tick?(): ProjectTick;
+ applyRef?(node: N, ref: (target: unknown) => void): void;
}
diff --git a/packages/jsx/src/index.ts b/packages/jsx/src/index.ts
index 01aab8d6..f56fae2e 100644
--- a/packages/jsx/src/index.ts
+++ b/packages/jsx/src/index.ts
@@ -22,6 +22,8 @@ export type {
AnimationSpec,
AnimationType,
AudioProps,
+ CanvasPaintProps,
+ CanvasProps,
CaptionPreset,
CaptionsProps,
ColorStopProps,
diff --git a/packages/jsx/src/jsx-runtime.ts b/packages/jsx/src/jsx-runtime.ts
index f5f1e856..6f2f1f03 100644
--- a/packages/jsx/src/jsx-runtime.ts
+++ b/packages/jsx/src/jsx-runtime.ts
@@ -12,6 +12,8 @@
import type { JSX as SolidJSX } from "solid-js";
import type {
AudioProps,
+ CanvasPaintProps,
+ CanvasProps,
CaptionsProps,
ColorStopProps,
GradientPaintProps,
@@ -27,9 +29,10 @@ import type {
VideoProps,
} from "./types";
-// "html" is dropped alongside the media tags: in composition context it is
-// the html node element, and as a DOM tag it is useless inside a paint host.
-type HtmlElementTags = Omit<SolidJSX.HTMLElementTags, "audio" | "video" | "html">;
+// "html" and "canvas" are dropped alongside the media tags: in composition
+// context they are node elements, and as DOM tags they are useless inside a
+// paint host (a DOM canvas's content doesn't survive drawElementImage).
+type HtmlElementTags = Omit<SolidJSX.HTMLElementTags, "audio" | "video" | "html" | "canvas">;
// SVG vocabulary for <htmlPaint> content. Tags the editor also declares keep
// their composition types below; the runtime resolves the actual meaning by
@@ -68,5 +71,7 @@ export declare namespace JSX {
stop: ColorStopProps;
htmlPaint: HtmlPaintProps;
html: HtmlProps;
+ canvasPaint: CanvasPaintProps;
+ canvas: CanvasProps;
}
}
diff --git a/packages/jsx/src/renderer.ts b/packages/jsx/src/renderer.ts
index 6d52721b..5b657656 100644
--- a/packages/jsx/src/renderer.ts
+++ b/packages/jsx/src/renderer.ts
@@ -68,9 +68,23 @@ export const {
spread,
setProp,
mergeProps,
- use,
} = renderer;
+/**
+ * Ref application (compiled `ref={fn}`). Hosts that materialize elements
+ * lazily route refs through `applyRef` so the callback receives the backing
+ * object (e.g. a canvas paint's canvas) once it exists; hosts without
+ * `applyRef` keep the renderer's immediate call with the host node.
+ */
+export function use(fn: (target: unknown, arg?: unknown) => void, node: unknown, arg?: unknown): unknown {
+ const document = doc();
+ if (document.applyRef) {
+ document.applyRef(node, fn);
+ return node;
+ }
+ return renderer.use(fn, node, arg);
+}
+
export type Ticker = {
time: Accessor<number>;
frame: Accessor<number>;
diff --git a/packages/jsx/src/types.ts b/packages/jsx/src/types.ts
index 206596f3..f50dc563 100644
--- a/packages/jsx/src/types.ts
+++ b/packages/jsx/src/types.ts
@@ -298,6 +298,25 @@ export type HtmlPaintProps = Pick<PatchProps, "opacity"> & {
/** `<html>` — a rectangle carrying an `<htmlPaint>` with the given children. */
export type HtmlProps = CommonProps & Pick<HtmlPaintProps, "children">;
+// HTMLCanvasElement without requiring the DOM lib (this package also
+// type-checks in node contexts): the real type when present, a structural
+// stub otherwise.
+type HostCanvas = typeof globalThis extends { HTMLCanvasElement: new () => infer T } ? T
+ : { width: number; height: number; getContext(contextId: string, options?: unknown): unknown };
+
+export type CanvasPaintProps = Pick<PatchProps, "opacity"> & {
+ /**
+ * Receives the paint's backing canvas once the paint materializes. Draw to
+ * any context type (2d, webgl, webgpu); the engine samples the bitmap every
+ * frame and stretches it into the parent geometry's box. Callback form
+ * only — the renderer has no element to assign to a variable ref.
+ */
+ ref: (canvas: HostCanvas) => void;
+};
+
+/** `<canvas>` — a rectangle carrying a `<canvasPaint>` with the given ref. */
+export type CanvasProps = CommonProps & Pick<CanvasPaintProps, "ref">;
+
export type AudioProps = TimingProps &
Required<Pick<PatchProps, "src">> &
Pick<PatchProps, "name" | "volume" | "muted" | "syncTo" | "animations">;
diff --git a/samples/reconciler/08-canvas-paint.tsx b/samples/reconciler/08-canvas-paint.tsx
new file mode 100644
index 00000000..09ac339a
--- /dev/null
+++ b/samples/reconciler/08-canvas-paint.tsx
@@ -0,0 +1,51 @@
+/* @jsxImportSource @diffusionstudio/jsx */
+/* Ref-provided canvas: the <canvas> element hands its backing bitmap to the
+ * ref callback, which owns it from then on.
+ *
+ * dapi mount --live samples/reconciler/08-canvas-paint.tsx
+ *
+ * The ref runs once at materialization, inside the mount's reactive owner:
+ * the effect below redraws from a signal and the playhead ticker, and the
+ * engine samples the bitmap into the node's box every frame. The standalone
+ * <canvasPaint> on the right stacks over a solid fill inside a plain <rect>.
+ */
+
+import { createEffect, createSignal } from "solid-js";
+import { useTicker } from "@diffusionstudio/jsx";
+
+export default () => {
+ const { time } = useTicker();
+ const [canvas, setCanvas] = createSignal<HTMLCanvasElement>();
+
+ createEffect(() => {
+ const cvs = canvas();
+ const ctx = cvs?.getContext("2d");
+ if (!ctx || !cvs) return;
+ ctx.clearRect(0, 0, cvs.width, cvs.height);
+ ctx.fillStyle = "#1b1b22";
+ ctx.fillRect(0, 0, cvs.width, cvs.height);
+ ctx.beginPath();
+ ctx.arc(
+ cvs.width / 2 + Math.cos(time() * 2) * 140,
+ cvs.height / 2 + Math.sin(time() * 2) * 120,
+ 40,
+ 0,
+ Math.PI * 2,
+ );
+ ctx.fillStyle = "#44dd88";
+ ctx.fill();
+ });
+
+ return (
+ <scene key="sample-canvas" name="Canvas paint" width={960} height={540} fill="#101014">
+ <canvas
+ x={40}
+ y={60}
+ width={520}
+ height={420}
+ cornerRadius={16}
+ ref={setCanvas}
+ />
+ </scene>
+ );
+};
diff --git a/samples/reconciler/README.md b/samples/reconciler/README.md
index f6da5c14..052d94ab 100644
--- a/samples/reconciler/README.md
+++ b/samples/reconciler/README.md
@@ -12,6 +12,7 @@ Run from the repo root with the editor open:
| [05-html-paint.tsx](05-html-paint.tsx) | `dapi mount --live samples/reconciler/05-html-paint.tsx` | DOM/SVG materialization under \<html\>; needs html-in-canvas |
| [06-anime-timeline.tsx](06-anime-timeline.tsx) | `dapi mount --live samples/reconciler/06-anime-timeline.tsx` | anime.js timeline seeked from useTicker, driving an ECS node and \<html\> content in lockstep |
| [07-anime-mini.tsx](07-anime-mini.tsx) | `dapi mount --live samples/reconciler/07-anime-mini.tsx` | minimal anime.js + useTicker sync, sized for a social post |
+| [08-canvas-paint.tsx](08-canvas-paint.tsx) | `dapi mount --live samples/reconciler/08-canvas-paint.tsx` | ref-provided canvas: \<canvas\> element and \<canvasPaint\> child, effect-driven redraws |
Each root scene carries a stable key, so re-mounting a sample replaces its
scene in place. Inspect results with `dapi node tree`; entity ids in sample 03