Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Add scripted Motion documents and agent commands

Bittid0e5189368a3
diff --git a/src-tauri/assets/agent-motion.md b/src-tauri/assets/agent-motion.md
new file mode 100644
index 0000000..4330040
--- /dev/null
+++ b/src-tauri/assets/agent-motion.md
@@ -0,0 +1,28 @@
+## Motions
+
+Motions are inline procedural graphics assets, created and edited with JavaScript ES modules. No compilation, package installation or external libraries. An empty Motion can already be present because the user clicked Add Motion. Populate that asset with `motion.set`; do not create a duplicate unless asked. The full source and user controls appear in the timeline inspector.
+
+Commands (ordinary JSONL project commands followed by commit):
+
+- `{"op":"motion.add","id":"motion-title","name":"Title","motion":DOCUMENT}` creates an asset.
+- `{"op":"motion.set","id":"motion-title","name":"Optional new name","motion":DOCUMENT}` replaces its document. Omit name or motion to retain it.
+- `{"op":"clip.add","id":"clip-title","asset":"motion-title","track":"...","at":0,"seconds":5}` places an asset on the timeline. An empty Motion created in the UI is already placed. Reuse its existing clip.
+
+DOCUMENT shape:
+```json
+{"version":1,"space":"2d","width":1280,"height":720,"duration":5,"seed":1,"source":"export function update({ time, parameters, nodes }, draw) { const title = nodes.find(n => n.id === 'title'); title.position[0] += Math.sin(time * parameters.speed) * parameters.distance; }","nodes":[{"id":"title","name":"Title","kind":"text","position":[100,200,0],"width":900,"height":100,"text":"Hello","fontSize":72,"fill":"#ffffff"}],"parameters":[{"id":"speed","label":"Speed","type":"number","value":2,"min":0,"max":10},{"id":"distance","label":"Travel distance","type":"number","value":100,"min":0,"max":500}],"overrides":{}}
+```
+
+Document dimensions: integers 32..4096. Duration: 0.1..600 seconds. IDs start with an ASCII letter, use letters/digits/underscore/hyphen, maximum 80 characters. Up to 512 named nodes and 100 parameters. Source limit 200000 characters. `space:"2d"` is the only currently supported space; do not claim 3D support.
+
+Each node has id, name, kind (`rect`, `ellipse`, `text`, `path`). Optional defaults: position [0,0,0], rotation [0,0,0] in degrees, scale [1,1,1], width/height 100, opacity 1, visible true, fill #ffffff, stroke transparent, strokeWidth 0, text empty, fontSize 48, fontFamily sans-serif (or serif/monospace), path empty. Position is the upper-left anchor in document pixels. Ellipses fit the width/height rectangle. Paths use SVG path syntax. Text uses a top baseline and is constrained by width. Set height to a useful selection-box height. Version 1 uses only position x/y, rotation z and scale x/y. Reserve other axes for future 3D; keep them at identity values.
+
+Export `function update(context, draw)`. It must be synchronous and return nothing. Context includes time (seconds, clamped to duration), duration, width, height, parameters (id -> value), nodes (a fresh mutable copy), and random() (seeded identically on each evaluation). Update nodes directly. `draw.rect/ellipse/text/path({...nodeProperties})` appends temporary primitives; give IDs if needed, and do not duplicate existing IDs. Total output limit 1024 nodes. Named scene nodes are selectable/editable in the video view; prefer them over temporary primitives for user-editable elements.
+
+Compute the result from time and parameters, independent of previous frames. Avoid mutable module state, Date, Math.random, timers, fetch, DOM, imports, async code, and direct Canvas/WebGPU calls. The runtime handles rendering and frame scheduling. Use context.random for stable procedural variation. Pure helpers at module scope are fine.
+
+Parameter types: number, string, color, boolean. Every parameter needs id, label, type, value. Numbers can specify min, max, positive step. Expose useful movement/timing controls instead of hardcoding everything in source.
+
+`overrides` maps named node IDs to user adjustments: offset [x,y,0], scale [x,y,1], rotation [0,0,z], optional fill/text. These apply after script evaluation. Preserve existing overrides and parameter values when revising a Motion unless the requested change requires altering them. Keep IDs stable. Remove an override only when removing its corresponding node.
+
+Use timeline.capture to inspect actual Motion frames and correct composition or script errors. Render as reference video in the Motion inspector bakes the full Motion to H.264 at 24 fps and adds a video reference for MiniMax. Clips shorter than two seconds hold their final pose to reach two seconds; longer references initially select the first 15 seconds. The user can adjust the reference range. Never invent a baked file path or claim a render occurred just by emitting motion.add/set.
diff --git a/src-tauri/src/agent/prompt.rs b/src-tauri/src/agent/prompt.rs
index f179efc..cbc66fa 100644
--- a/src-tauri/src/agent/prompt.rs
+++ b/src-tauri/src/agent/prompt.rs
@@ -1,5 +1,5 @@
use crate::project::ProjectConfig;
-pub(super) const AGENT_SYSTEM_PROMPT: &str = include_str!("../../assets/agent-system.md");
+pub(super) const AGENT_SYSTEM_PROMPT: &str = concat!(include_str!("../../assets/agent-system.md"), "\n\n", include_str!("../../assets/agent-motion.md"));
pub(super) fn full_agent_system_prompt() -> String {
system_prompt_for(crate::generation::models::default_model())
}
diff --git a/src-tauri/src/project/commands/batch.rs b/src-tauri/src/project/commands/batch.rs
index 585427c..a8b221c 100644
--- a/src-tauri/src/project/commands/batch.rs
+++ b/src-tauri/src/project/commands/batch.rs
@@ -80,6 +80,7 @@ fn apply_command(
effects: &mut Effects,
) -> Result<(), String> {
match command {
+ ProjectCommand::MotionAdd { .. } | ProjectCommand::MotionSet { .. } => super::motions::apply(project, command, timestamp),
ProjectCommand::ImageDraftAdd { .. } | ProjectCommand::ImageDraftSelect { .. } => {
super::image_drafts::apply(project, command, timestamp)
}
diff --git a/src-tauri/src/project/commands/clips.rs b/src-tauri/src/project/commands/clips.rs
index 4fbfe6b..c5f02c2 100644
--- a/src-tauri/src/project/commands/clips.rs
+++ b/src-tauri/src/project/commands/clips.rs
@@ -62,7 +62,7 @@ pub(super) fn apply(
width: None,
height: None,
has_audio: None,
- image_generation: None, image_editing: None,
+ motion: None, image_generation: None, image_editing: None,
scene_segments: None,
image_draft: None,
parent_asset_id: None,
diff --git a/src-tauri/src/project/commands/mod.rs b/src-tauri/src/project/commands/mod.rs
index 26bc618..b9669c5 100644
--- a/src-tauri/src/project/commands/mod.rs
+++ b/src-tauri/src/project/commands/mod.rs
@@ -17,3 +17,5 @@ pub(crate) use types::{CommandBatch, ProjectCommand};
#[cfg(test)]
mod tests;
pub(crate) use batch::execute_checked;
+
+mod motions;
diff --git a/src-tauri/src/project/commands/motions.rs b/src-tauri/src/project/commands/motions.rs
new file mode 100644
index 0000000..4298cfa
--- /dev/null
+++ b/src-tauri/src/project/commands/motions.rs
@@ -0,0 +1,27 @@
+use super::types::ProjectCommand;
+use crate::project::{ProjectAsset, ProjectConfig};
+
+pub(super) fn apply(project: &mut ProjectConfig, command: &ProjectCommand, timestamp: &str) -> Result<(), String> {
+ match command {
+ ProjectCommand::MotionAdd { id, name, motion } => {
+ if id.trim().is_empty() || name.trim().is_empty() || project.assets.iter().any(|a| a.id == *id) { return Err("A Motion needs a unique asset id and name.".into()); }
+ crate::project::motion::validate(motion)?;
+ let asset: ProjectAsset = serde_json::from_value(serde_json::json!({
+ "id": id, "name": name, "kind":"motion", "mimeType":"application/x-slopus-motion", "motion":motion, "createdAt":timestamp,
+ "width":motion["width"], "height":motion["height"], "durationMs":(motion["duration"].as_f64().unwrap() * 1000.0).round() as u64, "hasAudio":false
+ })).map_err(|e| e.to_string())?;
+ project.assets.push(asset);
+ }
+ ProjectCommand::MotionSet { id, name, motion } => {
+ let asset = project.assets.iter_mut().find(|a| a.id == *id && a.kind == "motion").ok_or("Motion asset not found.")?;
+ if let Some(name) = name { if name.trim().is_empty() { return Err("Motion name cannot be empty.".into()); } asset.name = name.clone(); }
+ if let Some(motion) = motion { crate::project::motion::validate(motion)?;
+ asset.width = motion["width"].as_u64().map(|v| v as u32);
+ asset.height = motion["height"].as_u64().map(|v| v as u32);
+ asset.duration_ms = motion["duration"].as_f64().map(|v| (v * 1000.0).round() as u64);
+ asset.motion = Some(motion.clone()); }
+ }
+ _ => return Err("Unsupported Motion command.".into()),
+ }
+ Ok(())
+}
diff --git a/src-tauri/src/project/commands/tests.rs b/src-tauri/src/project/commands/tests.rs
index c116e48..8f509cd 100644
--- a/src-tauri/src/project/commands/tests.rs
+++ b/src-tauri/src/project/commands/tests.rs
@@ -297,3 +297,37 @@ fn commands_cannot_deserialize_protected_scene_fields() {
.unwrap_err();
assert!(error.to_string().contains("unknown field"), "{error}");
}
+
+#[test]
+fn motions_are_agent_editable_inline_assets_with_same_batch_clip_metadata() {
+ let current = fixture();
+ let motion = serde_json::json!({ "version":1, "space":"2d", "width":320, "height":180, "duration":3,
+ "source":"export function update({time,nodes}) {}", "nodes":[{"id":"title","name":"Title","kind":"text","text":"Hello"}],
+ "parameters":[], "overrides":{"title":{"offset":[12,24,0]}} });
+ let commands: Vec<ProjectCommand> = serde_json::from_value(serde_json::json!([
+ {"op":"motion.add", "id":"motion-title", "name":"Title", "motion":motion},
+ {"op":"clip.add", "id":"motion-clip", "asset":"motion-title"}
+ ])).unwrap();
+ let next = execute_commands_at(&current, &commands, "2026-10-25T00:00:00.000Z").unwrap();
+ let asset = next.assets.iter().find(|a| a.id == "motion-title").unwrap();
+ assert_eq!(asset.motion.as_ref(), Some(&motion));
+ assert_eq!(asset.duration_ms, Some(3000));
+ assert_eq!(asset.width, Some(320));
+ let clip = next.timeline.tracks.iter().flat_map(|t| &t.clips).find(|c| c.id == "motion-clip").unwrap();
+ assert_eq!(clip.duration_ms, 3000);
+ let restored: ProjectConfig = serde_json::from_value(serde_json::to_value(&next).unwrap()).unwrap();
+ let restored = validate_and_normalize_config(restored).unwrap();
+ assert_eq!(restored.assets, next.assets);
+ assert_eq!(restored.timeline, next.timeline);
+ let mut longer = motion.clone(); longer["duration"] = serde_json::json!(7);
+ let commands: Vec<ProjectCommand> = serde_json::from_value(serde_json::json!([
+ {"op":"motion.set", "id":"motion-title", "motion":longer},
+ {"op":"clip.add", "id":"motion-longer", "asset":"motion-title"}
+ ])).unwrap();
+ let edited = execute_commands_at(&next, &commands, "2026-10-25T00:00:00.000Z").unwrap();
+ assert_eq!(edited.timeline.tracks.iter().flat_map(|t| &t.clips).find(|c| c.id == "motion-longer").unwrap().duration_ms, 7000);
+ let mut invalid = motion.clone(); invalid["overrides"]["missing"] = serde_json::json!({});
+ let bad: ProjectCommand = serde_json::from_value(serde_json::json!({"op":"motion.set","id":"motion-title","motion":invalid})).unwrap();
+ assert!(execute_commands_at(&next, &[bad], "2026-10-25T00:00:00.000Z").is_err());
+ assert_eq!(current, fixture());
+}
diff --git a/src-tauri/src/project/commands/types.rs b/src-tauri/src/project/commands/types.rs
index c79f016..a59fd1d 100644
--- a/src-tauri/src/project/commands/types.rs
+++ b/src-tauri/src/project/commands/types.rs
@@ -12,6 +12,10 @@ pub const MAX_COMMANDS_PER_TURN: usize = 100;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "op", deny_unknown_fields)]
pub enum ProjectCommand {
+ #[serde(rename = "motion.add")]
+ MotionAdd { id: String, name: String, motion: serde_json::Value },
+ #[serde(rename = "motion.set")]
+ MotionSet { id: String, #[serde(default)] name: Option<String>, #[serde(default)] motion: Option<serde_json::Value> },
#[serde(rename = "image.draft.add")]
ImageDraftAdd {
id: String,
@@ -318,6 +322,8 @@ pub(super) struct CommitLine {
}
pub(super) fn command_name(command: &ProjectCommand) -> &'static str {
match command {
+ ProjectCommand::MotionAdd { .. } => "motion.add",
+ ProjectCommand::MotionSet { .. } => "motion.set",
ProjectCommand::ImageConfigure { .. } => "image.configure",
ProjectCommand::ImageDraftAdd { .. } => "image.draft.add",
ProjectCommand::ImageDraftSelect { .. } => "image.draft.select",
diff --git a/src-tauri/src/project/mod.rs b/src-tauri/src/project/mod.rs
index 5768f70..462d892 100644
--- a/src-tauri/src/project/mod.rs
+++ b/src-tauri/src/project/mod.rs
@@ -15,3 +15,5 @@ pub(crate) use scenes::*;
pub(crate) use timeline::*;
pub(crate) mod commands;
pub(crate) mod reference_tokens;
+
+pub(crate) mod motion;
diff --git a/src-tauri/src/project/model.rs b/src-tauri/src/project/model.rs
index 6022fd4..0489d04 100644
--- a/src-tauri/src/project/model.rs
+++ b/src-tauri/src/project/model.rs
@@ -101,6 +101,8 @@ pub(crate) struct ProjectAsset {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub(crate) source_path: Option<String>,
pub(crate) mime_type: String,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) motion: Option<serde_json::Value>,
// zod now spells these `.nullish()`, so an explicit `null` WOULD parse.
// Keep skipping anyway: it is the shape the fixtures pin, and it keeps the
// absent case unambiguous. Do not restate the old reason — the frontend
diff --git a/src-tauri/src/project/motion.rs b/src-tauri/src/project/motion.rs
new file mode 100644
index 0000000..769efe8
--- /dev/null
+++ b/src-tauri/src/project/motion.rs
@@ -0,0 +1,64 @@
+//! Portable Motion documents. Scripts are evaluated only in the UI worker;
+//! saving or agent preparation never executes project JavaScript.
+use serde_json::Value;
+
+fn number(v: &Value, low: f64, high: f64) -> bool {
+ v.as_f64().is_some_and(|n| n.is_finite() && n >= low && n <= high)
+}
+fn text(v: &Value, max: usize) -> bool { v.as_str().is_some_and(|s| s.chars().count() <= max) }
+fn identifier(v: &Value) -> bool {
+ v.as_str().is_some_and(|s| !s.is_empty() && s.len() <= 80 && s.as_bytes()[0].is_ascii_alphabetic()
+ && s.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'_' || b == b'-'))
+}
+fn vector(v: &Value) -> bool {
+ v.as_array().is_some_and(|a| a.len() == 3 && a.iter().all(|n| number(n, -1e6, 1e6)))
+}
+pub(crate) fn validate(m: &Value) -> Result<(), String> {
+ let invalid = || "Invalid Motion document. Use version 1, space 2d, valid nodes, parameters and source.".to_string();
+ if m["version"] != 1 || m["space"] != "2d" || !text(&m["source"], 200000)
+ || !number(&m["duration"], 0.1, 600.0)
+ || ["width", "height"].iter().any(|k| !m[k].as_u64().is_some_and(|n| (32..=4096).contains(&n)))
+ || m.get("seed").is_some_and(|v| !v.as_u64().is_some_and(|n| n <= u32::MAX as u64)) { return Err(invalid()); }
+ let nodes = m["nodes"].as_array().ok_or_else(invalid)?;
+ let params = m["parameters"].as_array().ok_or_else(invalid)?;
+ if nodes.len() > 512 || params.len() > 100 { return Err(invalid()); }
+ for entries in [nodes, params] {
+ let mut ids = std::collections::BTreeSet::new();
+ for entry in entries { if !identifier(&entry["id"]) || !ids.insert(entry["id"].as_str()) { return Err(invalid()); } }
+ }
+ for node in nodes {
+ if !text(&node["name"], 200) || node["name"] == ""
+ || !matches!(node["kind"].as_str(), Some("rect" | "ellipse" | "text" | "path")) { return Err(invalid()); }
+ for key in ["position", "rotation", "scale"] { if node.get(key).is_some_and(|v| !vector(v)) { return Err(invalid()); } }
+ for key in ["width", "height"] { if node.get(key).is_some_and(|v| !number(v, -1e6, 1e6)) { return Err(invalid()); } }
+ for (key, low, high) in [("opacity", 0.0, 1.0), ("strokeWidth", 0.0, 1000.0), ("fontSize", 1.0, 1000.0)] {
+ if node.get(key).is_some_and(|v| !number(v, low, high)) { return Err(invalid()); }
+ }
+ for (key, max) in [("fill", 100), ("stroke", 100), ("text", 10000), ("path", 50000)] {
+ if node.get(key).is_some_and(|v| !text(v, max)) { return Err(invalid()); }
+ }
+ if node.get("visible").is_some_and(|v| !v.is_boolean()) || node.get("fontFamily").is_some_and(|v| !matches!(v.as_str(), Some("sans-serif" | "serif" | "monospace"))) { return Err(invalid()); }
+ }
+ for p in params {
+ if !text(&p["label"], 200) || p["label"] == "" { return Err(invalid()); }
+ let valid = match p["type"].as_str() {
+ Some("number") => number(&p["value"], -1e6, 1e6),
+ Some("boolean") => p["value"].is_boolean(),
+ Some("string" | "color") => text(&p["value"], 10000), _ => false,
+ };
+ if !valid || ["min", "max"].iter().any(|k| p.get(k).is_some_and(|v| !number(v, -1e6, 1e6)))
+ || p.get("step").is_some_and(|v| !number(v, f64::MIN_POSITIVE, 1e6)) { return Err(invalid()); }
+ if let Some(min) = p["min"].as_f64() {
+ if p["max"].as_f64().is_some_and(|max| max < min) || p["value"].as_f64().is_some_and(|v| v < min) { return Err(invalid()); }
+ }
+ if let Some(max) = p["max"].as_f64() { if p["value"].as_f64().is_some_and(|v| v > max) { return Err(invalid()); } }
+ }
+ if let Some(overrides) = m.get("overrides") {
+ for (id, edit) in overrides.as_object().ok_or_else(invalid)? {
+ if !nodes.iter().any(|n| n["id"] == id.as_str()) || !edit.is_object() { return Err(invalid()); }
+ for key in ["offset", "scale", "rotation"] { if edit.get(key).is_some_and(|v| !vector(v)) { return Err(invalid()); } }
+ for (key, max) in [("fill", 100), ("text", 10000)] { if edit.get(key).is_some_and(|v| !text(v, max)) { return Err(invalid()); } }
+ }
+ }
+ Ok(())
+}
diff --git a/src-tauri/src/project/validation/assets.rs b/src-tauri/src/project/validation/assets.rs
index 81da5e6..f4686b5 100644
--- a/src-tauri/src/project/validation/assets.rs
+++ b/src-tauri/src/project/validation/assets.rs
@@ -3,6 +3,16 @@ use crate::project::paths::*;
use crate::project::*;
pub(super) fn validate(config: &mut ProjectConfig) -> Result<(), String> {
for asset in &mut config.assets {
+ if asset.kind == "motion" {
+ let motion = asset.motion.as_ref().ok_or("Motion asset needs its document.")?;
+ crate::project::motion::validate(motion)?;
+ if asset.relative_path.is_some() || asset.source_path.is_some() { return Err("Motion assets store their document inline.".into()); }
+ asset.width = motion["width"].as_u64().map(|v| v as u32);
+ asset.height = motion["height"].as_u64().map(|v| v as u32);
+ asset.duration_ms = motion["duration"].as_f64().map(|v| (v * 1000.0).round() as u64);
+ asset.has_audio = Some(false);
+ } else if asset.motion.is_some() { return Err("Only Motion assets can contain a Motion document.".into()); }
+
if let Some(segments) = &mut asset.scene_segments {
if asset.kind != "generated" || segments.is_empty() {
return Err("Scene ranges require a generated video.".into());
@@ -53,7 +63,7 @@ pub(super) fn validate(config: &mut ProjectConfig) -> Result<(), String> {
}
if !matches!(
asset.kind.as_str(),
- "video" | "audio" | "image" | "caption" | "generated"
+ "video" | "audio" | "image" | "caption" | "generated" | "motion"
) {
return Err(format!("Unsupported asset kind '{}'.", asset.kind));
}
@@ -82,7 +92,7 @@ pub(super) fn validate(config: &mut ProjectConfig) -> Result<(), String> {
.transpose()?;
// Generated scenes and blank image drafts have no file until generation
// completes. Imported assets still have exactly one stored location.
- if (asset.kind != "generated" && asset.image_draft != Some(true)) || asset.relative_path.is_some() || asset.source_path.is_some()
+ if (asset.kind != "generated" && asset.kind != "motion" && asset.image_draft != Some(true)) || asset.relative_path.is_some() || asset.source_path.is_some()
{
check_one_location(
&format!("Asset '{}'", asset.id),
diff --git a/src-tauri/src/tests/media_access.rs b/src-tauri/src/tests/media_access.rs
index feade30..a83af25 100644
--- a/src-tauri/src/tests/media_access.rs
+++ b/src-tauri/src/tests/media_access.rs
@@ -111,7 +111,7 @@ fn external_media_is_readable_only_because_the_project_names_it() {
width: None,
height: None,
has_audio: None,
- image_generation: None, image_editing: None, scene_segments: None,
+ motion: None, image_generation: None, image_editing: None, scene_segments: None,
image_draft: None,
parent_asset_id: None,
created_at: config.created_at.clone(),
@@ -285,7 +285,7 @@ fn a_project_the_user_only_opened_reads_any_media_file_it_names() {
width: None,
height: None,
has_audio: None,
- image_generation: None, image_editing: None, scene_segments: None,
+ motion: None, image_generation: None, image_editing: None, scene_segments: None,
image_draft: None,
parent_asset_id: None,
created_at: config.created_at.clone(),
diff --git a/src-tauri/src/tests/persistence.rs b/src-tauri/src/tests/persistence.rs
index 10a5e49..780f7e4 100644
--- a/src-tauri/src/tests/persistence.rs
+++ b/src-tauri/src/tests/persistence.rs
@@ -430,7 +430,7 @@ fn a_project_file_only_yields_media_a_hand_edit_cannot_widen() {
width: None,
height: None,
has_audio: None,
- image_generation: None, image_editing: None, scene_segments: None,
+ motion: None, image_generation: None, image_editing: None, scene_segments: None,
image_draft: None,
parent_asset_id: None,
created_at: config.created_at.clone(),
diff --git a/src-tauri/tauri.conf.json b/src-tauri/tauri.conf.json
index 5691675..df945e1 100644
--- a/src-tauri/tauri.conf.json
+++ b/src-tauri/tauri.conf.json
@@ -27,7 +27,7 @@
}
],
"security": {
- "csp": "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'self' blob: data:; media-src 'self' blob: data:; worker-src 'self' blob:; connect-src 'self' ipc: http://ipc.localhost blob: data:; object-src 'none'; base-uri 'self'; frame-ancestors 'none'; form-action 'none'"
+ "csp": "default-src 'self'; script-src 'self' blob:; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'self' blob: data:; media-src 'self' blob: data:; worker-src 'self' blob:; connect-src 'self' ipc: http://ipc.localhost blob: data:; object-src 'none'; base-uri 'self'; frame-ancestors 'none'; form-action 'none'"
}
},
"bundle": {
diff --git a/src/lib/motion.ts b/src/lib/motion.ts
new file mode 100644
index 0000000..cffcccd
--- /dev/null
+++ b/src/lib/motion.ts
@@ -0,0 +1,65 @@
+import { z } from "zod";
+
+const finite = z.number().finite().min(-1e6).max(1e6);
+const vector = z.tuple([finite, finite, finite]);
+const id = z.string().regex(/^[a-zA-Z][a-zA-Z0-9_-]{0,79}$/);
+export const motionNodeSchema = z.object({
+ id, name: z.string().min(1).max(200),
+ kind: z.enum(["rect", "ellipse", "text", "path"]),
+ position: vector.default([0, 0, 0]), rotation: vector.default([0, 0, 0]), scale: vector.default([1, 1, 1]),
+ width: finite.default(100), height: finite.default(100), opacity: z.number().min(0).max(1).default(1),
+ visible: z.boolean().default(true), fill: z.string().max(100).default("#ffffff"),
+ stroke: z.string().max(100).default("transparent"), strokeWidth: z.number().min(0).max(1000).default(0),
+ text: z.string().max(10000).default(""), fontSize: z.number().min(1).max(1000).default(48),
+ fontFamily: z.enum(["sans-serif", "serif", "monospace"]).default("sans-serif"),
+ path: z.string().max(50000).default(""),
+});
+export type MotionNode = z.infer<typeof motionNodeSchema>;
+export const motionOverrideSchema = z.object({
+ offset: vector.default([0, 0, 0]), scale: vector.default([1, 1, 1]), rotation: vector.default([0, 0, 0]),
+ fill: z.string().max(100).optional(), text: z.string().max(10000).optional(),
+});
+export type MotionOverride = z.infer<typeof motionOverrideSchema>;
+const parameterSchema = z.object({
+ id, label: z.string().min(1).max(200), type: z.enum(["number", "string", "color", "boolean"]),
+ value: z.union([finite, z.string().max(10000), z.boolean()]),
+ min: finite.optional(), max: finite.optional(), step: z.number().positive().max(1e6).optional(),
+}).superRefine((p, ctx) => {
+ if (typeof p.value !== (p.type === "number" ? "number" : p.type === "boolean" ? "boolean" : "string")
+ || p.min !== undefined && p.max !== undefined && p.min > p.max
+ || typeof p.value === "number" && (p.min !== undefined && p.value < p.min || p.max !== undefined && p.value > p.max))
+ ctx.addIssue({ code: "custom", message: "Motion parameter value must match its type and range." });
+});
+/** Versioned scene description, independent of Canvas/WebGPU. Three-component
+ * transforms leave a migration path to mesh/camera primitives; v1 is 2D. */
+export const motionSchema = z.object({
+ version: z.literal(1), space: z.literal("2d"),
+ width: z.number().int().min(32).max(4096), height: z.number().int().min(32).max(4096),
+ duration: z.number().finite().min(0.1).max(600), seed: z.number().int().min(0).max(0xffffffff).default(1),
+ source: z.string().max(200000), nodes: z.array(motionNodeSchema).max(512),
+ parameters: z.array(parameterSchema).max(100),
+ overrides: z.record(motionOverrideSchema).default({}),
+}).superRefine((motion, ctx) => {
+ for (const entries of [motion.nodes, motion.parameters]) {
+ if (new Set(entries.map(({ id }) => id)).size !== entries.length) ctx.addIssue({ code: "custom", message: "Motion IDs must be unique." });
+ }
+ if (Object.keys(motion.overrides).some((id) => !motion.nodes.some((node) => node.id === id)))
+ ctx.addIssue({ code: "custom", message: "Motion overrides must refer to scene elements." });
+});
+export type Motion = z.infer<typeof motionSchema>;
+export const MOTION_MIME = "application/x-slopus-motion";
+export function emptyMotion(width = 1280, height = 720): Motion {
+ return motionSchema.parse({ version: 1, space: "2d", width, height, duration: 5, nodes: [], parameters: [],
+ source: "// Ask the agent to create this Motion.\n// Time is in seconds. Modify named nodes or add primitives with draw.\nexport function update({ time, duration, parameters, nodes }, draw) {\n}\n" });
+}
+export const motionOverride = (motion: Motion, id: string): MotionOverride => motion.overrides[id] ?? motionOverrideSchema.parse({});
+export function applyMotionOverrides(nodes: MotionNode[], motion: Motion): MotionNode[] {
+ return nodes.map((node) => {
+ const edit = motion.overrides[node.id];
+ if (!edit) return node;
+ return { ...node, position: node.position.map((v, i) => v + edit.offset[i]) as MotionNode["position"],
+ rotation: node.rotation.map((v, i) => v + edit.rotation[i]) as MotionNode["rotation"],
+ scale: node.scale.map((v, i) => v * edit.scale[i]) as MotionNode["scale"],
+ ...(edit.fill !== undefined ? { fill: edit.fill } : {}), ...(edit.text !== undefined ? { text: edit.text } : {}) };
+ });
+}
diff --git a/src/lib/motion.worker.ts b/src/lib/motion.worker.ts
new file mode 100644
index 0000000..b83598d
--- /dev/null
+++ b/src/lib/motion.worker.ts
@@ -0,0 +1,21 @@
+import { motionSchema } from "./motion";
+import { evaluateMotion, type MotionUpdate } from "./motionEvaluation";
+
+let source: string | undefined;
+let update: MotionUpdate | undefined;
+self.onmessage = async ({ data }) => {
+ try {
+ const motion = motionSchema.parse(data.motion);
+ if (!Number.isFinite(data.time)) throw new Error("Invalid Motion time.");
+ if (source !== motion.source) {
+ const url = URL.createObjectURL(new Blob([motion.source], { type: "text/javascript" }));
+ try {
+ const module = await import(/* @vite-ignore */ url);
+ if (typeof module.update !== "function") throw new Error("Export a synchronous update(context, draw) function.");
+ update = module.update;
+ source = motion.source;
+ } finally { URL.revokeObjectURL(url); }
+ }
+ self.postMessage({ id: data.id, nodes: evaluateMotion(motion, data.time, update!) });
+ } catch (reason) { self.postMessage({ id: data.id, error: reason instanceof Error ? reason.message : String(reason) }); }
+};
diff --git a/src/lib/motionEvaluation.ts b/src/lib/motionEvaluation.ts
new file mode 100644
index 0000000..eddebb6
--- /dev/null
+++ b/src/lib/motionEvaluation.ts
@@ -0,0 +1,26 @@
+import { applyMotionOverrides, motionNodeSchema, type Motion, type MotionNode } from "./motion";
+
+export interface MotionContext { time: number; duration: number; width: number; height: number; parameters: Record<string, string | number | boolean>; nodes: MotionNode[]; random: () => number }
+export type MotionDraw = Record<MotionNode["kind"], (node: Partial<MotionNode>) => void>;
+export type MotionUpdate = (context: MotionContext, draw: MotionDraw) => void;
+
+/** Pure evaluation contract shared by the worker and tests. A fresh scene and
+ * seeded RNG on every request make seeking independent of playback order. */
+export function evaluateMotion(motion: Motion, seconds: number, update: MotionUpdate): MotionNode[] {
+ const nodes = structuredClone(motion.nodes);
+ let state = motion.seed >>> 0;
+ const draw = Object.fromEntries((["rect", "ellipse", "text", "path"] as const).map((kind) => [kind, (node: Partial<MotionNode>) => {
+ if (nodes.length >= 1024) throw new Error("Motion exceeds 1024 drawing elements.");
+ nodes.push(motionNodeSchema.parse({ id: `draw-${nodes.length}`, name: kind, ...node, kind }));
+ }])) as MotionDraw;
+ const result = update({ time: Math.max(0, Math.min(motion.duration, seconds)), duration: motion.duration,
+ width: motion.width, height: motion.height, nodes,
+ parameters: Object.fromEntries(motion.parameters.map((p) => [p.id, p.value])),
+ random: () => { state = (Math.imul(1664525, state) + 1013904223) >>> 0; return state / 4294967296; },
+ }, draw);
+ if (result !== undefined) throw new Error("Motion update must be synchronous and return nothing.");
+ if (nodes.length > 1024) throw new Error("Motion exceeds 1024 drawing elements.");
+ const checked = nodes.map((node) => motionNodeSchema.parse(node));
+ if (new Set(checked.map((node) => node.id)).size !== checked.length) throw new Error("Motion drawing IDs must be unique.");
+ return applyMotionOverrides(checked, motion);
+}
diff --git a/src/lib/motionRuntime.ts b/src/lib/motionRuntime.ts
new file mode 100644
index 0000000..1f30262
--- /dev/null
+++ b/src/lib/motionRuntime.ts
@@ -0,0 +1,85 @@
+import { motionNodeSchema, type Motion, type MotionNode } from "./motion";
+
+/** Scripts run off the UI thread. This is a responsiveness boundary, not an
+ * OS security sandbox. No interpreter, compiler or external renderer. */
+export class MotionRuntime {
+ private worker: Worker | null = null;
+ private sequence = 0;
+ private pending = new Map<number, { resolve: (nodes: MotionNode[]) => void; reject: (error: Error) => void; timer: ReturnType<typeof setTimeout> }>();
+ evaluate(motion: Motion, time: number): Promise<MotionNode[]> {
+ if (!this.worker) {
+ this.worker = new Worker(new URL("./motion.worker.ts", import.meta.url), { type: "module" });
+ this.worker.onmessage = ({ data }) => {
+ const task = this.pending.get(data.id);
+ if (!task) return;
+ this.pending.delete(data.id); clearTimeout(task.timer);
+ try {
+ if (data.error) throw new Error(data.error);
+ if (!Array.isArray(data.nodes) || data.nodes.length > 1024) throw new Error("Invalid Motion drawing output.");
+ task.resolve(data.nodes.map((node: unknown) => motionNodeSchema.parse(node)));
+ } catch (reason) { task.reject(reason instanceof Error ? reason : new Error(String(reason))); }
+ };
+ this.worker.onerror = (event) => this.dispose(new Error(event.message || "Motion script failed."));
+ }
+ const id = ++this.sequence;
+ return new Promise((resolve, reject) => {
+ const timer = setTimeout(() => this.dispose(new Error("Motion took too long. Simplify its script and try again.")), 3000);
+ this.pending.set(id, { resolve, reject, timer });
+ try { this.worker!.postMessage({ id, motion, time }); }
+ catch (reason) { this.pending.delete(id); clearTimeout(timer); reject(reason); }
+ });
+ }
+ dispose(error = new Error("Motion closed.")) {
+ this.worker?.terminate(); this.worker = null;
+ for (const task of this.pending.values()) { clearTimeout(task.timer); task.reject(error); }
+ this.pending.clear();
+ }
+}
+
+export interface MotionRenderer { render(nodes: MotionNode[], width: number, height: number): void }
+/** Portable 2D renderer. The resulting transparent surface is composited by
+ * Slopus's WebGPU path, or Canvas 2D when WebGPU is unavailable. A future 3D
+ * renderer can consume an extended scene without changing script scheduling. */
+export class CanvasMotionRenderer implements MotionRenderer {
+ constructor(private canvas: HTMLCanvasElement | OffscreenCanvas) {}
+ render(nodes: MotionNode[], width: number, height: number) {
+ const ctx = this.canvas.getContext("2d") as CanvasRenderingContext2D | OffscreenCanvasRenderingContext2D | null;
+ if (!ctx) throw new Error("Canvas rendering is unavailable for this Motion.");
+ ctx.resetTransform(); ctx.clearRect(0, 0, this.canvas.width, this.canvas.height);
+ ctx.save(); ctx.scale(this.canvas.width / width, this.canvas.height / height);
+ try { for (const node of nodes) {
+ if (!node.visible) continue;
+ ctx.save();
+ try {
+ ctx.translate(node.position[0], node.position[1]); ctx.rotate(node.rotation[2] * Math.PI / 180); ctx.scale(node.scale[0], node.scale[1]);
+ ctx.globalAlpha = node.opacity; ctx.fillStyle = "#ffffff"; ctx.fillStyle = node.fill;
+ ctx.strokeStyle = "transparent"; ctx.strokeStyle = node.stroke; ctx.lineWidth = node.strokeWidth;
+ if (node.kind === "text") {
+ ctx.font = `${node.fontSize}px ${node.fontFamily}`; ctx.textBaseline = "top";
+ ctx.fillText(node.text, 0, 0, Math.max(1, Math.abs(node.width)));
+ } else {
+ const path = new Path2D(node.kind === "path" ? node.path : undefined);
+ if (node.kind === "rect") path.rect(0, 0, node.width, node.height);
+ if (node.kind === "ellipse") path.ellipse(node.width / 2, node.height / 2, Math.abs(node.width / 2), Math.abs(node.height / 2), 0, 0, 2 * Math.PI);
+ ctx.fill(path); if (node.strokeWidth > 0) ctx.stroke(path);
+ }
+ } finally { ctx.restore(); }
+ } } finally { ctx.restore(); }
+ }
+}
+
+export function motionFrameReader(motion: Motion, check: () => void) {
+ const runtime = new MotionRuntime();
+ const canvas = new OffscreenCanvas(motion.width, motion.height);
+ const renderer = new CanvasMotionRenderer(canvas);
+ let frame: VideoFrame | null = null;
+ return {
+ async frameAt(timeUs: number) {
+ check(); const nodes = await runtime.evaluate(motion, timeUs / 1e6); check();
+ renderer.render(nodes, motion.width, motion.height);
+ frame?.close(); frame = new VideoFrame(canvas, { timestamp: Math.round(timeUs), alpha: "keep" });
+ return frame;
+ },
+ dispose() { runtime.dispose(); frame?.close(); frame = null; },
+ };
+}
diff --git a/src/lib/project.ts b/src/lib/project.ts
index 9c8257a..0c319a2 100644
--- a/src/lib/project.ts
+++ b/src/lib/project.ts
@@ -1,3 +1,4 @@
+import { motionSchema } from "./motion";
import { generationEffectsSchema, activeGenerationEffects, type GenerationEffects } from "./generationEffects";
import { MIN_SIGMA_SHIFT, MAX_SIGMA_SHIFT } from "./sampling";
import { z } from "zod";
@@ -184,7 +185,8 @@ export type SceneMediaSegment = z.infer<typeof sceneMediaSegmentSchema>;
export const projectAssetSchema = z.object({
id: idSchema,
- kind: z.enum(["video", "audio", "image", "caption", "generated"]),
+ kind: z.enum(["video", "audio", "image", "caption", "generated", "motion"]),
+ motion: motionSchema.nullish(),
name: z.string().min(1),
// Exactly one of the two. `relativePath` is a file COPIED into the project
// folder — images, and everything generated here; `sourcePath` is a file left
@@ -217,6 +219,9 @@ export const projectAssetSchema = z.object({
parentAssetId: idSchema.nullish(),
createdAt: isoDateSchema,
}).superRefine((asset, context) => {
+ if (asset.kind === "motion" ? !asset.motion || Boolean(asset.relativePath || asset.sourcePath) : Boolean(asset.motion)) {
+ context.addIssue({ code: z.ZodIssueCode.custom, message: "Motion assets require an inline Motion document without a media path." });
+ }
checkOneLocation(asset, context, `Asset '${asset.id}'`);
if (asset.sceneSegments) {
let end = 0;
@@ -241,7 +246,7 @@ export const projectAssetSchema = z.object({
}
/* Generated scenes and blank image drafts can exist before rendering.
* WorkQueue fills their file paths in when generation ends. */
- if (asset.kind !== "generated" && !asset.imageDraft && !asset.relativePath && !asset.sourcePath) {
+ if (asset.kind !== "generated" && asset.kind !== "motion" && !asset.imageDraft && !asset.relativePath && !asset.sourcePath) {
context.addIssue({
code: z.ZodIssueCode.custom, path: ["relativePath"],
message: `Asset '${asset.id}' must have either a project-relative path or an external source path.`,