Skip to content

AI media harness for planning, generating, and editing locally

TypeScript

Add Video library and timed scene references

Bitti4940719bb087
diff --git a/README.md b/README.md
index f945b41..96eb3a1 100644
--- a/README.md
+++ b/README.md
@@ -25,7 +25,7 @@ Slopus combines an AI assistant, local image, video and music generation, reusab
## Features
- **Agent.** Plan scenes, generate media, inspect frames and song inputs, and match Motions to music with Claude Code, Codex, OpenRouter or a local OpenAI-compatible model. Toggle the pane with Ctrl+Shift+A.
-- **Video.** Generate Prompt, Single Shot, First & last frame and other [scene types](docs/scene-types.md), with shot timing, dialogue, camera and sound controls. Keep up to four takes and an editable Draft; use **Make Primary** to choose the timeline video.
+- **Video.** Organize Videos → Scenes → Shots in folders, preview whole videos, and share prompts and timed music/video references across scenes. Generate with [multiple scene types](docs/scene-types.md), keep four takes plus a Draft, and choose each scene's timeline take with **Make Primary**.
- **Images.** Generate with MiniMax H3, MOSS Image or Fibo 2.0. Use whole-image, box, lasso and Zoom edits where supported, with drafts and versions in the image bar.
- **Music.** Generate with YuE2 or MiniMax Music3, edit lyrics and style, organize songs in folders, browse grid/list views, and play songs in sequence or reuse them as references and timeline audio.
- **References.** Reuse descriptions, images, trimmed video clips and [H3 RefMods](docs/refmods.md) across scenes.
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index bb6a33f..1fb1d84 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -4,6 +4,7 @@ Changes since 0.3.2.
## What's new
+- **Video library.** Organize Videos → Scenes → Shots in folders, play scenes sequentially on cards or in a popout, and apply shared prompts and timed music/video references. Split a complete song into scenes; timeline clips remain individual scenes.
- **Music export.** Select and preview songs in Export, then save WAV, AAC/M4A or Opus/WebM/Ogg with device-supported encoders, bitrate controls and background export.
- **Single Shot.** Generate a short ending first, then anchor a continuous shot to it. Supports references in both prompts and optional continuation from any generated scene.
- **Music.** YuE2 and MiniMax Music3 generation with folders, drag-and-drop organization, grid/list views, background playback, references and timeline audio. Playback appears in the Music tab and pauses when Timeline, Video or Export playback starts. YuE2 selects BF16 above 8 GiB VRAM, otherwise INT8.
diff --git a/docs/scene-types.md b/docs/scene-types.md
index 3fbc546..7421c39 100644
--- a/docs/scene-types.md
+++ b/docs/scene-types.md
@@ -1,5 +1,11 @@
# Scene types
+Videos contain ordered Scenes, which contain Shots. The Video library supports folders, grid/list views and sequential card or popout playback. Open a Video to edit its scenes; each scene remains separate in Timeline. Existing projects open with their scenes grouped in a default Video.
+
+Video settings provide a shared prompt with reference chips, background music and a reference video. Each scene receives the reference range at its cumulative start time: a 10-second first scene uses 0–10 seconds, and the next starts at 10 seconds. Imported/generated media uses the complete asset; reference entries use their saved trim range. Reordering or resizing scenes updates these ranges for the next generation. A source that ends too early blocks generation.
+
+For a music video, select a complete song on an empty Video and choose **Create scenes for song** to divide its full duration into roughly 10-second scenes. Add visual direction and an image reference to the shared prompt, or select a reference video: SlopFab sound references need visual conditioning and References/Singularity weights. Music guides generated audio; add the original song to Timeline when you want its exact soundtrack. Shared media references support Prompt, Single Shot, Continue, First & Last Frame, Backdrop, Pose and Long Shot scenes; each scene must last at least two seconds.
+
The Scene panel saves a type on each scene:
- **Prompt** is the default for ordinary video scenes. It supports shot descriptions, speech, Look, sound, music and references cited in prompts, without start- or last-frame selectors. Stored frame anchors and continuation links are inactive.
diff --git a/src-tauri/src/project/mod.rs b/src-tauri/src/project/mod.rs
index 1324ff1..7c58753 100644
--- a/src-tauri/src/project/mod.rs
+++ b/src-tauri/src/project/mod.rs
@@ -18,3 +18,4 @@ pub(crate) mod reference_tokens;
pub(crate) mod motion;
pub(crate) mod music;
+pub(crate) mod video;
diff --git a/src-tauri/src/project/model.rs b/src-tauri/src/project/model.rs
index a3c1e64..f77b1ab 100644
--- a/src-tauri/src/project/model.rs
+++ b/src-tauri/src/project/model.rs
@@ -18,6 +18,10 @@ pub(crate) enum GenerationType {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct ProjectConfig {
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ pub(crate) videos: Vec<super::video::ProjectVideo>,
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ pub(crate) video_folders: Vec<super::music::MusicFolder>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub(crate) music_songs: Vec<super::music::MusicSong>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
diff --git a/src-tauri/src/project/validation/mod.rs b/src-tauri/src/project/validation/mod.rs
index 33bfd65..06f02b7 100644
--- a/src-tauri/src/project/validation/mod.rs
+++ b/src-tauri/src/project/validation/mod.rs
@@ -21,6 +21,7 @@ pub(crate) fn validate_and_normalize_config(
pub(crate) fn validate_checked(
mut config: ProjectConfig,
) -> Result<ProjectConfig, issue::ValidationIssue> {
+ super::video::validate(&mut config).map_err(|message| issue::ValidationIssue::new("project.invalid", Some(config.id.clone()), "videos", message))?;
super::music::validate(&config).map_err(|message| issue::ValidationIssue::new("project.invalid", Some(config.id.clone()), "musicSongs", message))?;
if let Some(scene) = &config.image_scene {
scene.validate().map_err(|message| {
diff --git a/src-tauri/src/project/video.rs b/src-tauri/src/project/video.rs
new file mode 100644
index 0000000..74c5ef3
--- /dev/null
+++ b/src-tauri/src/project/video.rs
@@ -0,0 +1,90 @@
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct VideoSource { pub kind: String, pub id: String }
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct ProjectVideo {
+ pub id: String,
+ pub title: String,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub folder_id: Option<String>,
+ #[serde(default)] pub scene_ids: Vec<String>,
+ #[serde(default)] pub prompt: String,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub music_source: Option<VideoSource>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub video_source: Option<VideoSource>,
+ pub created_at: String,
+}
+
+pub(crate) fn validate(config: &mut super::ProjectConfig) -> Result<(), String> {
+ let mut folders = HashMap::new();
+ for folder in &config.video_folders {
+ if folder.id.is_empty() || folder.name.trim().is_empty() || folder.name.chars().count() > 200
+ || folders.insert(folder.id.as_str(), folder).is_some() { return Err("Invalid video folder.".into()); }
+ }
+ for folder in &config.video_folders {
+ let mut seen = HashSet::from([folder.id.as_str()]);
+ let mut parent = folder.parent_id.as_deref();
+ while let Some(id) = parent {
+ if !seen.insert(id) { return Err("Video folders cannot contain a cycle.".into()); }
+ parent = folders.get(id).ok_or("Video folder parent is missing.")?.parent_id.as_deref();
+ }
+ }
+ let scenes: HashSet<_> = config.generation_jobs.iter().map(|job| job.id.as_str()).collect();
+ let mut ids = HashSet::new();
+ let mut assigned = HashSet::new();
+ for video in &mut config.videos {
+ if video.id.is_empty() || !ids.insert(video.id.as_str()) || video.title.trim().is_empty() || video.title.chars().count() > 200
+ || video.prompt.chars().count() > 30000 { return Err("Invalid Video settings.".into()); }
+ if video.folder_id.as_deref().is_some_and(|id| !folders.contains_key(id)) { return Err("Video folder is missing.".into()); }
+ for source in [&video.music_source, &video.video_source].into_iter().flatten() {
+ if source.id.is_empty() || !matches!(source.kind.as_str(), "asset" | "reference") { return Err("Invalid Video reference source.".into()); }
+ }
+ // Scene deletion from the agent or timeline must not leave stale members.
+ video.scene_ids.retain(|id| scenes.contains(id.as_str()));
+ for id in &video.scene_ids {
+ if !assigned.insert(id.as_str()) { return Err("A scene can belong to only one Video.".into()); }
+ }
+ }
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ fn fixture() -> super::super::ProjectConfig {
+ serde_json::from_str(include_str!("../../../fixtures/project-v1-complete.json")).unwrap()
+ }
+ #[test]
+ fn video_settings_round_trip_without_merging_scene_media() {
+ let mut config = fixture();
+ let scene_id = config.generation_jobs[0].id.clone();
+ let timeline = serde_json::to_value(&config.timeline).unwrap();
+ config.videos = vec![ProjectVideo { id: "film".into(), title: "Film".into(), folder_id: None,
+ scene_ids: vec![scene_id.clone(), "removed-scene".into()], prompt: "Neon".into(),
+ music_source: Some(VideoSource { kind: "asset".into(), id: "song".into() }), video_source: None, created_at: config.created_at.clone() }];
+ validate(&mut config).unwrap();
+ assert_eq!(config.videos[0].scene_ids, [scene_id]);
+ let restored: super::super::ProjectConfig = serde_json::from_value(serde_json::to_value(&config).unwrap()).unwrap();
+ assert_eq!(restored.videos, config.videos);
+ assert_eq!(serde_json::to_value(&restored.timeline).unwrap(), timeline);
+ config.videos.push(ProjectVideo { id: "other".into(), ..config.videos[0].clone() });
+ assert!(validate(&mut config).unwrap_err().contains("only one Video"));
+ }
+ #[test]
+ fn video_folders_reject_missing_parents_and_cycles() {
+ let mut config = fixture();
+ assert!(config.videos.is_empty());
+ config.video_folders = vec![super::super::music::MusicFolder { id: "a".into(), name: "Album".into(), parent_id: Some("missing".into()) }];
+ assert!(validate(&mut config).unwrap_err().contains("parent is missing"));
+ config.video_folders[0].parent_id = Some("a".into());
+ assert!(validate(&mut config).unwrap_err().contains("cycle"));
+ config.video_folders[0].parent_id = None;
+ assert!(validate(&mut config).is_ok());
+ }
+}
diff --git a/src/components/ProjectWorkspace.music.test.tsx b/src/components/ProjectWorkspace.music.test.tsx
index 650cf54..8fa8097 100644
--- a/src/components/ProjectWorkspace.music.test.tsx
+++ b/src/components/ProjectWorkspace.music.test.tsx
@@ -15,7 +15,7 @@ vi.mock("../lib/persistence", async (original) => ({ ...await original<typeof im
readMediaFileUrl: vi.fn(async (_, asset) => `blob:${asset.relativePath}`),
}));
// Exercise workspace media-event routing, including React portals used by shot popouts.
-vi.mock("./workspace/GeneratorView", () => ({ GeneratorView: () => <><video aria-label="Shot" />{createPortal(<video aria-label="Popout" />, document.body)}</> }));
+vi.mock("./workspace/VideoLibraryView", () => ({ VideoLibraryView: () => <><video aria-label="Shot" />{createPortal(<video aria-label="Popout" />, document.body)}</> }));
vi.mock("./workspace/TimelineView", () => ({ TimelineView: () => <audio aria-label="Timeline preview" /> }));
vi.mock("./workspace/ExportView", () => ({ ExportView: () => <video aria-label="Export preview" /> }));
vi.mock("./workspace/ReferencesView", () => ({ ReferencesView: () => <div>References page</div> }));
diff --git a/src/components/ProjectWorkspace.tsx b/src/components/ProjectWorkspace.tsx
index 1664aaf..b12e69e 100644
--- a/src/components/ProjectWorkspace.tsx
+++ b/src/components/ProjectWorkspace.tsx
@@ -16,7 +16,7 @@ import { generateAgentMedia } from "../lib/agentGeneration";
import { ExportView } from "./workspace/ExportView";
import { ImageExportView } from "./workspace/ImageExportView";
import { MusicExportView } from "./workspace/MusicExportView";
-import { GeneratorView } from "./workspace/GeneratorView";
+import { VideoLibraryView } from "./workspace/VideoLibraryView";
import { ReferencesView } from "./workspace/ReferencesView";
import { TimelineView, type ConfigUpdate } from "./workspace/TimelineView";
import { UnsavedProjectDialog } from "./UnsavedProjectDialog";
@@ -151,6 +151,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
const [editingProject, setEditingProject] = useState(false);
const [savingProjectSettings, setSavingProjectSettings] = useState(false);
const [selectedGenerationJobId, setSelectedGenerationJobId] = useState<string | undefined>(project.config.generationJobs[0]?.id);
+ const [videoNavigation, setVideoNavigation] = useState<{ id: string }>();
const [imageNavigation, setImageNavigation] = useState<{ id: string }>();
const [musicNavigation, setMusicNavigation] = useState<{ id: string }>();
const [referenceNavigation, setReferenceNavigation] = useState<{ id: string }>();
@@ -158,7 +159,7 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
if (!navigation) return;
setView(navigation.view);
if (navigation.view === "export") setMediaMode(navigation.exportKind ?? "video");
- if (navigation.sceneId) setSelectedGenerationJobId(navigation.sceneId);
+ if (navigation.sceneId) { setSelectedGenerationJobId(navigation.sceneId); if (navigation.view === "generator") setVideoNavigation({ id: navigation.sceneId }); }
if (navigation.imageId) setImageNavigation({ id: navigation.imageId });
if (navigation.songId) setMusicNavigation({ id: navigation.songId });
if (navigation.referenceId) setReferenceNavigation({ id: navigation.referenceId });
@@ -301,10 +302,10 @@ export function ProjectWorkspace({ project, initialView, runtime = null, onBack,
{musicVisited && <div hidden={view !== "music"} style={{ height: "100%", display: view === "music" ? "block" : "none" }}>
<MusicView active={view === "music"} onPlayingChange={setMusicPlaying} openSong={musicNavigation} onSongOpened={() => setMusicNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onGenerate={(id, template) => queue.enqueueMusic(session, id, template)} onCancel={(id) => void queue.cancel(id)} workItems={projectItems.filter((item) => item.kind === "music")} />
</div>}
- {view === "editor" && <ImageEditor openImage={imageNavigation} onImageOpened={() => setImageNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onGenerate={(template, count) => queue.enqueueImage(session, template, count)} onGenerateCharacterSheet={(template, sourceId, options) => queue.enqueueCharacterSheet(session, template, sourceId, options)} onGenerateExtend={(template, sourceId, options) => queue.enqueueExtendImage(session, template, sourceId, options)} onCancel={(id) => queue.cancel(id)} workItems={projectItems.filter((item) => item.kind === "image")} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setView("generator"); }} onOpenReferences={() => setView("references")} />}
- {view === "timeline" && <TimelineView config={config} openSceneId={selectedGenerationJobId} folderPath={project.folderPath} generationCompletionTimes={generationCompletionTimes} onChange={changeConfig} onMeasured={recordMeasurement} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setView("generator"); }} />}
- {view === "generator" && <GeneratorView onGenerate={(submissions) => queue.enqueue(session, submissions)} onCancelGeneration={(ids) => queue.cancelScenes(session, ids)} cancellingJobIds={cancellingJobIds} config={config} folderPath={project.folderPath} generationCompletionTimes={generationCompletionTimes} runtime={runtime?.slopfab ?? null} onRuntimeChange={onGeneratorRuntimeChange} onChange={changeConfig} selectedJobId={selectedGenerationJobId} onSelectedJobChange={setSelectedGenerationJobId} onOpenTimeline={() => setView("timeline")} />}
- {view === "references" && <ReferencesView openReference={referenceNavigation} onReferenceOpened={() => setReferenceNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onRegenerateIcon={(id) => queue.regenerateReferenceIcon(session, id)} onExportRefmod={(id, outputPath) => queue.exportReferenceRefmod(session, id, outputPath)} onGenerateBuiltinIcons={() => queue.generateBuiltinReferenceIcons(session)} onRegenerateBuiltinIcon={(id) => queue.regenerateBuiltinReferenceIcon(session, id)} pendingBuiltinIconIds={queue.pendingBuiltinIconIds()} pendingIconIds={new Set(config.references.filter((reference) => queue.isReferenceIconPending(session, reference.id)).map((reference) => reference.id))} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setView("generator"); }} />}
+ {view === "editor" && <ImageEditor openImage={imageNavigation} onImageOpened={() => setImageNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onGenerate={(template, count) => queue.enqueueImage(session, template, count)} onGenerateCharacterSheet={(template, sourceId, options) => queue.enqueueCharacterSheet(session, template, sourceId, options)} onGenerateExtend={(template, sourceId, options) => queue.enqueueExtendImage(session, template, sourceId, options)} onCancel={(id) => queue.cancel(id)} workItems={projectItems.filter((item) => item.kind === "image")} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setVideoNavigation(jobId ? { id: jobId } : undefined); setView("generator"); }} onOpenReferences={() => setView("references")} />}
+ {view === "timeline" && <TimelineView config={config} openSceneId={selectedGenerationJobId} folderPath={project.folderPath} generationCompletionTimes={generationCompletionTimes} onChange={changeConfig} onMeasured={recordMeasurement} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setVideoNavigation(jobId ? { id: jobId } : undefined); setView("generator"); }} />}
+ {view === "generator" && <VideoLibraryView openScene={videoNavigation} onSceneOpened={() => setVideoNavigation(undefined)} onGenerate={(submissions) => queue.enqueue(session, submissions)} onCancelGeneration={(ids) => queue.cancelScenes(session, ids)} cancellingJobIds={cancellingJobIds} config={config} folderPath={project.folderPath} generationCompletionTimes={generationCompletionTimes} runtime={runtime?.slopfab ?? null} onRuntimeChange={onGeneratorRuntimeChange} onChange={changeConfig} selectedJobId={selectedGenerationJobId} onSelectedJobChange={setSelectedGenerationJobId} onOpenTimeline={() => setView("timeline")} />}
+ {view === "references" && <ReferencesView openReference={referenceNavigation} onReferenceOpened={() => setReferenceNavigation(undefined)} config={config} folderPath={project.folderPath} onChange={changeConfig} onRegenerateIcon={(id) => queue.regenerateReferenceIcon(session, id)} onExportRefmod={(id, outputPath) => queue.exportReferenceRefmod(session, id, outputPath)} onGenerateBuiltinIcons={() => queue.generateBuiltinReferenceIcons(session)} onRegenerateBuiltinIcon={(id) => queue.regenerateBuiltinReferenceIcon(session, id)} pendingBuiltinIconIds={queue.pendingBuiltinIconIds()} pendingIconIds={new Set(config.references.filter((reference) => queue.isReferenceIconPending(session, reference.id)).map((reference) => reference.id))} onOpenGenerator={(jobId) => { setSelectedGenerationJobId(jobId); setVideoNavigation(jobId ? { id: jobId } : undefined); setView("generator"); }} />}
{view === "export" && <div className="project-export">
<SelectorBar aria-label="Export type" className="project-export__types" value={mediaMode} onChange={setMediaMode} items={[{ value: "video", label: "Video" }, { value: "image", label: "Image" }, { value: "music", label: "Music" }]} />
{mediaMode === "music" ? <MusicExportView config={config} folderPath={project.folderPath} /> : mediaMode === "image"
diff --git a/src/components/workspace/GeneratorView.tsx b/src/components/workspace/GeneratorView.tsx
index 1517610..c40f5d2 100644
--- a/src/components/workspace/GeneratorView.tsx
+++ b/src/components/workspace/GeneratorView.tsx
@@ -15,7 +15,7 @@ import { askNative } from "../../lib/nativeShell";
import { ProjectStatus } from "./ProjectStatus";
import { GeneratorSelector } from "./GeneratorSelector";
import { CommandBar, CommandBarButton, CommandBarSeparator, ComboBox, EmptyState, ItemHeader, Splitter, tooltipProps, usePaneSize } from "../ui";
-import { shotReferenceIds, compileGenerationJobSegments, createDraftGenerationJob, GENERATION_FRAME_RATE, RANDOM_GENERATION_SEED, sceneDurationSeconds, sceneFrameInputs, sceneGenerationReferences, sceneGenerationSeed, sceneGenerationSnapshot, sceneShots, SCENE_MAX_SECONDS, SCENE_MIN_SECONDS, type GenerationJob, type ProjectConfig, type ProjectReference, type SceneShot } from "../../lib/project";
+import { sceneSiblings, shotReferenceIds, compileGenerationJobSegments, createDraftGenerationJob, GENERATION_FRAME_RATE, RANDOM_GENERATION_SEED, sceneDurationSeconds, sceneFrameInputs, sceneGenerationReferences, sceneGenerationSeed, sceneGenerationSnapshot, sceneShots, SCENE_MAX_SECONDS, SCENE_MIN_SECONDS, type GenerationJob, type ProjectConfig, type ProjectReference, type SceneShot } from "../../lib/project";
import { isTauri } from "../../lib/persistence";
import { getEngineStatus, type SlopfabGenerationRequest, type SlopfabStatus } from "../../lib/runtime";
import { selectedWorker } from "../../lib/workers";
@@ -27,8 +27,10 @@ import { purgeTimelineThumbnails } from "../../lib/timelineThumbnails";
import { defaultGeneratorTemplate, generationStepsForTemplate, loadDebugOptionsEnabled, loadGeneratorTemplateSettings, saveGeneratorTemplateSettings, subscribeDebugOptions, subscribeGeneratorTemplates, templateSupportsVideo, templateUsable } from "../../lib/settings";
import { refreshDownloadedWeights } from "../../lib/weightDownloads";
import { DebugPromptDialog } from "./DebugPromptDialog";
+import { videoGuidedScene } from "../../lib/videoLibrary";
interface GeneratorViewProps {
+ sceneIds?: readonly string[];
config: ProjectConfig;
folderPath: string;
runtime?: SlopfabStatus | null;
@@ -62,8 +64,8 @@ const readDensity = (): BoardDensity => {
* The panel is whatever is open — one shot, or the scene itself. Nothing is
* edited on the board, and nothing is duplicated in the panel: there is exactly
* one place to change any given thing. */
-export function GeneratorView({ config, folderPath, runtime = null, generationCompletionTimes = {}, onChange, selectedJobId, onSelectedJobChange, onRuntimeChange, onGenerate, onCancelGeneration, cancellingJobIds = new Set() }: GeneratorViewProps) {
- const jobs = config.generationJobs;
+export function GeneratorView({ config, sceneIds, folderPath, runtime = null, generationCompletionTimes = {}, onChange, selectedJobId, onSelectedJobChange, onRuntimeChange, onGenerate, onCancelGeneration, cancellingJobIds = new Set() }: GeneratorViewProps) {
+ const jobs = sceneIds ? sceneIds.flatMap((id) => { const job = config.generationJobs.find((job) => job.id === id); return job ? [job] : []; }) : config.generationJobs;
const [selection, setSelection] = useState<GeneratorSelection>(() => ({ jobId: selectedJobId ?? jobs.find((job) => job.status === "generating")?.id ?? jobs[0]?.id ?? "", shotId: null }));
const [templateSettings, setTemplateSettings] = useState(loadGeneratorTemplateSettings);
const templateFor = (job?: GenerationJob) => templateSettings.templates.find(t => t.id === job?.generatorTemplateId) ?? defaultGeneratorTemplate(templateSettings, "video");
@@ -162,7 +164,7 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
const active = jobs.filter((job) => job.status === "generating" || job.status === "ready");
const queued = jobs.filter((job) => job.status === "queued");
const cancellable = jobs.filter((job) => job.status === "queued" || job.status === "generating");
- const frameInputs = useMemo(() => selected ? sceneFrameInputs(selected, config) : null, [config, selected]);
+ const frameInputs = useMemo(() => { if (!selected) return null; const guided = videoGuidedScene(selected, config); return sceneFrameInputs(guided.job, guided.config); }, [config, selected]);
const boundRefs = frameInputs?.references ?? [];
const runtimeReady = Boolean(selectedTemplate.id) && generatorRuntime?.state === "ready";
@@ -365,7 +367,7 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
const newScene = () => {
const job = createDraftGenerationJob("", { steps: defaultGenerationSteps,
sceneType: selectedTemplate.mode === "animate" ? "animate" : selected?.sceneType === "single-shot" ? "single-shot" : selected?.sceneType === "long-shot" ? "long-shot" : "prompt" });
- onChange({ ...config, generationJobs: [...jobs, job] });
+ onChange({ ...config, generationJobs: [...config.generationJobs, job] });
setSelection({ jobId: job.id, shotId: null });
};
@@ -378,7 +380,8 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
const generationJobs = current.generationJobs.filter((item) => item.id !== job.id);
onChange({ ...current, generationJobs });
if (selection.jobId === job.id) {
- const fallback = generationJobs[Math.min(at, generationJobs.length - 1)];
+ const remaining = sceneIds ? generationJobs.filter((item) => sceneIds.includes(item.id)) : generationJobs;
+ const fallback = remaining[Math.min(jobs.findIndex((item) => item.id === job.id), remaining.length - 1)];
setSelection({ jobId: fallback?.id ?? "", shotId: null });
}
};
@@ -413,17 +416,17 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
const visiting = new Set<string>();
const visitForBatch = (job: GenerationJob): boolean => {
if (job.generatorTemplateId && !templateSettings.templates.some(t => t.id === job.generatorTemplateId) || !templateUsable(templateFor(job))) return false;
- if (continuationBlocker(job, jobs) || longShotBlocker(job, config, false) || templateSceneBlocker(sceneTypeFor(job), templateFor(job)) || sendBlocker(job, config.references)) return false;
+ if (continuationBlocker(job, config.generationJobs) || longShotBlocker(job, config, false) || templateSceneBlocker(sceneTypeFor(job), templateFor(job)) || sendBlocker(job, config.references, config)) return false;
if (visited.has(job.id)) return true;
if (visiting.has(job.id)) return false;
if (["queued", "generating", "ready"].includes(job.status)) { visited.add(job.id); return true; }
- const previous = jobs.find((candidate) => candidate.id === continuationSceneId(job, jobs));
- const dependencies = [...(previous ? [previous] : []), ...longShotDependencies(job, jobs)];
+ const previous = config.generationJobs.find((candidate) => candidate.id === continuationSceneId(job, config.generationJobs));
+ const dependencies = [...(previous ? [previous] : []), ...longShotDependencies(job, sceneSiblings(job, config))];
visiting.add(job.id);
if (dependencies.some((source) => !visitForBatch(source))) { visiting.delete(job.id); return false; }
visiting.delete(job.id);
visited.add(job.id);
- const priorBridge = longShotPosition(job, jobs)?.previousBridge;
+ const priorBridge = longShotPosition(job, sceneSiblings(job, config))?.previousBridge;
const previousWillRender = [...dependencies, ...(priorBridge ? [priorBridge] : [])].some((source) =>
batchScenesReady.includes(source) || ["queued", "generating", "ready"].includes(source.status));
if (previousWillRender || sceneGenerationSeed(job) === RANDOM_GENERATION_SEED || job.status !== "completed"
@@ -442,18 +445,18 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
if (!templateUsable(templateFor(job))) return "Prepare this scene's generator weights before generating.";
if (!runtimeReady) return runtimeError ?? generatorRuntime?.detail ?? "The video generator is not ready.";
if (job.status === "ready") return "Saving this scene.";
- const source = jobs.find((candidate) => candidate.id === continuationSceneId(job, jobs));
- return continuationBlocker(job, jobs)
+ const source = config.generationJobs.find((candidate) => candidate.id === continuationSceneId(job, config.generationJobs));
+ return continuationBlocker(job, config.generationJobs)
?? longShotBlocker(job, config)
?? (source && (!source.latentRelativePath || source.status !== "completed") ? "Generate the selected source scene first, or use Generate all." : null)
- ?? templateSceneBlocker(sceneTypeFor(job), templateFor(job)) ?? sendBlocker(job, config.references);
+ ?? templateSceneBlocker(sceneTypeFor(job), templateFor(job)) ?? sendBlocker(job, config.references, config);
};
const changedJobIds = new Set(jobs
.filter((job) => job.status === "completed"
&& Boolean(job.generationSnapshot)
&& (job.generationSnapshot !== snapshotFor(job)
- || jobs.some((source) => (source.id === continuationSceneId(job, jobs) || longShotDependencies(job, jobs).includes(source)) && ["queued", "generating", "ready"].includes(source.status))))
+ || config.generationJobs.some((source) => (source.id === continuationSceneId(job, config.generationJobs) || longShotDependencies(job, sceneSiblings(job, config)).includes(source)) && ["queued", "generating", "ready"].includes(source.status))))
.map((job) => job.id));
const selectedIndicator = selected && changedJobIds.has(selected.id) && !selected.generationHistory?.selectedId ? "changed" : selected?.status;
const selectedCancellable = selected?.status === "queued" || selected?.status === "generating";
@@ -685,7 +688,8 @@ export function GeneratorView({ config, folderPath, runtime = null, generationCo
folderPath={folderPath}
importAvailable={isTauri()}
importError={startFrameError}
- scenes={jobs}
+ scenes={config.generationJobs}
+ neighborScenes={jobs}
onAddStartFrame={() => void addFrame(selected, "start")}
onAddEndFrame={() => void addFrame(selected, "end")}
onChange={(updates) => changeSceneSettings(selected, updates)}
diff --git a/src/components/workspace/SceneEditor.tsx b/src/components/workspace/SceneEditor.tsx
index 4e00892..ebd8dee 100644
--- a/src/components/workspace/SceneEditor.tsx
+++ b/src/components/workspace/SceneEditor.tsx
@@ -315,7 +315,7 @@ const SCENE_TYPE_DESCRIPTIONS: Record<SceneType, string> = {
* §4.1), and the two sound fields defined per prompt (§4.6, §4.7). Length
* lives in the scene header where it stays visible. Simple label + control
* pairs are inspector rows; the free-text fields stay full width. */
-export function SceneInspector({ job, shots, references, folderPath = "", scenes = [], defaultSteps, effectiveSteps, template, defaultLook, disabled, importAvailable, importError, onAddStartFrame, onAddEndFrame, onChange, onShots, sceneType = sceneTypeFor(job) }: {
+export function SceneInspector({ job, shots, references, folderPath = "", scenes = [], neighborScenes = scenes, defaultSteps, effectiveSteps, template, defaultLook, disabled, importAvailable, importError, onAddStartFrame, onAddEndFrame, onChange, onShots, sceneType = sceneTypeFor(job) }: {
sceneType?: SceneType;
defaultLook?: string | null;
job: GenerationJob;
@@ -323,6 +323,7 @@ export function SceneInspector({ job, shots, references, folderPath = "", scenes
references: ProjectReference[];
folderPath?: string;
scenes?: GenerationJob[];
+ neighborScenes?: GenerationJob[];
defaultSteps: number;
template?: GeneratorTemplate;
effectiveSteps?: number;
@@ -337,7 +338,7 @@ export function SceneInspector({ job, shots, references, folderPath = "", scenes
const animate = sceneType === "animate";
const continuing = sceneType === "continue";
const backdrop = sceneType === "backdrop";
- const longShot = longShotPosition(job, scenes);
+ const longShot = longShotPosition(job, neighborScenes);
const pose = sceneType === "pose";
const characterReplace = sceneType === "character-replace";
const transition = isVideoTransition({ sceneType });
diff --git a/src/components/workspace/VideoLibraryView.test.tsx b/src/components/workspace/VideoLibraryView.test.tsx
new file mode 100644
index 0000000..b30f5e9
--- /dev/null
+++ b/src/components/workspace/VideoLibraryView.test.tsx
@@ -0,0 +1,88 @@
+// @vitest-environment jsdom
+import "@testing-library/jest-dom/vitest";
+import { act, cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
+import { useState } from "react";
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { createDraftGenerationJob, createProjectConfig, generationAssetId, type ProjectConfig } from "../../lib/project";
+import { newVideo } from "../../lib/videoLibrary";
+import { readMediaFileUrl } from "../../lib/persistence";
+import { VideoLibraryView } from "./VideoLibraryView";
+import { VideoPlayback } from "./VideoPlayback";
+import { choose } from "./comboTestUtils";
+
+vi.mock("../../lib/persistence", async (original) => ({ ...await original<typeof import("../../lib/persistence")>(), readMediaFileUrl: vi.fn(async (_, item) => `blob:${item.relativePath}`) }));
+vi.mock("../../lib/nativeShell", async (original) => ({ ...await original<typeof import("../../lib/nativeShell")>(), askNative: vi.fn(async () => true) }));
+vi.mock("./GeneratorView", () => ({ GeneratorView: ({ config, sceneIds, onChange }: { config: ProjectConfig; sceneIds: string[]; onChange: (next: ProjectConfig) => void }) => <div aria-label="Scenes editor">
+ <span>{sceneIds.join(",")}</span><span data-testid="all-scenes">{config.generationJobs.length}</span>
+ <button onClick={() => onChange({ ...config, generationJobs: [...config.generationJobs, createDraftGenerationJob("New", { id: "added" })] })}>Add scene</button>
+</div> }));
+
+beforeEach(() => {
+ localStorage.clear();
+ vi.spyOn(HTMLMediaElement.prototype, "play").mockResolvedValue(); vi.spyOn(HTMLMediaElement.prototype, "pause").mockImplementation(() => undefined);
+ vi.stubGlobal("URL", Object.assign(URL, { revokeObjectURL: vi.fn() }));
+});
+afterEach(() => { cleanup(); vi.restoreAllMocks(); vi.clearAllMocks(); vi.unstubAllGlobals(); });
+const project = () => createProjectConfig({ name: "Library", prompt: "", resolution: "416p", aspectRatio: "16:9", targetDurationSeconds: 30 });
+function mount(config = project(), openScene?: { id: string }) {
+ let latest = config;
+ function Harness() { const [value, setValue] = useState(config); latest = value; return <VideoLibraryView config={value} folderPath="D:/Project" onChange={setValue} onOpenTimeline={() => {}} openScene={openScene} />; }
+ return { ...render(<Harness />), config: () => latest };
+}
+
+it("creates nested folders and videos, moves cards to a folder, and opens only that video's scenes", () => {
+ const config = project(); config.generationJobs = [createDraftGenerationJob("Existing", { id: "legacy" })]; const view = mount(config);
+ fireEvent.click(screen.getByRole("button", { name: "New folder" })); fireEvent.change(screen.getByRole("textbox", { name: "Folder name" }), { target: { value: "Album" } }); fireEvent.click(screen.getByRole("button", { name: "Create folder" }));
+ fireEvent.click(within(screen.getByRole("toolbar", { name: "Video library actions" })).getByRole("button", { name: "New video" }));
+ const video = view.config().videos!.find((video) => video.title === "New video")!;
+ expect(video.folderId).toBe(view.config().videoFolders![0].id);
+ fireEvent.change(screen.getByRole("textbox", { name: "Video title" }), { target: { value: "Track one" } });
+ fireEvent.doubleClick(screen.getByRole("button", { name: "Select Track one" }));
+ expect(screen.getByTestId("all-scenes")).toHaveTextContent("1"); fireEvent.click(screen.getByRole("button", { name: "Add scene" }));
+ expect(view.config().videos!.find((item) => item.id === video.id)?.sceneIds).toEqual(["added"]);
+ expect(view.config().generationJobs.map((scene) => scene.id)).toEqual(["legacy", "added"]);
+ fireEvent.click(screen.getByRole("button", { name: "Back to Video" }));
+ fireEvent.click(within(screen.getByRole("navigation", { name: "Video path" })).getByRole("button", { name: "Video" }));
+ const dataTransfer = { types: ["application/x-slopus-video"], getData: () => JSON.stringify({ projectId: config.id, videoId: "video-default" }), dropEffect: "" };
+ fireEvent.drop(screen.getByRole("button", { name: "Open Album" }).closest('[role="listitem"]')!, { dataTransfer });
+ expect(view.config().videos!.find((item) => item.id === "video-default")?.folderId).toBe(video.folderId);
+ fireEvent.click(screen.getByRole("button", { name: "List view" })); fireEvent.click(screen.getByRole("button", { name: "Open Album" }));
+ expect(screen.getByRole("button", { name: "Select Track one" })).toBeInTheDocument(); expect(screen.getByRole("button", { name: "Select Library" })).toBeInTheDocument();
+});
+
+it("creates a whole-song scene sequence and opens the owner for a timeline navigation", () => {
+ const config = project(); config.videos = [{ ...newVideo(null), title: "Song video" }];
+ config.assets = [{ id: "song", kind: "audio", name: "Complete song", mimeType: "audio/wav", relativePath: "music/song.wav", durationMs: 25000, createdAt: config.createdAt }];
+ const view = mount(config); choose("Background music", "Complete song"); fireEvent.click(screen.getByRole("button", { name: "Create scenes for song" }));
+ expect(view.config().generationJobs).toHaveLength(3); expect(view.config().videos![0].sceneIds).toHaveLength(3);
+ expect(screen.getByLabelText("Scenes editor")).toBeInTheDocument();
+ const next = view.config(); view.unmount(); mount(next, { id: next.generationJobs[1].id }); expect(screen.getByLabelText("Scenes editor")).toBeInTheDocument();
+});
+
+it("removes folders without deleting their videos and removes video scenes without removing timeline media", async () => {
+ const config = project(); config.videoFolders = [{ id: "folder", name: "Collection" }]; config.videos = [{ ...newVideo("folder"), title: "Film", sceneIds: ["a"] }]; config.generationJobs = [createDraftGenerationJob("A", { id: "a" })];
+ const view = mount(config); fireEvent.contextMenu(screen.getByRole("button", { name: "Open Collection" })); fireEvent.click(screen.getByRole("menuitem", { name: "Remove folder" }));
+ expect(view.config().videoFolders).toHaveLength(0); expect(screen.getByRole("button", { name: "Select Film" })).toBeInTheDocument();
+ fireEvent.contextMenu(screen.getByRole("button", { name: "Select Film" })); fireEvent.click(screen.getByRole("menuitem", { name: "Remove video" }));
+ await waitFor(() => expect(view.config().videos).toHaveLength(0)); expect(view.config().generationJobs).toHaveLength(0); expect(view.config().timeline).toEqual(config.timeline);
+});
+
+it("plays successive scene ranges, replays the full video, closes outside the popout and releases media", async () => {
+ const config = project(); config.generationJobs = ["a", "b"].map((id) => createDraftGenerationJob("Dance", { id, durationSeconds: 3 }));
+ config.assets = config.generationJobs.map((job, index) => ({ id: generationAssetId(job.id), kind: "generated", name: job.id, relativePath: `media/${job.id}.mp4`, mimeType: "video/mp4", createdAt: config.createdAt, durationMs: 10000,
+ sceneSegments: [{ sceneId: job.id, startFrame: index * 96, frameCount: 96, latentRelativePath: `latents/${job.id}.safetensors` }] }));
+ const video = { ...newVideo(null), title: "Film", sceneIds: ["a", "b"] }, close = vi.fn();
+ const view = render(<VideoPlayback video={video} config={config} folderPath="D:/Project" popout onClose={close} />);
+ const player = view.container.querySelector("video")!;
+ await waitFor(() => expect(player).toHaveAttribute("src", "blob:media/a.mp4"));
+ await act(async () => { fireEvent.loadedMetadata(player); }); expect(player.currentTime).toBe(0);
+ player.currentTime = 3; fireEvent.timeUpdate(player);
+ await waitFor(() => expect(player).toHaveAttribute("src", "blob:media/b.mp4"));
+ await act(async () => { fireEvent.loadedMetadata(player); }); expect(player.currentTime).toBe(4);
+ fireEvent.ended(player); fireEvent.click(screen.getByRole("button", { name: "Replay Film" }));
+ await waitFor(() => expect(player).toHaveAttribute("src", "blob:media/a.mp4"));
+ fireEvent.pointerDown(screen.getByRole("dialog")); expect(close).not.toHaveBeenCalled();
+ fireEvent.pointerDown(view.container.querySelector(".shot-popout")!); expect(close).toHaveBeenCalledTimes(1);
+ view.unmount(); expect(URL.revokeObjectURL).toHaveBeenCalledWith("blob:media/a.mp4"); expect(URL.revokeObjectURL).toHaveBeenCalledWith("blob:media/b.mp4");
+ expect(readMediaFileUrl).toHaveBeenCalledTimes(3);
+});
diff --git a/src/components/workspace/VideoLibraryView.tsx b/src/components/workspace/VideoLibraryView.tsx
new file mode 100644
index 0000000..2ff4d3d
--- /dev/null
+++ b/src/components/workspace/VideoLibraryView.tsx
@@ -0,0 +1,147 @@
+import { useEffect, useRef, useState, type ComponentProps, type DragEvent } from "react";
+import { referenceTypeLabel, sceneDurationSeconds, type ProjectConfig, type ProjectVideo, type VideoSource } from "../../lib/project";
+import { citableReferences } from "./SceneEditor";
+import { referenceMediaTypes } from "../../lib/referenceSelection";
+import { mergeVideoScenes, newVideo, resolveVideoSource, songScenes, videoLibrary, videoScenes } from "../../lib/videoLibrary";
+import { newMusicFolder } from "../../lib/music";
+import { askNative } from "../../lib/nativeShell";
+import { CommandBarButton, ComboBox, PropRow, PropSection, Splitter, useContextMenu, usePaneSize, type MenuEntry } from "../ui";
+import { Add16, Delete16, Film32, FolderAdd16, FolderOpen16, GridView16, ListView16, OpenExternal16, Play16 } from "../ui/icons";
+import { GeneratorView } from "./GeneratorView";
+import { VideoPlayback, videoPlaylist } from "./VideoPlayback";
+import { PromptTextField } from "./PromptTextField";
+
+const VIDEO_DRAG = "application/x-slopus-video";
+const sourceValue = (source: VideoSource | null | undefined) => source ? `${source.kind}:${source.id}` : "";
+const parseSource = (value: string): VideoSource | null => value ? { kind: value.startsWith("asset:") ? "asset" : "reference", id: value.slice(value.indexOf(":") + 1) } : null;
+
+export function VideoLibraryView(props: ComponentProps<typeof GeneratorView> & { openScene?: { id: string }; onSceneOpened?: () => void }) {
+ const { config, onChange, folderPath } = props;
+ const latest = useRef(config); latest.current = config;
+ const videos = videoLibrary(config), folders = config.videoFolders ?? [];
+ const [folderId, setFolderId] = useState<string | null>(null);
+ const [selectedId, setSelectedId] = useState<string>();
+ const [openId, setOpenId] = useState<string>();
+ const [playback, setPlayback] = useState<{ id: string; popout: boolean }>();
+ const [creatingFolder, setCreatingFolder] = useState(false), [folderName, setFolderName] = useState("");
+ const [view, setView] = useState<"grid" | "list">(() => localStorage.getItem("slopus.video.view") === "list" ? "list" : "grid");
+ const pane = usePaneSize("video.inspector", 340, { min: 280, max: 560 });
+ const menu = useContextMenu();
+ const currentFolder = folders.find((folder) => folder.id === folderId), currentFolderId = currentFolder?.id ?? null;
+ const visible = videos.filter((video) => (video.folderId ?? null) === currentFolderId);
+ const selected = visible.find((video) => video.id === selectedId) ?? visible[0];
+ const open = videos.find((video) => video.id === openId);
+ const playing = videos.find((video) => video.id === playback?.id);
+ const breadcrumbs: { id: string | null; name: string }[] = [];
+ const visited = new Set<string>();
+ let ancestor = currentFolder;
+ while (ancestor && !visited.has(ancestor.id)) {
+ visited.add(ancestor.id); breadcrumbs.unshift({ id: ancestor.id, name: ancestor.name });
+ const parent = ancestor.parentId; ancestor = folders.find((folder) => folder.id === parent);
+ }
+ breadcrumbs.unshift({ id: null, name: "Video" });
+ useEffect(() => {
+ if (!props.openScene) return;
+ const owner = videoLibrary(latest.current).find((video) => video.sceneIds.includes(props.openScene!.id));
+ if (owner) { setFolderId(owner.folderId ?? null); setSelectedId(owner.id); setOpenId(owner.id); setPlayback(undefined); }
+ props.onSceneOpened?.();
+ }, [props.openScene]);
+ const update = (change: (current: ProjectConfig) => ProjectConfig, key?: string) => {
+ const current = latest.current;
+ onChange(change({ ...current, videos: videoLibrary(current) }), key);
+ };
+ const edit = (patch: Partial<ProjectVideo>) => { if (selected) update((current) => ({ ...current, videos: current.videos!.map((video) => video.id === selected.id ? { ...video, ...patch } : video) }), `video:${selected.id}`); };
+ const add = () => {
+ const video = newVideo(currentFolderId);
+ update((current) => ({ ...current, videos: [...current.videos!, video] })); setSelectedId(video.id);
+ };
+ const browse = (id: string | null) => { setFolderId(id); setPlayback(undefined); setCreatingFolder(false); menu.close(); };
+ const openVideo = (video: ProjectVideo) => { setSelectedId(video.id); setOpenId(video.id); setPlayback(undefined); menu.close(); };
+ const newItems = (): MenuEntry[] => [
+ { label: "New video", icon: <Add16 />, onSelect: add },
+ { label: "New folder", icon: <FolderAdd16 />, onSelect: () => { setFolderName(""); setCreatingFolder(true); } },
+ ];
+ const remove = async (video: ProjectVideo) => {
+ if (!await askNative({ message: `Remove ${video.title} and its scenes? Existing timeline clips and media will be kept.`, title: "Remove video", kind: "warning" })) return;
+ update((current) => ({ ...current, videos: current.videos!.filter((item) => item.id !== video.id), generationJobs: current.generationJobs.filter((scene) => !video.sceneIds.includes(scene.id)) }));
+ setPlayback(undefined);
+ };
+ const dropProps = (destination: string | null) => ({
+ onDragOver: (event: DragEvent<HTMLElement>) => { if (event.dataTransfer.types.includes(VIDEO_DRAG)) { event.preventDefault(); event.dataTransfer.dropEffect = "move"; } },
+ onDrop: (event: DragEvent<HTMLElement>) => {
+ event.preventDefault(); event.stopPropagation();
+ try {
+ const data = JSON.parse(event.dataTransfer.getData(VIDEO_DRAG));
+ if (data.projectId === config.id && typeof data.videoId === "string") update((current) => ({ ...current, videos: current.videos!.map((video) => video.id === data.videoId ? { ...video, folderId: destination } : video) }));
+ } catch { /* Ignore unrelated drags. */ }
+ },
+ });
+ const sourceOptions = (kind: "audio" | "video") => [
+ { value: "", label: "None" },
+ ...config.assets.filter((asset) => (asset.kind === kind || kind === "video" && asset.kind === "generated") && (asset.relativePath || asset.sourcePath)).map((asset) => ({ value: `asset:${asset.id}`, label: config.musicSongs?.find((song) => song.assetId === asset.id)?.title ?? asset.name })),
+ ...config.references.filter((reference) => reference.kind === kind && (reference.relativePath || reference.sourcePath)).map((reference) => ({ value: `reference:${reference.id}`, label: `${reference.name} (reference range)` })),
+ ];
+ if (open) return <div className="video-scenes">
+ <div className="music-toolbar video-scenes__path"><button type="button" className="secondary-button" onClick={() => setOpenId(undefined)}>Back to Video</button><strong>{open.title}</strong><span>Scenes</span></div>
+ <GeneratorView {...props} key={open.id} config={{ ...config, videos }} sceneIds={open.sceneIds} selectedJobId={open.sceneIds.includes(props.selectedJobId ?? "") ? props.selectedJobId : undefined}
+ onChange={(next, key) => onChange(mergeVideoScenes(latest.current, open.id, next), key)} />
+ </div>;
+ return <div className="video-library-view" style={pane.style}>
+ <div className="music-toolbar" role="toolbar" aria-label="Video library actions">
+ <CommandBarButton icon={<Add16 />} label="New video" showLabel onClick={add} />
+ <CommandBarButton icon={<FolderAdd16 />} label="New folder" showLabel onClick={() => { setFolderName(""); setCreatingFolder(true); }} />
+ <div className="music-toolbar__views" role="group" aria-label="Video view">{(["grid", "list"] as const).map((mode) => <button type="button" className="icon-button" key={mode} aria-label={`${mode === "grid" ? "Grid" : "List"} view`} aria-pressed={view === mode} onClick={() => { setView(mode); localStorage.setItem("slopus.video.view", mode); }}>{mode === "grid" ? <GridView16 /> : <ListView16 />}</button>)}</div>
+ </div>
+ <div className="video-library-body">
+ <div className="video-library-stage">
+ <div className="music-library" aria-label="Video library" tabIndex={0} onContextMenu={(event) => { if (!(event.target as HTMLElement).closest("input, textarea")) menu.open(event, newItems()); }}>
+ <div className="music-library__heading"><nav aria-label="Video path"><h2 className="music-path">{breadcrumbs.map((folder, index) => <span key={folder.id ?? "root"}>{index > 0 && <span className="music-path__separator">/</span>}<button type="button" aria-current={folder.id === currentFolderId ? "page" : undefined} onClick={() => browse(folder.id)} {...dropProps(folder.id)}>{folder.name}</button></span>)}</h2></nav><span>{visible.length} videos</span></div>
+ {creatingFolder && <form className="music-new-folder" onSubmit={(event) => { event.preventDefault(); if (!folderName.trim()) return; const folder = newMusicFolder(folderName, currentFolderId); update((current) => ({ ...current, videoFolders: [...(current.videoFolders ?? []), folder] })); browse(folder.id); }}>
+ <input className="text-field" aria-label="Folder name" maxLength={200} autoFocus value={folderName} onChange={(event) => setFolderName(event.target.value)} />
+ <button className="primary-button" disabled={!folderName.trim()}>Create folder</button><button type="button" className="secondary-button" onClick={() => setCreatingFolder(false)}>Cancel</button>
+ </form>}
+ <div className={`music-songs music-songs--${view}`} role="list" aria-label="Video contents">
+ {folders.filter((folder) => (folder.parentId ?? null) === currentFolderId).map((folder) => <div className="music-song music-folder" role="listitem" key={folder.id} {...dropProps(folder.id)} onContextMenu={(event) => menu.open(event, [...newItems(), { separator: true }, { label: "Remove folder", icon: <Delete16 />, onSelect: () => update((current) => ({ ...current, videoFolders: current.videoFolders?.filter((item) => item.id !== folder.id).map((item) => item.parentId === folder.id ? { ...item, parentId: folder.parentId } : item), videos: current.videos!.map((video) => video.folderId === folder.id ? { ...video, folderId: folder.parentId } : video) })) }])}>
+ <button type="button" className="music-song__select" aria-label={`Open ${folder.name}`} onClick={() => browse(folder.id)}><span className="music-song__art"><FolderOpen16 /></span><span className="music-song__info"><strong>{folder.name}</strong></span></button>
+ </div>)}
+ {visible.map((video) => {
+ const scenes = videoScenes(video, config), playable = videoPlaylist(video, config).length;
+ const busy = scenes.some((scene) => ["queued", "generating"].includes(scene.status));
+ return <div key={video.id} role="listitem" className={`music-song video-card${selected?.id === video.id ? " music-song--selected" : ""}${playable ? "" : " music-song--draft"}`} draggable onDragStart={(event) => { event.dataTransfer.effectAllowed = "move"; event.dataTransfer.setData(VIDEO_DRAG, JSON.stringify({ projectId: config.id, videoId: video.id })); }}
+ onContextMenu={(event) => menu.open(event, [{ label: "Open scenes", onSelect: () => openVideo(video) }, ...newItems(), { separator: true }, { label: "Remove video", icon: <Delete16 />, disabled: busy, danger: true, onSelect: () => { void remove(video); } }])}>
+ <button type="button" className="music-song__select" aria-label={`Select ${video.title}`} aria-pressed={selected?.id === video.id} onClick={() => setSelectedId(video.id)} onDoubleClick={() => openVideo(video)}>
+ <span className="music-song__art"><Film32 /></span><span className="music-song__info"><strong>{video.title}</strong><span>{scenes.length} scenes · {scenes.reduce((seconds, scene) => seconds + sceneDurationSeconds(scene), 0).toFixed(1)} s</span><span>{busy ? "Generating" : playable ? `${playable} / ${scenes.length} scenes ready` : "Draft"}</span></span>
+ </button>
+ {playable > 0 && <div className="video-card__controls"><button type="button" aria-label={`Play ${video.title}`} onClick={() => setPlayback({ id: video.id, popout: false })}><Play16 /></button><button type="button" aria-label={`Pop out ${video.title}`} onClick={() => setPlayback({ id: video.id, popout: true })}><OpenExternal16 /></button></div>}
+ <button type="button" className="video-card__open secondary-button" onClick={() => openVideo(video)}>Open scenes</button>
+ {playing?.id === video.id && !playback?.popout && <VideoPlayback key={video.id} video={video} config={config} folderPath={folderPath} popout={false} onClose={() => setPlayback(undefined)} />}
+ </div>;
+ })}
+ </div>
+ {!visible.length && !folders.some((folder) => (folder.parentId ?? null) === currentFolderId) && <div className="music-empty"><Film32 /><h3>Create a video</h3><p>Organize scenes and shots into a video, with shared prompts and references.</p><button type="button" className="primary-button" onClick={add}>New video</button></div>}
+ </div>
+ {playing && playback?.popout && <VideoPlayback key={playing.id} video={playing} config={config} folderPath={folderPath} popout onClose={() => setPlayback(undefined)} />}
+ </div>
+ <Splitter {...pane.splitterProps} reverse aria-label="Resize video inspector" />
+ <aside className="music-inspector" aria-label="Video settings"><div className="music-inspector__fields">{selected ? <>
+ <PropSection title="Video"><PropRow label="Title"><input className="text-field" aria-label="Video title" maxLength={200} value={selected.title} onChange={(event) => edit({ title: event.target.value || "New video" })} /></PropRow>
+ <div className="music-prompt"><span>Shared prompt</span><PromptTextField aria-label="Video prompt" rows={5} value={selected.prompt} references={citableReferences(config.references).map((reference) => ({ id: reference.id, name: reference.name, mediaTypes: referenceMediaTypes(reference), detail: referenceTypeLabel(reference) }))} placeholder="Visual style and references for every scene…" onChange={(prompt) => edit({ prompt: prompt.slice(0, 30000) })} /></div>
+ </PropSection>
+ <PropSection title="References">
+ <PropRow label="Music"><ComboBox aria-label="Background music" value={sourceValue(selected.musicSource)} options={sourceOptions("audio")} placeholder="Reference unavailable" onChange={(value) => edit({ musicSource: parseSource(value) })} /></PropRow>
+ <PropRow label="Video"><ComboBox aria-label="Reference video" value={sourceValue(selected.videoSource)} options={sourceOptions("video")} placeholder="Reference unavailable" onChange={(value) => edit({ videoSource: parseSource(value) })} /></PropRow>
+ <p className="video-reference-help">Each scene uses the matching time range of these references. Import media in Timeline or choose a generated song. Reference entries use their saved trim range.</p>
+ {selected.musicSource && <p className="video-reference-help">Music needs a visual reference: add an image to the shared prompt or select a reference video. Use a References or Singularity generator.</p>}
+ {selected.musicSource && !selected.sceneIds.length && (resolveVideoSource(selected.musicSource, config, "audio")?.durationSeconds ?? 0) >= 2 && <button type="button" className="secondary-button" onClick={() => {
+ const scenes = songScenes(selected, latest.current); if (!scenes.length) return;
+ update((current) => ({ ...current, generationJobs: [...current.generationJobs, ...scenes], videos: current.videos!.map((video) => video.id === selected.id ? { ...video, sceneIds: scenes.map((scene) => scene.id) } : video) })); openVideo(selected);
+ }}>Create scenes for song</button>}
+ </PropSection>
+ <PropSection title="Scenes"><button type="button" className="primary-button" onClick={() => openVideo(selected)}>Open scenes</button>
+ <p className="video-reference-help">{videoScenes(selected, config).map((scene, index, all) => { const start = all.slice(0, index).reduce((sum, item) => sum + sceneDurationSeconds(item), 0); return <span className="video-reference-range" key={scene.id}>{scene.title}: {start.toFixed(1)}–{(start + sceneDurationSeconds(scene)).toFixed(1)} s</span>; })}</p>
+ </PropSection>
+ </> : <p className="music-inspector__empty">Select a video to edit its settings.</p>}</div></aside>
+ </div>
+ {menu.element}
+ </div>;
+}
diff --git a/src/components/workspace/VideoPlayback.tsx b/src/components/workspace/VideoPlayback.tsx
new file mode 100644
index 0000000..e00607f
--- /dev/null
+++ b/src/components/workspace/VideoPlayback.tsx
@@ -0,0 +1,81 @@
+import { useEffect, useRef, useState, type CSSProperties } from "react";
+import { readMediaFileUrl } from "../../lib/persistence";
+import { sceneDurationSeconds, type ProjectConfig, type ProjectVideo } from "../../lib/project";
+import { scenePreviewAsset } from "../../lib/sceneHistory";
+import { sceneMediaStartSeconds, sceneMediaDurationMs } from "../../lib/continuationMedia";
+import { videoScenes } from "../../lib/videoLibrary";
+import { Dismiss16, Pause20, Play20Filled } from "../ui/icons";
+
+export function videoPlaylist(video: ProjectVideo, config: ProjectConfig) {
+ return videoScenes(video, config).flatMap((scene) => {
+ const asset = scenePreviewAsset(scene, config.assets);
+ if (!asset?.relativePath) return [];
+ const start = sceneMediaStartSeconds(asset);
+ const duration = Math.min(sceneDurationSeconds(scene), (sceneMediaDurationMs(asset) ?? sceneDurationSeconds(scene) * 1000) / 1000);
+ return [{ sceneId: scene.id, title: scene.title, asset, start, end: start + duration }];
+ });
+}
+
+/** Plays each scene's own range, excluding retained continuation prefixes. */
+export function VideoPlayback({ video: item, config, folderPath, popout, onClose }: {
+ video: ProjectVideo; config: ProjectConfig; folderPath: string; popout: boolean; onClose: () => void;
+}) {
+ const playlist = videoPlaylist(item, config);
+ const [index, setIndex] = useState(0), [url, setUrl] = useState<string | null>(null);
+ const [playing, setPlaying] = useState(false), [ended, setEnded] = useState(false), [error, setError] = useState<string | null>(null);
+ const [aspect, setAspect] = useState(16 / 9);
+ const media = useRef<HTMLVideoElement>(null), frame = useRef<HTMLDivElement>(null);
+ const close = useRef(onClose); close.current = onClose;
+ const timer = useRef<number>(), moving = useRef(false), epoch = useRef(0);
+ const current = playlist[index];
+ const advance = () => {
+ if (moving.current) return;
+ moving.current = true; clearTimeout(timer.current); media.current?.pause();
+ if (index + 1 < playlist.length) setIndex(index + 1);
+ else { setEnded(true); setPlaying(false); }
+ };
+ const play = async (restart = false) => {
+ const player = media.current;
+ if (!player || !current || !url) return;
+ if (restart || player.currentTime < current.start || player.currentTime >= current.end - 0.02) player.currentTime = current.start;
+ moving.current = false; setEnded(false); setError(null); clearTimeout(timer.current);
+ const token = ++epoch.current;
+ try {
+ await player.play();
+ if (token !== epoch.current) return;
+ setPlaying(true);
+ timer.current = window.setTimeout(advance, Math.max(0, (Math.min(current.end, player.duration || current.end) - player.currentTime) * 1000));
+ } catch { if (token === epoch.current) { setPlaying(false); setError("Could not play this scene. Try replaying it."); } }
+ };
+ useEffect(() => {
+ let disposed = false, owned: string | null = null;
+ epoch.current++; moving.current = false; setUrl(null); setError(null); setPlaying(false);
+ const player = media.current; player?.pause(); clearTimeout(timer.current);
+ if (current) void readMediaFileUrl(folderPath, current.asset, "video/mp4").then((value) => {
+ if (disposed) { if (value) URL.revokeObjectURL(value); return; }
+ owned = value; setUrl(value); if (!value) setError("This scene's video is unavailable.");
+ }).catch((reason) => { if (!disposed) setError(String(reason)); });
+ return () => { disposed = true; epoch.current++; clearTimeout(timer.current); player?.pause(); if (owned) URL.revokeObjectURL(owned); };
+ }, [folderPath, current?.sceneId, current?.asset.relativePath]);
+ useEffect(() => {
+ const focused = document.activeElement as HTMLElement | null;
+ if (popout) frame.current?.focus();
+ const escape = (event: KeyboardEvent) => { if (event.key === "Escape") { event.preventDefault(); close.current(); } };
+ document.addEventListener("keydown", escape);
+ return () => { document.removeEventListener("keydown", escape); if (popout && focused?.isConnected) focused.focus(); };
+ }, [popout]);
+ const player = <div className={popout ? "shot-popout__frame" : "video-card__player"} ref={frame} tabIndex={-1}
+ role={popout ? "dialog" : "group"} aria-label={`${item.title} preview`} style={{ "--shot-aspect": aspect } as CSSProperties}>
+ <video ref={media} src={url ?? undefined} playsInline onLoadedMetadata={(event) => { if (url) { setAspect(event.currentTarget.videoWidth / event.currentTarget.videoHeight || 16 / 9); void play(true); } }}
+ onTimeUpdate={(event) => { if (playing && current && event.currentTarget.currentTime >= current.end - 0.02) advance(); }} onEnded={advance}
+ onError={() => { if (url) { clearTimeout(timer.current); setPlaying(false); setError("This scene could not be played."); } }} />
+ <button type="button" className="shot-popout__close" aria-label="Close video preview" onClick={onClose}><Dismiss16 /></button>
+ <button type="button" className="shot-popout__play" aria-label={`${playing ? "Pause" : "Replay"} ${item.title}`} disabled={!url} onClick={() => {
+ if (playing) { epoch.current++; clearTimeout(timer.current); media.current?.pause(); setPlaying(false); }
+ else if (ended && index !== 0) { setEnded(false); setIndex(0); } else void play(ended);
+ }}>{playing ? <Pause20 /> : <Play20Filled />}</button>
+ {error && <p className="shot-popout__message" role="alert">{error}</p>}
+ <span className="video-card__position">{index + 1} / {playlist.length} · {current?.title}</span>
+ </div>;
+ return popout ? <div className="shot-popout" onPointerDown={(event) => { if (event.target === event.currentTarget) onClose(); }}>{player}</div> : player;
+}
diff --git a/src/lib/agentGeneration.ts b/src/lib/agentGeneration.ts
index 61a3ee8..af02fb7 100644
--- a/src/lib/agentGeneration.ts
+++ b/src/lib/agentGeneration.ts
@@ -130,7 +130,7 @@ export async function generateAgentScene(queue: WorkQueue, session: ProjectSessi
const blocker = continuationBlocker(job, config.generationJobs)
?? longShotBlocker(job, config)
?? (source && (!source.latentRelativePath || source.status !== "completed") ? "Generate the selected source scene first." : null)
- ?? templateSceneBlocker(sceneTypeFor(job), template) ?? sendBlocker(job, config.references);
+ ?? templateSceneBlocker(sceneTypeFor(job), template) ?? sendBlocker(job, config.references, config);
if (blocker) throw new Error(blocker);
const request = sceneGenerationRequest(job, config, session.record.folderPath, template.defaultSteps, template);
const [id] = queue.enqueue(session, [{ job, request, snapshot: sceneGenerationSnapshot(job, request) }], template);
diff --git a/src/lib/longShot.ts b/src/lib/longShot.ts
index 6788bef..64026b8 100644
--- a/src/lib/longShot.ts
+++ b/src/lib/longShot.ts
@@ -1,4 +1,4 @@
-import { generationAssetId, sceneDurationSeconds, sceneOutputFrames, type GenerationJob, type ProjectAsset, type ProjectConfig, type SceneGenerationInput, type SceneMediaSegment } from "./project";
+import { sceneSiblings, generationAssetId, sceneDurationSeconds, sceneOutputFrames, type GenerationJob, type ProjectAsset, type ProjectConfig, type SceneGenerationInput, type SceneMediaSegment } from "./project";
export const LONG_SHOT_CONTEXT = 22;
export const bridgeGapFrames = (frames: number) => Math.max(12, Math.ceil((frames - 12) / 17) * 17 + 12);
@@ -23,7 +23,7 @@ export function longShotDependencies(job: GenerationJob, jobs: readonly Generati
}
export function longShotBlocker(job: GenerationJob, config: ProjectConfig, requireCompleted = true): string | null {
- const position = longShotPosition(job, config.generationJobs);
+ const position = longShotPosition(job, sceneSiblings(job, config));
if (!position?.left) return null;
const neighbors: [GenerationJob, number][] = [[position.left, position.continuation ? 0 : job.bridgeLeftMargin ?? 17]];
if (position.right) neighbors.push([position.right, job.bridgeRightMargin ?? 17]);
@@ -41,25 +41,30 @@ export function longShotBlocker(job: GenerationJob, config: ProjectConfig, requi
return null;
}
-export function currentLongShotArchives(config: Pick<ProjectConfig, "assets" | "generationJobs">): ProjectAsset[] {
+export function currentLongShotArchives(config: Pick<ProjectConfig, "assets" | "generationJobs" | "videos">): ProjectAsset[] {
const candidates = config.assets.filter((asset) => {
const parts = asset.sceneSegments;
const own = parts?.findIndex((part) => generationAssetId(part.sceneId) === asset.id) ?? -1;
if (!parts || own <= 0) return false;
- const first = config.generationJobs.findIndex((job) => job.id === parts[0].sceneId);
- if (first < 0 || !parts.every((part, offset) => config.generationJobs[first + offset]?.id === part.sceneId && config.generationJobs[first + offset]?.sceneType === "long-shot")) return false;
+ const firstScene = config.generationJobs.find((job) => job.id === parts[0].sceneId);
+ if (!firstScene) return false;
+ const siblings = sceneSiblings(firstScene, config);
+ const first = siblings.findIndex((job) => job.id === parts[0].sceneId);
+ if (first < 0 || !parts.every((part, offset) => siblings[first + offset]?.id === part.sceneId && siblings[first + offset]?.sceneType === "long-shot")) return false;
if (!parts.every((part) => config.generationJobs.some((job) => job.id === part.sceneId && job.status === "completed" && job.latentRelativePath === part.latentRelativePath))) return false;
const owner = config.generationJobs.find((job) => generationAssetId(job.id) === asset.id)!;
- const position = longShotPosition(owner, config.generationJobs);
+ const position = longShotPosition(owner, sceneSiblings(owner, config));
if (!position?.left || position.left.id !== parts[own - 1].sceneId) return false;
if (position.continuation ? own !== parts.length - 1 : !position.bridge || position.right?.id !== parts[own + 1]?.sceneId) return false;
return true;
- }).sort((a, b) => config.generationJobs.findIndex((job) => generationAssetId(job.id) === a.id)
- - config.generationJobs.findIndex((job) => generationAssetId(job.id) === b.id));
+ }).sort((a, b) => {
+ const rank = (asset: ProjectAsset) => { const owner = config.generationJobs.find((job) => generationAssetId(job.id) === asset.id)!; return sceneSiblings(owner, config).findIndex((job) => job.id === owner.id); };
+ return rank(a) - rank(b);
+ });
const valid: ProjectAsset[] = [];
for (const asset of candidates) {
const owner = config.generationJobs.find((job) => generationAssetId(job.id) === asset.id)!;
- const previous = longShotPosition(owner, config.generationJobs)?.previousBridge;
+ const previous = longShotPosition(owner, sceneSiblings(owner, config))?.previousBridge;
// A newly completed earlier bridge changes the shared anchor's head.
// A suffix must incorporate it, but an invalid earlier bridge is optional.
if (previous && valid.some((candidate) => candidate.id === generationAssetId(previous.id))
@@ -67,7 +72,7 @@ export function currentLongShotArchives(config: Pick<ProjectConfig, "assets" | "
// Reject chains that embed an earlier, now-incompatible joined archive.
if (asset.sceneSegments!.some((part) => {
const job = config.generationJobs.find((job) => job.id === part.sceneId)!;
- return job.id !== owner.id && longShotPosition(job, config.generationJobs)?.bridge
+ return job.id !== owner.id && longShotPosition(job, sceneSiblings(job, config))?.bridge
&& !valid.some((candidate) => candidate.id === generationAssetId(job.id));
})) continue;
valid.push(asset);
@@ -75,7 +80,7 @@ export function currentLongShotArchives(config: Pick<ProjectConfig, "assets" | "
return valid.sort((a, b) => b.sceneSegments!.length - a.sceneSegments!.length);
}
-export function longShotPlaybackSource(job: GenerationJob, config: Pick<ProjectConfig, "assets" | "generationJobs">) {
+export function longShotPlaybackSource(job: GenerationJob, config: Pick<ProjectConfig, "assets" | "generationJobs" | "videos">) {
const source = currentLongShotArchives(config).find((asset) => asset.sceneSegments?.some((part) => part.sceneId === job.id && part.latentRelativePath === job.latentRelativePath));
const segment = source?.sceneSegments?.find((part) => part.sceneId === job.id);
return source && segment ? { source, segment } : null;
@@ -84,7 +89,7 @@ export function longShotPlaybackSource(job: GenerationJob, config: Pick<ProjectC
/** A final unpaired scene extends the latest joined archive, so earlier
* bridge edits to the preceding anchor survive in its video and audio. */
export function longShotContinuationInputs(job: GenerationJob, config: ProjectConfig) {
- const position = longShotPosition(job, config.generationJobs);
+ const position = longShotPosition(job, sceneSiblings(job, config));
if (!position?.continuation || !position.left) return null;
const { left, previousBridge } = position;
const prefix = previousBridge && currentLongShotArchives(config).find((asset) => asset.id === generationAssetId(previousBridge.id)
@@ -100,7 +105,7 @@ export function longShotContinuationInputs(job: GenerationJob, config: ProjectCo
}
export function longShotInputs(job: GenerationJob, config: ProjectConfig): { bridge: NonNullable<SceneGenerationInput["latentBridge"]>; leftSegments?: SceneMediaSegment[] } | null {
- const position = longShotPosition(job, config.generationJobs);
+ const position = longShotPosition(job, sceneSiblings(job, config));
if (!position?.bridge) return null;
const { left, right, previousBridge } = position;
const prefix = previousBridge && currentLongShotArchives(config).find((asset) => asset.id === generationAssetId(previousBridge.id)
diff --git a/src/lib/project.ts b/src/lib/project.ts
index bd179e3..eb1e3f6 100644
--- a/src/lib/project.ts
+++ b/src/lib/project.ts
@@ -691,12 +691,24 @@ export const providerSettingSchema = z.object({
});
export const providerSettingsSchema = z.record(idSchema, providerSettingSchema);
+export const videoSourceSchema = z.object({ kind: z.enum(["asset", "reference"]), id: idSchema });
+export const projectVideoSchema = z.object({
+ id: idSchema, title: z.string().min(1).max(200), folderId: idSchema.nullish(),
+ sceneIds: z.array(idSchema).default([]), prompt: z.string().max(30000).default(""),
+ musicSource: videoSourceSchema.nullish(), videoSource: videoSourceSchema.nullish(),
+ createdAt: isoDateSchema,
+});
+export type ProjectVideo = z.infer<typeof projectVideoSchema>;
+export type VideoSource = z.infer<typeof videoSourceSchema>;
+
export const projectConfigSchema = z.object({
schemaVersion: z.literal(CURRENT_SCHEMA_VERSION),
generationType: generationTypeSchema.default("video"),
imageScene: imageSceneSchema.nullish(),
musicSongs: z.array(musicSongSchema).optional(),
musicFolders: z.array(musicFolderSchema).optional(),
+ videos: z.array(projectVideoSchema).optional(),
+ videoFolders: z.array(musicFolderSchema).optional(),
imageSettings: z.object({
resolution: resolutionSchema,
aspectRatio: aspectRatioSchema,
@@ -1204,6 +1216,18 @@ export function sceneDurationSeconds(job: GenerationJob): number {
return Math.min(SCENE_MAX_SECONDS, Math.max(SCENE_MIN_SECONDS, stored));
}
+/** Ordered neighbors belong to one Video; explicit continuation links remain project-wide. */
+export function sceneSiblings(job: GenerationJob, config: Pick<ProjectConfig, "videos" | "generationJobs">): GenerationJob[] {
+ if (!config.videos?.length) return config.generationJobs;
+ const assigned = new Set(config.videos.flatMap((video) => video.sceneIds));
+ const owner = config.videos.find((video) => video.sceneIds.includes(job.id))
+ ?? (!assigned.has(job.id) ? config.videos.find((video) => video.id === "video-default") : undefined);
+ const unassigned = config.generationJobs.filter((scene) => !assigned.has(scene.id));
+ if (!owner) return unassigned;
+ const members = owner.sceneIds.flatMap((id) => { const scene = config.generationJobs.find((scene) => scene.id === id); return scene ? [scene] : []; });
+ return owner.id === "video-default" ? [...members, ...unassigned] : members;
+}
+
export function sceneGenerationSteps(job: GenerationJob, fallback = DEFAULT_GENERATION_STEPS): number {
return job.steps ?? fallback;
}
diff --git a/src/lib/sceneGeneration.ts b/src/lib/sceneGeneration.ts
index edf386e..91af476 100644
--- a/src/lib/sceneGeneration.ts
+++ b/src/lib/sceneGeneration.ts
@@ -9,9 +9,12 @@ import {
usableReferenceImages, compileScenePromptSegments, type GenerationJob, type ProjectConfig, type ProjectReference, type SceneType,
} from "./project";
import type { SlopfabGenerationRequest } from "./runtime";
+import { videoGuidedScene } from "./videoLibrary";
import { generationSigmaShifts, templateSupportsVideo, type GeneratorTemplate } from "./settings";
export function sceneGenerationRequest(job: GenerationJob, config: ProjectConfig, folderPath: string, defaultGenerationSteps: number, template?: GeneratorTemplate): SlopfabGenerationRequest {
+ const guided = videoGuidedScene(job, config);
+ job = guided.job; config = guided.config;
// Only the references actually bound to this scene, in list order. The same
// ordered list drives the <Subject N> / <Picture N> numbering inside the
// compiled prompt, because slopfab.rs adds reference_paths sequentially — so
@@ -66,7 +69,11 @@ export function sceneGenerationRequest(job: GenerationJob, config: ProjectConfig
}
/** Validate the inputs required by the selected generator before submission. */
-export function sendBlocker(job: GenerationJob, references: ProjectReference[]): string | null {
+export function sendBlocker(job: GenerationJob, references: ProjectReference[], config?: ProjectConfig): string | null {
+ if (config) {
+ const guided = videoGuidedScene(job, config);
+ return guided.blocker ?? sendBlocker(guided.job, guided.config.references);
+ }
const shots = sceneShots(job);
if (job.sceneType === "single-shot") {
if (sceneDurationSeconds(job) < 2) return "Single Shot needs at least 2 seconds.";
diff --git a/src/lib/videoLibrary.test.ts b/src/lib/videoLibrary.test.ts
new file mode 100644
index 0000000..4418364
--- /dev/null
+++ b/src/lib/videoLibrary.test.ts
@@ -0,0 +1,93 @@
+import { expect, it } from "vitest";
+import { createDraftGenerationJob, createProjectConfig, generationAssetId, parseProjectConfig, sceneDurationSeconds, type ProjectAsset } from "./project";
+import { mergeVideoScenes, newVideo, resolveVideoSource, songScenes, videoGuidedScene, videoLibrary, videoSceneOffset } from "./videoLibrary";
+import { sceneGenerationRequest, sendBlocker } from "./sceneGeneration";
+import { videoPlaylist } from "../components/workspace/VideoPlayback";
+import { longShotBlocker, longShotInputs } from "./longShot";
+
+const project = () => createProjectConfig({ name: "Videos", prompt: "", resolution: "416p", aspectRatio: "16:9", targetDurationSeconds: 30 });
+const scene = (id: string, seconds = 10) => createDraftGenerationJob("A dancer moves", { id, sceneType: "prompt", durationSeconds: seconds });
+const asset = (id: string, kind: ProjectAsset["kind"] = "audio", durationMs = 30000): ProjectAsset => ({ id, kind, name: id, relativePath: `media/${id}.${kind === "audio" ? "wav" : "mp4"}`, mimeType: kind === "audio" ? "audio/wav" : "video/mp4", durationMs, createdAt: "2026-10-27T00:00:00.000Z" });
+
+it("keeps Long Shot neighbors within their Video but permits explicit continuation across Videos", () => {
+ const config = project(); config.generationJobs = ["a", "other", "b", "c"].map((id) => ({ ...scene(id), sceneType: "long-shot" as const, latentRelativePath: `latents/${id}.safetensors` }));
+ config.videos = [{ ...newVideo(null), sceneIds: ["a", "b", "c"] }, { ...newVideo(null), sceneIds: ["other"] }];
+ expect(longShotBlocker(config.generationJobs[1], config)).toBeNull();
+ expect(longShotInputs(config.generationJobs[2], config)?.bridge).toMatchObject({ leftSceneId: "a", rightSceneId: "c" });
+ const continued = { ...config.generationJobs[2], sceneType: "single-shot" as const, continuationSceneId: "other" };
+ expect(sceneGenerationRequest(continued, config, "D:/Project", 8).continuationRelativePath).toBe("latents/other.safetensors");
+});
+
+it("groups legacy and external scenes without changing scene or timeline identity", () => {
+ const config = project(); config.generationJobs = [scene("a"), scene("b")];
+ const original = structuredClone(config);
+ expect(videoLibrary(config)[0].sceneIds).toEqual(["a", "b"]);
+ expect(config).toEqual(original);
+ config.videos = videoLibrary(config); config.videoFolders = [{ id: "folder", name: "Campaign" }]; config.videos[0].folderId = "folder";
+ config.generationJobs.push(scene("external"));
+ expect(videoLibrary(config)[0].sceneIds).toEqual(["a", "b", "external"]);
+ const restored = parseProjectConfig(JSON.parse(JSON.stringify(config)));
+ expect(restored.videos).toEqual(config.videos); expect(restored.videoFolders).toEqual(config.videoFolders);
+ expect(restored.timeline).toEqual(config.timeline);
+});
+
+it("edits, adds, reorders and removes scenes within one video while retaining other videos", () => {
+ const config = project(); config.generationJobs = [scene("a"), scene("other"), scene("b")];
+ const first = { ...newVideo(null), sceneIds: ["a", "b"] }, second = { ...newVideo(null), sceneIds: ["other"] };
+ config.videos = [first, second];
+ const edited = { ...config, generationJobs: [scene("b"), scene("other"), scene("new")] };
+ const merged = mergeVideoScenes(config, first.id, edited);
+ expect(merged.videos?.map((video) => video.sceneIds)).toEqual([["b", "new"], ["other"]]);
+ expect(merged.generationJobs.map((job) => job.id)).toEqual(["b", "other", "new"]);
+ expect(merged.assets).toBe(config.assets); expect(merged.timeline).toBe(config.timeline);
+});
+
+it("slices music and video at cumulative scene times in video order and compiles consistent reference numbers", () => {
+ const config = project(); config.generationJobs = [scene("second", 8), scene("unrelated", 15), scene("first")];
+ const video = { ...newVideo(null), sceneIds: ["first", "second"], prompt: "Neon lights", musicSource: { kind: "asset" as const, id: "song" }, videoSource: { kind: "asset" as const, id: "guide" } };
+ config.videos = [video]; config.assets = [asset("song"), asset("guide", "video")];
+ const before = structuredClone(config), job = config.generationJobs[0];
+ expect(videoSceneOffset(video, job, config)).toBe(10);
+ expect(sendBlocker(job, config.references, config)).toBeNull();
+ const request = sceneGenerationRequest(job, config, "D:/Project", 8);
+ expect(request.referenceAudios?.[0]).toMatchObject({ startSeconds: 10, durationSeconds: 8, relativePath: "media/song.wav" });
+ expect(request.referenceVideos?.[0]).toMatchObject({ startSeconds: 10, durationSeconds: 8, includeAudio: false });
+ expect(request.prompt).toContain("Neon lights"); expect(request.prompt).toContain("<Audio 1>"); expect(request.prompt).toContain("<Video 1>");
+ expect(request.prompt).not.toContain("@[ref:"); expect(config).toEqual(before);
+ config.videos[0].sceneIds.reverse();
+ expect(sceneGenerationRequest(job, config, "D:/Project", 8).referenceAudios?.[0].startSeconds).toBe(0);
+});
+
+it("binds a shared image for a complete song and keeps Single Shot ending guidance", () => {
+ const config = project(), video = { ...newVideo(null), prompt: "@[ref:hero] dancing", musicSource: { kind: "asset" as const, id: "song" } };
+ config.assets = [asset("song", "audio", 201000)]; config.videos = [video];
+ config.references = [{ id: "hero", kind: "image", name: "Hero", description: "", intendedUse: [], relativePath: "references/hero.png", createdAt: config.createdAt }];
+ const scenes = songScenes(video, config); config.generationJobs = scenes; video.sceneIds = scenes.map((job) => job.id);
+ expect(scenes.length).toBe(21); expect(scenes.reduce((sum, job) => sum + sceneDurationSeconds(job), 0)).toBeCloseTo(201);
+ for (const job of scenes) expect(sendBlocker(job, config.references, config)).toBeNull();
+ const last = scenes.at(-1)!; const request = sceneGenerationRequest(last, config, "D:/Project", 8);
+ expect(request.referencePaths).toEqual(["D:/Project/references/hero.png"]);
+ expect(request.referenceAudios![0].startSeconds + request.referenceAudios![0].durationSeconds).toBeCloseTo(201);
+ const single = { ...scenes[0], sceneType: "single-shot" as const, endingPrompt: "The dancer bows." };
+ expect(sceneGenerationRequest(single, config, "D:/Project", 8).singleShot?.endingPrompt).toContain("<Audio 1>");
+ expect(songScenes(video, config)).toEqual([]);
+});
+
+it("respects reference trims, detects missing and exhausted sources, and requires a visual anchor for music", () => {
+ const config = project(), job = scene("first"); config.generationJobs = [job];
+ config.references = [{ id: "trim", kind: "audio", name: "Trimmed song", description: "", intendedUse: [], sourcePath: "D:/song.wav", audio: { startSeconds: 45, durationSeconds: 10 }, createdAt: config.createdAt }];
+ const video = { ...newVideo(null), sceneIds: [job.id], musicSource: { kind: "reference" as const, id: "trim" } }; config.videos = [video];
+ expect(resolveVideoSource(video.musicSource, config, "audio")?.startSeconds).toBe(45);
+ expect(sceneGenerationRequest(job, config, "D:/Project", 8).referenceAudios?.[0].startSeconds).toBe(45);
+ expect(sendBlocker(job, config.references, config)).toMatch(/image or video/);
+ expect(videoGuidedScene({ ...job, durationSeconds: 11 }, config).blocker).toMatch(/ends before/);
+ expect(videoGuidedScene({ ...job, durationSeconds: 1 }, config).blocker).toMatch(/at least 2/);
+ config.references = []; expect(sendBlocker(job, [], config)).toMatch(/unavailable/);
+});
+
+it("plays scenes in video order, skips drafts and excludes joined continuation prefixes", () => {
+ const config = project(); config.generationJobs = [scene("first"), scene("draft"), scene("last", 8)];
+ const video = { ...newVideo(null), sceneIds: ["last", "draft", "first"] };
+ config.assets = [asset(generationAssetId("first"), "generated", 10250), { ...asset(generationAssetId("last"), "generated", 19000), sceneSegments: [{ sceneId: "last", startFrame: 240, frameCount: 216, latentRelativePath: "latents/last.safetensors" }] }];
+ expect(videoPlaylist(video, config).map(({ sceneId, start, end }) => ({ sceneId, start, end }))).toEqual([{ sceneId: "last", start: 10, end: 18 }, { sceneId: "first", start: 0, end: 10 }]);
+});
diff --git a/src/lib/videoLibrary.ts b/src/lib/videoLibrary.ts
new file mode 100644
index 0000000..531784d
--- /dev/null
+++ b/src/lib/videoLibrary.ts
@@ -0,0 +1,84 @@
+import { createDraftGenerationJob, sceneDurationSeconds, sceneShots, type GenerationJob, type ProjectConfig, type ProjectReference, type ProjectVideo, type VideoSource } from "./project";
+import { sceneMediaDurationMs, sceneMediaStartSeconds } from "./continuationMedia";
+
+/** Older projects and scenes added by external tools remain visible. */
+export function videoLibrary(config: ProjectConfig): ProjectVideo[] {
+ const videos = config.videos ?? [];
+ const assigned = new Set(videos.flatMap((video) => video.sceneIds));
+ const unassigned = config.generationJobs.filter((scene) => !assigned.has(scene.id)).map((scene) => scene.id);
+ if (!unassigned.length) return videos;
+ const fallback = videos.find((video) => video.id === "video-default");
+ return fallback ? videos.map((video) => video === fallback ? { ...video, sceneIds: [...video.sceneIds, ...unassigned] } : video)
+ : [...videos, { id: "video-default", title: config.name || "Video 1", sceneIds: unassigned, prompt: "", createdAt: config.createdAt }];
+}
+export function videoScenes(video: ProjectVideo, config: ProjectConfig): GenerationJob[] {
+ return video.sceneIds.flatMap((id) => { const scene = config.generationJobs.find((job) => job.id === id); return scene ? [scene] : []; });
+}
+export function newVideo(folderId: string | null): ProjectVideo {
+ return { id: `video-${crypto.randomUUID()}`, title: "New video", folderId, sceneIds: [], prompt: "", createdAt: new Date().toISOString() };
+}
+export function resolveVideoSource(source: VideoSource | null | undefined, config: ProjectConfig, kind: "audio" | "video") {
+ if (!source) return undefined;
+ if (source.kind === "asset") {
+ const asset = config.assets.find((asset) => asset.id === source.id && (asset.kind === kind || kind === "video" && asset.kind === "generated") && (asset.relativePath || asset.sourcePath));
+ return asset ? { name: asset.name, relativePath: asset.relativePath, sourcePath: asset.sourcePath, startSeconds: kind === "video" ? sceneMediaStartSeconds(asset) : 0,
+ durationSeconds: (kind === "video" ? sceneMediaDurationMs(asset) : asset.durationMs) != null ? (kind === "video" ? sceneMediaDurationMs(asset)! : asset.durationMs!) / 1000 : undefined } : undefined;
+ }
+ const reference = config.references.find((reference) => reference.id === source.id && reference.kind === kind && (reference.relativePath || reference.sourcePath));
+ const range = kind === "audio" ? reference?.audio : reference?.video;
+ return reference ? { name: reference.name, relativePath: reference.relativePath, sourcePath: reference.sourcePath,
+ startSeconds: range?.startSeconds ?? 0, durationSeconds: range?.durationSeconds ?? undefined } : undefined;
+}
+export function videoSceneOffset(video: ProjectVideo, job: GenerationJob, config: ProjectConfig): number {
+ const scenes = videoScenes(video, config), index = scenes.findIndex((scene) => scene.id === job.id);
+ return scenes.slice(0, Math.max(0, index)).reduce((seconds, scene) => seconds + sceneDurationSeconds(scene), 0);
+}
+
+/** Generation-only copies: never write the injected prompt/reference bindings to the project. */
+export function videoGuidedScene(job: GenerationJob, config: ProjectConfig): { job: GenerationJob; config: ProjectConfig; blocker: string | null } {
+ const video = videoLibrary(config).find((video) => video.sceneIds.includes(job.id));
+ if (!video || !video.prompt.trim() && !video.musicSource && !video.videoSource) return { job, config, blocker: null };
+ const offset = videoSceneOffset(video, job, config), duration = sceneDurationSeconds(job);
+ const references: ProjectReference[] = [];
+ const guidance: string[] = video.prompt.trim() ? [video.prompt.trim()] : [];
+ let blocker: string | null = null;
+ if ((video.musicSource || video.videoSource) && ["animate", "character-replace", "extend", "bridge"].includes(job.sceneType ?? "")) {
+ blocker = "Video-level references require a Prompt, Single Shot, Continue, First & Last Frame, Backdrop, Pose or Long Shot scene.";
+ }
+ for (const kind of ["audio", "video"] as const) {
+ const chosen = kind === "audio" ? video.musicSource : video.videoSource;
+ if (!chosen) continue;
+ const source = resolveVideoSource(chosen, config, kind);
+ if (!source) { blocker = `The video's ${kind === "audio" ? "music" : "video"} reference is unavailable. Select it again in Video settings.`; continue; }
+ if (duration < 2) blocker = "Scenes using video-level references must last at least 2 seconds.";
+ if (source.durationSeconds !== undefined && offset + duration > source.durationSeconds + 0.001) blocker = `The ${kind} reference ends before ${job.title}. Shorten the video or choose a longer reference.`;
+ const id = `video-guide-${video.id}-${kind}`;
+ references.push({ id, kind, name: source.name, description: "", intendedUse: kind === "audio" ? ["audio"] : [], relativePath: source.relativePath, sourcePath: source.sourcePath, createdAt: video.createdAt,
+ ...(kind === "audio" ? { audio: { startSeconds: source.startSeconds + offset, durationSeconds: duration } }
+ : { video: { startSeconds: source.startSeconds + offset, durationSeconds: duration, includeAudio: !video.musicSource } }) });
+ guidance.push(kind === "audio" ? `Create visuals synchronized to the rhythm, mood and progression of @[ref:${id}]. This is the music segment for this scene.`
+ : `Use @[ref:${id}] as the visual reference for this scene's time interval.`);
+ }
+ const text = guidance.join("\n");
+ const shots = sceneShots(job).map((shot, index) => index === 0 ? { ...shot, action: [shot.action, text].filter(Boolean).join("\n") } : shot);
+ const guided = { ...job, shots, ...(job.sceneType === "single-shot" && job.endingPrompt?.trim() ? { endingPrompt: `${job.endingPrompt}\n${text}` } : {}) };
+ return { job: guided, config: { ...config, references: [...config.references, ...references] }, blocker };
+}
+
+/** Assign new scenes to the open Video and retain scene ordering after edits. */
+export function mergeVideoScenes(current: ProjectConfig, videoId: string, edited: ProjectConfig): ProjectConfig {
+ const videos = videoLibrary(current), video = videos.find((video) => video.id === videoId);
+ if (!video) return current;
+ const ids = new Set(video.sceneIds);
+ const before = new Set(current.generationJobs.map((job) => job.id)), after = new Set(edited.generationJobs.map((job) => job.id));
+ return { ...edited, videos: videos.map((item) => item.id === videoId
+ ? { ...item, sceneIds: edited.generationJobs.filter((job) => ids.has(job.id) || !before.has(job.id)).map((job) => job.id) }
+ : { ...item, sceneIds: item.sceneIds.filter((id) => after.has(id)) }) };
+}
+
+export function songScenes(video: ProjectVideo, config: ProjectConfig): GenerationJob[] {
+ const seconds = resolveVideoSource(video.musicSource, config, "audio")?.durationSeconds;
+ if (!seconds || seconds < 2 || video.sceneIds.length) return [];
+ const count = Math.ceil(seconds / 10), duration = seconds / count;
+ return Array.from({ length: count }, (_, index) => createDraftGenerationJob("", { sceneType: "prompt", title: `Scene ${index + 1}`, durationSeconds: duration }));
+}
diff --git a/src/styles/index.css b/src/styles/index.css
index 446a12e..32b1e7d 100644
--- a/src/styles/index.css
+++ b/src/styles/index.css
@@ -25,6 +25,7 @@
@import "./agent-dock.css";
@import "./generator.css";
@import "./music.css";
+@import "./video-library.css";
@import "./timeline.css";
@import "./references.css";
@import "./export.css";
diff --git a/src/styles/video-library.css b/src/styles/video-library.css
new file mode 100644
index 0000000..aa3d18a
--- /dev/null
+++ b/src/styles/video-library.css
@@ -0,0 +1,25 @@
+.video-library-view { display: grid; grid-template-rows: auto minmax(0, 1fr); height: 100%; min-height: 0; }
+.video-library-body { display: grid; grid-template-columns: minmax(180px, 1fr) auto minmax(280px, var(--pane-video-inspector)); min-height: 0; }
+.video-library-stage { position: relative; min-width: 0; min-height: 0; overflow: hidden; }
+.video-library-stage > .music-library { height: 100%; box-sizing: border-box; }
+.video-library-body > .ui-splitter { grid-column: 2; }
+.video-library-body > .music-inspector { grid-row: 1; }
+.video-scenes { display: flex; flex-direction: column; height: 100%; min-height: 0; }
+.video-scenes > .generator-view { flex: 1; min-height: 0; height: auto; }
+.video-scenes__path { flex: none; }
+.video-scenes__path strong { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.video-scenes__path span, .video-reference-help { color: var(--text-secondary); font-size: var(--text-xs); }
+.video-reference-range { display: block; margin-block: var(--space-1); }
+.video-card { overflow: hidden; }
+.video-card > .music-song__select { height: auto; padding-bottom: 44px; }
+.video-card .music-song__art { aspect-ratio: 16 / 9; }
+.video-card__open { position: absolute; bottom: var(--space-2); right: var(--space-2); }
+.video-card__controls { position: absolute; top: 30%; left: 50%; transform: translate(-50%, -50%); display: flex; gap: var(--space-2); }
+.video-card__controls button { display: grid; place-items: center; width: 32px; height: 32px; border: 1px solid #ffffff80; border-radius: 50%; background: #15191fe6; color: #fff; box-shadow: 0 1px 6px #0008; }
+.video-card__controls button:hover { background: #363d48; }
+.video-card__player { position: absolute; inset: 0; background: #000; z-index: 2; }
+.video-card__player video { width: 100%; height: 100%; object-fit: contain; }
+.video-card__position { position: absolute; left: var(--space-2); top: var(--space-2); color: #fff; background: #000b; padding: 2px 5px; font-size: var(--text-xs); pointer-events: none; max-width: 70%; }
+.music-songs--list .video-card > .music-song__select { padding-bottom: var(--space-3); padding-right: 190px; min-height: 70px; }
+.music-songs--list .video-card__controls { top: 50%; left: auto; right: 100px; transform: translateY(-50%); }
+.music-songs--list .video-card__open { bottom: 50%; transform: translateY(50%); }