Skip to content

Open source CapCut alternative.

TypeScript

feat(site): reference other projects in chat [rebuild]

David Lef840729ae66a
diff --git a/site/src/clients/chatgpt/server/catalog.ts b/site/src/clients/chatgpt/server/catalog.ts
index 81b882ca..74f11e80 100644
--- a/site/src/clients/chatgpt/server/catalog.ts
+++ b/site/src/clients/chatgpt/server/catalog.ts
@@ -75,6 +75,7 @@ export const READ_COMMANDS = new Set([
"notes_list",
"read_note",
"read_project",
+ "list_projects",
"check_facts",
]);
diff --git a/site/src/clients/chatgpt/server/guidance.ts b/site/src/clients/chatgpt/server/guidance.ts
index 18f5b2b7..e65fa52f 100644
--- a/site/src/clients/chatgpt/server/guidance.ts
+++ b/site/src/clients/chatgpt/server/guidance.ts
@@ -81,7 +81,7 @@ read_folder browses project or Library folders and accessible Donkey folder link
delete_asset removes project media and its timeline uses. Library deletions can affect the user's synced shelf. These deletions require the user's explicit request; undo covers timeline edits and does not restore deleted source files.`,
"replicating-a-project": `# Reusing a project
-read_project inspects a source project or share link. Use its returned ids and document to understand the edit; use watch_video and listen_audio for media you need to judge. Ask for a reference the user can share when access fails.
+read_project inspects a source project by id or link, or a share link; list_projects finds the id of a project named by its title. Use its returned ids and document to understand the edit; use watch_video and listen_audio for media you need to judge. Ask for a reference the user can share when access fails.
replicate_project copies the requested edit or items and can map source roles to assets already in this project. Match by content and measured timing. Unmapped media copies from the reference. copy_project_media brings across media for a new arrangement.
Read the resulting adjustments and placement: trims can clamp to shorter replacement sources, and clips append to the current cut. Inspect the result, revise the requested text and timing, and check frames and audio. Apply background removal to replacement footage when requested. save_template and template_add support reusable arrangements.`,
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.ts
index a81dd87b..65eedf4f 100644
--- a/site/src/cut/components/AiPanel.tools.ts
+++ b/site/src/cut/components/AiPanel.tools.ts
@@ -11,6 +11,7 @@ import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
import { TIMELINE_IDS, WATCH_DETAIL_NOTES, WATCH_DETAILS } from "@/cut/lib/types";
import { FACT_CLAIM_MAX_CHARS, FACT_CLAIMS_MAX } from "@/lib/inference/factCheck";
import { URL_BATCH_MAX } from "@/cut/lib/urlBatch";
+import { PROJECT_PAGE_DEFAULT, PROJECT_PAGE_MAX } from "@/cut/lib/projectLink";
import { FIND_MEDIA_DEFAULT, FIND_MEDIA_MAX, FIND_MEDIA_SNAPSHOT_CAP } from "@/cut/lib/findMedia";
import { COMPARE_MODES, MAX_COMPARE, SWEEP_TARGET } from "@/cut/lib/frameDiff";
import { EASE_IDS } from "@donkeycut/effects-kit";
@@ -410,10 +411,19 @@ export const AI_PANEL_TOOLS = [
["claims"]
),
},
+ {
+ name: "list_projects",
+ description:
+ "List the user's own Donkey Cut projects, newest edit first: each project's id, its current name, where it lives and when it was last edited, with the open project marked. Use it when the user or a note names another project by its title (\"copy the styles of the project named Graphics Ref\", \"the intro from my launch video\"): pick the project the name means, then pass its id to read_project. Names change; the id is what every project tool takes. A project attachment already carries its id, so it needs no lookup. `unreachable` names shelves this session could not read; a project the user names may live there, so say so.",
+ inputSchema: obj({
+ offset: num("Where the page starts; pass the previous page's next"),
+ limit: num(`Projects per page, 1..${PROJECT_PAGE_MAX} (default ${PROJECT_PAGE_DEFAULT})`),
+ }),
+ },
{
name: "read_project",
description:
- "Read another Donkey Cut project as a reference: its whole edit in the same shape as <editor_state> — every clip with its trims, speed, framing, grade, look, mask, keyframes and animations; the transition bars; titles, shapes and stickers with their keyframes; the soundtrack; the caption look and cues; aspect and background — plus each source's observed notes, transcript and asset id. Takes a project link (…/app/p/<id>), a share link (…/s/<token>), or a project id. The document IS the edit: never watch_video a reference to learn what its document already states; watch a reference source (project_link) only to see footage its notes leave dark. Every item carries its id, so replicate_project can bring the whole edit or just the items the user wants — a title, one clip's treatment, the music — and copy_project_media can bring files over on their own. Nothing in the open project changes.",
+ "Read another Donkey Cut project as a reference: its whole edit in the same shape as <editor_state> — every clip with its trims, speed, framing, grade, look, mask, keyframes and animations; the transition bars; titles, shapes and stickers with their keyframes; the soundtrack; the caption look and cues; aspect and background — plus each source's observed notes, transcript and asset id. Takes a project link (…/app/p/<id>), a share link (…/s/<token>), or a project id (a project the user names by title: its id from list_projects). The document IS the edit: never watch_video a reference to learn what its document already states; watch a reference source (project_link) only to see footage its notes leave dark. Every item carries its id, so replicate_project can bring the whole edit or just the items the user wants — a title, one clip's treatment, the music — and copy_project_media can bring files over on their own. Nothing in the open project changes.",
inputSchema: obj({ link: str("Project link, share link, or project id") }, ["link"]),
},
{
diff --git a/site/src/cut/components/AssetRefs.tsx b/site/src/cut/components/AssetRefs.tsx
index 9581c571..9c1011b6 100644
--- a/site/src/cut/components/AssetRefs.tsx
+++ b/site/src/cut/components/AssetRefs.tsx
@@ -2,7 +2,7 @@
import { useEffect, useLayoutEffect, useMemo, useRef, useState, type ReactNode, type RefObject } from "react";
import Link from "next/link";
-import { CUT_APP_BASE, homeHref } from "@/cut/lib/nav";
+import { CUT_APP_BASE, homeHref, projectHref } from "@/cut/lib/nav";
import { Check, Clock, Copy, FileText, X } from "lucide-react";
import {
highlightMentions,
@@ -16,7 +16,7 @@ import {
type AssetRef,
} from "@/cut/lib/assetRef";
import { MEDIA_CORS } from "@/cut/lib/mediaCors";
-import { useRefFor, useRefCandidates } from "@/cut/lib/assetRef";
+import { isCollectionRef, useRefFor, useRefCandidates } from "@/cut/lib/assetRef";
import { useInView } from "@/cut/hooks/useInView";
import { revealRef } from "@/cut/lib/refReveal";
import { formatTime } from "@/cut/lib/time";
@@ -45,8 +45,12 @@ import { ValueSlider } from "@/cut/components/ValueSlider";
import { cn } from "@/lib/utils";
function CollectionLink({ item, children, className }: { item: AssetRef; children: ReactNode; className?: string }) {
- return item.scope === "note"
- ? <Link href={`${homeHref(CUT_APP_BASE, "notes")}?note=${encodeURIComponent(item.id)}`} target="_blank" title={item.name} className={className}>{children}</Link>
+ // A note or another project opens in its own tab; a folder reveals in place.
+ const href = item.scope === "note"
+ ? `${homeHref(CUT_APP_BASE, "notes")}?note=${encodeURIComponent(item.id)}`
+ : item.scope === "projectDoc" ? projectHref(CUT_APP_BASE, item.id, "projects") : null;
+ return href
+ ? <Link href={href} target="_blank" title={item.name} className={className}>{children}</Link>
: <button type="button" onClick={() => revealRef(item)} title={item.name} className={className}>{children}</button>;
}
@@ -322,7 +326,7 @@ export function RefTokenChip({
const [peek, setPeek] = useState(false);
// Entity refs have no media to peek at and no card to jump to — the pill
// itself, icon plus name, is the whole story.
- if (item.scope === "folder" || item.scope === "note") {
+ if (isCollectionRef(item)) {
return <CollectionLink item={item} className="inline-flex items-center gap-1 rounded-md bg-blue-500/15 px-1 text-blue-500">{entityGlyph(item, "size-3")} {item.name}</CollectionLink>;
}
if (item.scope === "entity") {
@@ -514,7 +518,7 @@ export function RefChips({
// become candidates.
const handle =
candidates.find((c) => c.scope === r.scope && c.id === r.id)?.handle ?? r.handle;
- if (r.scope === "folder" || r.scope === "note") return (
+ if (isCollectionRef(r)) return (
<span key={r.id} className="inline-flex max-w-full items-center gap-1 rounded-md border border-border bg-muted px-2 py-1 text-xs">
{entityGlyph(r, "size-3 shrink-0")}
<CollectionLink item={r} className="truncate">{r.name}</CollectionLink>
@@ -1160,14 +1164,14 @@ function MentionPill({
)
);
const rich =
- (item.scope === "entity" || item.scope === "folder" || item.scope === "note") &&
+ (item.scope === "entity" || isCollectionRef(item)) &&
!editing &&
!wrapped &&
clippedFor !== text &&
!!(glyph || item.preview);
// Entity pills act as one object: a click selects the whole token, so the
// caret never lands inside and typing replaces it in one stroke.
- const atomic = (item.scope === "entity" || item.scope === "folder" || item.scope === "note") && !pinnable;
+ const atomic = (item.scope === "entity" || isCollectionRef(item)) && !pinnable;
return (
<span
ref={spanRef}
@@ -1552,7 +1556,7 @@ export function MentionTextarea({
};
const pick = (ref: AssetRef) => {
- if (ref.scope === "folder" || ref.scope === "note") onUpsertRef?.(ref);
+ if (isCollectionRef(ref)) onUpsertRef?.(ref);
writeToken(refToken(ref) + " ");
};
diff --git a/site/src/cut/components/entityIcons.ts b/site/src/cut/components/entityIcons.ts
index 6afc3987..7aeb6db7 100644
--- a/site/src/cut/components/entityIcons.ts
+++ b/site/src/cut/components/entityIcons.ts
@@ -21,6 +21,7 @@ import {
SplitSquareVertical,
Folder,
StickyNote,
+ Clapperboard,
Heart,
Hexagon,
Brush,
@@ -103,6 +104,7 @@ export function entityGlyph(ref: AssetRef, className: string): ReactNode {
export function entityIcon(ref: AssetRef): LucideIcon | null {
if (ref.scope === "note") return StickyNote;
if (ref.scope === "folder") return Folder;
+ if (ref.scope === "projectDoc") return Clapperboard;
switch (ref.entityKind) {
case "title":
return Type;
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index ccccdb8a..8ed0f379 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -231,8 +231,8 @@ import { STOCK_IMAGES } from "./stockManifest";
import { STOCK_VIDEOS } from "./stockVideoManifest";
import { STOCK_SFX } from "./stockSfxManifest";
import { storedMediaUrl } from "./mediaSync";
-import { parseProjectLink } from "./projectLink";
-import { landReferenceAssets as landReferenceAssetsInto, openReference, referenceMediaUrl, type LandedAsset, type ReferenceProject } from "./projectReference";
+import { parseProjectLink, PROJECT_PAGE_DEFAULT, PROJECT_PAGE_MAX } from "./projectLink";
+import { landReferenceAssets as landReferenceAssetsInto, listOwnProjects, openReference, referenceMediaUrl, type LandedAsset, type ReferenceProject } from "./projectReference";
import { clampLayersToAssets, mediaTypeFits, templateFromDoc } from "./projectTemplate";
import { isSoundPresetTemplate, listSoundPresets, saveSoundPreset } from "./soundPresets";
import { isGradePresetTemplate, listSavedGrades, saveGradePreset } from "./gradePresets";
@@ -5148,6 +5148,26 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
return factCheckToolOutput(results);
},
+ list_projects: async (s, input) => {
+ const { projects, unreachable } = await listOwnProjects();
+ const offset = typeof input.offset === "number" && Number.isSafeInteger(input.offset) && input.offset > 0 ? input.offset : 0;
+ const limit = typeof input.limit === "number" && input.limit >= 1 ? Math.min(Math.floor(input.limit), PROJECT_PAGE_MAX) : PROJECT_PAGE_DEFAULT;
+ const page = projects.slice(offset, offset + limit);
+ return {
+ projects: page.map((p) => ({
+ id: p.id,
+ name: p.name,
+ residency: p.residency,
+ edited: new Date(p.updatedAt).toISOString(),
+ ...(p.id === s.projectId ? { open: true } : {}),
+ })),
+ total: projects.length,
+ ...(offset + page.length < projects.length ? { next: offset + page.length } : {}),
+ // A shelf that did not answer may hold the project the user means.
+ ...(unreachable.length ? { unreachable } : {}),
+ };
+ },
+
read_project: async (_s, input) => {
const ref = await openReferenceLink(input.link);
const reference = describeDoc(ref.doc, { fullCues: true, chatId: chatOwner() });
diff --git a/site/src/cut/lib/assetRef.ts b/site/src/cut/lib/assetRef.ts
index 521f9300..0bf2a672 100644
--- a/site/src/cut/lib/assetRef.ts
+++ b/site/src/cut/lib/assetRef.ts
@@ -8,7 +8,8 @@ import { isLinkedAssetType } from "./types";
import type React from "react";
import { libraryRouteUrl, type LibraryAsset, type LibraryData } from "./library";
import { libraryLutId, LUT_MARK_ICON } from "./linkedLibrary/luts";
-import { useLibrary, useNotes } from "./queries";
+import { useLibrary, useNotes, useOwnProjects } from "./queries";
+import { parseProjectLink, projectDocRef } from "./projectLink";
import { stockAspectDims, stockTitle, type StockImage, type StockMusic, type StockSfx, type StockVideo } from "./stock";
import { STOCK_IMAGES } from "./stockManifest";
import { STOCK_MUSIC } from "./stockMusicManifest";
@@ -69,8 +70,9 @@ import type {
* (`overlay:<id>`, `key:clip:<id>:pose:<index>`, …) and its `url` is a
* data: text payload describing the entity — the parent's id included — so a
* mention lands in chat as grounded text the tools can act on.
- * "folder" carries a typed collection reference for read_folder. */
-export type AssetRefScope = "project" | "library" | "stock" | "file" | "clip" | "entity" | "folder" | "note";
+ * "folder" carries a typed collection reference for read_folder.
+ * "projectDoc" names another of the user's projects by id, for read_project. */
+export type AssetRefScope = "project" | "library" | "stock" | "file" | "clip" | "entity" | "folder" | "note" | "projectDoc";
export type AssetRefKind = "video" | "audio" | "image" | "text";
export interface AssetRef {
@@ -250,6 +252,11 @@ export async function refFromTextFile(file: File): Promise<AssetRef> {
};
}
+/** A ref that names a collection or a document the assistant reads by id —
+ * a folder, a note, another project — and reads as a pill with no media. */
+export const isCollectionRef = (ref: Pick<AssetRef, "scope">) =>
+ ref.scope === "folder" || ref.scope === "note" || ref.scope === "projectDoc";
+
export const sameRef = (a: AssetRef, b: AssetRef) => a.scope === b.scope && a.id === b.id;
export const addRefOnce = (list: AssetRef[], ref: AssetRef): AssetRef[] =>
@@ -293,6 +300,7 @@ export function normalizeRef(v: unknown): AssetRef | null {
const o = v as Partial<AssetRef> & { type?: AssetRefKind };
if (!o.id || !o.name || !o.url) return null;
if (o.scope === "note") return noteIdSchema.safeParse(o.id).success ? { ...noteRef({ id: o.id, title: "" }), name: o.name } : null;
+ if (o.scope === "projectDoc") return parseProjectLink(o.id)?.kind === "project" ? { ...projectDocRef({ id: o.id, name: "" }), name: o.name } : null;
const folder = o.scope === "folder" ? folderReferenceSchema.safeParse(o.folder) : null;
if (folder && !folder.success) return null;
if (folder?.success) return folderRef(folder.data, o.name);
@@ -945,12 +953,14 @@ export function useRefCandidates(enabled = true, includeCollections = false): As
const lib = useLibrary({ enabled }).data ?? EMPTY_LIBRARY;
const notes = useNotes(enabled && includeCollections).data;
+ const projects = useOwnProjects(enabled && includeCollections);
const folderRefs = useMemo(() => {
if (!includeCollections) return [];
const refs = [
...(projectId ? (folders ?? []).map((f) => projectFolderRef(f, projectId)) : []),
...libraryFolderRefs(lib.folders),
...(notes?.notes ?? []).map(noteRef),
+ ...projects.filter((p) => p.id !== projectId).map(projectDocRef),
...lib.assets
.filter((a) => LIBRARY_MENTION[a.type].menu === "chat")
.map(refFromLibrary)
@@ -960,7 +970,7 @@ export function useRefCandidates(enabled = true, includeCollections = false): As
for (const ref of refs) counts.set(ref.name.toLowerCase(), (counts.get(ref.name.toLowerCase()) ?? 0) + 1);
return refs.map((ref) => (counts.get(ref.name.toLowerCase()) ?? 0) > 1
? { ...ref, name: `${ref.name} [${ref.id}]` } : ref);
- }, [includeCollections, projectId, folders, lib.folders, lib.assets, notes]);
+ }, [includeCollections, projectId, folders, lib.folders, lib.assets, notes, projects]);
return useMemo(() => {
if (!assets || !clips || !audioClips || !overlays || !transitions || !subtitles || !projectTemplates) return [];
@@ -1020,7 +1030,7 @@ const NBSP = " ";
export const refToken = (ref: AssetRef) =>
ref.handle
? `@${ref.handle}`
- : ref.scope === "entity" || ref.scope === "folder" || ref.scope === "note"
+ : ref.scope === "entity" || isCollectionRef(ref)
? `@"${ref.name.replace(/"/g, "").replace(/ /g, NBSP)}"`
: mentionToken(ref.name);
diff --git a/site/src/cut/lib/cutClipboard.test.ts b/site/src/cut/lib/cutClipboard.test.ts
index 12899779..15979e7b 100644
--- a/site/src/cut/lib/cutClipboard.test.ts
+++ b/site/src/cut/lib/cutClipboard.test.ts
@@ -26,6 +26,7 @@ const rich = richDoc();
const docs = new Map<string, ProjectDoc>([["rich", rich]]);
const copies: string[] = [];
setHostDocStore({
+ listDocs: async () => [],
readDoc: async (id) => docs.get(id) ?? null,
copyMedia: async (_src, file) => {
copies.push(file);
@@ -192,6 +193,7 @@ describe("pasting into another project", () => {
test("a project that closes while the media lands takes none of it", async () => {
setHostDocStore({
+ listDocs: async () => [],
readDoc: async (id) => docs.get(id) ?? null,
copyMedia: async (_src, file) => {
// The user switched projects while the bytes were on their way.
@@ -207,6 +209,7 @@ describe("pasting into another project", () => {
expect(useEditor.getState().clips).toHaveLength(0);
} finally {
setHostDocStore({
+ listDocs: async () => [],
readDoc: async (id) => docs.get(id) ?? null,
copyMedia: async (_src, file) => {
copies.push(file);
diff --git a/site/src/cut/lib/pi/mutationLedger.ts b/site/src/cut/lib/pi/mutationLedger.ts
index dcef1370..b7d0ce83 100644
--- a/site/src/cut/lib/pi/mutationLedger.ts
+++ b/site/src/cut/lib/pi/mutationLedger.ts
@@ -27,6 +27,7 @@ const READ_ONLY = new Set([
"measure_level",
"read_color_stats",
"read_project",
+ "list_projects",
"library_list",
"library_list_folder",
"stock_search",
diff --git a/site/src/cut/lib/projectLink.ts b/site/src/cut/lib/projectLink.ts
index d1e55c13..d1bd6033 100644
--- a/site/src/cut/lib/projectLink.ts
+++ b/site/src/cut/lib/projectLink.ts
@@ -4,8 +4,13 @@
// the link opens — the project's home, the share's grant — is the reader's
// to resolve.
+import type { AssetRef } from "./assetRef";
import { normalizeLink } from "./link";
+/** How many projects one list_projects page carries by default, and at most. */
+export const PROJECT_PAGE_DEFAULT = 100;
+export const PROJECT_PAGE_MAX = 500;
+
export type ProjectLink = { kind: "project"; id: string } | { kind: "share"; token: string };
const ID_RE = /^[A-Za-z0-9_-]+$/;
@@ -33,3 +38,16 @@ export function parseProjectLink(value: string): ProjectLink | null {
if (share && ID_RE.test(share[1])) return { kind: "share", token: share[1] };
return null;
}
+
+/** Another of the user's projects as a chat attachment. It carries the id, so
+ * the reference holds through a rename; read_project returns the current
+ * title. */
+export function projectDocRef(project: { id: string; name: string }): AssetRef {
+ return {
+ scope: "projectDoc",
+ id: project.id,
+ name: `Projects / ${project.name || "Untitled"}`.replace(/"/g, "”").replace(/\s+/g, " "),
+ kind: "text",
+ url: `data:text/plain;charset=utf-8,${encodeURIComponent(JSON.stringify({ tool: "read_project", link: project.id }))}`,
+ };
+}
diff --git a/site/src/cut/lib/projectList.test.tsx b/site/src/cut/lib/projectList.test.tsx
new file mode 100644
index 00000000..29786967
--- /dev/null
+++ b/site/src/cut/lib/projectList.test.tsx
@@ -0,0 +1,106 @@
+import { afterAll, afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { renderToStaticMarkup } from "react-dom/server";
+import { UnmentionedRefPills } from "@/cut/components/AssetRefs";
+import { attachedAssetsBlock } from "@/cut/server/ai/catalog";
+import { runAiTool } from "./aiTools";
+import { normalizeRef, refToken } from "./assetRef";
+import { bindCutMode, releaseCutMode } from "./backend";
+import { projectDocRef } from "./projectLink";
+import { listOwnProjects, setHostDocStore } from "./projectReference";
+import { useEditor } from "./store";
+import type { ProjectSummary } from "./types";
+
+// Another project reaches the assistant two ways: an @ attachment that
+// carries its id, or a title the model looks up with list_projects.
+
+const summary = (id: string, name: string, updatedAt: number): ProjectSummary => ({
+ id,
+ name,
+ createdAt: 0,
+ updatedAt,
+ duration: 0,
+ clipCount: 0,
+ assetCount: 0,
+});
+
+const originalFetch = globalThis.fetch;
+afterEach(() => {
+ globalThis.fetch = originalFetch;
+});
+afterAll(() => setHostDocStore(null));
+
+describe("list_projects", () => {
+ beforeEach(() => {
+ setHostDocStore({
+ listDocs: async () => [summary("open", "Launch", 3), summary("gfx", "Graphics Ref", 2), summary("old", "Old cut", 1)],
+ readDoc: async () => null,
+ copyMedia: async (_src, file) => file,
+ });
+ useEditor.setState({ projectId: "open" });
+ });
+
+ test("names every project by its current title and marks the open one", async () => {
+ const out = (await runAiTool("list_projects", {})) as { projects: { id: string; name: string; open?: boolean }[]; total: number; unreachable?: string[] };
+ expect(out.total).toBe(3);
+ // The engine reads only its own disk, and says which shelves it skipped.
+ expect(out.unreachable).toEqual(["cloud", "browser"]);
+ expect(out.projects.map((p) => [p.id, p.name, p.open ?? false])).toEqual([
+ ["open", "Launch", true],
+ ["gfx", "Graphics Ref", false],
+ ["old", "Old cut", false],
+ ]);
+ });
+
+ test("pages through a long shelf", async () => {
+ const out = (await runAiTool("list_projects", { offset: 1, limit: 1 })) as { projects: { id: string }[]; next?: number };
+ expect(out.projects.map((p) => p.id)).toEqual(["gfx"]);
+ expect(out.next).toBe(2);
+ });
+});
+
+describe("the shelves a page lists", () => {
+ beforeEach(() => {
+ setHostDocStore(null);
+ bindCutMode("cloud");
+ });
+ afterEach(() => releaseCutMode());
+
+ test("the cloud shelf lists newest first", async () => {
+ globalThis.fetch = (async () => Response.json([summary("a", "A", 1), summary("b", "B", 5)])) as unknown as typeof fetch;
+ const { projects, unreachable } = await listOwnProjects();
+ expect(projects.map((p) => [p.id, p.residency])).toEqual([["b", "cloud"], ["a", "cloud"]]);
+ expect(unreachable).toEqual([]);
+ });
+
+ test("a shelf that does not answer is named", async () => {
+ globalThis.fetch = (async () => new Response("", { status: 503 })) as unknown as typeof fetch;
+ expect(await listOwnProjects()).toEqual({ projects: [], unreachable: ["cloud"] });
+ });
+});
+
+describe("a project attachment", () => {
+ const ref = projectDocRef({ id: "gfx", name: "Graphics Ref" });
+
+ test("carries the id read_project takes", () => {
+ const payload = JSON.parse(decodeURIComponent(ref.url.slice(ref.url.indexOf(",") + 1)));
+ expect(payload).toEqual({ tool: "read_project", link: "gfx" });
+ // Composer tokens carry no-break spaces so the pill wraps whole.
+ expect(refToken(ref).replace(/\u00a0/g, " ")).toBe(`@"Projects / Graphics Ref"`);
+ });
+
+ test("comes back from a stored thread", () => {
+ expect(normalizeRef({ ...ref })).toMatchObject({ scope: "projectDoc", id: "gfx", name: "Projects / Graphics Ref" });
+ expect(normalizeRef({ ...ref, id: "not an id" })).toBeNull();
+ });
+
+ test("tells the model to read it by id", () => {
+ expect(attachedAssetsBlock([ref])).toContain("Call read_project with that id");
+ });
+
+ test("reads as a pill that opens the project in a new tab", () => {
+ const html = renderToStaticMarkup(<UnmentionedRefPills text="copy its styles" refs={[ref]} />);
+ expect(html).toContain("Projects / Graphics Ref");
+ expect(html).toContain("/p/gfx");
+ expect(html).toContain("target=\"_blank\"");
+ });
+});
diff --git a/site/src/cut/lib/projectReference.test.ts b/site/src/cut/lib/projectReference.test.ts
index 50cfa0de..494a4887 100644
--- a/site/src/cut/lib/projectReference.test.ts
+++ b/site/src/cut/lib/projectReference.test.ts
@@ -12,6 +12,7 @@ import type { StoredAsset } from "./types";
const copies: string[] = [];
setHostDocStore({
+ listDocs: async () => [],
readDoc: async () => null,
copyMedia: async (_src, file) => {
copies.push(file);
diff --git a/site/src/cut/lib/projectReference.ts b/site/src/cut/lib/projectReference.ts
index 330564d5..7921cfea 100644
--- a/site/src/cut/lib/projectReference.ts
+++ b/site/src/cut/lib/projectReference.ts
@@ -17,13 +17,14 @@ import { registerFontAsset } from "./fontAssets";
import { enrichAsset, uploadProjectMediaTo } from "./media";
import { localMediaFile, storedMediaUrl } from "./mediaSync";
import type { ProjectLink } from "./projectLink";
-import { projectBackend } from "./residency";
+import { availableResidencies, backendFor, projectBackend, type Residency } from "./residency";
import { useEditor } from "./store";
-import type { MediaAsset, ProjectDoc, ShareFeatures, StoredAsset } from "./types";
+import type { MediaAsset, ProjectDoc, ProjectSummary, ShareFeatures, StoredAsset } from "./types";
-/** What a headless host does for itself: read a project it stores, copy a
- * media file between two of its projects. */
+/** What a headless host does for itself: list and read the projects it
+ * stores, copy a media file between two of them. */
export interface HostDocStore {
+ listDocs(): Promise<ProjectSummary[]>;
readDoc(projectId: string): Promise<ProjectDoc | null>;
copyMedia(srcProjectId: string, fileName: string, dstProjectId: string): Promise<string>;
}
@@ -51,6 +52,51 @@ export interface ReferenceProject {
const BROWSER_ONLY =
"A project kept in a browser's own storage is readable only from that browser.";
+/** One of the user's own projects as the assistant finds it by name: the id
+ * every reader takes, the title it carries now, and where it lives. */
+export interface ListedProject {
+ id: string;
+ name: string;
+ residency: Residency;
+ updatedAt: number;
+}
+
+/** Every project the user owns on the shelves this process can read, newest
+ * first. A shelf that does not answer is named in `unreachable` so the reader
+ * can say why a project is missing. */
+export async function listOwnProjects(): Promise<{ projects: ListedProject[]; unreachable: Residency[] }> {
+ // A headless host reads only its own disk; the other shelves need the
+ // editor page's sign-in.
+ if (host) {
+ const docs = await host.listDocs();
+ return { projects: docs.map((p) => listed(p, "local")), unreachable: HOST_UNREAD };
+ }
+
+ // Each live shelf lists on its own; one failing leaves the others standing.
+ const unreachable: Residency[] = [];
+ const shelves = await Promise.all(
+ availableResidencies().map(async (r) => {
+ const res = await backendFor(r).fetch("/api/cut/projects").catch(() => null);
+ if (!res?.ok) {
+ unreachable.push(r);
+ return [];
+ }
+ return ((await res.json()) as ProjectSummary[]).map((p) => listed(p, r));
+ })
+ );
+ const projects = shelves.flat().sort((a, b) => b.updatedAt - a.updatedAt);
+ return { projects, unreachable };
+}
+
+const HOST_UNREAD: Residency[] = ["cloud", "browser"];
+
+const listed = (p: ProjectSummary, residency: Residency): ListedProject => ({
+ id: p.id,
+ name: p.name,
+ residency,
+ updatedAt: p.updatedAt,
+});
+
export async function openReference(link: ProjectLink): Promise<ReferenceProject> {
if (link.kind === "share") return openShare(link.token, link);
const id = link.id;
diff --git a/site/src/cut/lib/queries.ts b/site/src/cut/lib/queries.ts
index 7e228407..0a97de56 100644
--- a/site/src/cut/lib/queries.ts
+++ b/site/src/cut/lib/queries.ts
@@ -16,7 +16,7 @@
* user asked for it.
*/
-import { useQuery, useQueryClient, type QueryClient } from "@tanstack/react-query";
+import { useQueries, useQuery, useQueryClient, type QueryClient, type UseQueryResult } from "@tanstack/react-query";
import { useEffect } from "react";
import { confirmedEngineUser, currentEngineUser, engineLost, onEngineUserConfirmed } from "./api";
@@ -102,6 +102,57 @@ async function fetchProjectsSection(r: Residency): Promise<ProjectsSection> {
};
}
+/** One residency's listing as the home reads it: stored for the next cold
+ * load and noted for the chat-history sweep. */
+async function loadProjectsSection(r: Residency): Promise<ProjectsSection> {
+ try {
+ const as = currentEngineUser();
+ const data = await fetchProjectsSection(r);
+ writeSnapshot(projectsSnapshotKey(r), data);
+ noteListing(r, data, as);
+ return data;
+ } catch (err) {
+ if (r === "local" && cutMode() === "local") {
+ // The engine this app is running on stopped answering; the
+ // ConnectGate falls back to the cloud and says so. A cloud-bound app
+ // whose local shelf hiccups keeps running — that failure is one
+ // section's error, not the ground moving.
+ engineLost();
+ }
+ throw err;
+ }
+}
+
+const NO_PROJECTS: ProjectSummary[] = [];
+
+/** How long the @ menu trusts a projects listing before reading it again. A
+ * rename in this tab patches the cache at once; this bounds a listing read on
+ * every shelf, which on the Mac reads every project on disk. */
+const PROJECT_REFS_STALE_MS = 5 * 60_000;
+
+// Module-level, so the combined list keeps its identity while no listing
+// changes and the @ menu's memo holds.
+const allProjects = (results: UseQueryResult<ProjectsSection>[]) => {
+ const projects = results.flatMap((q) => q.data?.projects ?? []);
+ return projects.length ? projects : NO_PROJECTS;
+};
+
+/** Every project on the shelves this browser can read now, for the chat's
+ * @ menu. The same cache entries as the home's sections, so a rename there
+ * renames the mention here. */
+export function useOwnProjects(enabled: boolean): ProjectSummary[] {
+ const live = enabled ? availableResidencies() : [];
+ return useQueries({
+ queries: live.map((r) => ({
+ queryKey: projectsKey(r),
+ queryFn: () => loadProjectsSection(r),
+ staleTime: PROJECT_REFS_STALE_MS,
+ retry: false,
+ })),
+ combine: allProjects,
+ });
+}
+
/**
* One residency's projects and folders. Errors surface as `isError` rather
* than an empty list, so one backend failing never blanks the other's cards.
@@ -122,24 +173,7 @@ export function useProjectsSection(r: Residency, { list, live }: { list: boolean
return useQuery<ProjectsSection>({
enabled: list && live,
queryKey: projectsKey(r),
- queryFn: async () => {
- try {
- const as = currentEngineUser();
- const data = await fetchProjectsSection(r);
- writeSnapshot(projectsSnapshotKey(r), data);
- noteListing(r, data, as);
- return data;
- } catch (err) {
- if (r === "local" && cutMode() === "local") {
- // The engine this app is running on stopped answering; the
- // ConnectGate falls back to the cloud and says so. A cloud-bound app
- // whose local shelf hiccups keeps running — that failure is one
- // section's error, not the ground moving.
- engineLost();
- }
- throw err;
- }
- },
+ queryFn: () => loadProjectsSection(r),
// The listing is cheap to re-read and its cards carry edit times and
// sizes, so every visit revalidates — behind the cached paint, never a
// spinner.
diff --git a/site/src/cut/lib/refMedia.ts b/site/src/cut/lib/refMedia.ts
index 73c45361..11dc05c9 100644
--- a/site/src/cut/lib/refMedia.ts
+++ b/site/src/cut/lib/refMedia.ts
@@ -412,6 +412,8 @@ export async function refsToParts(
text:
ref.scope === "note"
? `Referenced note "${ref.name}". Read its current text with read_note: ${JSON.stringify({ id: ref.id })}`
+ : ref.scope === "projectDoc"
+ ? `Referenced project "${ref.name}". Read its current document with read_project: ${JSON.stringify({ link: ref.id })}`
: ref.use
? `Referenced ${ref.use.noun} "${ref.name}" in the Library. Use it with ${ref.use.tool}: ${JSON.stringify(ref.use.args)}`
: ref.scope === "folder"
diff --git a/site/src/cut/lib/replicateProject.test.ts b/site/src/cut/lib/replicateProject.test.ts
index eb36b882..1a42446e 100644
--- a/site/src/cut/lib/replicateProject.test.ts
+++ b/site/src/cut/lib/replicateProject.test.ts
@@ -28,6 +28,7 @@ const docs = new Map<string, ProjectDoc>([["rich", richDoc()]]);
const copies: { src: string; file: string; dst: string }[] = [];
setHostDocStore({
+ listDocs: async () => [],
readDoc: async (id) => docs.get(id) ?? null,
copyMedia: async (src, file, dst) => {
copies.push({ src, file, dst });
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index 720b0371..d66a763b 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -630,7 +630,7 @@ The Library is shared across every project: folders, reusable assets, and templa
Attachments: media files dropped on the chat import into project media by themselves; library attachments wait for library_add.`,
"replicating-a-project": `# Replicating another project
-A link to a Donkey Cut project (…/app/p/<id>) or a share (…/s/<token>) is a document you can read whole. read_project returns it in the editor_state shape — every clip with its id, trims, rate, framing, grade, look, mask, keyframes and animations; the transition bars; titles, shapes and stickers with their keyframes; the soundtrack; the caption look and cues; aspect and background — and each source under \`media\` with its id, its observed notes and its transcript. \`unobserved\` lists video sources with neither. \`partial\` on a share names what its owner withheld.
+A link to a Donkey Cut project (…/app/p/<id>), a share (…/s/<token>), a project attachment, or a project named by its title (find its id with list_projects) is a document you can read whole. read_project returns it in the editor_state shape — every clip with its id, trims, rate, framing, grade, look, mask, keyframes and animations; the transition bars; titles, shapes and stickers with their keyframes; the soundtrack; the caption look and cues; aspect and background — and each source under \`media\` with its id, its observed notes and its transcript. \`unobserved\` lists video sources with neither. \`partial\` on a share names what its owner withheld.
The document is the edit. Never watch_video a reference to reconstruct what read_project already states. Watch a reference source (watch_video with project_link and its asset_id or clip_id) only to see footage whose notes leave its content unclear; that look decides a mapping and writes nothing to the reference.
**Scope is the user's.** "Replicate this" is the whole edit; "use the title from this", "bring over the music", "grade my clip like theirs", "the caption style" are items. replicate_project takes \`items\` (any ids from read_project: clips on any track, soundtrack clips, elements, cues, transition bars) and brings only those, timed from the earliest; a chosen clip brings the bars playing on it. A single clip's treatment onto footage already here is a mapping: items:[that clip], media:[{source_asset_id, asset_id: the clip here}] — its grade, look, mask, keys and animations land on a new clip cut from the user's asset; delete the old one only if they asked for a replacement. A style alone with no clip — a grade, a caption look — is also readable straight off read_project and applied with the ordinary tools (set_color_grade, set_caption_look). Files alone go through copy_project_media.
**Mapping roles.** A look meant to be reproduced — type, weight, spacing, colour, edge treatment — is watched with detail original; a contact-sheet cell has already thrown those away. Each reference source the chosen items use plays from one of this project's assets or copies across. Match by content — a talking-head source to the talking-head here, b-roll to b-roll — using the reference's notes and transcripts against this project's (\`media\` in editor_state, watch_video on anything unseen). One asset here may play several reference parts. Video and stills stand in for video; audio for audio. Leave unmapped whatever has no counterpart — music, stickers, images, fonts, a clip the user wants as-is — and it copies from the reference into this project, on a card in this chat.
@@ -663,7 +663,7 @@ Rules:
- A skill attached to the message (<skill_relevance>) is the reference for its area: read it before touching that area's settings. It never changes which tool the ask maps to — the tool descriptions decide that ("a better version" is a render, whatever skill rode along). list_skills / read_skill cover an area the attachment left out.
- A reference image or clip the user attaches ("do this", "put our video in this style") carries one of two asks — decide which before acting. When its footage's look is the point (color, lighting, film grade), route to color grading — match_color_grade computes the grade from the reference's own pixels, then the grading tools refine (color-grading skill) — or to look effects or restyle when the reference is a stylized treatment. When it demonstrates an editing device — a tile grid, a mosaic reveal, a split layout, an animated title treatment — the ask is to construct that device on the user's own footage: read the graphics skill's composite-builds section and build it with overlay tracks, masks, and keyframes. Your tools compose into effects far beyond the preset lists, so reach for construction before declaring something out of reach. Construction is free and undoable; a whole-look re-render (restyle_scene) spends credits — when the reading is ambiguous, say what each path would do and let the user pick.
- A video held up as a reference — a file they imported, a link you brought in, a clip on the timeline — is watched all the way through before anything is copied from it, then rebuilt out of this editor's parts: everything on the screen, every card of type, every shape, every colour and every move, at the time it happens. The footage inside the shots is the one thing you cannot rebuild — add_clip \`blocks\` holds each shot's place in the colour behind it, for the person to fill, or for Pexels footage you find when they ask for the shots filled — and when they want the reference's own picture playing, add_clip \`spans\` lays its shots down instead. What you built is a guess until you look at it beside the thing it copies: compare_to_source scores the cut against the source frame by frame and shows the worst moments side by side, and what differs gets fixed and checked again. The replicating-a-video skill has the flow.
-- A Donkey Cut project link or share link the user gives you is a document: read it with read_project (the document is the edit — never watch a reference to learn what its document already states), decide which of this project's sources plays each of its parts from the reference's observed notes and transcripts (a reference look with watch_video's project_link where those leave it dark), then call replicate_project once — the whole edit, or only the items they asked for: a title, one clip's treatment, the music, the caption look — and tune from there. copy_project_media brings files over on their own. The replicating-a-project skill has the flow.
+- A Donkey Cut project link, share link, project attachment, or project the user or a note names by title (its id from list_projects) is a document: read it with read_project (the document is the edit — never watch a reference to learn what its document already states), decide which of this project's sources plays each of its parts from the reference's observed notes and transcripts (a reference look with watch_video's project_link where those leave it dark), then call replicate_project once — the whole edit, or only the items they asked for: a title, one clip's treatment, the music, the caption look — and tune from there. copy_project_media brings files over on their own. The replicating-a-project skill has the flow.
- Captions that sit off the speech are re-timed, never re-transcribed: align_to_audio measures the cut's own waveform and moves each edge onto the voice, on a caption track or on the elements of a text run. Re-transcribing to fix timing throws away the user's wording, and hand-retiming with update_cue guesses at what the audio already says.
- Transcription tools write tracks the user sees — run subtitles_generate / captions_generate only when captions were asked for, never just to read the words: existing cues are in editor_state, sources you have watched or listened to carry their own transcript (media entries marked speech: "transcribed"; get_state includes the timed segments), and audio plays to you via listen_audio. Don't transcribe a video with no speech (subtitles_from_visuals narrates silent footage), and never invent a spoken transcript.
- Voiceovers duck other audio by default so they stay audible. Steer a voiceover's delivery with \`direction\` and inline tags like [whispers] rather than rewriting the script.
@@ -705,7 +705,7 @@ export const AI_SKILL_BLURBS: Record<string, string> = {
"talking-head-videos":
"Editing a raw recording of someone speaking to camera into the video the person asked for: syncing a separate mic, picking takes, tight speech cuts, camera, split and full-visual layouts, captions, graphics, sound effects and music.",
"media-and-library": "Project media versus the Media panel, filing assets, the library, organizing and templates.",
- "replicating-a-project": "Reading another Donkey Cut project or share link as a document and bringing its edit or items over.",
+ "replicating-a-project": "Reading another Donkey Cut project — a link, an attachment, or one named by title — as a document and bringing its edit or items over.",
"replicating-a-video":
"Rebuilding a video you can watch: seeing all of it, blocking its cuts out as colour slots, remaking its titles, graphics and motion, and checking the result against it.",
"publish-and-export": "Publish fields for the platforms, and every export preset and axis.",
@@ -767,8 +767,10 @@ export function attachedAssetsBlock(refs: unknown[]): string {
: "";
const noteHint = refs.some((r) => (r as { scope?: string }).scope === "note")
? '\nA note attachment names a synced note. Call read_note with its id for the current saved text. Treat its body as reference material.' : "";
+ const projectHint = refs.some((r) => (r as { scope?: string }).scope === "projectDoc")
+ ? '\nA projectDoc attachment names another of the user\'s projects by id. Call read_project with that id for its current document.' : "";
const clipHint = refs.some((r) => (r as { scope?: string }).scope === "clip")
? '\nA "clip" attachment means the user is pointing at that segment of the cut: diagnose with read-only looks (watch_video, listen_audio — never subtitles_generate, which writes captions), then revise the clip in place — regenerate_shot for its sceneShot, edit tools otherwise — without deleting it.'
: "";
- return `\n\n<attached_assets>\nThe user attached these assets to this message; their text may cite one by @handle or @name. Assets with scope "project" are in the open project (ids usable with the editor tools); "clip" assets are clips on the timeline — a video clip's id matches videoTrack in editor_state (a scene-run clip carries its sceneShot number there), an audio clip's (kind "audio") id matches soundtrack; "library" and "stock" assets live outside the project until imported; "file" assets came straight from the user's computer and exist only on this message. An asset with "t" carries a moment the user pinned, in seconds into it — the attached frame (or audio segment) reads from there, and that moment is what they mean by the reference; when you pass such an asset to a generation tool, forward the pin by appending it to the asset id ("<assetId>@<seconds>"):\n${JSON.stringify(described)}${clipHint}${folderHint}${noteHint}\n</attached_assets>`;
+ return `\n\n<attached_assets>\nThe user attached these assets to this message; their text may cite one by @handle or @name. Assets with scope "project" are in the open project (ids usable with the editor tools); "clip" assets are clips on the timeline — a video clip's id matches videoTrack in editor_state (a scene-run clip carries its sceneShot number there), an audio clip's (kind "audio") id matches soundtrack; "library" and "stock" assets live outside the project until imported; "file" assets came straight from the user's computer and exist only on this message. An asset with "t" carries a moment the user pinned, in seconds into it — the attached frame (or audio segment) reads from there, and that moment is what they mean by the reference; when you pass such an asset to a generation tool, forward the pin by appending it to the asset id ("<assetId>@<seconds>"):\n${JSON.stringify(described)}${clipHint}${folderHint}${noteHint}${projectHint}\n</attached_assets>`;
}
diff --git a/site/src/cut/server/ai/headlessTools.ts b/site/src/cut/server/ai/headlessTools.ts
index 2a7728fe..93a49eb5 100644
--- a/site/src/cut/server/ai/headlessTools.ts
+++ b/site/src/cut/server/ai/headlessTools.ts
@@ -11,7 +11,7 @@ import { serializeDoc, useEditor } from "../../lib/store";
import type { MediaAsset, ProjectDoc } from "../../lib/types";
import { convertToMp4 } from "../convert";
import { probeDuration } from "../frames";
-import { deleteMedia, mediaPath, readProject, writeProject } from "../projects";
+import { deleteMedia, listProjects, mediaPath, readProject, writeProject } from "../projects";
import { uniqueName } from "../util";
import { withFileColors } from "../fileColor";
import { drawnLists } from "../../lib/itemKinds";
@@ -108,10 +108,11 @@ setHostMediaStore({
},
});
-// Another project on this Mac, read as a reference: the doc off disk and a
-// file copied folder to folder. A cloud or shared reference needs the page's
+// Another project on this Mac, listed and read as a reference: the doc off
+// disk and a file copied folder to folder. A cloud or shared reference needs the page's
// sign-in, and the reader says so.
setHostDocStore({
+ listDocs: () => listProjects(),
readDoc: (projectId) => readProject(projectId),
async copyMedia(srcProjectId, fileName, dstProjectId) {
const target = await uniqueName(fileName, (n) => mediaPath(dstProjectId, n));