Skip to content

Open source CapCut alternative.

TypeScript

feat(site): a chat turn the page lost resumes itself

David Lebd3e6c3551c2
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.md
index 3b4ffb8e..860aa847 100644
--- a/docs/guides/cut/ai-assistant.md
+++ b/docs/guides/cut/ai-assistant.md
@@ -35,6 +35,8 @@ A Claude or Codex turn can outlive its tab while the Mac engine remains running.
Gemini's interactive turn runs in the browser. Its active turn survives panel and project navigation within that page, saves its final transcript after unmount, and waits for its original project before executing another editor tool. Opening another project waits for tools already using the store. Closing the browser tab ends this loop; the cloud worker's queued-turn API is a separate execution path. Project residency stays fixed throughout these transitions.
+A turn the page lost resumes itself when the thread next opens, replaying the engine journal or sending the model a continuation that finishes the ask with the failure gone from the transcript.
+
The Gemini path skips the engine entirely — the same carve-out as AI media generation. The loop runs on the pi agent harness (the `@earendil-works` packages) in the browser: a custom transport speaks Donkey's hosted responses route with the user's session, tools execute directly against the store, and each thread's conversation persists as the harness's own structured history — past tool calls replay as real tool records, so no bookkeeping rides as prose the model could imitate. Gemini's quirks live in that transport: thought signatures replay exactly with each call, tool media returns as its own following turn, and an empty round retries before it surfaces as an error.
The turn's route is decided by an intent gate while the turn is already running. A light-model call judges the newest message three ways — a turn that requests nothing, one self-contained ask, a composed job — and the first round goes out immediately on the light chat model with the full catalog while that call is in flight. Tool execution waits on the verdict and the round's output buffers: a simple verdict (the common case) releases the buffer and keeps the run, so the gate costs no wall time, while a chat or complex verdict discards the speculative run with nothing shown and nothing executed and restarts the turn routed — every tool declaration withheld for chat, the full model for a composed job or any classifier doubt. A message carrying attachments is complex on sight and runs routed from the start.
diff --git a/site/src/cut/components/AiPanel.tsx b/site/src/cut/components/AiPanel.tsx
index aefdb2eb..5c58ba97 100644
--- a/site/src/cut/components/AiPanel.tsx
+++ b/site/src/cut/components/AiPanel.tsx
@@ -5,7 +5,7 @@ import type { ChatStatus } from "./ChatStatusBadge";
import { chatRuntime } from "@/cut/lib/chatRuntime";
import { Fragment, memo, useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState, useSyncExternalStore, type ReactNode } from "react";
import { useChat } from "@ai-sdk/react";
-import { DefaultChatTransport, type ChatTransport, type UIMessage } from "ai";
+import { DefaultChatTransport, type ChatTransport, type UIMessage, type UIMessageChunk } from "ai";
import {
ArrowUp,
Brain,
@@ -101,6 +101,7 @@ import { runAiTool } from "@/cut/lib/aiTools";
import { holdEditorChat } from "@/cut/lib/editorWork";
import { recoverSceneCall } from "@/cut/lib/chatRecovery";
import { replayChatStream } from "@/cut/lib/chatReplay";
+import { INTERRUPTED_ERROR, isResumeMessage, RESUME_LIMIT, resumePrompt, scrubLostTurn, unfinishedAsk, type ResumeMetadata, type TurnSettled } from "@/cut/lib/chatResume";
import { claimEngineTool, cancelEngineChat } from "@/cut/lib/engineChat";
import { projectBackend } from "@/cut/lib/residency";
import { beginBrowserChat, browserChatRunning, cancelBrowserChat, watchBrowserChatTurns } from "@/cut/lib/browserChatTurns";
@@ -141,6 +142,11 @@ interface ChatThread {
updatedAt: number;
messages: UIMessage[];
completedMessageId?: string;
+ /** How the newest turn ended when it did not complete. Unset while it runs
+ * and when the page lost it, which is what resumes it on return. */
+ turnSettled?: TurnSettled;
+ /** Continuations the newest ask has had. */
+ resumed?: number;
/** Provider-native session ids so a resumed thread keeps its context. */
sessions: Record<string, string>;
/** The pi loop's LLM context (structured tool history included), when the
@@ -167,6 +173,22 @@ function readSeenReplies(projectId: string): Record<string, string> {
// How long a streaming turn's newest snapshot may park before it must land —
// the same cadence the cloud mirror debounces on.
const THREAD_SAVE_MS = 1500;
+// How long a dropped connection rests before the turn picks itself back up.
+const RESUME_RETRY_MS = 1500;
+// How long after its last save a lost turn still resumes; an older thread is
+// one the user walked away from, and it waits for them.
+const RESUME_WINDOW_MS = 24 * 60 * 60 * 1000;
+
+/** Attachments a restored message can still deliver: file blobs die with the
+ * page that made them, and a project asset may have been deleted since. */
+function liveRefs(refs: AssetRef[]): AssetRef[] {
+ return refs.filter(
+ (a) =>
+ !a.url?.startsWith("blob:") &&
+ (a.scope !== "project" ||
+ useEditor.getState().assets.some((x) => x.id === a.id)),
+ );
+}
function readThreads(projectId: string): ChatThread[] {
return readRawThreads(projectId).filter((t) => isStoredThread(t) && !t.deleted) as ChatThread[];
@@ -257,7 +279,7 @@ function settleInterruptedTools(messages: UIMessage[]): UIMessage[] {
const t = p as { type: string; state?: string };
if (!isToolPartType(t.type)) return p;
if (t.state === "output-available" || t.state === "output-error") return p;
- return { ...p, state: "output-error", errorText: "Interrupted." } as typeof p;
+ return { ...p, state: "output-error", errorText: INTERRUPTED_ERROR } as typeof p;
}),
}));
}
@@ -869,6 +891,14 @@ function ChatSession({
);
const seenThreadAt = useRef(initialThread?.updatedAt ?? 0);
const completedMessageId = useRef(initialThread?.completedMessageId);
+ const turnSettled = useRef<TurnSettled | undefined>(initialThread?.turnSettled);
+ const resumed = useRef(initialThread?.resumed ?? 0);
+ const userStopped = useRef(false);
+ // A dropped connection with continuations left: the turn rests, then resumes.
+ const [retryDisconnect, setRetryDisconnect] = useState(false);
+ // The engine journal reconnect for a turn the page lost: idle until the
+ // resume effect asks for one, pending while it runs, done once it answered.
+ const [reconnect, setReconnect] = useState<"idle" | "pending" | "done">("idle");
const providerSessions = useRef<Record<string, string>>({
...(initialThread?.sessions ?? {}),
});
@@ -940,6 +970,11 @@ function ChatSession({
// Claude/Codex chat through the local engine; Gemini goes straight from
// the page to Donkey's hosted inference with the user's session.
sendMessages: async (options) => {
+ // A new turn opens with no verdict; a continuation keeps its ask's count.
+ turnSettled.current = undefined;
+ userStopped.current = false;
+ const newest = options.messages.at(-1);
+ if (!newest || !isResumeMessage(newest)) resumed.current = 0;
if (provider(currentModel()) === "gemini") {
setClientTools(true);
const signal = registerBrowserTurn(options.abortSignal);
@@ -958,15 +993,22 @@ function ChatSession({
reconnectToStream: async (options) => {
if (provider(currentModel()) === "gemini") return Promise.resolve(null);
setClientTools(false);
- const stream = await engine().reconnectToStream(options);
- return stream ? replayChatStream(stream, prepareReplay, attachReplay, completedMessageId.current) : null;
+ let replay: ReadableStream<UIMessageChunk> | null = null;
+ try {
+ const stream = await engine().reconnectToStream(options);
+ replay = stream ? await replayChatStream(stream, prepareReplay, attachReplay, completedMessageId.current) : null;
+ } finally {
+ // A replay settles the reconnect when it finishes (onFinish); with
+ // nothing to replay the resume effect moves on to a continuation.
+ if (!replay) setReconnect("done");
+ }
+ return replay;
},
};
}, [threadId, projectId, currentModel, sessionFor, setClientTools, prepareReplay, attachReplay, registerBrowserTurn]);
- const { messages, setMessages, sendMessage, stop, status, error, clearError } = useChat({
+ const { messages, setMessages, sendMessage, stop, status, error, clearError, resumeStream } = useChat({
id: threadId,
- resume: !readOnly && provider(model) !== "gemini",
messages: initialThread && settleInterruptedTools(initialThread.messages),
transport,
onError: () => {
@@ -981,6 +1023,18 @@ function ChatSession({
finishBrowserTurn.current = null;
if (chatThreadDeleted(projectId, threadId)) return;
if (!isAbort && !isDisconnect && !isError) completedMessageId.current = message.id;
+ // A verdict holds the thread for the user. A detach (this panel closing
+ // its reader while the engine keeps working) and a dropped connection
+ // with continuations left leave none, so the turn resumes.
+ const retrying = isDisconnect && resumed.current < RESUME_LIMIT;
+ if (isAbort) {
+ if (userStopped.current) turnSettled.current = "stopped";
+ } else if (isError && !retrying) {
+ turnSettled.current = "failed";
+ }
+ userStopped.current = false;
+ setRetryDisconnect(retrying);
+ setReconnect(retrying ? "idle" : "done");
const stored = readThreads(projectId);
const previous = stored.find((thread) => thread.id === threadId);
const firstUser = finished.find((message) => message.role === "user");
@@ -990,6 +1044,8 @@ function ChatSession({
title: firstUser?.parts.map((part) => part.type === "text" ? part.text : "").join("").trim().slice(0, 80) || "New chat",
updatedAt: Date.now(),
completedMessageId: completedMessageId.current,
+ turnSettled: turnSettled.current,
+ resumed: resumed.current,
messages: recoverSceneCall(finished, threadId, useEditor.getState().projectId === projectId ? useEditor.getState().genvideo : undefined),
sessions: { ...providerSessions.current },
pi: readPiSession(threadId) ?? previous?.pi,
@@ -1118,6 +1174,8 @@ function ChatSession({
seenThreadAt.current = thread.updatedAt;
receivedThread.current = true;
completedMessageId.current = thread.completedMessageId;
+ turnSettled.current = thread.turnSettled;
+ resumed.current = thread.resumed ?? 0;
providerSessions.current = { ...thread.sessions };
dropPiSession(threadId);
hydratePiSession(threadId, thread.pi);
@@ -1172,7 +1230,13 @@ function ChatSession({
// session has been let go reads back as undefined, and writing that would
// erase the tool-call context the stored one still has. What is on disk
// stands until a live session replaces it.
- const merged = { ...thread, completedMessageId: completedMessageId.current, pi: thread.pi ?? stored.find((t) => t.id === thread.id)?.pi };
+ const merged = {
+ ...thread,
+ completedMessageId: completedMessageId.current,
+ turnSettled: turnSettled.current,
+ resumed: resumed.current,
+ pi: thread.pi ?? stored.find((t) => t.id === thread.id)?.pi,
+ };
seenThreadAt.current = merged.updatedAt;
writeThreads(projectId, [merged, ...stored.filter((t) => t.id !== thread.id)]);
// Cloud projects mirror the thread server-side (debounced while it streams).
@@ -1332,6 +1396,64 @@ function ChatSession({
setAttachments([]);
};
+ // A turn the page lost picks itself back up. With an engine journal the
+ // panel reconnects and replays it; when there is none, or the replay shows
+ // a tool that needed this page, the model gets a continuation quoting the
+ // ask. The count rides the thread, so a reload loop ends at RESUME_LIMIT;
+ // a turn the user stopped, one that failed outright, and one left longer
+ // than RESUME_WINDOW_MS all wait for them.
+ const projectLoaded = useEditor((s) => s.projectId === projectId && s.loaded);
+ const signedIn = useGenerate((s) => s.signedIn);
+ const resumeOwed = useCallback(
+ () => (resumed.current < RESUME_LIMIT ? unfinishedAsk(messages, completedMessageId.current, turnSettled.current) : undefined),
+ [messages],
+ );
+ // A dropped connection rests briefly; clearing its error settles the
+ // status to ready, and the resume below takes it from there.
+ useEffect(() => {
+ if (status !== "error" || !retryDisconnect) return;
+ const timer = window.setTimeout(() => {
+ setRetryDisconnect(false);
+ clearError();
+ }, RESUME_RETRY_MS);
+ return () => window.clearTimeout(timer);
+ }, [status, retryDisconnect, clearError]);
+ useEffect(() => {
+ if (status !== "ready" || busy || readOnly || !projectLoaded || !currentAvailable) return;
+ if (provider(model) === "gemini" ? signedIn !== true : !info) return;
+ if (Date.now() - seenThreadAt.current > RESUME_WINDOW_MS) return;
+ const ask = resumeOwed();
+ if (!ask) return;
+ if (provider(model) !== "gemini") {
+ if (reconnect === "pending") return;
+ if (reconnect === "idle") {
+ // eslint-disable-next-line react-hooks/set-state-in-effect -- the reconnect reacts to the thread settling unfinished; the state gates it to one attempt
+ setReconnect("pending");
+ void resumeStream();
+ return;
+ }
+ }
+ // Another tab holding this thread may have resumed it already.
+ const stored = readThreads(projectId).find((t) => t.id === threadId);
+ if ((stored?.resumed ?? 0) > resumed.current) {
+ resumed.current = stored?.resumed ?? 0;
+ return;
+ }
+ resumed.current += 1;
+ const refs = ((ask.metadata as { attachments?: unknown[] } | undefined)?.attachments ?? [])
+ .map(normalizeRef)
+ .filter((r): r is AssetRef => r !== null);
+ const live = liveRefs(refs);
+ setSendError(null);
+ pinnedRef.current = true;
+ // The continuation finishes the job in the lost turn's place, so the
+ // transcript drops that turn's failure before it goes out: the cut-off
+ // reply keeps the calls that landed and nothing else.
+ setMessages((current) => scrubLostTurn(current, ask.id));
+ const metadata: ResumeMetadata = { resume: true, ...(live.length > 0 ? { attachments: live } : {}) };
+ void sendMessage({ text: resumePrompt(ask), metadata });
+ }, [status, busy, readOnly, projectLoaded, currentAvailable, model, signedIn, info, reconnect, resumeOwed, resumeStream, setMessages, sendMessage, projectId, threadId]);
+
// Drain the queue as turns settle. drainedRef keeps it to one dispatch per
// ready period — the effect re-runs when the queue changes before useChat
// surfaces the new turn's status. A dispatched row stays in the queue as
@@ -1348,6 +1470,8 @@ function ChatSession({
return;
}
if (drainedRef.current || queuePaused || !currentAvailable) return;
+ // A thread still owed a reply resumes that first.
+ if (resumeOwed()) return;
const next = queue.find(
(m) => m.status === "queued" && m.id !== queueEditing,
);
@@ -1368,17 +1492,12 @@ function ChatSession({
// A restored row's attachments may point at assets deleted since it was
// parked (or at file blobs that died with the last page); dropping the
// dead refs keeps the message from claiming media it can't deliver.
- const live = next.attachments.filter(
- (a) =>
- !a.url?.startsWith("blob:") &&
- (a.scope !== "project" ||
- useEditor.getState().assets.some((x) => x.id === a.id)),
- );
+ const live = liveRefs(next.attachments);
void sendMessage({
text: next.text,
...(live.length > 0 && { metadata: { attachments: live } }),
});
- }, [status, queue, queuePaused, queueEditing, currentAvailable, sendMessage]);
+ }, [status, queue, queuePaused, queueEditing, currentAvailable, resumeOwed, sendMessage]);
// Mirror the waiting rows to storage as they change; an empty queue clears
// its slot.
@@ -1472,7 +1591,7 @@ function ChatSession({
<LiveElapsed />
</div>
)}
- {(error || (sendError && !currentAvailable)) && (
+ {((error && !retryDisconnect) || (sendError && !currentAvailable)) && (
<div className="ai-error mt-2 flex items-start gap-2 rounded-lg bg-red-50 px-2.5 py-2 text-[11.5px] leading-relaxed text-red-700">
<TriangleAlert className="mt-0.5 size-3.5 shrink-0" />
<span>
@@ -1668,6 +1787,7 @@ function ChatSession({
// pausing it anyway would leave a stale flag.
if (queue.some((m) => m.status === "queued"))
setQueuePaused(true);
+ userStopped.current = true;
toolAbort.current.abort();
if (provider(model) !== "gemini") void cancelEngineChat(projectId, threadId).catch((error) => showNotice(String(error)));
cancelBrowserChat(projectId, threadId);
@@ -2100,6 +2220,9 @@ const MessageView = memo(function MessageView({
}: {
message: UIMessage;
}) {
+ // A continuation is the panel's own doing; the reply it brings reads as the
+ // ask's reply.
+ if (isResumeMessage(message)) return null;
if (message.role === "user") {
const text = message.parts
.map((p) => (p.type === "text" ? p.text : ""))
diff --git a/site/src/cut/lib/chatRecovery.ts b/site/src/cut/lib/chatRecovery.ts
index 590157ec..9f341bea 100644
--- a/site/src/cut/lib/chatRecovery.ts
+++ b/site/src/cut/lib/chatRecovery.ts
@@ -1,4 +1,5 @@
import type { UIMessage } from "ai";
+import { INTERRUPTED_ERROR } from "./chatResume";
import type { VideoProject } from "./genvideo/types";
/** Reconcile the latest scene call with its persisted job after reconnecting. */
@@ -18,7 +19,7 @@ export function recoverSceneCall(
if (typeof tool.input?.brief === "string" && tool.input.brief.trim() !== scene.brief) return part;
if (typeof tool.input?.from_audio_asset_id === "string" && tool.input.from_audio_asset_id !== scene.audioAssetId) return part;
if (tool.state === "output-available") return part;
- if (tool.state === "output-error" && tool.errorText !== "Interrupted.") return part;
+ if (tool.state === "output-error" && tool.errorText !== INTERRUPTED_ERROR) return part;
const { errorText: _error, ...rest } = part as typeof part & { errorText?: string };
void _error;
if (scene.phase === "failed")
diff --git a/site/src/cut/lib/pi/cutAgent.ts b/site/src/cut/lib/pi/cutAgent.ts
index 043be475..0ca89909 100644
--- a/site/src/cut/lib/pi/cutAgent.ts
+++ b/site/src/cut/lib/pi/cutAgent.ts
@@ -2,6 +2,7 @@ import type { UIMessage, UIMessageChunk } from "ai";
import { Agent, type AgentMessage } from "@earendil-works/pi-agent-core";
import type { Message, UserMessage } from "@earendil-works/pi-ai";
import { AI_TOOLS, attachedAssetsBlock, systemPrompt } from "@/cut/server/ai/catalog";
+import { isResumeMessage } from "../chatResume";
import { parseTurnIntent, turnIntentInput, turnIntentPrompt, type TurnIntent } from "../turnIntent";
import { enforceContextBudget } from "./contextBudget";
import { donkeyModel, type ChatThinkingLevel } from "./donkeyModel";
@@ -310,17 +311,26 @@ function sessionFor(threadId: string, history: UIMessage[]): AgentMessage[] {
const GATE_PROMPT = turnIntentPrompt();
+/** A message that runs routed from the start, with no gate call and no
+ * speculative round: one carrying attachments, or the continuation of a turn
+ * the page lost, which picks up whatever tools that turn was using. */
+function complexOnSight(lastUser: UIMessage | undefined): boolean {
+ if (!lastUser) return false;
+ if (isResumeMessage(lastUser)) return true;
+ const attached = (lastUser.metadata as { attachments?: unknown[] } | undefined)?.attachments;
+ return Array.isArray(attached) && attached.length > 0;
+}
+
/** The turn's gate and router (ported from the legacy loop): judge the newest
* message; "chat" withholds every tool, "simple" downgrades the model. A
- * message with attachments is complex by construction. Fails open. */
+ * message complex on sight skips the call. Fails open. */
async function classifyTurnIntent(
messages: UIMessage[],
deps: CutAgentDeps,
abortSignal?: AbortSignal
): Promise<{ intent: TurnIntent; skipped: boolean }> {
const lastUser = messages.findLast((m) => m.role === "user");
- const attached = (lastUser?.metadata as { attachments?: unknown[] } | undefined)?.attachments;
- if (Array.isArray(attached) && attached.length > 0) return { intent: "complex", skipped: true };
+ if (complexOnSight(lastUser)) return { intent: "complex", skipped: true };
const turns = messages.map((m) => ({
role: m.role === "user" ? ("user" as const) : ("assistant" as const),
text: m.parts
@@ -597,12 +607,10 @@ export function streamCutChat({
// lands, so a chat or complex verdict aborts the run with nothing
// shown and nothing executed, and the turn restarts routed — while a
// simple verdict (the common case) keeps the run and pays no gate
- // wall time at all. A message with attachments resolves complex
- // instantly, so it runs routed from the start.
- const attached = (lastUser?.metadata as { attachments?: unknown[] } | undefined)
- ?.attachments;
+ // wall time at all. A message complex on sight runs routed from
+ // the start.
let kept = false;
- if (!(Array.isArray(attached) && attached.length > 0) && !abortSignal?.aborted) {
+ if (!complexOnSight(lastUser) && !abortSignal?.aborted) {
const buffer: Record<string, unknown>[] = [];
let mode: "buffering" | "live" | "discarded" = "buffering";
const send = (chunk: Record<string, unknown>) => {