Skip to content

Open source CapCut alternative.

TypeScript

feat(site): skills, tool areas, and turn intent routed by one judgment [rebuild]

David Lee724dae1cdda
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.md
index d669c73e..abe57065 100644
--- a/docs/guides/cut/ai-assistant.md
+++ b/docs/guides/cut/ai-assistant.md
@@ -39,7 +39,7 @@ A turn the page lost resumes itself when the thread next opens, replaying the en
The Gemini path skips the engine entirely — the same carve-out as AI media generation. The loop runs on the pi agent harness (the `@earendil-works` packages) in the browser: a custom transport speaks Donkey's hosted responses route with the user's session, tools execute directly against the store, and each thread's conversation persists as the harness's own structured history — past tool calls replay as real tool records, so no bookkeeping rides as prose the model could imitate. Gemini's quirks live in that transport: thought signatures replay exactly with each call, tool media returns as its own following turn, and an empty round retries before it surfaces as an error.
-The turn's route is decided by an intent gate while the turn is already running. A light-model call judges the newest message three ways — a turn that requests nothing, one self-contained ask, a composed job — and the first round goes out immediately on the light chat model with the full catalog while that call is in flight. Tool execution waits on the verdict and the round's output buffers: a simple verdict (the common case) releases the buffer and keeps the run, so the gate costs no wall time, while a chat or complex verdict discards the speculative run with nothing shown and nothing executed and restarts the turn routed — every tool declaration withheld for chat, the full model for a composed job or any classifier doubt. A message carrying attachments is complex on sight and runs routed from the start.
+The turn's route is one typed judgment: a System One judge scores the ask's size, the skill to open first, and the tool areas it needs, and code applies the thresholds in the `cutJudge` setting. The first round waits a bounded moment for it, then runs the light chat model with only the routed areas' tools; a simple verdict keeps the run, chat or complex restarts it routed (a composed job gets its skill attached), and a `request_tools` call widens the catalog for the next round.
The same loop code runs with no browser at all. A headless session opens a project doc into the editor store, binds both transports at an explicit origin with auth headers, and runs the production chat loop against it — which is how a chat turn becomes a durable job. The page or an API caller posts a thread's messages to the cloud project's turns route; the row is the project's agent write lease (one turn at a time per project); the worker container claims it, runs the turn under a runner grant that spends that user's credits and writes that user's project, pushes the doc back through the versioned PUT, and appends the finished exchange to the stored thread.
@@ -47,7 +47,7 @@ A job can do everything a tab can, because the same tool code runs over swapped
Two mechanisms keep a turn honest. A mutation ledger, computed in code from the tool results, rides every model call as an ephemeral context note: what ran, what failed, and what debris the edits left (a parked transition bar, an item stranded past the video's end) — the model must fix or plainly report each entry before its reply lands, and the note is never stored or rendered. And the round budget extends itself: a build that outgrows its 24 rounds gets fresh budget automatically (up to three extensions, so 96 rounds in all), so the user never types "keep going". At the ceiling tool calls stop and the model gets one round to say where it landed; a turn still reaching for tools ends with a line offering "keep going".
-The composer stays open while a turn runs. A message sent mid-turn waits in a small tray on the composer's top edge and goes out when the running turn settles — one turn at a time, each with a fresh editor snapshot, so a queued ask always sees the edits made before it. A light-model triage places each mid-turn message first: a refinement of the running work is steered into the live turn, an independent ask that cannot collide starts a parallel thread, and everything else — including whatever the call cannot place — waits in the tray, where the running agent can also park a message itself when it needs the finished result first. The tray is the review surface: waiting rows can be reordered, rewritten, or removed, and the queue pauses and resumes from its header. While a row is being rewritten the list holds still — the row under edit waits, the others keep draining and cross out in place as they finish. Stopping a turn or hitting an error pauses the queue automatically, so the remaining asks hold for review. Waiting rows are saved with the thread on the machine: a reload or a trip to another project brings them back paused, ready to resume.
+The composer stays open while a turn runs. A message sent mid-turn waits in a small tray on the composer's top edge and goes out when the running turn settles — one turn at a time, each with a fresh editor snapshot, so a queued ask always sees the edits made before it. A typed triage places each mid-turn message first: a refinement of the running work is steered into the live turn, an independent ask that cannot collide starts a parallel thread, and everything else — including whatever the call cannot place — waits in the tray, where the running agent can also park a message itself when it needs the finished result first. The tray is the review surface: waiting rows can be reordered, rewritten, or removed, and the queue pauses and resumes from its header. While a row is being rewritten the list holds still — the row under edit waits, the others keep draining and cross out in place as they finish. Stopping a turn or hitting an error pauses the queue automatically, so the remaining asks hold for review. Waiting rows are saved with the thread on the machine: a reload or a trip to another project brings them back paused, ready to resume.
Which path runs is chosen by the model id alone. The picker lives in the page so a site deploy updates it for everyone; the engine only reports which CLIs are installed and signed in (probed and cached for a minute), and the page overlays Gemini availability from its own sign-in probe.
@@ -58,8 +58,8 @@ Every knowledge surface is defined once — the catalog file ships in the engine
| Surface | Size today | When it enters context |
| --- | --- | --- |
| System prompt | ~13K chars (~3K tokens) | Claude: replaces the Agent SDK default. Codex: prepended to the first turn (a resumed session already has it). Gemini: the instructions field, every round. |
-| Tool catalog (117 tools) | ~140K chars (~35K tokens) | Claude/Codex: the MCP tool listing. Gemini: function declarations on each request. |
-| Skills library (16 docs) | ~97K chars (~24K tokens) total | On demand only, via the list-skills and read-skill tools. |
+| Tool catalog (127 tools in 19 areas) | ~140K chars (~35K tokens) whole | Claude/Codex: the MCP tool listing. Gemini: the core plus the routed areas per request. |
+| Skills library (16 docs) | ~97K chars (~24K tokens) total | The judge's pick rides the turn; list-skills and read-skill cover the rest. |
| Editor snapshot | Grows with the project; media list and subtitle cues capped at 60 each | Attached to the newest user message as `<editor_state>`, rebuilt fresh every turn. |
| Attachments | Metadata JSON per asset | `<attached_assets>` on the message that carried them; on the Gemini path the newest message also carries the actual payloads (frames, images, text contents). |
| Full state | Uncapped | The get-state tool — the model calls it when the snapshot is stale or truncated. |
@@ -72,6 +72,8 @@ The snapshot is a compact JSON picture of everything user-visible: project meta,
## The decision layer
+Decisions that gate, route, or pick (the turn route, queue triage, stock ranking, filler words, a described voice, a provider's failure class) are typed judgments from the judge route, thresholded in code. Reasoning and generation stay on the chat model.
+
Deciding what the user wants is prompt text, executed by the model. The prompt orders the calls it must make each turn:
1. **Deliverable first.** "Write me a caption / a script / a prompt" asks for words — the answer goes in chat and the project stays untouched until the user says "do it". A request to change the project gets acted on directly with tools.
diff --git a/site/scripts/eval-cut-chat.ts b/site/scripts/eval-cut-chat.ts
index 3de7829d..4641bc79 100644
--- a/site/scripts/eval-cut-chat.ts
+++ b/site/scripts/eval-cut-chat.ts
@@ -16,7 +16,6 @@
* [--runs N] default 1; use 3+ for latency stats
* [--model <registryKey|rawId>] pin BOTH chat roles to one model (unrouted)
* [--simple-model <id>] [--complex-model <id>] override one routed role
- * [--gate-model <registryKey|rawId>]
* [--matrix] run every CANDIDATES row, compare
* [--no-judge] skip the voice/taste judge call
* [--enforce-budgets] exit 1 when a case breaches its budget
@@ -54,24 +53,21 @@ import {
// Model configs the --matrix mode compares. Add a row when trialing a new
// model id; keep "baseline" matching geminiModelRoles so the comparison always
// includes what production runs (the routed pair).
-const CANDIDATES: { label: string; simple: string; complex: string; gate: string }[] = [
+const CANDIDATES: { label: string; simple: string; complex: string }[] = [
{
label: "baseline",
simple: geminiModelRoles.chatSimple,
complex: geminiModelRoles.chat,
- gate: geminiModelRoles.fastDecision,
},
{
label: "all-flash",
simple: geminiModels.flash,
complex: geminiModels.flash,
- gate: geminiModelRoles.fastDecision,
},
{
label: "all-flashLite",
simple: geminiModels.flashLite,
complex: geminiModels.flashLite,
- gate: geminiModelRoles.fastDecision,
},
];
@@ -82,6 +78,8 @@ const argValue = (flag: string) => {
};
const BASE = argValue("--base") ?? "http://localhost:3000";
const ONLY = argValue("--only");
+/** Print each tool call's arguments and the reply on a failing run. */
+const TRACE = args.includes("--trace");
const BUCKET = argValue("--bucket") as Bucket | undefined;
const RUNS = Number(argValue("--runs") ?? 1);
const MATRIX = args.includes("--matrix");
@@ -124,7 +122,7 @@ async function runConfig(
const simpleId = idOf(cfg.simpleModel);
const complexId = idOf(cfg.complexModel);
const chatDesc = simpleId === complexId ? simpleId : `${simpleId} | complex→${complexId}`;
- console.log(`\n== ${label} chat=${chatDesc} gate=${idOf(cfg.gateModel)}`);
+ console.log(`\n== ${label} chat=${chatDesc} judge=typesafe`);
const caseReports: CaseReport[] = [];
for (const c of selected) {
const runs: RunReport[] = [];
@@ -146,15 +144,17 @@ async function runConfig(
);
for (const r of runs.filter((r) => !r.pass)) {
for (const n of r.error ? [r.error] : r.notes) console.log(` - ${n}`);
- if (r.trace.length > 0)
+ if (r.trace.length > 0) {
console.log(` tools: ${r.trace.map((t) => t.name).join(" → ")}`);
+ if (TRACE)
+ for (const t of r.trace) console.log(` ${t.name} ${JSON.stringify(t.args).slice(0, 200)}`);
+ }
}
for (const r of runs.filter((r) => r.judgeNote)) console.log(` judge: ${r.judgeNote}`);
}
return {
label,
chatModel: chatDesc,
- gateModel: idOf(cfg.gateModel),
buckets: buildBucketSummaries(caseReports),
cases: caseReports,
};
@@ -174,7 +174,7 @@ async function main() {
// production's routed pair.
const both = argValue("--model");
const defaults = defaultRunConfig(BASE);
- const judgeModel = NO_JUDGE ? null : defaults.judgeModel;
+ const judge = !NO_JUDGE;
const configs = MATRIX
? CANDIDATES.map((cand) => ({
label: cand.label,
@@ -182,8 +182,7 @@ async function main() {
base: BASE,
simpleModel: cand.simple,
complexModel: cand.complex,
- gateModel: cand.gate,
- judgeModel,
+ judge,
},
}))
: [
@@ -197,10 +196,7 @@ async function main() {
complexModel: resolveModel(
argValue("--complex-model") ?? both ?? defaults.complexModel
),
- gateModel: resolveModel(
- argValue("--gate-model") ?? defaults.gateModel
- ),
- judgeModel,
+ judge,
},
},
];
diff --git a/site/scripts/eval-cut-queue.ts b/site/scripts/eval-cut-queue.ts
index e837ebd4..4b83bb16 100644
--- a/site/scripts/eval-cut-queue.ts
+++ b/site/scripts/eval-cut-queue.ts
@@ -18,7 +18,6 @@
* [--base http://localhost:3000]
* [--only <case>[,<case>…]]
* [--runs N] default 1; 3+ shows a flaky verdict
- * [--gate-model <registryKey|rawId>]
* [--no-live] skip the live fold turn
*
* Auth is the dev bypass header (scripts only — never the app), so runs are
@@ -27,7 +26,7 @@
import type { UIMessage } from "ai";
import type { Message } from "@earendil-works/pi-ai";
-import { geminiModelRoleNames, geminiModels, resolveGeminiModel } from "../src/lib/inference/gemini-models";
+import { geminiModelRoleNames } from "../src/lib/inference/gemini-models";
import {
dropPiSession,
foldIntoCutChat,
@@ -49,17 +48,13 @@ const BASE = argValue("--base") ?? "http://localhost:3000";
const ONLY = argValue("--only")?.split(",").map((s) => s.trim());
const RUNS = Number(argValue("--runs") ?? 1);
const LIVE = !args.includes("--no-live");
-const GATE = (() => {
- const v = argValue("--gate-model");
- if (!v) return geminiModelRoleNames.fastDecision;
- return (geminiModels as Record<string, string>)[v] ?? v;
-})();
/** The eval's deps: the dev server as the inference proxy. The triage uses
- * the post function and the gate model alone. */
-const deps: CutAgentDeps = {
- post: (payload, signal) =>
- fetch(`${BASE}/api/inference/responses`, {
+ * the judge post alone. */
+const devPost =
+ (path: string) =>
+ (payload: Record<string, unknown>, signal?: AbortSignal) =>
+ fetch(`${BASE}${path}`, {
method: "POST",
headers: {
"Content-Type": "application/json",
@@ -68,9 +63,12 @@ const deps: CutAgentDeps = {
},
body: JSON.stringify(payload),
signal,
- }),
+ });
+const deps: CutAgentDeps = {
+ post: devPost("/api/inference/responses"),
+ judge: devPost("/api/inference/judge"),
execTool: async () => ({}),
- models: { simple: GATE, complex: GATE, gate: GATE },
+ models: { simple: geminiModelRoleNames.chatSimple, complex: geminiModelRoleNames.chat },
buildContext: () => ({}),
resolveRefs: async () => [],
};
@@ -112,7 +110,6 @@ async function runLiveFold(): Promise<string[]> {
models: {
simple: geminiModelRoleNames.chatSimple,
complex: geminiModelRoleNames.chat,
- gate: geminiModelRoleNames.fastDecision,
},
buildContext: () => EDITOR_STATE,
execTool: async (name, args) => {
@@ -175,7 +172,6 @@ async function runLiveQueueBack(): Promise<string[]> {
models: {
simple: geminiModelRoleNames.chatSimple,
complex: geminiModelRoleNames.chat,
- gate: geminiModelRoleNames.fastDecision,
},
buildContext: () => AUDIO_STATE,
execTool: async (name, args) => {
@@ -226,7 +222,7 @@ async function runLiveQueueBack(): Promise<string[]> {
async function main() {
const selected = queueCases.filter((c) => !ONLY || ONLY.includes(c.name));
if (selected.length === 0 && !LIVE) throw new Error(`No case named "${ONLY?.join(", ")}".`);
- console.log(`== queue triage gate=${resolveGeminiModel(GATE)} runs=${RUNS}`);
+ console.log(`== queue triage judge=typesafe runs=${RUNS}`);
let failed = 0;
const times: number[] = [];
for (const c of selected) {
diff --git a/site/scripts/lib/cut-eval/README.md b/site/scripts/lib/cut-eval/README.md
index fa3041ae..e9edd7ee 100644
--- a/site/scripts/lib/cut-eval/README.md
+++ b/site/scripts/lib/cut-eval/README.md
@@ -5,7 +5,7 @@ One command measures the chat assistant's behavior and speed together:
```
npm run eval:cut-chat -- [--runs N] [--only <case>] [--bucket chat|single-tool|multi-tool]
[--model <id>] [--simple-model <id>] [--complex-model <id>]
- [--gate-model <id>] [--matrix] [--enforce-budgets] [--out <path>]
+ [--matrix] [--enforce-budgets] [--out <path>]
```
It replays real composer turns against the live chat model through the dev
@@ -24,13 +24,11 @@ alongside so a fast-but-wrong config stays visible.
Every case carries a bucket: `chat` turns (greetings, questions — the gated
fast path), `single-tool` edits (one decisive call), and `multi-tool` edits
-(composed cuts). The eval runs the gate before round 1; production overlaps it
-with input assembly, which is negligible, so this is the same critical path
-the user perceives.
+(composed cuts).
-The default config is production's: the gate's three-way verdict (chat /
-simple / complex) withholds tools on chat turns and routes simple turns to the
-light chat model. `--model` pins both roles to one model for an unrouted
+The default config is production's: the turn judge's verdict (chat / simple /
+complex) withholds tools on chat turns, routes simple turns to the light chat
+model, and declares only the routed tool areas. `--model` pins both roles to one model for an unrouted
comparison; each run's report records the model every turn actually ran on
(`roundModel`). `--matrix` runs every row of `CANDIDATES` in
`eval-cut-chat.ts` and prints a comparison table. Model flags and candidate
@@ -43,13 +41,13 @@ always exits 1 when a case has zero passing runs.
## The queue triage eval
-`npm run eval:cut-queue -- [--runs N] [--only <case>] [--gate-model <id>]`
+`npm run eval:cut-queue -- [--runs N] [--only <case>]`
checks where a message sent mid-turn goes. Each case in `queueCases.ts` is a
running turn (its ask and the tools it has run so far, plus what other chats
of the project are doing) and the messages the user sent while it ran, each
with the place it must get: `fold` into the running turn, `spawn` a parallel
thread, or `queue` in the tray. The cases run the production triage call
-against the dev server on the gate model and print one line per case; a wrong
+against the dev server's judge route and print one line per case; a wrong
verdict prints the message, what it got and what it should have got. It exits
1 when a case never passes. Add a case whenever a real message landed in the
wrong place.
diff --git a/site/scripts/lib/cut-eval/cases.ts b/site/scripts/lib/cut-eval/cases.ts
index a0ab2ff5..e01b0931 100644
--- a/site/scripts/lib/cut-eval/cases.ts
+++ b/site/scripts/lib/cut-eval/cases.ts
@@ -529,6 +529,8 @@ export function cases(audio: { dataBase64: string; mimeType: string }): EvalCase
state: CROSSFADED_STATE,
simulate: () => (name, args) => {
if (name === "set_transition") return { id: "c1", transition: 0, style: "crossfade" };
+ // Clearing the bar is the other way to get rid of a crossfade.
+ if (name === "remove_transition") return { removed: args.transitionId ?? "tr-1" };
if (name !== "set_animation") return undefined;
if (args.clipId !== "c2" || args.which !== "in")
throw new Error(`set_animation ${args.clipId}/${args.which} — expected c2 "in"`);
@@ -743,12 +745,14 @@ export function cases(audio: { dataBase64: string; mimeType: string }): EvalCase
// asked for lands as nodes over the clip's footage.
name: "speed-ramp-from-description",
bucket: "single-tool",
+ // A clean timeline: the debris cases own the parked bars, and a turn
+ // that can see them is right to tidy them.
input: () => [
- userTurn("ease the san francisco clip into slow motion at the end", { state: PARKED_STATE }),
+ userTurn("ease the san francisco clip into slow motion at the end", { state: STYLED_STATE }),
],
reply: /slow|ramp|curve|ease/i,
requiredTools: ["set_speed_curve"],
- state: PARKED_STATE,
+ state: STYLED_STATE,
simulate: () => (name, args) => {
if (name === "set_speed_curve") {
return {
@@ -758,7 +762,8 @@ export function cases(audio: { dataBase64: string; mimeType: string }): EvalCase
{ at: 6, speed: 0.4 },
],
speed: 0.7,
- len: 8.6,
+ lenBefore: 3.7,
+ lenAfter: 3.7,
};
}
return undefined;
diff --git a/site/scripts/lib/cut-eval/graders.ts b/site/scripts/lib/cut-eval/graders.ts
index a9ea45b9..7f9d6196 100644
--- a/site/scripts/lib/cut-eval/graders.ts
+++ b/site/scripts/lib/cut-eval/graders.ts
@@ -4,6 +4,8 @@
* pass/fail, so a flaky verdict never flips a baseline. --no-judge skips it.
*/
+import { noul, score } from "../../../src/lib/inference/judge";
+
// Tag blocks that belong to the model's input. A reply carrying one is showing
// the user internal scaffolding.
const LEAKED_TAG = /<\/?(tools_ran|turn_ledger|editor_state|attached_assets)\b/i;
@@ -37,42 +39,78 @@ export function replyShapeNotes(
return notes;
}
-const JUDGE_PROMPT = `You grade one reply from a video editor's built-in AI assistant. The turn is labeled: an editing turn ran tools against the project; a conversational turn answered in words alone.
+/** The rubric, one judged line each. A positive line passes at or above the
+ * floor; a negative one (self-praise, upsell) passes below it. */
+const RUBRIC = {
+ concise_factual: {
+ q: noul(
+ "Is `reply` concise and factual — an editing turn says what changed in concrete terms, a conversational turn answers the question directly — with no headings and no filler?",
+ ),
+ wantYes: true,
+ },
+ self_praise: {
+ q: noul("Does `reply` praise its own output — stunning, professional, a masterpiece, incredible, or similar?"),
+ wantYes: false,
+ },
+ upsell: {
+ q: noul(
+ "Does `reply` pitch exports, subtitles, music, or other features beyond at most one short suggestion phrased as a statement?",
+ ),
+ wantYes: false,
+ },
+ decisive: {
+ q: noul(
+ "Does `reply` avoid asking the user a question the assistant could have decided itself? (Yes when it asks nothing, or asks only about a genuine ambiguity.)",
+ ),
+ wantYes: true,
+ },
+ failures_owned: {
+ q: noul("Does `reply` report plainly any work that errored or was skipped, or was there nothing to own?", {
+ true: "Every failure is stated as a failure, or nothing failed.",
+ false: "A failure or a skipped step is hidden, softened, or described as done.",
+ }),
+ wantYes: true,
+ },
+ approval_gate: {
+ q: noul(
+ "When the turn produced a plan whose rendering spends the user's credits, does `reply` stop and state the approval step? (Yes when no such plan exists.)",
+ ),
+ wantYes: true,
+ },
+} as const;
-Grade against this rubric:
-- Concise and factual: an editing turn says what changed in concrete terms; a conversational turn answers the question directly. No headings, no filler.
-- No self-praise: never calls its own output stunning, professional, a masterpiece, or similar.
-- No upsell: pitches no exports, subtitles, music, or other features beyond at most one short suggestion phrased as a statement.
-- Decisive: exercises judgment where the request left details open; a question appears only for genuine ambiguity.
-- Failures owned: work that errored or was skipped is reported plainly as such.
-- Approval gates respected: a plan whose rendering spends the user's credits correctly stops and states the approval step; that is required behavior.
+const LENGTH = score("How long is `reply` for what it does?", [
+ "Too short: a bare fragment that leaves the result unclear.",
+ "Right: one to three sentences that say what happened.",
+ "Long: a paragraph or more, restating, listing, or explaining beyond the ask.",
+]);
-A short, plain reply that fits its label passes. Reply with exactly one line: "pass" or "fail: <short reason>".`;
+const FLOOR = 0.5;
export interface JudgeVerdict {
pass: boolean;
note: string | null;
judgeMs: number;
+ /** Each rubric line's probability of yes, and the length score. */
+ scores?: Record<string, number>;
}
-/** One rubric call on the judge model. A transport or parse failure returns a
- * pass with a note, so the eval never blocks on the judge. */
+/** One typed judgment over the reply. A transport failure returns a pass
+ * with a note, so the eval never blocks on the judge. */
export async function judgeReply({
base,
- model,
userAsk,
reply,
didWork,
}: {
base: string;
- model: string;
userAsk: string;
reply: string;
didWork: boolean;
}): Promise<JudgeVerdict> {
const t0 = performance.now();
try {
- const res = await fetch(`${base}/api/inference/responses`, {
+ const res = await fetch(`${base}/api/inference/judge`, {
method: "POST",
headers: {
"Content-Type": "application/json",
@@ -80,29 +118,36 @@ export async function judgeReply({
"x-donkey-dev-auth-bypass": "1",
},
body: JSON.stringify({
- donkeyProvider: "gemini",
- model,
- instructions: JUDGE_PROMPT,
- input: [
- {
- role: "user",
- content: [
- {
- text: `Turn kind: ${didWork ? "editing (tools ran)" : "conversational (words alone)"}\n\nUser asked:\n${userAsk}\n\nAssistant replied:\n${reply}`,
- },
- ],
- },
- ],
+ state: {
+ turnKind: didWork ? "editing (tools ran against the project)" : "conversational (words alone)",
+ userAsk,
+ reply,
+ },
+ questions: {
+ ...Object.fromEntries(Object.entries(RUBRIC).map(([k, v]) => [k, v.q])),
+ length: LENGTH,
+ },
}),
});
const judgeMs = performance.now() - t0;
if (!res.ok) return { pass: true, note: `judge unavailable (${res.status})`, judgeMs };
- const body = (await res.json()) as { output_text?: string };
- const verdict = (body.output_text ?? "").trim();
- if (/^pass\b/i.test(verdict)) return { pass: true, note: null, judgeMs };
- if (/^fail\b/i.test(verdict))
- return { pass: false, note: verdict.replace(/^fail:?\s*/i, ""), judgeMs };
- return { pass: true, note: `judge returned "${verdict.slice(0, 80)}"`, judgeMs };
+ const body = (await res.json()) as {
+ answers: Record<string, { type: string; noul?: number; score?: number }>;
+ };
+ const scores: Record<string, number> = {};
+ const failing: string[] = [];
+ for (const [k, v] of Object.entries(RUBRIC)) {
+ const p = body.answers[k]?.noul ?? (v.wantYes ? 1 : 0);
+ scores[k] = p;
+ if (v.wantYes ? p < FLOOR : p >= FLOOR) failing.push(`${k.replace(/_/g, " ")} ${p.toFixed(2)}`);
+ }
+ const length = body.answers.length?.score;
+ if (typeof length === "number") {
+ scores.length = length;
+ if (length >= 1.5) failing.push(`long ${length.toFixed(1)}`);
+ if (length < 0.5) failing.push(`too short ${length.toFixed(1)}`);
+ }
+ return { pass: failing.length === 0, note: failing.length > 0 ? failing.join("; ") : null, judgeMs, scores };
} catch (err) {
return {
pass: true,
diff --git a/site/scripts/lib/cut-eval/harness.test.ts b/site/scripts/lib/cut-eval/harness.test.ts
index 8cce4ea7..b6f50506 100644
--- a/site/scripts/lib/cut-eval/harness.test.ts
+++ b/site/scripts/lib/cut-eval/harness.test.ts
@@ -2,24 +2,19 @@ import { describe, expect, test } from "bun:test";
import { readFileSync } from "node:fs";
import { defaultRunConfig } from "./harness";
import { providerCreditPricing } from "../../../src/lib/credits/provider-pricing";
-import {
- geminiModelRoleNames,
- geminiModelRoles,
- resolveGeminiModel,
-} from "../../../src/lib/inference/gemini-models";
+import { geminiModelRoleNames, resolveGeminiModel } from "../../../src/lib/inference/gemini-models";
describe("chat eval model selection", () => {
test("the eval sends the same role names as the editor and headless runner", () => {
const config = defaultRunConfig("http://localhost:3000");
expect(config.simpleModel).toBe(geminiModelRoleNames.chatSimple);
expect(config.complexModel).toBe(geminiModelRoleNames.chat);
- expect(config.gateModel).toBe(geminiModelRoleNames.fastDecision);
- expect(config.judgeModel).toBe(geminiModelRoles.fastDecision);
+ expect(config.judge).toBe(true);
});
test("every eval role resolves to a priced hosted model", () => {
const config = defaultRunConfig("http://localhost:3000");
- for (const model of [config.simpleModel, config.complexModel, config.gateModel, config.judgeModel!]) {
+ for (const model of [config.simpleModel, config.complexModel]) {
expect(providerCreditPricing("gemini", resolveGeminiModel(model))).toBeTruthy();
}
});
@@ -28,13 +23,11 @@ describe("chat eval model selection", () => {
const report = JSON.parse(readFileSync(
new URL("../../../../evals/cut-chat.latest-report.json", import.meta.url),
"utf8",
- )) as { configs: { chatModel: string; gateModel: string }[] };
+ )) as { configs: { chatModel: string }[] };
const config = defaultRunConfig("http://localhost:3000");
const simple = resolveGeminiModel(config.simpleModel);
const complex = resolveGeminiModel(config.complexModel);
const chatModel = simple === complex ? simple : `${simple} | complex→${complex}`;
- expect(report.configs.some((baseline) =>
- baseline.chatModel === chatModel && baseline.gateModel === resolveGeminiModel(config.gateModel),
- )).toBe(true);
+ expect(report.configs.some((baseline) => baseline.chatModel === chatModel)).toBe(true);
});
});
diff --git a/site/scripts/lib/cut-eval/harness.ts b/site/scripts/lib/cut-eval/harness.ts
index 4437ca98..6740b7c5 100644
--- a/site/scripts/lib/cut-eval/harness.ts
+++ b/site/scripts/lib/cut-eval/harness.ts
@@ -12,13 +12,9 @@
import type { UIMessage } from "ai";
import { AI_SKILLS } from "../../../src/cut/server/ai/catalog";
-import {
- geminiModelRoleNames,
- geminiModelRoles,
- resolveGeminiModel,
-} from "../../../src/lib/inference/gemini-models";
+import { geminiModelRoleNames, resolveGeminiModel } from "../../../src/lib/inference/gemini-models";
import { streamCutChat, dropPiSession, type CutAgentDeps } from "../../../src/cut/lib/pi/cutAgent";
-import type { TurnIntent } from "../../../src/cut/lib/turnIntent";
+import type { TurnIntent } from "../../../src/cut/lib/turnJudge";
import { EDITOR_STATE, SAFE_TOOLS, serveSafeTool } from "./fixtures";
import { judgeReply, replyShapeNotes } from "./graders";
import type { EvalCase } from "./cases";
@@ -30,10 +26,10 @@ export interface RunConfig {
simpleModel: string;
/** Model a complex-verdict (or fail-open) turn's rounds run on. */
complexModel: string;
- /** Model the gate/router call runs on. */
- gateModel: string;
- /** Model the voice/taste judge runs on; null skips the judge. */
- judgeModel: string | null;
+ /** Whether the reply judge scores each turn (report-only). */
+ judge: boolean;
+ /** Threshold overrides for the turn judge; unset runs the registry defaults. */
+ judgeSettings?: CutAgentDeps["judgeSettings"];
}
/** Production's config: the three-way gate routes between the two chat roles,
@@ -42,8 +38,7 @@ export const defaultRunConfig = (base: string): RunConfig => ({
base,
simpleModel: geminiModelRoleNames.chatSimple,
complexModel: geminiModelRoleNames.chat,
- gateModel: geminiModelRoleNames.fastDecision,
- judgeModel: geminiModelRoles.fastDecision,
+ judge: true,
});
export interface RoundTiming {
@@ -84,6 +79,8 @@ export interface TraceEntry {
/** Turn start → this call being served. */
atMs: number;
execMs: number;
+ /** The call's arguments, for a trace print. */
+ args: Record<string, unknown>;
}
export interface CaseResult {
@@ -95,6 +92,12 @@ export interface CaseResult {
/** The judge's complaint (or transport note). Report-only: never flips pass. */
judgeNote: string | null;
intent: TurnIntent;
+ /** The skill the judge attached, the areas it declared, and how many
+ * tools that came to; request_tools calls widened the catalog (a miss). */
+ skill: string | null;
+ areas: string[];
+ declaredTools: number;
+ routingMisses: string[];
/** The model the rounds ran on — differs per turn under a router config. */
roundModel: string;
timings: RunTimings;
@@ -123,9 +126,10 @@ export async function runCase(c: EvalCase, cfg: RunConfig): Promise<CaseResult>
let streamError: string | null = null;
let extensions = 0;
- const deps: CutAgentDeps = {
- post: (payload, signal) =>
- fetch(`${cfg.base}/api/inference/responses`, {
+ const devPost =
+ (path: string) =>
+ (payload: Record<string, unknown>, signal?: AbortSignal) =>
+ fetch(`${cfg.base}${path}`, {
method: "POST",
headers: {
"Content-Type": "application/json",
@@ -134,14 +138,20 @@ export async function runCase(c: EvalCase, cfg: RunConfig): Promise<CaseResult>
},
body: JSON.stringify(payload),
signal,
- }),
+ });
+ let route: { skill: string | null; areas: string[]; declaredTools: number } | null = null;
+ const routingMisses: string[] = [];
+ const deps: CutAgentDeps = {
+ post: devPost("/api/inference/responses"),
+ judge: devPost("/api/inference/judge"),
+ judgeSettings: cfg.judgeSettings,
execTool: async (name, args) => {
const atMs = performance.now() - turnStart;
const e0 = performance.now();
const record = (execMs: number) => {
toolMs += execMs;
const round = Math.max(0, rounds.length - 1);
- trace.push({ name, round, atMs, execMs });
+ trace.push({ name, round, atMs, execMs, args });
rounds[round]?.calls.push({ name, execMs });
};
try {
@@ -165,17 +175,19 @@ export async function runCase(c: EvalCase, cfg: RunConfig): Promise<CaseResult>
record(performance.now() - e0);
}
},
- models: { simple: cfg.simpleModel, complex: cfg.complexModel, gate: cfg.gateModel },
+ models: { simple: cfg.simpleModel, complex: cfg.complexModel },
buildContext: () => lastUser?.__state ?? c.state ?? EDITOR_STATE,
resolveRefs: async () => lastUser?.__wireParts ?? [],
debris: () => c.debris ?? [],
limits: c.limits,
hooks: {
- onGate: (i, ms, skipped) => {
+ onGate: (i, ms, skipped, r) => {
intent = i;
gateMs = ms;
gateSkipped = skipped;
+ route = { skill: r.skill, areas: r.areas, declaredTools: r.declaredTools };
},
+ onRequestTools: (areas) => routingMisses.push(...areas),
onRound: (ms, firstDeltaMs) => {
modelMs += ms;
rounds.push({ roundMs: ms, firstDeltaMs, calls: [] });
@@ -252,10 +264,9 @@ export async function runCase(c: EvalCase, cfg: RunConfig): Promise<CaseResult>
let judgeNote: string | null = null;
let judgeMs: number | undefined;
- if (cfg.judgeModel && reply) {
+ if (cfg.judge && reply) {
const verdict = await judgeReply({
base: cfg.base,
- model: cfg.judgeModel,
userAsk: askText,
reply,
didWork: mutated,
@@ -284,6 +295,10 @@ export async function runCase(c: EvalCase, cfg: RunConfig): Promise<CaseResult>
notes,
judgeNote,
intent,
+ skill: route?.skill ?? null,
+ areas: route?.areas ?? [],
+ declaredTools: route?.declaredTools ?? 0,
+ routingMisses,
roundModel: resolveGeminiModel(intent === "simple" ? cfg.simpleModel : cfg.complexModel),
timings,
};
diff --git a/site/scripts/lib/cut-eval/report.ts b/site/scripts/lib/cut-eval/report.ts
index 2ea34935..5ab5b724 100644
--- a/site/scripts/lib/cut-eval/report.ts
+++ b/site/scripts/lib/cut-eval/report.ts
@@ -19,6 +19,10 @@ export interface Agg {
export interface RunReport {
pass: boolean;
intent: string;
+ skill?: string | null;
+ areas?: string[];
+ declaredTools?: number;
+ routingMisses?: string[];
/** The model the rounds ran on — differs per turn under a router config. */
roundModel?: string;
notes: string[];
@@ -61,7 +65,6 @@ export interface BucketSummary {
export interface ConfigReport {
label: string;
chatModel: string;
- gateModel: string;
buckets: Partial<Record<Bucket, BucketSummary>>;
cases: CaseReport[];
}
@@ -98,6 +101,10 @@ export function toRunReport(result: CaseResult | null, error?: string): RunRepor
return {
pass: result.pass,
intent: result.intent,
+ skill: result.skill,
+ areas: result.areas,
+ declaredTools: result.declaredTools,
+ routingMisses: result.routingMisses,
roundModel: result.roundModel,
notes: result.notes,
judgeNote: result.judgeNote,
diff --git a/site/src/clients/chatgpt/server/catalog.ts b/site/src/clients/chatgpt/server/catalog.ts
index 09dc4c39..83d0c95c 100644
--- a/site/src/clients/chatgpt/server/catalog.ts
+++ b/site/src/clients/chatgpt/server/catalog.ts
@@ -1,6 +1,6 @@
import type { AiToolDef } from "@/cut/lib/aiToolDef";
import { ADOPT_COMMAND } from "@/cut/lib/commandBatch";
-import { AI_SKILL_INDEX, AI_SKILLS, PROJECT_TOOLS } from "@/cut/server/ai/catalog";
+import { AI_SKILL_INDEX, PROJECT_TOOLS, readSkill } from "@/cut/server/ai/catalog";
// The editing commands ChatGPT can run, drawn from the editor's own tool
// catalog so a command the assistant gains is a command ChatGPT gains. A few
@@ -76,4 +76,4 @@ export function describeCommands(names: string[]): { found: AiToolDef[]; unknown
}
export const SKILL_INDEX = AI_SKILL_INDEX;
-export const readSkill = (name: string): string | undefined => AI_SKILLS[name];
+export { readSkill };
diff --git a/site/src/cut/components/AiPanel.tsx b/site/src/cut/components/AiPanel.tsx
index 74b8e678..e5f544c5 100644
--- a/site/src/cut/components/AiPanel.tsx
+++ b/site/src/cut/components/AiPanel.tsx
@@ -49,7 +49,6 @@ import {
import { engineReady } from "@/cut/lib/api";
import { useCutCaps, useLocalCompute } from "@/cut/lib/backend/hooks";
import { localBackend } from "@/cut/lib/backend/local";
-import { buildAiContext } from "@/cut/lib/aiContext";
import { setAssetDragData } from "@/cut/lib/assetDrag";
import { useRefCopy } from "@/cut/lib/refCopy";
import { registerChatIntake } from "@/cut/lib/chatIntake";
@@ -95,7 +94,7 @@ import {
useSignedIn,
} from "@/cut/lib/generate";
import { useCreditsRecheck, useOutOfCredits } from "@/cut/lib/hosted";
-import { cutChatLive, dropPiSession, foldIntoCutChat, hydratePiSession, readPiSession, streamCutChat, triageQueuedMessages } from "@/cut/lib/pi/cutAgent";
+import { cutChatLive, dropPiSession, foldIntoCutChat, hydratePiSession, judgeEngineSkill, readPiSession, streamCutChat, triageQueuedMessages } from "@/cut/lib/pi/cutAgent";
import { toolProgress } from "@/cut/lib/queueTriage";
import { registerQueueSink } from "@/cut/lib/chatQueue";
import { productionDeps } from "@/cut/lib/pi/prodDeps";
@@ -1065,7 +1064,13 @@ function ChatSession({
// localBackend (read after the origin resolves), which carries the
// account scope the engine requires on every data route.
prepareSendMessagesRequest: async ({ messages }) => {
- await engineReady();
+ // The judge picks the turn's skill here, since the engine never
+ // calls hosted models; the engine carries it on its prompt.
+ const deps = productionDeps(projectId);
+ const [skill] = await Promise.all([
+ judgeEngineSkill(messages, deps.buildContext(), deps),
+ engineReady(),
+ ]);
return {
api: localBackend.url("/api/cut/ai/chat"),
body: {
@@ -1073,7 +1078,8 @@ function ChatSession({
runtime: chatRuntime(),
messages,
model: currentModel(),
- context: buildAiContext(),
+ context: deps.buildContext(),
+ skill,
providerSession: sessionFor(currentModel()),
},
};
diff --git a/site/src/cut/components/RequireSession.tsx b/site/src/cut/components/RequireSession.tsx
index 582b7910..ab54c796 100644
--- a/site/src/cut/components/RequireSession.tsx
+++ b/site/src/cut/components/RequireSession.tsx
@@ -1,5 +1,5 @@
"use client";
-import { bindChatRuntime } from "@/cut/lib/chatRuntime";
+import { bindChatRuntime, bindCutJudge } from "@/cut/lib/chatRuntime";
import { useEffect, type ReactNode } from "react";
import { useEnvironment } from "@/cut/lib/environment";
@@ -42,6 +42,7 @@ export function RequireSession({ children }: { children: ReactNode }) {
useEffect(() => {
if (!userId || !config.data) return;
bindChatRuntime(config.data.settings.chatRuntime);
+ bindCutJudge(config.data.settings.cutJudge);
reportExposures(config.data);
}, [userId, config.data]);
diff --git a/site/src/cut/lib/chatRuntime.ts b/site/src/cut/lib/chatRuntime.ts
index 136c77fa..c4780601 100644
--- a/site/src/cut/lib/chatRuntime.ts
+++ b/site/src/cut/lib/chatRuntime.ts
@@ -6,3 +6,10 @@ export function bindChatRuntime(value: unknown): void {
if (parsed.success) runtime = parsed.data;
}
export function chatRuntime(): Settings["chatRuntime"] { return runtime; }
+
+let judge: Settings["cutJudge"] = SETTINGS.cutJudge.default;
+export function bindCutJudge(value: unknown): void {
+ const parsed = SETTINGS.cutJudge.schema.safeParse(value);
+ if (parsed.success) judge = parsed.data;
+}
+export function cutJudge(): Settings["cutJudge"] { return judge; }
diff --git a/site/src/cut/lib/judge.ts b/site/src/cut/lib/judge.ts
new file mode 100644
index 00000000..0c725344
--- /dev/null
+++ b/site/src/cut/lib/judge.ts
@@ -0,0 +1,60 @@
+import type { PostFn } from "./pi/donkeyStream";
+import type { Entry, JudgeQuestion, JudgeResult } from "@/lib/inference/judge";
+
+export { choice, noul, score } from "@/lib/inference/judge";
+export type { ChoiceAnswer, Entry, JudgeQuestion, JudgeResult, NoulAnswer, ScoreAnswer } from "@/lib/inference/judge";
+
+// The page's and the worker's side of a judgment: post the state and the
+// questions to the judge route through the bound transport (session and
+// credits ride on it) and hand back the typed answers. Callers compose the
+// probabilities under their own thresholds; a failed call throws, and each
+// caller decides what its safe verdict is.
+
+export class JudgeError extends Error {
+ constructor(
+ message: string,
+ public readonly status: number,
+ ) {
+ super(message);
+ this.name = "JudgeError";
+ }
+}
+
+export async function askJudge<Q extends Record<string, JudgeQuestion>>(
+ post: PostFn,
+ state: Entry,
+ questions: Q,
+ signal?: AbortSignal,
+): Promise<JudgeResult<Q>> {
+ const res = await post({ state, questions }, signal);
+ if (!res.ok) {
+ let message = `judge ${res.status}`;
+ try {
+ const body = (await res.json()) as { message?: string; error?: string };
+ message = body.message ?? body.error ?? message;
+ } catch {}
+ throw new JudgeError(message, res.status);
+ }
+ return (await res.json()) as JudgeResult<Q>;
+}
+
+/** Split a per-item question map into requests of at most `size` questions,
+ * ask them in parallel, and merge the answers back under their keys. */
+export async function askJudgeChunked<Q extends Record<string, JudgeQuestion>>(
+ post: PostFn,
+ state: (keys: (keyof Q & string)[]) => Entry,
+ questions: Q,
+ size: number,
+ signal?: AbortSignal,
+): Promise<JudgeResult<Q>["answers"]> {
+ const keys = Object.keys(questions) as (keyof Q & string)[];
+ const chunks: (keyof Q & string)[][] = [];
+ for (let i = 0; i < keys.length; i += size) chunks.push(keys.slice(i, i + size));
+ const results = await Promise.all(
+ chunks.map((chunk) => {
+ const subset = Object.fromEntries(chunk.map((k) => [k, questions[k]])) as unknown as Q;
+ return askJudge(post, state(chunk), subset, signal);
+ }),
+ );
+ return Object.assign({}, ...results.map((r) => r.answers)) as JudgeResult<Q>["answers"];
+}
diff --git a/site/src/cut/lib/pi/cutAgent.ts b/site/src/cut/lib/pi/cutAgent.ts
index 88e1e6b3..faec647d 100644
--- a/site/src/cut/lib/pi/cutAgent.ts
+++ b/site/src/cut/lib/pi/cutAgent.ts
@@ -1,31 +1,50 @@
import type { UIMessage, UIMessageChunk } from "ai";
import { Agent, type AgentMessage } from "@earendil-works/pi-agent-core";
import type { Message, UserMessage } from "@earendil-works/pi-ai";
-import { AI_TOOLS, attachedAssetsBlock, systemPrompt } from "@/cut/server/ai/catalog";
+import type { AgentTool } from "@earendil-works/pi-agent-core";
+import {
+ areaTools,
+ attachedAssetsBlock,
+ CORE_TOOLS,
+ REQUEST_TOOLS_DEF,
+ skillRelevanceBlock,
+ systemPrompt,
+ TOOL_AREA_NAMES,
+} from "@/cut/server/ai/catalog";
import { isResumeMessage } from "../chatResume";
+import { cutJudge } from "../chatRuntime";
+import { askJudge } from "../judge";
import {
- parseQueueTriage,
- QUEUE_TRIAGE_SCHEMA,
- queueTriageInput,
- queueTriagePrompt,
+ placeQueuedRows,
+ queueTriageQuestions,
+ queueTriageState,
+ type QueueTriageAnswers,
type QueueVerdict,
type TriageAnchor,
type TriageRow,
} from "../queueTriage";
-import { parseTurnIntent, turnIntentInput, turnIntentPrompt, type TurnIntent } from "../turnIntent";
+import {
+ judgeTurnState,
+ routeTurn,
+ TURN_JUDGE_QUESTIONS,
+ type CutJudgeSettings,
+ type TurnIntent,
+ type TurnJudgeAnswers,
+ type TurnRoute,
+} from "../turnJudge";
import { enforceContextBudget } from "./contextBudget";
import { donkeyModel, type ChatThinkingLevel } from "./donkeyModel";
import { ledgerText, recordCall, type LedgerRecord } from "./mutationLedger";
import { makeDonkeyStream, type DonkeyToolDetails, type PostFn, type WireCarrier, type WirePart } from "./donkeyStream";
-import { toAgentTools, type ExecTool } from "./tools";
+import { toAgentTools, toToolResult, type ExecTool } from "./tools";
import { subscribeUiChunks } from "./uiChunks";
-// The chat turn runner on the pi agent harness. Each turn: the intent gate
-// classifies the newest message (chat verdicts withhold every tool, simple
-// verdicts downgrade the model and narrow the tool catalog to the areas the
-// verdict names), the Agent loops model calls and tool
-// executions against the live editor store, and the events stream out as the
-// UIMessageChunks AiPanel already consumes. The thread's LLM context is pi's
+// The chat turn runner on the pi agent harness. Each turn: one typed judgment
+// routes the newest message (a chat verdict withholds every tool, a simple
+// verdict keeps the light model; the turn declares the tool areas the judge
+// picked and carries the skill document it picked), the Agent loops model
+// calls and tool executions against the live editor store, and the events
+// stream out as the UIMessageChunks AiPanel already consumes. The thread's LLM context is pi's
// own message list, kept per thread in the session registry — past tool calls
// replay as structured toolCall/toolResult messages, so no bookkeeping rides
// as prose the model could mimic.
@@ -39,8 +58,15 @@ const MAX_EXTENSIONS = 3;
export interface CutAgentDeps {
post: PostFn;
+ /** The judge route: typed questions over a state, answered as
+ * probabilities. Routes the turn and triages the queue. */
+ judge: PostFn;
+ /** The judgment thresholds. The page binds them from the account config;
+ * the worker reads them from the global settings; unset falls back to the
+ * bound value (the registry default when nothing bound one). */
+ judgeSettings?: CutJudgeSettings;
execTool: ExecTool;
- models: { simple: string; complex: string; gate: string };
+ models: { simple: string; complex: string };
/** The fresh editor snapshot for the newest message. */
buildContext: () => unknown;
/** Attachment metadata resolved into wire media parts for the newest
@@ -58,7 +84,11 @@ export interface CutAgentDeps {
/** Timing instrumentation: the eval asserts on it, production logs it at
* debug level. */
hooks?: {
- onGate?: (intent: TurnIntent, ms: number, skipped: boolean) => void;
+ /** The turn's route landed: the verdict, the judge's wall time, whether
+ * the intent was decided on sight, and what the turn declares. */
+ onGate?: (intent: TurnIntent, ms: number, skipped: boolean, route: TurnRoute & { declaredTools: number }) => void;
+ /** The model widened its catalog mid-turn: a routing miss. */
+ onRequestTools?: (areas: string[]) => void;
/** One LLM round settled: wall time, and time to its first visible delta. */
onRound?: (ms: number, firstDeltaMs: number | null) => void;
/** The round budget auto-extended (n = extensions so far). */
@@ -190,11 +220,9 @@ export async function foldIntoCutChat({
});
}
-const TRIAGE_PROMPT = queueTriagePrompt();
-
-/** Where each mid-turn message goes: one light-model call over the running
- * ask and the rows. Fails closed — every row the call cannot place stays in
- * the queue. */
+/** Where each mid-turn message goes: one judgment over the running ask and
+ * the rows. Fails closed — every row the call cannot place stays in the
+ * queue. */
export async function triageQueuedMessages({
anchor,
waiting,
@@ -211,26 +239,21 @@ export async function triageQueuedMessages({
// Short ids on the wire; the verdict quotes them back.
const wire = rows.map((r, i) => ({ id: `m${i + 1}`, text: r.text }));
const byWire = new Map(wire.map((w, i) => [w.id, rows[i].id] as const));
- const closed = () => new Map<string, QueueVerdict>(rows.map((r) => [r.id, "queue"]));
+ let answers: QueueTriageAnswers | null = null;
try {
- const res = await deps.post(
- {
- donkeyProvider: "gemini",
- model: deps.models.gate,
- instructions: TRIAGE_PROMPT,
- input: queueTriageInput(anchor, waiting, wire),
- text: { format: { type: "json_schema", schema: QUEUE_TRIAGE_SCHEMA } },
- },
+ const result = await askJudge(
+ deps.judge,
+ queueTriageState(anchor, waiting, wire),
+ queueTriageQuestions(wire),
abortSignal
);
- if (!res.ok) return closed();
- const body = (await res.json()) as { output_text?: string };
- const out = closed();
- for (const [w, v] of parseQueueTriage(body.output_text, wire)) out.set(byWire.get(w)!, v);
- return out;
+ answers = result.answers as QueueTriageAnswers;
} catch {
- return closed();
+ answers = null;
}
+ const out = new Map<string, QueueVerdict>();
+ for (const [w, v] of placeQueuedRows(answers, wire)) out.set(byWire.get(w)!, v);
+ return out;
}
/** Every toolCall in the session answered. An aborted batch leaves calls with
@@ -449,11 +472,10 @@ function sessionFor(threadId: string, history: UIMessage[]): AgentMessage[] {
return [...stored, ...legacySession(history.slice(firstMissing))];
}
-const GATE_PROMPT = turnIntentPrompt();
-
-/** A message that runs routed from the start, with no gate call and no
- * speculative round: one carrying attachments, or the continuation of a turn
- * the page lost, which picks up whatever tools that turn was using. */
+/** A message that runs on the full model from the start, with no speculative
+ * round: one carrying attachments, or the continuation of a turn the page
+ * lost, which picks up whatever tools that turn was using. The judge still
+ * picks its skill and areas. */
function complexOnSight(lastUser: UIMessage | undefined): boolean {
if (!lastUser) return false;
if (isResumeMessage(lastUser)) return true;
@@ -461,39 +483,73 @@ function complexOnSight(lastUser: UIMessage | undefined): boolean {
return Array.isArray(attached) && attached.length > 0;
}
-/** The turn's gate and router (ported from the legacy loop): judge the newest
- * message; "chat" withholds every tool, "simple" downgrades the model. A
- * message complex on sight skips the call. Fails open. */
-async function classifyTurnIntent(
+export interface TurnVerdict {
+ route: TurnRoute;
+ /** The intent was decided on sight (attachments, a resumed turn). */
+ skipped: boolean;
+}
+
+const sleep = (ms: number) => new Promise<void>((r) => setTimeout(r, ms));
+
+/** The turn's route: one judge call over the newest message, the recent
+ * turns, and the editor slice, composed under the thresholds. A message
+ * complex on sight keeps its intent and takes the skill and areas. Fails
+ * open — a call that fails runs the full model with every area and no
+ * skill. */
+export async function judgeTurn(
messages: UIMessage[],
+ context: unknown,
deps: CutAgentDeps,
abortSignal?: AbortSignal
-): Promise<{ intent: TurnIntent; skipped: boolean }> {
+): Promise<TurnVerdict> {
+ const settings = deps.judgeSettings ?? cutJudge();
const lastUser = messages.findLast((m) => m.role === "user");
- if (complexOnSight(lastUser)) return { intent: "complex", skipped: true };
- const turns = messages.map((m) => ({
- role: m.role === "user" ? ("user" as const) : ("assistant" as const),
- text: m.parts
- .map((p) => (p.type === "text" ? p.text : ""))
- .join("")
- .trim(),
- }));
+ const onSight = complexOnSight(lastUser);
+ if (onSight && !settings.skillSuggestion && !settings.toolRouting)
+ return { route: routeTurn(null, settings), skipped: true };
+ let route: TurnRoute;
try {
- const res = await deps.post(
- {
- donkeyProvider: "gemini",
- model: deps.models.gate,
- instructions: GATE_PROMPT,
- input: turnIntentInput(turns),
- },
+ const { answers } = await askJudge(
+ deps.judge,
+ judgeTurnState(messages, context),
+ TURN_JUDGE_QUESTIONS,
abortSignal
);
- if (!res.ok) return { intent: "complex", skipped: false };
- const body = (await res.json()) as { output_text?: string };
- return { intent: parseTurnIntent(body.output_text), skipped: false };
+ route = routeTurn(answers as unknown as TurnJudgeAnswers, settings);
} catch {
- return { intent: "complex", skipped: false };
+ route = routeTurn(null, settings);
}
+ if (onSight) route.intent = "complex";
+ return { route, skipped: onSight };
+}
+
+/** The skill the judge attaches to a turn the engine's own chat runs: the
+ * page judges before the send and the engine carries the block on its
+ * prompt. Null when no skill fits; undefined when the judge could not be
+ * asked. */
+export async function judgeEngineSkill(
+ messages: UIMessage[],
+ context: unknown,
+ deps: CutAgentDeps
+): Promise<string | null | undefined> {
+ const settings = deps.judgeSettings ?? cutJudge();
+ if (!settings.skillSuggestion) return undefined;
+ try {
+ const { answers } = await askJudge(deps.judge, judgeTurnState(messages, context), TURN_JUDGE_QUESTIONS);
+ return routeTurn(answers as unknown as TurnJudgeAnswers, settings).skill;
+ } catch {
+ return undefined;
+ }
+}
+
+/** The declared catalog for a run: the core on every work turn, the routed
+ * areas' tools, and the escape hatch whenever something was left out. */
+function toolsFor(areas: readonly string[], execTool: ExecTool, requestTools: AgentTool): AgentTool[] {
+ const full = areas.length === TOOL_AREA_NAMES.length;
+ return [
+ ...toAgentTools([...CORE_TOOLS, ...areaTools(areas)], execTool),
+ ...(full ? [] : [requestTools]),
+ ];
}
/** The newest composer message as the agent's prompt: text plus the
@@ -501,7 +557,8 @@ async function classifyTurnIntent(
* payloads as wire parts. */
async function buildPrompt(
lastUser: UIMessage | undefined,
- deps: CutAgentDeps
+ deps: CutAgentDeps,
+ context: unknown = deps.buildContext()
): Promise<UserMessage & WireCarrier> {
let text = (lastUser?.parts ?? [])
.map((p) => (p.type === "text" ? p.text : ""))
@@ -515,7 +572,7 @@ async function buildPrompt(
wireParts.push(...(await deps.resolveRefs(meta)));
} catch {}
}
- text += `\n\n<editor_state>\n${JSON.stringify(deps.buildContext())}\n</editor_state>`;
+ text += `\n\n<editor_state>\n${JSON.stringify(context)}\n</editor_state>`;
return { role: "user", content: [{ type: "text", text }], timestamp: Date.now(), wireParts };
}
@@ -544,11 +601,19 @@ export function streamCutChat({
try {
const gateStart = performance.now();
const lastUser = messages.findLast((m) => m.role === "user");
- const promptPromise = buildPrompt(lastUser, deps);
- const verdictPromise = classifyTurnIntent(messages, deps, abortSignal);
- void verdictPromise.then(({ intent, skipped }) =>
- deps.hooks?.onGate?.(intent, performance.now() - gateStart, skipped)
- );
+ const context = deps.buildContext();
+ const promptPromise = buildPrompt(lastUser, deps, context);
+ const settings = deps.judgeSettings ?? cutJudge();
+ const verdictPromise = judgeTurn(messages, context, deps, abortSignal);
+ // The route once it has landed, read by every round after.
+ let settled: TurnRoute | null = null;
+ void verdictPromise.then(({ route, skipped }) => {
+ settled = route;
+ deps.hooks?.onGate?.(route.intent, performance.now() - gateStart, skipped, {
+ ...route,
+ declaredTools: route.intent === "chat" ? 0 : CORE_TOOLS.length + areaTools(route.areas).length,
+ });
+ });
const roundBudget = deps.limits?.roundBudget ?? ROUND_BUDGET;
const maxExtensions = deps.limits?.maxExtensions ?? MAX_EXTENSIONS;
@@ -581,12 +646,53 @@ export function streamCutChat({
// Everything this turn ran, harvested off the tool results in code.
const records: LedgerRecord[] = [];
+ // The catalog this run declares. The speculative run waits a bounded
+ // moment for the route; a route that has landed narrows the tools
+ // to its areas, and one still in flight leaves the full catalog on
+ // this run (the model already sees those tools, so the run never
+ // narrows afterwards). The escape hatch widens between rounds.
+ const routed =
+ settled ?? (await Promise.race([verdictPromise.then((v) => v.route), sleep(settings.judgeWaitMs).then(() => null)]));
+ const declared = new Set<string>(routed?.areas ?? TOOL_AREA_NAMES);
+ let widened: string[] | null = null;
+ const requestTools: AgentTool = {
+ name: REQUEST_TOOLS_DEF.name,
+ label: REQUEST_TOOLS_DEF.name,
+ description: REQUEST_TOOLS_DEF.description,
+ parameters: REQUEST_TOOLS_DEF.inputSchema as never,
+ execute: async (_id, params) => {
+ const asked = ((params as { areas?: unknown }).areas ?? []) as unknown[];
+ const added = asked.filter((a): a is string => typeof a === "string" && TOOL_AREA_NAMES.includes(a) && !declared.has(a));
+ for (const a of added) declared.add(a);
+ if (added.length > 0) {
+ widened = [...(widened ?? []), ...added];
+ deps.hooks?.onRequestTools?.(added);
+ }
+ const tools = areaTools(added).map((t) => t.name);
+ return toToolResult(REQUEST_TOOLS_DEF.name, {
+ added,
+ tools,
+ note: added.length > 0 ? "Available from your next step." : "Nothing to add: those areas are already declared.",
+ });
+ },
+ };
+ const tools = withTools ? toolsFor([...declared], deps.execTool, requestTools) : [];
+
const agent = new Agent({
initialState: {
systemPrompt: systemPrompt(),
model: donkeyModel(roundModel, thinkingLevel),
messages: sessionFor(threadId, messages.filter((m) => m !== lastUser)),
- tools: withTools ? toAgentTools(AI_TOOLS, deps.execTool) : [],
+ tools,
+ },
+ // A widened catalog reaches the next round through the loop's own
+ // context, and the agent's state, so a follow-up sees it too.
+ prepareNextTurnWithContext: (ctx) => {
+ if (!widened) return undefined;
+ widened = null;
+ const next = toolsFor([...declared], deps.execTool, requestTools);
+ agent.state.tools = next;
+ return { context: { ...ctx.context, tools: next } };
},
streamFn: makeDonkeyStream({
post: deps.post,
@@ -598,9 +704,13 @@ export function streamCutChat({
// The turn ledger rides every call as an ephemeral tail message —
// the reply-writing call always sees the current record, and the
// stored session never carries it.
+ // The skill the judge attached rides the same way, from the
+ // first round it is known for.
+ const skill = withTools && settings.skillSuggestion && settled ? skillRelevanceBlock(settled.skill) : null;
const ledger = ledgerText(records, deps.debris?.() ?? []);
- if (!ledger) return out;
- return [...out, { role: "user", content: ledger, timestamp: 0 }];
+ const tail = [skill, ledger].filter((t): t is string => !!t).join("\n\n");
+ if (!tail) return out;
+ return [...out, { role: "user", content: tail, timestamp: 0 }];
},
toolExecution: "sequential",
beforeToolCall: async ({ toolCall }) => {
@@ -746,8 +856,9 @@ export function streamCutChat({
}
};
- // Speculative first round: the turn goes out immediately on the light
- // model with the full catalog while the gate classifies in parallel.
+ // Speculative first round: the turn goes out on the light model while
+ // the judge routes in parallel, waiting on it only a bounded moment for
+ // the narrowed catalog.
// Tool execution waits on the verdict and UI chunks buffer until it
// lands, so a chat or complex verdict aborts the run with nothing
// shown and nothing executed, and the turn restarts routed — while a
@@ -763,8 +874,8 @@ export function streamCutChat({
else if (mode === "buffering") buffer.push(chunk);
};
let speculating: Agent | null = null;
- const watcher = verdictPromise.then(({ intent }) => {
- if (intent === "simple") {
+ const watcher = verdictPromise.then(({ route }) => {
+ if (route.intent === "simple") {
mode = "live";
for (const c of buffer.splice(0)) emit(c);
return true;
@@ -778,7 +889,7 @@ export function streamCutChat({
roundModel: deps.models.simple,
withTools: true,
send,
- executionGate: async () => (await verdictPromise).intent === "simple",
+ executionGate: async () => (await verdictPromise).route.intent === "simple",
onAgent: (a) => {
speculating = a;
},
@@ -789,7 +900,7 @@ export function streamCutChat({
}
}
if (!kept && !abortSignal?.aborted) {
- const { intent } = await verdictPromise;
+ const { intent } = (await verdictPromise).route;
finalize(
await runAgentTurn({
roundModel: intent === "simple" ? deps.models.simple : model,
diff --git a/site/src/cut/lib/pi/mutationLedger.ts b/site/src/cut/lib/pi/mutationLedger.ts
index b330e727..28b3807b 100644
--- a/site/src/cut/lib/pi/mutationLedger.ts
+++ b/site/src/cut/lib/pi/mutationLedger.ts
@@ -10,6 +10,7 @@ const READ_ONLY = new Set([
"get_state",
"list_skills",
"read_skill",
+ "request_tools",
"capture_frame",
"watch_video",
"note_source",
diff --git a/site/src/cut/lib/pi/prodDeps.ts b/site/src/cut/lib/pi/prodDeps.ts
index 03896836..35f0e9cb 100644
--- a/site/src/cut/lib/pi/prodDeps.ts
+++ b/site/src/cut/lib/pi/prodDeps.ts
@@ -1,7 +1,7 @@
"use client";
import { geminiModelRoleNames } from "@/lib/inference/gemini-models";
-import { AI_SKILL_INDEX, AI_SKILLS } from "@/cut/server/ai/catalog";
+import { AI_SKILL_INDEX, readSkill } from "@/cut/server/ai/catalog";
import { buildAiContext } from "../aiContext";
import { runAiTool } from "../aiTools";
import { runProjectChatTool, withChatProject } from "../projectChatTools";
@@ -23,10 +23,11 @@ export function productionDeps(projectId?: string, signal?: AbortSignal): CutAge
const context = buildAiContext();
return {
post: (payload, signal) => hostedPost("/api/inference/responses", payload, signal),
+ judge: (payload, signal) => hostedPost("/api/inference/judge", payload, signal),
execTool: async (name, args) => {
if (name === "list_skills") return { skills: AI_SKILL_INDEX };
if (name === "read_skill") {
- const doc = AI_SKILLS[String(args.name ?? "")];
+ const doc = readSkill(String(args.name ?? ""));
if (!doc) throw new Error(`No such skill. Available: ${AI_SKILL_INDEX.join(", ")}`);
return doc;
}
@@ -35,7 +36,6 @@ export function productionDeps(projectId?: string, signal?: AbortSignal): CutAge
models: {
simple: geminiModelRoleNames.chatSimple,
complex: geminiModelRoleNames.chat,
- gate: geminiModelRoleNames.fastDecision,
},
buildContext: () => context,
resolveRefs: async (meta) => {
diff --git a/site/src/cut/lib/pi/serverDeps.ts b/site/src/cut/lib/pi/serverDeps.ts
index 894a26b8..fcfd35dc 100644
--- a/site/src/cut/lib/pi/serverDeps.ts
+++ b/site/src/cut/lib/pi/serverDeps.ts
@@ -1,6 +1,6 @@
import { geminiModelRoleNames } from "@/lib/inference/gemini-models";
import { UI_TOOLS } from "@/cut/lib/projectCommands";
-import { AI_SKILL_INDEX, AI_SKILLS } from "@/cut/server/ai/catalog";
+import { AI_SKILL_INDEX, readSkill } from "@/cut/server/ai/catalog";
import { projectOperation } from "../projectOperation";
import { useEditor } from "../store";
import { buildAiContext } from "../aiContext";
@@ -25,10 +25,11 @@ export function headlessDeps(session: HeadlessSession): CutAgentDeps {
bindHeadlessSession(session);
return {
post: (payload, signal) => hostedPost("/api/inference/responses", payload, signal),
+ judge: (payload, signal) => hostedPost("/api/inference/judge", payload, signal),
execTool: async (name, args) => {
if (name === "list_skills") return { skills: AI_SKILL_INDEX };
if (name === "read_skill") {
- const doc = AI_SKILLS[String(args.name ?? "")];
+ const doc = readSkill(String(args.name ?? ""));
if (!doc) throw new Error(`No such skill. Available: ${AI_SKILL_INDEX.join(", ")}`);
return doc;
}
@@ -49,7 +50,6 @@ export function headlessDeps(session: HeadlessSession): CutAgentDeps {
models: {
simple: geminiModelRoleNames.chatSimple,
complex: geminiModelRoleNames.chat,
- gate: geminiModelRoleNames.fastDecision,
},
buildContext: () => buildAiContext(),
resolveRefs: async (meta) => {
diff --git a/site/src/cut/lib/queueTriage.test.ts b/site/src/cut/lib/queueTriage.test.ts
index 980453cb..8acddf46 100644
--- a/site/src/cut/lib/queueTriage.test.ts
+++ b/site/src/cut/lib/queueTriage.test.ts
@@ -1,60 +1,80 @@
import { describe, expect, test } from "bun:test";
-import { parseQueueTriage, queueTriageInput, toolProgress } from "./queueTriage";
+import { placeQueuedRows, queueTriageQuestions, queueTriageState, toolProgress, type QueueTriageAnswers } from "./queueTriage";
const rows = [
{ id: "m1", text: "also make the title blue" },
{ id: "m2", text: "then export it" },
];
-describe("parseQueueTriage", () => {
+const place = (choice: "fold" | "spawn" | "queue", p = 0.9) => ({
+ type: "choice" as const,
+ choice,
+ probabilities: { fold: choice === "fold" ? p : (1 - p) / 2, spawn: choice === "spawn" ? p : (1 - p) / 2, queue: choice === "queue" ? p : (1 - p) / 2 },
+ confidence: p,
+});
+const collides = (p: number) => ({ type: "noul" as const, noul: p });
+
+describe("placeQueuedRows", () => {
test("reads one verdict per row", () => {
- const out = parseQueueTriage(
- JSON.stringify({ verdicts: [{ id: "m1", verdict: "fold" }, { id: "m2", verdict: "queue" }] }),
+ const out = placeQueuedRows(
+ { "place::m1": place("fold"), "collides::m1": collides(0.1), "place::m2": place("queue"), "collides::m2": collides(0.1) } as QueueTriageAnswers,
rows
);
expect(out.get("m1")).toBe("fold");
expect(out.get("m2")).toBe("queue");
});
- test("a row the reply skips, misnames, or answers with another word stays queued", () => {
- const out = parseQueueTriage(
- JSON.stringify({ verdicts: [{ id: "m9", verdict: "spawn" }, { id: "m2", verdict: "later" }] }),
+ test("a collision forbids a spawn; a fold on colliding work stands", () => {
+ const out = placeQueuedRows(
+ { "place::m1": place("spawn"), "collides::m1": collides(0.8), "place::m2": place("fold"), "collides::m2": collides(0.8) } as QueueTriageAnswers,
rows
);
expect(out.get("m1")).toBe("queue");
- expect(out.get("m2")).toBe("queue");
+ expect(out.get("m2")).toBe("fold");
+ });
+
+ test("an uncertain placement queues", () => {
+ const out = placeQueuedRows({ "place::m1": place("spawn", 0.3), "collides::m1": collides(0) } as QueueTriageAnswers, rows);
+ expect(out.get("m1")).toBe("queue");
});
- test("a garbled or empty reply queues every row", () => {
- expect([...parseQueueTriage("fold", rows).values()]).toEqual(["queue", "queue"]);
- expect([...parseQueueTriage(undefined, rows).values()]).toEqual(["queue", "queue"]);
- expect([...parseQueueTriage("{}", rows).values()]).toEqual(["queue", "queue"]);
+ test("a row the answers skip, and a failed call, queue", () => {
+ expect([...placeQueuedRows({ "place::m2": place("fold") } as QueueTriageAnswers, rows).values()]).toEqual(["queue", "fold"]);
+ expect([...placeQueuedRows(null, rows).values()]).toEqual(["queue", "queue"]);
});
});
-describe("queueTriageInput", () => {
+describe("queueTriageQuestions", () => {
+ test("asks a placement and a collision per row, keyed by the row's id", () => {
+ const q = queueTriageQuestions(rows);
+ expect(Object.keys(q)).toEqual(["place::m1", "collides::m1", "place::m2", "collides::m2"]);
+ expect(q["place::m1"].type).toBe("choice");
+ expect(q["collides::m2"].type).toBe("noul");
+ });
+});
+
+describe("queueTriageState", () => {
test("lays out the running ask, its progress, other chats, the waiting rows and the rows to place", () => {
- const [turn] = queueTriageInput(
+ const state = queueTriageState(
{ ask: "cut the silences", progress: ["detect_silence", "split_clip (running)"], elsewhere: ["add captions"] },
[{ id: "w1", text: "what's the total length?" }],
rows
- );
- const text = (turn.content as { text: string }[])[0].text;
- expect(text).toContain("Running ask:\ncut the silences");
- expect(text).toContain("Tools run so far: detect_silence, split_clip (running)");
- expect(text).toContain("Running in other chats:\n- add captions");
- expect(text).toContain("Already waiting in the queue:\n- what's the total length?");
- expect(text).toContain("[m1] also make the title blue\n[m2] then export it");
+ ) as Record<string, unknown>;
+ expect(state.running).toEqual({
+ ask: "cut the silences",
+ toolsRunSoFar: ["detect_silence", "split_clip (running)"],
+ inOtherChats: ["add captions"],
+ });
+ expect(state.alreadyWaiting).toEqual(["what's the total length?"]);
+ expect(state.messages).toEqual([
+ { id: "m1", text: "also make the title blue" },
+ { id: "m2", text: "then export it" },
+ ]);
});
test("caps a pasted wall of text", () => {
- const [turn] = queueTriageInput(
- { ask: "x".repeat(5000), progress: [], elsewhere: [] },
- [],
- [{ id: "m1", text: "y".repeat(5000) }]
- );
- const text = (turn.content as { text: string }[])[0].text;
- expect(text.length).toBeLessThan(4400);
+ const state = queueTriageState({ ask: "x".repeat(5000), progress: [], elsewhere: [] }, [], [{ id: "m1", text: "y".repeat(5000) }]);
+ expect(JSON.stringify(state).length).toBeLessThan(4400);
});
});
diff --git a/site/src/cut/lib/queueTriage.ts b/site/src/cut/lib/queueTriage.ts
index c3ea9ab8..ded815fe 100644
--- a/site/src/cut/lib/queueTriage.ts
+++ b/site/src/cut/lib/queueTriage.ts
@@ -1,10 +1,12 @@
-// Where a message sent mid-turn goes. A fast structured model call reads the
-// running ask, what it has done so far, and the message, and names one of
-// three places: fold (the running turn takes it in stride), spawn (a second
-// thread runs it in parallel because the work cannot collide), or queue (it
-// waits in the tray for the finished result). The page acts deterministically
-// on the verdict, and a garbled or failed call reads as queue — the tray is
-// the safe place, and it is exactly what the queue did before triage existed.
+import { choice, noul, type ChoiceAnswer, type Entry, type NoulAnswer } from "./judge";
+
+// Where a message sent mid-turn goes. One typed judgment reads the running
+// ask, what it has done so far, and each message, and names one of three
+// places: fold (the running turn takes it in stride), spawn (a second thread
+// runs it in parallel because the work cannot collide), or queue (it waits in
+// the tray for the finished result). The page acts deterministically on the
+// verdict, and a failed call reads as queue — the tray is the safe place, and
+// it is exactly what the queue did before triage existed.
export type QueueVerdict = "fold" | "queue" | "spawn";
@@ -22,80 +24,67 @@ export interface TriageRow {
text: string;
}
-/** The gate's instructions. */
-export function queueTriagePrompt(): string {
- return `You triage messages a user sends to the AI editor built into a video editor while it is already working on an earlier ask. The running ask, the tools it has run so far, asks running in other chats of the same project, and any messages already waiting in the queue are given as context. Judge each message listed under "triage" on its own and place it:
-
-fold — the working agent takes it in the same pass. The message refines, extends, narrows, corrects, or adds to the running ask ("also…", "and make it…", "actually…", "not that one", "bigger"), asks about the work in progress, or is another question when the running ask is a question. A further edit to the same items, the same job, or the same stretch of the timeline folds too: the agent already holds that context and does the extra piece after what is in flight.
-
-spawn — a second agent runs it in parallel right now, because it is a different job on things nothing running touches: a question about the footage, a source, or the project that changes nothing; or an edit to other items than the running work's — the look, color, or volume of a different clip, a caption style, the music level, a project setting — whose timing the running work cannot move and that does not need its result. Being in the same video is no collision; the same items or the same job is.
+const TEXT_CAP = 2000;
-queue — it waits for the running ask to finish and for a fresh view of the project: it needs the finished result (export, render, share, publish, a thumbnail or summary of the final cut), or it is sequenced after the running work ("then…", "after that…", "once it's done…", "when you're finished…"). Anything unclear.
+/** The judge's state: the running work and the rows to place, each under
+ * the short id its verdict is keyed by. */
+export function queueTriageState(anchor: TriageAnchor, waiting: TriageRow[], rows: TriageRow[]): Entry {
+ const clip = (s: string) => s.slice(0, TEXT_CAP);
+ return {
+ running: {
+ ask: clip(anchor.ask),
+ toolsRunSoFar: anchor.progress,
+ inOtherChats: anchor.elsewhere.map(clip),
+ },
+ alreadyWaiting: waiting.map((r) => clip(r.text)),
+ messages: rows.map((r) => ({ id: r.id, text: clip(r.text) })),
+ };
+}
-Collisions on the timeline: while any running work — in this chat or another — cuts, trims, moves, splits, deletes, or reorders clips, the times of everything after the cuts shift. A message that places or times something on the timeline (a sticker at 12 seconds, a title at the end, a sound effect on a moment) collides with it: fold it when it belongs with this chat's running job, and queue it in every other case — never spawn it.
+const PLACE_CRITERIA = {
+ fold: "The working agent takes it in the same pass: the message refines, extends, narrows, corrects, or adds to the running ask (\"also…\", \"and make it…\", \"actually…\", \"not that one\", \"bigger\"), asks about the work in progress, or is another question when the running ask is a question. A further edit to the same items, the same job, or the same stretch of the timeline folds too.",
+ spawn:
+ "A second agent runs it in parallel right now, because it is a different job on things nothing running touches: a question about the footage, a source, or the project that changes nothing; or an edit to other items than the running work's (the look, color, or volume of a different clip, a caption style, the music level, a project setting) whose timing the running work cannot move and that does not need its result. Being in the same video is no collision; the same items or the same job is.",
+ queue:
+ "It waits for the running ask to finish and for a fresh view of the project: it needs the finished result (export, render, share, publish, a thumbnail or summary of the final cut), or it is sequenced after the running work (\"then…\", \"after that…\", \"once it's done…\"). Anything unclear.",
+} as const;
-When unsure between fold and queue, queue. When unsure between spawn and queue, queue.`;
+/** One placement and one collision question per row. */
+export function queueTriageQuestions(rows: TriageRow[]) {
+ const out: Record<string, ReturnType<typeof choice<typeof PLACE_CRITERIA>> | ReturnType<typeof noul>> = {};
+ for (const r of rows) {
+ out[`place::${r.id}`] = choice(
+ `The user sent \`messages\` (id "${r.id}") to the AI editor built into a video editor while it was already working on \`running.ask\`; \`running.toolsRunSoFar\` is what that work has done, \`running.inOtherChats\` lists asks running in other chats of the same project, and \`alreadyWaiting\` the messages queued before it. Where does message "${r.id}" go?`,
+ PLACE_CRITERIA,
+ );
+ out[`collides::${r.id}`] = noul(
+ `Does message "${r.id}" place or time something on the timeline (a sticker at 12 seconds, a title at the end, a sound effect on a moment) while running work in this chat or another cuts, trims, moves, splits, deletes, or reorders clips, so the times it names would shift under it?`,
+ );
+ }
+ return out;
}
-/** The gate's input, as plain text turns. Rows carry short ids the verdict
- * quotes back, so a long message can never be mistaken for another. */
-export function queueTriageInput(
- anchor: TriageAnchor,
- waiting: TriageRow[],
- rows: TriageRow[]
-): Record<string, unknown>[] {
- const clip = (s: string) => s.slice(0, 2000);
- const lines: string[] = [`Running ask:\n${clip(anchor.ask) || "(none)"}`];
- lines.push(
- `Tools run so far: ${anchor.progress.length > 0 ? anchor.progress.join(", ") : "(none yet)"}`
- );
- if (anchor.elsewhere.length > 0)
- lines.push(`Running in other chats:\n${anchor.elsewhere.map((a) => `- ${clip(a)}`).join("\n")}`);
- if (waiting.length > 0)
- lines.push(`Already waiting in the queue:\n${waiting.map((r) => `- ${clip(r.text)}`).join("\n")}`);
- lines.push(`Triage:\n${rows.map((r) => `[${r.id}] ${clip(r.text)}`).join("\n")}`);
- return [{ role: "user", content: [{ text: lines.join("\n\n") }] }];
-}
+export type QueueTriageAnswers = Record<string, ChoiceAnswer<typeof PLACE_CRITERIA> | NoulAnswer>;
-/** The verdict's shape: one entry per triaged row. */
-export const QUEUE_TRIAGE_SCHEMA = {
- type: "object",
- properties: {
- verdicts: {
- type: "array",
- items: {
- type: "object",
- properties: {
- id: { type: "string" },
- verdict: { type: "string", enum: ["fold", "queue", "spawn"] },
- },
- required: ["id", "verdict"],
- },
- },
- },
- required: ["verdicts"],
-} as const;
+/** The probability a placement needs before the code commits to it; below
+ * it the row waits. The chosen option's own probability is the measure, since
+ * a clear pick over two also-rans reads as low concentration. */
+const PLACE_FLOOR = 0.5;
+const COLLISION_FLOOR = 0.5;
-/** Deterministic read of the verdict. A row the reply skips, misnames, or
- * answers with anything but the three words stays in the queue. */
-export function parseQueueTriage(
- outputText: string | undefined,
- rows: TriageRow[]
-): Map<string, QueueVerdict> {
+/** Each row's place, composed from the answers: a collision forbids a spawn,
+ * an uncertain placement queues, and a row with no answer queues. */
+export function placeQueuedRows(answers: QueueTriageAnswers | null, rows: TriageRow[]): Map<string, QueueVerdict> {
const out = new Map<string, QueueVerdict>(rows.map((r) => [r.id, "queue"]));
- let parsed: unknown;
- try {
- parsed = JSON.parse(outputText ?? "");
- } catch {
- return out;
- }
- const verdicts = (parsed as { verdicts?: unknown } | null)?.verdicts;
- if (!Array.isArray(verdicts)) return out;
- for (const v of verdicts) {
- const id = (v as { id?: unknown })?.id;
- const verdict = (v as { verdict?: unknown })?.verdict;
- if (typeof id !== "string" || !out.has(id)) continue;
- if (verdict === "fold" || verdict === "spawn" || verdict === "queue") out.set(id, verdict);
+ if (!answers) return out;
+ for (const r of rows) {
+ const place = answers[`place::${r.id}`];
+ if (!place || place.type !== "choice") continue;
+ const collides = answers[`collides::${r.id}`];
+ const collision = collides?.type === "noul" ? collides.noul : 0;
+ if ((place.probabilities[place.choice] ?? 0) < PLACE_FLOOR) continue;
+ if (place.choice === "spawn" && collision >= COLLISION_FLOOR) continue;
+ if (place.choice === "fold" || place.choice === "spawn" || place.choice === "queue") out.set(r.id, place.choice);
}
return out;
}
diff --git a/site/src/cut/lib/turnIntent.ts b/site/src/cut/lib/turnIntent.ts
deleted file mode 100644
index bb71e2cf..00000000
--- a/site/src/cut/lib/turnIntent.ts
+++ /dev/null
@@ -1,47 +0,0 @@
-// The chat harness's understanding boundary for one composer turn: a fast
-// structured model call judges the newest user message — does it ask for
-// anything, and how big is the job — and the harness gates and routes
-// deterministically on the verdict. A "chat" turn (greeting, thanks — social
-// filler that requests nothing) runs with no tool declarations at all, so
-// unsolicited edits are impossible by construction. A "simple" turn (one
-// self-contained ask) runs on the light chat model; a "complex" turn (a
-// composed job) runs on the full model. The turn runner overlaps this call
-// with the first model round and holds tool execution until the verdict
-// lands, so the gate costs a simple turn no wall time.
-
-export type TurnIntent = "chat" | "simple" | "complex";
-
-/** The gate's instructions. */
-export function turnIntentPrompt(): string {
- return `You gate an AI assistant built into a video editor. Read the conversation and judge only its newest user message.
-
-Reply with exactly one word:
-chat — pure social filler (a greeting, thanks, a sign-off, an acknowledgement) that requests nothing.
-simple — one self-contained ask: a single edit, a single generation, or a question to answer. Terse follow-ups ("yes", "do it", "the second one") that confirm one pending action are simple.
-complex — a composed job: several edits across the timeline, cutting or reorganizing many clips, assembling media into a cut, or anything needing a plan across steps. A sweep phrased as one ask still fans out into many cuts — removing filler words, tightening silences, cutting every pause — and is complex. An edit aimed at "this clip / that one / it" with no named target is also complex — resolving the referent takes the full model. So is an edit whose targets are picked by description — "the grey ones", "the short clips" — since deciding which items match takes the full model.
-
-If unsure between chat and the others, reply simple. If unsure between simple and complex, reply complex.`;
-}
-
-/** The conversation as plain text turns for the classifier — recent turns ride
- * along so follow-ups keep their referent, and each is capped so a pasted wall
- * of text can't blow the call's budget. */
-export function turnIntentInput(
- turns: { role: "user" | "assistant"; text: string }[]
-): Record<string, unknown>[] {
- return turns
- .filter((t) => t.text)
- .slice(-6)
- .map((t) => ({ role: t.role, content: [{ text: t.text.slice(0, 2000) }] }));
-}
-
-/** Deterministic read of the classifier's verdict line. Only a clear "chat"
- * withholds tools, and only a clear "simple" takes the light model — a garbled
- * reply runs the full model with every tool, so a classifier hiccup never
- * blocks or downgrades a real request. */
-export function parseTurnIntent(outputText: string | undefined): TurnIntent {
- const t = (outputText ?? "").trim();
- if (/^\W*chat\b/i.test(t)) return "chat";
- if (/^\W*simple\b/i.test(t)) return "simple";
- return "complex";
-}
diff --git a/site/src/cut/lib/turnJudge.test.ts b/site/src/cut/lib/turnJudge.test.ts
new file mode 100644
index 00000000..a36f4b54
--- /dev/null
+++ b/site/src/cut/lib/turnJudge.test.ts
@@ -0,0 +1,125 @@
+import { describe, expect, test } from "bun:test";
+import { SETTINGS } from "@/lib/config/registry";
+import { AI_SKILL_INDEX, TOOL_AREA_NAMES } from "@/cut/server/ai/catalog";
+import { judgeTurnState, pickSkill, routeTurn, TURN_JUDGE_QUESTIONS, type TurnJudgeAnswers } from "./turnJudge";
+
+const settings = SETTINGS.cutJudge.default;
+
+function answers(over: {
+ intent?: { chat: number; simple: number; complex: number };
+ skill?: string;
+ gate?: number;
+ unresolved?: number;
+ fits?: Record<string, number>;
+ areas?: Record<string, number>;
+}): TurnJudgeAnswers {
+ const p = over.intent ?? { chat: 0.05, simple: 0.8, complex: 0.15 };
+ const winner = (Object.entries(p).sort((a, b) => b[1] - a[1])[0][0]) as "chat" | "simple" | "complex";
+ const out: Record<string, unknown> = {
+ intent: { type: "choice", choice: winner, probabilities: p, confidence: 0.8 },
+ skill: {
+ type: "choice",
+ choice: over.skill ?? "none",
+ probabilities: Object.fromEntries(["none", ...AI_SKILL_INDEX].map((n) => [n, n === (over.skill ?? "none") ? 0.7 : 0.01])),
+ confidence: 0.7,
+ },
+ needs_reference: { type: "noul", noul: over.gate ?? 0.8 },
+ unresolved_target: { type: "noul", noul: over.unresolved ?? 0.05 },
+ };
+ for (const name of AI_SKILL_INDEX) out[`fits::${name}`] = { type: "noul", noul: over.fits?.[name] ?? 0.05 };
+ for (const area of TOOL_AREA_NAMES) out[`area::${area}`] = { type: "noul", noul: over.areas?.[area] ?? 0.05 };
+ return out as unknown as TurnJudgeAnswers;
+}
+
+describe("TURN_JUDGE_QUESTIONS", () => {
+ test("asks intent, skill, the gate, one fit per skill and one per area", () => {
+ const keys = Object.keys(TURN_JUDGE_QUESTIONS);
+ expect(keys.length).toBe(4 + AI_SKILL_INDEX.length + TOOL_AREA_NAMES.length);
+ expect(Object.keys(TURN_JUDGE_QUESTIONS.skill.criteria)).toEqual(["none", ...AI_SKILL_INDEX]);
+ });
+});
+
+describe("routeTurn", () => {
+ test("a clear chat verdict withholds every tool and carries no skill", () => {
+ const r = routeTurn(answers({ intent: { chat: 0.9, simple: 0.05, complex: 0.05 }, skill: "graphics", fits: { graphics: 0.9 } }), settings);
+ expect(r.intent).toBe("chat");
+ expect(r.skill).toBeNull();
+ });
+
+ test("simple needs a clear simple and little complex", () => {
+ expect(routeTurn(answers({ intent: { chat: 0.1, simple: 0.6, complex: 0.3 } }), settings).intent).toBe("simple");
+ expect(routeTurn(answers({ intent: { chat: 0.1, simple: 0.5, complex: 0.4 } }), settings).intent).toBe("complex");
+ expect(routeTurn(answers({ intent: { chat: 0.5, simple: 0.45, complex: 0.05 } }), settings).intent).toBe("simple");
+ expect(routeTurn(answers({ intent: { chat: 0.4, simple: 0.3, complex: 0.3 } }), settings).intent).toBe("complex");
+ });
+
+ const complex = { chat: 0.05, simple: 0.15, complex: 0.8 };
+
+ test("an edit at an unnamed target takes the full model", () => {
+ expect(routeTurn(answers({ unresolved: 0.8 }), settings).intent).toBe("complex");
+ });
+
+ test("a composed job carries the winning skill when the gate and its fit clear the floors", () => {
+ const r = routeTurn(answers({ intent: complex, skill: "color-grading", fits: { "color-grading": 0.8 }, areas: { color: 0.9 } }), settings);
+ expect(r.skill).toBe("color-grading");
+ for (const a of ["color", "inspector"]) expect(r.areas).toContain(a);
+ });
+
+ test("a simple turn runs on the tool descriptions alone", () => {
+ const a = answers({ skill: "color-grading", fits: { "color-grading": 0.9 }, areas: { color: 0.9 } });
+ expect(pickSkill(a, settings)).toBe("color-grading");
+ expect(routeTurn(a, settings).skill).toBeNull();
+ });
+
+ test("no skill when the ask needs nothing looked up", () => {
+ expect(pickSkill(answers({ skill: "timeline-editing", fits: { "timeline-editing": 0.9 }, gate: 0.2 }), settings)).toBeNull();
+ expect(pickSkill(answers({ skill: "none", fits: { graphics: 0.9 } }), settings)).toBeNull();
+ expect(pickSkill(answers({ skill: "graphics", fits: { graphics: 0.1 } }), settings)).toBeNull();
+ });
+
+ test("areas are the ones above the threshold plus the skill's own", () => {
+ const r = routeTurn(answers({ intent: complex, skill: "audio-and-subtitles", fits: { "audio-and-subtitles": 0.9 }, areas: { stock: 0.5, timeline: 0.2 } }), settings);
+ for (const a of ["stock", "audio", "subtitles", "inspector", "timeline"]) expect(r.areas).toContain(a);
+ expect(r.areas).not.toContain("scene");
+ // Catalog order, so the declared list is stable.
+ expect(r.areas).toEqual(TOOL_AREA_NAMES.filter((a) => r.areas.includes(a)));
+ });
+
+ test("a failed judgment fails open: the full model, every area, no skill", () => {
+ const r = routeTurn(null, settings);
+ expect(r).toEqual({ intent: "complex", skill: null, areas: [...TOOL_AREA_NAMES] });
+ });
+
+ test("routing off declares every area; suggestion off attaches nothing", () => {
+ const off = { ...settings, toolRouting: false, skillSuggestion: false };
+ const r = routeTurn(answers({ skill: "graphics", fits: { graphics: 0.9 }, areas: { color: 0.9 } }), off);
+ expect(r.areas).toEqual([...TOOL_AREA_NAMES]);
+ expect(r.skill).toBeNull();
+ });
+});
+
+describe("judgeTurnState", () => {
+ test("carries the newest ask, the turns before it, attachment metadata without urls, and an editor slice", () => {
+ const state = judgeTurnState(
+ [
+ { id: "1", role: "user", parts: [{ type: "text", text: "add captions" }] },
+ { id: "2", role: "assistant", parts: [{ type: "text", text: "Done." }] },
+ {
+ id: "3",
+ role: "user",
+ parts: [{ type: "text", text: "make them bigger" }],
+ metadata: { attachments: [{ kind: "image", scope: "file", name: "ref.png", url: "data:..." }] },
+ },
+ ],
+ { project: { aspect: "9:16", duration: 12 }, videoTrack: [{}, {}], overlays: [{ kind: "title" }], subtitles: { tracks: [{}] }, selection: { kind: "clip" } },
+ ) as Record<string, unknown>;
+ expect(state.request).toBe("make them bigger");
+ expect(state.recent).toEqual([
+ { role: "user", text: "add captions" },
+ { role: "assistant", text: "Done." },
+ ]);
+ expect(state.attachments).toEqual([{ kind: "image", scope: "file", name: "ref.png" }]);
+ expect(state.editor).toMatchObject({ aspect: "9:16", clips: 2, overlayKinds: ["title"], captionTracks: 1, selection: { kind: "clip" } });
+ expect(JSON.stringify(state)).not.toContain("data:");
+ });
+});
diff --git a/site/src/cut/lib/turnJudge.ts b/site/src/cut/lib/turnJudge.ts
new file mode 100644
index 00000000..217a6a03
--- /dev/null
+++ b/site/src/cut/lib/turnJudge.ts
@@ -0,0 +1,218 @@
+import type { UIMessage } from "ai";
+import type { Settings } from "@/lib/config/registry";
+import {
+ AI_SKILL_AREAS,
+ AI_SKILL_BLURBS,
+ AI_SKILL_INDEX,
+ skillTitle,
+ TOOL_AREA_NAMES,
+ TOOL_AREAS,
+} from "@/cut/server/ai/catalog";
+import { choice, noul, type ChoiceAnswer, type Entry, type NoulAnswer } from "./judge";
+
+// The chat harness's understanding boundary for one composer turn: one typed
+// judgment over the newest message, the recent turns, and a thin slice of the
+// editor answers three things at once — how big the ask is (chat / simple /
+// complex), which skill document the model should read first, and which tool
+// areas the turn needs — and the harness routes deterministically on the
+// probabilities under the thresholds in the `cutJudge` setting. A "chat" turn
+// runs with no tool declarations at all, so unsolicited edits are impossible
+// by construction; a "simple" turn keeps the light chat model; a "complex"
+// turn runs on the full model. The route is composed here, in code, from the
+// answers; nothing reads the user's words.
+
+export type TurnIntent = "chat" | "simple" | "complex";
+
+export interface TurnRoute {
+ intent: TurnIntent;
+ /** The skill document attached to the turn, or null for none. */
+ skill: string | null;
+ /** The tool areas declared to the turn. */
+ areas: string[];
+}
+
+export type CutJudgeSettings = Settings["cutJudge"];
+
+/** The judge's questions, built once: the roster of skills and areas is the
+ * catalog's, so a new skill or tool module joins the judgment on its own. */
+export const TURN_JUDGE_QUESTIONS = {
+ intent: choice(
+ "Judge only the newest user message, `request`, with `recent` as context. What does it ask of the AI assistant built into a video editor?",
+ {
+ chat: "Pure social filler that requests nothing: a greeting, thanks, a sign-off, an acknowledgement.",
+ simple:
+ "One self-contained ask: a single edit, a single generation, or a question to answer. A terse follow-up (\"yes\", \"do it\", \"the second one\") that confirms one pending action is simple.",
+ complex:
+ "A composed job: several edits across the timeline, cutting or reorganizing many clips, assembling media into a cut, building a whole video from a script, a song, or pasted text (a lyric video, quote cards, kinetic type), or anything needing a plan across steps. A sweep phrased as one ask still fans out into many cuts (removing filler words, tightening silences, cutting every pause). An edit aimed at \"this clip / that one / it\" with no named target, or whose targets are picked by description (\"the grey ones\", \"the short clips\"), takes the full model.",
+ },
+ ),
+ skill: choice(
+ "Which skill document would a careful editor open before acting on `request`? Each document teaches one area of the editor: its settings, where they live, and how they behave.",
+ Object.fromEntries([
+ ["none", "No document covers this request, or the request needs none: a greeting, a question answered from the state, an edit a single plainly named tool performs."],
+ ...AI_SKILL_INDEX.map((name) => [name, `${skillTitle(name)}. ${AI_SKILL_BLURBS[name] ?? ""}`]),
+ ]) as Record<string, Entry>,
+ ),
+ unresolved_target: noul(
+ "Does `request` aim an edit at an item it points to without naming — \"that clip\", \"it\", \"the one\", \"those\" — which neither `recent` nor `editor.selection` identifies?",
+ {
+ true: "The target is pointed at and nothing in the conversation or the selection says which item it is.",
+ false: "The target is named, described, selected, or was the subject of the recent turns; or the request is an undo, a redo, a confirmation, or edits nothing.",
+ },
+ ),
+ needs_reference: noul(
+ "Would the assistant need to read a reference document about this area of the editor to do `request` well?",
+ {
+ true: "The result depends on settings, modes, defaults, or a flow the tool descriptions do not spell out: grading, transitions, captions, voiceovers, generation, scene productions, graphics, exports, replicating a project, a taste question.",
+ false: "One plainly named action — mute, delete, hide, seek, undo, select, rename, set the aspect, open a dialog — or a fact read off the editor state, such as the cut's length or what is on the timeline.",
+ },
+ ),
+ ...Object.fromEntries(
+ AI_SKILL_INDEX.map((name) => [
+ `fits::${name}`,
+ noul(`Does the skill "${skillTitle(name)}" document the settings or the flow that \`request\` needs?`, {
+ true: AI_SKILL_BLURBS[name] ?? skillTitle(name),
+ false: "The request does not involve this area, or needs nothing from it.",
+ }),
+ ]),
+ ),
+ ...Object.fromEntries(
+ TOOL_AREA_NAMES.map((area) => [
+ `area::${area}`,
+ noul(`Does finishing \`request\` call for the ${area.replace("_", " ")} tools?`, {
+ true: TOOL_AREAS[area].blurb,
+ false: "The request can be finished without these tools.",
+ }),
+ ]),
+ ),
+} as const;
+
+type Answers = {
+ intent: ChoiceAnswer<Record<"chat" | "simple" | "complex", string>>;
+ skill: ChoiceAnswer<Record<string, string>>;
+ unresolved_target: NoulAnswer;
+ needs_reference: NoulAnswer;
+} & Record<string, NoulAnswer | ChoiceAnswer<Record<string, string>>>;
+
+export type TurnJudgeAnswers = Answers;
+
+const TEXT_CAP = 2000;
+const RECENT_TURNS = 5;
+
+/** The state the judge reads: the newest ask, the turns before it, what it
+ * attached, and the parts of the editor snapshot that change what an ask
+ * means. No urls, no media. */
+export function judgeTurnState(messages: UIMessage[], context: unknown): Entry {
+ const text = (m: UIMessage) =>
+ m.parts
+ .map((p) => (p.type === "text" ? p.text : ""))
+ .join("")
+ .trim()
+ .slice(0, TEXT_CAP);
+ const last = messages.findLast((m) => m.role === "user");
+ const before = last ? messages.slice(0, messages.indexOf(last)) : messages;
+ const recent = before
+ .map((m) => ({ role: m.role === "user" ? "user" : "assistant", text: text(m) }))
+ .filter((t) => t.text)
+ .slice(-RECENT_TURNS);
+ const meta = (last?.metadata as { attachments?: unknown[] } | undefined)?.attachments;
+ const attachments = Array.isArray(meta)
+ ? meta.map((a) => {
+ const r = (a ?? {}) as Record<string, unknown>;
+ return {
+ kind: typeof r.kind === "string" ? r.kind : null,
+ scope: typeof r.scope === "string" ? r.scope : null,
+ name: typeof r.name === "string" ? r.name.slice(0, 120) : null,
+ };
+ })
+ : [];
+ return {
+ request: last ? text(last) : "",
+ recent,
+ attachments,
+ editor: editorSlice(context),
+ };
+}
+
+function editorSlice(context: unknown): Entry {
+ const c = (context ?? {}) as Record<string, unknown>;
+ const project = (c.project ?? {}) as Record<string, unknown>;
+ const selection = c.selection as Record<string, unknown> | null | undefined;
+ const videoTrack = Array.isArray(c.videoTrack) ? c.videoTrack : [];
+ const soundtrack = Array.isArray(c.soundtrack) ? c.soundtrack : [];
+ const overlays = Array.isArray(c.overlays) ? (c.overlays as Record<string, unknown>[]) : [];
+ const subtitles = (c.subtitles ?? {}) as Record<string, unknown>;
+ const tracks = Array.isArray(subtitles.tracks) ? subtitles.tracks : [];
+ const media = Array.isArray(c.media) ? c.media : [];
+ return {
+ aspect: typeof project.aspect === "string" ? project.aspect : null,
+ duration: typeof project.duration === "number" ? project.duration : null,
+ selection: selection
+ ? { kind: typeof selection.kind === "string" ? selection.kind : null, name: typeof selection.name === "string" ? selection.name : null }
+ : null,
+ clips: videoTrack.length,
+ soundtrackClips: soundtrack.length,
+ overlayKinds: [...new Set(overlays.map((o) => (typeof o.kind === "string" ? o.kind : "overlay")))],
+ captionTracks: tracks.length,
+ mediaAssets: media.length,
+ };
+}
+
+// Only a clear chat withholds the tools; any real weight on complex, or a
+// flat read, takes the full model; the rest is the light model.
+const CHAT_FLOOR = 0.6;
+const COMPLEX_FLOOR = 0.35;
+const FLAT_CEILING = 0.45;
+const UNRESOLVED_FLOOR = 0.5;
+
+/** The skill the answers name, under the thresholds: the reference need
+ * clears the gate, the pick is a real skill, and its own fit clears the
+ * floor. Null when the turn wants none or suggestion is off. */
+export function pickSkill(answers: TurnJudgeAnswers, settings: CutJudgeSettings): string | null {
+ if (!settings.skillSuggestion) return null;
+ const pick = answers.skill.choice;
+ const fit = (answers[`fits::${pick}`] as NoulAnswer | undefined)?.noul ?? 0;
+ return answers.needs_reference.noul >= settings.skillGate &&
+ pick !== "none" &&
+ AI_SKILL_INDEX.includes(pick) &&
+ fit >= settings.skillFits
+ ? pick
+ : null;
+}
+
+/** The route for a turn, composed from the answers under the thresholds. A
+ * missing verdict (the call failed) fails open: the full model, no skill,
+ * every area. */
+export function routeTurn(answers: TurnJudgeAnswers | null, settings: CutJudgeSettings): TurnRoute {
+ if (!answers) return { intent: "complex", skill: null, areas: [...TOOL_AREA_NAMES] };
+
+ const p = answers.intent.probabilities;
+ // An edit whose target nothing identifies takes the full model, which asks.
+ const intent: TurnIntent =
+ p.chat >= CHAT_FLOOR
+ ? "chat"
+ : p.complex >= COMPLEX_FLOOR ||
+ Math.max(p.chat, p.simple, p.complex) < FLAT_CEILING ||
+ answers.unresolved_target.noul >= UNRESOLVED_FLOOR
+ ? "complex"
+ : "simple";
+
+ // A composed job gets its reference up front. A simple turn runs on the
+ // tool descriptions alone: the light model reads an attached reference as
+ // more to do, and the eval shows it over-acting with one.
+ const skill = intent === "complex" ? pickSkill(answers, settings) : null;
+
+ let areas: string[];
+ if (!settings.toolRouting) {
+ areas = [...TOOL_AREA_NAMES];
+ } else {
+ const wanted = new Set<string>();
+ for (const area of TOOL_AREA_NAMES) {
+ const a = answers[`area::${area}`] as NoulAnswer | undefined;
+ if (a && a.noul >= settings.toolArea) wanted.add(area);
+ }
+ for (const area of skill ? (AI_SKILL_AREAS[skill] ?? []) : []) wanted.add(area);
+ areas = TOOL_AREA_NAMES.filter((a) => wanted.has(a));
+ }
+ return { intent, skill, areas };
+}
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index edeb65ef..744d82a0 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -84,6 +84,136 @@ export const AI_TOOLS: AiToolDef[] = [
/** Shared project schemas for hosts that provide their own conversation UI. */
export const PROJECT_TOOLS: AiToolDef[] = AI_TOOLS.filter((tool) => !tool.server && !UI_TOOLS.has(tool.name));
+/** The assistant's own senses and flows, declared on every work turn. */
+export const CORE_TOOLS: AiToolDef[] = [...AI_PANEL_TOOLS];
+
+const COLOR_TOOL = /^(set_color_|read_color_stats$|match_color_grade$)/;
+
+/** The catalog by area: each `*.tools.ts` module is one area (the inspector's
+ * color tools stand on their own), named and described once here so the turn
+ * judge, the escape-hatch tool, and the eval all read the same map. */
+export const TOOL_AREAS: Record<string, { blurb: string; tools: AiToolDef[] }> = {
+ preview: {
+ blurb: "render a preview, capture the composited frame at a time, seek, play or pause, nudge the selected item",
+ tools: [...PREVIEW_TOOLS],
+ },
+ timeline: {
+ blurb:
+ "cut, split, trim, move, place, add and delete clips; speed and retimes; mute, hide, lock, and select; gaps, groups, tracks, overlay video, freeze frames; add titles and text runs; refine speech cuts; the transitions parked at cuts",
+ // A retime or a cut moves the transitions parked at its cuts, so the
+ // transition tools ride with the timeline's.
+ tools: [...TIMELINE_TOOLS, ...TRANSITIONS_TOOLS],
+ },
+ inspector: {
+ blurb:
+ "the selected item's settings: an overlay's text, position, size and style; masks; clip and overlay keyframes; a clip's audio, volume and sound; framing, clip style, speed",
+ tools: INSPECTOR_TOOLS.filter((t) => !COLOR_TOOL.test(t.name)),
+ },
+ color: {
+ blurb: "color grading: a clip's color stats, grades, presets, curves, wheels, HSL, matching a reference's grade",
+ tools: INSPECTOR_TOOLS.filter((t) => COLOR_TOOL.test(t.name)),
+ },
+ removal: {
+ blurb: "background removal (cutout): matte a clip's subject or background, strokes, what shows behind it",
+ tools: [...REMOVAL_TOOLS],
+ },
+ elements: {
+ blurb: "add shapes and stickers, or create a sticker from an image",
+ tools: [...ELEMENTS_TOOLS],
+ },
+ effects: {
+ blurb: "add a look effect or treatment to a clip",
+ tools: [...EFFECTS_TOOLS],
+ },
+ animation: {
+ blurb: "an overlay element's entrance, exit and loop animations and keyframed motion",
+ tools: [...OVERLAY_ANIMATION_TOOLS],
+ },
+ transitions: {
+ blurb: "transitions between clips, and a clip's own in and out animation",
+ tools: [...TRANSITIONS_TOOLS],
+ },
+ subtitles: {
+ blurb:
+ "captions and subtitles: transcribe, caption from visuals, sync lyrics, align to audio, the caption look, tracks, translation, editing cues",
+ tools: [...SUBTITLES_TOOLS],
+ },
+ audio: {
+ blurb: "voiceover, music generation, reading subtitles aloud, the voice list",
+ tools: [...AUDIO_TOOLS],
+ },
+ image_gen: {
+ blurb: "generate an image with a hosted model",
+ tools: [...IMAGE_GEN_TOOLS],
+ },
+ video_gen: {
+ blurb: "generate a video clip or a talking character video with a hosted model; a better version or another take of an existing clip is a fresh render",
+ tools: [...VIDEO_GEN_TOOLS],
+ },
+ scene: {
+ blurb: "scene productions: plan, approve, regenerate, recut and restyle a narrated multi-shot cut",
+ tools: [...SCENE_TOOLS],
+ },
+ stock: {
+ blurb: "search the bundled stock footage, images, characters and sound effects, and add one to the project",
+ tools: [...STOCK_TOOLS],
+ },
+ library: {
+ blurb: "the shared library: list, add, share, organize, notes, templates",
+ tools: [...LIBRARY_TOOLS],
+ },
+ side_panel: {
+ blurb:
+ "the side panel and project media: open a panel, file or convert assets, organize or delete media, publish metadata, the project background",
+ tools: [...SIDE_PANEL_TOOLS],
+ },
+ top_bar: {
+ blurb: "the aspect ratio, guides, the project name, export",
+ tools: [...TOP_BAR_TOOLS],
+ },
+ editor: {
+ blurb: "copy, paste, undo, redo",
+ tools: [...EDITOR_TOOLS],
+ },
+};
+
+export const TOOL_AREA_NAMES = Object.keys(TOOL_AREAS);
+
+/** The tools of the named areas, in catalog order, deduplicated. */
+export function areaTools(areas: Iterable<string>): AiToolDef[] {
+ const wanted = new Set(areas);
+ const names = new Set<string>();
+ for (const area of TOOL_AREA_NAMES) {
+ if (!wanted.has(area)) continue;
+ for (const t of TOOL_AREAS[area].tools) names.add(t.name);
+ }
+ return AI_TOOLS.filter((t) => names.has(t.name));
+}
+
+/** The tool a turn calls to widen its catalog when the routed areas left out
+ * what the ask needs. Loop-local: it lives outside the catalog, executes in
+ * the chat loop, and never reaches a host or the engine. */
+export const REQUEST_TOOLS_DEF: AiToolDef = {
+ name: "request_tools",
+ description:
+ "Add tool areas to this turn. The tools declared to you are the areas judged relevant to the ask; when the ask needs a tool you do not see, name its area here and it is available from your next step. Areas: " +
+ TOOL_AREA_NAMES.map((a) => `${a} (${TOOL_AREAS[a].blurb})`).join("; ") +
+ ".",
+ inputSchema: {
+ type: "object",
+ properties: {
+ areas: {
+ type: "array",
+ items: { type: "string", enum: TOOL_AREA_NAMES },
+ description: "Area names to add",
+ },
+ },
+ required: ["areas"],
+ additionalProperties: false,
+ },
+ server: true,
+};
+
/** Deep documentation the model can pull in on demand. */
export const AI_SKILLS: Record<string, string> = {
@@ -431,7 +561,7 @@ Rules:
- project.safeZones in editor_state gives, for this frame, the safe area of every guide preset (frame fractions, x/y/w/h) and the keepOut boxes where TikTok, Reels and Shorts draw their own UI or crop the frame's sides — known whether or not a guide shows. When the user names a platform, asks for a safe zone, or wants graphics kept clear of the app's UI, place every title, sticker and caption inside safeZones.shortform.safeArea and check with capture_frame; safeZones.margins is the general 5% inset. A safeArea is a rect — x/y its top-left corner, w/h its size — while an element's x/y is its center: centered in the safe area means x + w/2, y + h/2, and an element of width ew stays inside while its center is between x + ew/2 and x + w − ew/2. Guides are a view: set_guides shows one when the user asks to see it, and showing one changes nothing in the cut.
- Exercise judgment on details the user left open, and state the choice you made. Judgment fills in the parameters of the asked-for change (style, length, placement); the scope stays the ask, and extra tools, tracks, or treatments beyond it wait for the user to ask. Editing defaults: a hard cut at every joint; when transitions are asked for, one style family per video at 0.4–0.6s. The editing-taste skill carries the full doctrine.
- Styling is part of the ask, never a bonus. Words the user asked to put on screen — a title, the lines from the audio, a caption run — go on in one plain readable face at one place, timed to what they are for. Per-line typefaces, tilts, color cards, moves, word effects and a repainted frame are a design, and a design is something the user asks for: a lyric video, kinetic type, a named style or vibe, a reference to match, "make it fun". Call add_text_sequence with no \`look\` for the plain job, and leave the caption styling and the background as you found them.
-- Read list_skills / read_skill before working in an area you're unsure about — they document every setting. A tool whose description already answers the ask goes straight through.
+- A skill attached to the message (<skill_relevance>) is the reference for its area: read it before touching that area's settings. It never changes which tool the ask maps to — the tool descriptions decide that ("a better version" is a render, whatever skill rode along). list_skills / read_skill cover an area the attachment left out.
- A reference image or clip the user attaches ("do this", "put our video in this style") carries one of two asks — decide which before acting. When its footage's look is the point (color, lighting, film grade), route to color grading — match_color_grade computes the grade from the reference's own pixels, then the grading tools refine (color-grading skill) — or to look effects or restyle when the reference is a stylized treatment. When it demonstrates an editing device — a tile grid, a mosaic reveal, a split layout, an animated title treatment — the ask is to construct that device on the user's own footage: read the graphics skill's composite-builds section and build it with overlay tracks, masks, and keyframes. Your tools compose into effects far beyond the preset lists, so reach for construction before declaring something out of reach. Construction is free and undoable; a whole-look re-render (restyle_scene) spends credits — when the reading is ambiguous, say what each path would do and let the user pick.
- A Donkey Cut project link or share link the user gives you is a document: read it with read_project (the document is the edit — never watch a reference to learn what its document already states), decide which of this project's sources plays each of its parts from the reference's observed notes and transcripts (a reference look with watch_video's project_link where those leave it dark), then call replicate_project once — the whole edit, or only the items they asked for: a title, one clip's treatment, the music, the caption look — and tune from there. copy_project_media brings files over on their own. The replicating-a-project skill has the flow.
- Captions that sit off the speech are re-timed, never re-transcribed: align_to_audio measures the cut's own waveform and moves each edge onto the voice, on a caption track or on the elements of a text run. Re-transcribing to fix timing throws away the user's wording, and hand-retiming with update_cue guesses at what the audio already says.
@@ -444,6 +574,70 @@ Rules:
export const AI_SKILL_INDEX = Object.keys(AI_SKILLS);
+/** A skill document by name; undefined for a name the library lacks. */
+export function readSkill(name: string): string | undefined {
+ return Object.hasOwn(AI_SKILLS, name) ? AI_SKILLS[name] : undefined;
+}
+
+/** A skill's title: the heading its document opens with. */
+export function skillTitle(name: string): string {
+ return (AI_SKILLS[name] ?? "").split("\n")[0].replace(/^#\s*/, "").trim();
+}
+
+/** One sentence per skill, for the turn judge's roster. */
+export const AI_SKILL_BLURBS: Record<string, string> = {
+ "editor-overview": "The editor's layout: panels, tracks, the aspect ratio, autosave, undo, where media lives.",
+ "timeline-editing": "How clips, gaps, groups, tracks and moves behave on the timeline; placing and reordering.",
+ "color-grading":
+ "Grading like a colorist: reading color stats, correcting exposure and balance, presets, curves, wheels, matching a reference.",
+ "background-removal": "Cutouts: matting a clip's subject or background, auto and custom modes, strokes, refinement.",
+ "watching-and-cutting":
+ "Cutting by content: watching footage, listening, dead air, beats; the flow for cuts that depend on what the footage holds.",
+ "transitions-and-fades": "Which feature an ask maps to: transitions between clips, a clip's own animation, look effects, fades.",
+ graphics: "Titles, shapes and stickers: lanes, positioning, styling, masks, keyframes, composite builds like grids and reveals.",
+ "text-videos": "Videos whose picture is words: lyric videos, kinetic type, quote cards; looks, timing, backgrounds.",
+ "text-creativity": "Fifty text devices for a run of words that reads flat: groups of devices and how to combine them.",
+ "audio-and-subtitles": "Soundtrack clips, levels and matching, voiceover, lyric sync, caption tracks and their look.",
+ "ai-generation": "Getting footage the user lacks: stock, URL import, hosted image, video and music generation and their settings; a better version or another take of a clip is a fresh render.",
+ "scene-productions": "Planning and rendering a narrated multi-shot scene: the plan card, approval, look references, revisions.",
+ "editing-taste": "Deciding details the user left open: joints, transition families, pacing, when to add and when to leave alone.",
+ "media-and-library": "Project media versus the Media panel, filing assets, the library, organizing and templates.",
+ "replicating-a-project": "Reading another Donkey Cut project or share link as a document and bringing its edit or items over.",
+ "publish-and-export": "Publish fields for the platforms, and every export preset and axis.",
+};
+
+/** The tool areas a skill's guidance acts through; a suggested skill brings
+ * them along whatever the judge said about the areas. */
+export const AI_SKILL_AREAS: Record<string, string[]> = {
+ "editor-overview": [],
+ "timeline-editing": ["timeline"],
+ "color-grading": ["color", "inspector"],
+ "background-removal": ["removal"],
+ "watching-and-cutting": ["timeline"],
+ "transitions-and-fades": ["transitions", "animation", "effects"],
+ graphics: ["timeline", "inspector", "elements", "animation"],
+ "text-videos": ["timeline", "subtitles", "inspector", "elements", "side_panel"],
+ "text-creativity": ["timeline", "inspector", "elements", "animation"],
+ "audio-and-subtitles": ["audio", "subtitles", "inspector", "timeline"],
+ "ai-generation": ["stock", "image_gen", "video_gen", "timeline"],
+ "scene-productions": ["scene"],
+ "editing-taste": ["timeline", "transitions"],
+ "media-and-library": ["library", "side_panel", "timeline"],
+ "replicating-a-project": [],
+ "publish-and-export": ["top_bar", "side_panel"],
+};
+
+/** The turn's skill suggestion as prompt text: the judge's pick and the
+ * document itself, so the model reads the right reference without a tool
+ * round. Rides as an ephemeral message on the hosted loop and on the
+ * engine's prompt; never stored. */
+export function skillRelevanceBlock(skill: string | null): string {
+ const doc = skill ? readSkill(skill) : undefined;
+ if (!skill || !doc)
+ return "<skill_relevance>\nNo skill in the library appears relevant to this request.\n</skill_relevance>";
+ return `<skill_relevance>\nRelevant to the current request: ${skill}. It documents that area's settings; the tool descriptions still decide which tool the ask maps to. Ignore it if it does not fit what the user actually asked for.\n</skill_relevance>\n<skill name="${skill}">\n${doc}\n</skill>`;
+}
+
/** The <attached_assets> block a user turn carries when it has attachment
* refs. One builder serves hosted chat, the engine chat, and the eval mirror,
* so the doctrine — especially the clip-scope steering — can never drift
diff --git a/site/src/cut/server/http/ai.ts b/site/src/cut/server/http/ai.ts
index c6b89d8a..16a3fb61 100644
--- a/site/src/cut/server/http/ai.ts
+++ b/site/src/cut/server/http/ai.ts
@@ -19,13 +19,16 @@ import {
} from "../ai/bridge";
import { rewriteCaptions, translateCaptions } from "../ai/captions";
import { writeVisualCues, type VisualFrame } from "../ai/visualSubtitles";
-import { AI_SKILL_INDEX, AI_SKILLS, AI_TOOLS, attachedAssetsBlock, systemPrompt } from "../ai/catalog";
+import { AI_SKILL_INDEX, AI_TOOLS, attachedAssetsBlock, readSkill, skillRelevanceBlock, systemPrompt } from "../ai/catalog";
interface ChatBody {
threadId: string;
messages: UIMessage[];
model: string;
context?: unknown;
+ /** The skill the page's judge attached to this turn: a name, null for
+ * none, absent when the judge was not asked. */
+ skill?: string | null;
/** Provider-native session/thread id from the previous turn, if any. */
providerSession?: string;
}
@@ -409,7 +412,9 @@ export const aiApi = {
const sessionKey = crypto.randomUUID();
const userText = lastUserText(body.messages);
const attachments = lastUserAttachments(body.messages);
- const prompt = `${userText}${attachedAssetsBlock(attachments)}\n\n<editor_state>\n${JSON.stringify(body.context ?? {})}\n</editor_state>`;
+ const skill =
+ body.skill === undefined ? "" : `\n\n${skillRelevanceBlock(typeof body.skill === "string" ? body.skill : null)}`;
+ const prompt = `${userText}${attachedAssetsBlock(attachments)}${skill}\n\n<editor_state>\n${JSON.stringify(body.context ?? {})}\n</editor_state>`;
return startTurnStream(projectId, body.threadId, (signal) => createUIMessageStream({
execute: async ({ writer }) => {
@@ -520,7 +525,7 @@ export const aiApi = {
if (def.server) {
if (name === "list_skills") return Response.json(mcpText({ skills: AI_SKILL_INDEX }));
if (name === "read_skill") {
- const doc = AI_SKILLS[String(args?.name ?? "")];
+ const doc = readSkill(String(args?.name ?? ""));
return doc
? Response.json(mcpText(doc))
: Response.json({
diff --git a/site/src/cut/worker/turnJob.ts b/site/src/cut/worker/turnJob.ts
index 95d6d443..e50559c2 100644
--- a/site/src/cut/worker/turnJob.ts
+++ b/site/src/cut/worker/turnJob.ts
@@ -1,4 +1,5 @@
import type { UIMessage } from "ai";
+import { getGlobalSetting } from "@/lib/config/effective";
import { geminiModelRoleNames } from "@/lib/inference/gemini-models";
import { openCloudProject, pushCloudProject } from "../lib/headless/docSession";
import { dropPiSession, streamCutChat } from "../lib/pi/cutAgent";
@@ -102,6 +103,7 @@ export async function runTurnJob(
const doc = await openCloudProject(session, job.projectId);
const deps = headlessDeps(session);
+ deps.judgeSettings = await getGlobalSetting("cutJudge");
const toolCalls: string[] = [];
const exec = deps.execTool;
deps.execTool = async (name, args) => {