Skip to content

Show it a video you love. Get a new video in the same style. An AI crew (Claude Code or Codex) plans, builds and reviews it with you.

9.3 KB
Raw Blame

English · 繁體中文 · 简体中文

Extending ReelMimic

Extension points, from shallow to deep:

You want to Change Code?
Add a video style .claude/skills/video-clone/styles/<name>.md No
Add a production engine (a new drawing / rendering method) .claude/skills/<engine>/ (an ordinary agent skill) No
Change how an agent handles a step app/server/prompts.ts Text only, applies on the next turn
Give agents a new tool .claude/skills/video-clone/scripts/ Write the script, then mention it in a prompt / SKILL.md
Plug in another AI director (a new agent CLI) app/server/agents/index.ts Yes
Change the steps or order of the production line app/server/ (see §5) + prompts.ts + CONTRACT.md Yes

1. Add a style

A style = "for this kind of reference → use this engine with these defaults". During the styling step the director reads every file in styles/, matches the reference against each file's recognition cues and writes the best match into analysis/route.json.

  1. Copy styles/_TEMPLATE.md to styles/<name>.md.

  2. Fill in the frontmatter:

    ---
    name: lofi-anime-loop        # unique id, same as the file name
    engine: hyperframes          # folder name under .claude/skills/
    medium: 2d-vector            # 2d-painted · 2d-vector · 3d-stylized · 3d-photoreal (hard constraint when picking an engine)
    priority: 60                 # when two styles match, the higher one wins
    ---
    
  3. Write three sections:

    • Recognition cues: a sentence or two each for picture, editing, sound and text — things you can see or measure (e.g. average shot 2–3 s, cuts follow the narration, not the beat).
    • Production defaults: aspect ratio, length, fps, how characters are made (e.g. 2D vector characters always use assets/vector_rig), caption spec.
    • Known pitfalls: problems hit in earlier productions and their fixes. Add to it whenever a production teaches something new; the next video avoids it automatically.
  4. Create a project with a representative reference and check that analysis/route.json picks your style.

To take a style out of selection, move the file to styles/_disabled/.

2. Add a production engine

An engine is an ordinary agent skill: a folder with a SKILL.md (instructions for the agent) and the scripts and templates it needs.

.claude/skills/my-engine/
  SKILL.md          frontmatter (name, description) + how to create a project, preview a frame, export MP4, rules and pitfalls
  scripts/ …        render and preview tools
  template/ …       new-project skeleton (optional)

For it to plug into the production line, SKILL.md must make these clear (the director follows it in the setup step to build build/production.json):

  • One file per shot: parallel builders each edit only their own files, so they never collide.
  • Shared files: character definitions, palette, caption layer and audio live in shared files; shots call characters, they never redraw body parts.
  • Preview command: how to quickly render a single frame and a crop at a given time (all review depends on it). HyperFrames projects can use scripts/hf_frames.py directly.
  • Export command: how to render the full MP4 and mix audio.
  • Determinism: every frame depends only on time (no Math.random(), no cross-frame state), so frames can be rendered in parallel and reviewed out of order.

Then write a style file pointing to it (previous section). Before adding a third-party skill, read its scripts and add its license to THIRD_PARTY_NOTICES.md.

Starting from the engine kit

.claude/skills/video-clone/assets/engine-kit/ is the fastest way to build a new look. It has the shared runtime the pixel-art, paper-cutout, whiteboard and anime-cel engines are built on (timeline, motion helpers, camera, captions and the render contract), the headless-Chrome render.mjs, a page template, new_project.sh, and ENGINE_BRIEF.md, the checklist those engines were built from. Copy it into .claude/skills/<your-engine>/template/, write your drawing library and characters, and you get parallel rendering, contact sheets and crops for review, and resumable export for free.

Character systems

  • assets/vector_rig/: 2D vector characters — a skeleton (neck, shoulders, elbows, wrists, hips, knees) with one outline per depth layer, so limbs are always attached. See its README.
  • assets/cast_rig.js: character skeleton for painted-animation (hand-painted watercolor).

If a new engine has its own way of making characters, keep the same rule — "characters defined in shared files, one file per character, shots only pass pose parameters" — so the cast gate can review and fix them in parallel.

3. Tune agent behaviour (prompts.ts)

Each key in app/server/prompts.ts is one step of the pipeline:

Key Who Does
style · plan · replan director pick the style, write the plan core, revise from feedback
pre_cast · pre_assets · plan_frames character · assets · director parallel pre-production: per-character drafts, asset search, merge and paint style frames
setup director engine project, shared files, characters, segments
cast_qa · cast_fix reviewer · fixer cast gate
build_chunk · shot_qa · fix_chunk builder · reviewer · builder segment production; each finished shot gets its own reviewer (shots / out select the shots and output file)
shared_fix director shared-file problems reported by builders
assemble · critique · revise director · critic · director assembly, final critique, revisions

Shared fragments: RULES (content and licensing), SPEED (efficiency rules), EYE (human-eye checklist), QA_OUT (review output format and the blocker/polish definitions). The server checks this file for changes before every dispatch — edits apply from the next agent turn, no restart.

Suggested approach: run python .claude/skills/video-clone/scripts/timeline.py projects/<id> to see where time went and which gate kept sending work back, then decide what to change. Moving "what the final critic keeps rejecting" forward into the setup or build specs is the most effective speed-up.

4. Plug in another AI director

In app/server/agents/index.ts:

  1. Write xxxArgs(sessionId, cwd): command-line arguments for a non-interactive run that auto-approves file edits and can resume a session.
  2. Write parseXxx(obj, emit, st): turn the CLI's JSON stream into the common events: session {id}, text {text}, thinking {text}, tool {name, detail}, error {text}, and finally done {ok, text}.
  3. Add the new kind to runAgent and agentStatus, and an option to the picker in the web app's App.tsx.

The agent must be able to: read and write files in the repo, run a shell (python, node, ffmpeg), look at images (review depends on it) and resume a conversation.

5. Change the production line

Where each part lives in app/server/:

  • preprod.ts — preProduction(): plan core → characters and assets in parallel → plan frames.
  • production.ts — production(): setup → castGate() (runs alongside segment building) → runChunk() × N → assemble.
  • cast.ts — castGate() / castSerial(): per-character parallel review and fixing; the serial version is used for a single character or shared-file work.
  • final.ts — finalPanel(): final critic ⇄ revise.
  • review.ts — reviewFix(): the review → fix loop all of the above share. A new gate is a review, a verdict, a limit and a fix; the loop handles rounds, pauses and stopping at the limit.
  • turn.ts — turn(): dispatches one agent turn (session handling, global slots, required-file check, failure reason); step() is a turn that also moves the stage.
  • jobs.ts — the user's actions. Each runs as the project's single flow (runner.ts), so Cancel reaches all of it.

When changing a step, change three places together: the flow file above, prompts.ts (instructions) and CONTRACT.md (format of any new file; if the server acts on it, add its schema to contract.ts); plus Project.tsx if the web app should show it. Parameters (parallelism, rounds per gate) are in CONFIG and can also be set through environment variables.

6. Adding UI text / translations

Interface strings are written in Traditional Chinese in the components. app/web/src/i18n.ts translates the page at runtime: English comes from the EN table (plus EN_RE patterns for strings with numbers), Simplified Chinese is converted automatically with OpenCC. When you add a new string, add its English entry to EN; mark elements that must never be translated (user content, file names) with data-no-i18n.

7. Development

cd app
npm run server      # backend only (after editing server/, restart with: bash app/restart.sh)
npm run web         # UI dev server http://localhost:5173 (hot reload, /api proxied to 4318)
npm run build       # build app/dist for npm start / start.sh
npm run doctor      # environment check

Small tests: _smoke/ (local, not version-controlled) holds single-component check pages; the full check is to produce a real 30-second video, then look at the time breakdown with timeline.py and inspect the film frame by frame.