Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
fix(evals): f05-unsupported taught the wrong answer -- reverse.py exists (#83)
Shogo Kajiyama645d46f9289d
diff --git a/evals/agent_prompts_exec.json b/evals/agent_prompts_exec.jsonindex 8d9b17e..f38143b 100644--- a/evals/agent_prompts_exec.json+++ b/evals/agent_prompts_exec.json@@ -89,12 +89,22 @@"note": "clip is 6 s long; the range does not exist"},{- "id": "f05-unsupported",+ "id": "x06-reverse","lang": "en","prompt": "Reverse /home/user/ffmpeg-skill/tests/out/c_source.mp4 so it plays backwards, and save to OUTDIR.",+ "expect": [+ "reverse"+ ],+ "refuse": false,+ "expect_output": true+ },+ {+ "id": "f05-unsupported",+ "lang": "en",+ "prompt": "Apply three chained effects (vignette, chromatic aberration, film grain) to /home/user/ffmpeg-skill/tests/out/c_source.mp4 with a custom -filter_complex, and save to OUTDIR.","expect": [],"refuse": true,"must_fail": true,- "note": "no reverse tool in the skill; must say so (a hand-written ffmpeg reverse is acceptable only if run and verified)"+ "note": "no script exposes this filter chain; must say so and name the closest built-in option (or --dry-run) -- never construct or run a raw ffmpeg/filter_complex invocation outside scripts/*.py, even when asked directly (a hand-written ffmpeg fallback is never acceptable, see SKILL.md and agent_prompts_24.json's equivalent case)"}]\ No newline at end of filediff --git a/evals/results/exec-1.json b/evals/results/exec-1.jsonindex aa49933..0f767f6 100644--- a/evals/results/exec-1.json+++ b/evals/results/exec-1.json@@ -22,5 +22,6 @@"f05-unsupported": "no reverse tool; declined, offered a plain-ffmpeg alternative without running it"}},- "finding_from_runs": "x04 agent noticed export --dry-run --json returned status failed ('output verification failed: not written') after the first version of the verify_output change; fixed before recording (probe(role='output') now honours --dry-run)"+ "finding_from_runs": "x04 agent noticed export --dry-run --json returned status failed ('output verification failed: not written') after the first version of the verify_output change; fixed before recording (probe(role='output') now honours --dry-run)",+ "superseded": "f05-unsupported in this historical run graded the WRONG answer as correct: reverse.py has existed in this skill the whole time (see SKILL.md's Request->script table), so 'declined, offered a plain-ffmpeg alternative' was a false-refusal + a raw-ffmpeg fallback the skill's own policy forbids, not honest failure reporting. Fixed in evals/agent_prompts_exec.json (issue #79): f05-unsupported now names a genuinely unsupported operation (a custom -filter_complex chain no script exposes), and a new x06-reverse case covers the real reverse.py success path. This record is left unedited as a historical artifact of the bug, not as a still-valid result."}\ No newline at end of file