Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

test_contract.py: catch a docstring example naming a flag that doesn't exist (#30)

Shogo Kajiyamacb8af5f41676
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4009928..8f9c2fe 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,7 @@
## Unreleased — FFmpeg 8+ / Windows compatibility for caption and color
+- **Test: docstring examples can't drift from the parser.** `scenes.py`'s docstring once claimed a ranking option (motion) that was never implemented; that was prose, not an example, so nothing caught it. A new `test_contract.py` test at least closes the more common version of this gap: every `--flag` used in a script's own `Examples:` lines must exist in that script's real argparse parser, checked via the contract's `input_schema`. Verified to catch a deliberately introduced typo before writing this entry. Does not (and cannot) catch a false claim made only in prose.
- **SKILL.md: explicit language-matching instruction.** The agent has always been graded on replying in the user's language (see the 24-prompt eval), but SKILL.md never actually said to — it worked by the model's own default, not by instruction. The Report format section now says explicitly: reply in whatever language the request was written in (Japanese, English, Chinese, or any other), keep only the field labels (`Done:`, `Steps:`, ...) in English, and follow a language switch mid-conversation. No code change; this only affects the model-facing instructions.
- **`multicam.py` low-confidence warning.** `sync.py` warns on stderr when its cross-correlation confidence is below 0.1 ("check that both files contain the same audio event"); `multicam.py` used the same measurement per camera but never warned, even though it applies the offset to a rendered cut rather than just reporting it. It now warns per camera below the same threshold. SKILL.md's existing "check `confidence` before trusting a sync" guidance now names `multicam.py`'s per-camera confidence explicitly.
- **`sync.py`/`multicam.py` documentation: audio sync is not lip sync.** Both align audio tracks to each other by cross-correlation; neither has ever done any face or mouth detection, and a high `confidence` only means the audio matched well, not that the final picture looks in sync. This was previously undocumented; `sync.py`'s docstring, `multicam.py`'s docstring and SKILL.md now say so explicitly. No behaviour change.
diff --git a/tests/test_contract.py b/tests/test_contract.py
index fc830ab..f94b6a2 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -128,6 +128,31 @@ class ContractTests(unittest.TestCase):
self.assertEqual(p["lifecycle"], "EXPERIMENTAL")
self.assertEqual(set(p), {"id", "lifecycle", "tool_id"})
+ def test_docstring_examples_use_flags_that_actually_exist(self):
+ """A docstring's own runnable examples are the first thing a reader tries and trusts.
+ scenes.py once claimed (in prose, not an example) that highlight ranking used motion --
+ it never did. This can't catch a false prose claim, but it does catch the more common
+ drift: an Examples: line for script X naming a --flag that X's own parser doesn't have
+ (renamed, removed, or typo'd), which is exactly the kind of docs-vs-code gap that let
+ that claim go unnoticed for as long as it did."""
+ cli_by_tool = {t["name"]: set() for t in self.contract["tools"]}
+ for t in self.contract["tools"]:
+ for prop in t["input_schema"]["properties"].values():
+ cli_by_tool[t["name"]].update(prop.get("cli") or [])
+ flag_re = re.compile(r"(--[a-z][a-z0-9-]*)")
+ for script in sorted(SCRIPTS.glob("*.py")):
+ if script.name.startswith("_"):
+ continue
+ name = script.stem
+ doc = script.read_text(encoding="utf-8").split('"""')[1]
+ for line in doc.splitlines():
+ line = line.strip()
+ if not line.startswith(f"python3 {name}.py"):
+ continue
+ code = line.split("#", 1)[0]
+ for flag in flag_re.findall(code):
+ self.assertIn(flag, cli_by_tool[name], f"{name}.py's own docstring example uses {flag}, which its parser does not have: {line!r}")
+
def test_every_tool_executable_exists_and_internal_scripts_are_hidden(self):
for t in self.contract["tools"]:
self.assertTrue((ROOT / t["executable"]).is_file(), t["executable"])