World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
Python agentagentic-aiaiclaudecopilotcursorelevenlabsffmpegfluximage-generationopen-sourceopenaipythonremotionstable-diffusiontext-to-speechtext-to-videovideo-generationvideo-production
Remove internal capability audit document
calesthio8d1920cf249b
diff --git a/docs/CAPABILITY_AUDIT.md b/docs/CAPABILITY_AUDIT.mddeleted file mode 100644index 4a3d5374..00000000--- a/docs/CAPABILITY_AUDIT.md+++ /dev/null@@ -1,402 +0,0 @@-# OpenMontage Capability Audit--> 2026-03-28 | Code-verified. Every finding traced to specific file + line.--**Core question:** When an agent discovers its capabilities via the registry and reads-the skills/manifests, does it get an accurate picture of what it can and can't do?--**Verdict:** No. The agent receives a mostly-accurate tool inventory but is actively-misled at the skill-to-tool boundary. Skills promise inputs tools don't accept, schemas-reject structures skills tell the agent to produce, and the discovery mechanism omits-the one thing the agent needs most: what inputs each tool takes.-------## 1. Discovery: What the Agent Learns at Boot--The agent runs:-```python-from tools.tool_registry import registry-registry.discover()-envelope = registry.support_envelope()-```--**What it gets per tool** (via `BaseTool.get_info()`):-- Identity: name, version, tier, capability, provider, stability-- Status: available/unavailable/degraded, dependencies list-- Runtime: execution_mode, determinism, runtime type, resource_profile-- Decision hints: best_for, not_good_for, provider_matrix, fallback_tools-- Skill links: agent_skills, related_skills-- Side effects, resume_support, user_visible_verification--**What it does NOT get:**-| Missing Field | Why It Matters |-|---|---|-| `input_schema` | Agent can't know what parameters a tool accepts without reading source code |-| `output_schema` | Agent can't know what a tool returns |-| `artifact_schema` | Agent can't know what files a tool produces |-| `estimate_cost()` results | Agent can't compare costs without calling each tool individually |--**This is the root cause of most downstream issues.** The agent knows *what tools exist* but not *what they accept or produce*. It must rely on skills for that information — and skills get it wrong in several places.--### Fix--Add to `BaseTool.get_info()`:-```python-"input_schema": self.input_schema,-"output_schema": self.output_schema,-"artifact_schema": self.artifact_schema,-```--This single change would let the agent validate skill instructions against actual tool contracts at runtime.-------## 2. Skills Tell the Agent to Pass Inputs Tools Don't Accept--These are cases where a stage-director skill instructs the agent to call a tool with-specific parameters, but the tool's `input_schema` and `execute()` don't read them.-The agent follows the skill, the extra fields are silently ignored, and the output-doesn't reflect the agent's intent.--### 2a. tts_selector: voice style, speaker directions, pronunciation--**Skill says** (`skills/pipelines/explainer/asset-director.md` lines 54-62):-> Apply speaker directions from the script (pace, emphasis, emotion).-> Apply the playbook's `audio.voice_style`.-> Include a pronunciation map in the TTS request for technical terms.--**Tool accepts** (`tools/audio/tts_selector.py` input_schema):-`text`, `preferred_provider`, `allowed_providers`, `output_path`--No fields for voice_style, speaker_directions, pace, emphasis, emotion, or pronunciation.-None of the concrete TTS tools (elevenlabs, openai, piper) accept these either — except-elevenlabs_tts which accepts `stability`, `similarity_boost`, `style` (numeric 0-1 floats),-not the semantic concepts the skill describes.--**Agent impact:** Agent is told to express creative direction through tool parameters that-don't exist. The narration will be generated with default voice settings regardless of what-the script specifies.--### 2b. image_selector: negative_prompt, consistency_anchors--**Skill says** (`skills/pipelines/explainer/asset-director.md` lines 68-73):-> Build the prompt: playbook.image_prompt_prefix + scene description + style cues.-> Add negative prompt from playbook.-> Include consistency anchors (same palette, same style across all images).--**Tool accepts** (`tools/graphics/image_selector.py` input_schema):-`prompt`, `preferred_provider`, `allowed_providers`, `output_path`--No `negative_prompt` or `consistency_anchors` field. The selector passes `prompt` through-to whichever provider it selects. Some downstream providers (flux_image) DO accept-`negative_prompt`, but the selector doesn't forward it.--**Agent impact:** Agent must flatten everything into a single `prompt` string. Negative-prompts and consistency anchors are lost unless the agent bypasses the selector and calls-flux_image directly — which contradicts the skill's guidance.--### 2c. video_compose "render" operation: options, output_profile--**Skill says** (`skills/pipelines/explainer/compose-director.md` lines 66-80):-```json-{- "operation": "render",- "edit_decisions": {...},- "asset_manifest": {...},- "output_profile": "youtube_landscape",- "options": {- "subtitle_burn": true,- "audio_normalize": true,- "two_pass_encode": true- }-}-```--**Tool accepts** (`tools/video/video_compose.py` input_schema):-`operation`, `input_path`, `output_path`, `edit_decisions`, `subtitle_path`,-`subtitle_style`, `overlays`, `audio_path`, `profile`, `codec`, `crf`, `preset`--- `asset_manifest`: NOT in input_schema but IS read by `_render()` (line 271). Works but undocumented.-- `output_profile`: Not accepted. The field is `profile` and only used by `_encode()`, not `_render()`.-- `options`: Not accepted at all. `subtitle_burn`, `audio_normalize`, `two_pass_encode` are silently dropped.--**Agent impact:** Agent follows the skill, passes these fields, they vanish. Output won't-match target platform. No subtitle burn, no normalization, no two-pass encode.--### 2d. audio_mixer: skill implies single call, tool requires multiple--**Skill says** (`skills/pipelines/explainer/compose-director.md` lines 87-95):-> Call the audio_mixer tool to:-> 1. Layer narration segments in order-> 2. Mix background music at playbook volume-> 3. Apply ducking (music dips during narration)-> 4. Normalize overall audio levels--**Tool accepts** (`tools/audio/audio_mixer.py`):-Three discrete operations: `mix`, `duck`, `extract`. Each is a separate `execute()` call.-There is no single-call "mix everything together" mode.--**Agent impact:** Agent attempts a single call with all requirements. The tool handles one-operation per call. Agent must orchestrate: mix narration tracks first, then duck with-music, then the compose step muxes audio into video. The skill doesn't explain this-multi-step choreography.-------## 3. Skills Tell the Agent to Produce Artifacts Schemas Reject--These are cases where a skill shows the agent an example JSON structure for an artifact,-but the corresponding JSON Schema has `additionalProperties: false` and rejects the-skill's fields.--### 3a. edit_decisions: transforms and transitions--**Skill says** (`skills/pipelines/explainer/edit-director.md` lines 38-53):-```json-{- "id": "cut-1",- "source": "img-scene-1",- "in_seconds": 0,- "out_seconds": 10,- "layer": "primary",- "transform": {- "scale": 1.0,- "position": "center",- "animation": "ken-burns-slow-zoom"- },- "transition_in": "fade",- "transition_out": "dissolve",- "transition_duration": 0.4-}-```--**Schema allows** (`schemas/artifacts/edit_decisions.schema.json` lines 12-24):-```json-{- "required": ["id", "source", "in_seconds", "out_seconds"],- "properties": {- "id": {}, "source": {}, "in_seconds": {}, "out_seconds": {},- "speed": {}, "reason": {}- },- "additionalProperties": false-}-```--**Rejected fields:** `layer`, `transform` (and all sub-fields), `transition_in`,-`transition_out`, `transition_duration`.--**Agent impact:** Agent builds the edit_decisions artifact per skill instructions.-Checkpoint validation rejects it. Agent is stuck between following the skill and-passing schema validation. The schema also means `video_compose._compose()` will-never receive transition/transform data even if it could handle it — which it can't-(it only does hard cuts via concat).--### 3b. edit_decisions: subtitle styling--**Skill says** (edit-director.md lines 62-78):-```json-"subtitles": {- "enabled": true,- "style": "word-by-word",- "font": "Inter",- "font_size": 48,- "color": "#FFFFFF",- "background": "#00000088",- "position": "bottom-center",- "max_words_per_line": 8-}-```--**Schema allows** (edit_decisions.schema.json lines 51-58):-```json-"subtitles": {- "properties": {- "enabled": { "type": "boolean" },- "style": { "type": "string" },- "source": { "type": "string" }- }-}-```--**Rejected fields:** `font`, `font_size`, `color`, `background`, `position`,-`max_words_per_line`.--**Agent impact:** All subtitle styling information is lost at schema validation.-The compose stage gets `enabled: true` and `style: "word-by-word"` but no font,-color, or positioning data. Subtitle styling must come from video_compose's-`subtitle_style` input instead — but the edit-director skill doesn't mention this path.--### 3c. edit_decisions: narration and SFX audio configuration--**Skill says** (edit-director.md lines 84-110):-```json-"audio": {- "narration": {- "segments": [- { "asset_id": "narration-s1", "start_seconds": 0 }- ]- },- "music": {- "asset_id": "...",- "ducking": {- "enabled": true,- "threshold_db": -3,- "reduction_db": -8,- "attack_ms": 200,- "release_ms": 500- }- },- "sfx": []-}-```--**Schema allows** (edit_decisions.schema.json lines 59-67):-`music` with `asset_id`, `volume`, `ducking` (boolean only), `fade_in_seconds`, `fade_out_seconds`.--**Rejected:** Entire `audio.narration` section, entire `audio.sfx` section, structured-`ducking` object (schema only accepts boolean). Narration segment timing and SFX-have no home in the validated artifact.-------## 4. Tools Lie About What They Can Do--### 4a. video_compose._render() claims image-to-video conversion--**Docstring** (`tools/video/video_compose.py` line 267):-> "It orchestrates: image-to-video conversion, concatenation, audio mixing,-> subtitle burn-in, and final encoding to target profile."--**Actual code** (lines 280-302): Resolves asset IDs from asset_manifest to file paths,-then calls `_compose()`. No image-to-video conversion. `_compose()` uses `ffmpeg -ss/-to`-on each source, which fails on still images (PNG/JPG have no temporal dimension to seek into).--**Agent impact:** Agent trusts that the "render" operation handles images. It doesn't.-Explainer pipelines where most cuts are generated images will produce broken output.--### 4b. subtitle_gen declares highlight_style but ignores it--**Input schema** (`tools/subtitle/subtitle_gen.py`): includes `highlight_style` parameter.--**Execute method:** Never reads `highlight_style`. It's a phantom parameter.--**Agent impact:** Low — agent might pass highlight_style expecting highlighted captions-and get plain ones.--### 4c. video_selector declares operation parameter but ignores it--**Input schema** (`tools/video/video_selector.py`): includes `operation` field.--**Execute method:** Reads it but never passes it to the selected provider tool.--**Agent impact:** Low — operation context is lost during delegation but most video-providers only do one thing anyway.-------## 5. Phantom Tools in Pipeline Manifests--`pipeline_defs/framework-smoke.yaml` references three tools that don't exist:--| Tool Name | Stage | Field |-|---|---|---|-| `idea_explorer_llm` | research | preferred_tools |-| `script_writer_llm` | script | preferred_tools |-| `script_writer_template` | script | fallback_tools |--`registry.discover()` will not find these. If the agent tries to resolve them, it gets-nothing. These appear to be placeholders from an earlier design where LLM capabilities-were modeled as tools.--**Agent impact:** framework-smoke pipeline is broken. Other 10 pipelines are clean —-all their tool references resolve to real tools.-------## 6. Unused Tools the Agent Doesn't Know to Use--Five tools exist in `tools/` but no pipeline manifest references them:--| Tool | What It Does | Why It's Missing |-|---|---|---|-| `bg_remove` | Background removal (rembg/U2Net) | Useful for compositing, overlays. No skill mentions it. |-| `face_restore` | Face restoration (CodeFormer/GFPGAN) | Useful for low-res faces in talking-head. No skill mentions it. |-| `upscale` | Image/video upscaling (Real-ESRGAN) | Useful for enhancement pass. No skill mentions it. |-| `video_understand` | Vision-language analysis (CLIP/BLIP-2) | Could be used for quality review. No skill mentions it. |-| `image_gen` | Legacy multi-provider image gen | Deprecated, replaced by image_selector. Expected. |--The agent discovers these via `support_envelope()` but has no skill guidance on when or-how to use them. They're invisible capabilities — the agent CAN use them but WON'T-unless it independently decides to.-------## 7. The Playbook Gap: Declared Knowledge the Agent Can't Apply--Style playbooks (`styles/*.yaml`) define detailed visual contracts:-`image_prompt_prefix`, `negative_prompt`, `consistency_anchors`, `typography`,-`motion.transition_duration_seconds`, `audio.music_mood`, `quality_rules`.--Skills extensively reference these: "apply the playbook's voice_style", "use playbook's-image_prompt_prefix", "set music volume per playbook". **30+ skill references across-all pipelines.**--But:-- Zero tools import `playbook_loader` or read playbook data-- The selector tools (tts_selector, image_selector) have no playbook-aware parameters-- The agent must: load the YAML, extract values, manually inject them into tool calls--This works IF the agent knows to do it. But the tools give no signal that they expect-playbook-derived values. The playbook system is a skill-layer convention that the tool-layer is unaware of. This is architecturally fine for an agent-first system — but the-skills need to be explicit about HOW the agent should bridge this (e.g., "read-`playbook.asset_generation.image_prompt_prefix` and prepend it to the `prompt` field-when calling image_selector"). Currently skills say WHAT to apply but not HOW to-translate playbook values into tool parameters.-------## 8. Missing Skill: Talking-Head Executive Producer--`pipeline_defs/talking-head.yaml` declares `orchestration.skill: pipelines/talking-head/executive-producer` but the file `skills/pipelines/talking-head/executive-producer.md` does not exist. All other 9 pipelines with orchestration have their EP skill.--**Agent impact:** Agent following the talking-head pipeline reads the manifest, tries to-load the EP skill, gets a file-not-found. Must improvise orchestration without guidance.-------## Summary: What the Agent Gets Right vs. Wrong--### Accurate (agent can trust these)-- Tool inventory: 44/47 tool references in manifests resolve correctly (3 phantoms in framework-smoke only)-- Tool status: `get_status()` accurately reports availability based on real dependency checks-- Fallback chains: All `fallback_tools` references point to tools that exist-- Tool self-description: 10/12 tools audited have accurate input_schema vs execute() behavior-- Pipeline stage ordering and artifact flow: checkpoint system matches manifests-- Cost estimates: Tools that implement `estimate_cost()` return realistic numbers--### Inaccurate (agent will be misled by these)-| Issue | Count | Root Cause |-|---|---|---|-| Skills pass inputs tools don't accept | 4 cases | Skills written to aspirational API, not actual tool schema |-| Skills produce artifacts schemas reject | 3 cases | Schemas stricter than skills expect (additionalProperties: false) |-| Tool overclaims capability | 1 case | video_compose._render() docstring lies about image handling |-| Discovery omits input/output schemas | All tools | get_info() doesn't include schemas |-| Phantom tools in manifests | 3 tools | framework-smoke has placeholder references |-| Missing EP skill | 1 pipeline | talking-head pipeline incomplete |--### Recommended Fix Priority--| # | Fix | Effort | Effect |-|---|---|---|---|-| 1 | Add input_schema/output_schema to `get_info()` | 3 lines | Agent can self-validate against skills |-| 2 | Align edit_decisions.schema.json with edit-director skills | Medium | Unblocks transitions, subtitle styling, narration config |-| 3 | Add image-to-video loop in video_compose._compose() | Small | Unblocks all image-based pipelines |-| 4 | Add negative_prompt passthrough to image_selector | Small | Enables playbook integration for image gen |-| 5 | Add voice_id/style passthrough to tts_selector | Small | Enables creative direction for narration |-| 6 | Fix compose-director to match video_compose's actual interface | Small | Agent stops passing phantom options |-| 7 | Fix compose-director to document multi-call audio_mixer flow | Small | Agent knows to call mix then duck |-| 8 | Remove phantom tools from framework-smoke.yaml | Trivial | Clean manifest |-| 9 | Create talking-head executive-producer.md | Medium | Completes pipeline |-| 10 | Document playbook-to-tool bridging pattern in skills | Medium | Agent knows HOW to apply playbook values |