Skip to content

World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.

Add Higgsfield provider and update Runway to v0.2.0

calesthio4f682c8b0a2b
diff --git a/AGENT_GUIDE.md b/AGENT_GUIDE.md
index 74dd4c7e..3508302b 100644
--- a/AGENT_GUIDE.md
+++ b/AGENT_GUIDE.md
@@ -257,7 +257,7 @@ This returns every capability grouped by status — how many providers the user
```
YOUR CAPABILITIES
- Video Generation: 0/12 configured
+ Video Generation: 0/13 configured
Image Generation: 1/7 configured
Text-to-Speech: 1/3 configured
Music Generation: 1/1 configured
@@ -272,7 +272,7 @@ For EACH capability with unavailable providers, read the `install_instructions`
```
QUICK SETUP OPTIONS (1-minute each — set an env var in .env)
- Video Generation (0/12 -> unlock the biggest upgrade):
+ Video Generation (0/13 -> unlock the biggest upgrade):
Each unavailable provider lists its own install_instructions.
Read them from the provider_menu output and present grouped by env var.
Example: if 3 tools need FAL_KEY, group them: "FAL_KEY unlocks 3 providers"
diff --git a/README.md b/README.md
index 7d88eccf..5c7a37da 100644
--- a/README.md
+++ b/README.md
@@ -273,7 +273,7 @@ Most AI video tools give you a single clip from a prompt. OpenMontage gives you
Edit your own talking-head footage. Generate a fully animated explainer from scratch. Cut a 2-hour podcast into a dozen social clips. Translate and dub your content into 10 languages. Build a cinematic brand teaser from stock footage and AI-generated scenes. **If a production team can make it, OpenMontage can orchestrate it.**
- **11 production pipelines** — explainers, talking heads, screen demos, cinematic trailers, animations, podcasts, localization, and more
-- **51 production tools** — spanning video generation, image creation, text-to-speech, music, audio mixing, subtitles, enhancement, and analysis
+- **52 production tools** — spanning video generation, image creation, text-to-speech, music, audio mixing, subtitles, enhancement, and analysis
- **400+ agent skills** — production skills, pipeline directors, creative techniques, quality checklists, and deep technology knowledge packs that teach the agent how to use every tool like an expert
- **Reference-driven creation** — paste a video you like and the agent turns it into a grounded, differentiated production plan instead of forcing you to invent the perfect prompt from scratch
- **Live web research built in** — before writing a single word of script, the agent runs 15-25+ web searches across YouTube, Reddit, news sites, and academic sources to ground your video in real, current data
@@ -370,14 +370,15 @@ Each tool declares which Layer 3 skills it relies on. The agent reads Layer 1 to
> **Full setup guide with pricing and free tiers:** [`docs/PROVIDERS.md`](docs/PROVIDERS.md)
<details>
-<summary><strong>Video Generation — 12 providers</strong></summary>
+<summary><strong>Video Generation — 13 providers</strong></summary>
| Provider | Type | Notes |
|----------|------|-------|
| **Kling** | Cloud API | High quality, fast |
-| **Runway Gen-4** | Cloud API | Cinematic quality |
+| **Runway Gen-4** | Cloud API | Cinematic quality, Gen-3 Alpha Turbo / Gen-4 Turbo / Gen-4 Aleph |
| **Google Veo 3** | Cloud API | Long-form, cinematic. Via fal.ai or HeyGen. |
| **Grok Imagine Video** | Cloud API | Strong reference-image video and xAI-native short-form generation |
+| **Higgsfield** | Cloud API | Multi-model orchestrator with Soul ID for character consistency |
| **MiniMax** | Cloud API | Cost-effective |
| **HeyGen** | Cloud API | Multi-model gateway |
| **WAN 2.1** | Local GPU | Free, 1.3B and 14B variants |
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 0ac2b5df..ce3b3f6c 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -51,7 +51,7 @@ OpenMontage/
│ ├── graphics/ # Image gen (FLUX, DALL-E, Recraft, local diffusion), stock, diagrams, code snippets, math animation
│ ├── publishers/ # (Reserved)
│ ├── subtitle/ # SRT/VTT generation from timestamps
-│ └── video/ # 12 video gen providers, composition, stitching, trimming
+│ └── video/ # 13 video gen providers, composition, stitching, trimming
│
├── pipeline_defs/ # 11 YAML pipeline manifests
├── schemas/ # JSON Schema definitions for validation
@@ -162,7 +162,7 @@ Selectors route based on: user preference > availability > fallback order. They
**Subtitle (1):** subtitle_gen
-**Video (17):** grok_video, heygen_video, veo_video, kling_video, runway_video, minimax_video, wan_video, hunyuan_video, cogvideo_video, ltx_video_local, ltx_video_modal, pexels_video, pixabay_video, video_selector, video_compose (FFmpeg), video_stitch, video_trimmer
+**Video (18):** grok_video, heygen_video, higgsfield_video, veo_video, kling_video, runway_video, minimax_video, wan_video, hunyuan_video, cogvideo_video, ltx_video_local, ltx_video_modal, pexels_video, pixabay_video, video_selector, video_compose (FFmpeg), video_stitch, video_trimmer
---
@@ -384,7 +384,8 @@ All config is validated via Pydantic models in `lib/config_model.py`.
| `PEXELS_API_KEY` | pexels_image, pexels_video | Stock media |
| `PIXABAY_API_KEY` | pixabay_image, pixabay_video | Stock media |
| `GOOGLE_API_KEY` | google_imagen, google_tts | Google Imagen images, Google Cloud TTS |
-| `RUNWAY_API_KEY` | runway_video | Runway Gen-4 direct |
+| `RUNWAY_API_KEY` | runway_video | Runway Gen-3/Gen-4 direct |
+| `HIGGSFIELD_API_KEY` + `HIGGSFIELD_API_SECRET` | higgsfield_video | Higgsfield multi-model video |
| `MODAL_LTX2_ENDPOINT_URL` | ltx_video_modal | Self-hosted LTX-2 |
| `VIDEO_GEN_LOCAL_ENABLED` | local video tools | Enable local GPU generation |
| `VIDEO_GEN_LOCAL_MODEL` | wan, hunyuan, ltx, cogvideo | Select local model |
diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md
index 3e61988b..ed3ebce1 100644
--- a/docs/PROVIDERS.md
+++ b/docs/PROVIDERS.md
@@ -259,20 +259,19 @@ Google TTS offers 700+ voices across 50+ languages. Voice names follow the patte
---
-### Runway — Gen-4 Video
+### Runway — Gen-3/Gen-4 Video
-> **Highest-rated AI video quality.** #1 on Elo rankings. Professional-grade video generation.
+> **Highest-rated AI video quality.** #1 on Elo rankings. Professional-grade video generation with Gen-3 Alpha Turbo, Gen-4 Turbo, and Gen-4 Aleph models.
**Tools unlocked:** `runway_video`
**Env var:** `RUNWAY_API_KEY`
#### Setup
-1. Go to [app.runwayml.com/signup](https://app.runwayml.com/signup) and create an account
+1. Go to [dev.runwayml.com](https://dev.runwayml.com/) and create a developer account
2. Subscribe to a paid plan (Standard or above — API requires subscription)
-3. Go to **Settings > API Keys** at [app.runwayml.com/settings/api-keys](https://app.runwayml.com/settings/api-keys)
-4. Click **Create API Key**, copy it
-5. Add to `.env`: `RUNWAY_API_KEY=key_...`
+3. Generate an API key from the developer portal
+4. Add to `.env`: `RUNWAY_API_KEY=key_...`
#### Pricing
@@ -287,14 +286,56 @@ Google TTS offers 700+ voices across 50+ languages. Voice names follow the patte
| Model | Price per second |
|-------|-----------------|
+| Gen-3 Alpha Turbo | ~$0.05 |
| Gen-4 Turbo | ~$0.05 |
-| Gen-4 | ~$0.10 |
-| Gen-4.5 | ~$0.25 |
+| Gen-4 Aleph | ~$0.15 |
**Free tier:** 125 one-time credits (no monthly renewal). Enough for about 5 seconds of Gen-4 video. API access requires a paid subscription.
---
+### Higgsfield — Multi-Model Video Orchestrator
+
+> **Multi-model video platform.** Routes to Kling 3.0, Veo 3.1, Sora 2, WAN 2.5, and proprietary Soul Cinema through a single API. Includes Soul ID for character consistency across clips.
+
+**Tools unlocked:** `higgsfield_video`
+**Env vars:** `HIGGSFIELD_API_KEY` + `HIGGSFIELD_API_SECRET` (or combined `HIGGSFIELD_KEY=key:secret`)
+
+#### Setup
+
+1. Go to [cloud.higgsfield.ai](https://cloud.higgsfield.ai/) and create an account
+2. Subscribe to a plan (Starter or above for API access)
+3. Navigate to API Keys section at [cloud.higgsfield.ai/api-keys](https://cloud.higgsfield.ai/api-keys)
+4. Generate an API key and secret
+5. Add to `.env`:
+ ```
+ HIGGSFIELD_API_KEY=your-api-key
+ HIGGSFIELD_API_SECRET=your-api-secret
+ ```
+
+#### Pricing
+
+| Plan | Price | Notes |
+|------|-------|-------|
+| Free | $0 | Limited credits |
+| Starter | $15/mo | Basic allocation |
+| Plus | $34/mo | Mid-tier, ~33-56 Kling 3.0 clips |
+| Ultra | $84/mo | High volume |
+
+**Per-generation costs (approximate, via credits):**
+
+| Model | Cost per clip |
+|-------|--------------|
+| Kling 3.0 | ~$0.10 (cheapest) |
+| WAN 2.5 | ~$0.10 |
+| Soul Cinema | ~$0.15 |
+| Veo 3.1 | ~$0.50 |
+| Sora 2 | ~$0.50 |
+
+**Free tier:** Limited credits on signup. No monthly renewal on free plan.
+
+---
+
### HeyGen — Avatar Video Gateway
> **Multi-model video gateway.** Access VEO, Sora, Runway, Kling, and Seedance through a single API.
@@ -603,6 +644,7 @@ These tools require only FFmpeg or Python packages — no GPU, no API key.
| **OpenAI** | `OPENAI_API_KEY` | `openai_tts`, `openai_image` | Paid only |
| **xAI** | `XAI_API_KEY` | `grok_image`, `grok_video` | Paid only |
| **Runway** | `RUNWAY_API_KEY` | `runway_video` | Free trial + paid |
+| **Higgsfield** | `HIGGSFIELD_API_KEY` + `HIGGSFIELD_API_SECRET` | `higgsfield_video` | Subscription ($15-84/mo) |
| **HeyGen** | `HEYGEN_API_KEY` | `heygen_video` | Pay-as-you-go |
| **Suno** | `SUNO_API_KEY` | `suno_music` | Pay-as-you-go |
| **Local GPU** | `VIDEO_GEN_LOCAL_ENABLED` | `wan_video`, `hunyuan_video`, `cogvideo_video`, `ltx_video_local` | Free (GPU required) |
@@ -618,7 +660,7 @@ How many providers cover each capability:
| Capability | Cloud Providers | Local Providers | Free Options |
|-----------|----------------|-----------------|--------------|
| **Image Generation** | FLUX, Grok, Google Imagen, DALL-E 3, Recraft | Local Diffusion | Pexels, Pixabay (stock) |
-| **Video Generation** | Grok, Kling, Runway, Veo, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
+| **Video Generation** | Grok, Kling, Runway, Veo, Higgsfield, MiniMax, HeyGen | WAN, Hunyuan, CogVideo, LTX | Pexels, Pixabay (stock) |
| **Text-to-Speech** | ElevenLabs, Google TTS, OpenAI | Piper | Piper, Google free tier, ElevenLabs free tier |
| **Music Generation** | ElevenLabs, Suno | — | ElevenLabs free tier |
| **Post-Production** | — | FFmpeg (compose, stitch, trim, mix, enhance, grade) | All free |
diff --git a/tools/video/higgsfield_video.py b/tools/video/higgsfield_video.py
new file mode 100644
index 00000000..aaf83a2c
--- /dev/null
+++ b/tools/video/higgsfield_video.py
@@ -0,0 +1,243 @@
+"""Higgsfield video generation via Higgsfield Cloud API.
+
+Multi-model orchestrator with proprietary Soul model for character-consistent,
+photorealistic video generation. Routes to Kling, Veo, Sora, and WAN under the hood.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+from pathlib import Path
+from typing import Any
+
+from tools.base_tool import (
+ BaseTool,
+ Determinism,
+ ExecutionMode,
+ ResourceProfile,
+ RetryPolicy,
+ ToolResult,
+ ToolRuntime,
+ ToolStability,
+ ToolStatus,
+ ToolTier,
+)
+
+
+class HiggsFieldVideo(BaseTool):
+ name = "higgsfield_video"
+ version = "0.1.0"
+ tier = ToolTier.GENERATE
+ capability = "video_generation"
+ provider = "higgsfield"
+ stability = ToolStability.EXPERIMENTAL
+ execution_mode = ExecutionMode.SYNC
+ determinism = Determinism.STOCHASTIC
+ runtime = ToolRuntime.API
+
+ dependencies = []
+ install_instructions = (
+ "Set HIGGSFIELD_API_KEY and HIGGSFIELD_API_SECRET for your Higgsfield Cloud credentials.\n"
+ " Get them at https://cloud.higgsfield.ai/api-keys\n"
+ " Alternatively, set HIGGSFIELD_KEY as a combined key:secret value."
+ )
+ agent_skills = ["ai-video-gen"]
+
+ capabilities = ["text_to_video", "image_to_video"]
+ supports = {
+ "text_to_video": True,
+ "image_to_video": True,
+ "character_consistency": True,
+ "multi_model_routing": True,
+ }
+ best_for = [
+ "character-consistent video generation (Soul ID)",
+ "multi-model access through a single API",
+ "photorealistic and fashion-aware content",
+ ]
+ not_good_for = ["offline generation", "fine-grained model control", "budget projects without subscription"]
+ fallback_tools = ["kling_video", "veo_video", "minimax_video"]
+
+ input_schema = {
+ "type": "object",
+ "required": ["prompt"],
+ "properties": {
+ "prompt": {"type": "string"},
+ "operation": {
+ "type": "string",
+ "enum": ["text_to_video", "image_to_video"],
+ "default": "text_to_video",
+ },
+ "model": {
+ "type": "string",
+ "enum": [
+ "kling_3.0",
+ "veo_3.1",
+ "sora_2",
+ "wan_2.5",
+ "soul_cinema",
+ ],
+ "default": "kling_3.0",
+ "description": "Underlying model to use for generation",
+ },
+ "duration": {
+ "type": "string",
+ "enum": ["5", "10", "15"],
+ "default": "5",
+ "description": "Duration in seconds (availability varies by model)",
+ },
+ "aspect_ratio": {
+ "type": "string",
+ "enum": ["16:9", "9:16", "1:1", "21:9"],
+ "default": "16:9",
+ },
+ "image_url": {"type": "string", "description": "Reference image URL for image_to_video"},
+ "output_path": {"type": "string"},
+ },
+ }
+
+ resource_profile = ResourceProfile(
+ cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
+ )
+ retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
+ idempotency_key_fields = ["prompt", "model", "operation", "duration"]
+ side_effects = ["writes video file to output_path", "calls Higgsfield Cloud API"]
+ user_visible_verification = ["Watch generated clip for motion coherence and visual quality"]
+
+ def _get_credentials(self) -> tuple[str, str] | None:
+ """Return (api_key, api_secret) or None if not configured."""
+ combined = os.environ.get("HIGGSFIELD_KEY")
+ if combined and ":" in combined:
+ key, secret = combined.split(":", 1)
+ return key, secret
+ key = os.environ.get("HIGGSFIELD_API_KEY")
+ secret = os.environ.get("HIGGSFIELD_API_SECRET")
+ if key and secret:
+ return key, secret
+ return None
+
+ def get_status(self) -> ToolStatus:
+ if self._get_credentials():
+ return ToolStatus.AVAILABLE
+ return ToolStatus.UNAVAILABLE
+
+ def estimate_cost(self, inputs: dict[str, Any]) -> float:
+ model = inputs.get("model", "kling_3.0")
+ duration = int(inputs.get("duration", "5"))
+ # Approximate per-clip costs based on Higgsfield credit pricing
+ base_costs = {
+ "kling_3.0": 0.10,
+ "wan_2.5": 0.10,
+ "veo_3.1": 0.50,
+ "sora_2": 0.50,
+ "soul_cinema": 0.15,
+ }
+ base = base_costs.get(model, 0.15)
+ return base * (duration / 5)
+
+ def estimate_runtime(self, inputs: dict[str, Any]) -> float:
+ model = inputs.get("model", "kling_3.0")
+ if model in ("veo_3.1", "sora_2"):
+ return 120.0
+ return 60.0
+
+ def execute(self, inputs: dict[str, Any]) -> ToolResult:
+ creds = self._get_credentials()
+ if not creds:
+ return ToolResult(
+ success=False,
+ error="Higgsfield credentials not set. " + self.install_instructions,
+ )
+
+ import requests
+
+ api_key, api_secret = creds
+ start = time.time()
+ operation = inputs.get("operation", "text_to_video")
+ model = inputs.get("model", "kling_3.0")
+
+ payload: dict[str, Any] = {
+ "prompt": inputs["prompt"],
+ "model": model,
+ "task": operation.replace("_", "-"),
+ }
+ if inputs.get("duration"):
+ payload["duration"] = int(inputs["duration"])
+ if inputs.get("aspect_ratio"):
+ payload["aspect_ratio"] = inputs["aspect_ratio"]
+ if operation == "image_to_video" and inputs.get("image_url"):
+ payload["image_url"] = inputs["image_url"]
+
+ headers = {
+ "Authorization": f"Bearer {api_key}",
+ "X-API-Secret": api_secret,
+ "Content-Type": "application/json",
+ }
+
+ try:
+ # Submit generation request
+ submit_resp = requests.post(
+ "https://platform.higgsfield.ai/v1/generations",
+ headers=headers,
+ json=payload,
+ timeout=30,
+ )
+ submit_resp.raise_for_status()
+ gen_data = submit_resp.json()
+ generation_id = gen_data["id"]
+ status_url = gen_data.get("status_url", f"https://platform.higgsfield.ai/v1/generations/{generation_id}")
+
+ # Poll for completion
+ video_url = None
+ for _ in range(72): # max ~6 minutes
+ time.sleep(5)
+ poll_resp = requests.get(status_url, headers=headers, timeout=15)
+ poll_resp.raise_for_status()
+ poll_data = poll_resp.json()
+ status = poll_data.get("status", "Unknown")
+
+ if status in ("Completed", "COMPLETED"):
+ video_url = poll_data.get("output_url") or poll_data.get("url")
+ break
+ if status in ("Failed", "FAILED", "NSFW", "Cancelled", "CANCELLED"):
+ return ToolResult(
+ success=False,
+ error=f"Higgsfield generation {status}: {poll_data.get('error', 'unknown')}",
+ )
+
+ if not video_url:
+ return ToolResult(success=False, error="Higgsfield generation timed out.")
+
+ # Download video
+ video_response = requests.get(video_url, timeout=120)
+ video_response.raise_for_status()
+
+ output_path = Path(inputs.get("output_path", "higgsfield_output.mp4"))
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ output_path.write_bytes(video_response.content)
+
+ except Exception as e:
+ return ToolResult(success=False, error=f"Higgsfield video generation failed: {e}")
+
+ from tools.video._shared import probe_output
+
+ probed = probe_output(output_path)
+ return ToolResult(
+ success=True,
+ data={
+ "provider": "higgsfield",
+ "model": model,
+ "prompt": inputs["prompt"],
+ "operation": operation,
+ "aspect_ratio": inputs.get("aspect_ratio", "16:9"),
+ "output": str(output_path),
+ "output_path": str(output_path),
+ "format": "mp4",
+ **probed,
+ },
+ artifacts=[str(output_path)],
+ cost_usd=self.estimate_cost(inputs),
+ duration_seconds=round(time.time() - start, 2),
+ model=model,
+ )
diff --git a/tools/video/runway_video.py b/tools/video/runway_video.py
index 8ed84460..6de0eef3 100644
--- a/tools/video/runway_video.py
+++ b/tools/video/runway_video.py
@@ -1,6 +1,7 @@
"""Runway Gen-4 video generation via Runway API.
Highest Elo-rated video generation model — professional quality and control.
+Supports Gen-3 Alpha Turbo, Gen-4 Turbo, and Gen-4 Aleph (highest fidelity).
"""
from __future__ import annotations
@@ -23,22 +24,40 @@ from tools.base_tool import (
ToolTier,
)
+_RATIO_MAP = {
+ "16:9": "1280:720",
+ "9:16": "720:1280",
+ "1:1": "720:720",
+}
+
+_COST_PER_SECOND = {
+ "gen3a_turbo": 0.05,
+ "gen4_turbo": 0.05,
+ "gen4_aleph": 0.15,
+}
+
+_RUNTIME_SECONDS = {
+ "gen3a_turbo": 25.0,
+ "gen4_turbo": 30.0,
+ "gen4_aleph": 60.0,
+}
+
class RunwayVideo(BaseTool):
name = "runway_video"
- version = "0.1.0"
+ version = "0.2.0"
tier = ToolTier.GENERATE
capability = "video_generation"
provider = "runway"
- stability = ToolStability.EXPERIMENTAL
+ stability = ToolStability.BETA
execution_mode = ExecutionMode.SYNC
determinism = Determinism.STOCHASTIC
runtime = ToolRuntime.API
dependencies = []
install_instructions = (
- "Set RUNWAY_API_KEY to your Runway API key.\n"
- " Get one at https://app.runwayml.com/settings/api-keys"
+ "Set RUNWAY_API_KEY to your Runway API secret.\n"
+ " Get one at https://dev.runwayml.com/"
)
agent_skills = ["ai-video-gen"]
@@ -68,8 +87,9 @@ class RunwayVideo(BaseTool):
},
"model": {
"type": "string",
- "enum": ["gen4_turbo", "gen4"],
+ "enum": ["gen4_turbo", "gen4_aleph", "gen3a_turbo"],
"default": "gen4_turbo",
+ "description": "gen4_aleph is highest fidelity, gen4_turbo is balanced, gen3a_turbo is cheapest",
},
"duration": {
"type": "integer",
@@ -82,6 +102,11 @@ class RunwayVideo(BaseTool):
"enum": ["16:9", "9:16", "1:1"],
"default": "16:9",
},
+ "watermark": {
+ "type": "boolean",
+ "default": False,
+ "description": "Include Runway watermark on output",
+ },
"image_url": {"type": "string", "description": "Reference image URL for image_to_video"},
"output_path": {"type": "string"},
},
@@ -90,29 +115,30 @@ class RunwayVideo(BaseTool):
resource_profile = ResourceProfile(
cpu_cores=1, ram_mb=512, vram_mb=0, disk_mb=500, network_required=True
)
- retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout"])
+ retry_policy = RetryPolicy(max_retries=2, retryable_errors=["rate_limit", "timeout", "THROTTLED"])
idempotency_key_fields = ["prompt", "model", "operation", "duration"]
side_effects = ["writes video file to output_path", "calls Runway API"]
user_visible_verification = ["Watch generated clip for visual quality and motion coherence"]
def get_status(self) -> ToolStatus:
- if os.environ.get("RUNWAY_API_KEY"):
+ if os.environ.get("RUNWAY_API_KEY") or os.environ.get("RUNWAYML_API_SECRET"):
return ToolStatus.AVAILABLE
return ToolStatus.UNAVAILABLE
+ def _get_api_key(self) -> str | None:
+ return os.environ.get("RUNWAY_API_KEY") or os.environ.get("RUNWAYML_API_SECRET")
+
def estimate_cost(self, inputs: dict[str, Any]) -> float:
+ model = inputs.get("model", "gen4_turbo")
duration = inputs.get("duration", 5)
- # Runway charges per second of generated video
- return 0.05 * duration # ~$0.25 for 5s, ~$0.50 for 10s
+ return _COST_PER_SECOND.get(model, 0.05) * duration
def estimate_runtime(self, inputs: dict[str, Any]) -> float:
model = inputs.get("model", "gen4_turbo")
- if "turbo" in model:
- return 30.0
- return 60.0
+ return _RUNTIME_SECONDS.get(model, 30.0)
def execute(self, inputs: dict[str, Any]) -> ToolResult:
- api_key = os.environ.get("RUNWAY_API_KEY")
+ api_key = self._get_api_key()
if not api_key:
return ToolResult(
success=False,
@@ -124,17 +150,26 @@ class RunwayVideo(BaseTool):
start = time.time()
model = inputs.get("model", "gen4_turbo")
operation = inputs.get("operation", "text_to_video")
+ ratio_friendly = inputs.get("ratio", "16:9")
+ ratio_pixels = _RATIO_MAP.get(ratio_friendly, "1280:720")
- # Runway API v1 — submit task
task_payload: dict[str, Any] = {
"model": model,
"promptText": inputs["prompt"],
"duration": inputs.get("duration", 5),
- "ratio": inputs.get("ratio", "16:9"),
+ "ratio": ratio_pixels,
+ "watermark": inputs.get("watermark", False),
}
if operation == "image_to_video" and inputs.get("image_url"):
task_payload["promptImage"] = inputs["image_url"]
+ # Choose endpoint based on operation
+ endpoint = (
+ "https://api.dev.runwayml.com/v1/image_to_video"
+ if operation == "image_to_video"
+ else "https://api.dev.runwayml.com/v1/text_to_video"
+ )
+
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
@@ -144,8 +179,7 @@ class RunwayVideo(BaseTool):
try:
# Submit generation task
submit_response = requests.post(
- "https://api.dev.runwayml.com/v1/image_to_video" if operation == "image_to_video"
- else "https://api.dev.runwayml.com/v1/text_to_video",
+ endpoint,
headers=headers,
json=task_payload,
timeout=30,
@@ -153,9 +187,9 @@ class RunwayVideo(BaseTool):
submit_response.raise_for_status()
task_id = submit_response.json()["id"]
- # Poll for completion
+ # Poll for completion (max ~5 minutes)
video_url = None
- for _ in range(60): # max 5 minutes
+ for _ in range(60):
time.sleep(5)
poll_response = requests.get(
f"https://api.dev.runwayml.com/v1/tasks/{task_id}",
@@ -164,20 +198,23 @@ class RunwayVideo(BaseTool):
)
poll_response.raise_for_status()
task_data = poll_response.json()
+ status = task_data["status"]
- if task_data["status"] == "SUCCEEDED":
+ if status == "SUCCEEDED":
video_url = task_data["output"][0]
break
- if task_data["status"] == "FAILED":
+ if status == "FAILED":
+ failure_code = task_data.get("failureCode", "unknown")
return ToolResult(
success=False,
- error=f"Runway generation failed: {task_data.get('failure', 'unknown error')}",
+ error=f"Runway generation failed ({failure_code}): {task_data.get('failure', 'unknown error')}",
)
+ # PENDING, THROTTLED, RUNNING — keep polling
if not video_url:
- return ToolResult(success=False, error="Runway generation timed out.")
+ return ToolResult(success=False, error="Runway generation timed out after 5 minutes.")
- # Download video
+ # Download video — URLs are ephemeral (expire in 24-48h)
video_response = requests.get(video_url, timeout=120)
video_response.raise_for_status()
@@ -188,14 +225,22 @@ class RunwayVideo(BaseTool):
except Exception as e:
return ToolResult(success=False, error=f"Runway video generation failed: {e}")
+ from tools.video._shared import probe_output
+
+ probed = probe_output(output_path)
return ToolResult(
success=True,
data={
"provider": "runway",
"model": model,
"prompt": inputs["prompt"],
+ "operation": operation,
+ "ratio": ratio_friendly,
"output": str(output_path),
+ "output_path": str(output_path),
"task_id": task_id,
+ "format": "mp4",
+ **probed,
},
artifacts=[str(output_path)],
cost_usd=self.estimate_cost(inputs),