The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
docs: resolve videodb review findings
Affaan Mustafadb2bf1642741
diff --git a/skills/videodb/SKILL.md b/skills/videodb/SKILL.mdindex 277870360..e9195d7da 100644--- a/skills/videodb/SKILL.md+++ b/skills/videodb/SKILL.md@@ -10,59 +10,48 @@ argument-hint: "[task description]"**Perception + memory + actions for video, live streams, and desktop sessions.**-Use this skill when you need to:+## When to Use-## 1) Desktop Perception+### Desktop Perception- Start/stop a **desktop session** capturing **screen, mic, and system audio**- Stream **live context** and store **episodic session memory**- Run **real-time alerts/triggers** on what's spoken and what's happening on screen- Produce **session summaries**, a searchable timeline, and **playable evidence links**-## 2) Video ingest + stream+### Video ingest + stream- Ingest a **file or URL** and return a **playable web stream link**- Transcode/normalize: **codec, bitrate, fps, resolution, aspect ratio**-## 3) Index + search (timestamps + evidence)+### Index + search (timestamps + evidence)- Build **visual**, **spoken**, and **keyword** indexes- Search and return exact moments with **timestamps** and **playable evidence**- Auto-create **clips** from search results-## 4) Timeline editing + generation+### Timeline editing + generation- Subtitles: **generate**, **translate**, **burn-in**- Overlays: **text/image/branding**, motion captions- Audio: **background music**, **voiceover**, **dubbing**- Programmatic composition and exports via **timeline operations**-## 5) Live streams (RTSP) + monitoring+### Live streams (RTSP) + monitoring- Connect **RTSP/live feeds**- Run **real-time visual and spoken understanding** and emit **events/alerts** for monitoring workflows----+## How It Works-## Common inputs+### Common inputs- Local **file path**, public **URL**, or **RTSP URL**- Desktop capture request: **start / stop / summarize session**- Desired operations: get context for understanding, transcode spec, index spec, search query, clip ranges, timeline edits, alert rules-## Common outputs+### Common outputs- **Stream URL**- Search results with **timestamps** and **evidence links**- Generated assets: subtitles, audio, images, clips- **Event/alert payloads** for live streams- Desktop **session summaries** and memory entries------## Canonical prompts (examples)-- "Start desktop capture and alert when a password field appears."-- "Record my session and produce an actionable summary when it ends."-- "Ingest this file and return a playable stream link."-- "Index this folder and find every scene with people, return timestamps."-- "Generate subtitles, burn them in, and add light background music."-- "Connect this RTSP URL and alert when a person enters the zone."---## Running Python code+### Running Python codeBefore running any VideoDB code, change to the project directory and load environment variables:@@ -96,7 +85,7 @@ print(f"Videos: {len(coll.get_videos())}")EOF```-## Setup+### SetupWhen the user asks to "setup videodb" or similar:@@ -123,7 +112,7 @@ Get a free API key at https://console.videodb.io (50 free uploads, no credit car**Do NOT** read, write, or handle the API key yourself. Always let the user set it.-## Quick Reference+### Quick Reference### Upload media@@ -298,39 +287,37 @@ except InvalidRequestError as e:| Negative timestamps on Timeline | Silently produces broken stream | Always validate `start >= 0` before creating `VideoAsset` || `generate_video()` / `create_collection()` fails | `Operation not allowed` or `maximum limit` | Plan-gated features — inform the user about plan limits |-## Additional docs+## Examples-Reference documentation is in the `reference/` directory adjacent to this SKILL.md file. Use the Glob tool to locate it if needed.--- [reference/api-reference.md](reference/api-reference.md) - Complete VideoDB Python SDK API reference-- [reference/search.md](reference/search.md) - In-depth guide to video search (spoken word and scene-based)-- [reference/editor.md](reference/editor.md) - Timeline editing, assets, and composition-- [reference/streaming.md](reference/streaming.md) - HLS streaming and instant playback-- [reference/generative.md](reference/generative.md) - AI-powered media generation (images, video, audio)-- [reference/rtstream.md](reference/rtstream.md) - Live stream ingestion workflow (RTSP/RTMP)-- [reference/rtstream-reference.md](reference/rtstream-reference.md) - RTStream SDK methods and AI pipelines-- [reference/capture.md](reference/capture.md) - Desktop capture workflow-- [reference/capture-reference.md](reference/capture-reference.md) - Capture SDK and WebSocket events-- [reference/use-cases.md](reference/use-cases.md) - Common video processing patterns and examples+### Canonical prompts+- "Start desktop capture and alert when a password field appears."+- "Record my session and produce an actionable summary when it ends."+- "Ingest this file and return a playable stream link."+- "Index this folder and find every scene with people, return timestamps."+- "Generate subtitles, burn them in, and add light background music."+- "Connect this RTSP URL and alert when a person enters the zone."-## Screen Recording (Desktop Capture)+### Screen Recording (Desktop Capture)Use `ws_listener.py` to capture WebSocket events during recording sessions. Desktop capture supports **macOS** only.-### Quick Start+#### Quick Start++1. **Start listener**: `python scripts/ws_listener.py --clear &`+2. **Get WebSocket ID**: `cat "${VIDEODB_EVENTS_DIR:-$HOME/.local/state/videodb}/videodb_ws_id"`+3. **Run capture code** (see reference/capture.md for the full workflow)+4. **Events written to**: `${VIDEODB_EVENTS_DIR:-$HOME/.local/state/videodb}/videodb_events.jsonl`-1. **Start listener**: `python scripts/ws_listener.py &`-2. **Get WebSocket ID**: `cat /tmp/videodb_ws_id`-3. **Run capture code** (see reference/capture.md for full workflow)-4. **Events written to**: `/tmp/videodb_events.jsonl`+Use `--clear` whenever you start a fresh capture run so stale transcript and visual events do not leak into the new session.-### Query Events+#### Query Events```pythonimport json+import timefrom pathlib import Path-events_file = Path("/tmp/videodb_events.jsonl")+events_file = Path.home() / ".local" / "state" / "videodb" / "videodb_events.jsonl"events = []if events_file.exists():@@ -341,21 +328,28 @@ if events_file.exists():except json.JSONDecodeError:continue-# Get all transcriptstranscripts = [e["data"]["text"] for e in events if e.get("channel") == "transcript"]--# Get visual descriptions from last 5 minutes-import timecutoff = time.time() - 300-recent_visual = [e for e in events- if e.get("channel") == "visual_index" and e["unix_ts"] > cutoff]+recent_visual = [+ e for e in events+ if e.get("channel") == "visual_index" and e["unix_ts"] > cutoff+]```-### Utility Scripts+## Additional docs-- [scripts/ws_listener.py](scripts/ws_listener.py) - WebSocket event listener (dumps to JSONL)+Reference documentation is in the `reference/` directory adjacent to this SKILL.md file. Use the Glob tool to locate it if needed.-For complete capture workflow, see [reference/capture.md](reference/capture.md).+- [reference/api-reference.md](reference/api-reference.md) - Complete VideoDB Python SDK API reference+- [reference/search.md](reference/search.md) - In-depth guide to video search (spoken word and scene-based)+- [reference/editor.md](reference/editor.md) - Timeline editing, assets, and composition+- [reference/streaming.md](reference/streaming.md) - HLS streaming and instant playback+- [reference/generative.md](reference/generative.md) - AI-powered media generation (images, video, audio)+- [reference/rtstream.md](reference/rtstream.md) - Live stream ingestion workflow (RTSP/RTMP)+- [reference/rtstream-reference.md](reference/rtstream-reference.md) - RTStream SDK methods and AI pipelines+- [reference/capture.md](reference/capture.md) - Desktop capture workflow+- [reference/capture-reference.md](reference/capture-reference.md) - Capture SDK and WebSocket events+- [reference/use-cases.md](reference/use-cases.md) - Common video processing patterns and examples**Do not use ffmpeg, moviepy, or local encoding tools** when VideoDB supports the operation. The following are all handled server-side by VideoDB — trimming, combining clips, overlaying audio or music, adding subtitles, text/image overlays, transcoding, resolution changes, aspect-ratio conversion, resizing for platform requirements, transcription, and media generation. Only fall back to local tools for operations listed under Limitations in reference/editor.md (transitions, speed changes, crop/zoom, colour grading, volume mixing).diff --git a/skills/videodb/reference/capture-reference.md b/skills/videodb/reference/capture-reference.mdindex 4427aae9c..302e0ef12 100644--- a/skills/videodb/reference/capture-reference.md+++ b/skills/videodb/reference/capture-reference.md@@ -8,7 +8,7 @@ Code-level details for VideoDB capture sessions. For workflow guide, see [capturReal-time events from capture sessions and AI pipelines. No webhooks or polling required.-Use [scripts/ws_listener.py](../scripts/ws_listener.py) to connect and dump events to `/tmp/videodb_events.jsonl`.+Use [scripts/ws_listener.py](../scripts/ws_listener.py) to connect and dump events to `${VIDEODB_EVENTS_DIR:-$HOME/.local/state/videodb}/videodb_events.jsonl`.### Event Channels@@ -137,12 +137,12 @@ The script outputs `WS_ID=<connection_id>` on the first line, then listens indef**Get the ws_id:**```bash-cat /tmp/videodb_ws_id+cat "${VIDEODB_EVENTS_DIR:-$HOME/.local/state/videodb}/videodb_ws_id"```**Stop the listener:**```bash-kill $(cat /tmp/videodb_ws_pid)+kill "$(cat "${VIDEODB_EVENTS_DIR:-$HOME/.local/state/videodb}/videodb_ws_pid")"```**Functions that accept `ws_connection_id`:**@@ -152,7 +152,7 @@ kill $(cat /tmp/videodb_ws_pid)| `conn.create_capture_session()` | Session lifecycle events || RTStream methods | See [rtstream-reference.md](rtstream-reference.md) |-**Output files** (in output directory, default `/tmp`):+**Output files** (in output directory, default `${XDG_STATE_HOME:-$HOME/.local/state}/videodb`):- `videodb_ws_id` - WebSocket connection ID- `videodb_events.jsonl` - All events- `videodb_ws_pid` - Process ID for easy termination@@ -176,20 +176,27 @@ Each line is a JSON object with added timestamps:```pythonimport json-events = [json.loads(l) for l in open("/tmp/videodb_events.jsonl")]+import time+from pathlib import Path-# Filter by channel-transcripts = [e for e in events if e.get("channel") == "transcript"]+events_path = Path.home() / ".local" / "state" / "videodb" / "videodb_events.jsonl"+transcripts = []+recent = []+visual = []-# Filter by time (last 10 minutes)-import timecutoff = time.time() - 600-recent = [e for e in events if e["unix_ts"] > cutoff]--# Filter visual events containing keyword-visual = [e for e in events- if e.get("channel") == "visual_index"- and "code" in e.get("data", {}).get("text", "").lower()]+with events_path.open(encoding="utf-8") as handle:+ for line in handle:+ event = json.loads(line)+ if event.get("channel") == "transcript":+ transcripts.append(event)+ if event.get("unix_ts", 0) > cutoff:+ recent.append(event)+ if (+ event.get("channel") == "visual_index"+ and "code" in event.get("data", {}).get("text", "").lower()+ ):+ visual.append(event)```---@@ -224,7 +231,9 @@ ws_id = ws.connection_id### Create a Capture Session```python-ws_id = open("/tmp/videodb_ws_id").read().strip()+from pathlib import Path++ws_id = (Path.home() / ".local" / "state" / "videodb" / "videodb_ws_id").read_text().strip()session = conn.create_capture_session(end_user_id="user-123", # required@@ -390,7 +399,8 @@ For RTStream methods (indexing, transcription, alerts, batch config), see [rtstr┌───────────────┐ WebSocket: capture_session.exported│ exported │ ──> Access video_id, stream_url, player_url└───────────────┘-++ unrecoverable capture error│v┌───────────────┐ WebSocket: capture_session.faileddiff --git a/skills/videodb/reference/editor.md b/skills/videodb/reference/editor.mdindex 22262f5b5..6f772af0a 100644--- a/skills/videodb/reference/editor.md+++ b/skills/videodb/reference/editor.md@@ -280,6 +280,7 @@ For more streaming options (segment streams, search-to-stream, audio playback),```pythonimport videodbfrom videodb import SearchType+from videodb.exceptions import InvalidRequestErrorfrom videodb.timeline import Timelinefrom videodb.asset import VideoAsset, TextAsset, TextStyle@@ -289,8 +290,14 @@ video = coll.get_video("your-video-id")# 1. Search for key momentsvideo.index_spoken_words(force=True)-results = video.search("product announcement", search_type=SearchType.semantic)-shots = results.get_shots() # may be empty if no results+try:+ results = video.search("product announcement", search_type=SearchType.semantic)+ shots = results.get_shots()+except InvalidRequestError as exc:+ if "No results found" in str(exc):+ shots = []+ else:+ raise# 2. Build timelinetimeline = Timeline(conn)diff --git a/skills/videodb/reference/streaming.md b/skills/videodb/reference/streaming.mdindex 55e653c5b..cf591d923 100644--- a/skills/videodb/reference/streaming.md+++ b/skills/videodb/reference/streaming.md@@ -47,10 +47,10 @@ video.play()```python# Index and add subtitles firstvideo.index_spoken_words(force=True)-video.add_subtitle()+stream_url = video.add_subtitle()-# Stream now includes subtitles-stream_url = video.generate_stream()+# Returned URL already includes subtitles+print(f"Subtitled stream: {stream_url}")```### Specific Segmentsdiff --git a/skills/videodb/scripts/ws_listener.py b/skills/videodb/scripts/ws_listener.pyindex 968038fe6..456bfb203 100644--- a/skills/videodb/scripts/ws_listener.py+++ b/skills/videodb/scripts/ws_listener.py@@ -6,7 +6,7 @@ Usage:python scripts/ws_listener.py [OPTIONS] [output_dir]Arguments:- output_dir Directory for output files (default: /tmp or VIDEODB_EVENTS_DIR env var)+ output_dir Directory for output files (default: XDG_STATE_HOME/videodb or ~/.local/state/videodb)Options:--clear Clear the events file before starting (use when starting a new session)@@ -20,10 +20,10 @@ Output (first line, for parsing):WS_ID=<connection_id>Examples:- python scripts/ws_listener.py & # Run in background- python scripts/ws_listener.py --clear # Clear events and start fresh- python scripts/ws_listener.py --clear /tmp/mydir # Custom dir with clear- kill $(cat /tmp/videodb_ws_pid) # Stop the listener+ python scripts/ws_listener.py & # Run in background+ python scripts/ws_listener.py --clear # Clear events and start fresh+ python scripts/ws_listener.py --clear /tmp/mydir # Custom dir with clear+ kill "$(cat ~/.local/state/videodb/videodb_ws_pid)" # Stop the listener"""import osimport sys@@ -31,6 +31,7 @@ import jsonimport signalimport asyncioimport logging+import contextlibfrom datetime import datetime, timezonefrom pathlib import Path@@ -52,6 +53,27 @@ logging.basicConfig(LOGGER = logging.getLogger(__name__)# Parse arguments+RETRYABLE_ERRORS = (ConnectionError, TimeoutError)+++def default_output_dir() -> Path:+ """Return a private per-user state directory for listener artifacts."""+ xdg_state_home = os.environ.get("XDG_STATE_HOME")+ if xdg_state_home:+ return Path(xdg_state_home) / "videodb"+ return Path.home() / ".local" / "state" / "videodb"+++def ensure_private_dir(path: Path) -> Path:+ """Create the listener state directory with private permissions."""+ path.mkdir(parents=True, exist_ok=True, mode=0o700)+ try:+ path.chmod(0o700)+ except OSError:+ pass+ return path++def parse_args() -> tuple[bool, Path]:clear = Falseoutput_dir: str | None = None@@ -64,9 +86,9 @@ def parse_args() -> tuple[bool, Path]:output_dir = argif output_dir is None:- output_dir = os.environ.get("VIDEODB_EVENTS_DIR", "/tmp")-- return clear, Path(output_dir)+ return clear, ensure_private_dir(default_output_dir())++ return clear, ensure_private_dir(Path(output_dir))CLEAR_EVENTS, OUTPUT_DIR = parse_args()EVENTS_FILE = OUTPUT_DIR / "videodb_events.jsonl"@@ -93,7 +115,7 @@ def append_event(event: dict):def write_pid():"""Write PID file for easy process management."""- OUTPUT_DIR.mkdir(parents=True, exist_ok=True)+ OUTPUT_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)PID_FILE.write_text(str(os.getpid()))@@ -118,43 +140,10 @@ async def listen_with_retry():ws_wrapper = conn.connect_websocket()ws = await ws_wrapper.connect()ws_id = ws.connection_id-- # Ensure output directory exists- OUTPUT_DIR.mkdir(parents=True, exist_ok=True)-- # Clear events file only on first connection if --clear flag is set- if _first_connection and CLEAR_EVENTS:- EVENTS_FILE.unlink(missing_ok=True)- log("Cleared events file")- _first_connection = False-- # Write ws_id to file for easy retrieval- WS_ID_FILE.write_text(ws_id)-- # Print ws_id (parseable format for LLM)- if retry_count == 0:- print(f"WS_ID={ws_id}", flush=True)- log(f"Connected (ws_id={ws_id})")-- # Reset retry state on successful connection- retry_count = 0- backoff = INITIAL_BACKOFF-- # Listen for messages- async for msg in ws.receive():- append_event(msg)- channel = msg.get("channel", msg.get("event", "unknown"))- text = msg.get("data", {}).get("text", "")- if text:- print(f"[{channel}] {text[:80]}", flush=True)-- # If we exit the loop normally, connection was closed- log("Connection closed by server")-except asyncio.CancelledError:log("Shutdown requested")raise- except Exception as e:+ except RETRYABLE_ERRORS as e:retry_count += 1log(f"Connection error: {e}")@@ -165,6 +154,52 @@ async def listen_with_retry():log(f"Reconnecting in {backoff}s (attempt {retry_count}/{MAX_RETRIES})...")await asyncio.sleep(backoff)backoff = min(backoff * 2, MAX_BACKOFF)+ continue++ OUTPUT_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)++ if _first_connection and CLEAR_EVENTS:+ EVENTS_FILE.unlink(missing_ok=True)+ log("Cleared events file")+ _first_connection = False++ WS_ID_FILE.write_text(ws_id)++ if retry_count == 0:+ print(f"WS_ID={ws_id}", flush=True)+ log(f"Connected (ws_id={ws_id})")++ retry_count = 0+ backoff = INITIAL_BACKOFF++ receiver = ws.receive().__aiter__()+ while True:+ try:+ msg = await anext(receiver)+ except StopAsyncIteration:+ log("Connection closed by server")+ break+ except asyncio.CancelledError:+ log("Shutdown requested")+ raise+ except RETRYABLE_ERRORS as e:+ retry_count += 1+ log(f"Connection error: {e}")++ if retry_count >= MAX_RETRIES:+ log(f"Max retries ({MAX_RETRIES}) exceeded, exiting")+ return++ log(f"Reconnecting in {backoff}s (attempt {retry_count}/{MAX_RETRIES})...")+ await asyncio.sleep(backoff)+ backoff = min(backoff * 2, MAX_BACKOFF)+ break++ append_event(msg)+ channel = msg.get("channel", msg.get("event", "unknown"))+ text = msg.get("data", {}).get("text", "")+ if text:+ print(f"[{channel}] {text[:80]}", flush=True)async def main_async():@@ -178,7 +213,8 @@ async def main_async():# Register signal handlersfor sig in (signal.SIGINT, signal.SIGTERM):- loop.add_signal_handler(sig, handle_signal)+ with contextlib.suppress(NotImplementedError):+ loop.add_signal_handler(sig, handle_signal)# Run listener with cancellation supportlisten_task = asyncio.create_task(listen_with_retry())@@ -188,6 +224,9 @@ async def main_async():[listen_task, shutdown_task],return_when=asyncio.FIRST_COMPLETED,)++ if listen_task.done():+ await listen_task# Cancel remaining tasksfor task in pending:@@ -196,6 +235,10 @@ async def main_async():await taskexcept asyncio.CancelledError:pass++ for sig in (signal.SIGINT, signal.SIGTERM):+ with contextlib.suppress(NotImplementedError):+ loop.remove_signal_handler(sig)log("Shutdown complete")