Skip to content

The fastest, cheapest, most efficient open-source agent harness. Run more than 500 agents on a $10 VPS.

6.5 KB
Raw Blame

media

Family root for media agent tools. Today it has one member, generation, which gives the agent image and video generation over generators a host supplies (the TinyHumans one is in openhuman-tinyhumans). The family is tools-only: there is no RPC controller, no store, and no bus subscriber. A host's tool source is the only caller of the tool builder.

How it works

The work is split across four owners. The core holds the tools and the reference policy; the generators (endpoint, credential, egress and budget policy) come from the host that owns the media service.

 host tool source (openhuman-tinyhumans: integrations/media)
     |  managed_generators(config)  -> MediaGenerators { image, video }
     |     each wrapped in a Guard:
     |       admit():  local-only enforcement, then egress disclosure
     |       budget(): refuse a billed submit when managed credits are out
     v
 media::generation::media_tools_for_config(generators, config, action_dir, allowlist)
     |
     v
 tinyagents_harness::media::{GenerateImageTool, GenerateVideoTool}
     renamed media_generate_image / media_generate_video
     wrapped in MediaArtifactTool (files each output as an artifact)
   + MediaListModelsTool (media_list_models)
     |
     v
 tinyinference-image / tinyinference-video
     wire contract, submit -> poll -> download job loop

tools/ops.rs only records the files roots the reference policy reads (#[cfg(feature = "media")], DomainSet::media); it builds no media tools. A core with no tool source registered has no media tools at all, rather than broken ones.

When the agent calls media_generate_image, the TinyAgents tool parses the arguments and calls the guarded generator. The guard checks the privacy policy and announces the external transfer, checks the managed-credit budget, then hands the request to TinyInference. TinyAgents stores generated bytes in its media blob store. MediaArtifactTool copies each blob into the visible files folder, records it as an artifact (metadata under the workspace's artifacts/<id>/), and updates the JSON and Markdown result paths plus artifact_id so the chat UI can show a card. Video works the same way, except the generator submits a job and polls it (every 5 s, for up to 600 s) before downloading the clip. A timed-out video can be resumed with resume_job_id instead of paying for a new job.

Layout

Path What it does
mod.rs Declares generation.
generation/ The three media_* tools, the guarded OpenRouter generators, the local-reference policy, and the artifact wrapper.

Inside generation/:

File What it does
provider.rs MediaGenerators (the image + video pair, with an optional per-chat router) and GeneratorRouter.
tools.rs media_tools_for_config (the registry entry point over host-supplied generators), media_tools_from (the same assembly, used by tests), reference_policy, MediaListModelsTool, and the tool-name constants.
artifact_tool.rs MediaArtifactTool<T>: runs the inner tool unchanged, then files every reported output as an artifact, or annotates that one entry with artifact_error on failure.

Key types and entry points

  • media_tools_for_config(generators, &Config, &Path, allowlist) (generation/tools.rs) is the entry point a host's tool source calls.
  • MediaGenerators (generation/provider.rs) is the image + video pair the tools run over; MediaGenerators::with_router lets a host rebuild them per chat.
  • MediaArtifactTool (generation/artifact_tool.rs) is the artifact-tracking wrapper, following the same create_artifact_for_call / finalize_artifact / fail_artifact pattern the document and presentation tools use.
  • IMAGE_TOOL_NAME, VIDEO_TOOL_NAME, LIST_MODELS_TOOL_NAME pin the names the media tool pack and agent allowlists depend on.

Agent tools

Tool Permission Effect
media_generate_image Execute Generates or edits images from a prompt, optionally with reference images (URLs or allowed local paths). Billed per call.
media_generate_video Execute Generates a short clip, optionally from a first or last frame. Blocks until ready. Billed per call.
media_list_models ReadOnly Lists image and/or video models with an optional substring filter.

Boundaries

  • The OpenRouter wire contract, reference and output-shape standards, and the job loop belong to TinyInference (tinyinference-image, tinyinference-video, in vendor/tinyagents/vendor/tinyinference, upstream tinyhumansai/tinyinference), reached through tinyagents_harness.
  • Argument parsing, writing files into the workspace, and result wording belong to TinyAgents (tinyagents_harness::media, in vendor/tinyagents).
  • Provider keys, billing, the proxy route and the generators that reach it belong to the host (openhuman-tinyhumans: integrations/media).
  • Artifact storage belongs to artifacts.

Gotchas

  • Two gates. The media Cargo feature (in the core's default set, listed in scripts/ci/product-features.txt, forwarded from crates/openhuman-app/Cargo.toml) decides whether openhuman::media compiles. It turns on tinyagents-harness/media. At runtime, DomainSet::media in core/runtime/builder.rs filters any tool whose name starts with media_ (tool_group() in tools/ops.rs). Nothing else is tagged DomainGroup::Media.
  • Local reference files are limited by reference_policy to the action directory or workspace, with no .. and nothing under an always-forbidden path (credential stores, system roots). Anything else must be an https URL.

Tests

generation/tools_tests.rs (tool schemas, reference policy, flows over mock generators) and generation/artifact_tool_tests.rs (artifact filing and per-entry failure).

cargo test -p openhuman media::

User-facing docs: docs/gitbooks/en/features/native-tools/media-generation.md.

Further reading