The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]
Python ai-gatewayanthropicazure-openaibedrockgatewaylangchainlitellmllmllm-gatewayllmopsmcp-gatewayopenaiopenai-proxyrustrust-aivertex-ai
{
"model": "gpt-4o",
"input": "List files in /mnt/data and run python --version.",
"context_management": [
{
"type": "compaction",
"compact_threshold": 200000
}
],
"tools": [
{
"type": "shell",
"environment": {
"type": "container_auto"
}
}
],
"tool_choice": "auto",
"max_output_tokens": 256
}