Skip to content

Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.

docs: update accordions and fix layout (#2110)

Tony Kipkemboi4eaa8755ebb3
diff --git a/docs/concepts/llms.mdx b/docs/concepts/llms.mdx
index 117face04..12061d1a6 100644
--- a/docs/concepts/llms.mdx
+++ b/docs/concepts/llms.mdx
@@ -27,157 +27,6 @@ Large Language Models (LLMs) are the core intelligence behind CrewAI agents. The
</Card>
</CardGroup>
-## Available Models and Their Capabilities
-
-Here's a detailed breakdown of supported models and their capabilities, you can compare performance at [lmarena.ai](https://lmarena.ai/?leaderboard) and [artificialanalysis.ai](https://artificialanalysis.ai/):
-
-<Tabs>
- <Tab title="OpenAI">
- | Model | Context Window | Best For |
- |-------|---------------|-----------|
- | GPT-4 | 8,192 tokens | High-accuracy tasks, complex reasoning |
- | GPT-4 Turbo | 128,000 tokens | Long-form content, document analysis |
- | GPT-4o & GPT-4o-mini | 128,000 tokens | Cost-effective large context processing |
- | o3-mini | 200,000 tokens | Fast reasoning, complex reasoning |
-
- <Note>
- 1 token ≈ 4 characters in English. For example, 8,192 tokens ≈ 32,768 characters or about 6,000 words.
- </Note>
- </Tab>
- <Tab title="Nvidia NIM">
- | Model | Context Window | Best For |
- |-------|---------------|-----------|
- | nvidia/mistral-nemo-minitron-8b-8k-instruct | 8,192 tokens | State-of-the-art small language model delivering superior accuracy for chatbot, virtual assistants, and content generation. |
- | nvidia/nemotron-4-mini-hindi-4b-instruct| 4,096 tokens | A bilingual Hindi-English SLM for on-device inference, tailored specifically for Hindi Language. |
- | "nvidia/llama-3.1-nemotron-70b-instruct | 128k tokens | Llama-3.1-Nemotron-70B-Instruct is a large language model customized by NVIDIA in order to improve the helpfulness of LLM generated responses. |
- | nvidia/llama3-chatqa-1.5-8b | 128k tokens | Advanced LLM to generate high-quality, context-aware responses for chatbots and search engines. |
- | nvidia/llama3-chatqa-1.5-70b | 128k tokens | Advanced LLM to generate high-quality, context-aware responses for chatbots and search engines. |
- | nvidia/vila | 128k tokens | Multi-modal vision-language model that understands text/img/video and creates informative responses |
- | nvidia/neva-22| 4,096 tokens | Multi-modal vision-language model that understands text/images and generates informative responses |
- | nvidia/nemotron-mini-4b-instruct | 8,192 tokens | General-purpose tasks |
- | nvidia/usdcode-llama3-70b-instruct | 128k tokens | State-of-the-art LLM that answers OpenUSD knowledge queries and generates USD-Python code. |
- | nvidia/nemotron-4-340b-instruct | 4,096 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
- | meta/codellama-70b | 100k tokens | LLM capable of generating code from natural language and vice versa. |
- | meta/llama2-70b | 4,096 tokens | Cutting-edge large language AI model capable of generating text and code in response to prompts. |
- | meta/llama3-8b-instruct | 8,192 tokens | Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation. |
- | meta/llama3-70b-instruct | 8,192 tokens | Powers complex conversations with superior contextual understanding, reasoning and text generation. |
- | meta/llama-3.1-8b-instruct | 128k tokens | Advanced state-of-the-art model with language understanding, superior reasoning, and text generation. |
- | meta/llama-3.1-70b-instruct | 128k tokens | Powers complex conversations with superior contextual understanding, reasoning and text generation. |
- | meta/llama-3.1-405b-instruct | 128k tokens | Advanced LLM for synthetic data generation, distillation, and inference for chatbots, coding, and domain-specific tasks. |
- | meta/llama-3.2-1b-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
- | meta/llama-3.2-3b-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
- | meta/llama-3.2-11b-vision-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
- | meta/llama-3.2-90b-vision-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
- | meta/llama-3.1-70b-instruct | 128k tokens | Powers complex conversations with superior contextual understanding, reasoning and text generation. |
- | google/gemma-7b | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
- | google/gemma-2b | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
- | google/codegemma-7b | 8,192 tokens | Cutting-edge model built on Google's Gemma-7B specialized for code generation and code completion. |
- | google/codegemma-1.1-7b | 8,192 tokens | Advanced programming model for code generation, completion, reasoning, and instruction following. |
- | google/recurrentgemma-2b | 8,192 tokens | Novel recurrent architecture based language model for faster inference when generating long sequences. |
- | google/gemma-2-9b-it | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
- | google/gemma-2-27b-it | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
- | google/gemma-2-2b-it | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
- | google/deplot | 512 tokens | One-shot visual language understanding model that translates images of plots into tables. |
- | google/paligemma | 8,192 tokens | Vision language model adept at comprehending text and visual inputs to produce informative responses. |
- | mistralai/mistral-7b-instruct-v0.2 | 32k tokens | This LLM follows instructions, completes requests, and generates creative text. |
- | mistralai/mixtral-8x7b-instruct-v0.1 | 8,192 tokens | An MOE LLM that follows instructions, completes requests, and generates creative text. |
- | mistralai/mistral-large | 4,096 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
- | mistralai/mixtral-8x22b-instruct-v0.1 | 8,192 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
- | mistralai/mistral-7b-instruct-v0.3 | 32k tokens | This LLM follows instructions, completes requests, and generates creative text. |
- | nv-mistralai/mistral-nemo-12b-instruct | 128k tokens | Most advanced language model for reasoning, code, multilingual tasks; runs on a single GPU. |
- | mistralai/mamba-codestral-7b-v0.1 | 256k tokens | Model for writing and interacting with code across a wide range of programming languages and tasks. |
- | microsoft/phi-3-mini-128k-instruct | 128K tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
- | microsoft/phi-3-mini-4k-instruct | 4,096 tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
- | microsoft/phi-3-small-8k-instruct | 8,192 tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
- | microsoft/phi-3-small-128k-instruct | 128K tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
- | microsoft/phi-3-medium-4k-instruct | 4,096 tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
- | microsoft/phi-3-medium-128k-instruct | 128K tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
- | microsoft/phi-3.5-mini-instruct | 128K tokens | Lightweight multilingual LLM powering AI applications in latency bound, memory/compute constrained environments |
- | microsoft/phi-3.5-moe-instruct | 128K tokens | Advanced LLM based on Mixture of Experts architecure to deliver compute efficient content generation |
- | microsoft/kosmos-2 | 1,024 tokens | Groundbreaking multimodal model designed to understand and reason about visual elements in images. |
- | microsoft/phi-3-vision-128k-instruct | 128k tokens | Cutting-edge open multimodal model exceling in high-quality reasoning from images. |
- | microsoft/phi-3.5-vision-instruct | 128k tokens | Cutting-edge open multimodal model exceling in high-quality reasoning from images. |
- | databricks/dbrx-instruct | 12k tokens | A general-purpose LLM with state-of-the-art performance in language understanding, coding, and RAG. |
- | snowflake/arctic | 1,024 tokens | Delivers high efficiency inference for enterprise applications focused on SQL generation and coding. |
- | aisingapore/sea-lion-7b-instruct | 4,096 tokens | LLM to represent and serve the linguistic and cultural diversity of Southeast Asia |
- | ibm/granite-8b-code-instruct | 4,096 tokens | Software programming LLM for code generation, completion, explanation, and multi-turn conversion. |
- | ibm/granite-34b-code-instruct | 8,192 tokens | Software programming LLM for code generation, completion, explanation, and multi-turn conversion. |
- | ibm/granite-3.0-8b-instruct | 4,096 tokens | Advanced Small Language Model supporting RAG, summarization, classification, code, and agentic AI |
- | ibm/granite-3.0-3b-a800m-instruct | 4,096 tokens | Highly efficient Mixture of Experts model for RAG, summarization, entity extraction, and classification |
- | mediatek/breeze-7b-instruct | 4,096 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
- | upstage/solar-10.7b-instruct | 4,096 tokens | Excels in NLP tasks, particularly in instruction-following, reasoning, and mathematics. |
- | writer/palmyra-med-70b-32k | 32k tokens | Leading LLM for accurate, contextually relevant responses in the medical domain. |
- | writer/palmyra-med-70b | 32k tokens | Leading LLM for accurate, contextually relevant responses in the medical domain. |
- | writer/palmyra-fin-70b-32k | 32k tokens | Specialized LLM for financial analysis, reporting, and data processing |
- | 01-ai/yi-large | 32k tokens | Powerful model trained on English and Chinese for diverse tasks including chatbot and creative writing. |
- | deepseek-ai/deepseek-coder-6.7b-instruct | 2k tokens | Powerful coding model offering advanced capabilities in code generation, completion, and infilling |
- | rakuten/rakutenai-7b-instruct | 1,024 tokens | Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation. |
- | rakuten/rakutenai-7b-chat | 1,024 tokens | Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation. |
- | baichuan-inc/baichuan2-13b-chat | 4,096 tokens | Support Chinese and English chat, coding, math, instruction following, solving quizzes |
-
- <Note>
- NVIDIA's NIM support for models is expanding continuously! For the most up-to-date list of available models, please visit build.nvidia.com.
- </Note>
- </Tab>
- <Tab title="Gemini">
- | Model | Context Window | Best For |
- |-------|---------------|-----------|
- | gemini-2.0-flash-exp | 1M tokens | Higher quality at faster speed, multimodal model, good for most tasks |
- | gemini-1.5-flash | 1M tokens | Balanced multimodal model, good for most tasks |
- | gemini-1.5-flash-8B | 1M tokens | Fastest, most cost-efficient, good for high-frequency tasks |
- | gemini-1.5-pro | 2M tokens | Best performing, wide variety of reasoning tasks including logical reasoning, coding, and creative collaboration |
-
- <Tip>
- Google's Gemini models are all multimodal, supporting audio, images, video and text, supporting context caching, json schema, function calling, etc.
-
- These models are available via API_KEY from
- [The Gemini API](https://ai.google.dev/gemini-api/docs) and also from
- [Google Cloud Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/migrate-google-ai) as part of the
- [Model Garden](https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/explore-models).
- </Tip>
- </Tab>
- <Tab title="Groq">
- | Model | Context Window | Best For |
- |-------|---------------|-----------|
- | Llama 3.1 70B/8B | 131,072 tokens | High-performance, large context tasks |
- | Llama 3.2 Series | 8,192 tokens | General-purpose tasks |
- | Mixtral 8x7B | 32,768 tokens | Balanced performance and context |
-
- <Tip>
- Groq is known for its fast inference speeds, making it suitable for real-time applications.
- </Tip>
- </Tab>
- <Tab title="SambaNova">
- | Model | Context Window | Best For |
- |-------|---------------|-----------|
- | Llama 3.1 70B/8B | Up to 131,072 tokens | High-performance, large context tasks |
- | Llama 3.1 405B | 8,192 tokens | High-performance and output quality |
- | Llama 3.2 Series | 8,192 tokens | General-purpose tasks, multimodal |
- | Llama 3.3 70B | Up to 131,072 tokens | High-performance and output quality|
- | Qwen2 familly | 8,192 tokens | High-performance and output quality |
-
- <Tip>
- [SambaNova](https://cloud.sambanova.ai/) has several models with fast inference speed at full precision.
- </Tip>
- </Tab>
- <Tab title="Others">
- | Provider | Context Window | Key Features |
- |----------|---------------|--------------|
- | Deepseek Chat | 64,000 tokens | Specialized in technical discussions |
- | Deepseek R1 | 64,000 tokens | Affordable reasoning model |
- | Claude 3 | Up to 200K tokens | Strong reasoning, code understanding |
- | Gemma Series | 8,192 tokens | Efficient, smaller-scale tasks |
-
- <Info>
- Provider selection should consider factors like:
- - API availability in your region
- - Pricing structure
- - Required features (e.g., streaming, function calling)
- - Performance requirements
- </Info>
- </Tab>
-</Tabs>
-
## Setting Up Your LLM
There are three ways to configure LLMs in CrewAI. Choose the method that best fits your workflow:
@@ -206,102 +55,12 @@ There are three ways to configure LLMs in CrewAI. Choose the method that best fi
```yaml
researcher:
- # Agent Definition
role: Research Specialist
goal: Conduct comprehensive research and analysis
backstory: A dedicated research professional with years of experience
verbose: true
-
- # Model Selection (uncomment your choice)
-
- # OpenAI Models - Known for reliability and performance
- llm: openai/gpt-4o-mini
- # llm: openai/gpt-4 # More accurate but expensive
- # llm: openai/gpt-4-turbo # Fast with large context
- # llm: openai/gpt-4o # Optimized for longer texts
- # llm: openai/o1-preview # Latest features
- # llm: openai/o1-mini # Cost-effective
-
- # Azure Models - For enterprise deployments
- # llm: azure/gpt-4o-mini
- # llm: azure/gpt-4
- # llm: azure/gpt-35-turbo
-
- # Anthropic Models - Strong reasoning capabilities
- # llm: anthropic/claude-3-opus-20240229-v1:0
- # llm: anthropic/claude-3-sonnet-20240229-v1:0
- # llm: anthropic/claude-3-haiku-20240307-v1:0
- # llm: anthropic/claude-2.1
- # llm: anthropic/claude-2.0
-
- # Google Models - Strong reasoning, large cachable context window, multimodal
- # llm: gemini/gemini-1.5-pro-latest
- # llm: gemini/gemini-1.5-flash-latest
- # llm: gemini/gemini-1.5-flash-8b-latest
-
- # AWS Bedrock Models - Enterprise-grade
- # llm: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
- # llm: bedrock/anthropic.claude-v2:1
- # llm: bedrock/amazon.titan-text-express-v1
- # llm: bedrock/meta.llama2-70b-chat-v1
-
- # Amazon SageMaker Models - Enterprise-grade
- # llm: sagemaker/<my-endpoint>
-
- # Mistral Models - Open source alternative
- # llm: mistral/mistral-large-latest
- # llm: mistral/mistral-medium-latest
- # llm: mistral/mistral-small-latest
-
- # Groq Models - Fast inference
- # llm: groq/mixtral-8x7b-32768
- # llm: groq/llama-3.1-70b-versatile
- # llm: groq/llama-3.2-90b-text-preview
- # llm: groq/gemma2-9b-it
- # llm: groq/gemma-7b-it
-
- # IBM watsonx.ai Models - Enterprise features
- # llm: watsonx/ibm/granite-13b-chat-v2
- # llm: watsonx/meta-llama/llama-3-1-70b-instruct
- # llm: watsonx/bigcode/starcoder2-15b
-
- # Ollama Models - Local deployment
- # llm: ollama/llama3:70b
- # llm: ollama/codellama
- # llm: ollama/mistral
- # llm: ollama/mixtral
- # llm: ollama/phi
-
- # Fireworks AI Models - Specialized tasks
- # llm: fireworks_ai/accounts/fireworks/models/llama-v3-70b-instruct
- # llm: fireworks_ai/accounts/fireworks/models/mixtral-8x7b
- # llm: fireworks_ai/accounts/fireworks/models/zephyr-7b-beta
-
- # Perplexity AI Models - Research focused
- # llm: pplx/llama-3.1-sonar-large-128k-online
- # llm: pplx/mistral-7b-instruct
- # llm: pplx/codellama-34b-instruct
- # llm: pplx/mixtral-8x7b-instruct
-
- # Hugging Face Models - Community models
- # llm: huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct
- # llm: huggingface/mistralai/Mixtral-8x7B-Instruct-v0.1
- # llm: huggingface/tiiuae/falcon-180B-chat
- # llm: huggingface/google/gemma-7b-it
-
- # Nvidia NIM Models - GPU-optimized
- # llm: nvidia_nim/meta/llama3-70b-instruct
- # llm: nvidia_nim/mistral/mixtral-8x7b
- # llm: nvidia_nim/google/gemma-7b
-
- # SambaNova Models - Enterprise AI
- # llm: sambanova/Meta-Llama-3.1-8B-Instruct
- # llm: sambanova/BioMistral-7B
- # llm: sambanova/Falcon-180B
-
- # Open Router Models - Affordable reasoning
- # llm: openrouter/deepseek/deepseek-r1
- # llm: openrouter/deepseek/deepseek-chat
+ llm: openai/gpt-4o-mini # your model here
+ # (see provider configuration examples below for more)
```
<Info>
@@ -349,79 +108,17 @@ There are three ways to configure LLMs in CrewAI. Choose the method that best fi
</Tab>
</Tabs>
-## Advanced Features and Optimization
-
-Learn how to get the most out of your LLM configuration:
-
-<AccordionGroup>
- <Accordion title="Context Window Management">
- CrewAI includes smart context management features:
-
- ```python
- from crewai import LLM
-
- # CrewAI automatically handles:
- # 1. Token counting and tracking
- # 2. Content summarization when needed
- # 3. Task splitting for large contexts
-
- llm = LLM(
- model="gpt-4",
- max_tokens=4000, # Limit response length
- )
- ```
-
- <Info>
- Best practices for context management:
- 1. Choose models with appropriate context windows
- 2. Pre-process long inputs when possible
- 3. Use chunking for large documents
- 4. Monitor token usage to optimize costs
- </Info>
- </Accordion>
-
- <Accordion title="Performance Optimization">
- <Steps>
- <Step title="Token Usage Optimization">
- Choose the right context window for your task:
- - Small tasks (up to 4K tokens): Standard models
- - Medium tasks (between 4K-32K): Enhanced models
- - Large tasks (over 32K): Large context models
-
- ```python
- # Configure model with appropriate settings
- llm = LLM(
- model="openai/gpt-4-turbo-preview",
- temperature=0.7, # Adjust based on task
- max_tokens=4096, # Set based on output needs
- timeout=300 # Longer timeout for complex tasks
- )
- ```
- <Tip>
- - Lower temperature (0.1 to 0.3) for factual responses
- - Higher temperature (0.7 to 0.9) for creative tasks
- </Tip>
- </Step>
-
- <Step title="Best Practices">
- 1. Monitor token usage
- 2. Implement rate limiting
- 3. Use caching when possible
- 4. Set appropriate max_tokens limits
- </Step>
- </Steps>
+## Provider Configuration Examples
- <Info>
- Remember to regularly monitor your token usage and adjust your configuration as needed to optimize costs and performance.
- </Info>
- </Accordion>
-</AccordionGroup>
-## Provider Configuration Examples
+CrewAI supports a multitude of LLM providers, each offering unique features, authentication methods, and model capabilities.
+In this section, you'll find detailed examples that help you select, configure, and optimize the LLM that best fits your project's needs.
<AccordionGroup>
<Accordion title="OpenAI">
- ```python Code
+ Set the following environment variables in your `.env` file:
+
+ ```toml Code
# Required
OPENAI_API_KEY=sk-...
@@ -430,12 +127,12 @@ Learn how to get the most out of your LLM configuration:
OPENAI_ORGANIZATION=<your-org-id>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
from crewai import LLM
llm = LLM(
- model="gpt-4",
+ model="openai/gpt-4", # call model by provider/model_name
temperature=0.8,
max_tokens=150,
top_p=0.9,
@@ -445,14 +142,26 @@ Learn how to get the most out of your LLM configuration:
seed=42
)
```
+
+ OpenAI is one of the leading providers of LLMs with a wide range of models and features.
+
+ | Model | Context Window | Best For |
+ |---------------------|------------------|-----------------------------------------------|
+ | GPT-4 | 8,192 tokens | High-accuracy tasks, complex reasoning |
+ | GPT-4 Turbo | 128,000 tokens | Long-form content, document analysis |
+ | GPT-4o & GPT-4o-mini | 128,000 tokens | Cost-effective large context processing |
+ | o3-mini | 200,000 tokens | Fast reasoning, complex reasoning |
+ | o1-mini | 128,000 tokens | Fast reasoning, complex reasoning |
+ | o1-preview | 128,000 tokens | Fast reasoning, complex reasoning |
+ | o1 | 200,000 tokens | Fast reasoning, complex reasoning |
</Accordion>
<Accordion title="Anthropic">
- ```python Code
+ ```toml Code
ANTHROPIC_API_KEY=sk-ant-...
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="anthropic/claude-3-sonnet-20240229-v1:0",
@@ -462,7 +171,9 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Google">
- ```python Code
+ Set the following environment variables in your `.env` file:
+
+ ```toml Code
# Option 1: Gemini accessed with an API key.
# https://ai.google.dev/gemini-api/docs/api-key
GEMINI_API_KEY=<your-api-key>
@@ -471,7 +182,7 @@ Learn how to get the most out of your LLM configuration:
# https://cloud.google.com/vertex-ai/generative-ai/docs/overview
```
- Get credentials:
+ Get credentials from your Google Cloud Console and save it to a JSON file with the following code:
```python Code
import json
@@ -485,7 +196,7 @@ Learn how to get the most out of your LLM configuration:
vertex_credentials_json = json.dumps(vertex_credentials)
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
from crewai import LLM
@@ -495,10 +206,18 @@ Learn how to get the most out of your LLM configuration:
vertex_credentials=vertex_credentials_json
)
```
+ Google offers a range of powerful models optimized for different use cases:
+
+ | Model | Context Window | Best For |
+ |-----------------------|----------------|------------------------------------------------------------------|
+ | gemini-2.0-flash-exp | 1M tokens | Higher quality at faster speed, multimodal model, good for most tasks |
+ | gemini-1.5-flash | 1M tokens | Balanced multimodal model, good for most tasks |
+ | gemini-1.5-flash-8B | 1M tokens | Fastest, most cost-efficient, good for high-frequency tasks |
+ | gemini-1.5-pro | 2M tokens | Best performing, wide variety of reasoning tasks including logical reasoning, coding, and creative collaboration |
</Accordion>
<Accordion title="Azure">
- ```python Code
+ ```toml Code
# Required
AZURE_API_KEY=<your-api-key>
AZURE_API_BASE=<your-resource-url>
@@ -509,7 +228,7 @@ Learn how to get the most out of your LLM configuration:
AZURE_API_TYPE=<your-azure-api-type>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="azure/gpt-4",
@@ -519,13 +238,13 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="AWS Bedrock">
- ```python Code
+ ```toml Code
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
AWS_DEFAULT_REGION=<your-region>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
@@ -534,13 +253,13 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Amazon SageMaker">
- ```python Code
+ ```toml Code
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
AWS_DEFAULT_REGION=<your-region>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="sagemaker/<my-endpoint>"
@@ -549,11 +268,12 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Mistral">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
MISTRAL_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="mistral/mistral-large-latest",
@@ -563,35 +283,115 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Nvidia NIM">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
NVIDIA_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="nvidia_nim/meta/llama3-70b-instruct",
temperature=0.7
)
```
+
+ Nvidia NIM provides a comprehensive suite of models for various use cases, from general-purpose tasks to specialized applications.
+
+ | Model | Context Window | Best For |
+ |-------------------------------------------------------------------------|----------------|-------------------------------------------------------------------|
+ | nvidia/mistral-nemo-minitron-8b-8k-instruct | 8,192 tokens | State-of-the-art small language model delivering superior accuracy for chatbot, virtual assistants, and content generation. |
+ | nvidia/nemotron-4-mini-hindi-4b-instruct | 4,096 tokens | A bilingual Hindi-English SLM for on-device inference, tailored specifically for Hindi Language. |
+ | nvidia/llama-3.1-nemotron-70b-instruct | 128k tokens | Customized for enhanced helpfulness in responses |
+ | nvidia/llama3-chatqa-1.5-8b | 128k tokens | Advanced LLM to generate high-quality, context-aware responses for chatbots and search engines. |
+ | nvidia/llama3-chatqa-1.5-70b | 128k tokens | Advanced LLM to generate high-quality, context-aware responses for chatbots and search engines. |
+ | nvidia/vila | 128k tokens | Multi-modal vision-language model that understands text/img/video and creates informative responses |
+ | nvidia/neva-22 | 4,096 tokens | Multi-modal vision-language model that understands text/images and generates informative responses |
+ | nvidia/nemotron-mini-4b-instruct | 8,192 tokens | General-purpose tasks |
+ | nvidia/usdcode-llama3-70b-instruct | 128k tokens | State-of-the-art LLM that answers OpenUSD knowledge queries and generates USD-Python code. |
+ | nvidia/nemotron-4-340b-instruct | 4,096 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
+ | meta/codellama-70b | 100k tokens | LLM capable of generating code from natural language and vice versa. |
+ | meta/llama2-70b | 4,096 tokens | Cutting-edge large language AI model capable of generating text and code in response to prompts. |
+ | meta/llama3-8b-instruct | 8,192 tokens | Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation. |
+ | meta/llama3-70b-instruct | 8,192 tokens | Powers complex conversations with superior contextual understanding, reasoning and text generation. |
+ | meta/llama-3.1-8b-instruct | 128k tokens | Advanced state-of-the-art model with language understanding, superior reasoning, and text generation. |
+ | meta/llama-3.1-70b-instruct | 128k tokens | Powers complex conversations with superior contextual understanding, reasoning and text generation. |
+ | meta/llama-3.1-405b-instruct | 128k tokens | Advanced LLM for synthetic data generation, distillation, and inference for chatbots, coding, and domain-specific tasks. |
+ | meta/llama-3.2-1b-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
+ | meta/llama-3.2-3b-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
+ | meta/llama-3.2-11b-vision-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
+ | meta/llama-3.2-90b-vision-instruct | 128k tokens | Advanced state-of-the-art small language model with language understanding, superior reasoning, and text generation. |
+ | google/gemma-7b | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
+ | google/gemma-2b | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
+ | google/codegemma-7b | 8,192 tokens | Cutting-edge model built on Google's Gemma-7B specialized for code generation and code completion. |
+ | google/codegemma-1.1-7b | 8,192 tokens | Advanced programming model for code generation, completion, reasoning, and instruction following. |
+ | google/recurrentgemma-2b | 8,192 tokens | Novel recurrent architecture based language model for faster inference when generating long sequences. |
+ | google/gemma-2-9b-it | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
+ | google/gemma-2-27b-it | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
+ | google/gemma-2-2b-it | 8,192 tokens | Cutting-edge text generation model text understanding, transformation, and code generation. |
+ | google/deplot | 512 tokens | One-shot visual language understanding model that translates images of plots into tables. |
+ | google/paligemma | 8,192 tokens | Vision language model adept at comprehending text and visual inputs to produce informative responses. |
+ | mistralai/mistral-7b-instruct-v0.2 | 32k tokens | This LLM follows instructions, completes requests, and generates creative text. |
+ | mistralai/mixtral-8x7b-instruct-v0.1 | 8,192 tokens | An MOE LLM that follows instructions, completes requests, and generates creative text. |
+ | mistralai/mistral-large | 4,096 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
+ | mistralai/mixtral-8x22b-instruct-v0.1 | 8,192 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
+ | mistralai/mistral-7b-instruct-v0.3 | 32k tokens | This LLM follows instructions, completes requests, and generates creative text. |
+ | nv-mistralai/mistral-nemo-12b-instruct | 128k tokens | Most advanced language model for reasoning, code, multilingual tasks; runs on a single GPU. |
+ | mistralai/mamba-codestral-7b-v0.1 | 256k tokens | Model for writing and interacting with code across a wide range of programming languages and tasks. |
+ | microsoft/phi-3-mini-128k-instruct | 128K tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
+ | microsoft/phi-3-mini-4k-instruct | 4,096 tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
+ | microsoft/phi-3-small-8k-instruct | 8,192 tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
+ | microsoft/phi-3-small-128k-instruct | 128K tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
+ | microsoft/phi-3-medium-4k-instruct | 4,096 tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
+ | microsoft/phi-3-medium-128k-instruct | 128K tokens | Lightweight, state-of-the-art open LLM with strong math and logical reasoning skills. |
+ | microsoft/phi-3.5-mini-instruct | 128K tokens | Lightweight multilingual LLM powering AI applications in latency bound, memory/compute constrained environments |
+ | microsoft/phi-3.5-moe-instruct | 128K tokens | Advanced LLM based on Mixture of Experts architecure to deliver compute efficient content generation |
+ | microsoft/kosmos-2 | 1,024 tokens | Groundbreaking multimodal model designed to understand and reason about visual elements in images. |
+ | microsoft/phi-3-vision-128k-instruct | 128k tokens | Cutting-edge open multimodal model exceling in high-quality reasoning from images. |
+ | microsoft/phi-3.5-vision-instruct | 128k tokens | Cutting-edge open multimodal model exceling in high-quality reasoning from images. |
+ | databricks/dbrx-instruct | 12k tokens | A general-purpose LLM with state-of-the-art performance in language understanding, coding, and RAG. |
+ | snowflake/arctic | 1,024 tokens | Delivers high efficiency inference for enterprise applications focused on SQL generation and coding. |
+ | aisingapore/sea-lion-7b-instruct | 4,096 tokens | LLM to represent and serve the linguistic and cultural diversity of Southeast Asia |
+ | ibm/granite-8b-code-instruct | 4,096 tokens | Software programming LLM for code generation, completion, explanation, and multi-turn conversion. |
+ | ibm/granite-34b-code-instruct | 8,192 tokens | Software programming LLM for code generation, completion, explanation, and multi-turn conversion. |
+ | ibm/granite-3.0-8b-instruct | 4,096 tokens | Advanced Small Language Model supporting RAG, summarization, classification, code, and agentic AI |
+ | ibm/granite-3.0-3b-a800m-instruct | 4,096 tokens | Highly efficient Mixture of Experts model for RAG, summarization, entity extraction, and classification |
+ | mediatek/breeze-7b-instruct | 4,096 tokens | Creates diverse synthetic data that mimics the characteristics of real-world data. |
+ | upstage/solar-10.7b-instruct | 4,096 tokens | Excels in NLP tasks, particularly in instruction-following, reasoning, and mathematics. |
+ | writer/palmyra-med-70b-32k | 32k tokens | Leading LLM for accurate, contextually relevant responses in the medical domain. |
+ | writer/palmyra-med-70b | 32k tokens | Leading LLM for accurate, contextually relevant responses in the medical domain. |
+ | writer/palmyra-fin-70b-32k | 32k tokens | Specialized LLM for financial analysis, reporting, and data processing |
+ | 01-ai/yi-large | 32k tokens | Powerful model trained on English and Chinese for diverse tasks including chatbot and creative writing. |
+ | deepseek-ai/deepseek-coder-6.7b-instruct | 2k tokens | Powerful coding model offering advanced capabilities in code generation, completion, and infilling |
+ | rakuten/rakutenai-7b-instruct | 1,024 tokens | Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation. |
+ | rakuten/rakutenai-7b-chat | 1,024 tokens | Advanced state-of-the-art LLM with language understanding, superior reasoning, and text generation. |
+ | baichuan-inc/baichuan2-13b-chat | 4,096 tokens | Support Chinese and English chat, coding, math, instruction following, solving quizzes |
</Accordion>
<Accordion title="Groq">
- ```python Code
+ Set the following environment variables in your `.env` file:
+
+ ```toml Code
GROQ_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="groq/llama-3.2-90b-text-preview",
temperature=0.7
)
```
+ | Model | Context Window | Best For |
+ |-------------------|------------------|--------------------------------------------|
+ | Llama 3.1 70B/8B | 131,072 tokens | High-performance, large context tasks |
+ | Llama 3.2 Series | 8,192 tokens | General-purpose tasks |
+ | Mixtral 8x7B | 32,768 tokens | Balanced performance and context |
</Accordion>
<Accordion title="IBM watsonx.ai">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
# Required
WATSONX_URL=<your-url>
WATSONX_APIKEY=<your-apikey>
@@ -602,7 +402,7 @@ Learn how to get the most out of your LLM configuration:
WATSONX_DEPLOYMENT_SPACE_ID=<your-space-id>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="watsonx/meta-llama/llama-3-1-70b-instruct",
@@ -625,11 +425,12 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Fireworks AI">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
FIREWORKS_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="fireworks_ai/accounts/fireworks/models/llama-v3-70b-instruct",
@@ -639,11 +440,12 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Perplexity AI">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
PERPLEXITY_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="llama-3.1-sonar-large-128k-online",
@@ -653,11 +455,12 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Hugging Face">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
HUGGINGFACE_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct",
@@ -667,26 +470,36 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="SambaNova">
- ```python Code
+ Set the following environment variables in your `.env` file:
+
+ ```toml Code
SAMBANOVA_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="sambanova/Meta-Llama-3.1-8B-Instruct",
temperature=0.7
)
```
+ | Model | Context Window | Best For |
+ |--------------------|------------------------|----------------------------------------------|
+ | Llama 3.1 70B/8B | Up to 131,072 tokens | High-performance, large context tasks |
+ | Llama 3.1 405B | 8,192 tokens | High-performance and output quality |
+ | Llama 3.2 Series | 8,192 tokens | General-purpose, multimodal tasks |
+ | Llama 3.3 70B | Up to 131,072 tokens | High-performance and output quality |
+ | Qwen2 familly | 8,192 tokens | High-performance and output quality |
</Accordion>
<Accordion title="Cerebras">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
# Required
CEREBRAS_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="cerebras/llama3.1-70b",
@@ -705,11 +518,12 @@ Learn how to get the most out of your LLM configuration:
</Accordion>
<Accordion title="Open Router">
- ```python Code
+ Set the following environment variables in your `.env` file:
+ ```toml Code
OPENROUTER_API_KEY=<your-api-key>
```
- Example usage:
+ Example usage in your CrewAI project:
```python Code
llm = LLM(
model="openrouter/deepseek/deepseek-r1",
@@ -748,8 +562,79 @@ response = llm.call(
"Meet Kona! She is 3 years old and is a black german shepherd."
)
print(response)
+
+# Output:
+# Dog(name='Kona', age=3, breed='black german shepherd')
```
+## Advanced Features and Optimization
+
+Learn how to get the most out of your LLM configuration:
+
+<AccordionGroup>
+ <Accordion title="Context Window Management">
+ CrewAI includes smart context management features:
+
+ ```python
+ from crewai import LLM
+
+ # CrewAI automatically handles:
+ # 1. Token counting and tracking
+ # 2. Content summarization when needed
+ # 3. Task splitting for large contexts
+
+ llm = LLM(
+ model="gpt-4",
+ max_tokens=4000, # Limit response length
+ )
+ ```
+
+ <Info>
+ Best practices for context management:
+ 1. Choose models with appropriate context windows
+ 2. Pre-process long inputs when possible
+ 3. Use chunking for large documents
+ 4. Monitor token usage to optimize costs
+ </Info>
+ </Accordion>
+
+ <Accordion title="Performance Optimization">
+ <Steps>
+ <Step title="Token Usage Optimization">
+ Choose the right context window for your task:
+ - Small tasks (up to 4K tokens): Standard models
+ - Medium tasks (between 4K-32K): Enhanced models
+ - Large tasks (over 32K): Large context models
+
+ ```python
+ # Configure model with appropriate settings
+ llm = LLM(
+ model="openai/gpt-4-turbo-preview",
+ temperature=0.7, # Adjust based on task
+ max_tokens=4096, # Set based on output needs
+ timeout=300 # Longer timeout for complex tasks
+ )
+ ```
+ <Tip>
+ - Lower temperature (0.1 to 0.3) for factual responses
+ - Higher temperature (0.7 to 0.9) for creative tasks
+ </Tip>
+ </Step>
+
+ <Step title="Best Practices">
+ 1. Monitor token usage
+ 2. Implement rate limiting
+ 3. Use caching when possible
+ 4. Set appropriate max_tokens limits
+ </Step>
+ </Steps>
+
+ <Info>
+ Remember to regularly monitor your token usage and adjust your configuration as needed to optimize costs and performance.
+ </Info>
+ </Accordion>
+</AccordionGroup>
+
## Common Issues and Solutions
<Tabs>