Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
add knowledge demo + improve knowledge docs (#1706)
Tony Kipkemboifa373f966041
diff --git a/docs/concepts/knowledge.mdx b/docs/concepts/knowledge.mdxindex a00b2c2f0..06c2eb947 100644--- a/docs/concepts/knowledge.mdx+++ b/docs/concepts/knowledge.mdx@@ -1,6 +1,6 @@---title: Knowledge-description: Understand what knowledge is in CrewAI and how to effectively use it.+description: What is knowledge in CrewAI and how to use it.icon: book---@@ -8,7 +8,8 @@ icon: book## What is Knowledge?-Knowledge in CrewAI is a powerful system that allows AI agents to access and utilize external information sources during their tasks. Think of it as giving your agents a reference library they can consult while working.+Knowledge in CrewAI is a powerful system that allows AI agents to access and utilize external information sources during their tasks.+Think of it as giving your agents a reference library they can consult while working.<Info>Key benefits of using Knowledge:@@ -37,151 +38,275 @@ CrewAI supports various types of knowledge sources out of the box:## Quick Start-Here's a simple example using string-based knowledge:+Here's an example using string-based knowledge:-```python-from crewai import Agent, Task, Crew-from crewai.knowledge import StringKnowledgeSource--# 1. Create a knowledge source-product_info = StringKnowledgeSource(- content="""Our product X1000 has the following features:- - 10-hour battery life- - Water-resistant- - Available in black and silver- Price: $299.99""",- metadata={"category": "product"}-)+```python Code+from crewai import Agent, Task, Crew, Process, LLM+from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource-# 2. Create an agent with knowledge-sales_agent = Agent(- role="Sales Representative",- goal="Accurately answer customer questions about products",- backstory="Expert in product features and customer service",- knowledge_sources=[product_info] # Attach knowledge to agent+# Create a knowledge source+content = "Users name is John. He is 30 years old and lives in San Francisco."+string_source = StringKnowledgeSource(+ content=content,+ metadata={"preference": "personal"})-# 3. Create a task-answer_task = Task(- description="Answer: What colors is the X1000 available in and how much does it cost?",- agent=sales_agent+# Create an LLM with a temperature of 0 to ensure deterministic outputs+llm = LLM(model="gpt-4o-mini", temperature=0)++# Create an agent with the knowledge store+agent = Agent(+ role="About User",+ goal="You know everything about the user.",+ backstory="""You are a master at understanding people and their preferences.""",+ verbose=True,+ allow_delegation=False,+ llm=llm,+)+task = Task(+ description="Answer the following questions about the user: {question}",+ expected_output="An answer to the question.",+ agent=agent,)-# 4. Create and run the crewcrew = Crew(- agents=[sales_agent],- tasks=[answer_task]+ agents=[agent],+ tasks=[task],+ verbose=True,+ process=Process.sequential,+ knowledge={+ "sources": [string_source],+ "metadata": {"preference": "personal"}+ }, # Enable knowledge by adding the sources here. You can also add more sources to the sources list.)-result = crew.kickoff()+result = crew.kickoff(inputs={"question": "What city does John live in and how old is he?"})```## Knowledge Configuration-### Collection Names--Knowledge sources are organized into collections for better management:+### Metadata and Filtering-```python-# Create knowledge sources with specific collections-tech_specs = StringKnowledgeSource(- content="Technical specifications...",- collection_name="product_tech_specs"-)+Knowledge sources support metadata for better organization and filtering. Metadata is used to filter the knowledge sources when querying the knowledge store.-pricing_info = StringKnowledgeSource(- content="Pricing information...",- collection_name="product_pricing"+```python Code+knowledge_source = StringKnowledgeSource(+ content="Users name is John. He is 30 years old and lives in San Francisco.",+ metadata={"preference": "personal"} # Metadata is used to filter the knowledge sources)```-### Metadata and Filtering+### Chunking Configuration-Add metadata to organize and filter knowledge:+Control how content is split for processing by setting the chunk size and overlap.-```python+```python Codeknowledge_source = StringKnowledgeSource(- content="Product details...",- metadata={- "category": "electronics",- "product_line": "premium",- "last_updated": "2024-03"- }+ content="Long content...",+ chunk_size=4000, # Characters per chunk (default)+ chunk_overlap=200 # Overlap between chunks (default))```-### Chunking Configuration+## Embedder Configuration-Control how your content is split for processing:+You can also configure the embedder for the knowledge store. This is useful if you want to use a different embedder for the knowledge store than the one used for the agents.-```python-knowledge_source = PDFKnowledgeSource(- file_path="product_manual.pdf",- chunk_size=2000, # Characters per chunk- chunk_overlap=200 # Overlap between chunks+```python Code+...+string_source = StringKnowledgeSource(+ content="Users name is John. He is 30 years old and lives in San Francisco.",+ metadata={"preference": "personal"}+)+crew = Crew(+ ...+ knowledge={+ "sources": [string_source],+ "metadata": {"preference": "personal"},+ "embedder_config": {+ "provider": "openai", # Default embedder provider; can be "ollama", "gemini", e.t.c.+ "config": {"model": "text-embedding-3-small"} # Default embedder model; can be "mxbai-embed-large", "nomic-embed-tex", e.t.c.+ },+ },)```-## Advanced Usage+## Custom Knowledge Sources-### Custom Knowledge Sources+CrewAI allows you to create custom knowledge sources for any type of data by extending the `BaseKnowledgeSource` class. Let's create a practical example that fetches and processes space news articles.-Create your own knowledge source by extending the base class:+#### Space News Knowledge Source Example-```python-from crewai.knowledge.source import BaseKnowledgeSource+<CodeGroup>-class APIKnowledgeSource(BaseKnowledgeSource):- def __init__(self, api_endpoint: str, **kwargs):- super().__init__(**kwargs)- self.api_endpoint = api_endpoint+```python Code+from crewai import Agent, Task, Crew, Process, LLM+from crewai.knowledge.source.base_knowledge_source import BaseKnowledgeSource+import requests+from datetime import datetime+from typing import Dict, Any+from pydantic import BaseModel, Field++class SpaceNewsKnowledgeSource(BaseKnowledgeSource):+ """Knowledge source that fetches data from Space News API."""++ api_endpoint: str = Field(description="API endpoint URL")+ limit: int = Field(default=10, description="Number of articles to fetch")- def load_content(self):- # Implement API data fetching- response = requests.get(self.api_endpoint)- return response.json()+ def load_content(self) -> Dict[Any, str]:+ """Fetch and format space news articles."""+ try:+ response = requests.get(+ f"{self.api_endpoint}?limit={self.limit}"+ )+ response.raise_for_status()++ data = response.json()+ articles = data.get('results', [])++ formatted_data = self._format_articles(articles)+ return {self.api_endpoint: formatted_data}+ except Exception as e:+ raise ValueError(f"Failed to fetch space news: {str(e)}")- def add(self):+ def _format_articles(self, articles: list) -> str:+ """Format articles into readable text."""+ formatted = "Space News Articles:\n\n"+ for article in articles:+ formatted += f"""+Title: {article['title']}+Published: {article['published_at']}+Summary: {article['summary']}+News Site: {article['news_site']}+URL: {article['url']}+-------------------"""+ return formatted++ def add(self) -> None:+ """Process and store the articles."""content = self.load_content()- # Process and store content- self.save_documents({"source": "api"})-```+ for _, text in content.items():+ chunks = self._chunk_text(text)+ self.chunks.extend(chunks)++ self.save_documents(metadata={+ "source": "space_news_api",+ "timestamp": datetime.now().isoformat(),+ "article_count": self.limit+ })++# Create knowledge source+recent_news = SpaceNewsKnowledgeSource(+ api_endpoint="https://api.spaceflightnewsapi.net/v4/articles",+ limit=10,+ metadata={"category": "recent_news", "source": "spaceflight_news"}+)-### Embedder Configuration+# Create specialized agent+space_analyst = Agent(+ role="Space News Analyst",+ goal="Answer questions about space news accurately and comprehensively",+ backstory="""You are a space industry analyst with expertise in space exploration,+ satellite technology, and space industry trends. You excel at answering questions+ about space news and providing detailed, accurate information.""",+ knowledge_sources=[recent_news],+ llm=LLM(model="gpt-4", temperature=0.0)+)-Customize the embedding process:+# Create task that handles user questions+analysis_task = Task(+ description="Answer this question about space news: {user_question}",+ expected_output="A detailed answer based on the recent space news articles",+ agent=space_analyst+)-```python+# Create and run the crewcrew = Crew(- agents=[agent],- tasks=[task],- knowledge_sources=[source],- embedder={- "provider": "ollama",- "config": {"model": "nomic-embed-text:latest"},- }+ agents=[space_analyst],+ tasks=[analysis_task],+ verbose=True,+ process=Process.sequential+)++# Example usage+result = crew.kickoff(+ inputs={"user_question": "What are the latest developments in space exploration?"})```+```output Output+# Agent: Space News Analyst+## Task: Answer this question about space news: What are the latest developments in space exploration?+++# Agent: Space News Analyst+## Final Answer:+The latest developments in space exploration, based on recent space news articles, include the following:++1. SpaceX has received the final regulatory approvals to proceed with the second integrated Starship/Super Heavy launch, scheduled for as soon as the morning of Nov. 17, 2023. This is a significant step in SpaceX's ambitious plans for space exploration and colonization. [Source: SpaceNews](https://spacenews.com/starship-cleared-for-nov-17-launch/)++2. SpaceX has also informed the US Federal Communications Commission (FCC) that it plans to begin launching its first next-generation Starlink Gen2 satellites. This represents a major upgrade to the Starlink satellite internet service, which aims to provide high-speed internet access worldwide. [Source: Teslarati](https://www.teslarati.com/spacex-first-starlink-gen2-satellite-launch-2022/)++3. AI startup Synthetaic has raised $15 million in Series B funding. The company uses artificial intelligence to analyze data from space and air sensors, which could have significant applications in space exploration and satellite technology. [Source: SpaceNews](https://spacenews.com/ai-startup-synthetaic-raises-15-million-in-series-b-funding/)++4. The Space Force has formally established a unit within the U.S. Indo-Pacific Command, marking a permanent presence in the Indo-Pacific region. This could have significant implications for space security and geopolitics. [Source: SpaceNews](https://spacenews.com/space-force-establishes-permanent-presence-in-indo-pacific-region/)++5. Slingshot Aerospace, a space tracking and data analytics company, is expanding its network of ground-based optical telescopes to increase coverage of low Earth orbit. This could improve our ability to track and analyze objects in low Earth orbit, including satellites and space debris. [Source: SpaceNews](https://spacenews.com/slingshots-space-tracking-network-to-extend-coverage-of-low-earth-orbit/)++6. The National Natural Science Foundation of China has outlined a five-year project for researchers to study the assembly of ultra-large spacecraft. This could lead to significant advancements in spacecraft technology and space exploration capabilities. [Source: SpaceNews](https://spacenews.com/china-researching-challenges-of-kilometer-scale-ultra-large-spacecraft/)++7. The Center for AEroSpace Autonomy Research (CAESAR) at Stanford University is focusing on spacecraft autonomy. The center held a kickoff event on May 22, 2024, to highlight the industry, academia, and government collaboration it seeks to foster. This could lead to significant advancements in autonomous spacecraft technology. [Source: SpaceNews](https://spacenews.com/stanford-center-focuses-on-spacecraft-autonomy/)+```+</CodeGroup>+#### Key Components Explained++1. **Custom Knowledge Source (`SpaceNewsKnowledgeSource`)**:+ - Extends `BaseKnowledgeSource` for integration with CrewAI+ - Configurable API endpoint and article limit+ - Implements three key methods:+ - `load_content()`: Fetches articles from the API+ - `_format_articles()`: Structures the articles into readable text+ - `add()`: Processes and stores the content with metadata++2. **Agent Configuration**:+ - Specialized role as a Space News Analyst+ - Uses the knowledge source to access space news-### Referencing Sources+3. **Task Setup**:+ - Takes a user question as input through `{user_question}`+ - Designed to provide detailed answers based on the knowledge source-You can reference knowledge sources by their collection name or metadata.+4. **Crew Orchestration**:+ - Manages the workflow between agent and task+ - Handles input/output through the kickoff method-* Add a directory to your crew project called `knowledge`:-* File paths in knowledge can be referenced relative to the `knowledge` directory.+This example demonstrates how to:+- Create a custom knowledge source that fetches real-time data+- Process and format external data for AI consumption+- Use the knowledge source to answer specific user questions+- Integrate everything seamlessly with CrewAI's agent system-Example:-A file inside the `knowledge` directory called `example.txt` can be referenced as `example.txt`.+#### About the Spaceflight News API+The example uses the [Spaceflight News API](https://api.spaceflightnewsapi.net/v4/documentation), which:+- Provides free access to space-related news articles+- Requires no authentication+- Returns structured data about space news+- Supports pagination and filtering++You can customize the API query by modifying the endpoint URL:```python-source = TextFileKnowledgeSource(- file_path="example.txt", # or /example.txt- collection_name="example"+# Fetch more articles+recent_news = SpaceNewsKnowledgeSource(+ api_endpoint="https://api.spaceflightnewsapi.net/v4/articles",+ limit=20, # Increase the number of articles+ metadata={"category": "recent_news"})-crew = Crew(- agents=[agent],- tasks=[task],- knowledge_sources=[source],++# Add search parameters+recent_news = SpaceNewsKnowledgeSource(+ api_endpoint="https://api.spaceflightnewsapi.net/v4/articles?search=NASA", # Search for NASA news+ limit=10,+ metadata={"category": "nasa_news"})```@@ -189,43 +314,16 @@ crew = Crew(<AccordionGroup><Accordion title="Content Organization">- - Use meaningful collection names- - Add detailed metadata for filtering- - Keep chunk sizes appropriate for your content+ - Use descriptive metadata for better filtering+ - Keep chunk sizes appropriate for your content type- Consider content overlap for context preservation+ - Organize related information into separate knowledge sources</Accordion><Accordion title="Performance Tips">- - Use smaller chunk sizes for precise retrieval- - Implement metadata filtering for faster searches- - Choose appropriate embedding models for your use case- - Cache frequently accessed knowledge- </Accordion>-- <Accordion title="Error Handling">- - Validate knowledge source content- - Handle missing or corrupted files- - Monitor embedding generation- - Implement fallback options- </Accordion>-</AccordionGroup>--## Common Issues and Solutions--<AccordionGroup>- <Accordion title="Content Not Found">- If agents can't find relevant information:- - Check chunk sizes- - Verify knowledge source loading- - Review metadata filters- - Test with simpler queries first- </Accordion>-- <Accordion title="Performance Issues">- If knowledge retrieval is slow:- - Reduce chunk sizes- - Optimize metadata filtering- - Consider using a lighter embedding model- - Cache frequently accessed content+ - Use metadata filtering to narrow search scope+ - Adjust chunk sizes based on content complexity+ - Configure appropriate embedding models+ - Consider using local embedding providers for faster processing</Accordion>-</AccordionGroup>+</AccordionGroup>\ No newline at end of file