Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
Add documentation for mlflow tracing integration (#1988)
TomuHirata9cf3fadd0f08
diff --git a/docs/how-to/mlflow-observability.mdx b/docs/how-to/mlflow-observability.mdxnew file mode 100644index 000000000..f0c4a9ec4--- /dev/null+++ b/docs/how-to/mlflow-observability.mdx@@ -0,0 +1,206 @@+---+title: Agent Monitoring with MLflow+description: Quickly start monitoring your Agents with MLflow.+icon: bars-staggered+---++# MLflow Overview++[MLflow](https://mlflow.org/) is an open-source platform to assist machine learning practitioners and teams in handling the complexities of the machine learning process.++It provides a tracing feature that enhances LLM observability in your Generative AI applications by capturing detailed information about the execution of your application’s services.+Tracing provides a way to record the inputs, outputs, and metadata associated with each intermediate step of a request, enabling you to easily pinpoint the source of bugs and unexpected behaviors.++++### Features++- **Tracing Dashboard**: Monitor activities of your crewAI agents with detailed dashboards that include inputs, outputs and metadata of spans.+- **Automated Tracing**: A fully automated integration with crewAI, which can be enabled by running `mlflow.crewai.autolog()`.+- **Manual Trace Instrumentation with minor efforts**: Customize trace instrumentation through MLflow's high-level fluent APIs such as decorators, function wrappers and context managers.+- **OpenTelemetry Compatibility**: MLflow Tracing supports exporting traces to an OpenTelemetry Collector, which can then be used to export traces to various backends such as Jaeger, Zipkin, and AWS X-Ray.+- **Package and Deploy Agents**: Package and deploy your crewAI agents to an inference server with a variety of deployment targets.+- **Securely Host LLMs**: Host multiple LLM from various providers in one unified endpoint through MFflow gateway.+- **Evaluation**: Evaluate your crewAI agents with a wide range of metrics using a convenient API `mlflow.evaluate()`.++## Setup Instructions++<Steps>+ <Step title="Install MLflow package">+ ```shell+ # The crewAI integration is available in mlflow>=2.19.0+ pip install mlflow+ ```+ </Step>+ <Step title="Start MFflow tracking server">+ ```shell+ # This process is optional, but it is recommended to use MLflow tracking server for better visualization and broader features.+ mlflow server+ ```+ </Step>+ <Step title="Initialize MLflow in Your Application">+ Add the following two lines to your application code:++ ```python+ import mlflow++ mlflow.crewai.autolog()++ # Optional: Set a tracking URI and an experiment name if you have a tracking server+ mlflow.set_tracking_uri("http://localhost:5000")+ mlflow.set_experiment("CrewAI")+ ```++ Example Usage for tracing CrewAI Agents:++ ```python+ from crewai import Agent, Crew, Task+ from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource+ from crewai_tools import SerperDevTool, WebsiteSearchTool++ from textwrap import dedent++ content = "Users name is John. He is 30 years old and lives in San Francisco."+ string_source = StringKnowledgeSource(+ content=content, metadata={"preference": "personal"}+ )++ search_tool = WebsiteSearchTool()+++ class TripAgents:+ def city_selection_agent(self):+ return Agent(+ role="City Selection Expert",+ goal="Select the best city based on weather, season, and prices",+ backstory="An expert in analyzing travel data to pick ideal destinations",+ tools=[+ search_tool,+ ],+ verbose=True,+ )++ def local_expert(self):+ return Agent(+ role="Local Expert at this city",+ goal="Provide the BEST insights about the selected city",+ backstory="""A knowledgeable local guide with extensive information+ about the city, it's attractions and customs""",+ tools=[search_tool],+ verbose=True,+ )+++ class TripTasks:+ def identify_task(self, agent, origin, cities, interests, range):+ return Task(+ description=dedent(+ f"""+ Analyze and select the best city for the trip based+ on specific criteria such as weather patterns, seasonal+ events, and travel costs. This task involves comparing+ multiple cities, considering factors like current weather+ conditions, upcoming cultural or seasonal events, and+ overall travel expenses.+ Your final answer must be a detailed+ report on the chosen city, and everything you found out+ about it, including the actual flight costs, weather+ forecast and attractions.++ Traveling from: {origin}+ City Options: {cities}+ Trip Date: {range}+ Traveler Interests: {interests}+ """+ ),+ agent=agent,+ expected_output="Detailed report on the chosen city including flight costs, weather forecast, and attractions",+ )++ def gather_task(self, agent, origin, interests, range):+ return Task(+ description=dedent(+ f"""+ As a local expert on this city you must compile an+ in-depth guide for someone traveling there and wanting+ to have THE BEST trip ever!+ Gather information about key attractions, local customs,+ special events, and daily activity recommendations.+ Find the best spots to go to, the kind of place only a+ local would know.+ This guide should provide a thorough overview of what+ the city has to offer, including hidden gems, cultural+ hotspots, must-visit landmarks, weather forecasts, and+ high level costs.+ The final answer must be a comprehensive city guide,+ rich in cultural insights and practical tips,+ tailored to enhance the travel experience.++ Trip Date: {range}+ Traveling from: {origin}+ Traveler Interests: {interests}+ """+ ),+ agent=agent,+ expected_output="Comprehensive city guide including hidden gems, cultural hotspots, and practical travel tips",+ )+++ class TripCrew:+ def __init__(self, origin, cities, date_range, interests):+ self.cities = cities+ self.origin = origin+ self.interests = interests+ self.date_range = date_range++ def run(self):+ agents = TripAgents()+ tasks = TripTasks()++ city_selector_agent = agents.city_selection_agent()+ local_expert_agent = agents.local_expert()++ identify_task = tasks.identify_task(+ city_selector_agent,+ self.origin,+ self.cities,+ self.interests,+ self.date_range,+ )+ gather_task = tasks.gather_task(+ local_expert_agent, self.origin, self.interests, self.date_range+ )++ crew = Crew(+ agents=[city_selector_agent, local_expert_agent],+ tasks=[identify_task, gather_task],+ verbose=True,+ memory=True,+ knowledge={+ "sources": [string_source],+ "metadata": {"preference": "personal"},+ },+ )++ result = crew.kickoff()+ return result+++ trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")+ result = trip_crew.run()++ print(result)+ ```+ Refer to [MLflow Tracing Documentation](https://mlflow.org/docs/latest/llms/tracing/index.html) for more configurations and use cases.+ </Step>+ <Step title="Visualize Activities of Agents">+ Now traces for your crewAI agents are captured by MLflow.+ Let's visit MLflow tracking server to view the traces and get insights into your Agents.++ Open `127.0.0.1:5000` on your browser to visit MLflow tracking server.+ <Frame caption="MLflow Tracing Dashboard">+ <img src="/images/mlflow1.png" alt="MLflow tracing example with crewai" />+ </Frame>+ </Step>+</Steps>+diff --git a/docs/images/mlflow-tracing.gif b/docs/images/mlflow-tracing.gifnew file mode 100644index 000000000..e968c0924Binary files /dev/null and b/docs/images/mlflow-tracing.gif differdiff --git a/docs/images/mlflow1.png b/docs/images/mlflow1.pngnew file mode 100644index 000000000..b2c6f3b80Binary files /dev/null and b/docs/images/mlflow1.png differdiff --git a/docs/mint.json b/docs/mint.jsonindex 585fc0abd..a49f60946 100644--- a/docs/mint.json+++ b/docs/mint.json@@ -101,6 +101,7 @@"how-to/conditional-tasks","how-to/agentops-observability","how-to/langtrace-observability",+ "how-to/mlflow-observability","how-to/openlit-observability","how-to/portkey-observability"]diff --git a/mkdocs.yml b/mkdocs.ymlindex 948744de2..511078fff 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -152,6 +152,7 @@ nav:- Agent Monitoring with AgentOps: 'how-to/AgentOps-Observability.md'- Agent Monitoring with LangTrace: 'how-to/Langtrace-Observability.md'- Agent Monitoring with OpenLIT: 'how-to/openlit-Observability.md'+ - Agent Monitoring with MLflow: 'how-to/mlflow-Observability.md'- Tools Docs:- Browserbase Web Loader: 'tools/BrowserbaseLoadTool.md'- Code Docs RAG Search: 'tools/CodeDocsSearchTool.md'