Skip to content

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

feat(docs): add LlamaIndex integration documentation (#1064)

devin-ai-integration[bot]d48ef4429850
diff --git a/docs/mint.json b/docs/mint.json
index 38041ae0..2e80ea8c 100644
--- a/docs/mint.json
+++ b/docs/mint.json
@@ -106,6 +106,7 @@
"v1/integrations/groq",
"v1/integrations/haystack",
"v1/integrations/langchain",
+ "v1/integrations/llamaindex",
"v1/integrations/llama_stack",
"v1/integrations/litellm",
"v1/integrations/mistral",
@@ -175,6 +176,7 @@
"v2/integrations/google_adk",
"v2/integrations/google_generative_ai",
"v2/integrations/langchain",
+ "v2/integrations/llamaindex",
"v2/integrations/litellm",
"v2/integrations/mem0",
"v2/integrations/openai",
diff --git a/docs/v1/integrations/llamaindex.mdx b/docs/v1/integrations/llamaindex.mdx
new file mode 100644
index 00000000..b83d119c
--- /dev/null
+++ b/docs/v1/integrations/llamaindex.mdx
@@ -0,0 +1,88 @@
+---
+title: 'LlamaIndex'
+description: 'AgentOps works seamlessly with LlamaIndex, a framework for building context-augmented generative AI applications with LLMs.'
+---
+
+import CodeTooltip from '/snippets/add-code-tooltip.mdx'
+import EnvTooltip from '/snippets/add-env-tooltip.mdx'
+
+[LlamaIndex](https://www.llamaindex.ai/) is a framework for building context-augmented generative AI applications with LLMs. AgentOps provides observability into your LlamaIndex applications through automatic instrumentation.
+
+<Steps>
+ <Step title="Install the AgentOps SDK">
+ <CodeGroup>
+ ```bash pip
+ pip install agentops
+ ```
+ ```bash poetry
+ poetry add agentops
+ ```
+ </CodeGroup>
+ </Step>
+ <Step title="Install LlamaIndex AgentOps Instrumentation">
+ <CodeGroup>
+ ```bash pip
+ pip install llama-index-instrumentation-agentops
+ ```
+ ```bash poetry
+ poetry add llama-index-instrumentation-agentops
+ ```
+ </CodeGroup>
+ </Step>
+ <Step title="Add 2 lines of code">
+ <CodeTooltip/>
+ <CodeGroup>
+ ```python python
+ from llama_index.core import set_global_handler
+
+ # NOTE: Feel free to set your AgentOps environment variables (e.g., 'AGENTOPS_API_KEY')
+ # as outlined in the AgentOps documentation, or pass the equivalent keyword arguments
+ # anticipated by AgentOps' AOClient as **eval_params in set_global_handler.
+
+ set_global_handler("agentops")
+ ```
+ </CodeGroup>
+ <EnvTooltip />
+ <CodeGroup>
+ ```python .env
+ AGENTOPS_API_KEY=<YOUR API KEY>
+ ```
+ </CodeGroup>
+ Read more about environment variables in [Advanced Configuration](/v1/usage/advanced-configuration)
+ </Step>
+ <Step title="Run your LlamaIndex application">
+ Execute your program and visit [app.agentops.ai/drilldown](https://app.agentops.ai/drilldown) to observe your LlamaIndex application! 🕵️
+ <Tip>
+ After your run, AgentOps prints a clickable url to console linking directly to your session in the Dashboard
+ </Tip>
+ <div/>{/* Intentionally blank div for newline */}
+ <Frame type="glass" caption="Clickable link to session">
+ <img height="200" src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/link-to-session.gif?raw=true" />
+ </Frame>
+ </Step>
+</Steps>
+
+## Usage Pattern
+
+Here's a simple example of how to use AgentOps with LlamaIndex:
+
+```python
+from llama_index.core import set_global_handler
+import llama_index.core
+
+# Set the global handler to AgentOps
+set_global_handler("agentops")
+
+# Your LlamaIndex application code here
+# AgentOps will automatically track LLM calls and other operations
+```
+
+## Additional Resources
+
+For more detailed information about LlamaIndex's observability features and AgentOps integration, check out the [LlamaIndex documentation](https://docs.llamaindex.ai/en/stable/module_guides/observability/#agentops).
+
+<script type="module" src="/scripts/github_stars.js"></script>
+<script type="module" src="/scripts/scroll-img-fadein-animation.js"></script>
+<script type="module" src="/scripts/button_heartbeat_animation.js"></script>
+<script type="css" src="/styles/styles.css"></script>
+<script type="module" src="/scripts/adjust_api_dynamically.js"></script>
diff --git a/docs/v2/integrations/llamaindex.mdx b/docs/v2/integrations/llamaindex.mdx
new file mode 100644
index 00000000..79a149e6
--- /dev/null
+++ b/docs/v2/integrations/llamaindex.mdx
@@ -0,0 +1,79 @@
+---
+title: 'LlamaIndex'
+description: 'AgentOps works seamlessly with LlamaIndex, a framework for building context-augmented generative AI applications with LLMs.'
+---
+
+[LlamaIndex](https://www.llamaindex.ai/) is a framework for building context-augmented generative AI applications with LLMs. AgentOps provides comprehensive observability into your LlamaIndex applications through automatic instrumentation, allowing you to monitor LLM calls, track performance, and analyze your application's behavior.
+
+## Installation
+
+Install AgentOps and the LlamaIndex AgentOps instrumentation package:
+
+<CodeGroup>
+ ```bash pip
+ pip install agentops llama-index-instrumentation-agentops
+ ```
+ ```bash poetry
+ poetry add agentops llama-index-instrumentation-agentops
+ ```
+ ```bash uv
+ uv add agentops llama-index-instrumentation-agentops
+ ```
+</CodeGroup>
+
+## Setting Up API Keys
+
+You'll need an AgentOps API key from your [AgentOps Dashboard](https://app.agentops.ai/):
+
+<CodeGroup>
+ ```bash Export to CLI
+ export AGENTOPS_API_KEY="your_agentops_api_key_here"
+ ```
+ ```txt Set in .env file
+ AGENTOPS_API_KEY="your_agentops_api_key_here"
+ ```
+</CodeGroup>
+
+## Usage
+
+Simply set the global handler to "agentops" at the beginning of your LlamaIndex application. AgentOps will automatically instrument LlamaIndex to track your LLM interactions and application performance.
+
+```python
+from llama_index.core import set_global_handler
+from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
+
+# Set the global handler to AgentOps
+# NOTE: Feel free to set your AgentOps environment variables (e.g., 'AGENTOPS_API_KEY')
+# as outlined in the AgentOps documentation, or pass the equivalent keyword arguments
+# anticipated by AgentOps' AOClient as **eval_params in set_global_handler.
+set_global_handler("agentops")
+
+# Your LlamaIndex application code here
+documents = SimpleDirectoryReader("data").load_data()
+index = VectorStoreIndex.from_documents(documents)
+
+# Create a query engine
+query_engine = index.as_query_engine()
+
+# Query your data - AgentOps will automatically track this
+response = query_engine.query("What is the main topic of these documents?")
+print(response)
+```
+
+## What Gets Tracked
+
+When you use AgentOps with LlamaIndex, the following operations are automatically tracked:
+
+- **LLM Calls**: All interactions with language models including prompts, completions, and token usage
+- **Embeddings**: Vector embedding generation and retrieval operations
+- **Query Operations**: Search and retrieval operations on your indexes
+- **Performance Metrics**: Response times, token costs, and success/failure rates
+
+## Additional Resources
+
+For more detailed information about LlamaIndex's observability features and AgentOps integration, check out the [LlamaIndex documentation](https://docs.llamaindex.ai/en/stable/module_guides/observability/#agentops).
+
+<script type="module" src="/scripts/github_stars.js"></script>
+<script type="module" src="/scripts/scroll-img-fadein-animation.js"></script>
+<script type="module" src="/scripts/button_heartbeat_animation.js"></script>
+<script type="css" src="/styles/styles.css"></script>
diff --git a/examples/llamaindex_examples/README.md b/examples/llamaindex_examples/README.md
new file mode 100644
index 00000000..1e73b5f2
--- /dev/null
+++ b/examples/llamaindex_examples/README.md
@@ -0,0 +1,35 @@
+# LlamaIndex AgentOps Integration Example
+
+This example demonstrates how to use AgentOps with LlamaIndex for observability and monitoring of your context-augmented generative AI applications.
+
+## Setup
+
+1. Install required packages:
+```bash
+pip install agentops llama-index-instrumentation-agentops llama-index python-dotenv
+```
+
+2. Set your API keys:
+```bash
+export AGENTOPS_API_KEY="your_agentops_api_key"
+export OPENAI_API_KEY="your_openai_api_key"
+```
+
+## Files
+
+- `llamaindex_example.py` - Python script example
+- `llamaindex_example.ipynb` - Jupyter notebook example
+
+## Usage
+
+Run the Python script:
+```bash
+python llamaindex_example.py
+```
+
+Or open and run the Jupyter notebook:
+```bash
+jupyter notebook llamaindex_example.ipynb
+```
+
+After running, check your AgentOps dashboard for the recorded session.
diff --git a/examples/llamaindex_examples/llamaindex_example.ipynb b/examples/llamaindex_examples/llamaindex_example.ipynb
new file mode 100644
index 00000000..594d5350
--- /dev/null
+++ b/examples/llamaindex_examples/llamaindex_example.ipynb
@@ -0,0 +1,196 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a886f0ad",
+ "metadata": {},
+ "source": [
+ "# LlamaIndex AgentOps Integration Example\n",
+ "\n",
+ "This notebook demonstrates how to use AgentOps with LlamaIndex for observability and monitoring of your context-augmented generative AI applications.\n",
+ "\n",
+ "## Setup\n",
+ "\n",
+ "First, install the required packages:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "002718a0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install required packages\n",
+ "!pip install agentops llama-index-instrumentation-agentops llama-index-embeddings-huggingface llama-index-llms-huggingface python-dotenv"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bc009657",
+ "metadata": {},
+ "source": [
+ "## Initialize AgentOps Handler\n",
+ "\n",
+ "Set up the AgentOps handler for LlamaIndex instrumentation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1d8a0c2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "from dotenv import load_dotenv\n",
+ "from llama_index.core import VectorStoreIndex, Document, Settings\n",
+ "from llama_index.instrumentation.agentops import AgentOpsHandler\n",
+ "\n",
+ "# Initialize AgentOps handler\n",
+ "handler = AgentOpsHandler()\n",
+ "handler.init()\n",
+ "\n",
+ "# Load environment variables\n",
+ "load_dotenv()\n",
+ "\n",
+ "# Set API keys (replace with your actual keys)\n",
+ "os.environ[\"AGENTOPS_API_KEY\"] = os.getenv(\"AGENTOPS_API_KEY\", \"your_agentops_api_key_here\")\n",
+ "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\", \"your_openai_api_key_here\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8794d1f2",
+ "metadata": {},
+ "source": [
+ "## Configure Local Models (Optional)\n",
+ "\n",
+ "For this example, we'll use local HuggingFace models to avoid requiring external API keys:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "35804326",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
+ "from llama_index.llms.huggingface import HuggingFaceLLM\n",
+ "\n",
+ "# Configure local embeddings and LLM\n",
+ "Settings.embed_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\")\n",
+ "Settings.llm = HuggingFaceLLM(model_name=\"microsoft/DialoGPT-medium\")\n",
+ "print(\"Using local HuggingFace embeddings and LLM\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b23271d3",
+ "metadata": {},
+ "source": [
+ "## Create Sample Documents and Index\n",
+ "\n",
+ "Create some sample documents and build a vector index:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "80b1c26a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print(\"🚀 Starting LlamaIndex AgentOps Integration Example\")\n",
+ "print(\"=\" * 50)\n",
+ "\n",
+ "# Create sample documents\n",
+ "documents = [\n",
+ " Document(text=\"LlamaIndex is a framework for building context-augmented generative AI applications with LLMs.\"),\n",
+ " Document(text=\"AgentOps provides observability into your AI applications, tracking LLM calls, performance metrics, and more.\"),\n",
+ " Document(text=\"The integration between LlamaIndex and AgentOps allows you to monitor your RAG applications seamlessly.\"),\n",
+ " Document(text=\"Vector databases are used to store and retrieve embeddings for similarity search in RAG applications.\"),\n",
+ " Document(text=\"Context-augmented generation combines retrieval and generation to provide more accurate and relevant responses.\")\n",
+ "]\n",
+ "\n",
+ "print(\"📚 Creating vector index from sample documents...\")\n",
+ "index = VectorStoreIndex.from_documents(documents)\n",
+ "print(\"✅ Vector index created successfully\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e07f1d6a",
+ "metadata": {},
+ "source": [
+ "## Perform Queries\n",
+ "\n",
+ "Now let's perform some queries to demonstrate the AgentOps integration:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5882dd2a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create query engine\n",
+ "query_engine = index.as_query_engine()\n",
+ "\n",
+ "print(\"🔍 Performing queries...\")\n",
+ "\n",
+ "# Sample queries\n",
+ "queries = [\n",
+ " \"What is LlamaIndex?\",\n",
+ " \"How does AgentOps help with AI applications?\",\n",
+ " \"What are the benefits of using vector databases in RAG?\"\n",
+ "]\n",
+ "\n",
+ "for i, query in enumerate(queries, 1):\n",
+ " print(f\"\\n📝 Query {i}: {query}\")\n",
+ " response = query_engine.query(query)\n",
+ " print(f\"💬 Response: {response}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b46b944c",
+ "metadata": {},
+ "source": [
+ "## Results\n",
+ "\n",
+ "After running this notebook, you should see:\n",
+ "\n",
+ "1. **AgentOps Session Link**: A URL to view the session in your AgentOps dashboard\n",
+ "2. **Cost Tracking**: Information about the cost of LLM calls (if using paid APIs)\n",
+ "3. **Operation Tracking**: All LlamaIndex operations are automatically tracked\n",
+ "\n",
+ "Check your AgentOps dashboard to see detailed information about:\n",
+ "- LLM calls and responses\n",
+ "- Performance metrics\n",
+ "- Cost analysis\n",
+ "- Session replay\n",
+ "\n",
+ "The session link will be printed in the output above by AgentOps."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c141973e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print(\"\\n\" + \"=\" * 50)\n",
+ "print(\"🎉 Example completed successfully!\")\n",
+ "print(\"📊 Check your AgentOps dashboard to see the recorded session with LLM calls and operations.\")\n",
+ "print(\"🔗 The session link should be printed above by AgentOps.\")"
+ ]
+ }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/llamaindex_examples/llamaindex_example.py b/examples/llamaindex_examples/llamaindex_example.py
new file mode 100644
index 00000000..ed7826a5
--- /dev/null
+++ b/examples/llamaindex_examples/llamaindex_example.py
@@ -0,0 +1,66 @@
+#
+#
+
+import os
+from dotenv import load_dotenv
+from llama_index.core import VectorStoreIndex, Document, Settings
+from llama_index.instrumentation.agentops import AgentOpsHandler
+from llama_index.embeddings.huggingface import HuggingFaceEmbedding
+from llama_index.llms.huggingface import HuggingFaceLLM
+
+handler = AgentOpsHandler()
+handler.init()
+
+load_dotenv()
+
+os.environ["AGENTOPS_API_KEY"] = os.getenv("AGENTOPS_API_KEY", "your_agentops_api_key_here")
+os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY", "your_openai_api_key_here")
+
+Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
+Settings.llm = HuggingFaceLLM(model_name="microsoft/DialoGPT-medium")
+print("Using local HuggingFace embeddings and LLM")
+
+print("🚀 Starting LlamaIndex AgentOps Integration Example")
+print("=" * 50)
+
+documents = [
+ Document(text="LlamaIndex is a framework for building context-augmented generative AI applications with LLMs."),
+ Document(
+ text="AgentOps provides observability into your AI applications, tracking LLM calls, performance metrics, and more."
+ ),
+ Document(
+ text="The integration between LlamaIndex and AgentOps allows you to monitor your RAG applications seamlessly."
+ ),
+ Document(
+ text="Vector databases are used to store and retrieve embeddings for similarity search in RAG applications."
+ ),
+ Document(
+ text="Context-augmented generation combines retrieval and generation to provide more accurate and relevant responses."
+ ),
+]
+
+print("📚 Creating vector index from sample documents...")
+
+index = VectorStoreIndex.from_documents(documents)
+
+print("✅ Vector index created successfully")
+
+query_engine = index.as_query_engine()
+
+print("🔍 Performing queries...")
+
+queries = [
+ "What is LlamaIndex?",
+ "How does AgentOps help with AI applications?",
+ "What are the benefits of using vector databases in RAG?",
+]
+
+for i, query in enumerate(queries, 1):
+ print(f"\n📝 Query {i}: {query}")
+ response = query_engine.query(query)
+ print(f"💬 Response: {response}")
+
+print("\n" + "=" * 50)
+print("🎉 Example completed successfully!")
+print("📊 Check your AgentOps dashboard to see the recorded session with LLM calls and operations.")
+print("🔗 The session link should be printed above by AgentOps.")