Skip to content

armgit / camel

Public Imported from GitHub
0

đŸ« CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org

4.3 KB
Raw Blame

title: "Retrievers" icon: code-compare

doc_code_map: - "camel/retrievers/**/*.py"

What Are Retrievers?

Retrievers are your AI search engine for large text collections or knowledge bases. They let you find the most relevant information based on a query—using either advanced embeddings (semantic search) or classic keyword matching. Want a deep dive? Check out the RAG Cookbook for advanced agent + retrieval use cases.

Types of Retrievers

Converts documents into vectors using an embedding model, stores them, and retrieves by semantic similarity. Best for “meaning-based” search, RAG, and LLM workflows.
  • Chunks data, embeds with OpenAI or custom model
  • Stores in vector DB (like Qdrant)
  • Finds the most relevant info even with different wording
Classic keyword search! Breaks documents and queries into tokens/keywords, and matches on those.
  • Tokenizes documents
  • Indexes by keyword
  • Fast, transparent, great for exact matches

How To Use

This example uses OpenAI embeddings and Qdrant vector storage for semantic search. ```python vector_retriever_setup.py from camel.embeddings import OpenAIEmbedding from camel.retrievers import VectorRetriever from camel.storages.vectordb_storages import QdrantStorage

Set up vector DB for embeddings

vector_storage = QdrantStorage( vector_dim=OpenAIEmbedding().get_output_dim(), collection_name="my first collection", path="storage_customized_run", ) vr = VectorRetriever(embedding_model=OpenAIEmbedding(), storage=vector_storage)

</CodeGroup>

<CodeGroup>
```python vector_retriever_ingest.py
# Embed and store your data (URL or file)
content_input_path = "https://www.camel-ai.org/"
vr.process(content=content_input_path)
```python vector_retriever_query.py # Run a query for semantic search query = "What is CAMEL" results = vr.query(query=query, similarity_threshold=0) print(results) ``` ```markdown vector_retriever_output.md >>> [{'similarity score': '0.81...', 'content path': 'https://www.camel-ai.org/', 'metadata': {...}, 'text': '...CAMEL-AI.org is an open-source community dedicated to the study of autonomous and communicative agents...'}] ``` AutoRetriever simplifies everything: just specify storage and content, and it handles embedding, storage, and querying. ```python auto_retriever.py from camel.retrievers import AutoRetriever from camel.types import StorageType

ar = AutoRetriever( vector_storage_local_path="camel/retrievers", storage_type=StorageType.QDRANT, )

retrieved_info = ar.run_vector_retriever( contents=["https://www.camel-ai.org/"], # One or many URLs/files query="What is CAMEL-AI", return_detailed_info=True, ) print(retrieved_info)

```markdown auto_retriever_output.md
>>> Original Query: {What is CAMEL-AI}
>>> Retrieved Context:
>>> {'similarity score': '0.83...', 'content path': 'https://www.camel-ai.org/', 'metadata': {...}, 'text': 'Mission\n\nCAMEL-AI.org is an open-source community dedicated to the study of autonomous and communicative agents...'}
Use AutoRetriever for fast experiments and RAG workflows; for advanced control, use VectorRetriever directly. For simple, blazing-fast search by keyword—use KeywordRetriever. Great for small data, transparency, or keyword-driven tasks. *(API and code example coming soon—see RAG Cookbook for details.)* Build retrieval-augmented agents using these retrievers. Full configuration and options for all retriever classes.