🐫 CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org
Python agentai-societiesartificial-intelligencecommunicative-aicooperative-aideep-learninglarge-language-modelsmulti-agent-systemsnatural-language-processing
title: "Storages" icon: database
doc_code_map: - "camel/storages/**/*.py"
What Are Storages in CAMEL-AI?
The Storage module in CAMEL-AI gives you a **unified interface for saving, searching, and managing your data** from simple key-value records to high-performance vector databases and modern graph engines. It’s your plug-and-play toolkit for building robust, AI-ready storage layers.Types of Storages
**BaseKeyValueStorage** - Abstract base for all key-value storage backends. - **Standardizes:** Save, load, clear operations. - **Interface:** Python dicts. - **Use cases:** - JSON file storage - NoSQL (MongoDB, Redis) - In-memory caches**InMemoryKeyValueStorage**
- Fast, simple, *not persistent* (resets on restart)
- Ideal for caching, development, or quick prototyping
**JsonStorage**
- Human-readable, portable JSON file storage
- Supports custom JSON encoder (for Enums, etc)
- Good for configs, small persistent datasets, export/import flows
**BaseVectorStorage**
- Abstract base for vector database backends
- **Core operations:** Add/query/delete vectors, check DB status
- **Customizable:** Vector dimensions, collections, distance metrics
**MilvusStorage**
- For [Milvus](https://milvus.io/docs/overview.md/) (cloud-native vector search engine)
- High scalability, real-time search
**TiDBStorage**
- For [TiDB](https://pingcap.com/ai) (hybrid vector/relational database)
- Handles embeddings, knowledge graphs, ops data
**QdrantStorage**
- For [Qdrant](https://qdrant.tech/) (open-source vector DB)
- Fast similarity search for AI/ML
**OceanBaseStorage**
- For [OceanBase](https://www.oceanbase.com/) (cloud and on-prem vector DB)
- Supports large-scale, distributed deployments
**WeaviateStorage**
- For [Weaviate](https://weaviate.io/) (open-source vector engine)
- Schema-based, semantic search, hybrid queries
**ChromaStorage**
- For [ChromaDB](https://www.trychroma.com/) (AI-native open-source embedding database)
- Simple API, scales from notebook to production
**SurrealStorage**
- For [SurrealDB](https://surrealdb.com/) (scalable, distributed database with WebSocket support)
- Efficient vector storage and similarity search with real-time updates
**PgVectorStorage**
- For [PostgreSQL with pgvector](https://github.com/pgvector/pgvector) (open-source vector engine)
- Leverages PostgreSQL for vector search
**BaseGraphStorage**
- Abstract base for graph database integrations
- **Supports:**
- Schema queries and refresh
- Adding/deleting/querying triplets
**NebulaGraph**
- For [NebulaGraph](https://www.nebula-graph.io/) (distributed, high-performance graph DB)
- Scalable, open source
**Neo4jGraph**
- For [Neo4jGraph](https://neo4j.com/) (most popular enterprise graph DB)
- Widely used for graph analytics, recommendations
Get Started
Here are practical usage patterns for each storage type—pick the ones you need and mix them as you like.
Use for: Fast, temporary storage. Data is lost when your program exits. Perfect for: Prototyping, testing, in-memory caching. ```python from camel.storages.key_value_storages import InMemoryKeyValueStorage
memory_storage = InMemoryKeyValueStorage()
memory_storage.save([{'key1': 'value1'}, {'key2': 'value2'}])
records = memory_storage.load()
print(records)
memory_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> [{'key1': 'value1'}, {'key2': 'value2'}]
```
</Tab>
Use for: Persistent, human-readable storage on disk. Perfect for: Logs, local settings, configs, or sharing small data sets. ```python from camel.storages.key_value_storages import JsonStorage from pathlib import Path
json_storage = JsonStorage(Path("my_data.json"))
json_storage.save([{'key1': 'value1'}, {'key2': 'value2'}])
records = json_storage.load()
print(records)
json_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> [{'key1': 'value1'}, {'key2': 'value2'}]
```
</Tab>
Use for: Scalable, high-performance vector search (RAG, embeddings). Perfect for: Semantic search and production AI retrieval. ```python from camel.storages import MilvusStorage, VectorDBQuery, VectorRecord
milvus_storage = MilvusStorage(
url_and_api_key=("Your Milvus URI","Your Milvus Token"),
vector_dim=4,
collection_name="my_collection"
)
milvus_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
milvus_storage.load()
query_results = milvus_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
milvus_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.5669466853141785
```
</Tab>
Use for: Hybrid cloud-native storage, vectors + SQL in one. Perfect for: Combining AI retrieval with your business database. ```python import os from camel.storages import TiDBStorage, VectorDBQuery, VectorRecord
os.environ["TIDB_DATABASE_URL"] = "The database url of your TiDB cluster."
tidb_storage = TiDBStorage(
url_and_api_key=(os.getenv("TIDB_DATABASE_URL"), ''),
vector_dim=4,
collection_name="my_collection"
)
tidb_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
tidb_storage.load()
query_results = tidb_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
tidb_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.5669466755703252
```
</Tab>
Use for: Fast, scalable open-source vector search.
Perfect for: RAG, document search, and high-scale retrieval tasks.
```python
from camel.storages import QdrantStorage, VectorDBQuery, VectorRecord
# Create an instance of QdrantStorage with dimension = 4
qdrant_storage = QdrantStorage(vector_dim=4, collection_name="my_collection")
# Add two vector records
qdrant_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
# Query similar vectors
query_results = qdrant_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
# Clear all vectors
qdrant_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.5669467095138407
```
</Tab>
Use for: Fastest way to build LLM apps with memory and embeddings. Perfect for: From prototyping in notebooks to production clusters with the same simple API. ```python from camel.storages import ChromaStorage, VectorDBQuery, VectorRecord from camel.types import VectorDistance
# Create ChromaStorage instance with ephemeral (in-memory) client
chroma_storage = ChromaStorage(
vector_dim=4,
collection_name="camel_example_vectors",
client_type="ephemeral", # or "persistent", "http", "cloud"
distance=VectorDistance.COSINE,
)
# Add vector records
chroma_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
# Query similar vectors
query_results = chroma_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
# Clear all vectors
chroma_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.7834733426570892
```
</Tab>
Use for: Scalable, distributed vector storage with WebSocket support. Perfect for: Real-time vector search with distributed deployments and SQL-like querying. ```python import os from camel.storages.vectordb_storages import SurrealStorage from camel.storages import VectorDBQuery, VectorRecord
# Set environment variables for SurrealDB connection
os.environ["SURREAL_URL"] = "ws://localhost:8000/rpc"
os.environ["SURREAL_PASSWORD"] = "your_password"
# Create SurrealStorage instance with WebSocket connection
surreal_storage = SurrealStorage(
url=os.getenv("SURREAL_URL"),
table="camel_vectors",
namespace="ns",
database="db",
user="root",
password=os.getenv("SURREAL_PASSWORD"),
vector_dim=4,
)
# Add vector records
surreal_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
# Query similar vectors
query_results = surreal_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
# Clear all vectors
surreal_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.5669467095138407
```
</Tab>
Use for: Massive vector storage with advanced analytics. Perfect for: Batch operations, cloud or on-prem setups, and high-throughput search. ```python import random
from camel.storages.vectordb_storages import (
OceanBaseStorage,
VectorDBQuery,
VectorRecord,
)
# Replace these with your OceanBase connection parameters
OB_URI = "127.0.0.1:2881"
OB_USER = "root@sys"
OB_PASSWORD = ""
OB_DB_NAME = "oceanbase"
def main():
ob_storage = OceanBaseStorage(
vector_dim=4,
table_name="my_ob_vector_table",
uri=OB_URI,
user=OB_USER,
password=OB_PASSWORD,
db_name=OB_DB_NAME,
distance="cosine",
)
status = ob_storage.status()
print(f"Vector dimension: {status.vector_dim}")
print(f"Initial vector count: {status.vector_count}")
random.seed(20241023)
large_batch = []
for i in range(1000):
large_batch.append(
VectorRecord(
vector=[random.uniform(-1, 1) for _ in range(4)],
payload={'idx': i, 'batch': 'example'},
)
)
ob_storage.add(large_batch, batch_size=100)
status = ob_storage.status()
print(f"Vector count after adding batch: {status.vector_count}")
query_vector = [random.uniform(-1, 1) for _ in range(4)]
query_results = ob_storage.query(
VectorDBQuery(query_vector=query_vector, top_k=5)
)
for i, result in enumerate(query_results):
print(f"Result {i+1}:")
print(f" ID: {result.record.id}")
print(f" Payload: {result.record.payload}")
print(f" Similarity: {result.similarity}")
ob_storage.clear()
status = ob_storage.status()
print(f"Vector count after clearing: {status.vector_count}")
if __name__ == "__main__":
main()
```
</Tab>
<Tab title="Output">
```markdown
'''
===============================================================================
Vector dimension: 4
Initial vector count: 0
Adding vectors in batches...
Vector count after adding batch: 1000
Querying similar vectors...
Result 1:
ID: f33f008d-688a-468a-9a2d-e005d27ad9d9
Payload: {'idx': 496, 'batch': 'example'}
Similarity: 0.9847431876706196
Result 2:
ID: 0946ca4a-9129-4343-b339-b2f13a64c827
Payload: {'idx': 306, 'batch': 'example'}
Similarity: 0.9598140307734809
...
Clearing vectors...
Vector count after clearing: 0
===============================================================================
'''
```
</Tab>
Use for: Vector search with hybrid (vector + keyword) capabilities. Perfect for: Document retrieval and multimodal AI apps. ```python from camel.storages import WeaviateStorage, VectorDBQuery, VectorRecord
# Create WeaviateStorage instance with dimension = 4 using Weaviate Cloud
weaviate_storage = WeaviateStorage(
vector_dim=4,
collection_name="camel_example_vectors",
connection_type="cloud",
wcd_cluster_url="your-weaviate-cloud-url",
wcd_api_key="your-weaviate-api-key",
vector_index_type="hnsw",
distance_metric="cosine",
)
weaviate_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
query_results = weaviate_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
weaviate_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.7834733128547668
```
</Tab>
Use for: Open-source, distributed graph storage and querying. Perfect for: Knowledge graphs, relationships, and fast distributed queries. ```python from camel.storages.graph_storages import NebulaGraph
nebula_graph = NebulaGraph("your_host", "your_username", "your_password", "your_space")
# Show existing tags
query = 'SHOW TAGS;'
print(nebula_graph.query(query))
```
Use for: Industry-standard graph database for large-scale relationships. Perfect for: Enterprise graph workloads, Cypher queries, analytics. ```python from camel.storages import Neo4jGraph
neo4j_graph = Neo4jGraph(
url="your_url",
username="your_username",
password="your_password",
)
query = "MATCH (n) DETACH DELETE n"
print(neo4j_graph.query(query))
```
</CodeGroup>
Use for: Storing and querying vectors in PostgreSQL. Perfect for: Leveraging an existing PostgreSQL database for vector search. ```python from camel.storages import PgVectorStorage, VectorDBQuery, VectorRecord
# Replace with your PostgreSQL connection details
PG_CONN_INFO = {
"host": "127.0.0.1",
"port": 5432,
"user": "postgres",
"password": "postgres",
"dbname": "postgres",
}
# Create PgVectorStorage instance
pg_storage = PgVectorStorage(
vector_dim=4,
conn_info=PG_CONN_INFO,
table_name="camel_example_vectors",
)
# Add vector records
pg_storage.add([
VectorRecord(vector=[-0.1, 0.1, -0.1, 0.1], payload={'key1': 'value1'}),
VectorRecord(vector=[-0.1, 0.1, 0.1, 0.1], payload={'key2': 'value2'}),
])
# Query similar vectors
query_results = pg_storage.query(VectorDBQuery(query_vector=[0.1, 0.2, 0.1, 0.1], top_k=1))
for result in query_results:
print(result.record.payload, result.similarity)
# Clear all vectors
pg_storage.clear()
```
</Tab>
<Tab title="Output">
```markdown
>>> {'key2': 'value2'} 0.5669467
```
</Tab>