The Memory Layer for AI Agents - Drop-in memory infrastructure for AI agents and apps. Context that persists. Built for production.
Python agentic-memoryagentic-memory-systemagentsaiai-agentschatgptgenaillmlong-term-memorymemorymemory-managementpythonragstate-management
Chunker config docs (#913)
Sidharth Mohanty2b8b6d3ea9a8
diff --git a/configs/chunker.yaml b/configs/chunker.yamlnew file mode 100644index 000000000..63cf3f82c--- /dev/null+++ b/configs/chunker.yaml@@ -0,0 +1,4 @@+chunker:+ chunk_size: 100+ chunk_overlap: 20+ length_function: 'len'diff --git a/configs/full-stack.yaml b/configs/full-stack.yamlindex ec3c7a2ea..1da28209f 100644--- a/configs/full-stack.yaml+++ b/configs/full-stack.yaml@@ -2,6 +2,11 @@ app:config:id: 'full-stack-app'+chunker:+ chunk_size: 100+ chunk_overlap: 20+ length_function: 'len'+llm:provider: openaiconfig:diff --git a/docs/advanced/configuration.mdx b/docs/advanced/configuration.mdxindex 4ab7f38a0..44b4a9ecb 100644--- a/docs/advanced/configuration.mdx+++ b/docs/advanced/configuration.mdx@@ -11,6 +11,11 @@ app:config:id: 'full-stack-app'+chunker:+ chunk_size: 100+ chunk_overlap: 20+ length_function: 'len'+llm:provider: openaiconfig:@@ -49,7 +54,11 @@ Alright, let's dive into what each key means in the yaml config above:1. `app` Section:- `config`:- `id` (String): The ID or name of your full-stack application.-2. `llm` Section:+2. `chunker` Section:+ - `chunk_size` (Integer): The size of each chunk of text that is sent to the language model.+ - `chunk_overlap` (Integer): The amount of overlap between each chunk of text.+ - `length_function` (String): The function used to calculate the length of each chunk of text. In this case, it's set to 'len'. You can also use any function import directly as a string here.+3. `llm` Section:- `provider` (String): The provider for the language model, which is set to 'openai'. You can find the full list of llm providers in [our docs](/components/llms).- `model` (String): The specific model being used, 'gpt-3.5-turbo'.- `config`:@@ -59,13 +68,13 @@ Alright, let's dive into what each key means in the yaml config above:- `stream` (Boolean): Controls if the response is streamed back to the user (set to false).- `template` (String): A custom template for the prompt that the model uses to generate responses.- `system_prompt` (String): A system prompt for the model to follow when generating responses, in this case, it's set to the style of William Shakespeare.-3. `vectordb` Section:+4. `vectordb` Section:- `provider` (String): The provider for the vector database, set to 'chroma'. You can find the full list of vector database providers in [our docs](/components/vector-databases).- `config`:- `collection_name` (String): The initial collection name for the database, set to 'full-stack-app'.- `dir` (String): The directory for the database, set to 'db'.- `allow_reset` (Boolean): Indicates whether resetting the database is allowed, set to true.-4. `embedder` Section:+5. `embedder` Section:- `provider` (String): The provider for the embedder, set to 'openai'. You can find the full list of embedding model providers in [our docs](/components/embedding-models).- `config`:- `model` (String): The specific model used for text embedding, 'text-embedding-ada-002'.