~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(embedding-settings): clarify which splitter respects which settings (#5660)
Sa1f9328c0ef7
diff --git a/changelog.d/5660.bugfix.md b/changelog.d/5660.bugfix.mdnew file mode 100644index 0000000000..87da7e2855--- /dev/null+++ b/changelog.d/5660.bugfix.md@@ -0,0 +1 @@+Clarified the help text and tooltips on the embedding settings page so they match which chunk-size, chunk-overlap, and text-separator settings each text splitter type actually uses. Previously, the descriptions implied every splitter honored every setting — for example, the chunk-size field was described as "characters per chunk" even though the `token` and `sentence` splitters interpret it as tokens, and the `semantic` splitter silently forces chunk overlap to 0.diff --git a/src/local_deep_research/web/templates/pages/embedding_settings.html b/src/local_deep_research/web/templates/pages/embedding_settings.htmlindex ced9c354ec..59a51d7bc6 100644--- a/src/local_deep_research/web/templates/pages/embedding_settings.html+++ b/src/local_deep_research/web/templates/pages/embedding_settings.html@@ -16,7 +16,7 @@{% call help_panel('embedding-how', 'Understanding Embeddings', icon='brain', collapsed=true, dismissible=true) %}<div class="ldr-help-steps">{{ help_step(1, "What are Embeddings?", "Embeddings convert text into numbers (vectors) that capture meaning. Similar concepts have similar vectors.") }}- {{ help_step(2, "Document Chunk Size", "Splits documents into character chunks. Larger chunks keep more context per search result but less precision; smaller chunks are more granular but may lose context.") }}+ {{ help_step(2, "Document Chunk Size", "Maximum size of each chunk — characters for the recursive and semantic splitters, tokens for the token and sentence splitters. Larger chunks keep more context per search result but less precision; smaller chunks are more granular but may lose context.") }}{{ help_step(3, "Text Splitter", "Recursive: uses natural breaks. Semantic: AI-powered grouping (slower). Token-based: respects LLM limits.") }}{{ help_step(4, "Why Lock Settings?", "Mixing different embedding models causes inconsistent search results. Each collection uses one model.") }}</div>@@ -127,20 +127,20 @@<div class="ldr-form-row"><div class="ldr-form-group ldr-col-md-6">- <label for="chunk-size">{{ tooltip("Document Chunk Size", "Splits documents into character chunks before embedding. Larger chunks (1000+) keep more context but search is less precise. Smaller chunks (500-) are more granular but may cut off important context.") }}</label>+ <label for="chunk-size">{{ tooltip("Document Chunk Size", "Maximum size of each chunk. Larger values keep more context but reduce search precision; smaller values are more granular but may cut off important context.") }}</label><input type="number" id="chunk-size" class="ldr-form-control" value="1000" min="100" max="5000" step="100">- <small class="ldr-form-text ldr-text-muted">Characters per document chunk</small>+ <small class="ldr-form-text ldr-text-muted">Characters for recursive/semantic; tokens for token and sentence (sentence is capped by the tokenizer's max length).</small></div><div class="ldr-form-group ldr-col-md-6">- <label for="chunk-overlap">{{ tooltip("Chunk Overlap", "Characters shared between adjacent document chunks. Prevents important info from being cut at boundaries. Usually 10-20% of the document chunk size.") }}</label>+ <label for="chunk-overlap">{{ tooltip("Chunk Overlap", "Content shared between adjacent chunks to prevent important info from being cut at boundaries. Usually 10-20% of the chunk size.") }}</label><input type="number" id="chunk-overlap" class="ldr-form-control" value="200" min="0" max="1000" step="50">- <small class="ldr-form-text ldr-text-muted">Overlap between chunks</small>+ <small class="ldr-form-text ldr-text-muted">Ignored by the semantic splitter (always 0).</small></div></div><div class="ldr-form-row"><div class="ldr-form-group ldr-col-md-6">- <label for="splitter-type">{{ tooltip("Text Splitter Type", "How to divide documents. Recursive: natural breaks (paragraphs, sentences). Semantic: AI groups related content (slow but smart). Token: respects model limits.") }}</label>+ <label for="splitter-type">{{ tooltip("Text Splitter Type", "Method used to split documents into chunks.") }}</label><select id="splitter-type" class="ldr-form-control"><option value="recursive">Recursive (Default - uses separators)</option><option value="token">Token-based (Good for LLM context)</option>@@ -148,7 +148,7 @@<option value="semantic">Semantic (Smart grouping - slower)</option></select><small class="ldr-form-text ldr-text-muted">- How documents are split into chunks. Semantic uses AI to find natural breakpoints (slower but smarter)+ Recursive and semantic respect text separators; token and sentence ignore them.</small></div><div class="ldr-form-group ldr-col-md-6">@@ -190,8 +190,7 @@<textarea id="text-separators" class="ldr-form-control" rows="3"placeholder='["\n\n", "\n", ". ", " ", ""]'></textarea><small class="ldr-form-text ldr-text-muted">- JSON array of separators. Default works for 80% of use cases.- For code: add language-specific separators like "\nclass", "\ndef"+ JSON array of separators. Default works for 80% of use cases. Used by recursive and semantic splitters only — token and sentence ignore this. For code, add language-specific separators like "\nclass", "\ndef".</small></div>diff --git a/tests/embeddings/test_text_splitter_registry.py b/tests/embeddings/test_text_splitter_registry.pyindex 13a6179f19..d97923cf14 100644--- a/tests/embeddings/test_text_splitter_registry.py+++ b/tests/embeddings/test_text_splitter_registry.py@@ -157,6 +157,25 @@ class TestGetTextSplitterToken:)assert isinstance(splitter, TokenTextSplitter)+ assert splitter._chunk_size == 256+ assert splitter._chunk_overlap == 32++ def test_get_text_splitter_token_large_chunk_size_has_no_cap(self):+ """Test token splitter constructs at maximum chunk size with no tokenizer cap."""+ from local_deep_research.embeddings.splitters.text_splitter_registry import (+ get_text_splitter,+ )+ from langchain_text_splitters import TokenTextSplitter++ splitter = get_text_splitter(+ splitter_type="token",+ chunk_size=5000,+ chunk_overlap=200,+ )++ assert isinstance(splitter, TokenTextSplitter)+ assert splitter._chunk_size == 5000+ assert splitter._chunk_overlap == 200@pytest.mark.skipif(