Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(embedding-settings): clarify which splitter respects which settings (#5660)

Sa1f9328c0ef7
diff --git a/changelog.d/5660.bugfix.md b/changelog.d/5660.bugfix.md
new file mode 100644
index 0000000000..87da7e2855
--- /dev/null
+++ b/changelog.d/5660.bugfix.md
@@ -0,0 +1 @@
+Clarified the help text and tooltips on the embedding settings page so they match which chunk-size, chunk-overlap, and text-separator settings each text splitter type actually uses. Previously, the descriptions implied every splitter honored every setting — for example, the chunk-size field was described as "characters per chunk" even though the `token` and `sentence` splitters interpret it as tokens, and the `semantic` splitter silently forces chunk overlap to 0.
diff --git a/src/local_deep_research/web/templates/pages/embedding_settings.html b/src/local_deep_research/web/templates/pages/embedding_settings.html
index ced9c354ec..59a51d7bc6 100644
--- a/src/local_deep_research/web/templates/pages/embedding_settings.html
+++ b/src/local_deep_research/web/templates/pages/embedding_settings.html
@@ -16,7 +16,7 @@
{% call help_panel('embedding-how', 'Understanding Embeddings', icon='brain', collapsed=true, dismissible=true) %}
<div class="ldr-help-steps">
{{ help_step(1, "What are Embeddings?", "Embeddings convert text into numbers (vectors) that capture meaning. Similar concepts have similar vectors.") }}
- {{ help_step(2, "Document Chunk Size", "Splits documents into character chunks. Larger chunks keep more context per search result but less precision; smaller chunks are more granular but may lose context.") }}
+ {{ help_step(2, "Document Chunk Size", "Maximum size of each chunk — characters for the recursive and semantic splitters, tokens for the token and sentence splitters. Larger chunks keep more context per search result but less precision; smaller chunks are more granular but may lose context.") }}
{{ help_step(3, "Text Splitter", "Recursive: uses natural breaks. Semantic: AI-powered grouping (slower). Token-based: respects LLM limits.") }}
{{ help_step(4, "Why Lock Settings?", "Mixing different embedding models causes inconsistent search results. Each collection uses one model.") }}
</div>
@@ -127,20 +127,20 @@
<div class="ldr-form-row">
<div class="ldr-form-group ldr-col-md-6">
- <label for="chunk-size">{{ tooltip("Document Chunk Size", "Splits documents into character chunks before embedding. Larger chunks (1000+) keep more context but search is less precise. Smaller chunks (500-) are more granular but may cut off important context.") }}</label>
+ <label for="chunk-size">{{ tooltip("Document Chunk Size", "Maximum size of each chunk. Larger values keep more context but reduce search precision; smaller values are more granular but may cut off important context.") }}</label>
<input type="number" id="chunk-size" class="ldr-form-control" value="1000" min="100" max="5000" step="100">
- <small class="ldr-form-text ldr-text-muted">Characters per document chunk</small>
+ <small class="ldr-form-text ldr-text-muted">Characters for recursive/semantic; tokens for token and sentence (sentence is capped by the tokenizer's max length).</small>
</div>
<div class="ldr-form-group ldr-col-md-6">
- <label for="chunk-overlap">{{ tooltip("Chunk Overlap", "Characters shared between adjacent document chunks. Prevents important info from being cut at boundaries. Usually 10-20% of the document chunk size.") }}</label>
+ <label for="chunk-overlap">{{ tooltip("Chunk Overlap", "Content shared between adjacent chunks to prevent important info from being cut at boundaries. Usually 10-20% of the chunk size.") }}</label>
<input type="number" id="chunk-overlap" class="ldr-form-control" value="200" min="0" max="1000" step="50">
- <small class="ldr-form-text ldr-text-muted">Overlap between chunks</small>
+ <small class="ldr-form-text ldr-text-muted">Ignored by the semantic splitter (always 0).</small>
</div>
</div>
<div class="ldr-form-row">
<div class="ldr-form-group ldr-col-md-6">
- <label for="splitter-type">{{ tooltip("Text Splitter Type", "How to divide documents. Recursive: natural breaks (paragraphs, sentences). Semantic: AI groups related content (slow but smart). Token: respects model limits.") }}</label>
+ <label for="splitter-type">{{ tooltip("Text Splitter Type", "Method used to split documents into chunks.") }}</label>
<select id="splitter-type" class="ldr-form-control">
<option value="recursive">Recursive (Default - uses separators)</option>
<option value="token">Token-based (Good for LLM context)</option>
@@ -148,7 +148,7 @@
<option value="semantic">Semantic (Smart grouping - slower)</option>
</select>
<small class="ldr-form-text ldr-text-muted">
- How documents are split into chunks. Semantic uses AI to find natural breakpoints (slower but smarter)
+ Recursive and semantic respect text separators; token and sentence ignore them.
</small>
</div>
<div class="ldr-form-group ldr-col-md-6">
@@ -190,8 +190,7 @@
<textarea id="text-separators" class="ldr-form-control" rows="3"
placeholder='["\n\n", "\n", ". ", " ", ""]'></textarea>
<small class="ldr-form-text ldr-text-muted">
- JSON array of separators. Default works for 80% of use cases.
- For code: add language-specific separators like "\nclass", "\ndef"
+ JSON array of separators. Default works for 80% of use cases. Used by recursive and semantic splitters only — token and sentence ignore this. For code, add language-specific separators like "\nclass", "\ndef".
</small>
</div>
diff --git a/tests/embeddings/test_text_splitter_registry.py b/tests/embeddings/test_text_splitter_registry.py
index 13a6179f19..d97923cf14 100644
--- a/tests/embeddings/test_text_splitter_registry.py
+++ b/tests/embeddings/test_text_splitter_registry.py
@@ -157,6 +157,25 @@ class TestGetTextSplitterToken:
)
assert isinstance(splitter, TokenTextSplitter)
+ assert splitter._chunk_size == 256
+ assert splitter._chunk_overlap == 32
+
+ def test_get_text_splitter_token_large_chunk_size_has_no_cap(self):
+ """Test token splitter constructs at maximum chunk size with no tokenizer cap."""
+ from local_deep_research.embeddings.splitters.text_splitter_registry import (
+ get_text_splitter,
+ )
+ from langchain_text_splitters import TokenTextSplitter
+
+ splitter = get_text_splitter(
+ splitter_type="token",
+ chunk_size=5000,
+ chunk_overlap=200,
+ )
+
+ assert isinstance(splitter, TokenTextSplitter)
+ assert splitter._chunk_size == 5000
+ assert splitter._chunk_overlap == 200
@pytest.mark.skipif(