💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update docs
davidmezzettic50d8fd96d4c
diff --git a/docs/pipeline/data/segmentation.md b/docs/pipeline/data/segmentation.mdindex 9e16b5c4..7e1bae6c 100644--- a/docs/pipeline/data/segmentation.md+++ b/docs/pipeline/data/segmentation.md@@ -16,14 +16,12 @@ from txtai.pipeline import Segmentationsegment = Segmentation(sentences=True)segment("This is a test. And another test.")-# Segment text with Chonkie chunker (word, sentence, semantic, late etc)+# Load third-party chunkerssegment = Segmentation(chunker="semantic")segment("This is a test. And another test.")```-The Segmentation pipeline supports segmenting `sentences`, `lines`, `paragraphs` and `sections` using a rules-based approach. Each of these modes can be set when creating the pipeline.--More advanced functionality is supported via a [Chonkie chunker](https://docs.chonkie.ai/chunkers/overview). The `chunker` keyword dynamically creates a Chonkie chunker. For example, `chunker='token'` creates a [TokenChunker](https://docs.chonkie.ai/chunkers/token-chunker), `chunker='semantic'` creates a [SemanticChunker](https://docs.chonkie.ai/chunkers/semantic-chunker) and so forth. Additional keyword arguments are passed to the chunker.+The Segmentation pipeline supports segmenting `sentences`, `lines`, `paragraphs` and `sections` using a rules-based approach. Each of these modes can be set when creating the pipeline. Third-party chunkers are also supported via the `chunker` parameter.## Configuration-driven example