Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

1.9 KB
Raw Blame

Segmentation

pipeline pipeline

The Segmentation pipeline segments text into semantic units.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import Segmentation

# Create and run pipeline
segment = Segmentation(sentences=True)
segment("This is a test. And another test.")

# Load third-party chunkers
segment = Segmentation(chunker="semantic")
segment("This is a test. And another test.")

The Segmentation pipeline supports segmenting sentences, lines, paragraphs and sections using a rules-based approach. Each of these modes can be set when creating the pipeline. Third-party chunkers are also supported via the chunker parameter.

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

# Create pipeline using lower case class name
segmentation:
  sentences: true

# Run pipeline with workflow
workflow:
  segment:
    tasks:
      - action: segmentation

Run with Workflows

from txtai import Application

# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("segment", ["This is a test. And another test."]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &

curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"segment", "elements":["This is a test. And another test."]}'

Methods

Python documentation for the pipeline.

::: txtai.pipeline.Segmentation.init

::: txtai.pipeline.Segmentation.call