💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update documentation, closes #156
davidmezzettid78278317cfa
diff --git a/docs/pipelines/transcription.md b/docs/pipelines/audio/transcription.mdsimilarity index 100%rename from docs/pipelines/transcription.mdrename to docs/pipelines/audio/transcription.mddiff --git a/docs/pipelines/segmentation.md b/docs/pipelines/data/segmentation.mdsimilarity index 100%rename from docs/pipelines/segmentation.mdrename to docs/pipelines/data/segmentation.mddiff --git a/docs/pipelines/tabular.md b/docs/pipelines/data/tabular.mdsimilarity index 100%rename from docs/pipelines/tabular.mdrename to docs/pipelines/data/tabular.mddiff --git a/docs/pipelines/textractor.md b/docs/pipelines/data/textractor.mdsimilarity index 100%rename from docs/pipelines/textractor.mdrename to docs/pipelines/data/textractor.mddiff --git a/docs/pipelines/image/caption.md b/docs/pipelines/image/caption.mdnew file mode 100644index 00000000..b9c905dc--- /dev/null+++ b/docs/pipelines/image/caption.md@@ -0,0 +1,10 @@+# Caption++A caption pipeline reads a list of images and returns a list of captions for those images.++```python+Caption()+```++::: txtai.pipeline.Caption.__init__+::: txtai.pipeline.Caption.__call__diff --git a/docs/pipelines/image/objects.md b/docs/pipelines/image/objects.mdnew file mode 100644index 00000000..6cfa3fc2--- /dev/null+++ b/docs/pipelines/image/objects.md@@ -0,0 +1,14 @@+# Objects++A objects pipeline reads a list of images and returns a list of detected objects.++```python+# Default configuration+Objects()++# Detect objects with an image classificaton model+Objects(classification=True, threshold=0.95)+```++::: txtai.pipeline.Objects.__init__+::: txtai.pipeline.Objects.__call__diff --git a/docs/pipelines/overview.md b/docs/pipelines/overview.mdindex ae00c639..b344ef60 100644--- a/docs/pipelines/overview.md+++ b/docs/pipelines/overview.md@@ -5,17 +5,22 @@ process various types of data. Pipelines can wrap machine learning models as welThe following pipeline types are currently supported.-- Hugging Face Transformers pipelines- - [Extractive QA](../extractor)- - [Labeling](../labels)- - [Similarity](../similarity)- - [Summary](../summary)-- Hugging Face Transformers models- - [ONNX](../onnx)- - [Trainer](../trainer)- - [Transcription](../transcription)- - [Translation](../translation)-- Data processing- - [Segmentation](../segmentation)- - [Tabular](../tabular)- - [Text extraction](../textractor)+- Audio+ - [Transcription](../audio/transcription)+- Data Processing+ - [Segmentation](../data/segmentation)+ - [Tabular](../data/tabular)+ - [Text extraction](../data/textractor)+- Image+ - [Caption](../image/caption)+ - [Objects](../image/objects)+- Text+ - [Extractive QA](../text/extractor)+ - [Labeling](../text/labels)+ - [Similarity](../text/similarity)+ - [Summary](../text/summary)+ - [Translation](../text/translation)+- Training+ - [HF ONNX](../train/hfonnx)+ - [ML ONNX](../train/mlonnx)+ - [Trainer](../train/trainer)diff --git a/docs/pipelines/extractor.md b/docs/pipelines/text/extractor.mdsimilarity index 100%rename from docs/pipelines/extractor.mdrename to docs/pipelines/text/extractor.mddiff --git a/docs/pipelines/labels.md b/docs/pipelines/text/labels.mdsimilarity index 100%rename from docs/pipelines/labels.mdrename to docs/pipelines/text/labels.mddiff --git a/docs/pipelines/similarity.md b/docs/pipelines/text/similarity.mdsimilarity index 100%rename from docs/pipelines/similarity.mdrename to docs/pipelines/text/similarity.mddiff --git a/docs/pipelines/summary.md b/docs/pipelines/text/summary.mdsimilarity index 100%rename from docs/pipelines/summary.mdrename to docs/pipelines/text/summary.mddiff --git a/docs/pipelines/translation.md b/docs/pipelines/text/translation.mdsimilarity index 100%rename from docs/pipelines/translation.mdrename to docs/pipelines/text/translation.mddiff --git a/docs/pipelines/onnx.md b/docs/pipelines/train/hfonnx.mdsimilarity index 100%rename from docs/pipelines/onnx.mdrename to docs/pipelines/train/hfonnx.mddiff --git a/docs/pipelines/train/mlonnx.md b/docs/pipelines/train/mlonnx.mdnew file mode 100644index 00000000..20e1494b--- /dev/null+++ b/docs/pipelines/train/mlonnx.md@@ -0,0 +1,6 @@+# MLOnnx++Exports a traditional machine learning model (i.e. scikit-learn) to ONNX.++::: txtai.pipeline.MLOnnx.__init__+::: txtai.pipeline.MLOnnx.__call__diff --git a/docs/pipelines/trainer.md b/docs/pipelines/train/trainer.mdsimilarity index 100%rename from docs/pipelines/trainer.mdrename to docs/pipelines/train/trainer.mddiff --git a/docs/workflows.md b/docs/workflows.mddeleted file mode 100644index c4c5bc61..00000000--- a/docs/workflows.md+++ /dev/null@@ -1,74 +0,0 @@-# Workflows--Workflows are a simple yet powerful construct that takes a callable and returns elements. Workflows don't know they are working with pipelines but enable efficient processing of pipeline data. Workflows are streaming by nature and work on data in batches, allowing large volumes of data to be processed efficiently.--An example Workflow is shown below. This workflow will work with both documents and audio files. Documents will have text extracted and summarized. Audio files will be transcribed. Both results will be joined, translated into French and loaded into an Embeddings index.--```python-# file:// prefixes are required to signal to the workflow this is a file and not a text string-files = [- "file://txtai/article.pdf",- "file://txtai/US_tops_5_million.wav",- "file://txtai/Canadas_last_fully.wav",- "file://txtai/Beijing_mobilises.wav",- "file://txtai/The_National_Park.wav",- "file://txtai/Maine_man_wins_1_mil.wav",- "file://txtai/Make_huge_profits.wav"-]--data = [(x, element, None) for x, element in enumerate(files)]--# Workflow that extracts text and builds a summary-articles = Workflow([- FileTask(textractor),- Task(lambda x: summary([y[:1024] for y in x]))-])--# Define workflow tasks. Workflows can also be tasks!-tasks = [- WorkflowTask(articles, r".\.pdf$"),- FileTask(transcribe, r"\.wav$"),- Task(lambda x: translate(x, "fr")),- Task(index, unpack=False)-]--# Workflow that translates text to French-workflow = Workflow(tasks)-for _ in workflow(data):- pass-```--::: txtai.workflow.Workflow.__init__-::: txtai.workflow.Workflow.__call__--## Tasks--::: txtai.workflow.Task.__init__--### File Task--Task that processes file urls--### Image Task--Task that processes image urls--### Service Task--Task that runs content against a http service--::: txtai.workflow.ServiceTask.__init__--### Storage Task--Task that expands a local directory or cloud storage bucket into a list of URLs to process--::: txtai.workflow.StorageTask.__init__--### URL Task--Task that processes urls--### Workflow Task--Task that runs a Workflow. Allows creating workflows of workflows.diff --git a/docs/workflows/file.md b/docs/workflows/file.mdnew file mode 100644index 00000000..0815ce1b--- /dev/null+++ b/docs/workflows/file.md@@ -0,0 +1,10 @@+## File Task++Task that processes file urls.++```python+workflow = Workflow([FileTask()])+workflow(["/path/to/file", "file:///path/to/file"])+```++::: txtai.workflow.FileTask.__init__diff --git a/docs/workflows/image.md b/docs/workflows/image.mdnew file mode 100644index 00000000..4da58a8a--- /dev/null+++ b/docs/workflows/image.md@@ -0,0 +1,10 @@+## Image Task++Task that processes image urls.++```python+workflow = Workflow([ImageTask()])+workflow(["image.jpg", "image.gif"])+```++::: txtai.workflow.ImageTask.__init__diff --git a/docs/workflows/overview.md b/docs/workflows/overview.mdnew file mode 100644index 00000000..0df99dd5--- /dev/null+++ b/docs/workflows/overview.md@@ -0,0 +1,70 @@+# Workflows++Workflows are a simple yet powerful construct that takes a callable and returns elements. Workflows operate well with pipelines but can work with any callable object. Workflows are streaming by nature and work on data in batches, allowing large volumes of data to be processed efficiently.++Given that pipelines are callable objects, workflows enable efficient processing of pipeline data. Transformers models typically work with smaller batches of data, workflows are well suited to feed a series of transformers pipelines.++An example of the most basic workflow:++```python+workflow = Workflow([Task(lambda x: [y * 2 for y in x])])+list(workflow([1, 2, 3]))+```++This example simply multiplies each input value and returns a outputs via a generator.++A more complex example is shown below. This workflow will work with both documents and audio files. Documents will have text extracted and summarized. Audio files will be transcribed. Both results will be joined, translated into French and loaded into an Embeddings index.++```python+# file:// prefixes are required to signal to the workflow this is a file and not a text string+files = [+ "file://txtai/article.pdf",+ "file://txtai/US_tops_5_million.wav",+ "file://txtai/Canadas_last_fully.wav",+ "file://txtai/Beijing_mobilises.wav",+ "file://txtai/The_National_Park.wav",+ "file://txtai/Maine_man_wins_1_mil.wav",+ "file://txtai/Make_huge_profits.wav"+]++data = [(x, element, None) for x, element in enumerate(files)]++# Workflow that extracts text and builds a summary+articles = Workflow([+ FileTask(textractor),+ Task(lambda x: summary([y[:1024] for y in x]))+])++# Define workflow tasks. Workflows can also be tasks!+tasks = [+ WorkflowTask(articles, r".\.pdf$"),+ FileTask(transcribe, r"\.wav$"),+ Task(lambda x: translate(x, "fr")),+ Task(index, unpack=False)+]++# Workflow that translates text to French+workflow = Workflow(tasks)+for _ in workflow(data):+ pass+```++Workflows can be defined using Python as described here but they can also be created as YAML configuration and run as shown below.++```python+with open("workflow.yml") as config:+ workflow = yaml.safe_load(config)++# Create and run the workflow+app = API(workflow)+data = list(app.workflow("workflow", ["input text"]))+```++[Read more here on creating workflow YAML](../../api).++# Workflow++Workflows are callable objects. Workflows take an input of iterable data elements and output iterable data elements.++### ::: txtai.workflow.Workflow.__init__+### ::: txtai.workflow.Workflow.__call__diff --git a/docs/workflows/retrieve.md b/docs/workflows/retrieve.mdnew file mode 100644index 00000000..36eaaef2--- /dev/null+++ b/docs/workflows/retrieve.md@@ -0,0 +1,10 @@+## Retrieve Task++Task that connects to a url and downloads the content locally.++```python+workflow = Workflow([RetrieveTask(directory="/tmp")])+workflow(["https://file.to.download", "/local/file/to/copy"])+```++::: txtai.workflow.RetrieveTask.__init__diff --git a/docs/workflows/service.md b/docs/workflows/service.mdnew file mode 100644index 00000000..be2a135b--- /dev/null+++ b/docs/workflows/service.md@@ -0,0 +1,5 @@+## Service Task++Task that runs content against a http service.++::: txtai.workflow.ServiceTask.__init__diff --git a/docs/workflows/storage.md b/docs/workflows/storage.mdnew file mode 100644index 00000000..2602ada0--- /dev/null+++ b/docs/workflows/storage.md@@ -0,0 +1,5 @@+## Storage Task++Task that expands a local directory or cloud storage bucket into a list of URLs to process.++::: txtai.workflow.StorageTask.__init__diff --git a/docs/workflows/task.md b/docs/workflows/task.mdnew file mode 100644index 00000000..5f2b5599--- /dev/null+++ b/docs/workflows/task.md@@ -0,0 +1,61 @@+# Tasks++Workflows execute tasks. Tasks are callable objects with a number of parameters to control the processing of data at a given step.++A simple task is shown below.++```python+Task(lambda x: [y * 2 for y in x])+```++The task above executes the function above for all input elements.++Tasks work well with pipelines, since pipelines are callable objects. The example below will summarize each input element.++```python+summary = Summary()+Task(summary)+```++Tasks can operate independently but work best with workflows, as workflows add large-scale stream processing.++```python+summary = Summary()+task = Task(summary)+task(["Very long text here"])++workflow = Workflow([task])+list(workflow(["Very long text here"]))+```++::: txtai.workflow.Task.__init__++## Multi-action task merges++Multi-action tasks will generate parallel outputs for the input data. The task output can be merged together in a couple different ways.++### ::: txtai.workflow.Task.hstack+### ::: txtai.workflow.Task.vstack+### ::: txtai.workflow.Task.concat++## Extract task output columns++With column-wise merging, each output row will be a tuple of output values for each task action. This can be fed as input to a downstream task and that task can have separate tasks work with each element.++A simple example:++```python+workflow = Workflow([Task(lambda x: [y * 3 for y in x], unpack=False, column=0)])+list(workflow([(2, 8)]))+```++For the example input tuple of (2, 2), the workflow will only select the first element (2) and run the task against that element.++```python+workflow = Workflow([Task([lambda x: [y * 3 for y in x],+ lambda x: [y - 1 for y in x]],+ unpack=False, column={0:0, 1:1})])+list(workflow([(2, 8)]))+```++The example above applies a separate action to each input column. This simple construct can help build extremely powerful workflow graphs!diff --git a/docs/workflows/url.md b/docs/workflows/url.mdnew file mode 100644index 00000000..2928625a--- /dev/null+++ b/docs/workflows/url.md@@ -0,0 +1,10 @@+## Url Task++Task that processes urls.++```python+workflow = Workflow([UrlTask()])+workflow(["https://file.to.download", "file:////local/file/to/copy"])+```++::: txtai.workflow.UrlTask.__init__diff --git a/docs/workflows/workflow.md b/docs/workflows/workflow.mdnew file mode 100644index 00000000..8bb0d56d--- /dev/null+++ b/docs/workflows/workflow.md@@ -0,0 +1,3 @@+## Workflow Task++Task that runs a Workflow. Allows creating workflows of workflows.diff --git a/mkdocs.yml b/mkdocs.ymlindex 83932527..13126cfd 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -35,17 +35,35 @@ nav:- Embeddings: embeddings.md- Pipelines:- Overview: pipelines/overview.md- - Extractor: pipelines/extractor.md- - Labels: pipelines/labels.md- - ONNX: pipelines/onnx.md- - Segmentation: pipelines/segmentation.md- - Similarity: pipelines/similarity.md- - Summary: pipelines/summary.md- - Tabular: pipelines/tabular.md- - Textractor: pipelines/textractor.md- - Trainer: pipelines/trainer.md- - Transcription: pipelines/transcription.md- - Translation: pipelines/translation.md- - Workflows: workflows.md+ - Audio:+ - Transcription: pipelines/audio/transcription.md+ - Data:+ - Segmentation: pipelines/data/segmentation.md+ - Tabular: pipelines/data/tabular.md+ - Textractor: pipelines/data/textractor.md+ - Image:+ - Caption: pipelines/image/caption.md+ - Objects: pipelines/image/objects.md+ - Text:+ - Extractor: pipelines/text/extractor.md+ - Labels: pipelines/text/labels.md+ - Similarity: pipelines/text/similarity.md+ - Summary: pipelines/text/summary.md+ - Translation: pipelines/text/translation.md+ - Train:+ - HF ONNX: pipelines/train/hfonnx.md+ - ML ONNX: pipelines/train/mlonnx.md+ - Trainer: pipelines/train/trainer.md+ - Workflows:+ - Overview: workflows/overview.md+ - Task: workflows/task.md+ - File: workflows/file.md+ - Image: workflows/image.md+ - Retrieve: workflows/retrieve.md+ - Service: workflows/service.md+ - Storage: workflows/storage.md+ - Url: workflows/url.md+ - Workflow: workflows/workflow.md+- API: api.md- Further Reading: articles.md