💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update documentation
davidmezzettibd73a3f8d950
diff --git a/docs/api/cluster.md b/docs/api/cluster.mdnew file mode 100644index 00000000..696fe2df--- /dev/null+++ b/docs/api/cluster.md@@ -0,0 +1,14 @@+# Distributed embeddings clusters++The API supports combining multiple API instances into a single logical embeddings index. An example configuration is shown below.++```yaml+cluster:+ shards:+ - http://127.0.0.1:8002+ - http://127.0.0.1:8003+```++This configuration aggregates the API instances above as index shards. Data is evenly split among each of the shards at index time. Queries are run in parallel against each shard and the results are joined together. This method allows horizontal scaling and supports very large index clusters.++This method is only recommended for data sets in the 1 billion+ records. The ANN libraries can easily support smaller data sizes and this method is not worth the additional complexity. At this time, new shards can not be added after building the initial index.diff --git a/docs/api/configuration.md b/docs/api/configuration.mdnew file mode 100644index 00000000..80adabea--- /dev/null+++ b/docs/api/configuration.md@@ -0,0 +1,109 @@+# Configuration++Configuration is set through YAML. In most cases, YAML keys map to fields names in Python. The [example in the previous section](../) gave a full-featured example covering a wide array of configuration options.++Each section below describes the available configuration settings.++## Embeddings++The configuration parser expects a top level `embeddings` key to be present in the YAML. All [embeddings configuration](../../embeddings/configuration) is supported.++The following example defines an embeddings index.++```yaml+path: index path+writable: true++embeddings:+ path: vector model+ content: true+```++Two top level settings are available to control where indexes are saved and if an index is a read-only index.++### path+```yaml+path: string+```++Path to save and load the embeddings index. Each API instance can only access a single index at a time.++### writable+```yaml+path: boolean+```++Determines if the input embeddings index is writable (true) or read-only (false). This allows serving a read-only index.++## Pipeline++Pipelines are loaded as top level configuration parameters. Pipeline names are automatically detected in the YAML configuration and created upon startup. All [pipelines](../../pipeline) are supported.++The following example defines a series of pipelines.++```yaml+caption:++extractor:+ path: model path++labels:++summary:++tabular:++translation:+```++Under each pipeline name, configuration settings for the pipeline can be set.++## Workflow++Workflows are defined under a top level `workflow` key. Each key under the `workflow` key is the name of the workflow. Under that is a `tasks` key with each task definition.++The following example defines a workflow.++```yaml+workflow:+ sumtranslate:+ tasks:+ - action: summary+ - action: translation+```++### tasks+```yaml+tasks: list+```++Expects a list of workflow tasks. Each element defines a single workflow task. All [task configuration](../../workflow/task) is supported.++Each task element supports the following additional arguments.++### action+```yaml+task: string|list+```++Both single and multi-action tasks are supported.++The action parameter works slightly different when passed via configuration. The parameter(s) needs to be converted into callable method(s). If action is pipeline that has been defined in the current configuration, it will use that pipeline as the action.++There are two special action names `index` and `upsert`. If those are used as the action, the task will collect workflow data elements and load them into defined the embeddings index.++Otherwise, the action must be a path to a static callable function. The configuration parser will resolve the function name and use that as the task action.++### task+```yaml+task: string+```++Optionally sets the type of task to create. For example, this could be a `file` task or a `retrieve` task. If this is not specified, a generic task is created. [The list of workflow tasks can be found here](../../workflow).++### args+```yaml+args: list+```++Optional list of static arguments to pass to the workflow task. These are combined with workflow data to pass to each `__call__`.diff --git a/docs/api/docker.md b/docs/api/docker.mdnew file mode 100644index 00000000..ab141451--- /dev/null+++ b/docs/api/docker.md@@ -0,0 +1,16 @@+# Docker++A Dockerfile with commands to install txtai, all dependencies and default configuration is available in this repository.++The Dockerfile can be copied from the docker directory on GitHub locally. The following commands show how to run the API process.++```bash+docker build -t txtai.api -f docker/api.Dockerfile .+docker run --name txtai.api -p 8000:8000 --rm -it txtai.api++# Alternatively, if nvidia-docker is installed, the build will support+# GPU runtimes+docker run --name txtai.api --runtime=nvidia -p 8000:8000 --rm -it txtai.api+```++This will bring up an API instance without having to install Python, txtai or any dependencies on your machine!diff --git a/docs/api.md b/docs/api/index.mdsimilarity index 61%rename from docs/api.mdrename to docs/api/index.mdindex a5739dfd..f3457540 100644--- a/docs/api.md+++ b/docs/api/index.md@@ -1,7 +1,7 @@# API--++txtai has a full-featured API that can optionally be enabled for any txtai process. All functionality found in txtai can be accessed via the API. The following is an example configuration and startup script for the API.@@ -74,37 +74,6 @@ CONFIG=index.yml uvicorn "txtai.api:app"uvicorn is a full-featured production ready server with support for SSL and more. See the [uvicorn deployment guide](https://www.uvicorn.org/deployment/) for details.-## Docker--A Dockerfile with commands to install txtai, all dependencies and default configuration is available in this repository.--The Dockerfile can be copied from the docker directory on GitHub locally. The following commands show how to run the API process.--```bash-docker build -t txtai.api -f docker/api.Dockerfile .-docker run --name txtai.api -p 8000:8000 --rm -it txtai.api--# Alternatively, if nvidia-docker is installed, the build will support GPU runtimes-docker run --name txtai.api --runtime=nvidia -p 8000:8000 --rm -it txtai.api-```--This will bring up an API instance without having to install Python, txtai or any dependencies on your machine!--## Distributed embeddings clusters--The API supports combining multiple API instances into a single logical embeddings index. An example configuration is shown below.--```yaml-cluster:- shards:- - http://127.0.0.1:8002- - http://127.0.0.1:8003-```--This configuration aggregates the API instances above as index shards. Data is evenly split among each of the shards at index time. Queries are run in parallel against each shard and the results are joined together. This method allows horizontal scaling and supports very large index clusters.--This method is only recommended for data sets in the 1 billion+ records. The ANN libraries can easily support smaller data sizes and this method is not worth the additional complexity. At this time, new shards can not be added after building the initial index.-## Differences between Python and APIThe txtai API provides all the major functionality found in this project. But there are differences due to the nature of JSON and differences across the supported programming languages. For example, any Python callable method is available at a named endpoint (i.e. instead of summary() the method call would be summary.summary()).@@ -120,5 +89,3 @@ The following programming languages have txtai bindings:- [Go](https://github.com/neuml/txtai.go)See each of the projects above for details on how to install and use. Please add an issue to request additional language bindings!--::: txtai.api.application\ No newline at end of filediff --git a/docs/embeddings/index.md b/docs/embeddings/index.mdindex ed920a53..33a43147 100644--- a/docs/embeddings/index.md+++ b/docs/embeddings/index.md@@ -83,9 +83,3 @@ The search method takes two parameters, the query and query limit. The results f- List of `(id, score)` when content is disabled- List of `{**query columns}` when content is enabled--## Links--- [Configuration](../configuration)-- [Methods](../methods)-- [Query Guide](../query)diff --git a/docs/embeddings/query.md b/docs/embeddings/query.mdindex 0e898cce..6cf1ad03 100644--- a/docs/embeddings/query.md+++ b/docs/embeddings/query.md@@ -15,7 +15,7 @@ The queries above search the index for similarity matches on `feel good story` a## SQL-txtai supports more complex queries with [SQL](https://en.wikipedia.org/wiki/SQL). This is only supported if [content storage](../configuration/#content) is enabled. txtai has a translation layer that analyzes input SQL statements and combines similarity results with content stored in a relational database.+txtai supports more complex queries with SQL. This is only supported if [content storage](../configuration/#content) is enabled. txtai has a translation layer that analyzes input SQL statements and combines similarity results with content stored in a relational database.SQL queries are run through `embeddings.search` like natural language queries but the examples below only show the SQL query for conciseness.@@ -112,7 +112,7 @@ result = embeddings.search(query)[0]["object"]## Combined index architecture-When content storage is enabled, txtai becomes a dual storage engine. Content is stored in an underlying database (currently supports [SQLite](https://en.wikipedia.org/wiki/SQLite)) along with an Approximate Nearest Neighbor (ANN) index. These components combine to deliver similarity search alongside traditional structured search.+When content storage is enabled, txtai becomes a dual storage engine. Content is stored in an underlying database (currently supports SQLite) along with an Approximate Nearest Neighbor (ANN) index. These components combine to deliver similarity search alongside traditional structured search.The ANN index stores ids and vectors for each input element. When a natural language query is run, the query is translated into a vector and a similarity query finds the best matching ids. When a database is added into the mix, an additional step is applied. This step takes those ids and effectively inserts them as part of the underlying database query.diff --git a/docs/images/api-dark.png b/docs/images/api-dark.pngindex 7ffebe06..50b6b99b 100644Binary files a/docs/images/api-dark.png and b/docs/images/api-dark.png differdiff --git a/docs/pipeline/text/similarity.md b/docs/pipeline/text/similarity.mdindex 028d30c4..5657ba49 100644--- a/docs/pipeline/text/similarity.md+++ b/docs/pipeline/text/similarity.md@@ -1,4 +1,4 @@-## Similarity+# SimilarityA Similarity pipeline is also a zero shot classifier model where the labels are the queries. The results are transposed to get scores per query/label vs scores per input text.diff --git a/docs/workflow/file.md b/docs/workflow/file.mdindex 0815ce1b..77df270a 100644--- a/docs/workflow/file.md+++ b/docs/workflow/file.md@@ -1,4 +1,4 @@-## File Task+# File TaskTask that processes file urls.diff --git a/docs/workflow/image.md b/docs/workflow/image.mdindex 4da58a8a..07a8cd03 100644--- a/docs/workflow/image.md+++ b/docs/workflow/image.md@@ -1,4 +1,4 @@-## Image Task+# Image TaskTask that processes image urls.diff --git a/docs/workflow/index.md b/docs/workflow/index.mdindex 1f7063b7..1187d406 100644--- a/docs/workflow/index.md+++ b/docs/workflow/index.md@@ -3,7 +3,7 @@-Workflows are a simple yet powerful construct that takes a callable and returns elements. Workflows operate well with pipelines but can work with any callable object. Workflows are streaming by nature and work on data in batches, allowing large volumes of data to be processed efficiently.+Workflows are a simple yet powerful construct that takes a callable and returns elements. Workflows operate well with pipelines but can work with any callable object. Workflows are streaming and work on data in batches, allowing large volumes of data to be processed efficiently.Given that pipelines are callable objects, workflows enable efficient processing of pipeline data. Transformers models typically work with smaller batches of data, workflows are well suited to feed a series of transformers pipelines.@@ -16,53 +16,107 @@ list(workflow([1, 2, 3]))This example simply multiplies each input value and returns a outputs via a generator.-A more complex example is shown below. This workflow will work with both documents and audio files. Documents will have text extracted and summarized. Audio files will be transcribed. Both results will be joined, translated into French and loaded into an Embeddings index.+## Example++A full-featured example is shown below. This workflow transcribes a set of audio files, translates the text into French and indexes the data.```python-# file:// prefixes are required to signal to the workflow this is a file and not a text string-files = [- "file://txtai/article.pdf",- "file://txtai/US_tops_5_million.wav",- "file://txtai/Canadas_last_fully.wav",- "file://txtai/Beijing_mobilises.wav",- "file://txtai/The_National_Park.wav",- "file://txtai/Maine_man_wins_1_mil.wav",- "file://txtai/Make_huge_profits.wav"-]+from txtai.embeddings import Embeddings+from txtai.pipeline import Transcription, Translation+from txtai.workflow import FileTask, Task, Workflow++# Embeddings instance+embeddings = Embeddings({+ "path": "sentence-transformers/paraphrase-MiniLM-L3-v2",+ "content": True+})-data = [(x, element, None) for x, element in enumerate(files)]+# Transcription instance+transcribe = Transcription()-# Workflow that extracts text and builds a summary-articles = Workflow([- FileTask(textractor),- Task(lambda x: summary([y[:1024] for y in x]))-])+# Translation instance+translate = Translation()-# Define workflow tasks. Workflows can also be tasks!tasks = [- WorkflowTask(articles, r".\.pdf$"),FileTask(transcribe, r"\.wav$"),- Task(lambda x: translate(x, "fr")),- Task(index, unpack=False)+ Task(lambda x: translate(x, "fr"))+]++# List of files to process+data = [+ "US_tops_5_million.wav",+ "Canadas_last_fully.wav",+ "Beijing_mobilises.wav",+ "The_National_Park.wav",+ "Maine_man_wins_1_mil.wav",+ "Make_huge_profits.wav"]-# Workflow that translates text to French+# Workflow that translate text to Frenchworkflow = Workflow(tasks)-for _ in workflow(data):- pass++# Index data+embeddings.index((uid, text, None) for uid, text in enumerate(workflow(data)))++# Search+embeddings.search("wildlife", 1)```-Workflows can be defined using Python as described here but they can also be created as YAML configuration and run as shown below.+## Configuration-driven example++Workflows can be defined using Python as shown above but they can also run with YAML configuration.++```yaml+writable: true+embeddings:+ path: sentence-transformers/paraphrase-MiniLM-L3-v2+ content: true++# Transcribe audio to text+transcription:++# Translate text between languages+translation:++workflow:+ index:+ tasks:+ - action: transcription+ select: "\\.wav$"+ task: file+ - action: translation+ args: ["fr"]+ - action: index+``````python+# Create and run the workflow+from txtai.api import API+# Create and run the workflowapp = API("workflow.yml")-data = list(app.workflow("workflow", ["input text"]))+list(app.workflow("index", [+ "US_tops_5_million.wav",+ "Canadas_last_fully.wav",+ "Beijing_mobilises.wav",+ "The_National_Park.wav",+ "Maine_man_wins_1_mil.wav",+ "Make_huge_profits.wav"+]))++# Search+app.search("wildlife")```-[Read more here on creating workflow YAML](../../api).+The code above executes a workflow defined in the file `workflow.yml`. The API is used to run the workflow locally, there is minimal overhead running workflows in this manner. It's a matter of preference.-# Workflow+See the following links for more information.++- [Workflow builder application](../../examples/#applications)+- [Workflow YAML](../../api/configuration/#workflow)+- [Workflow examples](https://huggingface.co/spaces/NeuML/txtai/tree/main/workflows)++## MethodsWorkflows are callable objects. Workflows take an input of iterable data elements and output iterable data elements.diff --git a/docs/workflow/retrieve.md b/docs/workflow/retrieve.mdindex 6241b84f..55d9b003 100644--- a/docs/workflow/retrieve.md+++ b/docs/workflow/retrieve.md@@ -1,4 +1,4 @@-## Retrieve Task+# Retrieve TaskTask that connects to a url and downloads the content locally.diff --git a/docs/workflow/service.md b/docs/workflow/service.mdindex 09668784..ef07c131 100644--- a/docs/workflow/service.md+++ b/docs/workflow/service.md@@ -1,4 +1,4 @@-## Service Task+# Service TaskTask that runs content against a http service.diff --git a/docs/workflow/storage.md b/docs/workflow/storage.mdindex 9d4ed836..769ddb06 100644--- a/docs/workflow/storage.md+++ b/docs/workflow/storage.md@@ -1,4 +1,4 @@-## Storage Task+# Storage TaskTask that expands a local directory or cloud storage bucket into a list of URLs to process.diff --git a/docs/workflow/url.md b/docs/workflow/url.mdindex 2928625a..9ca5f6ce 100644--- a/docs/workflow/url.md+++ b/docs/workflow/url.md@@ -1,4 +1,4 @@-## Url Task+# Url TaskTask that processes urls.diff --git a/docs/workflow/workflow.md b/docs/workflow/workflow.mdindex ef61f624..50691c11 100644--- a/docs/workflow/workflow.md+++ b/docs/workflow/workflow.md@@ -1,4 +1,4 @@-## Workflow Task+# Workflow TaskTask that runs a Workflow. Allows creating workflows of workflows.diff --git a/mkdocs.yml b/mkdocs.ymlindex 18f27e87..6b20d43d 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -82,5 +82,9 @@ nav:- Storage: workflow/storage.md- Url: workflow/url.md- Workflow: workflow/workflow.md- - API: api.md+ - API:+ - api/index.md+ - Configuration: api/configuration.md+ - Cluster: api/cluster.md+ - Docker: api/docker.md- Further Reading: articles.md