Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Update documentation

davidmezzetti089c96bf14ef
diff --git a/README.md b/README.md
index 0915b918..90f36598 100644
--- a/README.md
+++ b/README.md
@@ -63,40 +63,8 @@ You can also install txtai directly from GitHub. Using a Python Virtual Environm
pip install git+https://github.com/neuml/txtai
-Python 3.6+ is supported.
-
-txtai has the following optional dependencies that can be installed as extras.
-
-```
-# Install all dependencies (mirrors txtai < 3.2)
-pip install txtai[all]
-
-# Serve txtai via a web API
-pip install txtai[api]
-
-# All pipelines - default install comes with most common pipelines
-pip install txtai[pipeline]
-
-# Word vectors, support for sentence-transformer models not on the HF Hub and additional ANN libraries
-pip install txtai[similarity]
-
-# All workflow tasks - default install comes with most common workflow tasks
-pip install txtai[workflow]
-```
-
-Additional environment specific prerequisites are below.
-
-### Linux
-
-Optional audio transcription requires a [system library to be installed](https://github.com/bastibe/python-soundfile#installation)
-
-### macOS
-
-Run `brew install libomp` see [this link](https://github.com/kyamagu/faiss-wheels#prerequisite)
-
-### Windows
-
-Install [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
+Python 3.6+ is supported. See the detailed [install instructions](https://neuml.github.io/txtai) for more information covering optional
+dependencies and environment specific prerequisites.
## Examples
diff --git a/docs/examples.md b/docs/examples.md
index 3c0c0606..ce7a00d0 100644
--- a/docs/examples.md
+++ b/docs/examples.md
@@ -2,7 +2,9 @@
The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
-## Notebooks
+## Similarity Search
+
+Build similarity/semantic/vector search applications with txtai.
| Notebook | Description | |
|:----------|:-------------|------:|
@@ -10,20 +12,29 @@ The examples directory has a series of notebooks and applications giving an over
| [Build an Embeddings index with Hugging Face Datasets](https://github.com/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) | Index and search Hugging Face Datasets | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) |
| [Build an Embeddings index from a data source](https://github.com/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) | Index and search a data source with word embeddings | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) |
| [Add semantic search to Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) | Add semantic search to existing search systems | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) |
+| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
+| [Similarity search with images](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | Embed images and text into the same space for search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) |
+| [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |
+
+## Pipelines and Workflows
+
+NLP-backed data transformation pipelines and workflows. Run existing models and train new ones.
+
+| Notebook | Description | |
+|:----------|:-------------|------:|
| [Extractive QA with txtai](https://github.com/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) | Introduction to extractive question-answering with txtai | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) |
| [Extractive QA with Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) | Run extractive question-answering queries with Elasticsearch | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) |
| [Apply labels with zero shot classification](https://github.com/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) | Use zero shot learning for labeling, classification and topic modeling | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) |
-| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
| [Building abstractive text summaries](https://github.com/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) | Run abstractive text summarization | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) |
| [Extract text from documents](https://github.com/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) | Extract text from PDF, Office, HTML and more | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) |
| [Transcribe audio to text](https://github.com/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) | Convert audio files to text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) |
| [Translate text between languages](https://github.com/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) | Streamline machine translation and language detection | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) |
-| [Similarity search with images](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | Embed images and text into the same space for search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) |
| [Run pipeline workflows](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) | Simple yet powerful constructs to efficiently process data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |
-| [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |
## Applications
+Series of example applications with txtai.
+
| Application | Description |
|:----------|:-------------|
| [Demo query shell](https://github.com/neuml/txtai/blob/master/examples/demo.py) | Basic similarity search example. Used in the original txtai demo. |
diff --git a/docs/install.md b/docs/install.md
index d18f756a..5c8aff4b 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -8,7 +8,32 @@ You can also install txtai directly from GitHub. Using a Python Virtual Environm
pip install git+https://github.com/neuml/txtai
-Python 3.6+ is supported. txtai has the following environment specific prerequisites.
+Python 3.6+ is supported.
+
+txtai has the following optional dependencies that can be installed as extras. The patterns below are supported
+in setup.py install_requires sections.
+
+```
+# Install all dependencies (mirrors txtai < 3.2)
+pip install txtai[all]
+
+# Serve txtai via a web API
+pip install txtai[api]
+
+# All pipelines - default install comes with most common pipelines
+pip install txtai[pipeline]
+
+# Word vectors, support for sentence-transformer models not on the HF Hub and additional ANN libraries
+pip install txtai[similarity]
+
+# All workflow tasks - default install comes with most common workflow tasks
+pip install txtai[workflow]
+
+# Pipelines and workflows
+pip install txtai[pipeline,workflow]
+```
+
+Additional environment specific prerequisites are below.
## Linux
@@ -20,4 +45,4 @@ Run `brew install libomp` see [this link](https://github.com/kyamagu/faiss-wheel
## Windows
-Install [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
+Optional dependencies require [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)