💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update documentation
davidmezzetti089c96bf14ef
diff --git a/README.md b/README.mdindex 0915b918..90f36598 100644--- a/README.md+++ b/README.md@@ -63,40 +63,8 @@ You can also install txtai directly from GitHub. Using a Python Virtual Environmpip install git+https://github.com/neuml/txtai-Python 3.6+ is supported.--txtai has the following optional dependencies that can be installed as extras.--```-# Install all dependencies (mirrors txtai < 3.2)-pip install txtai[all]--# Serve txtai via a web API-pip install txtai[api]--# All pipelines - default install comes with most common pipelines-pip install txtai[pipeline]--# Word vectors, support for sentence-transformer models not on the HF Hub and additional ANN libraries-pip install txtai[similarity]--# All workflow tasks - default install comes with most common workflow tasks-pip install txtai[workflow]-```--Additional environment specific prerequisites are below.--### Linux--Optional audio transcription requires a [system library to be installed](https://github.com/bastibe/python-soundfile#installation)--### macOS--Run `brew install libomp` see [this link](https://github.com/kyamagu/faiss-wheels#prerequisite)--### Windows--Install [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)+Python 3.6+ is supported. See the detailed [install instructions](https://neuml.github.io/txtai) for more information covering optional+dependencies and environment specific prerequisites.## Examplesdiff --git a/docs/examples.md b/docs/examples.mdindex 3c0c0606..ce7a00d0 100644--- a/docs/examples.md+++ b/docs/examples.md@@ -2,7 +2,9 @@The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.-## Notebooks+## Similarity Search++Build similarity/semantic/vector search applications with txtai.| Notebook | Description | ||:----------|:-------------|------:|@@ -10,20 +12,29 @@ The examples directory has a series of notebooks and applications giving an over| [Build an Embeddings index with Hugging Face Datasets](https://github.com/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) | Index and search Hugging Face Datasets | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) || [Build an Embeddings index from a data source](https://github.com/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) | Index and search a data source with word embeddings | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) || [Add semantic search to Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) | Add semantic search to existing search systems | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) |+| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |+| [Similarity search with images](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | Embed images and text into the same space for search | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) |+| [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |++## Pipelines and Workflows++NLP-backed data transformation pipelines and workflows. Run existing models and train new ones.++| Notebook | Description | |+|:----------|:-------------|------:|| [Extractive QA with txtai](https://github.com/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) | Introduction to extractive question-answering with txtai | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) || [Extractive QA with Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) | Run extractive question-answering queries with Elasticsearch | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) || [Apply labels with zero shot classification](https://github.com/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) | Use zero shot learning for labeling, classification and topic modeling | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) |-| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) || [Building abstractive text summaries](https://github.com/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) | Run abstractive text summarization | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) || [Extract text from documents](https://github.com/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) | Extract text from PDF, Office, HTML and more | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) || [Transcribe audio to text](https://github.com/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) | Convert audio files to text | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) || [Translate text between languages](https://github.com/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) | Streamline machine translation and language detection | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) |-| [Similarity search with images](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | Embed images and text into the same space for search | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) || [Run pipeline workflows](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) | Simple yet powerful constructs to efficiently process data | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |-| [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |## Applications+Series of example applications with txtai.+| Application | Description ||:----------|:-------------|| [Demo query shell](https://github.com/neuml/txtai/blob/master/examples/demo.py) | Basic similarity search example. Used in the original txtai demo. |diff --git a/docs/install.md b/docs/install.mdindex d18f756a..5c8aff4b 100644--- a/docs/install.md+++ b/docs/install.md@@ -8,7 +8,32 @@ You can also install txtai directly from GitHub. Using a Python Virtual Environmpip install git+https://github.com/neuml/txtai-Python 3.6+ is supported. txtai has the following environment specific prerequisites.+Python 3.6+ is supported.++txtai has the following optional dependencies that can be installed as extras. The patterns below are supported+in setup.py install_requires sections.++```+# Install all dependencies (mirrors txtai < 3.2)+pip install txtai[all]++# Serve txtai via a web API+pip install txtai[api]++# All pipelines - default install comes with most common pipelines+pip install txtai[pipeline]++# Word vectors, support for sentence-transformer models not on the HF Hub and additional ANN libraries+pip install txtai[similarity]++# All workflow tasks - default install comes with most common workflow tasks+pip install txtai[workflow]++# Pipelines and workflows+pip install txtai[pipeline,workflow]+```++Additional environment specific prerequisites are below.## Linux@@ -20,4 +45,4 @@ Run `brew install libomp` see [this link](https://github.com/kyamagu/faiss-wheel## Windows-Install [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)+Optional dependencies require [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)