π‘ All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update notebook
davidmezzetti9fc2ebe23ca8
diff --git a/examples/01_Introducing_txtai.ipynb b/examples/01_Introducing_txtai.ipynbindex 9b473011..cafff33f 100644--- a/examples/01_Introducing_txtai.ipynb+++ b/examples/01_Introducing_txtai.ipynb@@ -34,21 +34,23 @@"source": ["# Part 1: Introducing txtai\n","\n",- "[txtai](https://github.com/neuml/txtai) builds an AI-powered index over sections of text. txtai supports building text indices to perform similarity searches and create extractive question-answering based systems. txtai also has functionality for zero-shot classification. \n",+ "[txtai](https://github.com/neuml/txtai) executes machine-learning workflows to transform data and build AI-powered text indices to perform similarity search. The following is a summary of key features:\n","\n",- "NeuML uses txtai and/or the concepts behind it to power all of our Natural Language Processing (NLP) applications. Example applications:\n",+ "- π Large-scale similarity search with multiple index backends ([Faiss](https://github.com/facebookresearch/faiss), [Annoy](https://github.com/spotify/annoy), [Hnswlib](https://github.com/nmslib/hnswlib))\n",+ "- π Create embeddings for text snippets, documents, audio and images. Supports transformers and word vectors.\n",+ "- π‘ Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction\n",+ "- βͺοΈοΈ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.\n",+ "- π API bindings for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go)\n",+ "- βοΈ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)\n",+ "\n",+ "txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below:\n","\n","- [paperai](https://github.com/neuml/paperai) - AI-powered literature discovery and review engine for medical/scientific papers\n","- [tldrstory](https://github.com/neuml/tldrstory) - AI-powered understanding of headlines and story text\n","- [neuspo](https://neuspo.com) - Fact-driven, real-time sports event and news site\n","- [codequestion](https://github.com/neuml/codequestion) - Ask coding questions directly from the terminal\n","\n",- "txtai is built on the following stack:\n",- "\n",- "- [sentence-transformers](https://github.com/UKPLab/sentence-transformers)\n",- "- [transformers](https://github.com/huggingface/transformers)\n",- "- [faiss](https://github.com/facebookresearch/faiss)\n",- "- Python 3.6+\n",+ "txtai is built with Python 3.6+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) and [FastAPI](https://github.com/tiangolo/fastapi)\n","\n","This notebook gives an overview of txtai and how to run similarity searches."]@@ -77,7 +79,7 @@"%%capture\n","!pip install git+https://github.com/neuml/txtai"],- "execution_count": 1,+ "execution_count": null,"outputs": []},{@@ -105,7 +107,7 @@"# Create embeddings model, backed by sentence-transformers & transformers\n","embeddings = Embeddings({\"method\": \"transformers\", \"path\": \"sentence-transformers/bert-base-nli-mean-tokens\"})"],- "execution_count": 2,+ "execution_count": null,"outputs": []},{@@ -129,7 +131,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "420366d1-3215-4817-9269-461d43ba493b"+ "outputId": "1c58e588-2c7e-4202-a636-9f67e24e69b3"},"source": ["data = [\"US tops 5 million confirmed virus cases\",\n",@@ -148,7 +150,7 @@"\n"," print(\"%-20s %s\" % (query, data[uid]))"],- "execution_count": 3,+ "execution_count": null,"outputs": [{"output_type": "stream",@@ -198,7 +200,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "89c727e8-bc46-40e4-d411-7afc83d678da"+ "outputId": "ed87f588-2cef-4717-b753-8cd152fa4ee7"},"source": ["# Create an index for the list of text\n",@@ -216,7 +218,7 @@" # Print text\n"," print(\"%-20s %s\" % (query, data[uid]))"],- "execution_count": 4,+ "execution_count": null,"outputs": [{"output_type": "stream",@@ -255,7 +257,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "8af2eaf8-de58-49c4-8218-8b3beca090c4"+ "outputId": "07e6ce2c-c88c-4955-d210-ce2585e5a79f"},"source": ["embeddings.save(\"index\")\n",@@ -266,7 +268,7 @@"uid = embeddings.search(\"climate change\", 1)[0][0]\n","print(data[uid])"],- "execution_count": 5,+ "execution_count": null,"outputs": [{"output_type": "stream",@@ -285,12 +287,12 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "1dc1b488-a9e8-472d-b348-9d7a13b72296"+ "outputId": "c248984d-e855-4585-bc6b-5451aa892403"},"source": ["!ls index"],- "execution_count": 6,+ "execution_count": null,"outputs": [{"output_type": "stream",