Skip to content

armgit / txtai

Public Imported from GitHub
0

πŸ’‘ All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Update notebook

davidmezzetti9fc2ebe23ca8
diff --git a/examples/01_Introducing_txtai.ipynb b/examples/01_Introducing_txtai.ipynb
index 9b473011..cafff33f 100644
--- a/examples/01_Introducing_txtai.ipynb
+++ b/examples/01_Introducing_txtai.ipynb
@@ -34,21 +34,23 @@
"source": [
"# Part 1: Introducing txtai\n",
"\n",
- "[txtai](https://github.com/neuml/txtai) builds an AI-powered index over sections of text. txtai supports building text indices to perform similarity searches and create extractive question-answering based systems. txtai also has functionality for zero-shot classification. \n",
+ "[txtai](https://github.com/neuml/txtai) executes machine-learning workflows to transform data and build AI-powered text indices to perform similarity search. The following is a summary of key features:\n",
"\n",
- "NeuML uses txtai and/or the concepts behind it to power all of our Natural Language Processing (NLP) applications. Example applications:\n",
+ "- πŸ”Ž Large-scale similarity search with multiple index backends ([Faiss](https://github.com/facebookresearch/faiss), [Annoy](https://github.com/spotify/annoy), [Hnswlib](https://github.com/nmslib/hnswlib))\n",
+ "- πŸ“„ Create embeddings for text snippets, documents, audio and images. Supports transformers and word vectors.\n",
+ "- πŸ’‘ Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction\n",
+ "- β†ͺ️️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.\n",
+ "- πŸ”— API bindings for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go)\n",
+ "- ☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)\n",
+ "\n",
+ "txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below:\n",
"\n",
"- [paperai](https://github.com/neuml/paperai) - AI-powered literature discovery and review engine for medical/scientific papers\n",
"- [tldrstory](https://github.com/neuml/tldrstory) - AI-powered understanding of headlines and story text\n",
"- [neuspo](https://neuspo.com) - Fact-driven, real-time sports event and news site\n",
"- [codequestion](https://github.com/neuml/codequestion) - Ask coding questions directly from the terminal\n",
"\n",
- "txtai is built on the following stack:\n",
- "\n",
- "- [sentence-transformers](https://github.com/UKPLab/sentence-transformers)\n",
- "- [transformers](https://github.com/huggingface/transformers)\n",
- "- [faiss](https://github.com/facebookresearch/faiss)\n",
- "- Python 3.6+\n",
+ "txtai is built with Python 3.6+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) and [FastAPI](https://github.com/tiangolo/fastapi)\n",
"\n",
"This notebook gives an overview of txtai and how to run similarity searches."
]
@@ -77,7 +79,7 @@
"%%capture\n",
"!pip install git+https://github.com/neuml/txtai"
],
- "execution_count": 1,
+ "execution_count": null,
"outputs": []
},
{
@@ -105,7 +107,7 @@
"# Create embeddings model, backed by sentence-transformers & transformers\n",
"embeddings = Embeddings({\"method\": \"transformers\", \"path\": \"sentence-transformers/bert-base-nli-mean-tokens\"})"
],
- "execution_count": 2,
+ "execution_count": null,
"outputs": []
},
{
@@ -129,7 +131,7 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "420366d1-3215-4817-9269-461d43ba493b"
+ "outputId": "1c58e588-2c7e-4202-a636-9f67e24e69b3"
},
"source": [
"data = [\"US tops 5 million confirmed virus cases\",\n",
@@ -148,7 +150,7 @@
"\n",
" print(\"%-20s %s\" % (query, data[uid]))"
],
- "execution_count": 3,
+ "execution_count": null,
"outputs": [
{
"output_type": "stream",
@@ -198,7 +200,7 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "89c727e8-bc46-40e4-d411-7afc83d678da"
+ "outputId": "ed87f588-2cef-4717-b753-8cd152fa4ee7"
},
"source": [
"# Create an index for the list of text\n",
@@ -216,7 +218,7 @@
" # Print text\n",
" print(\"%-20s %s\" % (query, data[uid]))"
],
- "execution_count": 4,
+ "execution_count": null,
"outputs": [
{
"output_type": "stream",
@@ -255,7 +257,7 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "8af2eaf8-de58-49c4-8218-8b3beca090c4"
+ "outputId": "07e6ce2c-c88c-4955-d210-ce2585e5a79f"
},
"source": [
"embeddings.save(\"index\")\n",
@@ -266,7 +268,7 @@
"uid = embeddings.search(\"climate change\", 1)[0][0]\n",
"print(data[uid])"
],
- "execution_count": 5,
+ "execution_count": null,
"outputs": [
{
"output_type": "stream",
@@ -285,12 +287,12 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "1dc1b488-a9e8-472d-b348-9d7a13b72296"
+ "outputId": "c248984d-e855-4585-bc6b-5451aa892403"
},
"source": [
"!ls index"
],
- "execution_count": 6,
+ "execution_count": null,
"outputs": [
{
"output_type": "stream",