π‘ All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Refresh example notebooks. Add API Gallery notebook
davidmezzetti98c0a98a11da
diff --git a/README.md b/README.mdindex 42cb884b..61ca4011 100644--- a/README.md+++ b/README.md@@ -79,6 +79,7 @@ The examples directory has a series of examples and notebooks giving an overview| [Build an Embeddings index from a data source](https://github.com/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) | Embeddings index from a data source backed by word embeddings | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) || [Extractive QA with Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/04_Extractive_QA_with_Elasticsearch.ipynb) | Extractive question-answering with Elasticsearch | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Extractive_QA_with_Elasticsearch.ipynb) || [Labeling with zero-shot classification](https://github.com/neuml/txtai/blob/master/examples/05_Labeling_with_zero_shot_classification.ipynb) | Labeling with zero-shot classification | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Labeling_with_zero_shot_classification.ipynb) |+| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/06_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/06_API_Gallery.ipynb) |## Configurationdiff --git a/examples/01_Introducing_txtai.ipynb b/examples/01_Introducing_txtai.ipynbindex 4aefbca9..31a82f43 100644--- a/examples/01_Introducing_txtai.ipynb+++ b/examples/01_Introducing_txtai.ipynb@@ -128,7 +128,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "b481b5d9-3c61-413a-a0d8-472f30c8d2d1"+ "outputId": "b50e3b2c-337a-4bbe-cf7b-72eb2e75bada"},"source": ["import numpy as np\n",@@ -199,7 +199,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "bd7f32c4-7905-4de3-c8c2-8c6c8bdf5a77"+ "outputId": "cf66b86f-9c46-441d-e24c-2d00a455de59"},"source": ["# Create an index for the list of sections\n",@@ -256,7 +256,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "85314afd-0772-49ef-9b94-e4a0bedc4153"+ "outputId": "52db38d6-4a0f-4d81-bb77-60b5cf62eccd"},"source": ["embeddings.save(\"index\")\n",@@ -286,7 +286,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "09393584-f149-4a61-94bb-9e5fbf309132"+ "outputId": "abf029ec-9bf0-48c2-c2bd-07326b073010"},"source": ["!ls index"diff --git a/examples/02_Extractive_QA_with_txtai.ipynb b/examples/02_Extractive_QA_with_txtai.ipynbindex f6518bd5..9762847c 100644--- a/examples/02_Extractive_QA_with_txtai.ipynb+++ b/examples/02_Extractive_QA_with_txtai.ipynb@@ -88,7 +88,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "d3427fe6-b04d-4cde-a7e4-824fc1430447"+ "outputId": "0e802561-f115-4f5e-ada3-f2605c2e458f"},"source": ["sections = [\"Giants hit 3 HRs to down Dodgers\",\n",diff --git a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynbindex 0f5be883..f54cba66 100644--- a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb+++ b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb@@ -71,7 +71,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "f0c7ee46-f427-458e-f3d3-913cd13bd473"+ "outputId": "86f08372-8418-4f02-c932-6e4db6e710bb"},"source": ["!wget https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",@@ -83,21 +83,21 @@{"output_type": "stream","text": [- "--2020-11-20 17:28:20-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",+ "--2020-12-24 15:51:34-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n","Resolving github.com (github.com)... 140.82.113.3\n","Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n",- "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201120T172820Z&X-Amz-Expires=300&X-Amz-Signature=6fc939f24fb1b982c6c87602c2ed115447314011f9c8eda31f2a107b3d374d54&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",- "--2020-11-20 17:28:20-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201120T172820Z&X-Amz-Expires=300&X-Amz-Signature=6fc939f24fb1b982c6c87602c2ed115447314011f9c8eda31f2a107b3d374d54&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",- "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.163.3\n",- "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.163.3|:443... connected.\n",+ "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201224T155134Z&X-Amz-Expires=300&X-Amz-Signature=e9edf741855ab7e67d29082569baf7ce5ff3dd727f02bbd9420105b3bf892669&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",+ "--2020-12-24 15:51:34-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201224T155134Z&X-Amz-Expires=300&X-Amz-Signature=e9edf741855ab7e67d29082569baf7ce5ff3dd727f02bbd9420105b3bf892669&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",+ "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.104.99\n",+ "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.104.99|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 2456199 (2.3M) [application/octet-stream]\n","Saving to: βtests.gzβ\n","\n",- "tests.gz 100%[===================>] 2.34M --.-KB/s in 0.1s \n",+ "tests.gz 100%[===================>] 2.34M --.-KB/s in 0.09s \n","\n",- "2020-11-20 17:28:20 (22.9 MB/s) - βtests.gzβ saved [2456199/2456199]\n",+ "2020-12-24 15:51:35 (25.3 MB/s) - βtests.gzβ saved [2456199/2456199]\n","\n"],"name": "stdout"@@ -127,7 +127,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "af389b28-4016-4d1b-8ade-643c1b6ef8dd"+ "outputId": "a0b20a83-ab38-49da-8148-afa31c1f64cd"},"source": ["import os\n",@@ -173,9 +173,9 @@"Converting vectors to magnitude format\n","total 78948\n","-rw-r--r-- 1 root root 8065024 Aug 25 01:44 articles.sqlite\n",- "-rw-r--r-- 1 root root 24145920 Nov 20 17:31 cord19-300d.magnitude\n",- "-rw-r--r-- 1 root root 48625387 Nov 20 17:31 cord19-300d.txt\n",- "drwxr-xr-x 1 root root 4096 Nov 13 17:33 sample_data\n"+ "-rw-r--r-- 1 root root 24145920 Dec 24 15:55 cord19-300d.magnitude\n",+ "-rw-r--r-- 1 root root 48625387 Dec 24 15:54 cord19-300d.txt\n",+ "drwxr-xr-x 1 root root 4096 Dec 21 17:29 sample_data\n"],"name": "stdout"}@@ -199,7 +199,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "dae51d29-676f-41b4-c9a0-0e2b7e837405"+ "outputId": "a05b496a-6fe3-44a9-fed3-1e1ff66e53a9"},"source": ["import sqlite3\n",@@ -285,7 +285,7 @@"base_uri": "https://localhost:8080/","height": 293},- "outputId": "b982c2b4-97fd-4d22-b275-78eeba3ba04e"+ "outputId": "97229a51-0b97-46d6-9e53-f29439cd0c9f"},"source": ["import pandas as pd\n",@@ -413,7 +413,7 @@"base_uri": "https://localhost:8080/","height": 293},- "outputId": "550db0d6-2a86-4766-8bf0-14c18f8eb5f3"+ "outputId": "00332ff6-13e3-4af2-ce27-dc0670d6878c"},"source": ["db = sqlite3.connect(\"articles.sqlite\")\n",diff --git a/examples/04_Extractive_QA_with_Elasticsearch.ipynb b/examples/04_Extractive_QA_with_Elasticsearch.ipynbindex 1b790cdf..8a6c5f13 100644--- a/examples/04_Extractive_QA_with_Elasticsearch.ipynb+++ b/examples/04_Extractive_QA_with_Elasticsearch.ipynb@@ -93,7 +93,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "f9ccc4f4-dad6-477a-e149-0f0bc3649be7"+ "outputId": "032600e4-3b80-4b22-e208-f97a89075b44"},"source": ["!wget https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",@@ -105,21 +105,21 @@{"output_type": "stream","text": [- "--2020-11-20 17:29:03-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",- "Resolving github.com (github.com)... 140.82.112.4\n",- "Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",+ "--2020-12-24 15:52:18-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",+ "Resolving github.com (github.com)... 140.82.113.3\n",+ "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n",- "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201120T172904Z&X-Amz-Expires=300&X-Amz-Signature=b7134321c17bee4f62cf6efdb2e3af1d0f97a294dc718450612ef543a7098cf0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",- "--2020-11-20 17:29:03-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201120T172904Z&X-Amz-Expires=300&X-Amz-Signature=b7134321c17bee4f62cf6efdb2e3af1d0f97a294dc718450612ef543a7098cf0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",- "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.108.36\n",- "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.108.36|:443... connected.\n",+ "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201224T155134Z&X-Amz-Expires=300&X-Amz-Signature=e9edf741855ab7e67d29082569baf7ce5ff3dd727f02bbd9420105b3bf892669&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",+ "--2020-12-24 15:52:18-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201224%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201224T155134Z&X-Amz-Expires=300&X-Amz-Signature=e9edf741855ab7e67d29082569baf7ce5ff3dd727f02bbd9420105b3bf892669&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",+ "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.96.132\n",+ "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.96.132|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 2456199 (2.3M) [application/octet-stream]\n","Saving to: βtests.gzβ\n","\n",- "tests.gz 100%[===================>] 2.34M --.-KB/s in 0.1s \n",+ "tests.gz 100%[===================>] 2.34M --.-KB/s in 0.09s \n","\n",- "2020-11-20 17:29:04 (21.3 MB/s) - βtests.gzβ saved [2456199/2456199]\n",+ "2020-12-24 15:52:18 (25.4 MB/s) - βtests.gzβ saved [2456199/2456199]\n","\n"],"name": "stdout"@@ -144,7 +144,7 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "0521f5c1-14d2-4a0a-aa11-82de676a7f5a"+ "outputId": "1b400e43-4852-427c-891b-01be632cc6ad"},"source": ["import sqlite3\n",@@ -226,7 +226,7 @@"base_uri": "https://localhost:8080/","height": 293},- "outputId": "2d70a5d6-61b6-49f7-d55d-ee3ac1b120db"+ "outputId": "b3c14060-0d9a-451c-af89-e01759851ef4"},"source": ["import pandas as pd\n",@@ -269,12 +269,6 @@" </thead>\n"," <tbody>\n"," <tr>\n",- " <td>Management of osteoarthritis during COVIDβ19 pandemic</td>\n",- " <td>2020-05-21 00:00:00</td>\n",- " <td>https://doi.org/10.1002/cpt.1910</td>\n",- " <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",- " </tr>\n",- " <tr>\n"," <td>Prevalence and Impact of Myocardial Injury in Patients Hospitalized with COVID-19 Infection</td>\n"," <td>2020-04-24 00:00:00</td>\n"," <td>http://medrxiv.org/cgi/content/short/2020.04.20.20072702v1?rss=1</td>\n",@@ -298,6 +292,12 @@" <td>https://doi.org/10.7150/ijbs.45134</td>\n"," <td>β’ Three major risk factors for COVID-19 were sex (male), age (β₯60), and severe pneumonia.</td>\n"," </tr>\n",+ " <tr>\n",+ " <td>Associations with covid-19 hospitalisation amongst 406,793 adults: the UK Biobank prospective cohort study</td>\n",+ " <td>2020-05-11 00:00:00</td>\n",+ " <td>http://medrxiv.org/cgi/content/short/2020.05.06.20092957v1?rss=1</td>\n",+ " <td>In addition, many risk factors for covid-19 documented in the literature are highly correlated and it is not clear which may be independently related to risk.</td>\n",+ " </tr>\n"," </tbody>\n","</table>"],@@ -338,7 +338,7 @@"base_uri": "https://localhost:8080/","height": 328},- "outputId": "d4a1ae7c-7708-4207-f9df-178dc7795ac3"+ "outputId": "fba7fd0b-5017-4c94-a15c-b691277d464f"},"source": ["document = {\n",diff --git a/examples/05_Labeling_with_zero_shot_classification.ipynb b/examples/05_Labeling_with_zero_shot_classification.ipynbindex 133b8de2..a61fdbc8 100644--- a/examples/05_Labeling_with_zero_shot_classification.ipynb+++ b/examples/05_Labeling_with_zero_shot_classification.ipynb@@ -103,7 +103,7 @@"base_uri": "https://localhost:8080/"},"id": "-K2YJJzsVtfq",- "outputId": "ce29f32c-49da-4178-ae27-7c444df62340"+ "outputId": "ce2e7c95-14cc-4972-c030-bcbf424cf1c0"},"source": ["sections = [\"Dodgers lose again, give up 3 HRs in a loss to the Giants\",\n",@@ -123,6 +123,7 @@"tags = [\"Baseball\", \"Football\", \"Hockey\", \"Basketball\"]\n","\n","print(\"%-75s %s\" % (\"Text\", \"Label\"))\n",+ "print(\"-\" * 100)\n","\n","for text in sections:\n"," print(\"%-75s %s\" % (text, labels(text, tags)[0][0]))"@@ -133,6 +134,7 @@"output_type": "stream","text": ["Text Label\n",+ "----------------------------------------------------------------------------------------------------\n","Dodgers lose again, give up 3 HRs in a loss to the Giants Baseball\n","Giants 5 Cardinals 4 final in extra innings Baseball\n","Dodgers drop Game 2 against the Giants, 5-4 Baseball\n",@@ -168,11 +170,14 @@"base_uri": "https://localhost:8080/"},"id": "uIf064M9pbjn",- "outputId": "6cf15ade-286f-4167-d7d6-449849bacd16"+ "outputId": "75e38914-6491-42cb-d915-6ab55f681f9d"},"source": ["tags = [\"π\", \"π‘\"]\n","\n",+ "print(\"%-75s %s\" % (\"Text\", \"Label\"))\n",+ "print(\"-\" * 100)\n",+ "\n","for text in sections:\n"," print(\"%-75s %s\" % (text, labels(text, tags)[0][0]))"],@@ -181,6 +186,8 @@{"output_type": "stream","text": [+ "Text Label\n",+ "----------------------------------------------------------------------------------------------------\n","Dodgers lose again, give up 3 HRs in a loss to the Giants π‘\n","Giants 5 Cardinals 4 final in extra innings π\n","Dodgers drop Game 2 against the Giants, 5-4 π‘\n",@@ -197,6 +204,16 @@"name": "stdout"}]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "ETnbw2dHbtRH"+ },+ "source": [+ "# Next\n",+ "Part 6 of this series shows how to use the txtai API."+ ]}]}\ No newline at end of filediff --git a/examples/06_API_Gallery.ipynb b/examples/06_API_Gallery.ipynbnew file mode 100644index 00000000..443e9fcc--- /dev/null+++ b/examples/06_API_Gallery.ipynb@@ -0,0 +1,842 @@+{+ "nbformat": 4,+ "nbformat_minor": 0,+ "metadata": {+ "colab": {+ "name": "06 - API Gallery",+ "provenance": [],+ "collapsed_sections": []+ },+ "kernelspec": {+ "name": "python3",+ "display_name": "Python 3"+ }+ },+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "4Pjmz-RORV8E"+ },+ "source": [+ "# Part 6: API Gallery\n",+ "\n",+ "The txtai API is a web-based service backed by [FastAPI](https://fastapi.tiangolo.com/). All txtai functionality including similarity search, extractive QA and zero-shot labeling is available via the API.\n",+ "\n",+ "This notebook installs the txtai API and shows an example using each of the supported language bindings for txtai.\n",+ "\n",+ "The examples in this notebook are **Christmas themed! π βοΈππ**"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "Dk31rbYjSTYm"+ },+ "source": [+ "# Install dependencies\n",+ "\n",+ "Install txtai and all dependencies"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "XMQuuun2R06J"+ },+ "source": [+ "%%capture\n",+ "!pip install git+https://github.com/neuml/txtai"+ ],+ "execution_count": 1,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "-vGR_piwZZO6"+ },+ "source": [+ "# Python\n",+ "\n",+ "The first method we'll try is direct access via Python. We'll use zero-shot labeling for all the examples here. See [this notebook](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Labeling_with_zero_shot_classification.ipynb) for more details on zero-shot classification. "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "P4q72tkRMMkR"+ },+ "source": [+ "## Configure Labels instance"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "8Dy_TJ0iM38Q"+ },+ "source": [+ "%%capture\n",+ "import os\n",+ "from IPython.core.display import display, HTML\n",+ "from txtai.pipeline import Labels\n",+ "\n",+ "def table(rows):\n",+ " html = \"\"\"\n",+ " <style type='text/css'>\n",+ " @import url('https://fonts.googleapis.com/css?family=Oswald&display=swap');\n",+ " table {\n",+ " border-collapse: collapse;\n",+ " width: 900px;\n",+ " }\n",+ " th, td {\n",+ " border: 1px solid #9e9e9e;\n",+ " padding: 10px;\n",+ " font: 20px Oswald;\n",+ " }\n",+ " </style>\n",+ " \"\"\"\n",+ "\n",+ " html += \"<table><thead><tr><th>Text</th><th>Label</th></tr></thead>\"\n",+ " for text, label in rows:\n",+ " html += \"<tr><td>%s</td><td>%s</td></tr>\" % (text, label)\n",+ " html += \"</table>\"\n",+ "\n",+ " display(HTML(html))\n",+ "\n",+ "# Create labels model\n",+ "labels = Labels()"+ ],+ "execution_count": 2,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "L4B73tGkMT6Q"+ },+ "source": [+ "## Apply labels to text"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/",+ "height": 324+ },+ "id": "-K2YJJzsVtfq",+ "outputId": "9e979db5-d25d-4f50-b7ee-e353e4ff9d23"+ },+ "source": [+ "sections = [\"Wears a red suit and says ho ho\",\n",+ " \"Pulls a flying sleigh\",\n",+ " \"This is cut down and decorated\",\n",+ " \"Santa puts these under the tree\",\n",+ " \"Best way to spend the holidays\"]\n",+ "\n",+ "# List of labels\n",+ "tags = [\"π Santa Clause\", \"π¦ Reindeer\", \"πͺ Cookies\", \"π Christmas Tree\", \"π Gifts\", \"πͺ Family\"]\n",+ "\n",+ "# Render output to table\n",+ "table([(text, labels(text, tags)[0][0]) for text in sections])"+ ],+ "execution_count": 3,+ "outputs": [+ {+ "output_type": "display_data",+ "data": {+ "text/html": [+ "\n",+ " <style type='text/css'>\n",+ " @import url('https://fonts.googleapis.com/css?family=Oswald&display=swap');\n",+ " table {\n",+ " border-collapse: collapse;\n",+ " width: 900px;\n",+ " }\n",+ " th, td {\n",+ " border: 1px solid #9e9e9e;\n",+ " padding: 10px;\n",+ " font: 20px Oswald;\n",+ " }\n",+ " </style>\n",+ " <table><thead><tr><th>Text</th><th>Label</th></tr></thead><tr><td>Wears a red suit and says ho ho</td><td>π Santa Clause</td></tr><tr><td>Pulls a flying sleigh</td><td>π¦ Reindeer</td></tr><tr><td>This is cut down and decorated</td><td>π Christmas Tree</td></tr><tr><td>Santa puts these under the tree</td><td>π Gifts</td></tr><tr><td>Best way to spend the holidays</td><td>πͺ Family</td></tr></table>"+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {+ "tags": []+ }+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "UF_bImkLHTMs"+ },+ "source": [+ "Once again we see the power of zero-shot labeling. The model wasn't trained on any data specific to this example. Still amazed with how much knowledge is stored in large NLP models."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "PNPJ95cdTKSS"+ },+ "source": [+ "# Start an API instance\n",+ "\n",+ "Now we'll start an API instance to run the remaining examples. The API needs a configuration file to run. The example below is simplified to only include labeling. See [this link](https://github.com/neuml/txtai#api) for a more detailed configuration example.\n",+ "\n",+ "The API instance is started in the background.\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "nTDwXOUeTH2-",+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "outputId": "c9b3aedf-63d5-45f0-c55a-c154bdfdba01"+ },+ "source": [+ "%%writefile index.yml\n",+ "\n",+ "# Labels settings\n",+ "labels:"+ ],+ "execution_count": 4,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Overwriting index.yml\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "nGITHxUyRzyp"+ },+ "source": [+ "!CONFIG=index.yml nohup uvicorn \"txtai.api:app\" &> api.log &\n",+ "!sleep 90"+ ],+ "execution_count": 5,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "NHvBFZeSd9AG"+ },+ "source": [+ "# JavaScript\n",+ "\n",+ "txtai.js is available via NPM and can be installed as follows.\n",+ "\n",+ "```bash\n",+ "npm install txtai\n",+ "```\n",+ "\n",+ "For this example, we'll clone the txtai.js project to import the example build configuration."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "b52knObEdcCr"+ },+ "source": [+ "%%capture\n",+ "!git clone https://github.com/neuml/txtai.js"+ ],+ "execution_count": 6,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "rUGS0t-JMsS9"+ },+ "source": [+ "## Create labels.js\n",+ "\n",+ "The following file is a JavaScript version of the labels example."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "zJbKRTSJV-kd",+ "outputId": "61847d78-69dc-4b03-afb0-e2107bd16265"+ },+ "source": [+ "%%writefile txtai.js/examples/node/src/labels.js\n",+ "import {Labels} from \"txtai\";\n",+ "import {sprintf} from \"sprintf-js\";\n",+ "\n",+ "const run = async () => {\n",+ " try {\n",+ " let labels = new Labels(\"http://localhost:8000\");\n",+ "\n",+ " let sections = [\"Wears a red suit and says ho ho\",\n",+ " \"Pulls a flying sleigh\",\n",+ " \"This is cut down and decorated\",\n",+ " \"Santa puts these under the tree\",\n",+ " \"Best way to spend the holidays\"];\n",+ "\n",+ " // List of labels\n",+ " let tags = [\"π Santa Clause\", \"π¦ Reindeer\", \"πͺ Cookies\", \"π Christmas Tree\", \"π Gifts\", \"πͺ Family\"];\n",+ "\n",+ " console.log(sprintf(\"%-40s %s\", \"Text\", \"Label\"));\n",+ " console.log(\"-\".repeat(75))\n",+ "\n",+ " for (let text of sections) {\n",+ " let label = await labels.label(text, tags);\n",+ " label = label[0][0];\n",+ "\n",+ " console.log(sprintf(\"%-40s %s\", text, label));\n",+ " }\n",+ " }\n",+ " catch (e) {\n",+ " console.trace(e);\n",+ " }\n",+ "};\n",+ "\n",+ "run();\n"+ ],+ "execution_count": 7,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Overwriting txtai.js/examples/node/src/labels.js\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "nTBs11j-GtD-"+ },+ "source": [+ "## Build and run labels example\n",+ "\n",+ "\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "kC5Oub6wa1nK"+ },+ "source": [+ "%%capture\n",+ "os.chdir(\"txtai.js/examples/node\")\n",+ "!npm install\n",+ "!npm run build"+ ],+ "execution_count": 8,+ "outputs": []+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "ckOHNqyaeL-B",+ "outputId": "8e12e280-da43-4ef7-9e91-121b9569802c"+ },+ "source": [+ "!node dist/labels.js"+ ],+ "execution_count": 9,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Text Label\n",+ "---------------------------------------------------------------------------\n",+ "Wears a red suit and says ho ho π Santa Clause\n",+ "Pulls a flying sleigh π¦ Reindeer\n",+ "This is cut down and decorated π Christmas Tree\n",+ "Santa puts these under the tree π Gifts\n",+ "Best way to spend the holidays πͺ Family\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "1yukBIMYG5OE"+ },+ "source": [+ "The JavaScript program is showing the same results as when natively running through Python!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "nNiMgvg0p2BG"+ },+ "source": [+ "# Java\n",+ "\n",+ "txtai.java integrates with standard Java build tools (Gradle, Maven, SBT). The following shows how to add txtai as a dependency to Gradle.\n",+ "\n",+ "```gradle\n",+ "implementation 'com.github.neuml:txtai.java:v1.0.0'\n",+ "```\n",+ "\n",+ "For this example, we'll clone the txtai.java project to import the example build configuration."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "qs2ai8lhqmga"+ },+ "source": [+ "%%capture\n",+ "os.chdir(\"/content\")\n",+ "!git clone https://github.com/neuml/txtai.java"+ ],+ "execution_count": 10,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "o8QFvzXkNFgq"+ },+ "source": [+ "## Create LabelsDemo.java\n",+ "\n",+ "The following file is a Java version of the labels example."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "v73L8Gw0p6fh",+ "outputId": "76a7d9d8-029c-4dd5-e47a-e1414a2d4930"+ },+ "source": [+ "%%writefile txtai.java/examples/src/main/java/LabelsDemo.java\n",+ "import java.util.Arrays;\n",+ "import java.util.ArrayList;\n",+ "import java.util.List;\n",+ "\n",+ "import txtai.Labels;\n",+ "import txtai.Labels.Score;\n",+ "\n",+ "public class LabelsDemo {\n",+ " public static void main(String[] args) {\n",+ " try {\n",+ " Labels labels = new Labels(\"http://localhost:8000\");\n",+ "\n",+ " List <String> sections = \n",+ " Arrays.asList(\"Wears a red suit and says ho ho\",\n",+ " \"Pulls a flying sleigh\",\n",+ " \"This is cut down and decorated\",\n",+ " \"Santa puts these under the tree\",\n",+ " \"Best way to spend the holidays\");\n",+ "\n",+ " // List of labels\n",+ " List<String> tags = Arrays.asList(\"π Santa Clause\", \"π¦ Reindeer\", \"πͺ Cookies\", \"π Christmas Tree\", \"π Gifts\", \"πͺ Family\");\n",+ "\n",+ " System.out.printf(\"%-40s %s%n\", \"Text\", \"Label\");\n",+ " System.out.println(new String(new char[75]).replace(\"\\0\", \"-\"));\n",+ "\n",+ " for (String text: sections) {\n",+ " List<Score> scores = labels.label(text, tags);\n",+ " System.out.printf(\"%-40s %s%n\", text, scores.get(0).id);\n",+ " }\n",+ " }\n",+ " catch (Exception ex) {\n",+ " ex.printStackTrace();\n",+ " }\n",+ " }\n",+ "}\n"+ ],+ "execution_count": 11,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Overwriting txtai.java/examples/src/main/java/LabelsDemo.java\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "wZv7eMIOLnRC"+ },+ "source": [+ "## Build and run labels example"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "N2Mm3Gl5sH1z",+ "outputId": "084ed3f1-2f23-4d7d-be44-fba8700b9f24"+ },+ "source": [+ "os.chdir(\"txtai.java/examples\")\n",+ "!../gradlew -q --console=plain labels 2> /dev/null"+ ],+ "execution_count": 12,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Text Label\n",+ "---------------------------------------------------------------------------\n",+ "Wears a red suit and says ho ho π Santa Clause\n",+ "Pulls a flying sleigh π¦ Reindeer\n",+ "This is cut down and decorated π Christmas Tree\n",+ "Santa puts these under the tree π Gifts\n",+ "Best way to spend the holidays πͺ Family\n",+ "\u001b[m"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "iHpQvUAgNp7j"+ },+ "source": [+ "The Java program is showing the same results as when natively running through Python!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "zU6jK2UL7D5H"+ },+ "source": [+ "# Rust\n",+ "\n",+ "txtai.rs is available via crates.io and can be installed by adding the following to your cargo.toml file.\n",+ "\n",+ "```toml\n",+ "[dependencies]\n",+ "txtai = { version = \"1.0\" }\n",+ "tokio = { version = \"0.2\", features = [\"full\"] }\n",+ "```\n",+ "\n",+ "For this example, we'll clone the txtai.rs project to import the example build configuration. First we need to install Rust."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "Ob4aswkx7jRh"+ },+ "source": [+ "%%capture\n",+ "os.chdir(\"/content\")\n",+ "!apt-get install rustc\n",+ "!git clone https://github.com/neuml/txtai.rs"+ ],+ "execution_count": 13,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "evEQQXBuObZn"+ },+ "source": [+ "## Create labels.rs\n",+ "\n",+ "The following file is a Rust version of the labels example."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "jjggKnKQ7jQO",+ "outputId": "c8b3ab7d-2699-43e9-c2d2-c46c9d09b3e6"+ },+ "source": [+ "%%writefile txtai.rs/examples/demo/src/labels.rs\n",+ "use std::error::Error;\n",+ "\n",+ "use txtai::labels::Labels;\n",+ "\n",+ "pub async fn labels() -> Result<(), Box<dyn Error>> {\n",+ " let labels = Labels::new(\"http://localhost:8000\");\n",+ "\n",+ " let sections = [\"Wears a red suit and says ho ho\",\n",+ " \"Pulls a flying sleigh\",\n",+ " \"This is cut down and decorated\",\n",+ " \"Santa puts these under the tree\",\n",+ " \"Best way to spend the holidays\"];\n",+ "\n",+ " println!(\"{:<40} {}\", \"Text\", \"Label\");\n",+ " println!(\"{}\", \"-\".repeat(75));\n",+ "\n",+ " for text in sections.iter() {\n",+ " let tags = vec![\"π Santa Clause\", \"π¦ Reindeer\", \"πͺ Cookies\", \"π Christmas Tree\", \"π Gifts\", \"πͺ Family\"];\n",+ " let label = &labels.label(text, &tags.to_vec()).await?[0].id;\n",+ "\n",+ " println!(\"{:<40} {}\", text, label);\n",+ " }\n",+ "\n",+ " Ok(())\n",+ "}"+ ],+ "execution_count": 14,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Overwriting txtai.rs/examples/demo/src/labels.rs\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "gFFPZO8sQZC4"+ },+ "source": [+ "## Build and run labels example\n",+ "\n",+ "\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "wuoAidGz9T4g"+ },+ "source": [+ "%%capture\n",+ "os.chdir(\"txtai.rs/examples/demo\")\n",+ "!cargo build"+ ],+ "execution_count": 15,+ "outputs": []+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "-_v_FbL0-yPk",+ "outputId": "ad31b672-0022-4482-d9b5-9cab38e9b3d6"+ },+ "source": [+ "!cargo run labels"+ ],+ "execution_count": 16,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "\u001b[0m\u001b[0m\u001b[1m\u001b[32m Finished\u001b[0m dev [unoptimized + debuginfo] target(s) in 0.07s\n",+ "\u001b[0m\u001b[0m\u001b[1m\u001b[32m Running\u001b[0m `target/debug/demo labels`\n",+ "Text Label\n",+ "---------------------------------------------------------------------------\n",+ "Wears a red suit and says ho ho π Santa Clause\n",+ "Pulls a flying sleigh π¦ Reindeer\n",+ "This is cut down and decorated π Christmas Tree\n",+ "Santa puts these under the tree π Gifts\n",+ "Best way to spend the holidays πͺ Family\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "kDmS89TPS3kb"+ },+ "source": [+ "The Rust program is showing the same results as when natively running through Python!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "ezznN4I8_CCQ"+ },+ "source": [+ "# Go\n",+ "\n",+ "txtai.go can be installed by adding the following import statement. When using modules, txtai.go will automatically be installed. Otherwise use `go get`.\n",+ "\n",+ "```golang\n",+ "import \"github.com/neuml/txtai.go\"\n",+ "```\n",+ "\n",+ "For this example, we'll create a standalone process for labeling. First we need to install Go."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "id": "b-b6fhLQ_DpQ"+ },+ "source": [+ "%%capture\n",+ "os.chdir(\"/content\")\n",+ "!apt install golang-go\n",+ "!go get \"github.com/neuml/txtai.go\""+ ],+ "execution_count": 17,+ "outputs": []+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "Dw-I6jOGR6vA"+ },+ "source": [+ "## Create labels.go\n",+ "\n",+ "The following file is a Go version of the labels example."+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "bLBJwkN4ANpi",+ "outputId": "8a02aa7f-21a4-4434-eceb-54f4a2bf80c3"+ },+ "source": [+ "%%writefile labels.go\n",+ "package main\n",+ "\n",+ "import (\n",+ "\t\"fmt\"\n",+ "\t\"strings\"\n",+ "\t\"github.com/neuml/txtai.go\"\n",+ ")\n",+ "\n",+ "func main() {\n",+ "\tlabels := txtai.Labels(\"http://localhost:8000\")\n",+ "\n",+ "\tsections := []string{\"Wears a red suit and says ho ho\",\n",+ " \"Pulls a flying sleigh\",\n",+ " \"This is cut down and decorated\",\n",+ " \"Santa puts these under the tree\",\n",+ " \"Best way to spend the holidays\"}\n",+ "\n",+ "\t// List of labels\n",+ "\ttags := []string{\"π Santa Clause\", \"π¦ Reindeer\", \"πͺ Cookies\", \"π Christmas Tree\", \"π Gifts\", \"πͺ Family\"}\n",+ "\n",+ "\tfmt.Printf(\"%-40s %s\\n\", \"Text\", \"Label\")\n",+ "\tfmt.Println(strings.Repeat(\"-\", 75))\n",+ "\n",+ "\tfor _, text := range sections {\n",+ "\t\tscores := labels.Label(text, tags)\n",+ "\t\tfmt.Printf(\"%-40s %s\\n\", text, scores[0].Id)\n",+ "\t}\n",+ "}"+ ],+ "execution_count": 18,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Overwriting labels.go\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "PJ2XzzDbSeZh"+ },+ "source": [+ "## Build and run labels example\n"+ ]+ },+ {+ "cell_type": "code",+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "l1xnUbtdAy0p",+ "outputId": "0bcdad5d-3e2c-4823-e7ab-87532fe06ae5"+ },+ "source": [+ "!go run labels.go"+ ],+ "execution_count": 19,+ "outputs": [+ {+ "output_type": "stream",+ "text": [+ "Text Label\n",+ "---------------------------------------------------------------------------\n",+ "Wears a red suit and says ho ho π Santa Clause\n",+ "Pulls a flying sleigh π¦ Reindeer\n",+ "This is cut down and decorated π Christmas Tree\n",+ "Santa puts these under the tree π Gifts\n",+ "Best way to spend the holidays πͺ Family\n"+ ],+ "name": "stdout"+ }+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "oml43X5eS6YB"+ },+ "source": [+ "The Go program is showing the same results as when natively running through Python!"+ ]+ }+ ]+}\ No newline at end of file