š” All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Re-run notebooks, move recently added comments to codeblocks
davidmezzetti2337954a154b
diff --git a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynbindex f9058ebf..14226870 100644--- a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb+++ b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb@@ -68,7 +68,7 @@"cell_type": "code","metadata": {"id": "TONQ4_Kv9dtd",- "outputId": "20a392d5-379a-4f91-91fb-87dd94ce164c",+ "outputId": "5d7b5fdf-388c-43c2-c2ac-68dac1028811","colab": {"base_uri": "https://localhost:8080/","height": 315@@ -84,21 +84,21 @@{"output_type": "stream","text": [- "--2020-10-01 13:24:07-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",- "Resolving github.com (github.com)... 140.82.113.3\n",- "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n",+ "--2020-10-20 21:43:52-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",+ "Resolving github.com (github.com)... 192.30.255.112\n",+ "Connecting to github.com (github.com)|192.30.255.112|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n",- "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201001%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201001T132408Z&X-Amz-Expires=300&X-Amz-Signature=3568e1f02e57ce5d1335e74a7b30ead95385f1d635e6469da40937ea5065e999&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",- "--2020-10-01 13:24:08-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201001%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201001T132408Z&X-Amz-Expires=300&X-Amz-Signature=3568e1f02e57ce5d1335e74a7b30ead95385f1d635e6469da40937ea5065e999&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",- "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.145.139\n",- "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.145.139|:443... connected.\n",+ "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201020%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201020T214229Z&X-Amz-Expires=300&X-Amz-Signature=4add756b1621527d4e29cc454f5e0d1546f7f2fa145c03d3b706a8ea75e84aa4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",+ "--2020-10-20 21:43:52-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201020%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201020T214229Z&X-Amz-Expires=300&X-Amz-Signature=4add756b1621527d4e29cc454f5e0d1546f7f2fa145c03d3b706a8ea75e84aa4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",+ "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.32.148\n",+ "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.32.148|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 2456199 (2.3M) [application/octet-stream]\n","Saving to: ātests.gzā\n","\n",- "tests.gz 100%[===================>] 2.34M 1.85MB/s in 1.3s \n",+ "tests.gz 100%[===================>] 2.34M 5.23MB/s in 0.4s \n","\n",- "2020-10-01 13:24:10 (1.85 MB/s) - ātests.gzā saved [2456199/2456199]\n",+ "2020-10-20 21:43:53 (5.23 MB/s) - ātests.gzā saved [2456199/2456199]\n","\n"],"name": "stdout"@@ -125,7 +125,7 @@"cell_type": "code","metadata": {"id": "fJcn-CAH-u3K",- "outputId": "94d7f874-e49b-4f66-e827-421e682e133c",+ "outputId": "72feece5-e3e6-4c67-8d83-fe4b73dc2064","colab": {"base_uri": "https://localhost:8080/","height": 156@@ -175,9 +175,9 @@"Converting vectors to magnitude format\n","total 78948\n","-rw-r--r-- 1 root root 8065024 Aug 25 01:44 articles.sqlite\n",- "-rw-r--r-- 1 root root 24145920 Oct 1 13:27 cord19-300d.magnitude\n",- "-rw-r--r-- 1 root root 48625387 Oct 1 13:26 cord19-300d.txt\n",- "drwxr-xr-x 1 root root 4096 Sep 28 16:35 sample_data\n"+ "-rw-r--r-- 1 root root 24145920 Oct 20 21:46 cord19-300d.magnitude\n",+ "-rw-r--r-- 1 root root 48625387 Oct 20 21:46 cord19-300d.txt\n",+ "drwxr-xr-x 1 root root 4096 Oct 14 16:31 sample_data\n"],"name": "stdout"}@@ -198,7 +198,7 @@"cell_type": "code","metadata": {"id": "5PrrxGRPGHqX",- "outputId": "0354f102-df50-4c6e-d369-d9f18a934254",+ "outputId": "7c56c254-d0cd-452a-85ad-d6be2b2f454e","colab": {"base_uri": "https://localhost:8080/","height": 52@@ -284,7 +284,7 @@"cell_type": "code","metadata": {"id": "CRbDhvvDKEl-",- "outputId": "789b06fc-7507-402f-82d7-87367468ba91",+ "outputId": "cda3fdff-bd63-4de3-b20f-7b79832e0ce2","colab": {"base_uri": "https://localhost:8080/","height": 293@@ -313,15 +313,15 @@"\n","df = pd.DataFrame(results, columns=[\"Title\", \"Published\", \"Reference\", \"Match\"])\n","\n",+ "# It has been reported that displaying HTML within VSCode doesn't work.\n",+ "# When using VSCode, the data can be exported to an external HTML file to view.\n",+ "# See example below.\n",+ "\n",+ "# htmlData = df.to_html(index=False)\n",+ "# with open(\"data.html\", \"w\") as file:\n",+ "# file.write(htmlData)\n",+ "\n","display(HTML(df.to_html(index=False)))"- #If this python script is run in vscode environment, the above line of code will not display the results.- #One option is to save the results in a local file data.html, and use the browser to open it to view the result.- #This method can be applied to all the other occurences of "display(HTML(df.to_html(index=False)))" in the rest of the example codes.- #Uncomment the following 4 lines of code if this is your situation:- #htmlData = df.to_html(index=False)- #with open("data.html", "w") as file:- # file.write(htmlData)- #print("done writing result to data.html.")],"execution_count": 5,"outputs": [@@ -412,7 +412,7 @@"cell_type": "code","metadata": {"id": "19fmKawThs6d",- "outputId": "7676702d-7894-4e25-eb65-df9e730a2d03",+ "outputId": "4092bbbe-f152-4471-f69a-c8097eb74ea1","colab": {"base_uri": "https://localhost:8080/","height": 293@@ -540,4 +540,4 @@]}]-}+}\ No newline at end of filediff --git a/examples/04_Extractive_QA_with_Elasticsearch.ipynb b/examples/04_Extractive_QA_with_Elasticsearch.ipynbindex 5f9496a6..6c28bc0b 100644--- a/examples/04_Extractive_QA_with_Elasticsearch.ipynb+++ b/examples/04_Extractive_QA_with_Elasticsearch.ipynb@@ -15,8 +15,7 @@{"cell_type": "markdown","metadata": {- "id": "zzZbP0LM6m5z",- "colab_type": "text"+ "id": "zzZbP0LM6m5z"},"source": ["# Extractive QA with Elasticsearch\n",@@ -27,8 +26,7 @@{"cell_type": "markdown","metadata": {- "id": "xk7t5Jcd6reO",- "colab_type": "text"+ "id": "xk7t5Jcd6reO"},"source": ["# Install txtai and Elasticsearch\n",@@ -39,9 +37,7 @@{"cell_type": "code","metadata": {- "id": "0y1UA4-q-YdA",- "colab_type": "code",- "colab": {}+ "id": "0y1UA4-q-YdA"},"source": ["%%capture\n",@@ -60,14 +56,15 @@{"cell_type": "code","metadata": {- "id": "3ZfJeWbM6wmj",- "colab_type": "code",- "colab": {}+ "id": "3ZfJeWbM6wmj"},"source": ["import os\n","from subprocess import Popen, PIPE, STDOUT\n","\n",+ "# If issues are encountered with this section, ES can be manually started as follows:\n",+ "# ./elasticsearch-7.8.1/bin/elasticsearch\n",+ "\n","# Start and wait for server\n","server = Popen(['elasticsearch-7.8.1/bin/elasticsearch'], stdout=PIPE, stderr=STDOUT, preexec_fn=lambda: os.setuid(1))\n","!sleep 30"@@ -75,13 +72,10 @@"execution_count": 2,"outputs": []},- Note: if this is tested in a vscode or local developmeent environment, one can go to elasticsearch-7.8.1/bin directory,- directly start the elasticsearch server by typing "./elasticsearch" in the command line.{"cell_type": "markdown","metadata": {- "id": "TWEn4w68-D1y",- "colab_type": "text"+ "id": "TWEn4w68-D1y"},"source": ["# Download data\n",@@ -95,12 +89,11 @@"cell_type": "code","metadata": {"id": "8tVrIqSq-KBa",- "colab_type": "code",+ "outputId": "13d7c544-f0ff-4930-8a84-936fc8d9d68a","colab": {"base_uri": "https://localhost:8080/","height": 315- },- "outputId": "d4b8b54d-1ede-45c1-adef-94bfa9a104cf"+ }},"source": ["!wget https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",@@ -112,21 +105,21 @@{"output_type": "stream","text": [- "--2020-08-25 01:56:54-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",- "Resolving github.com (github.com)... 140.82.112.4\n",- "Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",+ "--2020-10-20 22:00:03-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",+ "Resolving github.com (github.com)... 52.69.186.44\n",+ "Connecting to github.com (github.com)|52.69.186.44|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n",- "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200825T015604Z&X-Amz-Expires=300&X-Amz-Signature=e135dc0b8f0d7774019d6525aad528b99160aff4007b2a539330c090c29ef9b5&X-Amz-SignedHeaders=host&actor_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",- "--2020-08-25 01:56:54-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200825T015604Z&X-Amz-Expires=300&X-Amz-Signature=e135dc0b8f0d7774019d6525aad528b99160aff4007b2a539330c090c29ef9b5&X-Amz-SignedHeaders=host&actor_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",- "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.140.68\n",- "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.140.68|:443... connected.\n",+ "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201020%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201020T220003Z&X-Amz-Expires=300&X-Amz-Signature=171812f33d30980a4ab40547678d0ca1622cb5e884ab9ce18cff2763b7d28606&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",+ "--2020-10-20 22:00:03-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201020%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201020T220003Z&X-Amz-Expires=300&X-Amz-Signature=171812f33d30980a4ab40547678d0ca1622cb5e884ab9ce18cff2763b7d28606&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",+ "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.133.75\n",+ "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.133.75|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 2456199 (2.3M) [application/octet-stream]\n","Saving to: ātests.gzā\n","\n",- "tests.gz 100%[===================>] 2.34M 11.2MB/s in 0.2s \n",+ "tests.gz 100%[===================>] 2.34M 1.87MB/s in 1.3s \n","\n",- "2020-08-25 01:56:55 (11.2 MB/s) - ātests.gzā saved [2456199/2456199]\n",+ "2020-10-20 22:00:05 (1.87 MB/s) - ātests.gzā saved [2456199/2456199]\n","\n"],"name": "stdout"@@ -136,8 +129,7 @@{"cell_type": "markdown","metadata": {- "id": "hSWFzkCn61tM",- "colab_type": "text"+ "id": "hSWFzkCn61tM"},"source": ["# Load data into Elasticsearch\n",@@ -149,12 +141,11 @@"cell_type": "code","metadata": {"id": "So-OBvUT61QD",- "colab_type": "code",+ "outputId": "c9059774-1c3f-49bd-9d6c-4bd060e6bfb2","colab": {"base_uri": "https://localhost:8080/","height": 34- },- "outputId": "c4d447fd-4103-4cb6-d16e-83c84fc1b689"+ }},"source": ["import sqlite3\n",@@ -219,8 +210,7 @@{"cell_type": "markdown","metadata": {- "id": "X5RO-VNwzMAo",- "colab_type": "text"+ "id": "X5RO-VNwzMAo"},"source": ["# Query data\n",@@ -233,12 +223,11 @@"cell_type": "code","metadata": {"id": "ucd9mwSfFTMm",- "colab_type": "code",+ "outputId": "7609d9dc-f169-49a3-fbbc-2c22e0f41b1a","colab": {"base_uri": "https://localhost:8080/","height": 293- },- "outputId": "582b3aa1-9eeb-431b-b7d1-509b7c3c28f2"+ }},"source": ["import pandas as pd\n",@@ -281,12 +270,6 @@" </thead>\n"," <tbody>\n"," <tr>\n",- " <td>Management of osteoarthritis during COVIDā19 pandemic</td>\n",- " <td>2020-05-21 00:00:00</td>\n",- " <td>https://doi.org/10.1002/cpt.1910</td>\n",- " <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",- " </tr>\n",- " <tr>\n"," <td>Prevalence and Impact of Myocardial Injury in Patients Hospitalized with COVID-19 Infection</td>\n"," <td>2020-04-24 00:00:00</td>\n"," <td>http://medrxiv.org/cgi/content/short/2020.04.20.20072702v1?rss=1</td>\n",@@ -299,16 +282,22 @@" <td>Risk factors associated with subsequent death include older age, hypertension, diabetes, ischemic heart disease, obesity and chronic lung disease; however, sometimes there are no obvious risk factors .</td>\n"," </tr>\n"," <tr>\n",- " <td>COVID-19 and associations with frailty and multimorbidity: a prospective analysis of UK Biobank participants</td>\n",- " <td>2020-07-23 00:00:00</td>\n",- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32705587/</td>\n",- " <td>BACKGROUND: Frailty and multimorbidity have been suggested as risk factors for severe COVID-19 disease.</td>\n",+ " <td>Early Safety Indicators of COVID-19 ConvalescentPlasma in 5,000 Patients</td>\n",+ " <td>2020-05-14 00:00:00</td>\n",+ " <td>https://www.ncbi.nlm.nih.gov/pubmed/32511566/</td>\n",+ " <td>TRALI often presents as bilateral pulmonary edema with little evidence of circulatory overload, and TRALI is further categorized into two types based on the absence of acute respiratory distress syndrome (ARDS) risk factors (type I) or presence of ARDS risk factors (type II) (18) .</td>\n"," </tr>\n"," <tr>\n",- " <td>COVID-19: what has been learned and to be learned about the novel coronavirus disease</td>\n",- " <td>2020-03-15 00:00:00</td>\n",- " <td>https://doi.org/10.7150/ijbs.45134</td>\n",- " <td>⢠Three major risk factors for COVID-19 were sex (male), age (ā„60), and severe pneumonia.</td>\n",+ " <td>Associations with covid-19 hospitalisation amongst 406,793 adults: the UK Biobank prospective cohort study</td>\n",+ " <td>2020-05-11 00:00:00</td>\n",+ " <td>http://medrxiv.org/cgi/content/short/2020.05.06.20092957v1?rss=1</td>\n",+ " <td>In addition, many risk factors for covid-19 documented in the literature are highly correlated and it is not clear which may be independently related to risk.</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <td>Associations with covid-19 hospitalisation amongst 406,793 adults: the UK Biobank prospective cohort study</td>\n",+ " <td>2020-05-11 00:00:00</td>\n",+ " <td>http://medrxiv.org/cgi/content/short/2020.05.06.20092957v1?rss=1</td>\n",+ " <td>The large numbers of covariables available in this cohort also enabled multivariable adjustment, permitting assessment of independent risk factors.</td>\n"," </tr>\n"," </tbody>\n","</table>"@@ -326,9 +315,7 @@{"cell_type": "code","metadata": {- "id": "mwBTrCkcOM_H",- "colab_type": "code",- "colab": {}+ "id": "mwBTrCkcOM_H"},"source": ["%%capture\n",@@ -348,12 +335,11 @@"cell_type": "code","metadata": {"id": "Yv75Lh-cOpL9",- "colab_type": "code",+ "outputId": "b66df081-6be7-47ee-8fdd-396a9cacce94","colab": {"base_uri": "https://localhost:8080/","height": 328- },- "outputId": "6d1c81fc-2da4-480c-9799-061297fca8a8"+ }},"source": ["document = {\n",@@ -466,4 +452,4 @@]}]-}+}\ No newline at end of file