š” All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Fix tokenization error in notebook, addresses #28
davidmezzettie45c4d59f4c8
diff --git a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynbindex c6651e13..c80427c6 100644--- a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb+++ b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb@@ -16,8 +16,7 @@{"cell_type": "markdown","metadata": {- "id": "WDbhGHtG8jFE",- "colab_type": "text"+ "id": "WDbhGHtG8jFE"},"source": ["# Part 3: Build an Embeddings index from a data source\n",@@ -32,8 +31,7 @@{"cell_type": "markdown","metadata": {- "id": "UQ0fCwXn9bcH",- "colab_type": "text"+ "id": "UQ0fCwXn9bcH"},"source": ["# Install dependencies\n",@@ -44,9 +42,7 @@{"cell_type": "code","metadata": {- "id": "czPYSA2Q9ZHO",- "colab_type": "code",- "colab": {}+ "id": "czPYSA2Q9ZHO"},"source": ["%%capture\n",@@ -58,8 +54,7 @@{"cell_type": "markdown","metadata": {- "id": "SN9SCZKQ9fJF",- "colab_type": "text"+ "id": "SN9SCZKQ9fJF"},"source": ["# Download data\n",@@ -73,12 +68,11 @@"cell_type": "code","metadata": {"id": "TONQ4_Kv9dtd",- "colab_type": "code",+ "outputId": "20a392d5-379a-4f91-91fb-87dd94ce164c","colab": {"base_uri": "https://localhost:8080/","height": 315- },- "outputId": "55f85ebf-946e-44aa-e27e-026436be909e"+ }},"source": ["!wget https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",@@ -90,21 +84,21 @@{"output_type": "stream","text": [- "--2020-08-25 01:56:04-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",- "Resolving github.com (github.com)... 140.82.112.4\n",- "Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",+ "--2020-10-01 13:24:07-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",+ "Resolving github.com (github.com)... 140.82.113.3\n",+ "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n",- "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200825T015604Z&X-Amz-Expires=300&X-Amz-Signature=e135dc0b8f0d7774019d6525aad528b99160aff4007b2a539330c090c29ef9b5&X-Amz-SignedHeaders=host&actor_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",- "--2020-08-25 01:56:04-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200825T015604Z&X-Amz-Expires=300&X-Amz-Signature=e135dc0b8f0d7774019d6525aad528b99160aff4007b2a539330c090c29ef9b5&X-Amz-SignedHeaders=host&actor_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",- "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.110.243\n",- "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.110.243|:443... connected.\n",+ "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201001%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201001T132408Z&X-Amz-Expires=300&X-Amz-Signature=3568e1f02e57ce5d1335e74a7b30ead95385f1d635e6469da40937ea5065e999&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",+ "--2020-10-01 13:24:08-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201001%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201001T132408Z&X-Amz-Expires=300&X-Amz-Signature=3568e1f02e57ce5d1335e74a7b30ead95385f1d635e6469da40937ea5065e999&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",+ "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.145.139\n",+ "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.145.139|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 2456199 (2.3M) [application/octet-stream]\n","Saving to: ātests.gzā\n","\n",- "tests.gz 100%[===================>] 2.34M --.-KB/s in 0.09s \n",+ "tests.gz 100%[===================>] 2.34M 1.85MB/s in 1.3s \n","\n",- "2020-08-25 01:56:05 (24.7 MB/s) - ātests.gzā saved [2456199/2456199]\n",+ "2020-10-01 13:24:10 (1.85 MB/s) - ātests.gzā saved [2456199/2456199]\n","\n"],"name": "stdout"@@ -114,8 +108,7 @@{"cell_type": "markdown","metadata": {- "id": "bzdaJiZYBIHE",- "colab_type": "text"+ "id": "bzdaJiZYBIHE"},"source": ["# Build Word Vectors\n",@@ -132,12 +125,11 @@"cell_type": "code","metadata": {"id": "fJcn-CAH-u3K",- "colab_type": "code",+ "outputId": "94d7f874-e49b-4f66-e827-421e682e133c","colab": {"base_uri": "https://localhost:8080/","height": 156- },- "outputId": "8ad62b92-55b7-4f2e-af55-f313402574a8"+ }},"source": ["import os\n",@@ -159,7 +151,7 @@" cur.execute(\"SELECT Text from sections\")\n","\n"," for row in cur:\n",- " output.write(\" \".join(row[0]) + \"\\n\")\n",+ " output.write(\" \".join(Tokenizer.tokenize(row[0])) + \"\\n\")\n","\n"," # Free database resources\n"," db.close()\n",@@ -181,11 +173,11 @@"Streaming tokens to temporary file\n","Building 300 dimension model\n","Converting vectors to magnitude format\n",- "total 9024\n",- "-rw-r--r-- 1 root root 8065024 Aug 25 01:44 articles.sqlite\n",- "-rw-r--r-- 1 root root 360448 Aug 25 01:57 cord19-300d.magnitude\n",- "-rw-r--r-- 1 root root 807886 Aug 25 01:57 cord19-300d.txt\n",- "drwxr-xr-x 1 root root 4096 Jul 30 16:30 sample_data\n"+ "total 78948\n",+ "-rw-r--r-- 1 root root 8065024 Aug 25 01:44 articles.sqlite\n",+ "-rw-r--r-- 1 root root 24145920 Oct 1 13:27 cord19-300d.magnitude\n",+ "-rw-r--r-- 1 root root 48625387 Oct 1 13:26 cord19-300d.txt\n",+ "drwxr-xr-x 1 root root 4096 Sep 28 16:35 sample_data\n"],"name": "stdout"}@@ -194,8 +186,7 @@{"cell_type": "markdown","metadata": {- "id": "_UxcC1-JGH-d",- "colab_type": "text"+ "id": "_UxcC1-JGH-d"},"source": ["# Build an embeddings index\n",@@ -207,12 +198,11 @@"cell_type": "code","metadata": {"id": "5PrrxGRPGHqX",- "colab_type": "code",+ "outputId": "0354f102-df50-4c6e-d369-d9f18a934254","colab": {"base_uri": "https://localhost:8080/","height": 52- },- "outputId": "36c78d6a-1f01-4e66-d0a8-7d27d3698ab3"+ }},"source": ["import sqlite3\n",@@ -282,8 +272,7 @@{"cell_type": "markdown","metadata": {- "id": "zHk24su3e_gb",- "colab_type": "text"+ "id": "zHk24su3e_gb"},"source": ["# Query data\n",@@ -295,12 +284,11 @@"cell_type": "code","metadata": {"id": "CRbDhvvDKEl-",- "colab_type": "code",+ "outputId": "789b06fc-7507-402f-82d7-87367468ba91","colab": {"base_uri": "https://localhost:8080/","height": 293- },- "outputId": "3f52b2b9-5307-47af-8dee-fdd9e694250b"+ }},"source": ["import pandas as pd\n",@@ -344,34 +332,34 @@" </thead>\n"," <tbody>\n"," <tr>\n",- " <td>Prevalence and Impact of Myocardial Injury in Patients Hospitalized with COVID-19 Infection</td>\n",- " <td>2020-04-24 00:00:00</td>\n",- " <td>http://medrxiv.org/cgi/content/short/2020.04.20.20072702v1?rss=1</td>\n",- " <td>This risk was consistent across patients stratified by history of CVD, risk factors but no CVD, and neither CVD nor risk factors.</td>\n",+ " <td>Management of osteoarthritis during COVIDā19 pandemic</td>\n",+ " <td>2020-05-21 00:00:00</td>\n",+ " <td>https://doi.org/10.1002/cpt.1910</td>\n",+ " <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n"," </tr>\n"," <tr>\n",- " <td>COVID-19 and associations with frailty and multimorbidity: a prospective analysis of UK Biobank participants</td>\n",- " <td>2020-07-23 00:00:00</td>\n",- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32705587/</td>\n",- " <td>The identification of risk factors for contracting COVID-19 is crucial, to inform public health policy and to facilitate the appropriate distribution of healthcare resources.</td>\n",+ " <td>Work-related and Personal Factors Associated with Mental Well-being during COVID-19 Response: A Survey of Health Care and Other Workers</td>\n",+ " <td>2020-06-11 00:00:00</td>\n",+ " <td>http://medrxiv.org/cgi/content/short/2020.06.09.20126722v1?rss=1</td>\n",+ " <td>Poor family supportive behaviors by supervisors were also associated with these outcomes [1.40 (1.21 - 1.62), 1.69 (1.48 - 1.92), 1.54 (1.44 - 1.64)].</td>\n"," </tr>\n"," <tr>\n",- " <td>Quantitative evaluation of olfactory dysfunction in hospitalized patients with Coronavirus [2] (COVID-19)</td>\n",- " <td>2020-05-25 00:00:00</td>\n",- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32451613/</td>\n",- " <td>In addition, these reports included patients with minor COVID-19 symptoms and low-risk factor burden.</td>\n",+ " <td>No evidence that androgen regulation of pulmonary TMPRSS2 explains sex-discordant COVID-19 outcomes</td>\n",+ " <td>2020-04-21 00:00:00</td>\n",+ " <td>https://doi.org/10.1101/2020.04.21.051201</td>\n",+ " <td>In addition to male sex, smoking is a risk factor for COVID-19 susceptibility and poor clinical outcomes .</td>\n"," </tr>\n"," <tr>\n",- " <td>COVID-19 from the perspective of urban and rural general adult mental health services</td>\n",- " <td>2020-05-21 00:00:00</td>\n",- " <td>https://doi.org/10.1017/ipm.2020.62</td>\n",- " <td>At-risk groups among staff members and service users were identified early and prioritised in service changes.</td>\n",+ " <td>Current status of potential therapeutic candidates for the COVID-19 crisis</td>\n",+ " <td>2020-04-22 00:00:00</td>\n",+ " <td>https://doi.org/10.1016/j.bbi.2020.04.046</td>\n",+ " <td>There was no difference on 28-day mortality between heparin users and nonusers.</td>\n"," </tr>\n"," <tr>\n",- " <td>Management of osteoarthritis during COVIDā19 pandemic</td>\n",- " <td>2020-05-21 00:00:00</td>\n",- " <td>https://doi.org/10.1002/cpt.1910</td>\n",- " <td>Consistently, a recent report indicated diabetes as a risk factor significantly associated with COVID-19 unfavourable clinical outcomes (37) .</td>\n",+ " <td>COVID-19: what has been learned and to be learned about the novel coronavirus disease</td>\n",+ " <td>2020-03-15 00:00:00</td>\n",+ " <td>https://doi.org/10.7150/ijbs.45134</td>\n",+ " <td>⢠Three major risk factors for COVID-19 were sex (male), age (ā„60), and severe pneumonia.</td>\n"," </tr>\n"," </tbody>\n","</table>"@@ -389,8 +377,7 @@{"cell_type": "markdown","metadata": {- "id": "XSf68I-ZfXOG",- "colab_type": "text"+ "id": "XSf68I-ZfXOG"},"source": ["# Extracting additional columns from query results\n",@@ -401,9 +388,7 @@{"cell_type": "code","metadata": {- "id": "TLVOTQJchvTi",- "colab_type": "code",- "colab": {}+ "id": "TLVOTQJchvTi"},"source": ["%%capture\n",@@ -419,12 +404,11 @@"cell_type": "code","metadata": {"id": "19fmKawThs6d",- "colab_type": "code",+ "outputId": "7676702d-7894-4e25-eb65-df9e730a2d03","colab": {"base_uri": "https://localhost:8080/","height": 293- },- "outputId": "e5c24448-97b4-444f-d47e-d268ef18aec5"+ }},"source": ["db = sqlite3.connect(\"articles.sqlite\")\n",@@ -476,43 +460,43 @@" </thead>\n"," <tbody>\n"," <tr>\n",- " <td>Prevalence and Impact of Myocardial Injury in Patients Hospitalized with COVID-19 Infection</td>\n",- " <td>2020-04-24 00:00:00</td>\n",- " <td>http://medrxiv.org/cgi/content/short/2020.04.20.20072702v1?rss=1</td>\n",- " <td>This risk was consistent across patients stratified by history of CVD, risk factors but no CVD, and neither CVD nor risk factors.</td>\n",- " <td>neither CVD nor risk factors</td>\n",- " <td>New York City hospitals</td>\n",+ " <td>Management of osteoarthritis during COVIDā19 pandemic</td>\n",+ " <td>2020-05-21 00:00:00</td>\n",+ " <td>https://doi.org/10.1002/cpt.1910</td>\n",+ " <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",+ " <td>sex, obesity, genetic factors and mechanical factors</td>\n",+ " <td>None</td>\n"," </tr>\n"," <tr>\n",- " <td>COVID-19 and associations with frailty and multimorbidity: a prospective analysis of UK Biobank participants</td>\n",- " <td>2020-07-23 00:00:00</td>\n",- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32705587/</td>\n",- " <td>The identification of risk factors for contracting COVID-19 is crucial, to inform public health policy and to facilitate the appropriate distribution of healthcare resources.</td>\n",- " <td>Frailty and multimorbidity</td>\n",- " <td>hospital settings</td>\n",+ " <td>Work-related and Personal Factors Associated with Mental Well-being during COVID-19 Response: A Survey of Health Care and Other Workers</td>\n",+ " <td>2020-06-11 00:00:00</td>\n",+ " <td>http://medrxiv.org/cgi/content/short/2020.06.09.20126722v1?rss=1</td>\n",+ " <td>Poor family supportive behaviors by supervisors were also associated with these outcomes [1.40 (1.21 - 1.62), 1.69 (1.48 - 1.92), 1.54 (1.44 - 1.64)].</td>\n",+ " <td>Poor family supportive behaviors</td>\n",+ " <td>None</td>\n"," </tr>\n"," <tr>\n",- " <td>Quantitative evaluation of olfactory dysfunction in hospitalized patients with Coronavirus [2] (COVID-19)</td>\n",- " <td>2020-05-25 00:00:00</td>\n",- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32451613/</td>\n",- " <td>In addition, these reports included patients with minor COVID-19 symptoms and low-risk factor burden.</td>\n",- " <td>patients with minor COVID-19 symptoms and low-risk factor burden</td>\n",- " <td>COVID-19 wards</td>\n",+ " <td>No evidence that androgen regulation of pulmonary TMPRSS2 explains sex-discordant COVID-19 outcomes</td>\n",+ " <td>2020-04-21 00:00:00</td>\n",+ " <td>https://doi.org/10.1101/2020.04.21.051201</td>\n",+ " <td>In addition to male sex, smoking is a risk factor for COVID-19 susceptibility and poor clinical outcomes .</td>\n",+ " <td>Higher morbidity and mortality</td>\n",+ " <td>None</td>\n"," </tr>\n"," <tr>\n",- " <td>COVID-19 from the perspective of urban and rural general adult mental health services</td>\n",- " <td>2020-05-21 00:00:00</td>\n",- " <td>https://doi.org/10.1017/ipm.2020.62</td>\n",- " <td>At-risk groups among staff members and service users were identified early and prioritised in service changes.</td>\n",- " <td>At-risk groups among staff members and service users</td>\n",- " <td>rural regions</td>\n",+ " <td>Current status of potential therapeutic candidates for the COVID-19 crisis</td>\n",+ " <td>2020-04-22 00:00:00</td>\n",+ " <td>https://doi.org/10.1016/j.bbi.2020.04.046</td>\n",+ " <td>There was no difference on 28-day mortality between heparin users and nonusers.</td>\n",+ " <td>elicited strong inflammatory responses are favorable or detrimental</td>\n",+ " <td>None</td>\n"," </tr>\n"," <tr>\n",- " <td>Management of osteoarthritis during COVIDā19 pandemic</td>\n",- " <td>2020-05-21 00:00:00</td>\n",- " <td>https://doi.org/10.1002/cpt.1910</td>\n",- " <td>Consistently, a recent report indicated diabetes as a risk factor significantly associated with COVID-19 unfavourable clinical outcomes (37) .</td>\n",- " <td>sex, obesity, genetic factors and mechanical factors</td>\n",+ " <td>COVID-19: what has been learned and to be learned about the novel coronavirus disease</td>\n",+ " <td>2020-03-15 00:00:00</td>\n",+ " <td>https://doi.org/10.7150/ijbs.45134</td>\n",+ " <td>⢠Three major risk factors for COVID-19 were sex (male), age (ā„60), and severe pneumonia.</td>\n",+ " <td>sex (male), age (ā„60), and severe pneumonia</td>\n"," <td>None</td>\n"," </tr>\n"," </tbody>\n",@@ -531,8 +515,7 @@{"cell_type": "markdown","metadata": {- "id": "ColTLy--rWfR",- "colab_type": "text"+ "id": "ColTLy--rWfR"},"source": ["In the example above, the Embeddings index is used to find the top N results for a given query. On top of that, a question-answer extractor is used to derive additional columns based on a list of questions. In this case, the \"Risk Factors\" and \"Location\" columns were pulled from the document text."@@ -541,8 +524,7 @@{"cell_type": "markdown","metadata": {- "id": "KWyoysauy7Pr",- "colab_type": "text"+ "id": "KWyoysauy7Pr"},"source": ["# Next\n",