Skip to content

armgit / txtai

Public Imported from GitHub
0

šŸ’” All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Fix tokenization error in notebook, addresses #28

davidmezzettie45c4d59f4c8
diff --git a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb
index c6651e13..c80427c6 100644
--- a/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb
+++ b/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb
@@ -16,8 +16,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "WDbhGHtG8jFE",
- "colab_type": "text"
+ "id": "WDbhGHtG8jFE"
},
"source": [
"# Part 3: Build an Embeddings index from a data source\n",
@@ -32,8 +31,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "UQ0fCwXn9bcH",
- "colab_type": "text"
+ "id": "UQ0fCwXn9bcH"
},
"source": [
"# Install dependencies\n",
@@ -44,9 +42,7 @@
{
"cell_type": "code",
"metadata": {
- "id": "czPYSA2Q9ZHO",
- "colab_type": "code",
- "colab": {}
+ "id": "czPYSA2Q9ZHO"
},
"source": [
"%%capture\n",
@@ -58,8 +54,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "SN9SCZKQ9fJF",
- "colab_type": "text"
+ "id": "SN9SCZKQ9fJF"
},
"source": [
"# Download data\n",
@@ -73,12 +68,11 @@
"cell_type": "code",
"metadata": {
"id": "TONQ4_Kv9dtd",
- "colab_type": "code",
+ "outputId": "20a392d5-379a-4f91-91fb-87dd94ce164c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 315
- },
- "outputId": "55f85ebf-946e-44aa-e27e-026436be909e"
+ }
},
"source": [
"!wget https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",
@@ -90,21 +84,21 @@
{
"output_type": "stream",
"text": [
- "--2020-08-25 01:56:04-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",
- "Resolving github.com (github.com)... 140.82.112.4\n",
- "Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",
+ "--2020-10-01 13:24:07-- https://github.com/neuml/txtai/releases/download/v1.1.0/tests.gz\n",
+ "Resolving github.com (github.com)... 140.82.113.3\n",
+ "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n",
"HTTP request sent, awaiting response... 302 Found\n",
- "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200825T015604Z&X-Amz-Expires=300&X-Amz-Signature=e135dc0b8f0d7774019d6525aad528b99160aff4007b2a539330c090c29ef9b5&X-Amz-SignedHeaders=host&actor_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",
- "--2020-08-25 01:56:04-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200825%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200825T015604Z&X-Amz-Expires=300&X-Amz-Signature=e135dc0b8f0d7774019d6525aad528b99160aff4007b2a539330c090c29ef9b5&X-Amz-SignedHeaders=host&actor_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",
- "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.110.243\n",
- "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.110.243|:443... connected.\n",
+ "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201001%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201001T132408Z&X-Amz-Expires=300&X-Amz-Signature=3568e1f02e57ce5d1335e74a7b30ead95385f1d635e6469da40937ea5065e999&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream [following]\n",
+ "--2020-10-01 13:24:08-- https://github-production-release-asset-2e65be.s3.amazonaws.com/286301447/080d8800-e653-11ea-8d02-c0c858a09e7a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201001%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201001T132408Z&X-Amz-Expires=300&X-Amz-Signature=3568e1f02e57ce5d1335e74a7b30ead95385f1d635e6469da40937ea5065e999&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=286301447&response-content-disposition=attachment%3B%20filename%3Dtests.gz&response-content-type=application%2Foctet-stream\n",
+ "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.145.139\n",
+ "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.145.139|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 2456199 (2.3M) [application/octet-stream]\n",
"Saving to: ā€˜tests.gz’\n",
"\n",
- "tests.gz 100%[===================>] 2.34M --.-KB/s in 0.09s \n",
+ "tests.gz 100%[===================>] 2.34M 1.85MB/s in 1.3s \n",
"\n",
- "2020-08-25 01:56:05 (24.7 MB/s) - ā€˜tests.gz’ saved [2456199/2456199]\n",
+ "2020-10-01 13:24:10 (1.85 MB/s) - ā€˜tests.gz’ saved [2456199/2456199]\n",
"\n"
],
"name": "stdout"
@@ -114,8 +108,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "bzdaJiZYBIHE",
- "colab_type": "text"
+ "id": "bzdaJiZYBIHE"
},
"source": [
"# Build Word Vectors\n",
@@ -132,12 +125,11 @@
"cell_type": "code",
"metadata": {
"id": "fJcn-CAH-u3K",
- "colab_type": "code",
+ "outputId": "94d7f874-e49b-4f66-e827-421e682e133c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 156
- },
- "outputId": "8ad62b92-55b7-4f2e-af55-f313402574a8"
+ }
},
"source": [
"import os\n",
@@ -159,7 +151,7 @@
" cur.execute(\"SELECT Text from sections\")\n",
"\n",
" for row in cur:\n",
- " output.write(\" \".join(row[0]) + \"\\n\")\n",
+ " output.write(\" \".join(Tokenizer.tokenize(row[0])) + \"\\n\")\n",
"\n",
" # Free database resources\n",
" db.close()\n",
@@ -181,11 +173,11 @@
"Streaming tokens to temporary file\n",
"Building 300 dimension model\n",
"Converting vectors to magnitude format\n",
- "total 9024\n",
- "-rw-r--r-- 1 root root 8065024 Aug 25 01:44 articles.sqlite\n",
- "-rw-r--r-- 1 root root 360448 Aug 25 01:57 cord19-300d.magnitude\n",
- "-rw-r--r-- 1 root root 807886 Aug 25 01:57 cord19-300d.txt\n",
- "drwxr-xr-x 1 root root 4096 Jul 30 16:30 sample_data\n"
+ "total 78948\n",
+ "-rw-r--r-- 1 root root 8065024 Aug 25 01:44 articles.sqlite\n",
+ "-rw-r--r-- 1 root root 24145920 Oct 1 13:27 cord19-300d.magnitude\n",
+ "-rw-r--r-- 1 root root 48625387 Oct 1 13:26 cord19-300d.txt\n",
+ "drwxr-xr-x 1 root root 4096 Sep 28 16:35 sample_data\n"
],
"name": "stdout"
}
@@ -194,8 +186,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "_UxcC1-JGH-d",
- "colab_type": "text"
+ "id": "_UxcC1-JGH-d"
},
"source": [
"# Build an embeddings index\n",
@@ -207,12 +198,11 @@
"cell_type": "code",
"metadata": {
"id": "5PrrxGRPGHqX",
- "colab_type": "code",
+ "outputId": "0354f102-df50-4c6e-d369-d9f18a934254",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
- },
- "outputId": "36c78d6a-1f01-4e66-d0a8-7d27d3698ab3"
+ }
},
"source": [
"import sqlite3\n",
@@ -282,8 +272,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "zHk24su3e_gb",
- "colab_type": "text"
+ "id": "zHk24su3e_gb"
},
"source": [
"# Query data\n",
@@ -295,12 +284,11 @@
"cell_type": "code",
"metadata": {
"id": "CRbDhvvDKEl-",
- "colab_type": "code",
+ "outputId": "789b06fc-7507-402f-82d7-87367468ba91",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 293
- },
- "outputId": "3f52b2b9-5307-47af-8dee-fdd9e694250b"
+ }
},
"source": [
"import pandas as pd\n",
@@ -344,34 +332,34 @@
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <td>Prevalence and Impact of Myocardial Injury in Patients Hospitalized with COVID-19 Infection</td>\n",
- " <td>2020-04-24 00:00:00</td>\n",
- " <td>http://medrxiv.org/cgi/content/short/2020.04.20.20072702v1?rss=1</td>\n",
- " <td>This risk was consistent across patients stratified by history of CVD, risk factors but no CVD, and neither CVD nor risk factors.</td>\n",
+ " <td>Management of osteoarthritis during COVID‐19 pandemic</td>\n",
+ " <td>2020-05-21 00:00:00</td>\n",
+ " <td>https://doi.org/10.1002/cpt.1910</td>\n",
+ " <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>COVID-19 and associations with frailty and multimorbidity: a prospective analysis of UK Biobank participants</td>\n",
- " <td>2020-07-23 00:00:00</td>\n",
- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32705587/</td>\n",
- " <td>The identification of risk factors for contracting COVID-19 is crucial, to inform public health policy and to facilitate the appropriate distribution of healthcare resources.</td>\n",
+ " <td>Work-related and Personal Factors Associated with Mental Well-being during COVID-19 Response: A Survey of Health Care and Other Workers</td>\n",
+ " <td>2020-06-11 00:00:00</td>\n",
+ " <td>http://medrxiv.org/cgi/content/short/2020.06.09.20126722v1?rss=1</td>\n",
+ " <td>Poor family supportive behaviors by supervisors were also associated with these outcomes [1.40 (1.21 - 1.62), 1.69 (1.48 - 1.92), 1.54 (1.44 - 1.64)].</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>Quantitative evaluation of olfactory dysfunction in hospitalized patients with Coronavirus [2] (COVID-19)</td>\n",
- " <td>2020-05-25 00:00:00</td>\n",
- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32451613/</td>\n",
- " <td>In addition, these reports included patients with minor COVID-19 symptoms and low-risk factor burden.</td>\n",
+ " <td>No evidence that androgen regulation of pulmonary TMPRSS2 explains sex-discordant COVID-19 outcomes</td>\n",
+ " <td>2020-04-21 00:00:00</td>\n",
+ " <td>https://doi.org/10.1101/2020.04.21.051201</td>\n",
+ " <td>In addition to male sex, smoking is a risk factor for COVID-19 susceptibility and poor clinical outcomes .</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>COVID-19 from the perspective of urban and rural general adult mental health services</td>\n",
- " <td>2020-05-21 00:00:00</td>\n",
- " <td>https://doi.org/10.1017/ipm.2020.62</td>\n",
- " <td>At-risk groups among staff members and service users were identified early and prioritised in service changes.</td>\n",
+ " <td>Current status of potential therapeutic candidates for the COVID-19 crisis</td>\n",
+ " <td>2020-04-22 00:00:00</td>\n",
+ " <td>https://doi.org/10.1016/j.bbi.2020.04.046</td>\n",
+ " <td>There was no difference on 28-day mortality between heparin users and nonusers.</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>Management of osteoarthritis during COVID‐19 pandemic</td>\n",
- " <td>2020-05-21 00:00:00</td>\n",
- " <td>https://doi.org/10.1002/cpt.1910</td>\n",
- " <td>Consistently, a recent report indicated diabetes as a risk factor significantly associated with COVID-19 unfavourable clinical outcomes (37) .</td>\n",
+ " <td>COVID-19: what has been learned and to be learned about the novel coronavirus disease</td>\n",
+ " <td>2020-03-15 00:00:00</td>\n",
+ " <td>https://doi.org/10.7150/ijbs.45134</td>\n",
+ " <td>• Three major risk factors for COVID-19 were sex (male), age (≄60), and severe pneumonia.</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
@@ -389,8 +377,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "XSf68I-ZfXOG",
- "colab_type": "text"
+ "id": "XSf68I-ZfXOG"
},
"source": [
"# Extracting additional columns from query results\n",
@@ -401,9 +388,7 @@
{
"cell_type": "code",
"metadata": {
- "id": "TLVOTQJchvTi",
- "colab_type": "code",
- "colab": {}
+ "id": "TLVOTQJchvTi"
},
"source": [
"%%capture\n",
@@ -419,12 +404,11 @@
"cell_type": "code",
"metadata": {
"id": "19fmKawThs6d",
- "colab_type": "code",
+ "outputId": "7676702d-7894-4e25-eb65-df9e730a2d03",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 293
- },
- "outputId": "e5c24448-97b4-444f-d47e-d268ef18aec5"
+ }
},
"source": [
"db = sqlite3.connect(\"articles.sqlite\")\n",
@@ -476,43 +460,43 @@
" </thead>\n",
" <tbody>\n",
" <tr>\n",
- " <td>Prevalence and Impact of Myocardial Injury in Patients Hospitalized with COVID-19 Infection</td>\n",
- " <td>2020-04-24 00:00:00</td>\n",
- " <td>http://medrxiv.org/cgi/content/short/2020.04.20.20072702v1?rss=1</td>\n",
- " <td>This risk was consistent across patients stratified by history of CVD, risk factors but no CVD, and neither CVD nor risk factors.</td>\n",
- " <td>neither CVD nor risk factors</td>\n",
- " <td>New York City hospitals</td>\n",
+ " <td>Management of osteoarthritis during COVID‐19 pandemic</td>\n",
+ " <td>2020-05-21 00:00:00</td>\n",
+ " <td>https://doi.org/10.1002/cpt.1910</td>\n",
+ " <td>Indeed, risk factors are sex, obesity, genetic factors and mechanical factors (3) .</td>\n",
+ " <td>sex, obesity, genetic factors and mechanical factors</td>\n",
+ " <td>None</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>COVID-19 and associations with frailty and multimorbidity: a prospective analysis of UK Biobank participants</td>\n",
- " <td>2020-07-23 00:00:00</td>\n",
- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32705587/</td>\n",
- " <td>The identification of risk factors for contracting COVID-19 is crucial, to inform public health policy and to facilitate the appropriate distribution of healthcare resources.</td>\n",
- " <td>Frailty and multimorbidity</td>\n",
- " <td>hospital settings</td>\n",
+ " <td>Work-related and Personal Factors Associated with Mental Well-being during COVID-19 Response: A Survey of Health Care and Other Workers</td>\n",
+ " <td>2020-06-11 00:00:00</td>\n",
+ " <td>http://medrxiv.org/cgi/content/short/2020.06.09.20126722v1?rss=1</td>\n",
+ " <td>Poor family supportive behaviors by supervisors were also associated with these outcomes [1.40 (1.21 - 1.62), 1.69 (1.48 - 1.92), 1.54 (1.44 - 1.64)].</td>\n",
+ " <td>Poor family supportive behaviors</td>\n",
+ " <td>None</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>Quantitative evaluation of olfactory dysfunction in hospitalized patients with Coronavirus [2] (COVID-19)</td>\n",
- " <td>2020-05-25 00:00:00</td>\n",
- " <td>https://www.ncbi.nlm.nih.gov/pubmed/32451613/</td>\n",
- " <td>In addition, these reports included patients with minor COVID-19 symptoms and low-risk factor burden.</td>\n",
- " <td>patients with minor COVID-19 symptoms and low-risk factor burden</td>\n",
- " <td>COVID-19 wards</td>\n",
+ " <td>No evidence that androgen regulation of pulmonary TMPRSS2 explains sex-discordant COVID-19 outcomes</td>\n",
+ " <td>2020-04-21 00:00:00</td>\n",
+ " <td>https://doi.org/10.1101/2020.04.21.051201</td>\n",
+ " <td>In addition to male sex, smoking is a risk factor for COVID-19 susceptibility and poor clinical outcomes .</td>\n",
+ " <td>Higher morbidity and mortality</td>\n",
+ " <td>None</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>COVID-19 from the perspective of urban and rural general adult mental health services</td>\n",
- " <td>2020-05-21 00:00:00</td>\n",
- " <td>https://doi.org/10.1017/ipm.2020.62</td>\n",
- " <td>At-risk groups among staff members and service users were identified early and prioritised in service changes.</td>\n",
- " <td>At-risk groups among staff members and service users</td>\n",
- " <td>rural regions</td>\n",
+ " <td>Current status of potential therapeutic candidates for the COVID-19 crisis</td>\n",
+ " <td>2020-04-22 00:00:00</td>\n",
+ " <td>https://doi.org/10.1016/j.bbi.2020.04.046</td>\n",
+ " <td>There was no difference on 28-day mortality between heparin users and nonusers.</td>\n",
+ " <td>elicited strong inflammatory responses are favorable or detrimental</td>\n",
+ " <td>None</td>\n",
" </tr>\n",
" <tr>\n",
- " <td>Management of osteoarthritis during COVID‐19 pandemic</td>\n",
- " <td>2020-05-21 00:00:00</td>\n",
- " <td>https://doi.org/10.1002/cpt.1910</td>\n",
- " <td>Consistently, a recent report indicated diabetes as a risk factor significantly associated with COVID-19 unfavourable clinical outcomes (37) .</td>\n",
- " <td>sex, obesity, genetic factors and mechanical factors</td>\n",
+ " <td>COVID-19: what has been learned and to be learned about the novel coronavirus disease</td>\n",
+ " <td>2020-03-15 00:00:00</td>\n",
+ " <td>https://doi.org/10.7150/ijbs.45134</td>\n",
+ " <td>• Three major risk factors for COVID-19 were sex (male), age (≄60), and severe pneumonia.</td>\n",
+ " <td>sex (male), age (≄60), and severe pneumonia</td>\n",
" <td>None</td>\n",
" </tr>\n",
" </tbody>\n",
@@ -531,8 +515,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "ColTLy--rWfR",
- "colab_type": "text"
+ "id": "ColTLy--rWfR"
},
"source": [
"In the example above, the Embeddings index is used to find the top N results for a given query. On top of that, a question-answer extractor is used to derive additional columns based on a list of questions. In this case, the \"Risk Factors\" and \"Location\" columns were pulled from the document text."
@@ -541,8 +524,7 @@
{
"cell_type": "markdown",
"metadata": {
- "id": "KWyoysauy7Pr",
- "colab_type": "text"
+ "id": "KWyoysauy7Pr"
},
"source": [
"# Next\n",