Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Update tabular notebook, closes #297

davidmezzettia31cc4d5ab2b
diff --git a/examples/22_Transform_tabular_data_with_composable_workflows.ipynb b/examples/22_Transform_tabular_data_with_composable_workflows.ipynb
index 29c3d842..325bcd3d 100644
--- a/examples/22_Transform_tabular_data_with_composable_workflows.ipynb
+++ b/examples/22_Transform_tabular_data_with_composable_workflows.ipynb
@@ -47,9 +47,9 @@
},
"source": [
"%%capture\n",
- "!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline,workflow]"
+ "!pip install git+https://github.com/neuml/txtai#egg=txtai[api,pipeline,workflow] sacremoses"
],
- "execution_count": null,
+ "execution_count": 66,
"outputs": []
},
{
@@ -77,7 +77,7 @@
"!head -1 metadata.csv > input.csv\n",
"!tail -10000 metadata.csv >> input.csv"
],
- "execution_count": null,
+ "execution_count": 67,
"outputs": []
},
{
@@ -98,7 +98,7 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "67f97e6b-7866-447f-b114-7db1342b5690"
+ "outputId": "a30ffeb2-1d85-467b-a0a3-134c8cbac19f"
},
"source": [
"from txtai.pipeline import Tabular\n",
@@ -113,7 +113,7 @@
"# Print 5 rows of input.csv via workflow\n",
"list(workflow([\"input.csv\"]))[:5]"
],
- "execution_count": null,
+ "execution_count": 68,
"outputs": [
{
"output_type": "execute_result",
@@ -137,7 +137,7 @@
]
},
"metadata": {},
- "execution_count": 3
+ "execution_count": 68
}
]
},
@@ -153,7 +153,11 @@
{
"cell_type": "code",
"metadata": {
- "id": "G7M34puLWeZm"
+ "id": "G7M34puLWeZm",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "0e9e771b-d53d-4c4e-8ac8-257c33992f66"
},
"source": [
"from txtai.embeddings import Embeddings\n",
@@ -169,7 +173,7 @@
" title = [text for url, text, _ in data if url == uid][0]\n",
" print(title, uid)"
],
- "execution_count": null,
+ "execution_count": 69,
"outputs": [
{
"output_type": "stream",
@@ -178,8 +182,8 @@
"Importance of diabetes management during the COVID-19 pandemic. https://doi.org/10.1080/00325481.2021.1978704; https://www.ncbi.nlm.nih.gov/pubmed/34602003/\n",
"Position Statement on How to Manage Patients with Diabetes and COVID-19 https://www.ncbi.nlm.nih.gov/pubmed/33442169/; https://doi.org/10.15605/jafes.035.01.03\n",
"Successful blood glucose management of a severe COVID-19 patient with diabetes: A case report https://www.ncbi.nlm.nih.gov/pubmed/32590779/; https://doi.org/10.1097/md.0000000000020844\n"
- ]
- }
+ ]
+ }
]
},
{
@@ -191,6 +195,82 @@
"The example searched for the term `insulin`. The top results mention diabetes and blood glucose which are a closely associated terms for diabetes."
]
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Workflow with stored content\n",
+ "\n",
+ "Next we'll re-run the same example adding in full content storage. Full content storage enables SQL queries."
+ ],
+ "metadata": {
+ "id": "BQeLRC1md6GR"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import json\n",
+ "\n",
+ "# Create tabular instance mapping input.csv fields\n",
+ "tabular = Tabular(\"url\", [\"title\"], True)\n",
+ "\n",
+ "# Create workflow\n",
+ "workflow = Workflow([Task(tabular)])\n",
+ "\n",
+ "# Embeddings with sentence-transformers backend\n",
+ "embeddings = Embeddings({\"method\": \"transformers\", \"path\": \"sentence-transformers/paraphrase-mpnet-base-v2\", \"content\": True})\n",
+ "\n",
+ "# Index subset of CORD-19 data\n",
+ "data = list(workflow([\"input.csv\"]))\n",
+ "embeddings.index(data)\n",
+ "\n",
+ "for result in embeddings.search(\"select title, abstract, authors, doi from txtai where similar('insulin')\"):\n",
+ " print(json.dumps(result, default=str, indent=2))"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Ln61lB7QeDEq",
+ "outputId": "ae364e17-36a5-480e-c614-c7e58d8b8462"
+ },
+ "execution_count": 70,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "{\n",
+ " \"title\": \"Importance of diabetes management during the COVID-19 pandemic.\",\n",
+ " \"abstract\": \"Uncontrolled diabetes and/or hyperglycemia is associated with severe COVID-19 disease and increased mortality. It is now known that poor glucose control before hospital admission can be associated with a high risk of in-hospital death. By achieving and maintaining glycemic control, primary care physicians (PCPs) play a critical role in limiting this potentially devastating outcome. Further, despite the hope that mass vaccination will help control the pandemic, genetic variants of the virus are causing surges in some countries. As such, PCPs will treat an increasing number of patients with diabetes who have symptoms of post-COVID-19 infection, or even have new-onset type 2 diabetes as a result of COVID-19 infection. However, much of the literature published focuses on the effects of COVID-19 in hospitalized patients, with few publications providing information and advice to those caring for people with diabetes in the primary care setting. This manuscript reviews the current knowledge of the risk and outcomes of individuals with diabetes who are infected with COVID-19 and provides information for PCPs on the importance of glucose control, appropriate treatment, and use of telemedicine and online prescription delivery systems to limit the potentially devastating effects of COVID-19 in people with hyperglycemia.\",\n",
+ " \"authors\": \"Pettus, Jeremy; Skolnik, Neil\",\n",
+ " \"doi\": \"10.1080/00325481.2021.1978704\"\n",
+ "}\n",
+ "{\n",
+ " \"title\": \"Position Statement on How to Manage Patients with Diabetes and COVID-19\",\n",
+ " \"abstract\": null,\n",
+ " \"authors\": null,\n",
+ " \"doi\": \"10.15605/jafes.035.01.03\"\n",
+ "}\n",
+ "{\n",
+ " \"title\": \"Successful blood glucose management of a severe COVID-19 patient with diabetes: A case report\",\n",
+ " \"abstract\": \"RATIONALE: Coronavirus disease 2019 (COVID-19) has emerged as a rapidly spreading communicable disease affecting individuals worldwide. Patients with diabetes are more vulnerable to the disease, and the mortality is higher than in those without diabetes. We reported a severe COVID-19 patient with diabetes and shared our experience with blood glucose management. PATIENT CONCERNS: A 64-year-old female diabetes patient was admitted to the intensive care unit due to productive coughing for 8 days without any obvious cause. The results of blood gas analysis indicated that the partial pressure of oxygen was 84 mm Hg with oxygen 8 L/min, and the oxygenation index was less than 200 mm Hg. In addition, postprandial blood glucose levels were abnormal (29.9 mmol/L). DIAGNOSES: The patient was diagnosed with COVID-19 (severe type) and type 2 diabetes. INTERVENTIONS: Comprehensive interventions including establishing a multidisciplinary team, closely monitoring her blood glucose level, an individualized diabetes diet, early activities, psychological care, etc, were performed to control blood glucose while actively treating COVID-19 infection. OUTCOMES: After the comprehensive measures, the patient's blood glucose level gradually became stable, and the patient was discharged after 20 days of hospitalization. LESSONS: This case indicated that the comprehensive measures performed by a multidisciplinary team achieved good treatment effects on a COVID-19 patient with diabetes. Targeted treatment and nursing methods should be performed based on patients\\u2019 actual situations in clinical practice.\",\n",
+ " \"authors\": \"Hu, Rujun; Gao, Huiming; Huang, Di; Jiang, Deyu; Chen, Fang; Fu, Bao; Yuan, Xiaoli; Li, Jin; Jiang, Zhixia\",\n",
+ " \"doi\": \"10.1097/md.0000000000020844\"\n",
+ "}\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Note how the same results are returned with additional content fields."
+ ],
+ "metadata": {
+ "id": "yTvn0O1v_eHT"
+ }
+ },
{
"cell_type": "markdown",
"metadata": {
@@ -211,7 +291,7 @@
"base_uri": "https://localhost:8080/"
},
"id": "bA8SihkeZqbJ",
- "outputId": "d990fa1a-8cf7-4026-d3d0-4602ebd7897e"
+ "outputId": "05962fa0-7e79-49a8-b6ac-3d535ace3fad"
},
"source": [
"from txtai.workflow import ServiceTask\n",
@@ -219,9 +299,9 @@
"service = ServiceTask(url=\"https://hn.algolia.com/api/v1/search\", method=\"get\", params={\"tags\": None}, batch=False, extract=\"hits\")\n",
"workflow = Workflow([service])\n",
"\n",
- "list(workflow([\"front_page\"]))[4]"
+ "list(workflow([\"front_page\"]))[0][2]"
],
- "execution_count": null,
+ "execution_count": 71,
"outputs": [
{
"output_type": "execute_result",
@@ -229,32 +309,32 @@
"text/plain": [
"{'_highlightResult': {'author': {'matchLevel': 'none',\n",
" 'matchedWords': [],\n",
- " 'value': 'carlesfe'},\n",
+ " 'value': 'cheesestain'},\n",
" 'title': {'matchLevel': 'none',\n",
" 'matchedWords': [],\n",
- " 'value': 'The reMarkable as a notebook and eBook reader'},\n",
+ " 'value': 'Ante: A low-level functional language'},\n",
" 'url': {'matchLevel': 'none',\n",
" 'matchedWords': [],\n",
- " 'value': 'https://cfenollosa.com/blog/no-notebook-is-perfect-but-the-remarkable-comes-really-close.html'}},\n",
- " '_tags': ['story', 'author_carlesfe', 'story_29304365', 'front_page'],\n",
- " 'author': 'carlesfe',\n",
+ " 'value': 'https://antelang.org/'}},\n",
+ " '_tags': ['story', 'author_cheesestain', 'story_31775216', 'front_page'],\n",
+ " 'author': 'cheesestain',\n",
" 'comment_text': None,\n",
- " 'created_at': '2021-11-22T09:07:12.000Z',\n",
- " 'created_at_i': 1637572032,\n",
- " 'num_comments': 231,\n",
- " 'objectID': '29304365',\n",
+ " 'created_at': '2022-06-17T07:39:40.000Z',\n",
+ " 'created_at_i': 1655451580,\n",
+ " 'num_comments': 109,\n",
+ " 'objectID': '31775216',\n",
" 'parent_id': None,\n",
- " 'points': 198,\n",
+ " 'points': 207,\n",
" 'story_id': None,\n",
" 'story_text': None,\n",
" 'story_title': None,\n",
" 'story_url': None,\n",
- " 'title': 'The reMarkable as a notebook and eBook reader',\n",
- " 'url': 'https://cfenollosa.com/blog/no-notebook-is-perfect-but-the-remarkable-comes-really-close.html'}"
+ " 'title': 'Ante: A low-level functional language',\n",
+ " 'url': 'https://antelang.org/'}"
]
},
"metadata": {},
- "execution_count": 5
+ "execution_count": 71
}
]
},
@@ -270,34 +350,35 @@
{
"cell_type": "code",
"metadata": {
+ "id": "YAbwhsaveKo1",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "YAbwhsaveKo1",
- "outputId": "487d7bb4-8560-4aec-896c-7a9b381b6658"
+ "outputId": "15c1b28a-3c16-4d38-c586-6b8eae244dbb"
},
"source": [
"from txtai.workflow import Task\n",
"\n",
+ "# Create tabular instance mapping input.csv fields\n",
+ "tabular = Tabular(\"url\", [\"title\"])\n",
+ "\n",
"# Recreate service applying the tabular pipeline to each result\n",
"service = ServiceTask(action=tabular, url=\"https://hn.algolia.com/api/v1/search\", method=\"get\", params={\"tags\": None}, batch=False, extract=\"hits\")\n",
"workflow = Workflow([service])\n",
"\n",
- "list(workflow([\"front_page\"]))[4]"
+ "list(workflow([\"front_page\"]))[2]"
],
- "execution_count": null,
+ "execution_count": 72,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
- "('https://cfenollosa.com/blog/no-notebook-is-perfect-but-the-remarkable-comes-really-close.html',\n",
- " 'The reMarkable as a notebook and eBook reader',\n",
- " None)"
+ "('https://antelang.org/', 'Ante: A low-level functional language', None)"
]
},
"metadata": {},
- "execution_count": 6
+ "execution_count": 72
}
]
},
@@ -317,7 +398,7 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "807a55ed-089f-42d0-92df-338f74eade83"
+ "outputId": "99e05da7-b013-4d17-d82b-c7e13e79a563"
},
"source": [
"# Embeddings with sentence-transformers backend\n",
@@ -331,15 +412,15 @@
" title = [text for url, text, _ in data if url == uid][0]\n",
" print(title, uid)"
],
- "execution_count": null,
+ "execution_count": 73,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "Rust programming for web developers, by a web developer https://codeamigo.dev/lessons/161\n",
- "Low overhead C++ interface for Apple's Metal API https://developer.apple.com/metal/cpp/\n",
- "Show HN: Run Python, Ruby, Node.js, C++, Lua in the Browser via x86 to WASM JIT https://repl.leaningtech.com/?python3\n"
+ "Bundling binary tools in Python wheels https://simonwillison.net/2022/May/23/bundling-binary-tools-in-python-wheels/\n",
+ "Ante: A low-level functional language https://antelang.org/\n",
+ "Adding a Rust compiler front end to GCC [video] https://www.youtube.com/watch?v=R8Pr21nlhig\n"
]
}
]
@@ -360,61 +441,57 @@
{
"cell_type": "code",
"metadata": {
+ "id": "K6CbS2QwltGi",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "K6CbS2QwltGi",
- "outputId": "61946750-c19c-4990-faba-a365952769e4"
+ "outputId": "2d9a0336-9680-4ca1-e2cd-68f6c5b4d1d3"
},
"source": [
"service = ServiceTask(url=\"http://export.arxiv.org/api/query\", method=\"get\", params={\"search_query\": None, \"max_results\": 25}, batch=False, extract=[\"feed\", \"entry\"])\n",
"workflow = Workflow([service])\n",
"\n",
- "list(workflow([\"all:aliens\"]))[:1]"
+ "list(workflow([\"all:aliens\"]))[0][:1]"
],
- "execution_count": null,
+ "execution_count": 74,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
- "[OrderedDict([('id', 'http://arxiv.org/abs/2102.01522v3'),\n",
- " ('updated', '2021-09-06T14:18:23Z'),\n",
- " ('published', '2021-02-01T18:27:12Z'),\n",
- " ('title',\n",
- " 'If Loud Aliens Explain Human Earliness, Quiet Aliens Are Also Rare'),\n",
- " ('summary',\n",
- " \"If life on Earth had to achieve n 'hard steps' to reach humanity's level,\\nthen the chance of this event rose as time to the n-th power. Integrating this\\nover habitable star formation and planet lifetime distributions predicts >99%\\nof advanced life appears after today, unless n<3 and max planet duration\\n<50Gyr. That is, we seem early. We offer this explanation: a deadline is set by\\n'loud' aliens who are born according to a hard steps power law, expand at a\\ncommon rate, change their volumes' appearances, and prevent advanced life like\\nus from appearing in their volumes. 'Quiet' aliens, in contrast, are much\\nharder to see. We fit this three-parameter model of loud aliens to data: 1)\\nbirth power from the number of hard steps seen in Earth history, 2) birth\\nconstant by assuming a inform distribution over our rank among loud alien birth\\ndates, and 3) expansion speed from our not seeing alien volumes in our sky. We\\nestimate that loud alien civilizations now control 40-50% of universe volume,\\neach will later control ~10^5 - 3x10^7 galaxies, and we could meet them in\\n~200Myr - 2Gyr. If loud aliens arise from quiet ones, a depressingly low\\ntransition chance (~10^-4) is required to expect that even one other quiet\\nalien civilization has ever been active in our galaxy. Which seems bad news for\\nSETI. But perhaps alien volume appearances are subtle, and their expansion\\nspeed lower, in which case we predict many long circular arcs to find in our\\nsky.\"),\n",
- " ('author',\n",
- " [OrderedDict([('name', 'Robin Hanson')]),\n",
- " OrderedDict([('name', 'Daniel Martin')]),\n",
- " OrderedDict([('name', 'Calvin McCarter')]),\n",
- " OrderedDict([('name', 'Jonathan Paulson')])]),\n",
- " ('arxiv:comment',\n",
- " OrderedDict([('@xmlns:arxiv', 'http://arxiv.org/schemas/atom'),\n",
- " ('#text', 'To appear in Astrophysical Journal')])),\n",
- " ('link',\n",
- " [OrderedDict([('@href', 'http://arxiv.org/abs/2102.01522v3'),\n",
- " ('@rel', 'alternate'),\n",
- " ('@type', 'text/html')]),\n",
- " OrderedDict([('@title', 'pdf'),\n",
- " ('@href', 'http://arxiv.org/pdf/2102.01522v3'),\n",
- " ('@rel', 'related'),\n",
- " ('@type', 'application/pdf')])]),\n",
- " ('arxiv:primary_category',\n",
- " OrderedDict([('@xmlns:arxiv', 'http://arxiv.org/schemas/atom'),\n",
- " ('@term', 'q-bio.OT'),\n",
- " ('@scheme', 'http://arxiv.org/schemas/atom')])),\n",
- " ('category',\n",
- " [OrderedDict([('@term', 'q-bio.OT'),\n",
- " ('@scheme', 'http://arxiv.org/schemas/atom')]),\n",
- " OrderedDict([('@term', 'physics.pop-ph'),\n",
- " ('@scheme',\n",
- " 'http://arxiv.org/schemas/atom')])])])]"
+ "[{'arxiv:comment': {'#text': 'To appear in Astrophysical Journal',\n",
+ " '@xmlns:arxiv': 'http://arxiv.org/schemas/atom'},\n",
+ " 'arxiv:doi': {'#text': '10.3847/1538-4357/ac2369',\n",
+ " '@xmlns:arxiv': 'http://arxiv.org/schemas/atom'},\n",
+ " 'arxiv:primary_category': {'@scheme': 'http://arxiv.org/schemas/atom',\n",
+ " '@term': 'q-bio.OT',\n",
+ " '@xmlns:arxiv': 'http://arxiv.org/schemas/atom'},\n",
+ " 'author': [{'name': 'Robin Hanson'},\n",
+ " {'name': 'Daniel Martin'},\n",
+ " {'name': 'Calvin McCarter'},\n",
+ " {'name': 'Jonathan Paulson'}],\n",
+ " 'category': [{'@scheme': 'http://arxiv.org/schemas/atom',\n",
+ " '@term': 'q-bio.OT'},\n",
+ " {'@scheme': 'http://arxiv.org/schemas/atom', '@term': 'physics.pop-ph'}],\n",
+ " 'id': 'http://arxiv.org/abs/2102.01522v3',\n",
+ " 'link': [{'@href': 'http://dx.doi.org/10.3847/1538-4357/ac2369',\n",
+ " '@rel': 'related',\n",
+ " '@title': 'doi'},\n",
+ " {'@href': 'http://arxiv.org/abs/2102.01522v3',\n",
+ " '@rel': 'alternate',\n",
+ " '@type': 'text/html'},\n",
+ " {'@href': 'http://arxiv.org/pdf/2102.01522v3',\n",
+ " '@rel': 'related',\n",
+ " '@title': 'pdf',\n",
+ " '@type': 'application/pdf'}],\n",
+ " 'published': '2021-02-01T18:27:12Z',\n",
+ " 'summary': \"If life on Earth had to achieve n 'hard steps' to reach humanity's level,\\nthen the chance of this event rose as time to the n-th power. Integrating this\\nover habitable star formation and planet lifetime distributions predicts >99%\\nof advanced life appears after today, unless n<3 and max planet duration\\n<50Gyr. That is, we seem early. We offer this explanation: a deadline is set by\\n'loud' aliens who are born according to a hard steps power law, expand at a\\ncommon rate, change their volumes' appearances, and prevent advanced life like\\nus from appearing in their volumes. 'Quiet' aliens, in contrast, are much\\nharder to see. We fit this three-parameter model of loud aliens to data: 1)\\nbirth power from the number of hard steps seen in Earth history, 2) birth\\nconstant by assuming a inform distribution over our rank among loud alien birth\\ndates, and 3) expansion speed from our not seeing alien volumes in our sky. We\\nestimate that loud alien civilizations now control 40-50% of universe volume,\\neach will later control ~10^5 - 3x10^7 galaxies, and we could meet them in\\n~200Myr - 2Gyr. If loud aliens arise from quiet ones, a depressingly low\\ntransition chance (~10^-4) is required to expect that even one other quiet\\nalien civilization has ever been active in our galaxy. Which seems bad news for\\nSETI. But perhaps alien volume appearances are subtle, and their expansion\\nspeed lower, in which case we predict many long circular arcs to find in our\\nsky.\",\n",
+ " 'title': 'If Loud Aliens Explain Human Earliness, Quiet Aliens Are Also Rare',\n",
+ " 'updated': '2021-09-06T14:18:23Z'}]"
]
},
"metadata": {},
- "execution_count": 8
+ "execution_count": 74
}
]
},
@@ -430,11 +507,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "DyIetJ7OmJjP",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "DyIetJ7OmJjP",
- "outputId": "7a28df0e-12bb-41df-fc98-1aa39b1e9002"
+ "outputId": "7e730b22-256d-4b1b-9064-2479e26d50c9"
},
"source": [
"from txtai.workflow import Task\n",
@@ -448,7 +525,7 @@
"\n",
"list(workflow([\"all:aliens\"]))[:1]"
],
- "execution_count": null,
+ "execution_count": 75,
"outputs": [
{
"output_type": "execute_result",
@@ -460,7 +537,7 @@
]
},
"metadata": {},
- "execution_count": 9
+ "execution_count": 75
}
]
},
@@ -476,11 +553,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "NX2oR5dhm_99",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "NX2oR5dhm_99",
- "outputId": "179d660f-3371-483b-d5fe-998064809535"
+ "outputId": "399c4316-bce6-443b-c383-aedd7ff9a1ec"
},
"source": [
"# Embeddings with sentence-transformers backend\n",
@@ -494,7 +571,7 @@
" title = [text for url, text, _ in data if url == uid][0]\n",
" print(title, uid)"
],
- "execution_count": null,
+ "execution_count": 76,
"outputs": [
{
"output_type": "stream",
@@ -522,11 +599,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "1eF5IJlzpNbw",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "1eF5IJlzpNbw",
- "outputId": "189c5921-8e8a-45c5-9076-94046707b3e0"
+ "outputId": "4ff0bfdc-fb01-46a6-b61a-142a370dc273"
},
"source": [
"%%writefile workflow.yml\n",
@@ -555,7 +632,7 @@
" extract: [feed, entry]\n",
" - action: upsert"
],
- "execution_count": null,
+ "execution_count": 77,
"outputs": [
{
"output_type": "stream",
@@ -584,7 +661,7 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "576e39a3-4533-4860-ec16-65f023decc6d"
+ "outputId": "7e170445-bc93-4660-b944-8b10b8b02d98"
},
"source": [
"!killall -9 uvicorn\n",
@@ -592,23 +669,20 @@
"!sleep 30\n",
"!cat api.log"
],
- "execution_count": null,
+ "execution_count": 78,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "uvicorn: no process found\n",
- "INFO: Started server process [260]\n",
+ "INFO: Started server process [754]\n",
+ "2022-06-17 15:05:58,554 [INFO] serve: Started server process [754]\n",
"INFO: Waiting for application startup.\n",
- "Downloading: 100%|██████████| 587/587 [00:00<00:00, 522kB/s]\n",
- "Downloading: 100%|██████████| 418M/418M [00:12<00:00, 34.3MB/s]\n",
- "Downloading: 100%|██████████| 1.16k/1.16k [00:00<00:00, 938kB/s]\n",
- "Downloading: 100%|██████████| 226k/226k [00:00<00:00, 1.93MB/s]\n",
- "Downloading: 100%|██████████| 455k/455k [00:00<00:00, 3.07MB/s]\n",
- "Downloading: 100%|██████████| 239/239 [00:00<00:00, 190kB/s]\n",
+ "2022-06-17 15:05:58,554 [INFO] startup: Waiting for application startup.\n",
"INFO: Application startup complete.\n",
- "INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n"
+ "2022-06-17 15:06:07,707 [INFO] startup: Application startup complete.\n",
+ "INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n",
+ "2022-06-17 15:06:07,707 [INFO] _log_started_message: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n"
]
}
]
@@ -625,23 +699,23 @@
{
"cell_type": "code",
"metadata": {
+ "id": "xt_qL6eA0SrS",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "xt_qL6eA0SrS",
- "outputId": "8edd7c2f-3a2d-4f25-8230-20f224e82e56"
+ "outputId": "3ca70a31-cf48-42b4-949f-f01aac14505c"
},
"source": [
"# Execute workflow via API call\n",
"!curl -X POST \"http://localhost:8000/workflow\" -H \"accept: application/json\" -H \"Content-Type: application/json\" -d \"{\\\"name\\\":\\\"index\\\",\\\"elements\\\":[\\\"all:aliens\\\"]}\""
],
- "execution_count": null,
+ "execution_count": 79,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "[[\"http://arxiv.org/abs/2102.01522v3\",\"If Loud Aliens Explain Human Earliness, Quiet Aliens Are Also Rare\",null],[\"http://arxiv.org/abs/cs/0306071v1\",\"AliEnFS - a Linux File System for the AliEn Grid Services\",null],[\"http://arxiv.org/abs/physics/0306103v1\",\"AliEn - EDG Interoperability in ALICE\",null],[\"http://arxiv.org/abs/2103.05559v1\",\"Oumuamua Is Not a Probe Sent to our Solar System by an Alien\\n Civilization\",null],[\"http://arxiv.org/abs/1403.3979v1\",\"Robust transitivity and density of periodic points of partially\\n hyperbolic diffeomorphisms\",null],[\"http://arxiv.org/abs/1712.09210v1\",\"Sampling alien species inside and outside protected areas: does it\\n matter?\",null],[\"http://arxiv.org/abs/cs/0306067v1\",\"The AliEn system, status and perspectives\",null],[\"http://arxiv.org/abs/0707.0011v2\",\"Calculating the probability of detecting radio signals from alien\\n civilizations\",null],[\"http://arxiv.org/abs/1805.04278v1\",\"Field Trial of Alien Wavelengths on GARR Optical Network\",null],[\"http://arxiv.org/abs/1808.00529v1\",\"Open Category Detection with PAC Guarantees\",null],[\"http://arxiv.org/abs/1206.3640v1\",\"The Study of Climate on Alien Worlds\",null],[\"http://arxiv.org/abs/1203.6805v2\",\"Aliens on Earth. Are reports of close encounters correct?\",null],[\"http://arxiv.org/abs/1604.05078v1\",\"The Imprecise Search for Habitability\",null],[\"http://arxiv.org/abs/1006.2613v1\",\"Resurgence, Stokes phenomenon and alien derivatives for level-one linear\\n differential systems\",null],[\"http://arxiv.org/abs/1307.0653v1\",\"General and alien solutions of a functional equation and of a functional\\n inequality\",null],[\"http://arxiv.org/abs/1701.02294v1\",\"Alien Calculus and non perturbative effects in Quantum Field Theory\",null],[\"http://arxiv.org/abs/1705.03394v1\",\"That is not dead which can eternal lie: the aestivation hypothesis for\\n resolving Fermi's paradox\",null],[\"http://arxiv.org/abs/1801.06180v1\",\"Are Alien Civilizations Technologically Advanced?\",null],[\"http://arxiv.org/abs/1902.05387v1\",\"Simultaneous x, y Pixel Estimation and Feature Extraction for Multiple\\n Small Objects in a Scene: A Description of the ALIEN Network\",null],[\"http://arxiv.org/abs/0711.4034v1\",\"The q-analogue of the wild fundamental group (II)\",null],[\"http://arxiv.org/abs/2111.07895v1\",\"Research Programs Arising from 'Oumuamua Considered as an Alien Craft\",null],[\"http://arxiv.org/abs/astro-ph/0501119v1\",\"Expanding advanced civilizations in the universe\",null],[\"http://arxiv.org/abs/cs/0306068v1\",\"AliEn Resource Brokers\",null],[\"http://arxiv.org/abs/hep-ph/9403231v2\",\"The Renormalization of Composite Operators in Yang-Mills Theories Using\\n General Covariant Gauge\",null],[\"http://arxiv.org/abs/1410.0501v1\",\"Alienation in Italian cities. Social network fragmentation from\\n collective data\",null]]"
+ "[[\"http://arxiv.org/abs/2102.01522v3\",\"If Loud Aliens Explain Human Earliness, Quiet Aliens Are Also Rare\",null],[\"http://arxiv.org/abs/cs/0306071v1\",\"AliEnFS - a Linux File System for the AliEn Grid Services\",null],[\"http://arxiv.org/abs/physics/0306103v1\",\"AliEn - EDG Interoperability in ALICE\",null],[\"http://arxiv.org/abs/2103.05559v1\",\"Oumuamua Is Not a Probe Sent to our Solar System by an Alien\\n Civilization\",null],[\"http://arxiv.org/abs/1403.3979v1\",\"Robust transitivity and density of periodic points of partially\\n hyperbolic diffeomorphisms\",null],[\"http://arxiv.org/abs/1712.09210v1\",\"Sampling alien species inside and outside protected areas: does it\\n matter?\",null],[\"http://arxiv.org/abs/cs/0306067v1\",\"The AliEn system, status and perspectives\",null],[\"http://arxiv.org/abs/0707.0011v2\",\"Calculating the probability of detecting radio signals from alien\\n civilizations\",null],[\"http://arxiv.org/abs/1805.04278v1\",\"Field Trial of Alien Wavelengths on GARR Optical Network\",null],[\"http://arxiv.org/abs/1808.00529v1\",\"Open Category Detection with PAC Guarantees\",null],[\"http://arxiv.org/abs/1206.3640v1\",\"The Study of Climate on Alien Worlds\",null],[\"http://arxiv.org/abs/1203.6805v2\",\"Aliens on Earth. Are reports of close encounters correct?\",null],[\"http://arxiv.org/abs/1604.05078v1\",\"The Imprecise Search for Habitability\",null],[\"http://arxiv.org/abs/1006.2613v1\",\"Resurgence, Stokes phenomenon and alien derivatives for level-one linear\\n differential systems\",null],[\"http://arxiv.org/abs/1307.0653v1\",\"General and alien solutions of a functional equation and of a functional\\n inequality\",null],[\"http://arxiv.org/abs/1705.03394v1\",\"That is not dead which can eternal lie: the aestivation hypothesis for\\n resolving Fermi's paradox\",null],[\"http://arxiv.org/abs/1701.02294v1\",\"Alien Calculus and non perturbative effects in Quantum Field Theory\",null],[\"http://arxiv.org/abs/1801.06180v1\",\"Are Alien Civilizations Technologically Advanced?\",null],[\"http://arxiv.org/abs/1902.05387v1\",\"Simultaneous x, y Pixel Estimation and Feature Extraction for Multiple\\n Small Objects in a Scene: A Description of the ALIEN Network\",null],[\"http://arxiv.org/abs/0711.4034v1\",\"The q-analogue of the wild fundamental group (II)\",null],[\"http://arxiv.org/abs/2111.07895v1\",\"Research Programs Arising from 'Oumuamua Considered as an Alien Craft\",null],[\"http://arxiv.org/abs/2112.15226v1\",\"Variations on the Resurgence of the Gamma Function\",null],[\"http://arxiv.org/abs/astro-ph/0501119v1\",\"Expanding advanced civilizations in the universe\",null],[\"http://arxiv.org/abs/cs/0306068v1\",\"AliEn Resource Brokers\",null],[\"http://arxiv.org/abs/hep-ph/9403231v2\",\"The Renormalization of Composite Operators in Yang-Mills Theories Using\\n General Covariant Gauge\",null]]"
]
}
]
@@ -660,23 +734,23 @@
{
"cell_type": "code",
"metadata": {
+ "id": "qbteIueJ1Fds",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "qbteIueJ1Fds",
- "outputId": "c24b10c2-6ba6-4957-c80f-0dbffdedb70c"
+ "outputId": "ad8ab84b-36a3-4346-d387-64321b933d25"
},
"source": [
"# Run a search\n",
"!curl -X GET \"http://localhost:8000/search?query=radio&limit=3\" -H \"accept: application/json\""
],
- "execution_count": null,
+ "execution_count": 80,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "[{\"id\":\"http://arxiv.org/abs/0707.0011v2\",\"score\":0.40350067615509033},{\"id\":\"http://arxiv.org/abs/1805.04278v1\",\"score\":0.34062114357948303},{\"id\":\"http://arxiv.org/abs/1902.05387v1\",\"score\":0.22262515127658844}]"
+ "[{\"id\":\"http://arxiv.org/abs/0707.0011v2\",\"score\":0.40350058674812317},{\"id\":\"http://arxiv.org/abs/1805.04278v1\",\"score\":0.3406212031841278},{\"id\":\"http://arxiv.org/abs/1902.05387v1\",\"score\":0.22262491285800934}]"
]
}
]
@@ -695,11 +769,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "j8rBVl17293q",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "j8rBVl17293q",
- "outputId": "a7178583-09c3-4dc2-a91e-fe6049e71802"
+ "outputId": "215dc2cf-3a57-4deb-820d-e8ac03e3e041"
},
"source": [
"%%writefile workflow.yml\n",
@@ -733,7 +807,7 @@
" args: [fr]\n",
" - action: upsert"
],
- "execution_count": null,
+ "execution_count": 81,
"outputs": [
{
"output_type": "stream",
@@ -747,11 +821,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "UQWvvgb2CwgG",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "UQWvvgb2CwgG",
- "outputId": "2366eafd-4317-4a2b-c05a-e439df56285b"
+ "outputId": "3b0c99d7-b5f6-4c81-eb6e-ac9055eaaae4"
},
"source": [
"!killall -9 uvicorn\n",
@@ -759,16 +833,20 @@
"!sleep 30\n",
"!cat api.log"
],
- "execution_count": null,
+ "execution_count": 82,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "INFO: Started server process [279]\n",
+ "INFO: Started server process [775]\n",
+ "2022-06-17 15:06:29,397 [INFO] serve: Started server process [775]\n",
"INFO: Waiting for application startup.\n",
+ "2022-06-17 15:06:29,397 [INFO] startup: Waiting for application startup.\n",
"INFO: Application startup complete.\n",
- "INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n"
+ "2022-06-17 15:06:40,198 [INFO] startup: Application startup complete.\n",
+ "INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n",
+ "2022-06-17 15:06:40,199 [INFO] _log_started_message: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n"
]
}
]
@@ -785,11 +863,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "npW3rjCw6_nx",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "npW3rjCw6_nx",
- "outputId": "faf16d5c-d604-4fd3-ab3a-3bb7672e3045"
+ "outputId": "fba0bd30-2340-4197-d68c-4fc925ce80b8"
},
"source": [
"# Execute workflow via API call\n",
@@ -798,13 +876,13 @@
"# Run a search\n",
"!curl -X GET \"http://localhost:8000/search?query=radio&limit=3\" -H \"accept: application/json\""
],
- "execution_count": null,
+ "execution_count": 83,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "[{\"id\":\"http://arxiv.org/abs/0707.0011v2\",\"score\":0.5328004956245422},{\"id\":\"http://arxiv.org/abs/0711.4034v1\",\"score\":0.2441330999135971},{\"id\":\"http://arxiv.org/abs/2102.01522v3\",\"score\":0.22881504893302917}]"
+ "[{\"id\":\"http://arxiv.org/abs/0707.0011v2\",\"score\":0.532800555229187},{\"id\":\"http://arxiv.org/abs/0711.4034v1\",\"score\":0.24413327872753143},{\"id\":\"http://arxiv.org/abs/2102.01522v3\",\"score\":0.22881504893302917}]"
]
}
]
@@ -823,11 +901,11 @@
{
"cell_type": "code",
"metadata": {
+ "id": "iXmkLDTlzPT3",
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "iXmkLDTlzPT3",
- "outputId": "0b238bef-fe04-41fe-f236-3649098ae7ae"
+ "outputId": "6b795963-2954-42e2-c42d-39c3f88c9ac3"
},
"source": [
"import yaml\n",
@@ -847,22 +925,15 @@
" text = [row[1] for row in data if row[0] == result[\"id\"]][0]\n",
" print(result[\"id\"], result[\"score\"], text)"
],
- "execution_count": null,
+ "execution_count": 84,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
- "http://arxiv.org/abs/0707.0011v2 0.5328004956245422 Calcul de la probabilité de détection des signaux radio de l'étrangercivilisations\n",
- "http://arxiv.org/abs/0711.4034v1 0.2441330999135971 Le q-analogue du groupe fondamental sauvage (II)\n",
- "http://arxiv.org/abs/2102.01522v3 0.22881504893302917 Si les étrangers louds expliquent le début de l'humanité, les étrangers tranquilles sont aussi rares\n",
- "http://arxiv.org/abs/physics/0306103v1 0.21307508647441864 Alien - EDG Interopérabilité en ALICE\n",
- "http://arxiv.org/abs/1006.2613v1 0.19786792993545532 Résurgence, phénomène Stokes et dérivés extraterrestres pour le niveau 1 linéairesystèmes différentiels\n",
- "http://arxiv.org/abs/2111.07895v1 0.1977267861366272 Programmes de recherche issus de 'Oumuamua Considéré comme un métier d'étranger\n",
- "http://arxiv.org/abs/1403.3979v1 0.1915999799966812 Transitivité robuste et densité des points périodiques en partiedifféomorphismes hyperboliques\n",
- "http://arxiv.org/abs/1805.04278v1 0.19029255211353302 Essai sur le terrain des longueurs d'onde aliens sur le réseau optique GARR\n",
- "http://arxiv.org/abs/1902.05387v1 0.17426751554012299 Simultanée x, y Pixel Estimation et Extraction de Caractéristiques pour MultiplePetits objets dans une scène : une description du réseau ALIEN\n",
- "http://arxiv.org/abs/cs/0306071v1 0.17286795377731323 AliEnFS - un système de fichiers Linux pour les services AliEn Grid\n"
+ "http://arxiv.org/abs/0707.0011v2 0.532800555229187 Calcul de la probabilité de détection des signaux radio de l'étrangercivilisations\n",
+ "http://arxiv.org/abs/0711.4034v1 0.24413327872753143 Le q-analogue du groupe fondamental sauvage (II)\n",
+ "http://arxiv.org/abs/2102.01522v3 0.22881504893302917 Si les étrangers louds expliquent le début de l'humanité, les étrangers tranquilles sont aussi rares\n"
]
}
]