Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Update translation notebook, closes #366

davidmezzetti8afd5efb1b0e
diff --git a/examples/12_Translate_text_between_languages.ipynb b/examples/12_Translate_text_between_languages.ipynb
index 21c7a9c3..4e7b1a9e 100644
--- a/examples/12_Translate_text_between_languages.ipynb
+++ b/examples/12_Translate_text_between_languages.ipynb
@@ -3,7 +3,6 @@
"nbformat_minor": 0,
"metadata": {
"colab": {
- "name": "12 - Translate text between languages",
"provenance": [],
"collapsed_sections": []
},
@@ -46,7 +45,7 @@
"%%capture\n",
"!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline]"
],
- "execution_count": null,
+ "execution_count": 1,
"outputs": []
},
{
@@ -76,7 +75,7 @@
"# Create translation model\n",
"translate = Translation()"
],
- "execution_count": null,
+ "execution_count": 14,
"outputs": []
},
{
@@ -98,26 +97,26 @@
"height": 36
},
"id": "-K2YJJzsVtfq",
- "outputId": "8aa5efb2-12d2-444b-c876-165f9ce8dd05"
+ "outputId": "44df5404-ea14-4746-fc8b-a2e205bd9466"
},
"source": [
"translation = translate(\"This is a test translation into Spanish\", \"es\")\n",
"translation"
],
- "execution_count": null,
+ "execution_count": 16,
"outputs": [
{
"output_type": "execute_result",
"data": {
- "application/vnd.google.colaboratory.intrinsic+json": {
- "type": "string"
- },
"text/plain": [
"'Esta es una traducción de prueba al español'"
- ]
+ ],
+ "application/vnd.google.colaboratory.intrinsic+json": {
+ "type": "string"
+ }
},
"metadata": {},
- "execution_count": 8
+ "execution_count": 16
}
]
},
@@ -129,25 +128,25 @@
"height": 36
},
"id": "K_UnAZQpetM8",
- "outputId": "65011616-bd63-49eb-d02c-952c79cdcdf2"
+ "outputId": "46c9c68c-ddcf-4f55-bac3-89f25931e91b"
},
"source": [
"translate(translation, \"en\")"
],
- "execution_count": null,
+ "execution_count": 17,
"outputs": [
{
"output_type": "execute_result",
"data": {
- "application/vnd.google.colaboratory.intrinsic+json": {
- "type": "string"
- },
"text/plain": [
"'This is a test translation into Spanish'"
- ]
+ ],
+ "application/vnd.google.colaboratory.intrinsic+json": {
+ "type": "string"
+ }
},
"metadata": {},
- "execution_count": 9
+ "execution_count": 17
}
]
},
@@ -169,22 +168,24 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "outputId": "09b174cd-4387-4883-bdbe-d536486d7e5c"
+ "outputId": "47040b2f-f6e5-482a-df81-7b758f47a7d5"
},
"source": [
- "languages = [\"fr\", \"es\", \"de\", \"hi\", \"ja\"]\n",
+ "def run():\n",
+ " languages = [\"fr\", \"es\", \"de\", \"hi\", \"ja\"]\n",
+ " translations = [translate(\"The sky is blue, the stars are far\", language) for language in languages]\n",
+ " english = translate(translations, \"en\")\n",
"\n",
- "translations = [translate(\"The sky is blue, the stars are far\", language) for language in languages]\n",
+ " for x, text in enumerate(translations):\n",
+ " print(\"Original Language: %s\" % languages[x])\n",
+ " print(\"Translation: %s\" % text)\n",
+ " print(\"Back to English: %s\" % english[x])\n",
+ " print()\n",
"\n",
- "english = translate(translations, \"en\")\n",
- "\n",
- "for x, text in enumerate(translations):\n",
- " print(\"Original Language: %s\" % languages[x])\n",
- " print(\"Translation: %s\" % text)\n",
- " print(\"Back to English: %s\" % english[x])\n",
- " print()\n"
+ "# Run multiple translations\n",
+ "run()"
],
- "execution_count": null,
+ "execution_count": 18,
"outputs": [
{
"output_type": "stream",
@@ -220,8 +221,157 @@
"id": "Xn3SlVE1LYvm"
},
"source": [
- "The translation quality overall is very high. Machine translation has made giant leaps and strides the last couple of years. These models give developers a solid alternative to cloud translation services if translating on local servers is preferred."
+ "The translation quality overall is very high!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Additional model types\n",
+ "\n",
+ "The translation pipeline is flexible and supports multiple model types. The default mode for the pipeline is to scan the Hugging Face Hub for models that best match the source-target translation pair. This often produces the best quality and is usually a smaller model than a large multi-language mode.\n",
+ "\n",
+ "There is a parameter that can override this and always use the base model."
+ ],
+ "metadata": {
+ "id": "3FdS5slz60eA"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "translate = Translation(\"t5-small\", findmodels=False)\n",
+ "translate(\"translate English to French: The sky is blue, the stars are far\", None)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 36
+ },
+ "id": "REb0X2Kz60Ew",
+ "outputId": "ae13d4d0-318f-4740-f725-0029ceeabeac"
+ },
+ "execution_count": 11,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "'Le ciel est bleu, les étoiles sont loin'"
+ ],
+ "application/vnd.google.colaboratory.intrinsic+json": {
+ "type": "string"
+ }
+ },
+ "metadata": {},
+ "execution_count": 11
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Translation isn't limited to spoken languages. txtai provides a text-to-sql model that converts English text into a txtai-compatible SQL statement. "
+ ],
+ "metadata": {
+ "id": "5vSJbW7zVJeH"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "translate = Translation(\"NeuML/t5-small-txtsql\", findmodels=False)\n",
+ "translate(\"translate English to SQL: feel good story since yesterday\", None)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 36
+ },
+ "id": "d5usam_jhKaz",
+ "outputId": "c12368a0-ea26-4191-be5a-f2de70711003"
+ },
+ "execution_count": 12,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "\"select id, text, score from txtai where similar('feel good story') and entry >= date('now', '-1 day')\""
+ ],
+ "application/vnd.google.colaboratory.intrinsic+json": {
+ "type": "string"
+ }
+ },
+ "metadata": {},
+ "execution_count": 12
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Last thing we'll do is run the multiple language example only using a single large language model."
+ ],
+ "metadata": {
+ "id": "cVb7uk7TXr_v"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "translate = Translation(\"facebook/mbart-large-50-many-to-many-mmt\", findmodels=False)\n",
+ "run()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "JkOLnvKZWI95",
+ "outputId": "fef6402c-e20e-43e1-a3eb-e4580913fa7e"
+ },
+ "execution_count": 13,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Original Language: fr\n",
+ "Translation: Le ciel est bleu, les étoiles sont loin\n",
+ "Back to English: The sky is blue, the stars are far away\n",
+ "\n",
+ "Original Language: es\n",
+ "Translation: El cielo es azul, las estrellas están lejos.\n",
+ "Back to English: The sky is blue, the stars are far away.\n",
+ "\n",
+ "Original Language: de\n",
+ "Translation: Der Himmel ist blau, die Sterne sind weit\n",
+ "Back to English: The sky is blue, the stars are far.\n",
+ "\n",
+ "Original Language: hi\n",
+ "Translation: आकाश नीली है, तारे दूर हैं।\n",
+ "Back to English: The sky is blue, and the stars are far away.\n",
+ "\n",
+ "Original Language: ja\n",
+ "Translation: 空は青い、星は遠い\n",
+ "Back to English: the sky is blue, the stars are far away.\n",
+ "\n"
+ ]
+ }
]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Wrapping up\n",
+ "\n",
+ "Machine translation has made giant leaps and strides the last couple of years. These models give developers a solid, locally-hosted alternative to cloud translation services. Additionally, there are models built for low resource languages that cloud translation services don't support.\n",
+ "\n",
+ "A number of different models and configurations are supported, give it a try!"
+ ],
+ "metadata": {
+ "id": "TvWx9PS-X32c"
+ }
}
]
}
\ No newline at end of file