💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update translation notebook, closes #366
davidmezzetti8afd5efb1b0e
diff --git a/examples/12_Translate_text_between_languages.ipynb b/examples/12_Translate_text_between_languages.ipynbindex 21c7a9c3..4e7b1a9e 100644--- a/examples/12_Translate_text_between_languages.ipynb+++ b/examples/12_Translate_text_between_languages.ipynb@@ -3,7 +3,6 @@"nbformat_minor": 0,"metadata": {"colab": {- "name": "12 - Translate text between languages","provenance": [],"collapsed_sections": []},@@ -46,7 +45,7 @@"%%capture\n","!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline]"],- "execution_count": null,+ "execution_count": 1,"outputs": []},{@@ -76,7 +75,7 @@"# Create translation model\n","translate = Translation()"],- "execution_count": null,+ "execution_count": 14,"outputs": []},{@@ -98,26 +97,26 @@"height": 36},"id": "-K2YJJzsVtfq",- "outputId": "8aa5efb2-12d2-444b-c876-165f9ce8dd05"+ "outputId": "44df5404-ea14-4746-fc8b-a2e205bd9466"},"source": ["translation = translate(\"This is a test translation into Spanish\", \"es\")\n","translation"],- "execution_count": null,+ "execution_count": 16,"outputs": [{"output_type": "execute_result","data": {- "application/vnd.google.colaboratory.intrinsic+json": {- "type": "string"- },"text/plain": ["'Esta es una traducción de prueba al español'"- ]+ ],+ "application/vnd.google.colaboratory.intrinsic+json": {+ "type": "string"+ }},"metadata": {},- "execution_count": 8+ "execution_count": 16}]},@@ -129,25 +128,25 @@"height": 36},"id": "K_UnAZQpetM8",- "outputId": "65011616-bd63-49eb-d02c-952c79cdcdf2"+ "outputId": "46c9c68c-ddcf-4f55-bac3-89f25931e91b"},"source": ["translate(translation, \"en\")"],- "execution_count": null,+ "execution_count": 17,"outputs": [{"output_type": "execute_result","data": {- "application/vnd.google.colaboratory.intrinsic+json": {- "type": "string"- },"text/plain": ["'This is a test translation into Spanish'"- ]+ ],+ "application/vnd.google.colaboratory.intrinsic+json": {+ "type": "string"+ }},"metadata": {},- "execution_count": 9+ "execution_count": 17}]},@@ -169,22 +168,24 @@"colab": {"base_uri": "https://localhost:8080/"},- "outputId": "09b174cd-4387-4883-bdbe-d536486d7e5c"+ "outputId": "47040b2f-f6e5-482a-df81-7b758f47a7d5"},"source": [- "languages = [\"fr\", \"es\", \"de\", \"hi\", \"ja\"]\n",+ "def run():\n",+ " languages = [\"fr\", \"es\", \"de\", \"hi\", \"ja\"]\n",+ " translations = [translate(\"The sky is blue, the stars are far\", language) for language in languages]\n",+ " english = translate(translations, \"en\")\n","\n",- "translations = [translate(\"The sky is blue, the stars are far\", language) for language in languages]\n",+ " for x, text in enumerate(translations):\n",+ " print(\"Original Language: %s\" % languages[x])\n",+ " print(\"Translation: %s\" % text)\n",+ " print(\"Back to English: %s\" % english[x])\n",+ " print()\n","\n",- "english = translate(translations, \"en\")\n",- "\n",- "for x, text in enumerate(translations):\n",- " print(\"Original Language: %s\" % languages[x])\n",- " print(\"Translation: %s\" % text)\n",- " print(\"Back to English: %s\" % english[x])\n",- " print()\n"+ "# Run multiple translations\n",+ "run()"],- "execution_count": null,+ "execution_count": 18,"outputs": [{"output_type": "stream",@@ -220,8 +221,157 @@"id": "Xn3SlVE1LYvm"},"source": [- "The translation quality overall is very high. Machine translation has made giant leaps and strides the last couple of years. These models give developers a solid alternative to cloud translation services if translating on local servers is preferred."+ "The translation quality overall is very high!"+ ]+ },+ {+ "cell_type": "markdown",+ "source": [+ "# Additional model types\n",+ "\n",+ "The translation pipeline is flexible and supports multiple model types. The default mode for the pipeline is to scan the Hugging Face Hub for models that best match the source-target translation pair. This often produces the best quality and is usually a smaller model than a large multi-language mode.\n",+ "\n",+ "There is a parameter that can override this and always use the base model."+ ],+ "metadata": {+ "id": "3FdS5slz60eA"+ }+ },+ {+ "cell_type": "code",+ "source": [+ "translate = Translation(\"t5-small\", findmodels=False)\n",+ "translate(\"translate English to French: The sky is blue, the stars are far\", None)"+ ],+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/",+ "height": 36+ },+ "id": "REb0X2Kz60Ew",+ "outputId": "ae13d4d0-318f-4740-f725-0029ceeabeac"+ },+ "execution_count": 11,+ "outputs": [+ {+ "output_type": "execute_result",+ "data": {+ "text/plain": [+ "'Le ciel est bleu, les étoiles sont loin'"+ ],+ "application/vnd.google.colaboratory.intrinsic+json": {+ "type": "string"+ }+ },+ "metadata": {},+ "execution_count": 11+ }+ ]+ },+ {+ "cell_type": "markdown",+ "source": [+ "Translation isn't limited to spoken languages. txtai provides a text-to-sql model that converts English text into a txtai-compatible SQL statement. "+ ],+ "metadata": {+ "id": "5vSJbW7zVJeH"+ }+ },+ {+ "cell_type": "code",+ "source": [+ "translate = Translation(\"NeuML/t5-small-txtsql\", findmodels=False)\n",+ "translate(\"translate English to SQL: feel good story since yesterday\", None)"+ ],+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/",+ "height": 36+ },+ "id": "d5usam_jhKaz",+ "outputId": "c12368a0-ea26-4191-be5a-f2de70711003"+ },+ "execution_count": 12,+ "outputs": [+ {+ "output_type": "execute_result",+ "data": {+ "text/plain": [+ "\"select id, text, score from txtai where similar('feel good story') and entry >= date('now', '-1 day')\""+ ],+ "application/vnd.google.colaboratory.intrinsic+json": {+ "type": "string"+ }+ },+ "metadata": {},+ "execution_count": 12+ }+ ]+ },+ {+ "cell_type": "markdown",+ "source": [+ "Last thing we'll do is run the multiple language example only using a single large language model."+ ],+ "metadata": {+ "id": "cVb7uk7TXr_v"+ }+ },+ {+ "cell_type": "code",+ "source": [+ "translate = Translation(\"facebook/mbart-large-50-many-to-many-mmt\", findmodels=False)\n",+ "run()"+ ],+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "JkOLnvKZWI95",+ "outputId": "fef6402c-e20e-43e1-a3eb-e4580913fa7e"+ },+ "execution_count": 13,+ "outputs": [+ {+ "output_type": "stream",+ "name": "stdout",+ "text": [+ "Original Language: fr\n",+ "Translation: Le ciel est bleu, les étoiles sont loin\n",+ "Back to English: The sky is blue, the stars are far away\n",+ "\n",+ "Original Language: es\n",+ "Translation: El cielo es azul, las estrellas están lejos.\n",+ "Back to English: The sky is blue, the stars are far away.\n",+ "\n",+ "Original Language: de\n",+ "Translation: Der Himmel ist blau, die Sterne sind weit\n",+ "Back to English: The sky is blue, the stars are far.\n",+ "\n",+ "Original Language: hi\n",+ "Translation: आकाश नीली है, तारे दूर हैं।\n",+ "Back to English: The sky is blue, and the stars are far away.\n",+ "\n",+ "Original Language: ja\n",+ "Translation: 空は青い、星は遠い\n",+ "Back to English: the sky is blue, the stars are far away.\n",+ "\n"+ ]+ }]+ },+ {+ "cell_type": "markdown",+ "source": [+ "# Wrapping up\n",+ "\n",+ "Machine translation has made giant leaps and strides the last couple of years. These models give developers a solid, locally-hosted alternative to cloud translation services. Additionally, there are models built for low resource languages that cloud translation services don't support.\n",+ "\n",+ "A number of different models and configurations are supported, give it a try!"+ ],+ "metadata": {+ "id": "TvWx9PS-X32c"+ }}]}\ No newline at end of file