DSPy: The framework for programming—not prompting—language models
Python
Add text to sql end-to-end pipeline from scratch using DSPy
Athe-kunal65bcf70c1698
diff --git a/examples/text_to_sql/text_to_sql.ipynb b/examples/text_to_sql/text_to_sql.ipynbnew file mode 100644index 000000000..611cb2811--- /dev/null+++ b/examples/text_to_sql/text_to_sql.ipynb@@ -0,0 +1,2183 @@+{+ "cells": [+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#Dependencies\n",+ "!pip install dspy-ai[chromadb] -Uqq\n",+ "!pip install termcolor -Uqq\n",+ "!pip install sqlalchemy -Uqq"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## AIM OF THE TUTORIAL\n",+ "\n",+ "* Build an end-to-end Text-to-SQL pipeline inspired from this [video](https://www.youtube.com/watch?v=L1o1VPVfbb0&pp=ygUYYWR2YW5jZWQgUkFHIGxsYW1hIGluZGV4) from Llama Index. In Llama Index, they used llama index Query Pipeline to build a Text-to-SQL pipeline. Here, we will build a Text-to-SQL pipeline based on our own dataset and from scratch. We will go from the dataset scraping, to building SQLlite database and using DSPy signatures to implementa a text-to-SQL pipeline\n",+ "\n",+ "## ABOUT THE DATASET\n",+ "* You can find the dataset [here](https://pages.stern.nyu.edu/~adamodar/New_Home_Page/datacurrent.html). The dataset has different industry based different financial metrics like WACC, tax rates, EBITDA, etc. There are multiple regions data `['US', 'Europe', 'Japan', 'AUS_NZ_CANADA', 'Emerging', 'China', 'India', 'Global']` where we have multiple tables for each region. There are nearly 250 tables with multiple columns in each table. We will build a text-to-SQL pipeline based on our own dataset and from scratch, starting from embedding tables schema and rows using ChromDB vector database."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 59,+ "metadata": {},+ "outputs": [],+ "source": [+ "#imports\n",+ "from bs4 import BeautifulSoup \n",+ "import urllib.request\n",+ "import ssl\n",+ "from dotenv import load_dotenv\n",+ "import openai\n",+ "import os\n",+ "import requests\n",+ "import warnings\n",+ "warnings.filterwarnings(\"ignore\")\n",+ "from tqdm import tqdm\n",+ "import pandas as pd\n",+ "import json\n",+ "from sqlalchemy import (\n",+ " create_engine,\n",+ " MetaData,\n",+ " Table,\n",+ " Column,\n",+ " String,\n",+ " Integer,\n",+ ")\n",+ "import re\n",+ "from sqlalchemy import inspect\n",+ "import sqlalchemy\n",+ "from sqlalchemy import text \n",+ "import dspy\n",+ "from typing import List\n",+ "from termcolor import colored"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## SCRAPING THE LINKS OF THE EXCEL FILES FROM THE WEBSITE"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {},+ "outputs": [],+ "source": [+ "ssl._create_default_https_context = ssl._create_stdlib_context\n",+ "html_link = \"https://pages.stern.nyu.edu/~adamodar/New_Home_Page/datacurrent.html\"\n",+ "\n",+ "with urllib.request.urlopen(html_link) as url:\n",+ " s = url.read()\n",+ " # I'm guessing this would output the html source code ?\n",+ " soup = BeautifulSoup(s,\"lxml\")\n",+ "\n",+ "html_table = soup.find_all(\"table\")\n",+ "req_table = html_table[1]\n",+ "hrefs_list = req_table.find_all('a')"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 42,+ "metadata": {},+ "outputs": [],+ "source": [+ "req_href = {\"US\":[],\"Europe\":[],\"Japan\":[],\"AUS_NZ_CANADA\":[],\"Emerging\":[],\"China\":[],\"India\":[],\"Global\":[]}\n",+ "\n",+ "for i in hrefs_list:\n",+ " name = i.get_text().strip()\n",+ " try:\n",+ " href_attr = i['href']\n",+ " # Only get the excel files\n",+ " if href_attr.endswith('.xls'):\n",+ " if \"US\" in name:\n",+ " req_href[\"US\"].append(href_attr)\n",+ " elif \"Europe\" in name:\n",+ " req_href[\"Europe\"].append(href_attr)\n",+ " elif \"Japan\" in name:\n",+ " req_href[\"Japan\"].append(href_attr)\n",+ " elif \"Aus\" in name:\n",+ " req_href['AUS_NZ_CANADA'].append(href_attr)\n",+ " elif \"Emerging\" in name:\n",+ " req_href['Emerging'].append(href_attr)\n",+ " elif \"China\" in name:\n",+ " req_href['China'].append(href_attr)\n",+ " elif \"India\" in name:\n",+ " req_href['India'].append(href_attr)\n",+ " elif \"Global\" in name: \n",+ " req_href['Global'].append(href_attr)\n",+ " except:\n",+ " pass"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 93,+ "metadata": {},+ "outputs": [],+ "source": [+ "#Download the excel files from the website and store it in a folder named DATA\n",+ "ssl._create_default_https_context = ssl._create_stdlib_context\n",+ "\n",+ "import os\n",+ "os.makedirs(\"DATA\",exist_ok=True)\n",+ "for country,excel_files in req_href.items():\n",+ " country_path = os.path.join(\"DATA\",country) \n",+ " os.makedirs(country_path,exist_ok=True)\n",+ " for file in excel_files:\n",+ " file_name = file.split(\"/\")[-1].split(\".\")[0]\n",+ " full_file_name = os.path.join(country_path,f\"{file_name}.xls\")\n",+ " resp = requests.get(file,verify=False)\n",+ " output = open(full_file_name, 'wb')\n",+ " output.write(resp.content)\n",+ " output.close()"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 35,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "FOR Emerging WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n",+ "FOR Europe WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n",+ "FOR Global WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n",+ "FOR AUS_NZ_CANADA WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n",+ "FOR China WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n",+ "FOR Japan WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n",+ "FOR US WE HAVE DIRECTORY LEN = 24 and ACTUAL LEN = 24\n",+ "FOR India WE HAVE DIRECTORY LEN = 29 and ACTUAL LEN = 29\n"+ ]+ }+ ],+ "source": [+ "# Sanity check\n",+ "for country in os.listdir(\"DATA\"):\n",+ " dir_len = len(os.listdir(os.path.join(\"DATA\",country)))\n",+ " country_len = len(req_href[country])\n",+ " print(f'FOR {country} WE HAVE DIRECTORY LEN = {dir_len} and ACTUAL LEN = {country_len}')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## CLEANING THE DATASET"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 73,+ "metadata": {},+ "outputs": [],+ "source": [+ "sample_excel = pd.ExcelFile(\"DATA2/US/capex.xls\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 79,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>End Game</th>\n",+ " <th>To measure how much companies are reinvesting back into their long term assets, as a prelude to forecasting expected growth.</th>\n",+ " <th>Unnamed: 2</th>\n",+ " <th>Unnamed: 3</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>2</th>\n",+ " <td>Variable</td>\n",+ " <td>How it is measured</td>\n",+ " <td>What it measures</td>\n",+ " <td>Units</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>3</th>\n",+ " <td>Capital Expenditures</td>\n",+ " <td>Sum of the capital expenditures reported on st...</td>\n",+ " <td>Gross investment in long term assets, at least...</td>\n",+ " <td>$ millions</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>4</th>\n",+ " <td>Depreciation</td>\n",+ " <td>Sum of the depreciation reported on statement ...</td>\n",+ " <td>Loss in value of assets, from use, as measured...</td>\n",+ " <td>$ millions</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>5</th>\n",+ " <td>Net Cap Ex</td>\n",+ " <td>Sum of capital expenditures on statement of ca...</td>\n",+ " <td>Net investment in long term assets, at least a...</td>\n",+ " <td>$ millions</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>6</th>\n",+ " <td>Net R&D</td>\n",+ " <td>Sum of R&D reported as expense in most recent ...</td>\n",+ " <td>Net investment in long term assets, expanded t...</td>\n",+ " <td>$ millions</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>7</th>\n",+ " <td>Acquisitions</td>\n",+ " <td>Sum of acquisitions reported on statement of c...</td>\n",+ " <td>Augments investment to include acquisition.</td>\n",+ " <td>$ millions</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " End Game \\\n",+ "0 NaN \n",+ "1 NaN \n",+ "2 Variable \n",+ "3 Capital Expenditures \n",+ "4 Depreciation \n",+ "5 Net Cap Ex \n",+ "6 Net R&D \n",+ "7 Acquisitions \n",+ "\n",+ " To measure how much companies are reinvesting back into their long term assets, as a prelude to forecasting expected growth. \\\n",+ "0 NaN \n",+ "1 NaN \n",+ "2 How it is measured \n",+ "3 Sum of the capital expenditures reported on st... \n",+ "4 Sum of the depreciation reported on statement ... \n",+ "5 Sum of capital expenditures on statement of ca... \n",+ "6 Sum of R&D reported as expense in most recent ... \n",+ "7 Sum of acquisitions reported on statement of c... \n",+ "\n",+ " Unnamed: 2 Unnamed: 3 \n",+ "0 NaN NaN \n",+ "1 NaN NaN \n",+ "2 What it measures Units \n",+ "3 Gross investment in long term assets, at least... $ millions \n",+ "4 Loss in value of assets, from use, as measured... $ millions \n",+ "5 Net investment in long term assets, at least a... $ millions \n",+ "6 Net investment in long term assets, expanded t... $ millions \n",+ "7 Augments investment to include acquisition. $ millions "+ ]+ },+ "execution_count": 79,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "sn = 'Variables & FAQ'\n",+ "sample_excel.parse(sn).head(10)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 78,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>Date updated:</th>\n",+ " <th>2024-01-05 00:00:00</th>\n",+ " <th>Unnamed: 2</th>\n",+ " <th>Unnamed: 3</th>\n",+ " <th>Unnamed: 4</th>\n",+ " <th>Unnamed: 5</th>\n",+ " <th>Unnamed: 6</th>\n",+ " <th>Unnamed: 7</th>\n",+ " <th>Unnamed: 8</th>\n",+ " <th>Unnamed: 9</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>Created by:</td>\n",+ " <td>Aswath Damodaran, [email protected]</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>What is this data?</td>\n",+ " <td>Capital Expenditures, Acquisitions and R&D and...</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>US companies</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>2</th>\n",+ " <td>Home Page:</td>\n",+ " <td>http://www.damodaran.com</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>3</th>\n",+ " <td>Data website:</td>\n",+ " <td>https://pages.stern.nyu.edu/~adamodar/New_Home...</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>4</th>\n",+ " <td>Companies in each industry:</td>\n",+ " <td>https://pages.stern.nyu.edu/~adamodar/pc/datas...</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>5</th>\n",+ " <td>Variable definitions:</td>\n",+ " <td>https://pages.stern.nyu.edu/~adamodar/New_Home...</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " <td>NaN</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>6</th>\n",+ " <td>Industry Name</td>\n",+ " <td>Number of Firms</td>\n",+ " <td>Capital Expenditures (US $ millions)</td>\n",+ " <td>Depreciation & Amort ((US $ millions)</td>\n",+ " <td>Cap Ex/Deprecn</td>\n",+ " <td>Acquisitions (US $ millions)</td>\n",+ " <td>Net R&D (US $ millions)</td>\n",+ " <td>Net Cap Ex/Sales</td>\n",+ " <td>Net Cap Ex/ EBIT (1-t)</td>\n",+ " <td>Sales/ Invested Capital (LTM)</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>7</th>\n",+ " <td>Advertising</td>\n",+ " <td>57</td>\n",+ " <td>775.729</td>\n",+ " <td>1887.338</td>\n",+ " <td>0.411018</td>\n",+ " <td>322.559</td>\n",+ " <td>75.08</td>\n",+ " <td>-0.016886</td>\n",+ " <td>-0.21812</td>\n",+ " <td>3.283403</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>8</th>\n",+ " <td>Aerospace/Defense</td>\n",+ " <td>70</td>\n",+ " <td>10982.128</td>\n",+ " <td>13311.598</td>\n",+ " <td>0.825004</td>\n",+ " <td>10344.75</td>\n",+ " <td>830.4634</td>\n",+ " <td>0.022829</td>\n",+ " <td>0.318077</td>\n",+ " <td>1.98434</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>9</th>\n",+ " <td>Air Transport</td>\n",+ " <td>25</td>\n",+ " <td>25559.725</td>\n",+ " <td>10609.948</td>\n",+ " <td>2.409034</td>\n",+ " <td>368.21</td>\n",+ " <td>73.5486</td>\n",+ " <td>0.067203</td>\n",+ " <td>1.355173</td>\n",+ " <td>1.77732</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " Date updated: \\\n",+ "0 Created by: \n",+ "1 What is this data? \n",+ "2 Home Page: \n",+ "3 Data website: \n",+ "4 Companies in each industry: \n",+ "5 Variable definitions: \n",+ "6 Industry Name \n",+ "7 Advertising \n",+ "8 Aerospace/Defense \n",+ "9 Air Transport \n",+ "\n",+ " 2024-01-05 00:00:00 \\\n",+ "0 Aswath Damodaran, [email protected] \n",+ "1 Capital Expenditures, Acquisitions and R&D and... \n",+ "2 http://www.damodaran.com \n",+ "3 https://pages.stern.nyu.edu/~adamodar/New_Home... \n",+ "4 https://pages.stern.nyu.edu/~adamodar/pc/datas... \n",+ "5 https://pages.stern.nyu.edu/~adamodar/New_Home... \n",+ "6 Number of Firms \n",+ "7 57 \n",+ "8 70 \n",+ "9 25 \n",+ "\n",+ " Unnamed: 2 \\\n",+ "0 NaN \n",+ "1 NaN \n",+ "2 NaN \n",+ "3 NaN \n",+ "4 NaN \n",+ "5 NaN \n",+ "6 Capital Expenditures (US $ millions) \n",+ "7 775.729 \n",+ "8 10982.128 \n",+ "9 25559.725 \n",+ "\n",+ " Unnamed: 3 Unnamed: 4 \\\n",+ "0 NaN NaN \n",+ "1 NaN NaN \n",+ "2 NaN NaN \n",+ "3 NaN NaN \n",+ "4 NaN NaN \n",+ "5 NaN NaN \n",+ "6 Depreciation & Amort ((US $ millions) Cap Ex/Deprecn \n",+ "7 1887.338 0.411018 \n",+ "8 13311.598 0.825004 \n",+ "9 10609.948 2.409034 \n",+ "\n",+ " Unnamed: 5 Unnamed: 6 Unnamed: 7 \\\n",+ "0 NaN NaN NaN \n",+ "1 US companies NaN NaN \n",+ "2 NaN NaN NaN \n",+ "3 NaN NaN NaN \n",+ "4 NaN NaN NaN \n",+ "5 NaN NaN NaN \n",+ "6 Acquisitions (US $ millions) Net R&D (US $ millions) Net Cap Ex/Sales \n",+ "7 322.559 75.08 -0.016886 \n",+ "8 10344.75 830.4634 0.022829 \n",+ "9 368.21 73.5486 0.067203 \n",+ "\n",+ " Unnamed: 8 Unnamed: 9 \n",+ "0 NaN NaN \n",+ "1 NaN NaN \n",+ "2 NaN NaN \n",+ "3 NaN NaN \n",+ "4 NaN NaN \n",+ "5 NaN NaN \n",+ "6 Net Cap Ex/ EBIT (1-t) Sales/ Invested Capital (LTM) \n",+ "7 -0.21812 3.283403 \n",+ "8 0.318077 1.98434 \n",+ "9 1.355173 1.77732 "+ ]+ },+ "execution_count": 78,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "sample_excel.parse(sample_excel.sheet_names[1]).head(10)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "* In the dataset above, there are two sheets. The first sheet is the variables and summary, and the second sheet is the table with the data. \n",+ "* We will clean the first sheet to get the table name and summary"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 38,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Emerging\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 86.52it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Europe\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 94.75it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Global\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 102.75it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "AUS_NZ_CANADA\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 104.23it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "China\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 103.50it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Japan\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 66.34it/s] \n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "US\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 24/24 [00:00<00:00, 102.91it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "India\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 29/29 [00:00<00:00, 100.35it/s]\n"+ ]+ }+ ],+ "source": [+ "import pandas as pd\n",+ "from tqdm import tqdm\n",+ "import json\n",+ "pd.set_option('display.max_rows', 50)\n",+ "\n",+ "def sanitize_column_name(col_name):\n",+ " # Remove special characters and replace spaces with underscores\n",+ " return re.sub(r\"\\W+\", \"_\", col_name)\n",+ "\n",+ "dir = \"DATA\"\n",+ "processed_dir = \"Processed Data\"\n",+ "all_infos_dict = []\n",+ "os.makedirs(processed_dir,exist_ok=True)\n",+ "for country in os.listdir(dir):\n",+ " print(country)\n",+ " file_name = os.path.join(dir,country)\n",+ " os.makedirs(os.path.join(processed_dir,country),exist_ok=True)\n",+ " os.makedirs(file_name,exist_ok=True)\n",+ " for excel_file in tqdm(os.listdir(file_name)):\n",+ " full_file_name = os.path.join(file_name,excel_file)\n",+ " xls = pd.ExcelFile(full_file_name)\n",+ " sns = xls.sheet_names\n",+ " for sheet_name in sns:\n",+ " if \"Var\" in sheet_name or \"var\" in sheet_name:\n",+ " info_df = xls.parse(sheet_name)\n",+ " info_df.dropna(how=\"all\",inplace=True)\n",+ " info_dict = {}\n",+ " for cols in info_df.columns:\n",+ " if \"End\" not in cols and 'Unnamed' not in cols:\n",+ " info_dict['Summary'] = cols\n",+ " info_dict['Vars'] = info_df.values[1:].tolist()\n",+ " all_infos_dict.append(info_dict)\n",+ " elif \"Industry\" in sheet_name or \"industry\" in sheet_name:\n",+ " data_df = xls.parse(sheet_name)\n",+ " try:\n",+ " data_df.dropna(axis=1,thresh=5,inplace=True)\n",+ " data_df.dropna(inplace=True)\n",+ " new_header = data_df.iloc[0] #grab the first row for the header\n",+ " except:\n",+ " print(full_file_name)\n",+ " print(data_df)\n",+ " data_df = data_df[1:] #take the data less the header row\n",+ " data_df.reset_index(inplace=True,drop=True)\n",+ " new_header = [sanitize_column_name(str(col)) for col in new_header]\n",+ " data_df.columns = new_header #set the header row as the df header\n",+ " save_name = full_file_name.split(\".\")[0].split(\"/\")[-1]\n",+ " save_file_path = os.path.join(os.path.join(processed_dir,country),save_name)\n",+ " data_df.to_csv(save_file_path+\".csv\",index=False)\n",+ " with open(save_file_path+\".json\", \"w\") as outfile: \n",+ " json.dump(info_dict, outfile)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## A SAMPLE METADATA JSON"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 196,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "{'Summary': 'Measures of accounting returns, to all claim holders (and from operations)',\n",+ " 'Vars': [['Number of firms',\n",+ " 'Number of firms in the industry grouping.',\n",+ " 'Law of large numbers?'],\n",+ " ['R&D Capitalized',\n",+ " 'My estimate of R&D capitalization, based upon a 5-year straight line amortization period, aggregated across firms in the group',\n",+ " 'Capitalized value of R&D gets added on to book equity and to invested capital'],\n",+ " ['Capitalized R&D as percent of invested capital',\n",+ " 'My R&D capitalization estimate, as a percent of invested capital including that number, based upon aggregated values across firms',\n",+ " 'Magnitude of investment in R&D, relative to investment in more traditional capital expenditures.'],\n",+ " ['R&D -LTM',\n",+ " 'Aggregated R&D expenses across the last twelve months, across companies in the group.',\n",+ " 'Spending on R&D in most recent year'],\n",+ " ['R&D: Years minus 1 to minus 5',\n",+ " 'Aggregated R&D expenses for each of the previous five years, across companies in the group.',\n",+ " 'Sepnding on R&D over last five years']]}"+ ]+ },+ "execution_count": 196,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "all_infos_dict[0]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## DATAFRAME AFTER PREPROCESSING"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 197,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>Industry_Name</th>\n",+ " <th>Number_of_Firms</th>\n",+ " <th>Capital_Expenditures_(US_$_millions)</th>\n",+ " <th>Depreciation_&_Amort_((US_$_millions)</th>\n",+ " <th>Cap_Ex/Deprecn</th>\n",+ " <th>Acquisitions_(US_$_millions)</th>\n",+ " <th>Net_R&D_(US_$_millions)</th>\n",+ " <th>Net_Cap_Ex/Sales</th>\n",+ " <th>Net_Cap_Ex/_EBIT_(1-t)</th>\n",+ " <th>Sales/_Invested_Capital_(LTM)</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>Advertising</td>\n",+ " <td>57</td>\n",+ " <td>775.729</td>\n",+ " <td>1887.338</td>\n",+ " <td>0.411018</td>\n",+ " <td>322.559</td>\n",+ " <td>75.0800</td>\n",+ " <td>-0.016886</td>\n",+ " <td>-0.218120</td>\n",+ " <td>3.283403</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>Aerospace/Defense</td>\n",+ " <td>70</td>\n",+ " <td>10982.128</td>\n",+ " <td>13311.598</td>\n",+ " <td>0.825004</td>\n",+ " <td>10344.750</td>\n",+ " <td>830.4634</td>\n",+ " <td>0.022829</td>\n",+ " <td>0.318077</td>\n",+ " <td>1.984340</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>2</th>\n",+ " <td>Air Transport</td>\n",+ " <td>25</td>\n",+ " <td>25559.725</td>\n",+ " <td>10609.948</td>\n",+ " <td>2.409034</td>\n",+ " <td>368.210</td>\n",+ " <td>73.5486</td>\n",+ " <td>0.067203</td>\n",+ " <td>1.355173</td>\n",+ " <td>1.777320</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>3</th>\n",+ " <td>Apparel</td>\n",+ " <td>38</td>\n",+ " <td>1730.981</td>\n",+ " <td>1386.480</td>\n",+ " <td>1.248472</td>\n",+ " <td>38.739</td>\n",+ " <td>0.9474</td>\n",+ " <td>0.005365</td>\n",+ " <td>0.072557</td>\n",+ " <td>1.773076</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>4</th>\n",+ " <td>Auto & Truck</td>\n",+ " <td>34</td>\n",+ " <td>29899.486</td>\n",+ " <td>18677.668</td>\n",+ " <td>1.600815</td>\n",+ " <td>193.220</td>\n",+ " <td>983.0696</td>\n",+ " <td>0.026577</td>\n",+ " <td>0.675918</td>\n",+ " <td>1.048732</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " Industry_Name Number_of_Firms Capital_Expenditures_(US_$_millions) \\\n",+ "0 Advertising 57 775.729 \n",+ "1 Aerospace/Defense 70 10982.128 \n",+ "2 Air Transport 25 25559.725 \n",+ "3 Apparel 38 1730.981 \n",+ "4 Auto & Truck 34 29899.486 \n",+ "\n",+ " Depreciation_&_Amort_((US_$_millions) Cap_Ex/Deprecn \\\n",+ "0 1887.338 0.411018 \n",+ "1 13311.598 0.825004 \n",+ "2 10609.948 2.409034 \n",+ "3 1386.480 1.248472 \n",+ "4 18677.668 1.600815 \n",+ "\n",+ " Acquisitions_(US_$_millions) Net_R&D_(US_$_millions) Net_Cap_Ex/Sales \\\n",+ "0 322.559 75.0800 -0.016886 \n",+ "1 10344.750 830.4634 0.022829 \n",+ "2 368.210 73.5486 0.067203 \n",+ "3 38.739 0.9474 0.005365 \n",+ "4 193.220 983.0696 0.026577 \n",+ "\n",+ " Net_Cap_Ex/_EBIT_(1-t) Sales/_Invested_Capital_(LTM) \n",+ "0 -0.218120 3.283403 \n",+ "1 0.318077 1.984340 \n",+ "2 1.355173 1.777320 \n",+ "3 0.072557 1.773076 \n",+ "4 0.675918 1.048732 "+ ]+ },+ "execution_count": 197,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "df = pd.read_csv(\"Processed Data/US/capex.csv\")\n",+ "df.head()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## BUILD TABLE NAMES AND METADATA\n",+ "\n",+ "* Here we use a DSPy signature given the first 10 rows of the dataframe, we generate the table name and table explanation. It will help us to dynamically select the correct table based on the query."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 199,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "',Industry_Name,Number_of_Firms,Capital_Expenditures_(US_$_millions),Depreciation_&_Amort_((US_$_millions),Cap_Ex/Deprecn,Acquisitions_(US_$_millions),Net_R&D_(US_$_millions),Net_Cap_Ex/Sales,Net_Cap_Ex/_EBIT_(1-t),Sales/_Invested_Capital_(LTM)\\n0,Advertising,57,775.729,1887.338,0.4110175283918408,322.559,75.07999999999993,-0.0168860862168837,-0.2181203933865563,3.283403041333076\\n1,Aerospace/Defense,70,10982.128,13311.597999999998,0.8250044810547916,10344.749999999998,830.4633999999933,0.0228292321038578,0.318077061754813,1.9843395804666923\\n2,Air Transport,25,25559.725,10609.948,2.4090339556800844,368.21,73.5486000000019,0.0672027546276203,1.3551729988499104,1.7773199625336142\\n3,Apparel,38,1730.9809999999998,1386.4799999999996,1.2484716692631703,38.739,0.9473999999991064,0.0053650838666078,0.0725567415076527,1.7730762352050575\\n4,Auto & Truck,34,29899.486,18677.668,1.6008147269776931,193.22,983.0695999999988,0.0265766732899557,0.6759175844836917,1.048731678622114\\n5,Auto Parts,39,3565.4210000000007,2034.471,1.7525051966825778,1061.7600000000002,454.9352000000008,0.0316936036463534,0.8398185079860373,2.004790963758861\\n6,Beverage (Alcoholic),19,2244.8160000000007,864.0629999999999,2.59797723082692,1370.0,0.0,0.0939030584900288,0.6367443614850201,0.8968231153352131\\n7,Beverage (Soft),29,7926.687000000001,4543.389,1.7446639501922463,674.6669999999999,7.705600000001141,0.0238699281532688,0.1517545513863501,1.6931990505110932\\n8,Broadcasting,22,1837.548,3387.6400000000003,0.5424271764414165,43.856,0.2399999999997817,-0.0206121524427381,-0.2104150803584717,1.1159893806472496\\n9,Brokerage & Investment Banking,27,7783.945999999999,4646.382,1.675270350134793,876.0999999999999,-189.38079999999945,0.0167661627913481,3.4298253908840737,0.2794992243703733\\n'"+ ]+ },+ "execution_count": 199,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "df.head(10).to_csv()"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {},+ "outputs": [],+ "source": [+ "load_dotenv(override=True)\n",+ "openai.api_key = os.environ['OPENAI_API_KEY']"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [],+ "source": [+ "import dspy\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo-instruct', max_tokens=250)\n",+ "dspy.settings.configure(lm=turbo)\n",+ "\n",+ "class SQLTableMetadata(dspy.Signature):\n",+ " \"\"\"Give a suitable table name and description about the given table\"\"\"\n",+ " pandas_dataframe_str = dspy.InputField(desc=\"First 10 rows of a pandas dataframe delimited by newline character\")\n",+ " table_name = dspy.OutputField(desc=\"suitable table name\")\n",+ " table_summary = dspy.OutputField(desc=\"a summary about the table\")\n",+ "\n",+ "class CoT(dspy.Module):\n",+ " def __init__(self):\n",+ " super().__init__()\n",+ " self.prog = dspy.ChainOfThought(SQLTableMetadata)\n",+ " \n",+ " def forward(self, pandas_dataframe_str):\n",+ " return self.prog(pandas_dataframe_str=pandas_dataframe_str)\n",+ "\n",+ "cot = CoT()\n",+ "\n",+ "cot(pandas_dataframe_str = df.head(10).to_csv())\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "US\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 48/48 [08:18<00:00, 10.39s/it]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "India\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 58/58 [12:00<00:00, 12.43s/it]\n"+ ]+ }+ ],+ "source": [+ "processed_dir = \"Processed Data\"\n",+ "\n",+ "for country in os.listdir(processed_dir):\n",+ " country_folder = os.path.join(processed_dir,country)\n",+ " print(f\"{country}\")\n",+ " for files in tqdm(os.listdir(country_folder)):\n",+ " # print(files)\n",+ " if files.endswith(\".csv\"):\n",+ " file_name = files.split(\".\")[0]\n",+ " csv_file_path = os.path.join(country_folder,files)\n",+ " df = pd.read_csv(csv_file_path,index_col=False)\n",+ " json_file_path = os.path.join(country_folder,f\"{file_name}.json\")\n",+ " with open(json_file_path,'r') as f:\n",+ " data = json.loads(f.read())\n",+ " if 'table_name' in data and 'table_summary' in data:\n",+ " # if data['table_name'] == \"\" or data['table_summary'] == \"\":\n",+ " if data['table_summary'] == \"\":\n",+ " pass\n",+ " else:\n",+ " continue\n",+ " table_preds = cot(pandas_dataframe_str = df.head(10).to_csv())\n",+ " data['table_name'] = table_preds.table_name\n",+ " data['table_summary'] = table_preds.table_summary\n",+ " with open(json_file_path,'w') as f:\n",+ " json.dump(data, f)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## NEXT TASKS\n",+ "1. Build database with each region for each table\n",+ "2. Embed the table summary and table SCHEMA. Also, embed the table rows\n",+ "3. Retrieval at table level and embed the rows to retrieve relevant rows from the retrieved schema of table\n",+ "4. Text-to-SQL pipeline"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## BUILD THE SQLITE DATABASE FROM THE CSV FILES\n",+ "\n",+ "It was taken from the [tutorial](https://docs.llamaindex.ai/en/stable/examples/pipeline/query_pipeline_sql/)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Function to create a sanitized column name\n",+ "def sanitize_column_name(col_name):\n",+ " # Remove special characters and replace spaces with underscores\n",+ " return re.sub(r\"\\W+\", \"_\", col_name)\n",+ "\n",+ "\n",+ "# Function to create a table from a DataFrame using SQLAlchemy\n",+ "def create_table_from_dataframe(\n",+ " df: pd.DataFrame, table_name: str, engine, metadata_obj\n",+ "):\n",+ " # Sanitize column names\n",+ " sanitized_columns = {col: sanitize_column_name(col) for col in df.columns}\n",+ " df = df.rename(columns=sanitized_columns)\n",+ "\n",+ " # Dynamically create columns based on DataFrame columns and data types\n",+ " columns = [\n",+ " Column(col, String if dtype == \"object\" else Integer)\n",+ " for col, dtype in zip(df.columns, df.dtypes)\n",+ " ]\n",+ "\n",+ " # Create a table with the defined columns\n",+ " table = Table(table_name, metadata_obj, *columns)\n",+ "\n",+ " # Create the table in the database\n",+ " metadata_obj.create_all(engine)\n",+ "\n",+ " # Insert data from DataFrame into the table\n",+ " with engine.connect() as conn:\n",+ " for _, row in df.iterrows():\n",+ " insert_stmt = table.insert().values(**row.to_dict())\n",+ " conn.execute(insert_stmt)\n",+ " conn.commit()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## DATABASE FOR US"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {},+ "outputs": [],+ "source": [+ "processed_dir = \"Processed Data\"\n",+ "def sqlalchemy_engine(region:str):\n",+ " \"\"\"Create a SQLAlchemy engine for the given region\"\"\"\n",+ " assert region in os.listdir(processed_dir), f\"{region} is not a valid region from {os.listdir(processed_dir)}\"\n",+ " # Create a SQLAlchemy database for each region\n",+ " engine = create_engine(f\"sqlite:///{region}.db\")\n",+ " metadata_obj = MetaData()\n",+ " region_path = os.path.join(processed_dir,region)\n",+ " dfs = []\n",+ " for dataframes_path in os.listdir(region_path):\n",+ " if dataframes_path.endswith(\".csv\"):\n",+ " df = pd.read_csv(os.path.join(region_path,dataframes_path),index_col=False)\n",+ " dfs.append((dataframes_path,df))\n",+ " pbar = tqdm(total=len(dfs),desc=f\"Creating tables for {region}\")\n",+ " for _, df_table_name in enumerate(dfs):\n",+ " table_name = df_table_name[0]\n",+ " table_name = table_name.split(\".\")[0]\n",+ " df = df_table_name[1]\n",+ " # print(f\"Creating table: {table_name}\")\n",+ " create_table_from_dataframe(df,table_name, engine, metadata_obj)\n",+ " # print(f\"Done creating table for: {table_name}\")\n",+ " pbar.update(1)\n",+ " return engine"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Creating tables for US: 0%| | 0/24 [00:00<?, ?it/s]"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Creating tables for US: 100%|██████████| 24/24 [00:00<00:00, 32.14it/s]\n",+ "Creating tables for India: 100%|██████████| 29/29 [00:00<00:00, 34.59it/s]\n",+ "Creating tables for China: 100%|██████████| 29/29 [00:00<00:00, 34.04it/s]\n",+ "Creating tables for Europe: 100%|██████████| 29/29 [00:01<00:00, 27.87it/s]\n",+ "Creating tables for Global: 100%|██████████| 29/29 [00:00<00:00, 32.03it/s]\n",+ "Creating tables for AUS_NZ_CANADA: 100%|██████████| 29/29 [00:00<00:00, 35.59it/s]\n",+ "Creating tables for Japan: 100%|██████████| 29/29 [00:00<00:00, 29.57it/s]\n",+ "Creating tables for Emerging: 100%|██████████| 29/29 [00:00<00:00, 32.32it/s]\n"+ ]+ }+ ],+ "source": [+ "us_engine = sqlalchemy_engine(\"US\")\n",+ "india_engine = sqlalchemy_engine(\"India\")\n",+ "china_engine = sqlalchemy_engine(\"China\")\n",+ "europe_engine = sqlalchemy_engine(\"Europe\")\n",+ "global_engine = sqlalchemy_engine(\"Global\")\n",+ "aus_nz_canada_engine = sqlalchemy_engine(\"AUS_NZ_CANADA\")\n",+ "japan_engine = sqlalchemy_engine(\"Japan\")\n",+ "emerging_engine = sqlalchemy_engine(\"Emerging\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {},+ "outputs": [],+ "source": [+ "def get_table_infos(sql_engine:sqlalchemy.engine.base.Engine,region:str):\n",+ " \"\"\"Get all the tables info in the database based on the given region\"\"\"\n",+ " inspector = inspect(sql_engine)\n",+ " table_names = inspector.get_table_names()\n",+ " table_infos_dict = {tb: [] for tb in table_names}\n",+ " for tb in table_names:\n",+ " column_dict = inspector.get_columns(tb)\n",+ " schema_str = \"\"\n",+ " primary_keys = []\n",+ " for col in column_dict:\n",+ " schema_str += f\"{col['name']} ({col['type']}), \"\n",+ " if col[\"primary_key\"] not in primary_keys:\n",+ " primary_keys.append(col[\"name\"])\n",+ " with open(os.path.join(processed_dir,region,f\"{tb}.json\")) as f:\n",+ " table_info = json.loads(f.read())\n",+ " table_infos_dict[tb] = [\n",+ " {\n",+ " \"table_info\": f\"Table {tb} has columns: {schema_str[:-2]}\",\n",+ " \"table_summary\": f'{table_info[\"Summary\"]}. {table_info[\"table_summary\"]}. ',\n",+ " }\n",+ " ]\n",+ " return table_infos_dict"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {},+ "outputs": [],+ "source": [+ "us_tb_dict = get_table_infos(us_engine,\"US\")\n",+ "india_tb_dict = get_table_infos(india_engine,\"India\")\n",+ "china_tb_dict = get_table_infos(china_engine,\"China\")\n",+ "europe_tb_dict = get_table_infos(europe_engine,\"Europe\")\n",+ "global_tb_dict = get_table_infos(global_engine,\"Global\")\n",+ "aus_nz_canada_tb_dict = get_table_infos(aus_nz_canada_engine,\"AUS_NZ_CANADA\")\n",+ "japan_tb_dict = get_table_infos(japan_engine,\"Japan\")\n",+ "emerging_tb_dict = get_table_infos(emerging_engine,\"Emerging\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "[{'table_info': 'Table DollarUS has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Average_Company_Age_years_ (INTEGER), Market_Cap_millions_ (INTEGER), Book_Equity_millions_ (INTEGER), Enteprise_Value_millions_ (INTEGER), Invested_Capital_millions_ (INTEGER), Total_Debt_including_leases_millions_ (INTEGER), Revenues_millions_ (INTEGER), Gross_Profit_millions_ (INTEGER), EBITDA_millions_ (INTEGER), EBIT_Operating_Income_millions_ (INTEGER), Net_Income_millions_ (INTEGER)',\n",+ " 'table_summary': 'To report aggregated dollar value of key operating and marker numbers, by industry group, in millions of US $.. This table contains financial information about various industries such as their names, number of firms, average company age, market cap, book equity, enterprise value, invested capital, total debt, revenues, gross profit, EBITDA, EBIT, and net income. The data is delimited by a newline character and can be used for comparison and analysis of the financial performance of different industries.'}]"+ ]+ },+ "execution_count": 7,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "us_tb_dict['DollarUS']"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## EMBEDDINGS\n",+ "\n",+ "1. Embed the table summary and table SCHEMA to get the table that the user is looking for\n",+ "2. Embed the table rows for each table, so as to get relevant rows from the retrieved table "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## EMBED THE TABLE SUMMARY AND TABLE SCHEMA"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [],+ "source": [+ "import chromadb\n",+ "import chromadb.utils.embedding_functions as embedding_functions\n",+ "from chromadb.utils.batch_utils import create_batches\n",+ "\n",+ "load_dotenv(override=True)\n",+ "emb_fn = embedding_functions.OpenAIEmbeddingFunction(\n",+ " api_key=os.environ['OPENAI_API_KEY'],\n",+ " model_name=\"text-embedding-3-small\")\n",+ "# EMBEDDING_MODEL = \"mixedbread-ai/mxbai-embed-large-v1\"\n",+ "# emb_fn = embedding_functions.HuggingFaceEmbeddingFunction(model_name=EMBEDDING_MODEL,api_key=os.environ[\"HF_API_KEY\"])\n",+ "def embed_table_info(region:str,tb_dict):\n",+ " \"\"\"Embed the table summary and table SCHEMA to get the table that the user is looking for\"\"\"\n",+ " client = chromadb.PersistentClient(path=f\"{region}_TABLE\")\n",+ "\n",+ " table_collection = client.create_collection(name=\"table\",embedding_function=emb_fn)\n",+ "\n",+ " table_docs = []\n",+ " table_metadata = []\n",+ "\n",+ "\n",+ " for table_name,table_data in tb_dict.items():\n",+ " table_docs.append(table_data[0]['table_info'] + \". \" + table_data[0]['table_summary'])\n",+ " table_metadata.append({\"table_name\":table_name,'table_metadata':table_data[0]['table_info']})\n",+ " table_ids = [f\"id{i}\" for i in range(len(table_docs))]\n",+ " assert len(table_docs) == len(table_metadata)\n",+ " # Create a batch of data to be sent to OpenAI Embedding API\n",+ " batches = create_batches(api=client,ids=table_ids, documents=table_docs, metadatas=table_metadata)\n",+ " for batch in tqdm(batches,desc=\"Embedding table info\"):\n",+ " table_collection.add(ids=batch[0],\n",+ " documents=batch[3],\n",+ " metadatas=batch[2])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## For some strange reason, the `create_batches` was not batching the below documents, hence I had to do it manually"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [],+ "source": [+ "\n",+ "def embed_rows(region:str,batch_size:int=24):\n",+ " client = chromadb.PersistentClient(path=f\"{region}_TABLE\")\n",+ "\n",+ " rows_collection = client.create_collection(name=\"rows\",embedding_function=emb_fn)\n",+ "\n",+ " rows_docs = []\n",+ " rows_metadata = []\n",+ " region_path = os.path.join(processed_dir,region)\n",+ " for df_path in os.listdir(region_path):\n",+ " df_full_path = os.path.join(region_path,df_path)\n",+ " df = pd.read_csv(df_full_path,index_col=False)\n",+ " for idx,row in df.iterrows():\n",+ " row_str = \"\"\n",+ " for rv in row.values:\n",+ " row_str+=str(rv) + \", \"\n",+ " row_str = row_str.replace('\"',\"\")\n",+ " row_str = row_str.replace(\"'\",'\"')\n",+ " rows_docs.append(row_str[:-2])\n",+ " rows_metadata.append({\"table_name\":df_path.split(\".\")[0],\"region\":region,\"index\":idx})\n",+ " row_ids = [f\"id{i}\" for i in range(len(rows_docs))]\n",+ " # print(len(rows_docs),len(rows_metadata))\n",+ " assert len(rows_docs) == len(rows_metadata) == len(row_ids)\n",+ "\n",+ " for start in tqdm(range(0,len(rows_docs),batch_size),desc=\"Embedding rows\"):\n",+ " end = min(start+batch_size,len(rows_docs))\n",+ " batch_ids = row_ids[start:end]\n",+ " batch_rows = rows_docs[start:end]\n",+ " batch_metadatas = rows_metadata[start:end]\n",+ " rows_collection.add(ids=batch_ids,\n",+ " documents=batch_rows,\n",+ " metadatas=batch_metadatas)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:00<00:00, 1.81it/s]\n",+ "Embedding rows: 100%|██████████| 5/5 [01:09<00:00, 13.91s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"US\"\n",+ "embed_table_info(region,us_tb_dict)\n",+ "embed_rows(region,500)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:00<00:00, 1.85it/s]\n",+ "Embedding rows: 100%|██████████| 3/3 [00:29<00:00, 9.79s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"India\"\n",+ "embed_table_info(region,india_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 11,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:20<00:00, 20.67s/it]\n",+ "Embedding rows: 100%|██████████| 3/3 [01:09<00:00, 23.16s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"China\"\n",+ "embed_table_info(region,china_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:00<00:00, 2.06it/s]\n",+ "Embedding rows: 100%|██████████| 3/3 [01:09<00:00, 23.11s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"Europe\"\n",+ "embed_table_info(region,europe_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 13,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:20<00:00, 20.66s/it]\n",+ "Embedding rows: 100%|██████████| 3/3 [01:13<00:00, 24.50s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"Global\"\n",+ "embed_table_info(region,global_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 15,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding rows: 100%|██████████| 3/3 [00:07<00:00, 2.54s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"Emerging\"\n",+ "embed_table_info(region,emerging_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 16,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:20<00:00, 20.68s/it]\n",+ "Embedding rows: 100%|██████████| 3/3 [00:47<00:00, 15.96s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"Japan\"\n",+ "embed_table_info(region,japan_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 17,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 0%| | 0/1 [00:00<?, ?it/s]"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Embedding table info: 100%|██████████| 1/1 [00:00<00:00, 1.91it/s]\n",+ "Embedding rows: 100%|██████████| 3/3 [00:47<00:00, 15.94s/it]\n"+ ]+ }+ ],+ "source": [+ "region = \"AUS_NZ_CANADA\"\n",+ "embed_table_info(region,aus_nz_canada_tb_dict)\n",+ "embed_rows(region,1000)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## TEXT-TO-SQL PIPELINE"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### LOAD DATABASE"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 19,+ "metadata": {},+ "outputs": [],+ "source": [+ "db_dict = {\n",+ " \"US\":us_engine,\n",+ " \"India\":india_engine,\n",+ " \"China\":china_engine,\n",+ " \"Europe\":europe_engine,\n",+ " \"Global\":global_engine,\n",+ " \"AUS_NZ_CANADA\":aus_nz_canada_engine,\n",+ " \"Japan\":japan_engine,\n",+ " \"Emerging\":emerging_engine,\n",+ "}\n",+ "\n",+ "def get_collections_db(region:str):\n",+ " # Get the database for the given region, table collection and row collection\n",+ " client = chromadb.PersistentClient(path=f\"{region}_TABLE\")\n",+ " table_collection = client.get_collection(name=\"table\",embedding_function=emb_fn)\n",+ " row_collection = client.get_collection(name=\"rows\",embedding_function=emb_fn)\n",+ " return [db_dict[region],table_collection,row_collection]"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 20,+ "metadata": {},+ "outputs": [],+ "source": [+ "db_collection_dict = {\n",+ " \"US\":get_collections_db(\"US\"),\n",+ " \"India\":get_collections_db(\"India\"),\n",+ " \"China\":get_collections_db(\"China\"),\n",+ " \"Europe\":get_collections_db(\"Europe\"),\n",+ " \"Global\":get_collections_db(\"Global\"),\n",+ " \"AUS_NZ_CANADA\":get_collections_db(\"AUS_NZ_CANADA\"),\n",+ " \"Japan\":get_collections_db(\"Japan\"),\n",+ " \"Emerging\":get_collections_db(\"Emerging\"),\n",+ "}"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 31,+ "metadata": {},+ "outputs": [],+ "source": [+ "load_dotenv(override=True)\n",+ "text_to_sql = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=1024)\n",+ "sql_to_answer = dspy.OpenAI(model='gpt-3.5-turbo',max_tokens=1024)\n",+ "\n",+ "# DSPy signature for converting text to SQL query\n",+ "class TextToSQLAnswer(dspy.Signature):\n",+ " \"\"\"Convert natural language text to SQL using suitable schema(s) from multiple schema choices\"\"\"\n",+ "\n",+ " question:str = dspy.InputField(desc=\"natural language input which will be converted to SQL\")\n",+ " relevant_table_schemas_rows:str = dspy.InputField(desc=\"Multiple possible tables which has table name and corresponding columns, along with relevant rows from the table (values in the same order as columns above)\")\n",+ " sql:str = dspy.OutputField(desc=\"Generate syntactically correct sqlite query with correct column names using suitable tables(s) and its rows.\\n Don't forget to add distinct.\\n Please rename the returned columns into suitable names.\\n DON'T OUTPUT anything else other than the sqlite query\")\n",+ "\n",+ "# DSPy signature for converting SQL query and question to natural language text\n",+ "class SQLReturnToAnswer(dspy.Signature):\n",+ " \"\"\"Answer the question using the rows from the SQL query\"\"\"\n",+ " question:str = dspy.InputField()\n",+ " sql:str = dspy.InputField(desc=\"sqlite query that generated the rows\")\n",+ " relevant_rows:str = dspy.InputField(desc=\"relevant rows to answer the question\")\n",+ " answer:str = dspy.OutputField(desc=\"answer to the question using relevant rows and the sql query\")\n",+ "\n",+ "# If there is an SQLError, then rectify the error by trying again\n",+ "class SQLRectifier(dspy.Signature):\n",+ " \"\"\"Correct the SQL query to resolve the error using the proper table names, columns and rows\"\"\" \n",+ " input_sql:str = dspy.InputField(desc=\"sqlite query that needs to be fixed\")\n",+ " error_str: str = dspy.InputField(desc=\"error that needs to be resolved\")\n",+ " relevant_table_schemas_rows:str = dspy.InputField(desc=\"Multiple possible tables which has table name and corresponding columns, along with relevant rows from the table (values in the same order as columns above)\")\n",+ " sql:str = dspy.OutputField(desc=\"corrected sqlite query to resolve the error and remove and any invalid syntax in the query.\\n Don't output anything else other than the sqlite query\")\n",+ "\n",+ "dspy.settings.configure(lm=text_to_sql)\n",+ "\n",+ "# Filter out the SQL Query\n",+ "def process_sql_str(sql_str:str):\n",+ " sql_str = sql_str.replace(\"```\",\"\")\n",+ " sql_str = sql_str.replace(\"sql\",\"\")\n",+ " sql_str = sql_str.strip()\n",+ " return sql_str"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "<p align=\"center\">\n",+ " <img src=\"https://raw.githubusercontent.com/Athe-kunal/Text-to-SQL/main/Schema.png\" alt=\"Sublime's custom image\"/>\n",+ "</p>"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 67,+ "metadata": {},+ "outputs": [],+ "source": [+ "class TextToSQLQueryModule(dspy.Module):\n",+ " \"\"\"Text to SQL to final module\"\"\"\n",+ " def __init__(self,region:str,use_cot:bool=True,max_retries:int=3):\n",+ " \"\"\"Text to Answer init module\n",+ "\n",+ " Args:\n",+ " region (str): Region for which the module will be used.\n",+ " use_cot (bool, optional): Whether to use chain of thought for sql query generation. Defaults to True.\n",+ " max_retries (int, optional): Number of max retries for SQLError. Defaults to 3.\n",+ " \"\"\"\n",+ " super().__init__()\n",+ " self.region = region\n",+ " db,table_collection,row_collection = db_collection_dict[region]\n",+ " self.table_collection = table_collection\n",+ " self.use_cot = use_cot\n",+ " self.db = db\n",+ " self.row_collection = row_collection\n",+ " if self.use_cot == True:\n",+ " self.sqlAnswer = dspy.ChainOfThought(TextToSQLAnswer)\n",+ " else:\n",+ " self.sqlAnswer = dspy.Predict(TextToSQLAnswer)\n",+ " self.final_output = dspy.Predict(SQLReturnToAnswer)\n",+ " self.max_tries = max_retries\n",+ " # Initialize the sql rectifier with CoT reasoning\n",+ " self.sql_rectifier = dspy.ChainOfThought(SQLRectifier,rationale_type=dspy.OutputField(\n",+ " prefix=\"Reasoning: Let's think step by step in order to\",\n",+ " desc=\"${produce the answer}. We ...\"\n",+ " ))\n",+ " def forward(self,question:str):\n",+ " # Embed the question with embedding function\n",+ " question_emb = emb_fn.embed_with_retries(question)\n",+ " # Retrieve the relevant tables from table schema and table summary\n",+ " docs = self.table_collection.query(\n",+ " query_embeddings = question_emb,\n",+ " n_results = 5\n",+ " )\n",+ "\n",+ " relevant_rows_schemas = \"\"\n",+ " \n",+ " existing_table_names = []\n",+ "\n",+ " for table_idx,metadata_name in enumerate(docs['metadatas'][0]):\n",+ " table_metadata = metadata_name['table_metadata']\n",+ " table_name = metadata_name['table_name']\n",+ " # If the table name is already in the list of existing table names, skip it\n",+ " if table_name in existing_table_names: \n",+ " continue\n",+ " existing_table_names.append(table_name)\n",+ " # Retrieve the relevant rows from the current table\n",+ " rows = self.row_collection.query(\n",+ " query_embeddings = question_emb,\n",+ " n_results = 5,\n",+ " # where clause to filter the rows\n",+ " where = {\"table_name\":table_name}\n",+ " )\n",+ " # Retrieve the relevant table with the schema and summary\n",+ " relevant_rows_schemas += f'Table {table_idx+1}: {table_metadata}\\n'\n",+ " for row_idx,row in enumerate(rows['documents'][0]):\n",+ " # Get the relevant rows from the current table\n",+ " relevant_rows_schemas += f'\\tRow {row_idx+1} from table {table_name}: {row}\\n'\n",+ " relevant_rows_schemas += '\\n\\n'\n",+ " print(colored(relevant_rows_schemas,\"yellow\"))\n",+ " sql_query = self.sqlAnswer(question=question,relevant_table_schemas_rows=relevant_rows_schemas)\n",+ "\n",+ " num_tries = 0\n",+ " print(sql_query)\n",+ " while num_tries <= self.max_tries:\n",+ " with self.db.connect() as conn:\n",+ " try:\n",+ " # Try executing the sql query for the database\n",+ " result = conn.execute(text(process_sql_str(sql_query.sql)))\n",+ " num_tries = self.max_tries + 1\n",+ " except Exception as error:\n",+ " # If there is an sql error, then try again with the sql rectifier\n",+ " print(colored(str(error),'red'))\n",+ " sql_query = self.sql_rectifier(input_sql=sql_query.sql,error_str=str(error),relevant_table_schemas_rows=relevant_rows_schemas)\n",+ " print(colored(sql_query.rationale,'green'))\n",+ " print()\n",+ " print(colored(sql_query.sql,'green'))\n",+ " # If all the num_retries are exhausted, then exit the program\n",+ " num_tries += 1\n",+ " if num_tries == self.max_tries:\n",+ " return sql_query,error\n",+ " # With the retrieved rows from the database, then try to answer the question with dspy context\n",+ " with dspy.context(lm=sql_to_answer):\n",+ " row_str = \"\"\n",+ " key = tuple(result.keys())\n",+ " for row in result.fetchall():\n",+ " for r,k in zip(row,key):\n",+ " row_str += f\" {k} = {r},\"\n",+ " row_str = row_str[:-1]\n",+ " row_str += \"\\n\"\n",+ " print(f\"Extracted rows: {row_str}\")\n",+ " final_answer = self.final_output(question=question,sql=sql_query.sql,relevant_rows=row_str)\n",+ " return final_answer"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 64,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\u001b[33mTable 1: Table wcdataIndia has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Acc_Rec_Sales (INTEGER), Inventory_Sales (INTEGER), Acc_Pay_Sales (INTEGER), Non_cash_WC_Sales (INTEGER)\n",+ "\tRow 1 from table wcdataIndia: Environmental & Waste Services, 14, 0.2368116129658699, 0.1284028935316297, 0.1546890424481737, 0.2020390659136147\n",+ "\tRow 2 from table wcdataIndia: Software (Internet), 6, 0.1181990107652022, 0.0352051207448356, 0.1166715158568519, 0.0613907477451265\n",+ "\tRow 3 from table wcdataIndia: Utility (Water), 1, 0.595068330362448, 0.0150920974450386, 0.3196672608437313, 0.3573380867498515\n",+ "\tRow 4 from table wcdataIndia: Packaging & Container, 86, 0.174276766908648, 0.1695062246193888, 0.1144739059878643, 0.2228221671710124\n",+ "\tRow 5 from table wcdataIndia: Software (System & Application), 73, 0.2206082612797564, 0.0069600980424173, 0.0861934214050863, 0.1104281462037313\n",+ "\n",+ "\n",+ "Table 2: Table waccIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), Cost_of_Equity (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER), Cost_of_Capital (INTEGER), Cost_of_Capital_Local_Currency_ (INTEGER)\n",+ "\tRow 1 from table waccIndia: Environmental & Waste Services, 14, 0.9313133898177656, 0.1115355757447675, 0.9526657804735228, 0.4423714612861579, 0.074755, 0.2138353277091159, 0.0523285, 0.0473342195264773, 0.1087330550239439, 0.1302618522088747\n",+ "\tRow 2 from table waccIndia: Utility (Water), 1, 1.8233554710432027, 0.1812040622884741, 0.9332942097026604, 0.2649326184290872, 0.074755, 0.3184143222506394, 0.0523285, 0.0667057902973395, 0.1726073160560074, 0.1953763901541823\n",+ "\tRow 3 from table waccIndia: Software (Internet), 6, -0.0057433681694908, 0.0383514429459627, 0.967448014179086, 0.504931470076568, 0.077424, 0.0983609410926519, 0.0541968, 0.0325519858209141, 0.0388672407841131, 0.0590394202168143\n",+ "\tRow 4 from table waccIndia: Packaging & Container, 86, 0.7511524165484432, 0.0974650037324334, 0.7447674040631929, 0.3819516759847317, 0.074755, 0.156854620625978, 0.0523285, 0.2552325959368071, 0.085944696713293, 0.1070310015038424\n",+ "\tRow 5 from table waccIndia: Software (System & Application), 73, 1.164011103524074, 0.1297092671852301, 0.9861376752921508, 0.4048728665269909, 0.074755, 0.1609424721087531, 0.0523285, 0.0138623247078493, 0.128636589864366, 0.1505518634539653\n",+ "\n",+ "\n",+ "Table 3: Table goodwillIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Goodwill_in_millions_ (INTEGER), Change_in_Goodwill_in_last_year (INTEGER), Goodwill_as_of_Total_Assets (INTEGER), Impairment_of_Goodwill_in_LTM_in_millioins (INTEGER), Impairment_as_of_Goodwill (INTEGER)\n",+ "\tRow 1 from table goodwillIndia: Environmental & Waste Services, 14, 2.6699999999999995, 0.8459999999999994, 0.0038754626605704, 0.0, 0.0\n",+ "\tRow 2 from table goodwillIndia: Packaging & Container, 86, 20.343, -0.2359999999999971, 0.0032979266492997, 0.0, 0.0\n",+ "\tRow 3 from table goodwillIndia: Software (System & Application), 73, 558.5479999999999, 161.64099999999996, 0.0955326543942697, 22.520000000000003, 0.0403188266720138\n",+ "\tRow 4 from table goodwillIndia: Software (Entertainment), 5, 58.54, -45.82999999999999, 0.0165666789015957, 0.0, 0.0\n",+ "\tRow 5 from table goodwillIndia: Semiconductor, 8, 5.369000000000001, -0.1119999999999992, 0.0478990097243286, 0.0, 0.0\n",+ "\n",+ "\n",+ "Table 4: Table EVAIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), ROE (INTEGER), Cost_of_Equity (INTEGER), _ROE_COE_ (INTEGER), BV_of_Equity (INTEGER), Equity_EVA (INTEGER), ROC (INTEGER), Cost_of_Capital (INTEGER), _ROC_WACC_ (INTEGER), BV_of_Capital (INTEGER), EVA (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER)\n",+ "\tRow 1 from table EVAIndia: Environmental & Waste Services, 14, 0.9313133898177656, 0.1433968804159445, 0.1115355757447675, 0.031861304671177, 288.49999999999994, 9.191986397634574, 0.1954191018281763, 0.1087330550239439, 0.0866860468042324, 240.89843118324, 20.88253268061651, 0.9526657804735228, 0.4423714612861579, 0.074755, 0.2138353277091159, 0.0523285, 0.0473342195264773\n",+ "\tRow 2 from table EVAIndia: Utility (Water), 1, 1.8233554710432027, 0.1604009967477059, 0.1812040622884741, -0.0208030655407682, 197.4, -4.106525137747652, 0.0801573172339537, 0.1726073160560074, -0.0924499988220537, 223.97, -20.706026236175376, 0.9332942097026604, 0.2649326184290872, 0.074755, 0.3184143222506394, 0.0523285, 0.0667057902973395\n",+ "\tRow 3 from table EVAIndia: Software (Internet), 6, -0.0057433681694908, 0.2266569688153468, 0.0383514429459627, 0.188305525869384, 10.999, 2.071172479037356, 0.2475746446157044, 0.0388672407841131, 0.2087074038315913, 11.807, 2.464208317039599, 0.967448014179086, 0.504931470076568, 0.077424, 0.0983609410926519, 0.0541968, 0.0325519858209141\n",+ "\tRow 4 from table EVAIndia: Software (System & Application), 73, 1.164011103524074, 0.1613766136661462, 0.1297092671852301, 0.031667346480916, 3600.342, 114.01327756379422, 0.2357976428869992, 0.128636589864366, 0.1071610530226331, 2544.434300346837, 272.664258972074, 0.9861376752921508, 0.4048728665269909, 0.074755, 0.1609424721087531, 0.0523285, 0.0138623247078493\n",+ "\tRow 5 from table EVAIndia: Power, 32, 1.0186133519679434, 0.155738975833232, 0.1183537027886963, 0.0373852730445356, 59792.977, 2235.37677129064, 0.0874063037779819, 0.0983790585890078, -0.0109727548110259, 135720.51367414955, -1489.2279193729348, 0.6974694002286621, 0.3130009553928686, 0.074755, 0.1828436981118144, 0.0523285, 0.3025305997713379\n",+ "\n",+ "\n",+ "Table 5: Table divfcfeIndia has columns: Industry_name (VARCHAR), Number_of_firms (INTEGER), _Dividends_ (INTEGER), _Net_Income_ (INTEGER), Payout (INTEGER), Dividends_Buybacks (INTEGER), Cash_Return_as_of_Net_Income (INTEGER), Dividends_Buybacks_Stock_Issuances (INTEGER), FCFE_before_debt_cash_flows_ (INTEGER), FCFE_after_debt_cash_flows_ (INTEGER), Net_Cash_Returned_FCFE_pre_debt_ (INTEGER), Net_Cash_Returned_FCFE_post_debt_ (INTEGER), Net_Cash_Returned_Net_Income (INTEGER), Cash_Firm_Value (INTEGER)\n",+ "\tRow 1 from table divfcfeIndia: Software (Internet), 6, 0.057, 2.493, 0.0228640192539109, 0.057, 0.0228640192539109, -0.649, 2.109000000000001, 2.9510000000000005, -0.3077287814129918, -0.2199254490003388, -0.2603289209787405, 0.0328328215881574\n",+ "\tRow 2 from table divfcfeIndia: Power, 32, 3389.4065, 9312.097, 0.3639788653404276, 3265.88, 0.3507137006841746, 2709.826, 7083.193999999999, 5267.906000000003, 0.3825711959887024, 0.5144028765889138, 0.2910006199462914, 0.0262393785332812\n",+ "\tRow 3 from table divfcfeIndia: Recreation, 12, 2.625, 67.85199999999999, 0.0386871426044921, 2.625, 0.0386871426044921, 2.127, 64.694, 54.191, 0.0328778557516925, 0.0392500599730582, 0.0313476389789541, 0.0244925013082297\n",+ "\tRow 4 from table divfcfeIndia: Advertising, 18, 2.1670000000000003, 216.254, 0.0100206238959741, 2.1670000000000003, 0.0100206238959741, -34.924, 214.557, 224.701, -0.1627725965594224, -0.15542431942893, -0.1614952787000471, 0.0217870502946718\n",+ "\tRow 5 from table divfcfeIndia: Information Services, 20, 58.482, 231.44600000000003, 0.2526809709392255, 106.471, 0.4600252326676632, 86.724, 186.4650000000001, 188.9690000000001, 0.4650953262006272, 0.4589324174864657, 0.3747051147999965, 0.0319488066345242\n",+ "\n",+ "\n",+ "\u001b[0m\n",+ "Prediction(\n",+ " rationale='produce the SQL. We need to find the effective tax rate of the software industry, which can be calculated using the tax rate from the waccIndia table.',\n",+ " sql=\"```sql\\nSELECT DISTINCT waccIndia.Industry_Name AS Industry_Name, waccIndia.Tax_Rate AS Effective_Tax_Rate\\nFROM waccIndia\\nWHERE waccIndia.Industry_Name LIKE '%Software%'\\n```\"\n",+ ")\n",+ " Industry_Name = Software (Entertainment), Effective_Tax_Rate = 0.2752644087938206\n",+ " Industry_Name = Software (Internet), Effective_Tax_Rate = 0.0983609410926519\n",+ " Industry_Name = Software (System & Application), Effective_Tax_Rate = 0.1609424721087531\n",+ "\n"+ ]+ }+ ],+ "source": [+ "tsql = TextToSQLQueryModule(\"India\")\n",+ "sq = tsql(question=\"What is the effective tax rate of software industry?\")\n",+ "# sq = tsql(question=\"What is the beta value and number of firms for all the Software industries and semiconductor industry?\")\n",+ "# sq = tsql(\"What is the debt to EBITDA ratio for software industry?\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 62,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "The effective tax rate of the software industry varies depending on the specific sector within the industry. For example, the effective tax rate for Software (Entertainment) is 27.53%, for Software (Internet) it is 9.84%, and for Software (System & Application) it is 16.09%.\n"+ ]+ }+ ],+ "source": [+ "print(sq.answer)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 68,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\u001b[33mTable 1: Table wcdataIndia has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Acc_Rec_Sales (INTEGER), Inventory_Sales (INTEGER), Acc_Pay_Sales (INTEGER), Non_cash_WC_Sales (INTEGER)\n",+ "\tRow 1 from table wcdataIndia: Environmental & Waste Services, 14, 0.2368116129658699, 0.1284028935316297, 0.1546890424481737, 0.2020390659136147\n",+ "\tRow 2 from table wcdataIndia: Software (Internet), 6, 0.1181990107652022, 0.0352051207448356, 0.1166715158568519, 0.0613907477451265\n",+ "\tRow 3 from table wcdataIndia: Utility (Water), 1, 0.595068330362448, 0.0150920974450386, 0.3196672608437313, 0.3573380867498515\n",+ "\tRow 4 from table wcdataIndia: Packaging & Container, 86, 0.174276766908648, 0.1695062246193888, 0.1144739059878643, 0.2228221671710124\n",+ "\tRow 5 from table wcdataIndia: Software (System & Application), 73, 0.2206082612797564, 0.0069600980424173, 0.0861934214050863, 0.1104281462037313\n",+ "\n",+ "\n",+ "Table 2: Table waccIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), Cost_of_Equity (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER), Cost_of_Capital (INTEGER), Cost_of_Capital_Local_Currency_ (INTEGER)\n",+ "\tRow 1 from table waccIndia: Environmental & Waste Services, 14, 0.9313133898177656, 0.1115355757447675, 0.9526657804735228, 0.4423714612861579, 0.074755, 0.2138353277091159, 0.0523285, 0.0473342195264773, 0.1087330550239439, 0.1302618522088747\n",+ "\tRow 2 from table waccIndia: Utility (Water), 1, 1.8233554710432027, 0.1812040622884741, 0.9332942097026604, 0.2649326184290872, 0.074755, 0.3184143222506394, 0.0523285, 0.0667057902973395, 0.1726073160560074, 0.1953763901541823\n",+ "\tRow 3 from table waccIndia: Software (Internet), 6, -0.0057433681694908, 0.0383514429459627, 0.967448014179086, 0.504931470076568, 0.077424, 0.0983609410926519, 0.0541968, 0.0325519858209141, 0.0388672407841131, 0.0590394202168143\n",+ "\tRow 4 from table waccIndia: Packaging & Container, 86, 0.7511524165484432, 0.0974650037324334, 0.7447674040631929, 0.3819516759847317, 0.074755, 0.156854620625978, 0.0523285, 0.2552325959368071, 0.085944696713293, 0.1070310015038424\n",+ "\tRow 5 from table waccIndia: Software (System & Application), 73, 1.164011103524074, 0.1297092671852301, 0.9861376752921508, 0.4048728665269909, 0.074755, 0.1609424721087531, 0.0523285, 0.0138623247078493, 0.128636589864366, 0.1505518634539653\n",+ "\n",+ "\n",+ "Table 3: Table goodwillIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Goodwill_in_millions_ (INTEGER), Change_in_Goodwill_in_last_year (INTEGER), Goodwill_as_of_Total_Assets (INTEGER), Impairment_of_Goodwill_in_LTM_in_millioins (INTEGER), Impairment_as_of_Goodwill (INTEGER)\n",+ "\tRow 1 from table goodwillIndia: Environmental & Waste Services, 14, 2.6699999999999995, 0.8459999999999994, 0.0038754626605704, 0.0, 0.0\n",+ "\tRow 2 from table goodwillIndia: Packaging & Container, 86, 20.343, -0.2359999999999971, 0.0032979266492997, 0.0, 0.0\n",+ "\tRow 3 from table goodwillIndia: Software (System & Application), 73, 558.5479999999999, 161.64099999999996, 0.0955326543942697, 22.520000000000003, 0.0403188266720138\n",+ "\tRow 4 from table goodwillIndia: Software (Entertainment), 5, 58.54, -45.82999999999999, 0.0165666789015957, 0.0, 0.0\n",+ "\tRow 5 from table goodwillIndia: Semiconductor, 8, 5.369000000000001, -0.1119999999999992, 0.0478990097243286, 0.0, 0.0\n",+ "\n",+ "\n",+ "Table 4: Table EVAIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), ROE (INTEGER), Cost_of_Equity (INTEGER), _ROE_COE_ (INTEGER), BV_of_Equity (INTEGER), Equity_EVA (INTEGER), ROC (INTEGER), Cost_of_Capital (INTEGER), _ROC_WACC_ (INTEGER), BV_of_Capital (INTEGER), EVA (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER)\n",+ "\tRow 1 from table EVAIndia: Environmental & Waste Services, 14, 0.9313133898177656, 0.1433968804159445, 0.1115355757447675, 0.031861304671177, 288.49999999999994, 9.191986397634574, 0.1954191018281763, 0.1087330550239439, 0.0866860468042324, 240.89843118324, 20.88253268061651, 0.9526657804735228, 0.4423714612861579, 0.074755, 0.2138353277091159, 0.0523285, 0.0473342195264773\n",+ "\tRow 2 from table EVAIndia: Utility (Water), 1, 1.8233554710432027, 0.1604009967477059, 0.1812040622884741, -0.0208030655407682, 197.4, -4.106525137747652, 0.0801573172339537, 0.1726073160560074, -0.0924499988220537, 223.97, -20.706026236175376, 0.9332942097026604, 0.2649326184290872, 0.074755, 0.3184143222506394, 0.0523285, 0.0667057902973395\n",+ "\tRow 3 from table EVAIndia: Software (Internet), 6, -0.0057433681694908, 0.2266569688153468, 0.0383514429459627, 0.188305525869384, 10.999, 2.071172479037356, 0.2475746446157044, 0.0388672407841131, 0.2087074038315913, 11.807, 2.464208317039599, 0.967448014179086, 0.504931470076568, 0.077424, 0.0983609410926519, 0.0541968, 0.0325519858209141\n",+ "\tRow 4 from table EVAIndia: Software (System & Application), 73, 1.164011103524074, 0.1613766136661462, 0.1297092671852301, 0.031667346480916, 3600.342, 114.01327756379422, 0.2357976428869992, 0.128636589864366, 0.1071610530226331, 2544.434300346837, 272.664258972074, 0.9861376752921508, 0.4048728665269909, 0.074755, 0.1609424721087531, 0.0523285, 0.0138623247078493\n",+ "\tRow 5 from table EVAIndia: Power, 32, 1.0186133519679434, 0.155738975833232, 0.1183537027886963, 0.0373852730445356, 59792.977, 2235.37677129064, 0.0874063037779819, 0.0983790585890078, -0.0109727548110259, 135720.51367414955, -1489.2279193729348, 0.6974694002286621, 0.3130009553928686, 0.074755, 0.1828436981118144, 0.0523285, 0.3025305997713379\n",+ "\n",+ "\n",+ "Table 5: Table divfcfeIndia has columns: Industry_name (VARCHAR), Number_of_firms (INTEGER), _Dividends_ (INTEGER), _Net_Income_ (INTEGER), Payout (INTEGER), Dividends_Buybacks (INTEGER), Cash_Return_as_of_Net_Income (INTEGER), Dividends_Buybacks_Stock_Issuances (INTEGER), FCFE_before_debt_cash_flows_ (INTEGER), FCFE_after_debt_cash_flows_ (INTEGER), Net_Cash_Returned_FCFE_pre_debt_ (INTEGER), Net_Cash_Returned_FCFE_post_debt_ (INTEGER), Net_Cash_Returned_Net_Income (INTEGER), Cash_Firm_Value (INTEGER)\n",+ "\tRow 1 from table divfcfeIndia: Software (Internet), 6, 0.057, 2.493, 0.0228640192539109, 0.057, 0.0228640192539109, -0.649, 2.109000000000001, 2.9510000000000005, -0.3077287814129918, -0.2199254490003388, -0.2603289209787405, 0.0328328215881574\n",+ "\tRow 2 from table divfcfeIndia: Power, 32, 3389.4065, 9312.097, 0.3639788653404276, 3265.88, 0.3507137006841746, 2709.826, 7083.193999999999, 5267.906000000003, 0.3825711959887024, 0.5144028765889138, 0.2910006199462914, 0.0262393785332812\n",+ "\tRow 3 from table divfcfeIndia: Recreation, 12, 2.625, 67.85199999999999, 0.0386871426044921, 2.625, 0.0386871426044921, 2.127, 64.694, 54.191, 0.0328778557516925, 0.0392500599730582, 0.0313476389789541, 0.0244925013082297\n",+ "\tRow 4 from table divfcfeIndia: Advertising, 18, 2.1670000000000003, 216.254, 0.0100206238959741, 2.1670000000000003, 0.0100206238959741, -34.924, 214.557, 224.701, -0.1627725965594224, -0.15542431942893, -0.1614952787000471, 0.0217870502946718\n",+ "\tRow 5 from table divfcfeIndia: Information Services, 20, 58.482, 231.44600000000003, 0.2526809709392255, 106.471, 0.4600252326676632, 86.724, 186.4650000000001, 188.9690000000001, 0.4650953262006272, 0.4589324174864657, 0.3747051147999965, 0.0319488066345242\n",+ "\n",+ "\n",+ "\u001b[0m\n",+ "Prediction(\n",+ " rationale='produce the SQL. We need to retrieve the beta value for Software industries and the semiconductor industry from the waccIndia table.',\n",+ " sql=\"```sql\\nSELECT DISTINCT waccIndia.Industry_Name AS Industry, waccIndia.Beta AS Beta, waccIndia.Number_of_Firms AS Number_of_Firms\\nFROM waccIndia\\nWHERE waccIndia.Industry_Name IN ('Software (Internet)', 'Software (System & Application)', 'Semiconductor');\\n```\"\n",+ ")\n",+ " Industry = Semiconductor, Beta = 1.611640106554315, Number_of_Firms = 8\n",+ " Industry = Software (Internet), Beta = -0.0057433681694908, Number_of_Firms = 6\n",+ " Industry = Software (System & Application), Beta = 1.164011103524074, Number_of_Firms = 73\n",+ "\n",+ "For the Software industries and semiconductor industry, the beta values and number of firms are as follows:\n",+ "\n",+ "- Industry: Semiconductor, Beta: 1.611640106554315, Number of Firms: 8\n",+ "- Industry: Software (Internet), Beta: -0.0057433681694908, Number of Firms: 6\n",+ "- Industry: Software (System & Application), Beta: 1.164011103524074, Number of Firms: 73\n"+ ]+ }+ ],+ "source": [+ "sq = tsql(question=\"What is the beta value and number of firms for all the Software industries and semiconductor industry?\")\n",+ "print(sq.answer)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 69,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\u001b[33mTable 1: Table wcdataIndia has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Acc_Rec_Sales (INTEGER), Inventory_Sales (INTEGER), Acc_Pay_Sales (INTEGER), Non_cash_WC_Sales (INTEGER)\n",+ "\tRow 1 from table wcdataIndia: Environmental & Waste Services, 14, 0.2368116129658699, 0.1284028935316297, 0.1546890424481737, 0.2020390659136147\n",+ "\tRow 2 from table wcdataIndia: Software (Internet), 6, 0.1181990107652022, 0.0352051207448356, 0.1166715158568519, 0.0613907477451265\n",+ "\tRow 3 from table wcdataIndia: Utility (Water), 1, 0.595068330362448, 0.0150920974450386, 0.3196672608437313, 0.3573380867498515\n",+ "\tRow 4 from table wcdataIndia: Packaging & Container, 86, 0.174276766908648, 0.1695062246193888, 0.1144739059878643, 0.2228221671710124\n",+ "\tRow 5 from table wcdataIndia: Software (System & Application), 73, 0.2206082612797564, 0.0069600980424173, 0.0861934214050863, 0.1104281462037313\n",+ "\n",+ "\n",+ "Table 2: Table waccIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), Cost_of_Equity (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER), Cost_of_Capital (INTEGER), Cost_of_Capital_Local_Currency_ (INTEGER)\n",+ "\tRow 1 from table waccIndia: Environmental & Waste Services, 14, 0.9313133898177656, 0.1115355757447675, 0.9526657804735228, 0.4423714612861579, 0.074755, 0.2138353277091159, 0.0523285, 0.0473342195264773, 0.1087330550239439, 0.1302618522088747\n",+ "\tRow 2 from table waccIndia: Utility (Water), 1, 1.8233554710432027, 0.1812040622884741, 0.9332942097026604, 0.2649326184290872, 0.074755, 0.3184143222506394, 0.0523285, 0.0667057902973395, 0.1726073160560074, 0.1953763901541823\n",+ "\tRow 3 from table waccIndia: Software (Internet), 6, -0.0057433681694908, 0.0383514429459627, 0.967448014179086, 0.504931470076568, 0.077424, 0.0983609410926519, 0.0541968, 0.0325519858209141, 0.0388672407841131, 0.0590394202168143\n",+ "\tRow 4 from table waccIndia: Packaging & Container, 86, 0.7511524165484432, 0.0974650037324334, 0.7447674040631929, 0.3819516759847317, 0.074755, 0.156854620625978, 0.0523285, 0.2552325959368071, 0.085944696713293, 0.1070310015038424\n",+ "\tRow 5 from table waccIndia: Software (System & Application), 73, 1.164011103524074, 0.1297092671852301, 0.9861376752921508, 0.4048728665269909, 0.074755, 0.1609424721087531, 0.0523285, 0.0138623247078493, 0.128636589864366, 0.1505518634539653\n",+ "\n",+ "\n",+ "Table 3: Table goodwillIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Goodwill_in_millions_ (INTEGER), Change_in_Goodwill_in_last_year (INTEGER), Goodwill_as_of_Total_Assets (INTEGER), Impairment_of_Goodwill_in_LTM_in_millioins (INTEGER), Impairment_as_of_Goodwill (INTEGER)\n",+ "\tRow 1 from table goodwillIndia: Environmental & Waste Services, 14, 2.6699999999999995, 0.8459999999999994, 0.0038754626605704, 0.0, 0.0\n",+ "\tRow 2 from table goodwillIndia: Packaging & Container, 86, 20.343, -0.2359999999999971, 0.0032979266492997, 0.0, 0.0\n",+ "\tRow 3 from table goodwillIndia: Software (System & Application), 73, 558.5479999999999, 161.64099999999996, 0.0955326543942697, 22.520000000000003, 0.0403188266720138\n",+ "\tRow 4 from table goodwillIndia: Software (Entertainment), 5, 58.54, -45.82999999999999, 0.0165666789015957, 0.0, 0.0\n",+ "\tRow 5 from table goodwillIndia: Semiconductor, 8, 5.369000000000001, -0.1119999999999992, 0.0478990097243286, 0.0, 0.0\n",+ "\n",+ "\n",+ "Table 4: Table EVAIndia has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), ROE (INTEGER), Cost_of_Equity (INTEGER), _ROE_COE_ (INTEGER), BV_of_Equity (INTEGER), Equity_EVA (INTEGER), ROC (INTEGER), Cost_of_Capital (INTEGER), _ROC_WACC_ (INTEGER), BV_of_Capital (INTEGER), EVA (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER)\n",+ "\tRow 1 from table EVAIndia: Environmental & Waste Services, 14, 0.9313133898177656, 0.1433968804159445, 0.1115355757447675, 0.031861304671177, 288.49999999999994, 9.191986397634574, 0.1954191018281763, 0.1087330550239439, 0.0866860468042324, 240.89843118324, 20.88253268061651, 0.9526657804735228, 0.4423714612861579, 0.074755, 0.2138353277091159, 0.0523285, 0.0473342195264773\n",+ "\tRow 2 from table EVAIndia: Utility (Water), 1, 1.8233554710432027, 0.1604009967477059, 0.1812040622884741, -0.0208030655407682, 197.4, -4.106525137747652, 0.0801573172339537, 0.1726073160560074, -0.0924499988220537, 223.97, -20.706026236175376, 0.9332942097026604, 0.2649326184290872, 0.074755, 0.3184143222506394, 0.0523285, 0.0667057902973395\n",+ "\tRow 3 from table EVAIndia: Software (Internet), 6, -0.0057433681694908, 0.2266569688153468, 0.0383514429459627, 0.188305525869384, 10.999, 2.071172479037356, 0.2475746446157044, 0.0388672407841131, 0.2087074038315913, 11.807, 2.464208317039599, 0.967448014179086, 0.504931470076568, 0.077424, 0.0983609410926519, 0.0541968, 0.0325519858209141\n",+ "\tRow 4 from table EVAIndia: Software (System & Application), 73, 1.164011103524074, 0.1613766136661462, 0.1297092671852301, 0.031667346480916, 3600.342, 114.01327756379422, 0.2357976428869992, 0.128636589864366, 0.1071610530226331, 2544.434300346837, 272.664258972074, 0.9861376752921508, 0.4048728665269909, 0.074755, 0.1609424721087531, 0.0523285, 0.0138623247078493\n",+ "\tRow 5 from table EVAIndia: Power, 32, 1.0186133519679434, 0.155738975833232, 0.1183537027886963, 0.0373852730445356, 59792.977, 2235.37677129064, 0.0874063037779819, 0.0983790585890078, -0.0109727548110259, 135720.51367414955, -1489.2279193729348, 0.6974694002286621, 0.3130009553928686, 0.074755, 0.1828436981118144, 0.0523285, 0.3025305997713379\n",+ "\n",+ "\n",+ "Table 5: Table divfcfeIndia has columns: Industry_name (VARCHAR), Number_of_firms (INTEGER), _Dividends_ (INTEGER), _Net_Income_ (INTEGER), Payout (INTEGER), Dividends_Buybacks (INTEGER), Cash_Return_as_of_Net_Income (INTEGER), Dividends_Buybacks_Stock_Issuances (INTEGER), FCFE_before_debt_cash_flows_ (INTEGER), FCFE_after_debt_cash_flows_ (INTEGER), Net_Cash_Returned_FCFE_pre_debt_ (INTEGER), Net_Cash_Returned_FCFE_post_debt_ (INTEGER), Net_Cash_Returned_Net_Income (INTEGER), Cash_Firm_Value (INTEGER)\n",+ "\tRow 1 from table divfcfeIndia: Software (Internet), 6, 0.057, 2.493, 0.0228640192539109, 0.057, 0.0228640192539109, -0.649, 2.109000000000001, 2.9510000000000005, -0.3077287814129918, -0.2199254490003388, -0.2603289209787405, 0.0328328215881574\n",+ "\tRow 2 from table divfcfeIndia: Power, 32, 3389.4065, 9312.097, 0.3639788653404276, 3265.88, 0.3507137006841746, 2709.826, 7083.193999999999, 5267.906000000003, 0.3825711959887024, 0.5144028765889138, 0.2910006199462914, 0.0262393785332812\n",+ "\tRow 3 from table divfcfeIndia: Recreation, 12, 2.625, 67.85199999999999, 0.0386871426044921, 2.625, 0.0386871426044921, 2.127, 64.694, 54.191, 0.0328778557516925, 0.0392500599730582, 0.0313476389789541, 0.0244925013082297\n",+ "\tRow 4 from table divfcfeIndia: Advertising, 18, 2.1670000000000003, 216.254, 0.0100206238959741, 2.1670000000000003, 0.0100206238959741, -34.924, 214.557, 224.701, -0.1627725965594224, -0.15542431942893, -0.1614952787000471, 0.0217870502946718\n",+ "\tRow 5 from table divfcfeIndia: Information Services, 20, 58.482, 231.44600000000003, 0.2526809709392255, 106.471, 0.4600252326676632, 86.724, 186.4650000000001, 188.9690000000001, 0.4650953262006272, 0.4589324174864657, 0.3747051147999965, 0.0319488066345242\n",+ "\n",+ "\n",+ "\u001b[0m\n",+ "Prediction(\n",+ " rationale='produce the SQL. We need to find the debt to EBITDA ratio for software industry and pharmaceutical companies. We will need to join multiple tables to get the required information.',\n",+ " sql=\"```sql\\nSELECT DISTINCT\\n wc.Industry_Name AS Industry,\\n (wacc.Cost_of_Debt / EVA.EBITDA) AS Debt_to_EBITDA_Ratio\\nFROM wcdataIndia wc\\nJOIN waccIndia wacc ON wc.Industry_Name = wacc.Industry_Name\\nJOIN EVAIndia EVA ON wc.Industry_Name = EVA.Industry_Name\\nWHERE wc.Industry_Name LIKE '%Software%' OR wc.Industry_Name LIKE '%Pharmaceutical%'\\n```\"\n",+ ")\n",+ "\u001b[31m(sqlite3.OperationalError) no such column: EVA.EBITDA\n",+ "[SQL: SELECT DISTINCT\n",+ " wc.Industry_Name AS Industry,\n",+ " (wacc.Cost_of_Debt / EVA.EBITDA) AS Debt_to_EBITDA_Ratio\n",+ "FROM wcdataIndia wc\n",+ "JOIN waccIndia wacc ON wc.Industry_Name = wacc.Industry_Name\n",+ "JOIN EVAIndia EVA ON wc.Industry_Name = EVA.Industry_Name\n",+ "WHERE wc.Industry_Name LIKE '%Software%' OR wc.Industry_Name LIKE '%Pharmaceutical%']\n",+ "(Background on this error at: https://sqlalche.me/e/20/e3q8)\u001b[0m\n",+ "\u001b[32mSql: ```sql\n",+ "SELECT DISTINCT wc.Industry_Name AS Industry, (wacc.Cost_of_Debt / EVA.Cost_of_Debt) AS Debt_to_EBITDA_Ratio \n",+ "FROM wcdataIndia wc \n",+ "JOIN waccIndia wacc ON wc.Industry_Name = wacc.Industry_Name \n",+ "JOIN EVAIndia EVA ON wc.Industry_Name = EVA.Industry_Name \n",+ "WHERE wc.Industry_Name LIKE '%Software%' OR wc.Industry_Name LIKE '%Pharmaceutical%'\n",+ "```\u001b[0m\n",+ "\n",+ "\u001b[32m```sql\n",+ "SELECT DISTINCT wc.Industry_Name AS Industry, (wacc.Cost_of_Debt / EVA.Cost_of_Debt) AS Debt_to_EBITDA_Ratio \n",+ "FROM wcdataIndia wc \n",+ "JOIN waccIndia wacc ON wc.Industry_Name = wacc.Industry_Name \n",+ "JOIN EVAIndia EVA ON wc.Industry_Name = EVA.Industry_Name \n",+ "WHERE wc.Industry_Name LIKE '%Software%' OR wc.Industry_Name LIKE '%Pharmaceutical%'\n",+ "```\u001b[0m\n",+ " Industry = Drugs (Pharmaceutical), Debt_to_EBITDA_Ratio = 1.0\n",+ " Industry = Software (Entertainment), Debt_to_EBITDA_Ratio = 1.0\n",+ " Industry = Software (Internet), Debt_to_EBITDA_Ratio = 1.0\n",+ " Industry = Software (System & Application), Debt_to_EBITDA_Ratio = 1.0\n",+ "\n",+ "The debt to EBITDA ratio for both the software industry and pharmaceutical companies is 1.0.\n"+ ]+ }+ ],+ "source": [+ "sq = tsql(\"What is the debt to EBITDA ratio for software industry and Pharmaceutical companies?\")\n",+ "print(sq.answer)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 70,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\u001b[33mTable 1: Table wcdata has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Acc_Rec_Sales (INTEGER), Inventory_Sales (INTEGER), Acc_Pay_Sales (INTEGER), Non_cash_WC_Sales (INTEGER)\n",+ "\tRow 1 from table wcdata: Environmental & Waste Services, 57, 0.1610239529643997, 0.0160835818498789, 0.0732441943895783, 0.1018059159059052\n",+ "\tRow 2 from table wcdata: Utility (Water), 13, 0.1519332278688117, 0.0326291583073621, 0.0835453907415389, 0.1135057399930943\n",+ "\tRow 3 from table wcdata: Software (Internet), 35, 0.1228885101454502, 1.6367264776917992e-05, 0.0423722824532171, 0.0966957543939344\n",+ "\tRow 4 from table wcdata: Packaging & Container, 22, 0.1374846253483106, 0.1265656973833159, 0.1305482832092476, 0.1112268155694879\n",+ "\tRow 5 from table wcdata: Semiconductor, 63, 0.1290909417966836, 0.1898317691334856, 0.0849115283669021, 0.2240302287823765\n",+ "\n",+ "\n",+ "Table 2: Table wacc has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), Cost_of_Equity (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER), Cost_of_Capital (INTEGER), Cost_of_Capital_Local_Currency_ (INTEGER)\n",+ "\tRow 1 from table wacc: Environmental & Waste Services, 57, 0.90699903446977, 0.0805219555856094, 0.8200309682356417, 0.4606641855124769, 0.050855, 0.0541917435398483, 0.03814125, 0.1799690317643583, 0.0728947410358769, 0.0728947410358769\n",+ "\tRow 2 from table wacc: Utility (Water), 13, 0.7139634340794939, 0.0716423179676567, 0.6627268690709837, 0.28198719816416, 0.050855, 0.1109172936995937, 0.03814125, 0.3372731309290163, 0.0603433078847393, 0.0603433078847392\n",+ "\tRow 3 from table wacc: Software (Internet), 35, 1.6160843271745926, 0.1131398790500312, 0.8930134645020447, 0.65221602599035, 0.060879, 0.0260557605345441, 0.04565925, 0.1069865354979553, 0.1059203603347457, 0.1059203603347458\n",+ "\tRow 4 from table wacc: Power, 50, 0.6506274182981785, 0.0687288612417162, 0.518339022067922, 0.2039437012455431, 0.045043, 0.1368565682061268, 0.03378225, 0.4816609779320779, 0.051896442295619, 0.0518964422956189\n",+ "\tRow 5 from table wacc: Packaging & Container, 22, 1.1327220379604754, 0.0909052137461818, 0.620115895714898, 0.2624270896574677, 0.050855, 0.1811910443993945, 0.03814125, 0.379884104285102, 0.0708610226399319, 0.0708610226399319\n",+ "\n",+ "\n",+ "Table 3: Table goodwill has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Goodwill_in_millions_ (INTEGER), Change_in_Goodwill_in_last_year (INTEGER), Goodwill_as_of_Total_Assets (INTEGER), Impairment_of_Goodwill_in_LTM_in_millioins (INTEGER), Impairment_as_of_Goodwill (INTEGER)\n",+ "\tRow 1 from table goodwill: Environmental & Waste Services, 57, 41368.444, 5543.546000000002, 0.3948996481872136, 15.278, 0.0003693153167665\n",+ "\tRow 2 from table goodwill: Utility (Water), 13, 4187.41, 13.519999999999529, 0.0702820236958313, 0.0, 0.0\n",+ "\tRow 3 from table goodwill: Software (Internet), 35, 21376.69, -1219.7700000000004, 0.3311515514892826, 484.91, 0.0226840544537063\n",+ "\tRow 4 from table goodwill: Semiconductor, 63, 173260.712, 1059.6019999999844, 0.2442931512253766, 429.63, 0.0024796735222928\n",+ "\tRow 5 from table goodwill: Power, 50, 71107.18000000001, 2563.4000000000087, 0.0438123430666786, 2078.7, 0.0292333348052897\n",+ "\n",+ "\n",+ "Table 4: Table DollarUS has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Average_Company_Age_years_ (INTEGER), Market_Cap_millions_ (INTEGER), Book_Equity_millions_ (INTEGER), Enteprise_Value_millions_ (INTEGER), Invested_Capital_millions_ (INTEGER), Total_Debt_including_leases_millions_ (INTEGER), Revenues_millions_ (INTEGER), Gross_Profit_millions_ (INTEGER), EBITDA_millions_ (INTEGER), EBIT_Operating_Income_millions_ (INTEGER), Net_Income_millions_ (INTEGER)\n",+ "\tRow 1 from table DollarUS: Environmental & Waste Services, 57, 36.204081632653065, 203179.789, 27942.29500000001, 245587.91613806767, 33584.18642975343, 44591.084138067694, 72586.38100000002, 25147.858, 16003.036, 10021.184772386465, 5871.133\n",+ "\tRow 2 from table DollarUS: Utility (Water), 13, 89.61538461538461, 46874.02, 17109.9, 69973.57501678159, 39265.4050167816, 23854.997016781603, 8862.38, 4918.995, 4115.236000000001, 2719.0665966436786, 1713.7600000000002\n",+ "\tRow 3 from table DollarUS: Packaging & Container, 22, 80.19047619047619, 132627.167, 49084.58099999999, 205727.3441716776, 86259.32917167754, 81247.63917167754, 146995.2, 31917.63, 23820.874, 14660.644165664493, 4193.327999999999\n",+ "\tRow 4 from table DollarUS: Software (System & Application), 351, 20.125, 5277974.335000003, 397835.59000000014, 5446717.280701609, 326815.0326293967, 327430.6527016032, 508015.76600000006, 363334.808, 170251.41100000014, 128741.80425967928, 97238.32000000008\n",+ "\tRow 5 from table DollarUS: Software (Internet), 35, 14.413793103448276, 220091.11299999992, 28399.863, 241745.7304766904, 25653.71492597519, 26367.783476690336, 28838.050000000003, 17046.417999999998, 1980.7540000000004, -835.1582953380669, -4129.525\n",+ "\n",+ "\n",+ "Table 5: Table divfcfe has columns: Industry_name (VARCHAR), Number_of_firms (INTEGER), _Dividends_ (INTEGER), _Net_Income_ (INTEGER), Payout (INTEGER), Dividends_Buybacks (INTEGER), Cash_Return_as_of_Net_Income (INTEGER), Dividends_Buybacks_Stock_Issuances (INTEGER), FCFE_before_debt_cash_flows_ (INTEGER), FCFE_after_debt_cash_flows_ (INTEGER), Net_Cash_Returned_FCFE_pre_debt_ (INTEGER), Net_Cash_Returned_FCFE_post_debt_ (INTEGER), Net_Cash_Returned_Net_Income (INTEGER), Cash_Firm_Value (INTEGER)\n",+ "\tRow 1 from table divfcfe: Environmental & Waste Services, 57, 2133.7000000000003, 5871.133, 0.3634221878468773, 4356.0830000000005, 0.7419492966689736, 3706.577, 4496.790999999999, 8844.003999999979, 0.824271574996481, 0.4191062102640398, 0.6313222677803416, 0.0088103858712382\n",+ "\tRow 2 from table divfcfe: Software (System & Application), 351, 26468.845, 97238.32000000008, 0.2722059060666615, 78137.651, 0.8035685005664427, 66388.049, 76723.88500000015, 92868.45200000016, 0.8652852889292542, 0.7148611565098542, 0.682735458613435, 0.0283097666177849\n",+ "\tRow 3 from table divfcfe: R.E.I.T., 193, 50197.926499999965, 23324.137999999984, 2.152187853630432, 43938.93199999997, 1.883839479941338, 18888.996999999978, 58792.15099999998, 108412.02399999648, 0.3212843326654264, 0.1742334134449938, 0.8098475922239866, 0.0188005563484407\n",+ "\tRow 4 from table divfcfe: Packaging & Container, 22, 2944.63, 4193.327999999999, 0.7022179042517068, 5178.039, 1.2348280411167458, 5090.014, 64.32999999999777, 2000.9439999999888, 79.12348826364335, 2.5438063234153625, 1.2138363610001417, 0.0380945383228542\n",+ "\tRow 5 from table divfcfe: Software (Entertainment), 84, 235.41, 93923.67300000004, 0.002506396869722, 90466.018, 0.9631865440356016, 89973.84199999999, 55413.70400000005, 62184.25100000004, 1.623674930663359, 1.44689114290369, 0.957946374179808, 0.0277904380642664\n",+ "\n",+ "\n",+ "\u001b[0m\n",+ "Prediction(\n",+ " rationale='produce the SQL. We need to retrieve the beta value and number of firms for the Software industries and semiconductor industry. We have the necessary information in the tables wcdata and wacc.',\n",+ " sql=\"```sql\\nSELECT DISTINCT wacc.Beta AS Beta_Value, wacc.Number_of_Firms AS Number_of_Firms\\nFROM wacc\\nWHERE wacc.Industry_Name = 'Software (Internet)' OR wacc.Industry_Name = 'Semiconductor';\\n```\"\n",+ ")\n",+ " Beta_Value = 1.495241185083536, Number_of_Firms = 63\n",+ " Beta_Value = 1.6160843271745926, Number_of_Firms = 35\n",+ "\n",+ "The beta value for the Software (Internet) industry is 1.495241185083536 with 63 firms, and for the Semiconductor industry, it is 1.6160843271745926 with 35 firms.\n"+ ]+ }+ ],+ "source": [+ "tsql = TextToSQLQueryModule(\"US\")\n",+ "sq = tsql(question=\"What is the beta value and number of firms for all the Software industries and semiconductor industry?\")\n",+ "print(sq.answer)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 87,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\u001b[33mTable 1: Table wcdataChina has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Acc_Rec_Sales (INTEGER), Inventory_Sales (INTEGER), Acc_Pay_Sales (INTEGER), Non_cash_WC_Sales (INTEGER)\n",+ "\tRow 1 from table wcdataChina: Environmental & Waste Services, 120, 0.7805756405836219, 0.1216584329026157, 0.4453190049448064, 0.4198965561444552\n",+ "\tRow 2 from table wcdataChina: Software (Internet), 19, 0.1839404184195781, 0.0865016001747371, 0.1431776526338781, 0.1385868130407118\n",+ "\tRow 3 from table wcdataChina: Utility (Water), 38, 0.5584269470144421, 0.056806454518549, 0.4650795448209008, 0.0671323545181202\n",+ "\tRow 4 from table wcdataChina: Power, 101, 0.1683105125531092, 0.0570632319985086, 0.1227652527863432, 0.0491055449848438\n",+ "\tRow 5 from table wcdataChina: Packaging & Container, 22, 0.2572060652570855, 0.2016428061075975, 0.2346039323612692, 0.193016274577258\n",+ "\n",+ "\n",+ "Table 2: Table waccChina has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Beta (INTEGER), Cost_of_Equity (INTEGER), E_D_E_ (INTEGER), Std_Dev_in_Stock (INTEGER), Cost_of_Debt (INTEGER), Tax_Rate (INTEGER), After_tax_Cost_of_Debt (INTEGER), D_D_E_ (INTEGER), Cost_of_Capital (INTEGER), Cost_of_Capital_Local_Currency_ (INTEGER)\n",+ "\tRow 1 from table waccChina: Environmental & Waste Services, 120, 1.0260843257446013, 0.096568547539421, 0.5693272218997147, 0.2807677994428052, 0.0585549999999999, 0.1004761945544854, 0.04391625, 0.4306727781002852, 0.0738926362847557, 0.0738926362847558\n",+ "\tRow 2 from table waccChina: Utility (Water), 38, 0.621853508338659, 0.0738103525194665, 0.4153411738771336, 0.2187327777480227, 0.052743, 0.1641446172200444, 0.03955725, 0.5846588261228663, 0.053783973809369, 0.053783973809369\n",+ "\tRow 3 from table waccChina: Software (Internet), 19, 0.9517387989546392, 0.0923828943811461, 0.7992627579619607, 0.3399832570802881, 0.0585549999999999, 0.0593218951020171, 0.04391625, 0.2007372420380393, 0.0826538338572364, 0.0826538338572364\n",+ "\tRow 4 from table waccChina: Packaging & Container, 22, 0.6632370473734861, 0.0761402457671272, 0.757479232841846, 0.2436783961826011, 0.052743, 0.1253655434552159, 0.03955725, 0.2425207671581539, 0.06726810956874, 0.0672681095687401\n",+ "\tRow 5 from table waccChina: Semiconductor, 173, 1.3766066410015918, 0.1163029538883896, 0.8916915589224178, 0.3505300755906819, 0.0585549999999999, 0.0455831075871332, 0.04391625, 0.1083084410775822, 0.1084628628354936, 0.1084628628354935\n",+ "\n",+ "\n",+ "Table 3: Table goodwillChina has columns: Industry_Name (VARCHAR), Number_of_Firms (INTEGER), Goodwill_in_millions_ (INTEGER), Change_in_Goodwill_in_last_year (INTEGER), Goodwill_as_of_Total_Assets (INTEGER), Impairment_of_Goodwill_in_LTM_in_millioins (INTEGER), Impairment_as_of_Goodwill (INTEGER)\n",+ "\tRow 1 from table goodwillChina: Environmental & Waste Services, 120, 1281.1550000000004, -305.801, 0.0098994856290149, 280.593, 0.2190156538436019\n",+ "\tRow 2 from table goodwillChina: Utility (Water), 38, 1178.812, -76.12300000000005, 0.0110438247444923, 2.713, 0.0023014696151718\n",+ "\tRow 3 from table goodwillChina: Power, 101, 12486.593, 553.064999999995, 0.0129974053264763, 284.26500000000004, 0.0227656174906958\n",+ "\tRow 4 from table goodwillChina: Software (Internet), 19, 437.1, -210.456, 0.0226844571766647, 209.421, 0.4791146190803019\n",+ "\tRow 5 from table goodwillChina: Packaging & Container, 22, 113.071, -31.805000000000007, 0.0080682012786847, 10.104, 0.0893597827913435\n",+ "\n",+ "\n",+ "Table 4: Table DollarChina has columns: Industry_Name (VARCHAR), Number_of_firms (INTEGER), Average_Company_Age_years_ (INTEGER), Market_Cap_millions_ (INTEGER), Book_Equity_millions_ (INTEGER), Enteprise_Value_millions_ (INTEGER), Invested_Capital_millions_ (INTEGER), Total_Debt_including_leases_millions_ (INTEGER), Revenues_millions_ (INTEGER), Gross_Profit_millions_ (INTEGER), EBITDA_millions_ (INTEGER), EBIT_Operating_Income_millions_ (INTEGER), Net_Income_millions_ (INTEGER)\n",+ "\tRow 1 from table DollarChina: Environmental & Waste Services, 120, 20.66379310344828, 63702.35200000002, 45104.79, 102308.07112384203, 89101.32412384207, 48188.22612384208, 35182.369999999995, 8751.014000000001, 5743.255999999997, 3536.412575231586, 711.52\n",+ "\tRow 2 from table DollarChina: Software (Internet), 19, 18.36842105263158, 22998.899999999998, 8376.390000000001, 26335.94280615911, 11691.322806159107, 5776.242806159108, 21060.2, 1836.089, 965.741, 463.86443876817833, 209.86\n",+ "\tRow 3 from table DollarChina: Utility (Water), 38, 24.257142857142856, 36310.41, 37787.3, 78750.12983004378, 87258.598, 51112.68283004377, 23313.9, 7664.893, 7358.603000000001, 4848.412433991248, 2454.96\n",+ "\tRow 4 from table DollarChina: Packaging & Container, 22, 20.454545454545453, 11669.89, 6646.999999999999, 13670.927747010572, 8812.14674701057, 3736.3277470105695, 9447.249999999996, 1447.605, 1022.056, 562.4454505978858, 412.2699999999999\n",+ "\tRow 5 from table DollarChina: Telecom (Wireless), 8, 18.25, 204487.52, 197714.1, 184902.649, 207578.19200000004, 21250.229, 188443.0, 51715.42999999999, 49397.89, 21422.96, 19285.28\n",+ "\n",+ "\n",+ "Table 5: Table divfcfeChina has columns: Industry_name (VARCHAR), Number_of_firms (INTEGER), _Dividends_ (INTEGER), _Net_Income_ (INTEGER), Payout (INTEGER), Dividends_Buybacks (INTEGER), Cash_Return_as_of_Net_Income (INTEGER), Dividends_Buybacks_Stock_Issuances (INTEGER), FCFE_before_debt_cash_flows_ (INTEGER), FCFE_after_debt_cash_flows_ (INTEGER), Net_Cash_Returned_FCFE_pre_debt_ (INTEGER), Net_Cash_Returned_FCFE_post_debt_ (INTEGER), Net_Cash_Returned_Net_Income (INTEGER), Cash_Firm_Value (INTEGER)\n",+ "\tRow 1 from table divfcfeChina: Telecom (Wireless), 8, 14213.16, 19285.28, 0.7369952627081381, 14321.986, 0.7426382194087926, 13860.877, 9217.517999999996, 2496.7279999999955, 1.503753721988935, 5.551616756010276, 0.7187283254378468, 0.1808961956114836\n",+ "\tRow 2 from table divfcfeChina: Utility (General), 4, 89.39999999999999, 142.57, 0.6270603913866872, 89.39999999999999, 0.6270603913866872, 89.39999999999999, 97.29000000000002, 340.03, 0.9189022510021582, 0.262917977825486, 0.6270603913866872, 0.1293218296009642\n",+ "\tRow 3 from table divfcfeChina: Oil/Gas (Production and Exploration), 15, 8865.4, 18678.828999999994, 0.4746229006111679, 8990.207, 0.4813046363880735, 8971.107, 9300.488999999998, 6030.410999999996, 0.9645844428180068, 1.4876443744879089, 0.4802820883471872, 0.1872653060931191\n",+ "\tRow 4 from table divfcfeChina: Office Equipment & Services, 10, 143.689, 227.36500000000007, 0.6319750181426339, 166.519, 0.7323862511820199, 98.474, 63.8070000000001, 37.09000000000015, 1.5433102951086848, 2.6550013480722465, 0.4331097574384798, 0.143033301916431\n",+ "\tRow 5 from table divfcfeChina: Total Market, 7504, 504086.7439999993, 798823.6939999987, 0.6310362947246281, 546579.9069999994, 0.6842309649868752, 297353.3259999999, 332582.1279999978, 531681.1990000112, 0.8940748794535401, 0.5592699658352854, 0.3722389911984763, 0.1603796454322562\n",+ "\n",+ "\n",+ "\u001b[0m\n",+ "Prediction(\n",+ " rationale='produce the SQL. We need to retrieve the beta value and number of firms for the Software industries and semiconductor industry. We have the necessary information in the tables wcdataChina and waccChina.',\n",+ " sql=\"```sql\\nSELECT DISTINCT waccChina.Industry_Name AS Industry, waccChina.Beta AS Beta_Value, waccChina.Number_of_Firms AS Number_of_Firms\\nFROM waccChina\\nWHERE waccChina.Industry_Name IN ('Software (Internet)', 'Semiconductor');\\n```\"\n",+ ")\n",+ " Industry = Semiconductor, Beta_Value = 1.3766066410015918, Number_of_Firms = 173\n",+ " Industry = Software (Internet), Beta_Value = 0.9517387989546392, Number_of_Firms = 19\n",+ "\n",+ "The beta value for the Semiconductor industry is 1.3766066410015918 with 173 firms, and for the Software (Internet) industry, the beta value is 0.9517387989546392 with 19 firms.\n"+ ]+ }+ ],+ "source": [+ "tsql = TextToSQLQueryModule(\"China\")\n",+ "sq = tsql(question=\"What is the beta value and number of firms for all the Software industries and semiconductor industry?\")\n",+ "print(sq.answer)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": []+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": ".venv",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.12"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}