DSPy: The framework for programming—not prompting—language models
Python
clear output in docs
Herumb Shandilyaa9a6aba7369a
diff --git a/examples/dataloaders/dataloaders_dolly.ipynb b/examples/dataloaders/dataloaders_dolly.ipynbindex 7612b58ff..fc86373f8 100644--- a/examples/dataloaders/dataloaders_dolly.ipynb+++ b/examples/dataloaders/dataloaders_dolly.ipynb@@ -1,49 +1,35 @@{- "nbformat": 4,- "nbformat_minor": 0,- "metadata": {- "colab": {- "provenance": []- },- "kernelspec": {- "name": "python3",- "display_name": "Python 3"- },- "language_info": {- "name": "python"- }- },"cells": [{"cell_type": "markdown",- "source": [- ""- ],"metadata": {"id": "i9ZPeh5Gk2pX"- }+ },+ "source": [+ ""+ ]},{"cell_type": "markdown",+ "metadata": {+ "id": "FH5S5Xm1lMC-"+ },"source": ["In the **DSPy** framework, the primary data type utilized for interactions with various modules `Example`. Given the huge repository of datasets available through Hugging Face, there's a significant opportunity to leverage these resources within DSPy. However, to make these datasets compatible and fully utilizable, they need to be converted into the `Example` data format.\n","\n","Moreover, CSV files are a common format for storing and exchanging data, recognized for their simplicity and wide application across various domains. The conversion of data from CSV format into the `Example` format is a necessary requirement. But can we we do something to avoid this tedious process?\n","\n","Introducing `DataLoaders`, a module aimed at simplifying the task of importing dataset from diverse sources, including HuggingFace and CSV files, into the `Example` format. `DataLoaders` are designed to abstract away the complexities and boilerplate involved in data conversion, offering a user-friendly interface for loading, transforming, and preparing data for use within the DSPy framework. This addition significantly enhances the ease of use of DSPy."- ],- "metadata": {- "id": "FH5S5Xm1lMC-"- }+ ]},{"cell_type": "markdown",- "source": [- "# Setting Up"- ],"metadata": {"id": "RqSZB2MqsTAQ"- }+ },+ "source": [+ "# Setting Up"+ ]},{"cell_type": "code",@@ -57,144 +43,14 @@},"outputs": [{- "output_type": "stream",- "name": "stdout",- "text": [- "Already up to date.\n",- "Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages (24.0)\n",- "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",- "\u001b[0mObtaining file:///content/dspy\n",- " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",- " Checking if build backend supports build_editable ... \u001b[?25l\u001b[?25hdone\n",- " Getting requirements to build editable ... \u001b[?25l\u001b[?25hdone\n",- " Preparing editable metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",- "Collecting backoff~=2.2.1 (from dspy-ai==2.1.10)\n",- " Downloading backoff-2.2.1-py3-none-any.whl (15 kB)\n",- "Requirement already satisfied: joblib~=1.3.2 in /usr/local/lib/python3.10/dist-packages (from dspy-ai==2.1.10) (1.3.2)\n",- "Collecting openai<2.0.0,>=0.28.1 (from dspy-ai==2.1.10)\n",- " Downloading openai-1.12.0-py3-none-any.whl.metadata (18 kB)\n",- "Collecting pandas~=2.1.1 (from dspy-ai==2.1.10)\n",- " Downloading pandas-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)\n",- "Collecting regex~=2023.10.3 (from dspy-ai==2.1.10)\n",- " Downloading regex-2023.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hCollecting ujson~=5.8.0 (from dspy-ai==2.1.10)\n",- " Downloading ujson-5.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.7 kB)\n",- "Requirement already satisfied: tqdm~=4.66.1 in /usr/local/lib/python3.10/dist-packages (from dspy-ai==2.1.10) (4.66.1)\n",- "Collecting datasets~=2.14.6 (from dspy-ai==2.1.10)\n",- " Downloading datasets-2.14.7-py3-none-any.whl.metadata (19 kB)\n",- "Requirement already satisfied: requests~=2.31.0 in /usr/local/lib/python3.10/dist-packages (from dspy-ai==2.1.10) (2.31.0)\n",- "Collecting optuna~=3.4.0 (from dspy-ai==2.1.10)\n",- " Downloading optuna-3.4.0-py3-none-any.whl.metadata (17 kB)\n",- "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (1.25.2)\n",- "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (10.0.1)\n",- "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (0.6)\n",- "Collecting dill<0.3.8,>=0.3.0 (from datasets~=2.14.6->dspy-ai==2.1.10)\n",- " Downloading dill-0.3.7-py3-none-any.whl.metadata (9.9 kB)\n",- "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (3.4.1)\n",- "Collecting multiprocess (from datasets~=2.14.6->dspy-ai==2.1.10)\n",- " Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n",- "Requirement already satisfied: fsspec<=2023.10.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets~=2.14.6->dspy-ai==2.1.10) (2023.6.0)\n",- "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (3.9.3)\n",- "Requirement already satisfied: huggingface-hub<1.0.0,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (0.20.3)\n",- "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (23.2)\n",- "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets~=2.14.6->dspy-ai==2.1.10) (6.0.1)\n",- "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (3.7.1)\n",- "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (1.7.0)\n",- "Collecting httpx<1,>=0.23.0 (from openai<2.0.0,>=0.28.1->dspy-ai==2.1.10)\n",- " Downloading httpx-0.26.0-py3-none-any.whl.metadata (7.6 kB)\n",- "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (2.6.1)\n",- "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (1.3.0)\n",- "Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (4.9.0)\n",- "Collecting alembic>=1.5.0 (from optuna~=3.4.0->dspy-ai==2.1.10)\n",- " Downloading alembic-1.13.1-py3-none-any.whl.metadata (7.4 kB)\n",- "Collecting colorlog (from optuna~=3.4.0->dspy-ai==2.1.10)\n",- " Downloading colorlog-6.8.2-py3-none-any.whl.metadata (10 kB)\n",- "Requirement already satisfied: sqlalchemy>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from optuna~=3.4.0->dspy-ai==2.1.10) (2.0.25)\n",- "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas~=2.1.1->dspy-ai==2.1.10) (2.8.2)\n",- "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas~=2.1.1->dspy-ai==2.1.10) (2023.4)\n",- "Collecting tzdata>=2022.1 (from pandas~=2.1.1->dspy-ai==2.1.10)\n",- " Downloading tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)\n",- "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests~=2.31.0->dspy-ai==2.1.10) (3.3.2)\n",- "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests~=2.31.0->dspy-ai==2.1.10) (3.6)\n",- "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests~=2.31.0->dspy-ai==2.1.10) (2.0.7)\n",- "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests~=2.31.0->dspy-ai==2.1.10) (2024.2.2)\n",- "Collecting Mako (from alembic>=1.5.0->optuna~=3.4.0->dspy-ai==2.1.10)\n",- " Downloading Mako-1.3.2-py3-none-any.whl.metadata (2.9 kB)\n",- "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (1.2.0)\n",- "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets~=2.14.6->dspy-ai==2.1.10) (1.3.1)\n",- "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets~=2.14.6->dspy-ai==2.1.10) (23.2.0)\n",- "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets~=2.14.6->dspy-ai==2.1.10) (1.4.1)\n",- "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets~=2.14.6->dspy-ai==2.1.10) (6.0.5)\n",- "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets~=2.14.6->dspy-ai==2.1.10) (1.9.4)\n",- "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets~=2.14.6->dspy-ai==2.1.10) (4.0.3)\n",- "Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai<2.0.0,>=0.28.1->dspy-ai==2.1.10)\n",- " Downloading httpcore-1.0.2-py3-none-any.whl.metadata (20 kB)\n",- "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=0.28.1->dspy-ai==2.1.10)\n",- " Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.14.0->datasets~=2.14.6->dspy-ai==2.1.10) (3.13.1)\n",- "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (0.6.0)\n",- "Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai<2.0.0,>=0.28.1->dspy-ai==2.1.10) (2.16.2)\n",- "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas~=2.1.1->dspy-ai==2.1.10) (1.16.0)\n",- "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from sqlalchemy>=1.3.0->optuna~=3.4.0->dspy-ai==2.1.10) (3.0.3)\n",- "INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.\n",- "Collecting multiprocess (from datasets~=2.14.6->dspy-ai==2.1.10)\n",- " Downloading multiprocess-0.70.15-py310-none-any.whl.metadata (7.2 kB)\n",- "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from Mako->alembic>=1.5.0->optuna~=3.4.0->dspy-ai==2.1.10) (2.1.5)\n",- "Downloading datasets-2.14.7-py3-none-any.whl (520 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m520.4/520.4 kB\u001b[0m \u001b[31m21.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading openai-1.12.0-py3-none-any.whl (226 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.7/226.7 kB\u001b[0m \u001b[31m22.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading optuna-3.4.0-py3-none-any.whl (409 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m409.6/409.6 kB\u001b[0m \u001b[31m34.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading pandas-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m92.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading regex-2023.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (773 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m773.9/773.9 kB\u001b[0m \u001b[31m52.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading ujson-5.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.9/53.9 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading alembic-1.13.1-py3-none-any.whl (233 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m233.4/233.4 kB\u001b[0m \u001b[31m22.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading dill-0.3.7-py3-none-any.whl (115 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading httpx-0.26.0-py3-none-any.whl (75 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.9/75.9 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m345.4/345.4 kB\u001b[0m \u001b[31m30.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading colorlog-6.8.2-py3-none-any.whl (11 kB)\n",- "Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m14.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hDownloading Mako-1.3.2-py3-none-any.whl (78 kB)\n",- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.7/78.7 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",- "\u001b[?25hBuilding wheels for collected packages: dspy-ai\n",- " Building editable for dspy-ai (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",- " Created wheel for dspy-ai: filename=dspy_ai-2.1.10-0.editable-py3-none-any.whl size=15381 sha256=a7d7a53dbcdb151dab69c7177d37bcf850bb4c2f965c51975c8aaf4b6959601b\n",- " Stored in directory: /tmp/pip-ephem-wheel-cache-ypxzl_a1/wheels/80/c7/dd/82bee9a1efa54dd8a664012a2537077eafe45c01781e3e9d68\n",- "Successfully built dspy-ai\n",- "Installing collected packages: ujson, tzdata, regex, Mako, h11, dill, colorlog, backoff, pandas, multiprocess, httpcore, alembic, optuna, httpx, openai, datasets, dspy-ai\n",- " Attempting uninstall: regex\n",- " Found existing installation: regex 2023.12.25\n",- " Uninstalling regex-2023.12.25:\n",- " Successfully uninstalled regex-2023.12.25\n",- " Attempting uninstall: pandas\n",- " Found existing installation: pandas 1.5.3\n",- " Uninstalling pandas-1.5.3:\n",- " Successfully uninstalled pandas-1.5.3\n",- "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",- "lida 0.0.10 requires fastapi, which is not installed.\n",- "lida 0.0.10 requires kaleido, which is not installed.\n",- "lida 0.0.10 requires python-multipart, which is not installed.\n",- "lida 0.0.10 requires uvicorn, which is not installed.\n",- "llmx 0.0.15a0 requires cohere, which is not installed.\n",- "llmx 0.0.15a0 requires tiktoken, which is not installed.\n",- "bigframes 0.20.1 requires pandas<2.1.4,>=1.5.0, but you have pandas 2.1.4 which is incompatible.\n",- "google-colab 1.0.0 requires pandas==1.5.3, but you have pandas 2.1.4 which is incompatible.\u001b[0m\u001b[31m\n",- "\u001b[0mSuccessfully installed Mako-1.3.2 alembic-1.13.1 backoff-2.2.1 colorlog-6.8.2 datasets-2.14.7 dill-0.3.7 dspy-ai-2.1.10 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 multiprocess-0.70.15 openai-1.12.0 optuna-3.4.0 pandas-2.1.4 regex-2023.10.3 tzdata-2024.1 ujson-5.8.0\n",- "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",- "\u001b[0m"+ "ename": "ModuleNotFoundError",+ "evalue": "No module named 'pkg_resources'",+ "output_type": "error",+ "traceback": [+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",+ "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",+ "Cell \u001b[0;32mIn[1], line 20\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[38;5;66;03m# Set up the cache for this notebook\u001b[39;00m\n\u001b[1;32m 18\u001b[0m os\u001b[38;5;241m.\u001b[39menviron[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDSP_NOTEBOOK_CACHEDIR\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mjoin(repo_path, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcache\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m---> 20\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpkg_resources\u001b[39;00m \u001b[38;5;66;03m# Install the package if it's not installed\u001b[39;00m\n\u001b[1;32m 21\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdspy-ai\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m {pkg\u001b[38;5;241m.\u001b[39mkey \u001b[38;5;28;01mfor\u001b[39;00m pkg \u001b[38;5;129;01min\u001b[39;00m pkg_resources\u001b[38;5;241m.\u001b[39mworking_set}:\n\u001b[1;32m 22\u001b[0m get_ipython()\u001b[38;5;241m.\u001b[39msystem(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mpip install -U pip\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",+ "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pkg_resources'"]}],@@ -227,6 +83,9 @@},{"cell_type": "markdown",+ "metadata": {+ "id": "PtBml664suSx"+ },"source": ["# Using `DataLoader`\n","\n",@@ -236,13 +95,24 @@"* CSV Files using `from_csv()` method.\n","\n","We'll start by initializing the `DataLoader` object that takes same argument as that of [`Dataset`](https://github.com/stanfordnlp/dspy/blob/main/dspy/datasets/dataset.py) class which are dataset size (in % format) and seed for each of the three splits: `train`, `dev` and `test`."- ],- "metadata": {- "id": "PtBml664suSx"- }+ ]},{"cell_type": "code",+ "execution_count": 2,+ "metadata": {+ "id": "UeB7AEvhshfe"+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "/home/krypticmouse/.cache/pypoetry/virtualenvs/dspy-74wouE_3-py3.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",+ " from .autonotebook import tqdm as notebook_tqdm\n"+ ]+ }+ ],"source": ["import dspy\n","from dspy.datasets import DataLoader\n",@@ -252,15 +122,13 @@" dev_size = 0.3,\n"," test_size = 0.2,\n",")"- ],- "metadata": {- "id": "UeB7AEvhshfe"- },- "execution_count": 21,- "outputs": []+ ]},{"cell_type": "markdown",+ "metadata": {+ "id": "RR8vp-IWv-N0"+ },"source": ["## Loading from HuggingFace\n","\n",@@ -271,46 +139,45 @@"* `fields`: An optional list of strings specifying which fields to include in the final datasets. If None, all fields in the dataset are included.\n","* `splits`: A string or list of strings specifying which splits of the dataset to load. If None, the entire dataset is loaded. If it's a list of splits, **it'll concatinate all the splits into 1 split**.\n","* `revision`: An optional string specifying the version of the dataset to load. Same parameter as that of datasets library."- ],- "metadata": {- "id": "RR8vp-IWv-N0"- }+ ]},{"cell_type": "code",+ "execution_count": 3,+ "metadata": {+ "id": "OslWdJbMvVYU"+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Downloading readme: 100%|██████████| 8.20k/8.20k [00:00<00:00, 10.7MB/s]\n",+ "Downloading data: 100%|██████████| 13.1M/13.1M [00:20<00:00, 649kB/s]\n",+ "Generating train split: 15011 examples [00:00, 518904.00 examples/s]\n"+ ]+ }+ ],"source": ["dl.from_huggingface(\n"," \"databricks/databricks-dolly-15k\",\n"," fields=[\"instruction\", \"context\", \"response\"],\n"," splits=\"train\" # or splits = [\"train\", \"test\"]\n",")"- ],- "metadata": {- "id": "OslWdJbMvVYU"- },- "execution_count": 22,- "outputs": []+ ]},{"cell_type": "markdown",- "source": [- "Once the above code is executed the data would be populated in the `train`, `dev`, and `test` properties of the object:"- ],"metadata": {"id": "RRyYfcTUF121"- }+ },+ "source": [+ "Once the above code is executed the data would be populated in the `train`, `dev`, and `test` properties of the object:"+ ]},{"cell_type": "code",- "source": [- "dolly_train_set = dl.train\n",- "dolly_dev_set = dl.dev\n",- "dolly_test_set = dl.test\n",- "\n",- "print(f\"Number of examples in train set: {len(dolly_train_set)}\")\n",- "print(f\"Number of examples in dev set: {len(dolly_dev_set)}\")\n",- "print(f\"Number of examples in test set: {len(dolly_test_set)}\")"- ],+ "execution_count": 4,"metadata": {"colab": {"base_uri": "https://localhost:8080/"@@ -318,21 +185,32 @@"id": "CIj3Tkj6zWTq","outputId": "6ff849e1-c6f5-453c-d328-ce78a24e15a9"},- "execution_count": 23,"outputs": [{- "output_type": "stream","name": "stdout",+ "output_type": "stream","text": ["Number of examples in train set: 7505\n","Number of examples in dev set: 2251\n","Number of examples in test set: 1501\n"]}+ ],+ "source": [+ "dolly_train_set = dl.train\n",+ "dolly_dev_set = dl.dev\n",+ "dolly_test_set = dl.test\n",+ "\n",+ "print(f\"Number of examples in train set: {len(dolly_train_set)}\")\n",+ "print(f\"Number of examples in dev set: {len(dolly_dev_set)}\")\n",+ "print(f\"Number of examples in test set: {len(dolly_test_set)}\")"]},{"cell_type": "markdown",+ "metadata": {+ "id": "WJ78GF1kPn9N"+ },"source": ["## Loading from CSV\n","\n",@@ -340,13 +218,23 @@"\n","* `file_path`: The file path of the CSV file to load.\n","* `fields`: An optional list of strings specifying which columns to include in the final datasets. If None, all columns in the dataset are included."- ],- "metadata": {- "id": "WJ78GF1kPn9N"- }+ ]},{"cell_type": "code",+ "execution_count": 5,+ "metadata": {+ "id": "tkK14QOVQwph"+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Generating train split: 100 examples [00:00, 27541.56 examples/s]\n"+ ]+ }+ ],"source": ["dl = DataLoader(\n"," train_size = 1.0, # Must be float\n",@@ -356,29 +244,20 @@" \"dolly_subset_100_rows.csv\",\n"," fields=[\"instruction\", \"context\", \"response\"],\n",")"- ],- "metadata": {- "id": "tkK14QOVQwph"- },- "execution_count": 26,- "outputs": []+ ]},{"cell_type": "markdown",- "source": [- "Once the above code is executed the data would be populated in the `train`, `dev`, and `test` properties of the object:"- ],"metadata": {"id": "q2Zcn1X2RPr_"- }+ },+ "source": [+ "Once the above code is executed the data would be populated in the `train`, `dev`, and `test` properties of the object:"+ ]},{"cell_type": "code",- "source": [- "dolly_train_set = dl.train\n",- "\n",- "print(f\"Number of examples in train set: {len(dolly_train_set)}\")"- ],+ "execution_count": 6,"metadata": {"colab": {"base_uri": "https://localhost:8080/"@@ -386,25 +265,52 @@"id": "TIrNy_wrQ4B5","outputId": "cd16d1d1-519a-4bbb-9c79-b89ce2f8e861"},- "execution_count": 27,"outputs": [{- "output_type": "stream","name": "stdout",+ "output_type": "stream","text": ["Number of examples in train set: 100\n"]}+ ],+ "source": [+ "dolly_train_set = dl.train\n",+ "\n",+ "print(f\"Number of examples in train set: {len(dolly_train_set)}\")"]},{"cell_type": "code",- "source": [],+ "execution_count": null,"metadata": {"id": "r4K2TWkRWdDR"},- "execution_count": null,- "outputs": []+ "outputs": [],+ "source": []}- ]-}\ No newline at end of file+ ],+ "metadata": {+ "colab": {+ "provenance": []+ },+ "kernelspec": {+ "display_name": "Python 3",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.12"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 0+}