DSPy: The framework for programming—not prompting—language models
Python
Adapters type casting & enforcement. Migration notebook.
Omar Khattab6812286e3a39
diff --git a/dspy/__init__.py b/dspy/__init__.pyindex 9ad0f1f74..0c9348641 100644--- a/dspy/__init__.py+++ b/dspy/__init__.py@@ -71,5 +71,6 @@ MIPROv2 = dspy.teleprompt.MIPROv2Ensemble = dspy.teleprompt.Ensemble+# TODO: Consider if this should access settings.lm *or* a list that's shared across all LMs in the program.def inspect_history(*args, **kwargs):return settings.lm.inspect_history(*args, **kwargs)\ No newline at end of filediff --git a/dspy/adapters/chat_adapter.py b/dspy/adapters/chat_adapter.pyindex 9acfafcc6..8ef73f9a4 100644--- a/dspy/adapters/chat_adapter.py+++ b/dspy/adapters/chat_adapter.py@@ -1,6 +1,11 @@import re+import ast+import jsonimport textwrap++from pydantic import TypeAdapterfrom .base import Adapter+from typing import get_origin, get_argsfield_header_pattern = re.compile(r'\[\[ ## (\w+) ## \]\]')@@ -43,7 +48,11 @@ class ChatAdapter(Adapter):fields = {}for k, v in sections:- if (k not in fields) and (k in signature.output_fields): fields[k] = v+ if (k not in fields) and (k in signature.output_fields):+ try:+ fields[k] = parse_value(v, signature.output_fields[k].annotation)+ except Exception as e:+ raise ValueError(f"Error parsing field {k}: {e}, with value ```{v}```")if fields.keys() != signature.output_fields.keys():raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")@@ -72,7 +81,17 @@ def format_fields(fields):return '\n\n'.join(output).strip()-++def parse_value(value, annotation):+ if annotation is str: return str(value)+ parsed_value = value+ if isinstance(value, str):+ try: parsed_value = json.loads(value)+ except json.JSONDecodeError:+ try: parsed_value = ast.literal_eval(value)+ except (ValueError, SyntaxError): parsed_value = value+ return TypeAdapter(annotation).validate_python(parsed_value)+def format_turn(signature, values, role, incomplete=False):content = []@@ -97,11 +116,24 @@ def format_turn(signature, values, role, incomplete=False):return {"role": role, "content": '\n\n'.join(content).strip()}++def get_annotation_name(annotation):+ origin = get_origin(annotation)+ args = get_args(annotation)+ if origin is None:+ if hasattr(annotation, '__name__'):+ return annotation.__name__+ else:+ return str(annotation)+ else:+ args_str = ', '.join(get_annotation_name(arg) for arg in args)+ return f"{origin.__name__}[{args_str}]"+def enumerate_fields(fields):parts = []for idx, (k, v) in enumerate(fields.items()):parts.append(f"{idx+1}. `{k}`")- parts[-1] += f" ({v.annotation.__name__})"+ parts[-1] += f" ({get_annotation_name(v.annotation)})"parts[-1] += f": {v.json_schema_extra['desc']}" if v.json_schema_extra['desc'] != f'${{{k}}}' else ''return '\n'.join(parts).strip()diff --git a/examples/migration.ipynb b/examples/migration.ipynbnew file mode 100644index 000000000..238e91610--- /dev/null+++ b/examples/migration.ipynb@@ -0,0 +1,348 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "<img src=\"../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"120\"/>\n",+ "\n",+ "### Migrating from DSPy 2.4 to 2.5\n",+ "\n",+ "**DSPy 2.5** focuses on improving the _pre-optimization_ developer experience, i.e. making it easier to obtain higher-quality outputs from various LMs _out of the box_ prior to setting up a DSPy optimizer."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### tl;dr Just set up the LM differently at the start.\n",+ "\n",+ "For most applications, you will only need to change 1-2 lines."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "/opt/anaconda3/envs/jun2024_py310/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",+ " from .autonotebook import tqdm as notebook_tqdm\n"+ ]+ }+ ],+ "source": [+ "import dspy\n",+ "\n",+ "lm = dspy.LM('openai/gpt-4o-mini')\n",+ "dspy.configure(lm=lm)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "The rest of your code can remain unchanged. However, make sure to experiment with your modules. Improvements to **Adapters** will affect the underlying prompts, before and after optimization, and will enforce stricter validation for LM outputs.\n",+ "\n",+ "If you want to dive deeper, here's are some new things and things to keep in mind when upgrading.\n",+ "\n",+ "1. **Use the `dspy.LM` class for setting up language models.**\n",+ "2. **Invoke LMs in the same way as before.**\n",+ "3. **This internally uses new DSPy `Adapters`, leading to better prompting out of the box.**\n",+ "4. **Advanced: If needed, you can set up your own Adapter for LMs.**"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 1) Use the `dspy.LM` class for setting up language models."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Earlier versions of DSPy involved tens of clients for different LM providers, e.g. `dspy.OpenAI`, `dspy.GoogleVertexAI`, and `dspy.HFClientTGI`, etc. These are now deprecated and will be removed in DSPy 2.6.\n",+ "\n",+ "Instead, use `dspy.LM` to access any LM endpoint for local and remote models. This relies on [LiteLLM](https://github.com/BerriAI/litellm) to translate the different client APIs into an OpenAI-compatible interface.\n",+ "\n",+ "Any [provider supported in LiteLLM](https://docs.litellm.ai/docs/providers) should work with `dspy.LM`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# OpenAI, which can authenticate via OPENAI_API_KEY.\n",+ "lm = dspy.LM('openai/gpt-4o-mini')\n",+ "\n",+ "# Anthropic, which can authenticate via ANTHROPIC_API_KEY.\n",+ "anthropic_lm = dspy.LM('anthropic/claude-3-opus-20240229')\n",+ "\n",+ "# You can also pass auth information directly.\n",+ "anthropic_lm = dspy.LM('anthropic/claude-3-opus-20240229', api_key=\"..\", api_base=\"..\")\n",+ "\n",+ "# Cohere, which can authenticate via COHERE_API_KEY.\n",+ "cohere_lm = dspy.LM('cohere/command-nightly')\n",+ "\n",+ "# Databricks, which can authenticate via DATABRICKS_API_KEY & DATABRICKS_API_BASE (or automatically on a DB workspace).\n",+ "databricks_llama3 = dspy.LM('databricks/databricks-meta-llama-3-1-70b-instruct')\n",+ "\n",+ "# or many others."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Any OpenAI-compatible endpoint is easy to set up with an `openai/` prefix as well. This works great for open LMs from HuggingFace hosted locally with SGLang, VLLM, or HF Text-Generation-Inference."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "sglang_port = 7501\n",+ "sglang_url = f\"http://localhost:{sglang_port}/v1\"\n",+ "sglang_llama = dspy.LM(\"openai/meta-llama/Meta-Llama-3-8B-Instruct\", api_base=sglang_url)\n",+ "\n",+ "# You could also use text mode, in which the prompts are *not* formatted as messages.\n",+ "sglang_llama_text = dspy.LM(\"openai/meta-llama/Meta-Llama-3-8B-Instruct\", api_base=sglang_url, model_type='text')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Configuring LM attributes\n",+ "\n",+ "For any LM, you can configure any of the following attributes at initialization or per call."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "gpt_4o_mini = dspy.LM('openai/gpt-4o-mini', temperature=0.9, max_tokens=3000, stop=None, cache=False)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Caching.\n",+ "\n",+ "Old clients still exist but you'll get a deprecation warning. They will be removed in DSPy 2.6."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 2) Invoke LMs in the same way as before."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Invoking the LM directly with a prompt is the same as before."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "lm(\"What is 2+2?\", temperature=0.9)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "For chat LMs, you can pass a list of messages."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "lm(messages=[{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",+ " {\"role\": \"user\", \"content\": \"What is 2+2?\"}])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Using DSPy modules is the same as before."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "module = dspy.ChainOfThought('question -> answer')\n",+ "module(question=\"What is 2+2?\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Every LM object maintains the history of its interactions, including inputs, outputs, token usage (and $$$ cost), and metadata."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "len(lm.history) # e.g., 3 calls to the LM\n",+ "\n",+ "lm.history[-1].keys() # access the last call to the LM, with all metadata"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 3) This internally uses new DSPy `Adapters`, leading to better prompting out of the box.\n",+ "\n",+ "Prompt optimizers in DSPy generate and tune the _instructions_ or the _examples_ in the prompts corresponding to your Signatures. DSPy 2.5 introduces **Adapters** as a layer between Signatures and LMs, responsible for formatting these pieces (Signature I/O fields, instructions, and examples) as well as generating and parsing the outputs. In DSPy 2.5, the default Adapters are now more natively aware of chat LMs and are responsible for enforcing types, building on earlier experimental features from `TypedPredictor` and `configure(experimental=True)`. In our experience, this change tends to deliver more consistent pre-optimization quality from various LMs, especially for sophisticated Signatures."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "fact_checking = dspy.ChainOfThought('claims -> verdict: list[bool]')\n",+ "fact_checking(claims=[\"Python was released in 1991.\", \"Python is a compiled language.\"])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We can as usual inspect the last prompt (and output) from an LM call."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "dspy.inspect_history()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Stricter validation for LM outputs.\n",+ "\n",+ "Previously, only `TypedPredictor`s had strict validation for LM outputs. This is now in all DSPy modules."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# a module with insufficient tokens to format the output\n",+ "bad_module = dspy.ChainOfThought('input -> output: list[str]', max_tokens=5)\n",+ "\n",+ "try:\n",+ " bad_module(input='oops')\n",+ "except ValueError as e:\n",+ " print(\"When parsing the output:\", e)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We are currently considering a \"soft failure\" mode, which fills failed fields with `None`s for cases that don't require explicit exception handling. We are also working on more accessible exceptions and error messages in future 2.5.* releases."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 4) Advanced: If needed, you can set up your own Adapter for LMs."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Adapters are the first step towards DSPy optimizers that tune the templates themselves. They also make it much easier for you to adapt the templates to your needs programatically, though that remains discouraged for most applications since such effort is often best spent on a program's control flow or optimization. At a minimum, an Adapter defines a method `format(signature, demos, inputs)` that prepares a list of messages (or a prompt string) and a method `parse(signature, completion)` that extract a `dict` of output fields. You can also overwrite `__call__(lm, lm_kwargs, signature, demos, inputs)` to refine the generation or decoding logic for your signatures."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "dspy.configure(adapter=dspy.ChatAdapter()) # ChatAdapter is the default adapter, but you can override it."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5) What's next?"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In the next few weeks, we will release into DSPy 2.5 a number of new updates, including: (1) optimizable adapters, smarter retries, better exceptions, support for images, support for multi-turn signatures, improved finetuning and assertions, concurrent with other ongoing efforts from the [DSPy Roadmap](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/roadmap.md), which include more powerful optimizers, human-in-the-loop processes, etc."+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "jun2024_py310",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.14"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}