Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

updated faqs and other todos

Arnav Singhvi7455bac77b01
diff --git a/docs/building-blocks/1-language_models.md b/docs/building-blocks/1-language_models.md
index 9ba6f59ee..87d772628 100644
--- a/docs/building-blocks/1-language_models.md
+++ b/docs/building-blocks/1-language_models.md
@@ -122,7 +122,11 @@ for idx in range(5):
## Remote LMs.
-These models are managed services. You just need to sign up and obtain an API key.
+These models are managed services. You just need to sign up and obtain an API key. Calling any of the remote LMs below assumes authentication and mirrors the following format for setting up the LM:
+
+```python
+lm = dspy.{provider_listed_below}(model="your model", model_request_kwargs="...")
+```
1. `dspy.OpenAI` for GPT-3.5 and GPT-4.
@@ -139,29 +143,33 @@ You need to host these models on your own GPU(s). Below, we include pointers for
1. `dspy.HFClientTGI`: for HuggingFace models through the Text Generation Inference (TGI) system. [Tutorial: How do I install and launch the TGI server?](/api/hosting_language_models_locally/TGI)
-2. `dspy.HFClientVLLM`: for HuggingFace models through vLLM. [Tutorial: How do I install and launch the vLLM server?](/api/hosting_language_models_locally/vLLM)
-
-3. `dspy.HFModel` (experimental)
+```python
+tgi_llama2 = dspy.HFClientTGI(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")
+```
-4. `dspy.Ollama` (experimental)
+2. `dspy.HFClientVLLM`: for HuggingFace models through vLLM. [Tutorial: How do I install and launch the vLLM server?](/api/hosting_language_models_locally/vLLM)
-5. `dspy.ChatModuleClient` (experimental): [How do I install and use MLC?](/api/hosting_language_models_locally/MLC)
+```python
+vllm_llama2 = dspy.HFClientVLLM(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")
+```
-6. `dspy.HFModel` (experimental)
+3. `dspy.HFModel` (experimental) [Tutorial: How do I initialize models using HFModel](/api/hosting_language_models_locally/HFModel.md)
-7. 7. `dspy.Ollama` (experimental) for open source models through [Ollama](https://ollama.com). [Tutorial: How do I install and use Ollama on a local computer?](/api/hosting_language_models_locally/Ollama.md)\n",
+```python
+llama = dspy.HFModel(model = 'meta-llama/Llama-2-7b-hf')
+```
-If there are other clients you want added, let us know or open a PR! Read the doc page on how to create a new LM client.
+4. `dspy.Ollama` (experimental) for open source models through [Ollama](https://ollama.com). [Tutorial: How do I install and use Ollama on a local computer?](/api/hosting_language_models_locally/Ollama.md)\n",
+```python
+mistral_ollama = dspy.OllamaLocal(model='mistral')
+```
-<!-- TODO: Usage examples for these all.
+5. `dspy.ChatModuleClient` (experimental): [How do I install and use MLC?](/api/hosting_language_models_locally/MLC)
```python
+model = 'dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1'
+model_path = 'dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so'
-# cohere = dspy.Cohere(...)
-# anyscale = dspy.Anyscale(...)
-# together = dspy.Together(...)
-# ollama = dspy.OllamaLocal(...)
-# tgi_llama2 = dspy.HFClientTGI(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")
-
-``` -->
\ No newline at end of file
+llama = dspy.ChatModuleClient(model=model, model_path=model_path)
+```
\ No newline at end of file
diff --git a/docs/building-blocks/solving_your_task.md b/docs/building-blocks/solving_your_task.md
index 14873f198..332500275 100644
--- a/docs/building-blocks/solving_your_task.md
+++ b/docs/building-blocks/solving_your_task.md
@@ -8,7 +8,7 @@ Using DSPy well for solving a new task is just doing good machine learning with
What this means is that it's an iterative process. You make some initial choices, which will be sub-optimal, and then you refine them incrementally.
-As we discuss below, you will define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers (`modules`) to use, giving each layer a `signature` (input/output spec), and then calling your modules freely in your Python code. Lastly, you use a DSPy `optimizer` to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM.
+As we discuss below, you will define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers [(`modules`)](/docs/building-blocks/3-modules.md) to use, giving each layer a [`signature` (input/output spec)](/docs/building-blocks/2-signatures.md), and then calling your modules freely in your Python code. Lastly, you use a DSPy [`optimizer`](/docs/building-blocks/6-optimizers.md) to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM.
## 1) Define your task.
@@ -31,7 +31,7 @@ What should your DSPy program do? Can it just be a simple chain-of-thought step?
Is there a typical workflow for solving your problem in multiple well-defined steps? Or do you want a fully open-ended LM (or open-ended tool use with agents) for your task?
-Think about this space but always start simple. Almost every task should probably start with just a single chain-of-thought module, and then add complexity incrementally as you go.
+Think about this space but always start simple. Almost every task should probably start with just a single [dspy.ChainofThought](/api/modules/ChainOfThought.md) module, and then add complexity incrementally as you go.
Then write your (initial) DSPy program. Again: start simple, and let the next few steps guide any complexity you will add.
@@ -39,7 +39,7 @@ Then write your (initial) DSPy program. Again: start simple, and let the next fe
By this point, you probably have a few examples of the task you're trying to solve.
-Run them through your pipeline. Consider using a large and powerful LM at this point, or a couple of different LMs, just to understand what's possible. (DSPy will make swapping these LMs pretty easy.)
+Run them through your pipeline. Consider using a large and powerful LM at this point, or a couple of different LMs, just to understand what's possible. (DSPy will make swapping these LMs pretty easy - [LM Guide](/docs/building-blocks/1-language_models.md).)
At this point, you're still using your pipeline zero-shot, so it will be far from perfect. DSPy will help you optimize the instructions, few-shot examples, and even weights of your LM calls below, but understanding where things go wrong in zero-shot usage will go a long way.
@@ -47,7 +47,7 @@ Record the interesting (both easy and hard) examples you try: even if you don't
## 4) Define your data.
-Now it's time to more formally declare your training and validation data for DSPy evaluation and optimization.
+Now it's time to more formally declare your training and validation data for DSPy evaluation and optimization - [Data Guide](/docs/building-blocks/4-data.md).
You can use DSPy optimizers usefully with as few as 10 examples, but having 50-100 examples (or even better, 300-500 examples) goes a long way.
@@ -62,7 +62,7 @@ If there's data whose licenses are permissive enough, we suggest you use them. O
What makes outputs from your system good or bad? Invest in defining metrics and improving them over time incrementally. It's really hard to consistently improve what you aren't able to define.
-A metric is just a function that will take examples from your data and take the output of your system, and return a score that quantifies how good the output is.
+A metric is just a function that will take examples from your data and take the output of your system, and return a score that quantifies how good the output is - [Metric Guide](/docs/building-blocks/5-metrics.md).
For simple tasks, this could be just "accuracy" or "exact match" or "F1 score". This may be the case for simple classification or short-form QA tasks.
@@ -79,17 +79,17 @@ Look at the outputs and the metric scores. This will probably allow you to spot
## 7) Compile with a DSPy optimizer.
-Given some data and a metric, we can now optimize the program you built.
+Given some data and a metric, we can now optimize the program you built - [Optimizer Guide](/docs/building-blocks/6-optimizers.md).
DSPy includes many optimizers that do different things. Remember: DSPy optimizers will create examples of each step, craft instructions, and/or update LM weights. In general, you don't need to have labels for your pipeline steps, but your data examples need to have input values and whatever labels your metric requires (e.g., no labels if your metric is reference-free, but final output labels otherwise in most cases).
Here's the general guidance on getting started:
-* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`.
+* If you have very little data, e.g. 10 examples of your task, use [`BootstrapFewShot`](/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx)
-* If you have slightly more data, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.
+* If you have slightly more data, e.g. 50 examples of your task, use [`BootstrapFewShotWithRandomSearch`](/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx).
-* If you have more data than that, e.g. 300 examples or more, use `BayesianSignatureOptimizer`.
+* If you have more data than that, e.g. 300 examples or more, use [`BayesianSignatureOptimizer`](/docs/deep-dive/teleprompter/signature-optimizer.mdx).
* If you have been able to use one of these with a large LM (e.g., 7B parameters or above) and need a very efficient program, compile that down to a small LM with `BootstrapFinetune`.
@@ -97,13 +97,8 @@ Here's the general guidance on getting started:
At this point, you are either very happy with everything (we've seen quite a few people get it right on first try with DSPy) or, more likely, you've made a lot of progress but you don't like something about the final program or the metric.
-At this point, go back to step 1 and revisit the major questions. Did you define your task well? Do you need to collect (or find online) more data for your problem? Do you want to update your metric? And do you want to use a more sophisticated optimizer? Do you need to consider advanced features like DSPy Assertions? Or, perhaps most importantly, do you want to add some more complexity or steps in your DSPy program itself? Do you want to use multiple optimizers in a sequence?
+At this point, go back to step 1 and revisit the major questions. Did you define your task well? Do you need to collect (or find online) more data for your problem? Do you want to update your metric? And do you want to use a more sophisticated optimizer? Do you need to consider advanced features like [DSPy Assertions](/docs/deep-dive/modules/assertions.mdx)? Or, perhaps most importantly, do you want to add some more complexity or steps in your DSPy program itself? Do you want to use multiple optimizers in a sequence?
Iterative development is key. DSPy gives you the pieces to do that incrementally: iterating on your data, your program structure, your assertions, your metric, and your optimization steps.
-Optimizing complex LM programs is an entirely new paradigm that only exists in DSPy at the time of writing, so naturally the norms around what to do are still emerging. If you need help, we recently created a [Discord server](https://discord.gg/VzS6RHHK6F) for the community.
-
-
-#### TODOs for this page
-
-1. Include links to a guide for each step with more concrete code and pointers.
+Optimizing complex LM programs is an entirely new paradigm that only exists in DSPy at the time of writing, so naturally the norms around what to do are still emerging. If you need help, we recently created a [Discord server](https://discord.gg/VzS6RHHK6F) for the community.
\ No newline at end of file
diff --git a/docs/deep-dive/modules/guide.mdx b/docs/deep-dive/modules/guide.mdx
index eb511f39c..04792dd8b 100644
--- a/docs/deep-dive/modules/guide.mdx
+++ b/docs/deep-dive/modules/guide.mdx
@@ -14,9 +14,7 @@ Remember that **DSPy program** is just Python code that calls one or more **DSPy
A **DSPy module** is a building block for programs that use LMs.
-#TODO link signatures docs
-<!-- - Each built-in module abstracts a **prompting technique** (like chain of thought or ReAct). Crucially, they are generalized to handle any [DSPy Signature](). -->
-- Each built-in module abstracts a **prompting technique** (like chain of thought or ReAct). Crucially, they are generalized to handle any DSPy Signature.
+- Each built-in module abstracts a **prompting technique** (like chain of thought or ReAct). Crucially, they are generalized to handle any [DSPy Signature](/docs/building-blocks/2-signatures.md).
- A DSPy module has **learnable parameters** (i.e., the little pieces comprising the prompt and the LM weights) and can be invoked (called) to process inputs and return outputs.
@@ -29,7 +27,7 @@ TODO. I typically take this as self-evident, but I'll spell it out here.
```python
# Install `dspy-ai` if needed. Then set up a default language model.
-# TODO: Add a graceful line for OPENAI_API_KEY.
+#openai.api_key = os.getenv("OPENAI_API_KEY")
try: import dspy
except ImportError:
@@ -60,10 +58,7 @@ We also have some function-style modules:
Let's start with the most fundamental one, `dspy.Predict`. Internally, all of the others are just built using it!
-#TODO link signatures docs
-<!-- We'll assume you are already at least a little familiar with [DSPy signatures](), which are declarative specs for defining the behavior of any module we use in DSPy. -->
-We'll assume you are already at least a little familiar with DSPy signatures, which are declarative specs for defining the behavior of any module we use in DSPy.
-
+We'll assume you are already at least a little familiar with [DSPy signatures](/docs/building-blocks/2-signatures.md), which are declarative specs for defining the behavior of any module we use in DSPy.
To use a module, we first **declare** it by giving it a signature. Then we **call** the module with the input arguments, and extract the output fields!
diff --git a/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx b/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx
index c2d85c9a1..5d30cdd09 100644
--- a/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx
+++ b/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx
@@ -10,7 +10,7 @@ When compiling a DSPy program, we generally invoke a teleprompter, which is an o
## Setting up a Sample Pipeline
-We'll be making a basic answer genration pipeline over GSM8K dataset that we saw in the [Minimal Example](/docs/quick-start/minimal-example), we won't be changing anything in it! So let's start by configuring the LM which will be OpenAI LM client with `gpt-3.5-turbo` as the LLM in use.
+We'll be making a basic answer generation pipeline over GSM8K dataset that we saw in the [Minimal Example](/docs/quick-start/minimal-example), we won't be changing anything in it! So let's start by configuring the LM which will be OpenAI LM client with `gpt-3.5-turbo` as the LLM in use.
```python
import dspy
@@ -107,11 +107,11 @@ cot_compiled.save('turbo_gsm8k.json')
`BootStrapFewShot` starts by doing this only, it starts by:
1. Initializing a student program which which is the one we are optimizing and a teacher program which unless specified otherwise is a clone of the student.
-2. Then to the teacher it add the demos by using `LabeledFewShot` telepromtpter.
+2. Then to the teacher it add the demos by using `LabeledFewShot` teleprompter.
3. Mappings are created between the names of predictors and their corresponding instances in both student and teacher models.
-4. The maximum number of bootstrap demonstrations (max_bootsraps) is determined. This limits the amount of initial training data generated.
+4. The maximum number of bootstrap demonstrations (max_bootstraps) is determined. This limits the amount of initial training data generated.
5. The process iterates over each example in the training set. For each example, the method checks if the maximum number of bootstraps has been reached. If so, the process stops.
diff --git a/docs/faqs.md b/docs/faqs.md
index 392bb6b18..dff2f4598 100644
--- a/docs/faqs.md
+++ b/docs/faqs.md
@@ -12,7 +12,7 @@ The **DSPy** philosophy and abstraction differ significantly from other librarie
**DSPy vs. application development libraries like LangChain, LlamaIndex** LangChain and LlamaIndex target high-level application development; they offer _batteries-included_, pre-built application modules that plug in with your data or configuration. If you'd be happy to use a generic, off-the-shelf prompt for question answering over PDFs or standard text-to-SQL, you will find a rich ecosystem in these libraries. **DSPy** doesn't internally contain hand-crafted prompts that target specific applications. Instead, **DSPy** introduces a small set of much more powerful and general-purpose modules _that can learn to prompt (or finetune) your LM within your pipeline on your data_. when you change your data, make tweaks to your program's control flow, or change your target LM, the **DSPy compiler** can map your program into a new set of prompts (or finetunes) that are optimized specifically for this pipeline. Because of this, you may find that **DSPy** obtains the highest quality for your task, with the least effort, provided you're willing to implement (or extend) your own short program. In short, **DSPy** is for when you need a lightweight but automatically-optimizing programming model — not a library of predefined prompts and integrations. If you're familiar with neural networks: This is like the difference between PyTorch (i.e., representing **DSPy**) and HuggingFace Transformers (i.e., representing the higher-level libraries).
-**DSPy vs. generation control libraries like Guidance, LMQL, RELM, Outlines** These are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression. This is very useful in many settings, but it's generally focused on low-level, structured control of a single LM call. It doesn't help ensure the JSON (or structured output) you get is going to be correct or useful for your task. In contrast, **DSPy** automatically optimizes the prompts in your programs to align them with various task needs, which may also include producing valid structured ouputs. That said, we are considering allowing **Signatures** in **DSPy** to express regex-like constraints that are implemented by these libraries.
+**DSPy vs. generation control libraries like Guidance, LMQL, RELM, Outlines** These are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression. This is very useful in many settings, but it's generally focused on low-level, structured control of a single LM call. It doesn't help ensure the JSON (or structured output) you get is going to be correct or useful for your task. In contrast, **DSPy** automatically optimizes the prompts in your programs to align them with various task needs, which may also include producing valid structured outputs. That said, we are considering allowing **Signatures** in **DSPy** to express regex-like constraints that are implemented by these libraries.
## Basic Usage
@@ -22,7 +22,7 @@ The **DSPy** philosophy and abstraction differ significantly from other librarie
**What do DSPy optimizers tune?** Or, _what does compiling actually do?_ Each optimizer is different, but they all seek to maximize a metric on your program by updating prompts or LM weights. Current DSPy `optimizers` can inspect your data, simulate traces through your program to generate good/bad examples of each step, propose or refine instructions for each step based on past results, finetune the weights of your LM on self-generated examples, or combine several of these to improve quality or cut cost. We'd love to merge new optimizers that explore a richer space: most manual steps you currently go through for prompt engineering, "synthetic data" generation, or self-improvement can probably generalized into a DSPy optimizer that acts on arbitrary LM programs.
-Other FAQs (TODO). We welcome PRs to add formal answers to each of these here. You will find the answer in existing issues, tutorials, or the papers for all or most of these.
+Other FAQs. We welcome PRs to add formal answers to each of these here. You will find the answer in existing issues, tutorials, or the papers for all or most of these.
- **How do I get multiple outputs?**
@@ -30,6 +30,8 @@ You can specify multiple output fields. For the short-form signature, you can li
- **How can I work with long responses?**
+You can specify the generation of long responses as a `dspy.OutputField`. To ensure comprehensive checks of the content within the long-form generations, you can indicate the inclusion of citations per referenced context. Such constraints such as response length or citation inclusion can be stated through Signature descriptions, or concretely enforced through DSPy Assertions. Check out the [LongFormQA notebook](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/longformqa/longformqa_assertions.ipynb) to learn more about **Generating long-form length responses to answer questions**.
+
- **How do I define my own metrics? Can metrics return a float?**
You can define metrics as simply Python functions that process model generations and evaluate them based on user-defined requirements. Metrics can compare existent data (e.g. gold labels) to model predictions or they can be used to assess various components of an output using validation feedback from LMs (e.g. LLMs-as-Judges). Metrics can return `bool`, `int`, and `float` types scores. Check out the official [Metrics docs](/docs/building-blocks/5-metrics.md) to learn more about defining custom metrics and advanced evaluations using AI feedback and/or DSPy programs.
@@ -59,7 +61,13 @@ cot.load('compiled_cot_gsm8k.json')
```
- **How do I export for deployment?**
+
+Exporting DSPy programs is simply saving them as highlighted above!
+
- **How do I search my own data?**
+
+Open source libraries such as [RAGautouille](https://github.com/bclavie/ragatouille) enable you to search for your own data through advanced retrieval models like ColBERT with tools to embdeed and index documents. Feel free to integrate such libraries to create searchable datasets while developing your DSPy programs!
+
- **How do I turn off the cache? How do I export the cache?**
You can turn off the cache by setting the [`cache_turn_on` flag to `False`](https://github.com/stanfordnlp/dspy/blob/9d8a40c477b9dd6dcdc007647b5b9ddad2b5657a/dsp/modules/cache_utils.py#L10).
@@ -73,6 +81,9 @@ Your local cache will be saved to the global env directory `os.environ["DSP_NOTE
You can parallelize DSPy programs during both compilation and evaluation by specifying multiple thread settings in the respective DSPy `optimizers` or within the `dspy.Evaluate` utility function.
- **How do freeze a module?**
+
+Modules can be frozen by setting their `._compiled` attribute to be True, indicating the module has gone through optimizer compilation and should not have its parameters adjusted. This is handled internally in optimizers such as `dspy.BootstrapFewShot` where the student program is ensured to be frozen before the teacher propagates the gathered few-shot demonstrations in the bootstrapping process.
+
- **How do I get JSON output?**
You can specify JSON-type descriptions in the `desc` field of the long-form signature `dspy.OutputField` (e.g. `output = dspy.OutputField(desc='key-value pairs')`).