Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

building blocks

Omar Khattab756ca9048708
diff --git a/docs/building-blocks/1-signatures.md b/docs/building-blocks/1-signatures.md
new file mode 100644
index 000000000..94607aff6
--- /dev/null
+++ b/docs/building-blocks/1-signatures.md
@@ -0,0 +1,168 @@
+---
+sidebar_position: 2
+---
+
+# [1] DSPy Signatures
+
+## 1) What is a DSPy Signature?
+
+When we assign tasks to LMs in DSPy, we specify the behavior we need as a Signature.
+
+**A signature is a declarative specification of input/output behavior of a DSPy module.**
+
+What does that mean? It means tht signatures allow you tell the LM _what_ it needs to do, rather than specify _how_ we should ask the LM to do it.
+
+
+## 2) Why are they called Signatures?
+
+You're probably familiar with function signatures, which specify the input and output arguments and their types.
+
+Ditto. DSPy signatures are similar, but the differences are that:
+
+- While typical function signatures just _describe_ things, DSPy Signatures _define and control the behavior_ of modules.
+
+- The field names matter in DSPy Signatures. You express semantic roles in plain English: a `question` is different from an `answer`, a `sql_query` is different from `python_code`.
+
+
+## 3) Why should I use a DSPy Signature?
+
+**tl;dr** For modular and clean code, in which LM calls can be optimized into high-quality prompts (or automatic finetunes).
+
+**Long Answer:** Most people coerce LMs to do tasks by hacking long, brittle prompts. Or by collecting/generating data for fine-tuning.
+
+Writing signatures is far more modular, adaptive, and reproducible than hacking at prompts or finetunes. The DSPy compiler will figure out how to build a highly-optimized prompt for your LM (or finetune your small LM) for your signature, on your data, and within your pipeline. In many cases, we found that compiling leads to better prompts than humans write. Not because DSPy optimizers are more creative than humans, but simply because they can try more things and tune the metrics directly.
+
+
+## 4) **Inline** DSPy Signatures
+
+Signatures can be defined as a short string, with argument names that define semantic roles for inputs/outputs.
+
+1. Question Answering: `"question -> answer"`
+
+2. Sentiment Classification: `"sentence -> sentiment"`
+
+3. Summarization: `"document -> summary"`
+
+Your signatures can also have multiple input/output fields.
+
+4. Retrieval-Augmented Question Answering: `"context, question -> answer"`
+
+5. Multiple-Choice Question Answering with Reasoning: `"question, choices -> reasoning, selection"`
+
+
+**Tip:** For fields, any valid variable names work! Field names should be semantically meaningful, but start simple and don't prematurely optimize keywords! Leave that kind of hacking to the DSPy compiler. For example, for summarization, it's probably fine to say `"document -> summary"`, `"text -> gist"`, or `"long_context -> tldr"`.
+
+
+### Example A: Sentiment Classification
+
+```python
+sentence = "it's a charming and often affecting journey." # example from the SST-2 dataset.
+
+classify = dspy.Predict('sentence -> sentiment')
+classify(sentence=sentence).sentiment
+```
+**Output:**
+```text
+'Positive'
+```
+
+
+### Example B: Summarization
+
+```python
+# Example from the XSum dataset.
+document = """The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page."""
+
+summarize = dspy.ChainOfThought('document -> summary')
+response = summarize(document=document)
+
+print(response.summary)
+```
+**Output:**
+```text
+The 21-year-old Lee made seven appearances and scored one goal for West Ham last season. He had loan spells in League One with Blackpool and Colchester United, scoring twice for the latter. He has now signed a contract with Barnsley, but the length of the contract has not been revealed.
+```
+
+
+Many DSPy modules (except `dspy.Predict`) return auxiliary information by expanding your signature under the hood.
+
+For example, `dspy.ChainOfThought` also adds a `rationale` field that includes the LM's reasoning before it generates the output `summary`.
+
+```python
+print("Rationale:", response.rationale)
+```
+**Output:**
+```text
+Rationale: produce the summary. We need to highlight the key points about Lee's performance for West Ham, his loan spells in League One, and his new contract with Barnsley. We also need to mention that his contract length has not been disclosed.
+```
+
+## 5) **Class-based** DSPy Signatures
+
+For some advanced tasks, you need more verbose signatures. This is typically to:
+
+1. Clarify something about the nature of the task (expressed below as a `docstring`).
+
+2. Supply hints on the nature of an input field, expressed as a `desc` keyword argument for `dspy.InputField`.
+
+3. Supply constraints on an output field, expressed as a `desc` keyword argument for `dspy.OutputField`.
+
+
+### Example C: Classification
+
+Notice how the docstring contains (minimal) instructions, which in this case are necessary to have a fully-defined task.
+
+Some optimizers in DSPy, like `SignatureOptimizer`, can take this simple docstring and then generate more effective variants if needed.
+
+```python
+class Emotion(dspy.Signature):
+ """Classify emotion among sadness, joy, love, anger, fear, surprise."""
+
+ sentence = dspy.InputField()
+ sentiment = dspy.OutputField()
+
+sentence = "i started feeling a little vulnerable when the giant spotlight started blinding me" # from dair-ai/emotion
+
+classify = dspy.Predict(Emotion)
+classify(sentence=sentence)
+```
+**Output:**
+```text
+Prediction(
+ sentiment='Fear'
+)
+```
+
+**Tip:** There's nothing wrong with specifying your requests to the LM more clearly. Class-based Signatures help you with that. However, don't prematurely tune the keywords of the your signature by hand. The DSPy optimizers will likely do a better job (and will transfer better across LMs).
+
+
+### Example D: A metric that evaluates faithfulness to citations
+
+```python
+class CheckCitationFaithfulness(dspy.Signature):
+ """Verify that the text is based on the provided context."""
+
+ context = dspy.InputField(desc="facts here are assumed to be true")
+ text = dspy.InputField()
+ faithfulness = dspy.OutputField(desc="True/False indicating if text is faithful to context")
+
+context = "The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page."
+
+text = "Lee scored 3 goals for Colchester United."
+
+faithfulness = dspy.ChainOfThought(CheckCitationFaithfulness)
+faithfulness(context=context, text=text)
+```
+**Output:**
+```text
+Prediction(
+ rationale="produce the faithfulness. We know that Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. However, there is no mention of him scoring three goals for Colchester United.",
+ faithfulness='False'
+)
+```
+
+
+## 6) Building modules & compiling them
+
+While signatures are covenient for prototyping with structured inputs/outputs, that's not the main reason to use them!
+
+You should compose multiple signatures into bigger [DSPy modules]() and [compile]() these modules into optimized prompts and finetunes.
diff --git a/docs/building-blocks/_category_.json b/docs/building-blocks/_category_.json
new file mode 100644
index 000000000..003f099b2
--- /dev/null
+++ b/docs/building-blocks/_category_.json
@@ -0,0 +1,8 @@
+{
+ "label": "DSPy Building Blocks",
+ "position": 3,
+ "link": {
+ "type": "generated-index",
+ "description": "DSPy introduces signatures (to abstract prompts), modules (to abstract prompting techniques), and optimizers that can tune the prompts (or weights) of modules."
+ }
+}
diff --git a/docs/solving_your_task.md b/docs/building-blocks/solving_your_task.md
similarity index 85%
rename from docs/solving_your_task.md
rename to docs/building-blocks/solving_your_task.md
index 45acbc87d..97ffacbdc 100644
--- a/docs/solving_your_task.md
+++ b/docs/building-blocks/solving_your_task.md
@@ -1,18 +1,19 @@
---
-sidebar_position: 3
+sidebar_position: 1
---
-# Overview: Solving Your Own Task in DSPy
+# Using DSPy in 7 Steps
-DSPy is built to allow you to build your own programs to solve your own tasks. It's not a drop-in replacement for any existing framework you might be using currently. It's a new paradigm for doing higher-level machine learning (ML) with language models.
+Using DSPy well for solving a new task is just doing good machine learning with LMs.
-The decisions you make in each stage will probably be "wrong" initially. That's fine, iterative development is at the core of idiomatic usage of DSPy.
+What this means is that it's an iterative process. You make some initial choices, which will be sub-optimal, and then you refine them incrementally.
+
+As we discuss below, you will define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers (`modules`) to use, giving each layer a `signature` (input/output spec), and then calling your modules freely in your Python code. Lastly, you use a DSPy `optimizer` to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM.
-**tl;dr** Using DSPy is an iterative process. You first define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers (`modules`) to use, giving each layer a `signature` (input/output spec), and then calling your modules freely in your Python code. Lastly, you use a DSPy `optimizer` to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM.
## 1) Define your task.
-You cannot use DSPy well if you can't define the problem you're trying to solve.
+You cannot use DSPy well if you haven't defined the problem you're trying to solve.
**Expected Input/Output Behavior:** Are you trying to build a chatbot over your data? A code assistant? A system for extracting information from papers? Or perhaps a translation system? Or a system for highlighting snippets from search results? Or a system to summarize information on a topic, with citations?
diff --git a/docs/components-in-dspy/_category_.json b/docs/components-in-dspy/_category_.json
index 398d3224f..61aeed075 100644
--- a/docs/components-in-dspy/_category_.json
+++ b/docs/components-in-dspy/_category_.json
@@ -1,8 +1,8 @@
{
- "label": "Components in DSPy",
+ "label": "Supplementary on Components in DSPy (by Herumb Shandilya)",
"position": 3,
"link": {
"type": "generated-index",
- "description": "DSPy introduces signatures (to abstract prompts), modules (to abstract prompting techniques), and optimizers that can tune the prompts (or weights) of modules.",
+ "description": "DSPy introduces signatures (to abstract prompts), modules (to abstract prompting techniques), and optimizers that can tune the prompts (or weights) of modules."
}
}
diff --git a/docs/tutorials/your_own_task.md b/docs/tutorials/your_own_task.md
deleted file mode 100644
index 479e0381f..000000000
--- a/docs/tutorials/your_own_task.md
+++ /dev/null
@@ -1,106 +0,0 @@
----
-sidebar_position: 3
----
-
-# [03] Solving Your Own Task in DSPy
-
-DSPy is built to allow you to build your own programs to solve your own tasks. It's not a drop-in replacement for any existing framework you might be using currently. It's a new paradigm for doing higher-level machine learning (ML) with language models.
-
-The decisions you make in each stage will probably be "wrong" initially. That's fine, iterative development is at the core of idiomatic usage of DSPy.
-
-**tl;dr** Using DSPy is an iterative process. You first define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers (`modules`) to use, giving each layer a `signature` (input/output spec), and then calling your modules freely in your Python code. Lastly, you use a DSPy `optimizer` to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM.
-
-## 1) Define your task.
-
-You cannot use DSPy well if you can't define the problem you're trying to solve.
-
-**Expected Input/Output Behavior:** Are you trying to build a chatbot over your data? A code assistant? A system for extracting information from papers? Or perhaps a translation system? Or a system for highlighting snippets from search results? Or a system to summarize information on a topic, with citations?
-
-It's often useful to come up with just 3-4 examples of the inputs and outputs of your program (e.g., questions and their answers, or topics and their summaries).
-
-If you need help thinking about your task, we recently created a [Discord server](https://discord.gg/VzS6RHHK6F) for the community.
-
-**Quality and Cost Specs:** You probably don't have infinite budget. Your final system can't be too expensive to run, and it should probably respond to users quickly enough.
-
-Take this as an opportunity to guess what kind of language model you'd like to use. Maybe GPT-3.5? Or a small open model, like Mistral-7B or Llama2-13B-chat? Or Mixtral? Or maybe you really need GPT-4-turbo? Or perhaps your resources are very constrained, and you want your final LM to be T5-base.
-
-## 2) Define your pipeline.
-
-What should your DSPy program do? Can it just be a simple chain-of-thought step? Or do you need the LM to use retrieval? Or maybe other tools, like a calculator or a calendar API?
-
-Is there a typical workflow for solving your problem in multiple well-defined steps? Or do you want a fully open-ended LM (or open-ended tool use with agents) for your task?
-
-Think about this space but always start simple. Almost every task should probably start with just a single chain-of-thought module, and then add complexity incrementally as you go.
-
-Then write your (initial) DSPy program. Again: start simple, and let the next few steps guide any complexity you will add.
-
-## 3) Explore a few examples.
-
-By this point, you probably have a few examples of the task you're trying to solve.
-
-Run them through your pipeline. Consider using a large and powerful LM at this point, or a couple of different LMs, just to understand what's possible. (DSPy will make swapping these LMs pretty easy.)
-
-At this point, you're still using your pipeline zero-shot, so it will be far from perfect. DSPy will help you optimize the instructions, few-shot examples, and even weights of your LM calls below, but understanding where things go wrong in zero-shot usage will go a long way.
-
-Record the interesting (both easy and hard) examples you try: even if you don't have labels, simply tracking the inputs you tried will be useful for DSPy optimizers below.
-
-## 4) Define your data.
-
-Now it's time to more formally declare your training and validation data for DSPy evaluation and optimization.
-
-You can use DSPy optimizers usefully with as few as 10 examples, but having 50-100 examples (or even better, 300-500 examples) goes a long way.
-
-How can you get examples like these? If your task is extremely unusual, please invest in preparing ~10 examples by hand. Often times, depending on your metric below, you just need inputs and not labels, so it's not that hard.
-
-However, chances are that your task is not actually that unique. You can almost always find somewhat adjacent datasets on, say, HuggingFace datasets or other forms of data that you can leverage here.
-
-If there's data whose licenses are permissive enough, we suggest you use them. Otherwise, you can also start using/deploying/demoing your system and collect some initial data that way.
-
-
-## 5) Define your metric.
-
-What makes outputs from your system good or bad? Invest in defining metrics and improving them over time incrementally. It's really hard to consistently improve what you aren't able to define.
-
-A metric is just a function that will take examples from your data and take the output of your system, and return a score that quantifies how good the output is.
-
-For simple tasks, this could be just "accuracy" or "exact match" or "F1 score". This may be the case for simple classification or short-form QA tasks.
-
-However, for most applications, your system will output long-form outputs. There, your metric should probably be a smaller DSPy program that checks multiple properties of the output (quite possibly using AI feedback from LMs).
-
-Getting this right on the first try is unlikely, but you should start with something simple and iterate. (If your metric is itself a DSPy program, notice that one of the most powerful ways to iterate is to compile (optimize) your metric itself. That's usually easy because the output of the metric is usually a simple value (e.g., a score out of 5) so the metric's metric is easy to define and optimize by collecting a few examples.)
-
-
-## 6) Collect preliminary "zero-shot" evaluations.
-
-Now that you have some data and a metric, run evaluation on your pipeline before any optimizer runs.
-
-Look at the outputs and the metric scores. This will probably allow you to spot any major issues, and it will define a baseline for your next step.
-
-## 6) Compile with a DSPy optimizer.
-
-Given some data and a metric, we can now optimize the program you built.
-
-DSPy includes many optimizers that do different things. Here's the general guidance on getting started:
-
-* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`.
-
-* If you have slightly more data, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.
-
-* If you have more data than that, e.g. 300 examples or more, use `BayesianSignatureOptimizer`.
-
-* If you have been able to use one of these with a large LM (e.g., 7B parameters or above) and need a very efficient program, compile that down to a small LM with `BootstrapFinetune`.
-
-## 7) Iterate.
-
-At this point, you are either very happy with everything (we've seen quite a few people get it right on first try with DSPy) or, more likely, you've made a lot of progress but you don't like something about the final program or the metric.
-
-At this point, go back to step 1 and revisit the major questions. Did you define your task well? Do you need to collect (or find online) more data for your problem? Do you want to update your metric? And do you want to use a more sophisticated optimizer? Do you need to consider advanced features like DSPy Assertions? Or, perhaps most importantly, do you want to add some more complexity or steps in your DSPy program itself? Do you want to use multiple optimizers in a sequence?
-
-Iterative development is key. DSPy gives you the pieces to do that incrementally: iterating on your data, your program structure, your assertions, your metric, and your optimization steps.
-
-Optimizing complex LM programs is an entirely new paradigm that only exists in DSPy at the time of writing, so naturally the norms around what to do are still emerging. If you need help, we recently created a [Discord server](https://discord.gg/VzS6RHHK6F) for the community.
-
-
-#### TODOs for this page
-
-1. Include links to a guide for each step with more concrete code and pointers.