Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

added the log size increase and update the documentation for the signatures (#1745)

sushmanth reddy92c440420589
diff --git a/docs/docs/building-blocks/2-signatures.md b/docs/docs/building-blocks/2-signatures.md
index bca787e93..43302ebad 100644
--- a/docs/docs/building-blocks/2-signatures.md
+++ b/docs/docs/building-blocks/2-signatures.md
@@ -8,14 +8,12 @@ When we assign tasks to LMs in DSPy, we specify the behavior we need as a Signat
**A signature is a declarative specification of input/output behavior of a DSPy module.** Signatures allow you to tell the LM _what_ it needs to do, rather than specify _how_ we should ask the LM to do it.
-
You're probably familiar with function signatures, which specify the input and output arguments and their types. DSPy signatures are similar, but the differences are that:
-- While typical function signatures just _describe_ things, DSPy Signatures _define and control the behavior_ of modules.
+- While typical function signatures just _describe_ things, DSPy Signatures _declare and initialize the behavior_ of modules.
- The field names matter in DSPy Signatures. You express semantic roles in plain English: a `question` is different from an `answer`, a `sql_query` is different from `python_code`.
-
## Why should I use a DSPy Signature?
**tl;dr** For modular and clean code, in which LM calls can be optimized into high-quality prompts (or automatic finetunes).
@@ -24,23 +22,21 @@ You're probably familiar with function signatures, which specify the input and o
Writing signatures is far more modular, adaptive, and reproducible than hacking at prompts or finetunes. The DSPy compiler will figure out how to build a highly-optimized prompt for your LM (or finetune your small LM) for your signature, on your data, and within your pipeline. In many cases, we found that compiling leads to better prompts than humans write. Not because DSPy optimizers are more creative than humans, but simply because they can try more things and tune the metrics directly.
-
## **Inline** DSPy Signatures
-Signatures can be defined as a short string, with argument names that define semantic roles for inputs/outputs.
+Signatures can be defined as a short string, with argument names and optional types that define semantic roles for inputs/outputs.
-1. Question Answering: `"question -> answer"`
+1. Question Answering: `"question -> answer"`, which is equivalent to `"question: str -> answer: str"` as the default type is always `str`
-2. Sentiment Classification: `"sentence -> sentiment"`
+2. Sentiment Classification: `"sentence -> sentiment: bool"`, e.g. `True` if positive
3. Summarization: `"document -> summary"`
-Your signatures can also have multiple input/output fields.
+Your signatures can also have multiple input/output fields with types:
-4. Retrieval-Augmented Question Answering: `"context, question -> answer"`
-
-5. Multiple-Choice Question Answering with Reasoning: `"question, choices -> reasoning, selection"`
+4. Retrieval-Augmented Question Answering: `"context: list[str], question: str -> answer: str"`
+5. Multiple-Choice Question Answering with Reasoning: `"question, choices: list[str] -> reasoning: str, selection: int"`
**Tip:** For fields, any valid variable names work! Field names should be semantically meaningful, but start simple and don't prematurely optimize keywords! Leave that kind of hacking to the DSPy compiler. For example, for summarization, it's probably fine to say `"document -> summary"`, `"text -> gist"`, or `"long_context -> tldr"`.
@@ -50,15 +46,14 @@ Your signatures can also have multiple input/output fields.
```python
sentence = "it's a charming and often affecting journey." # example from the SST-2 dataset.
-classify = dspy.Predict('sentence -> sentiment')
+classify = dspy.Predict('sentence -> sentiment: bool') # we'll see an example with Literal[] later
classify(sentence=sentence).sentiment
```
**Output:**
```text
-'Positive'
+True
```
-
### Example B: Summarization
```python
@@ -70,22 +65,22 @@ response = summarize(document=document)
print(response.summary)
```
-**Output:**
+**Possible Output:**
```text
The 21-year-old Lee made seven appearances and scored one goal for West Ham last season. He had loan spells in League One with Blackpool and Colchester United, scoring twice for the latter. He has now signed a contract with Barnsley, but the length of the contract has not been revealed.
```
-
Many DSPy modules (except `dspy.Predict`) return auxiliary information by expanding your signature under the hood.
+For example, `dspy.ChainOfThought` also adds a `reasoning` field that includes the LM's reasoning before it generates the output `summary`.
For example, `dspy.ChainOfThought` also adds a `reasoning` field that includes the LM's reasoning before it generates the output `summary`.
```python
-print("Rationale:", response.reasoning)
+print("Reasoning:", response.reasoning)
```
-**Output:**
+**Possible Output:**
```text
-Rationale: produce the summary. We need to highlight the key points about Lee's performance for West Ham, his loan spells in League One, and his new contract with Barnsley. We also need to mention that his contract length has not been disclosed.
+Reasoning: We need to highlight Lee's performance for West Ham, his loan spells in League One, and his new contract with Barnsley. We also need to mention that his contract length has not been disclosed.
```
## **Class-based** DSPy Signatures
@@ -98,44 +93,41 @@ For some advanced tasks, you need more verbose signatures. This is typically to:
3. Supply constraints on an output field, expressed as a `desc` keyword argument for `dspy.OutputField`.
-
### Example C: Classification
-Notice how the docstring contains (minimal) instructions, which in this case are necessary to have a fully-defined task.
-
-Some optimizers in DSPy, like `COPRO`, can take this simple docstring and then generate more effective variants if needed.
-
```python
+from typing import Literal
+
class Emotion(dspy.Signature):
- """Classify emotion among sadness, joy, love, anger, fear, surprise."""
+ """Classify emotion."""
- sentence = dspy.InputField()
- sentiment = dspy.OutputField()
+ sentence: str = dspy.InputField()
+ sentiment: Literal['sadness', 'joy', 'love', 'anger', 'fear', 'surprise'] = dspy.OutputField()
sentence = "i started feeling a little vulnerable when the giant spotlight started blinding me" # from dair-ai/emotion
classify = dspy.Predict(Emotion)
classify(sentence=sentence)
```
-**Output:**
+**Possible Output:**
```text
Prediction(
- sentiment='Fear'
+ sentiment='fear'
)
```
**Tip:** There's nothing wrong with specifying your requests to the LM more clearly. Class-based Signatures help you with that. However, don't prematurely tune the keywords of your signature by hand. The DSPy optimizers will likely do a better job (and will transfer better across LMs).
-
### Example D: A metric that evaluates faithfulness to citations
```python
class CheckCitationFaithfulness(dspy.Signature):
"""Verify that the text is based on the provided context."""
- context = dspy.InputField(desc="facts here are assumed to be true")
- text = dspy.InputField()
- faithfulness = dspy.OutputField(desc="True/False indicating if text is faithful to context")
+ context: str = dspy.InputField(desc="facts here are assumed to be true")
+ text: str = dspy.InputField()
+ faithfulness: bool = dspy.OutputField()
+ evidence: dict[str, list[str]] = dspy.OutputField(desc="Supporting evidence for claims")
context = "The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page."
@@ -144,17 +136,17 @@ text = "Lee scored 3 goals for Colchester United."
faithfulness = dspy.ChainOfThought(CheckCitationFaithfulness)
faithfulness(context=context, text=text)
```
-**Output:**
+**Possible Output:**
```text
Prediction(
- reasoning="produce the faithfulness. We know that Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. However, there is no mention of him scoring three goals for Colchester United.",
- faithfulness='False'
+ reasoning="Let's check the claims against the context. The text states Lee scored 3 goals for Colchester United, but the context clearly states 'He scored twice for the U's'. This is a direct contradiction.",
+ faithfulness=False,
+ evidence={'goal_count': ["scored twice for the U's"]}
)
```
-
## Using signatures to build modules & compiling them
-While signatures are convenient for prototyping with structured inputs/outputs, that's not the main reason to use them!
+While signatures are convenient for prototyping with structured inputs/outputs, that's not the only reason to use them!
You should compose multiple signatures into bigger [DSPy modules](/building-blocks/3-modules) and [compile these modules into optimized prompts](/building-blocks/6-optimizers#what-does-a-dspy-optimizer-tune-how-does-it-tune-them) and finetunes.
diff --git a/docs/docs/deep-dive/signature/executing-signatures.md b/docs/docs/deep-dive/signature/executing-signatures.md
deleted file mode 100644
index c1eeb0374..000000000
--- a/docs/docs/deep-dive/signature/executing-signatures.md
+++ /dev/null
@@ -1,122 +0,0 @@
----
-sidebar_position: 2
----
-
-# Executing Signatures
-
-So far we've understood what signatures are and how we can use them to craft our prompt, but now let's take a look at how to execute them.
-
-## Configuring LM
-
-To execute signatures, we require DSPy modules which are themselves dependent on a client connection to a language model (LM) client. DSPy supports LM APIs and local model hosting. Within this example, we will make use of the OpenAI client and configure the GPT-3.5 (`gpt-3.5-turbo`) model.
-
-```python
-turbo = dspy.OpenAI(model='gpt-3.5-turbo')
-dspy.settings.configure(lm=turbo)
-```
-
-## Executing Signatures
-
-Let's make use of the simplest module in DSPy - the `Predict` module that takes this signature as input to construct the prompt sent to the LM and generates a response for it.
-
-```python
-class BasicQA(dspy.Signature):
- """Answer questions with short factoid answers."""
-
- question = dspy.InputField()
- answer = dspy.OutputField(desc="often between 1 and 5 words")
-
-# Define the predictor.
-predictor = dspy.Predict(BasicQA)
-
-# Call the predictor on a particular input.
-pred = predictor(question=devset[0].question)
-
-# Print the input and the prediction.
-print(f"Question: {devset[0].question}")
-print(f"Predicted Answer: {pred.answer}")
-print(f"Actual Answer: {devset[0].answer}")
-```
-**Output:**
-```text
-Question: Are both Cangzhou and Qionghai in the Hebei province of China?
-Predicted Answer: No.
-Actual Answer: no
-```
-
-The `Predict` module generates a response via the LM we configured above and executes the prompt crafted by the signature. This returns the output i.e. `answer` which is present in the object returned by the `predictor` and can be accessed via `.` operator.
-
-## Inspecting Output
-
-Let's dive deeper into DSPy uses our signature to build up the prompt, which we can do through the `inspect_history` method on the configured LM following the program's execution. This method returns the last `n` prompts executed by LM.
-
-```python
-turbo.inspect_history(n=1)
-```
-**Output:**
-```text
-Answer questions with short factoid answers.
-
----
-
-Follow the following format.
-
-Question: ${question}
-Answer: often between 1 and 5 words
-
----
-
-Question: Are both Cangzhou and Qionghai in the Hebei province of China?
-Answer: No.
-```
-
-Additionally, if you want to store or use this prompt, you can access the `history` attribute of the LM object, which stores a list of dictionaries containing respective `prompt:response` entries for each LM generation.
-
-```python
-turbo.history[0]
-```
-**Output:**
-```text
-{'prompt': "Answer questions with short factoid answers.\n\n---\n\nFollow the following format.\n\nQuestion: ${question}\nQuestion's Answer: often between 1 and 5 words\n\n---\n\nQuestion: Are both Cangzhou and Qionghai in the Hebei province of China?\nQuestion's Answer:",
- 'response': <OpenAIObject chat.completion id=chatcmpl-8kCPsxikpVpmSaxdGLUIqubFZS05p at 0x7c3ba41fa840> JSON: {
- "id": "chatcmpl-8kCPsxikpVpmSaxdGLUIqubFZS05p",
- "object": "chat.completion",
- "created": 1706021508,
- "model": "gpt-3.5-turbo-0613",
- "choices": [
- {
- "index": 0,
- "message": {
- "role": "assistant",
- "content": "No."
- },
- "logprobs": null,
- "finish_reason": "stop"
- }
- ],
- "usage": {
- "prompt_tokens": 64,
- "completion_tokens": 2,
- "total_tokens": 66
- },
- "system_fingerprint": null
- },
- 'kwargs': {'stringify_request': '{"temperature": 0.0, "max_tokens": 150, "top_p": 1, "frequency_penalty": 0, "presence_penalty": 0, "n": 1, "model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Answer questions with short factoid answers.\\n\\n---\\n\\nFollow the following format.\\n\\nQuestion: ${question}\\nQuestion\'s Answer: often between 1 and 5 words\\n\\n---\\n\\nQuestion: Are both Cangzhou and Qionghai in the Hebei province of China?\\nQuestion\'s Answer:"}]}'},
- 'raw_kwargs': {}}
-```
-
-## How `Predict` works?
-
-The output of predictor is a `Prediction` class object which mirrors the `Example` class with additional functionalities for LM completion interactivity.
-
-How does `Predict` module actually 'predict' though? Here is a step-by-step breakdown:
-
-1. A call to the predictor will get executed in `__call__` method of `Predict` Module which executes the `forward` method of the class.
-
-2. In `forward` method, DSPy initializes the signature, LM call parameters and few-shot examples, if any.
-
-3. The `_generate` method formats the few shots example to mirror the signature and uses the LM object we configured to generate the output as a `Prediction` object.
-
-In case you are wondering how the prompt is constructed, the DSPy Signature framework internally handles the prompt structure, utilizing the DSP Template primitive to craft the prompt.
-
-Predict gives you a predefined pipeline to execute signature which is nice but you can build much more complicated pipelines with this by creating custom Modules.
diff --git a/docs/docs/deep-dive/signature/img/class_based_prompt_creation.png b/docs/docs/deep-dive/signature/img/class_based_prompt_creation.png
deleted file mode 100644
index 3972852a5..000000000
Binary files a/docs/docs/deep-dive/signature/img/class_based_prompt_creation.png and /dev/null differ
diff --git a/docs/docs/deep-dive/signature/img/dspy_signatures.png b/docs/docs/deep-dive/signature/img/dspy_signatures.png
deleted file mode 100644
index 5cfac02e4..000000000
Binary files a/docs/docs/deep-dive/signature/img/dspy_signatures.png and /dev/null differ
diff --git a/docs/docs/deep-dive/signature/img/prompt_creation.png b/docs/docs/deep-dive/signature/img/prompt_creation.png
deleted file mode 100644
index bd2712412..000000000
Binary files a/docs/docs/deep-dive/signature/img/prompt_creation.png and /dev/null differ
diff --git a/docs/docs/stylesheets/extra.css b/docs/docs/stylesheets/extra.css
new file mode 100644
index 000000000..572e0cab0
--- /dev/null
+++ b/docs/docs/stylesheets/extra.css
@@ -0,0 +1,33 @@
+/* Custom styles for logo */
+.md-logo {
+ width: auto !important;
+ height: 1.5rem !important; /* Reduced from 1.8rem */
+ padding: 0 !important;
+ margin: 0 !important;
+}
+
+.md-logo img {
+ width: auto !important;
+ height: 1.5rem !important; /* Reduced from 1.8rem */
+ object-fit: contain !important;
+}
+
+/* Adjust header to accommodate logo */
+.md-header__inner {
+ padding: 0.5rem !important;
+}
+
+/* Responsive adjustments */
+@media screen and (min-width: 76.25em) {
+ .md-logo {
+ height: 1.8rem !important; /* Reduced from 2.2rem */
+ }
+ .md-logo img {
+ height: 1.8rem !important; /* Reduced from 2.2rem */
+ }
+}
+
+/* Dark mode specific adjustments */
+[data-md-color-scheme="slate"] .md-logo img {
+ filter: brightness(0.9);
+}
\ No newline at end of file
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index e487236b9..526b184d8 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -1,4 +1,4 @@
-site_name: DSPy
+site_name: DSPy Documentation
repo_url: https://github.com/stanfordnlp/dspy
repo_name: stanfordnlp/dspy
@@ -17,7 +17,6 @@ nav:
- Signatures:
- Overview: building-blocks/2-signatures.md
- Understanding Signatures: deep-dive/signature/understanding-signatures.md
- - Executing Signatures: deep-dive/signature/executing-signatures.md
- Data Handling:
- Overview: building-blocks/4-data.md
- Examples in DSPy: deep-dive/data-handling/examples.md
@@ -81,12 +80,27 @@ nav:
- Summarization: tutorials/summarization.md
- Additional Resources: tutorials/other_tutorial.md
- Examples: tutorials/examples.md
- - Community:
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - '' : ''
+ - "Community":
- Community Resources: community/community-resources.md
- Use Cases: community/use-cases.md
- Roadmap: roadmap.md
- Contributing: community/how-to-contribute.md
- - FAQ:
+ - "FAQ":
- FAQ: faqs.md
- Cheatsheet: cheatsheet.md
@@ -127,6 +141,12 @@ theme:
logo: static/img/dspy_logo.png
favicon: static/img/logo.png
+
+extra_css:
+ - stylesheets/extra.css
+
+
+
plugins:
- social
- search
@@ -183,7 +203,6 @@ plugins:
# Signatures
'docs/deep-dive/signature/understanding-signatures.md': 'deep-dive/signature/understanding-signatures.md'
- 'docs/deep-dive/signature/executing-signatures.md': 'deep-dive/signature/executing-signatures.md'
# Modules
'docs/deep-dive/modules/predict.md': 'deep-dive/modules/predict.md'
@@ -242,6 +261,9 @@ plugins:
'docs/dspy-usecases.md': 'dspy-usecases.md'
+
+
+
extra:
social:
- icon: fontawesome/brands/github