DSPy: The framework for programming—not prompting—language models
Python
Improve docs (#1783)
Omar Khattab98c6ea57aae6
diff --git a/.gitignore b/.gitignoreindex d34d0c34e..b7aa2957c 100644--- a/.gitignore+++ b/.gitignore@@ -56,4 +56,5 @@ assertion.log/.devcontainer/.personalization.sh.mypy_cache-dummy.csv\ No newline at end of file+dummy.csv+docs/docs/**/*.json*\ No newline at end of filediff --git a/docs/docs/index.md b/docs/docs/index.mdindex 3fb1f8ede..41f1db15b 100644--- a/docs/docs/index.md+++ b/docs/docs/index.md@@ -1,6 +1,357 @@----template: home.html+sidebar_position: 1hide:- navigation- - toc+ # - toc+---++{ width="350", align=right }++# **DSPy**: _Programming_—not prompting—_Language Models_+++DSPy is the open-source framework for **building high-quality, modular AI systems** by _programming—rather than prompting—language models_. It provides abstractions and algorithms for **optimizing the prompts and weights** in LM programs, ranging from simple classifiers to sophisticated RAG pipelines and Agent loops.++Instead of writing brittle LM-specific prompts, you write compositional code and use DSPy optimizers to teach different models like `GPT-4o` or `Llama-3.2` to **deliver higher quality outputs** or avoid specific failure patterns. In essence, DSPy optimizers then _compile your high-level code_ into low-level computations, prompts, or weight updates that **align your LM with your program’s structure and metrics**.+++DSPy stands for Declarative Self-improving Python. This [recent lecture](https://www.youtube.com/watch?v=JEMYuzrKLUw) is a good conceptual introduction. Our [Discord server](https://discord.gg/XCGy2WDCQB) is a great place to meet the community, seek help, or start contributing.+++!!! info "Getting Started I: Install DSPy and set up your LM"++ ```bash+ > pip install -U dspy+ ```++ === "OpenAI"+ You can authenticate by setting the `OPENAI_API_KEY` env variable or passing `api_key` below.++ ```python linenums="1"+ import dspy+ lm = dspy.LM('openai/gpt-4o-mini', api_key='YOUR_OPENAI_API_KEY')+ dspy.configure(lm=lm)+ ```++ === "Anthropic"+ You can authenticate by setting the ANTHROPIC_API_KEY env variable or passing `api_key` below.++ ```python linenums="1"+ import dspy+ lm = dspy.LM('anthropic/claude-3-opus-20240229', api_key='YOUR_ANTHROPIC_API_KEY')+ dspy.configure(lm=lm)+ ```++ === "Databricks"+ If you're on the Databricks platform, authentication is automatic via their SDK. If not, you can set the env variables `DATABRICKS_API_KEY` and `DATABRICKS_API_BASE`, or pass `api_key` and `api_base` below.++ ```python linenums="1"+ import dspy+ lm = dspy.LM('databricks/databricks-meta-llama-3-1-70b-instruct')+ dspy.configure(lm=lm)+ ```++ === "Local LMs on a GPU server"+ First, install [SGLang](https://sgl-project.github.io/start/install.html) and launch its server with your LM.++ ```bash+ > pip install "sglang[all]"+ > pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/++ > CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --port 7501 --model-path meta-llama/Meta-Llama-3-8B-Instruct+ ```++ Then, connect to it from your DSPy code as an OpenAI-compatible endpoint.++ ```python linenums="1"+ llama3 = dspy.LM("openai/meta-llama/Meta-Llama-3-8B-Instruct",+ api_base="http://localhost:7501/v1", # ensure this points to your port+ api_key="", model_type='chat')+ dspy.configure(lm=llama3)+ ```++ === "Local LMs on your laptop"+ First, install [Ollama](https://github.com/ollama/ollama) and launch its server with your LM.++ ```bash+ > curl -fsSL https://ollama.ai/install.sh | sh+ > ollama run llama3.2:1b+ ```++ Then, connect to it from your DSPy code.++ ```python linenums="1"+ import dspy+ llama32 = dspy.LM('ollama_chat/llama3.2', api_base='http://localhost:11434', api_key='')+ dspy.configure(lm=llama32)+ ```++ === "Other providers"+ In DSPy, you can use any of the dozens of [LLM providers supported by LiteLLM](https://docs.litellm.ai/docs/providers). Simply follow their instructions for which `{PROVIDER}_API_KEY` to set and how to write pass the `{provider_name}/{model_name}` to the constructor.++ Some examples:++ - `anyscale/mistralai/Mistral-7B-Instruct-v0.1`, with `ANYSCALE_API_KEY`+ - `together_ai/togethercomputer/llama-2-70b-chat`, with `TOGETHERAI_API_KEY`+ - `sagemaker/<your-endpoint-name>`, with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION_NAME`+ - `azure/<your_deployment_name>`, with `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`, and the optional `AZURE_AD_TOKEN` and `AZURE_API_TYPE`+++ If your provider offers an OpenAI-compatible endpoint, just add an `openai/` prefix to your full model name.++ ```python linenums="1"+ import dspy+ lm = dspy.LM('openai/your-model-name', api_key='PROVIDER_API_KEY', api_base='YOUR_PROVIDER_URL')+ dspy.configure(lm=lm)+ ```+++## 1) **Modules** express portable, natural-language-typed behavior.++Building high-quality AI systems requires iterating fast, as you decompose your problem into modular LM components. But it’s really hard to iterate when you have to tinker with each component's messy prompts or synthetic data _every time you change the model, the metrics, or parts of the pipeline_. Having built more than a dozen state-of-the-art compound LM systems over the past five years, we learned this the hard way—and we built DSPy so you don't have to.++DSPy shifts your focus from tinkering with prompts to programming modular AI systems. It does that by **raising the level of AI abstraction from brittle strings to structured, declarative, and natural-language-typed modules**. For every module, you define a _signature_, a spec of input/output types and behavior, and you select an inference-time strategy. DSPy handles expanding your signatures into prompts and parsing your typed outputs, so you can write ergonomic, portable, and optimizable AI systems.+++!!! info "Getting Started II: Build DSPy modules for various tasks"+ After you've configured your preferred LM above, you can try the examples below. Adjust the field names or types to explore. Naturally, the ability of each LM to correctly solve a given task varies. Each example sets up a DSPy module, like `dspy.Predict`, `dspy.ChainOfThought`, or `dspy.ReAct`. The behavior of a module is defined by a _signature_. For example, `question -> answer: float` tells the module to take a question and to produce a `float` answer.++ === "Math"++ ```python linenums="1"+ math = dspy.ChainOfThought("question -> answer: float")+ math(question="Two dice are tossed. What is the probability that the sum equals two?")+ ```++ **Possible Output:**+ ```text+ Prediction(+ reasoning='When two dice are tossed, each die has 6 faces, resulting in a total of 6 x 6 = 36 possible outcomes. The sum of the numbers on the two dice equals two only when both dice show a 1. This is just one specific outcome: (1, 1). Therefore, there is only 1 favorable outcome. The probability of the sum being two is the number of favorable outcomes divided by the total number of possible outcomes, which is 1/36.',+ answer=0.0277776+ )+ ```++ === "Retrieval-Augmented Generation"++ ```python linenums="1"+ def search(query: str) -> list[str]:+ """Retrieves abstracts from Wikipedia."""+ results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)+ return [x['text'] for x in results]++ rag = dspy.ChainOfThought('context, question -> response')++ question = "What's the name of the castle that David Gregory inherited?"+ rag(context=search(question), question=question)+ ```++ **Possible Output:**+ ```text+ Prediction(+ reasoning='The context provides information about David Gregory, a Scottish physician and inventor. It specifically mentions that he inherited Kinnairdy Castle in 1664. This detail directly answers the question about the name of the castle that David Gregory inherited.',+ response='Kinnairdy Castle'+ )+ ```++ === "Classification"++ ```python linenums="1"+ from typing import Literal++ class Classify(dspy.Signature):+ """Classify sentiment of a given sentence."""++ sentence: str = dspy.InputField()+ sentiment: Literal['positive', 'negative', 'neutral'] = dspy.OutputField()+ confidence: float = dspy.OutputField()++ classify = dspy.Predict(Classify)+ classify(sentence="This book was super fun to read, though not the last chapter.")+ ```++ **Possible Output:**++ ```text+ Prediction(+ sentiment='positive',+ confidence=0.75+ )+ ```++ === "Information Extraction"++ ```python linenums="1"+ text = "Apple Inc. announced its latest iPhone 14 today. The CEO, Tim Cook, highlighted its new features in a press release."++ module = dspy.Predict("text -> title, headings: list[str], entities_and_metadata: list[dict[str, str]]")+ response = module(text=text)++ print(response.title)+ print(response.headings)+ print(response.entities_and_metadata)+ ```++ **Possible Output:**+ ```text+ Apple Unveils iPhone 14+ ['Introduction', 'Key Features', "CEO's Statement"]+ [{'entity': 'Apple Inc.', 'type': 'Organization'}, {'entity': 'iPhone 14', 'type': 'Product'}, {'entity': 'Tim Cook', 'type': 'Person'}]+ ```++ === "Agents"++ ```python linenums="1"+ def evaluate_math(expression: str) -> float:+ return dspy.PythonInterpreter({}).execute(expression)++ def search_wikipedia(query: str) -> str:+ results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)+ return [x['text'] for x in results]++ react = dspy.ReAct("question -> answer: float", tools=[evaluate_math, search_wikipedia])++ pred = react(question="What is 9362158 divided by the year of birth of David Gregory of Kinnairdy castle?")+ print(pred.answer)+ ```++ **Possible Output:**++ ```text+ 5761.328+ ```++??? "Using DSPy in practice: from quick scripting to building sophisticated systems."++ Although DSPy's _optimizers_ are often considered its most powerful feature, the ergonomic and portable nature of DSPy's _modules_ make them incredibly powerful for quick LM scripting, even when you don't have the data or metrics to optimize. We maintain large _signature test suites_, across many tasks and LMs, to assess the reliability of the built-in DSPy Adapters. Adapters are the components that map signatures to prompts prior to optimization. If you find a task where a simple prompt consistently outperforms idiomatic DSPy for your LM, consider that a bug and [file an issue](https://github.com/stanfordnlp/dspy/issues). We'll use this to improve the built-in adapters.+++## 2) **Optimizers** tune the prompts and weights of your Modules.++Given a few tens or hundreds of representative _inputs_ of your task and a _metric_ that can measure the quality of your system's outputs, you can use DSPy Optimizers. Different optimizers in DSPy will tune your program's quality by **synthesizing good few-shot examples** for every module like `dspy.BootstrapRS`,<sup>[1](https://arxiv.org/abs/2310.03714)</sup> **proposing and intelligently exploring better natural-language instructions** for every prompt like `dspy.MIPROv2`,<sup>[2](https://arxiv.org/abs/2406.11695)</sup> and **finetuning the LM weights** in your system like `dspy.BootstrapFinetune`.<sup>[3](https://arxiv.org/abs/2407.10930)</sup>+++!!! info "Getting Started III: Optimizing the LM prompts or weights in DSPy programs"+ Be careful when running optimizers with very large LMs or very large datasets. A typical simple optimizer run costs+ on the order of $2 USD and takes less than ten minutes, but optimizer runs can vary from as little as a few cents to+ tens of dollars, depending on your LM, dataset, and configuration.++ === "Optimizing prompts for a ReAct agent"+ This is a minimal but fully runnable example of setting up a `dspy.ReAct` agent that answers questions via+ search from Wikipedia and then optimizing it using `dspy.MIPROv2` in the cheap `light` mode on 500+ question-answer pairs sampled from the `HotPotQA` dataset.++ ```python linenums="1"+ import dspy+ from dspy.datasets import HotPotQA++ dspy.configure(lm=dspy.LM('openai/gpt-4o-mini'))++ def search(query: str) -> list[str]:+ """Retrieves abstracts from Wikipedia."""+ results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)+ return [x['text'] for x in results]++ trainset = [x.with_inputs('question') for x in HotPotQA(train_seed=2024, train_size=500).train]+ react = dspy.ReAct("question -> answer", tools=[search])++ tp = dspy.MIPROv2(metric=dspy.evaluate.answer_exact_match, auto="light", num_threads=24)+ optimized_react = tp.compile(react, trainset=trainset)+ ```++ An informal run very similar to this on DSPy 2.5.29 raises ReAct's score from 24% to 51%.++ === "Optimizing prompts for RAG"+ Given a retrieval index to `search`, your favorite `dspy.LM`, and a small `trainset` of questions and ground-truth responses, the following code snippet can optimize your RAG system with long outputs against the built-in `dspy.SemanticF1` metric, which is implemented as a DSPy module.++ ```python linenums="1"+ class RAG(dspy.Module):+ def __init__(self, num_docs=5):+ self.num_docs = num_docs+ self.respond = dspy.ChainOfThought('context, question -> response')++ def forward(self, question):+ context = search(question, k=self.num_docs) # not defined in this snippet, see link above+ return self.respond(context=context, question=question)++ tp = dspy.MIPROv2(metric=dspy.SemanticF1(), auto="medium", num_threads=24)+ optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)+ ```++ For a complete RAG example that you can run, start this [tutorial](http://127.0.0.1:8000/quick-start/getting-started-01/). It improves the quality of a RAG system over a subset of StackExchange communities from 53% to 61%.++ === "Optimizing weights for Classification"+ This is a minimal but fully runnable example of setting up a `dspy.ChainOfThought` module that classifies+ short texts into one of 77 banking labels and then using `dspy.BootstrapFinetune` with 2000 text-label pairs+ from the `Banking77` to finetune the weights of GPT-4o-mini for this task. We use the variant+ `dspy.ChainOfThoughtWithHint`, which takes an optional `hint` at bootstrapping time, to maximize the utility of+ the training data. Naturally, hints are not available at test time.++ <details><summary>Click to show dataset setup code.</summary>++ ```python linenums="1"+ import random+ from typing import Literal+ from dspy.datasets import DataLoader+ from datasets import load_dataset++ # Load the Banking77 dataset.+ CLASSES = load_dataset("PolyAI/banking77", split="train", trust_remote_code=True).features['label'].names+ kwargs = dict(fields=("text", "label"), input_keys=("text",), split="train", trust_remote_code=True)++ # Load the first 2000 examples from the dataset, and assign a hint to each *training* example.+ trainset = [+ dspy.Example(x, hint=CLASSES[x.label], label=CLASSES[x.label]).with_inputs("text", "hint")+ for x in DataLoader().from_huggingface(dataset_name="PolyAI/banking77", **kwargs)[:2000]+ ]+ random.Random(0).shuffle(trainset)+ ```+ </details>++ ```python linenums="1"+ import dspy+ dspy.configure(lm=dspy.LM('gpt-4o-mini-2024-07-18'))++ # Define the DSPy module for classification. It will use the hint at training time, if available.+ signature = dspy.Signature("text -> label").with_updated_fields('label', type_=Literal[tuple(CLASSES)])+ classify = dspy.ChainOfThoughtWithHint(signature)++ # Optimize via BootstrapFinetune.+ optimizer = dspy.BootstrapFinetune(metric=(lambda x, y, trace=None: x.label == y.label), num_threads=24)+ optimized = optimizer.compile(classify, trainset=trainset)++ optimized_classifier(text="What does a pending cash withdrawal mean?")+ ```++ **Possible Output (from the last line):**+ ```text+ Prediction(+ reasoning='A pending cash withdrawal indicates that a request to withdraw cash has been initiated but has not yet been completed or processed. This status means that the transaction is still in progress and the funds have not yet been deducted from the account or made available to the user.',+ label='pending_cash_withdrawal'+ )+ ```++ An informal run very similar to this on DSPy 2.5.29 raises GPT-4o-mini's score 66% to 87%.+++??? "What's an example of a DSPy optimizer? How do different optimizers work?"++ Take the `dspy.MIPROv2` optimizer as an example. First, MIPRO starts with the bootstrapping stage. It takes your initial program, which may be unoptimized, and running it many times across different inputs to collect traces of input/output behavior for each module in your program. These traces are filtered to keep only the traces that appear in trajectories that are scored highly by your metric. Second, MIPRO enters its grounded proposal stage. It previews your DSPy program's code, samples of your data, and traces from running your program, and uses them to draft many potential instructions for every prompt in your program. Third, MIPRO launches the search stage. It samples mini-batches from your training set, proposes a combination of instructions and traces to use for constructing every prompt in the pipeline, and evaluates the candidate program on the mini-batch. Using the resulting score, MIPRO updates a surrogate model that helps the proposals get better over time.++ One of the most powerful things about DSPy optimizers is that they can be composed. You can run `dspy.MIPROv2` and use the produced program as an input to `dspy.MIPROv2` again or, say, to `dspy.BootstrapFinetune` to get better results. This is partly the essence of `dspy.BetterTogether`. Alternatively, you can run the optimizer and then extract the top-5 candidate programs and build a `dspy.Ensemble` of them. This allows you to scale _inference-time compute_ (e.g., ensembles) and DSPy's unique _pre-inference time compute_ (i.e., optimization budget) in highly systematic ways.++++<!-- Future:+BootstrapRS or MIPRO on ??? with a local SGLang LM+BootstrapFS on MATH with a tiny LM like Llama-3.2 with Ollama (maybe with a big teacher) -->++++## 3) **DSPy's Ecosystem** advances open-source AI research.++By introducing structure into the space of LM programming, DSPy aims to enable a large, distributed community to independently improve program architectures, inference-time strategies, and LM program optimizers.++The initial DSPy research started at Stanford NLP in Feb 2022, building on what we learned from developing early compound LM systems like [ColBERT-QA](https://arxiv.org/abs/2007.00814), [Baleen](https://arxiv.org/abs/2101.00436), and [Hindsight](https://arxiv.org/abs/2110.07752). This work was first released as [Demonstrate-Search-Predict](https://arxiv.org/abs/2212.14024) (DSP) in Dec 2022 and then evolved in Oct 2023 into [DSPy](https://arxiv.org/abs/2310.03714), or Declarative Self-improving Python. Thanks to [nearly 250 wonderful contributors](https://github.com/stanfordnlp/dspy/graphs/contributors), DSPy has introduced tens of thousands of people to building modular LM programs and optimizing their prompts and weights automatically.++Since then, the broad community around DSPy has produced a large body of research. This includes work on optimizers, like [MIPROv2](https://arxiv.org/abs/2406.11695), [BetterTogether](https://arxiv.org/abs/2407.10930), and [LeReT](https://arxiv.org/abs/2410.23214). It also includes novel program architectures, like [STORM](https://arxiv.org/abs/2402.14207), [IReRa](https://arxiv.org/abs/2401.12178), and [DSPy Assertions](https://arxiv.org/abs/2312.13382). And it includes a large number of successful applications to new problems, like [PAPILLON](https://arxiv.org/abs/2410.17127), [PATH](https://arxiv.org/abs/2406.11706), [WangLab@MEDIQA](https://arxiv.org/abs/2404.14544), [UMD's Prompting Case Study](https://arxiv.org/abs/2406.06608), and [Haize's Red-Teaming Program](https://blog.haizelabs.com/posts/dspy/), in addition to many open-source projects and numerous production applications. You can learn about a few of these in the [Use Cases](/dspy-usecases/) page.diff --git a/docs/docs/index_.md b/docs/docs/index_.mdnew file mode 100644index 000000000..3fb1f8ede--- /dev/null+++ b/docs/docs/index_.md@@ -0,0 +1,6 @@+---+template: home.html+hide:+ - navigation+ - toc+---diff --git a/docs/docs/intro.md b/docs/docs/intro.mddeleted file mode 100644index b9537655f..000000000--- a/docs/docs/intro.md+++ /dev/null@@ -1,357 +0,0 @@-----sidebar_position: 1-hide:- - navigation- # - toc-------{ width="350", align=right }--# **DSPy**: _Programming_—not prompting—_Language Models_---DSPy is the open-source framework for **building high-quality, modular AI systems** by _programming—rather than prompting—language models_. It provides abstractions and algorithms for **optimizing the prompts and weights** in LM programs, ranging from simple classifiers to sophisticated RAG pipelines and Agent loops.--Instead of writing brittle LM-specific prompts, you write compositional code and use DSPy optimizers to teach different models like `GPT-4o` or `Llama-3.2` to **deliver higher quality outputs** or avoid specific failure patterns. In essence, DSPy optimizers then _compile your high-level code_ into low-level computations, prompts, or weight updates that **align your LM with your program’s structure and metrics**.---DSPy stands for Declarative Self-improving Python. This [recent lecture](https://www.youtube.com/watch?v=JEMYuzrKLUw) is a good conceptual introduction. Our [Discord server](https://discord.gg/XCGy2WDCQB) is a great place to meet the community, seek help, or start contributing.---!!! info "Getting Started I: Install DSPy and set up your LM"-- ```bash- > pip install -U dspy- ```-- === "OpenAI"- You can authenticate by setting the `OPENAI_API_KEY` env variable or passing `api_key` below.-- ```python linenums="1"- import dspy- lm = dspy.LM('openai/gpt-4o-mini', api_key='YOUR_OPENAI_API_KEY')- dspy.configure(lm=lm)- ```-- === "Anthropic"- You can authenticate by setting the ANTHROPIC_API_KEY env variable or passing `api_key` below.-- ```python linenums="1"- import dspy- lm = dspy.LM('anthropic/claude-3-opus-20240229', api_key='YOUR_ANTHROPIC_API_KEY')- dspy.configure(lm=lm)- ```-- === "Databricks"- If you're on the Databricks platform, authentication is automatic via their SDK. If not, you can set the env variables `DATABRICKS_API_KEY` and `DATABRICKS_API_BASE`, or pass `api_key` and `api_base` below.-- ```python linenums="1"- import dspy- lm = dspy.LM('databricks/databricks-meta-llama-3-1-70b-instruct')- dspy.configure(lm=lm)- ```-- === "Local LMs on a GPU server"- First, install [SGLang](https://sgl-project.github.io/start/install.html) and launch its server with your LM.-- ```bash- > pip install "sglang[all]"- > pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/-- > CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --port 7501 --model-path meta-llama/Meta-Llama-3-8B-Instruct- ```-- Then, connect to it from your DSPy code as an OpenAI-compatible endpoint.-- ```python linenums="1"- llama3 = dspy.LM("openai/meta-llama/Meta-Llama-3-8B-Instruct",- api_base="http://localhost:7501/v1", # ensure this points to your port- api_key="", model_type='chat')- dspy.configure(lm=llama3)- ```-- === "Local LMs on your laptop"- First, install [Ollama](https://github.com/ollama/ollama) and launch its server with your LM.-- ```bash- > curl -fsSL https://ollama.ai/install.sh | sh- > ollama run llama3.2:1b- ```-- Then, connect to it from your DSPy code.-- ```python linenums="1"- import dspy- llama32 = dspy.LM('ollama_chat/llama3.2', api_base='http://localhost:11434', api_key='')- dspy.configure(lm=llama32)- ```-- === "Other providers"- In DSPy, you can use any of the dozens of [LLM providers supported by LiteLLM](https://docs.litellm.ai/docs/providers). Simply follow their instructions for which `{PROVIDER}_API_KEY` to set and how to write pass the `{provider_name}/{model_name}` to the constructor.-- Some examples:-- - `anyscale/mistralai/Mistral-7B-Instruct-v0.1`, with `ANYSCALE_API_KEY`- - `together_ai/togethercomputer/llama-2-70b-chat`, with `TOGETHERAI_API_KEY`- - `sagemaker/<your-endpoint-name>`, with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION_NAME`- - `azure/<your_deployment_name>`, with `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`, and the optional `AZURE_AD_TOKEN` and `AZURE_API_TYPE`--- If your provider offers an OpenAI-compatible endpoint, just add an `openai/` prefix to your full model name.-- ```python linenums="1"- import dspy- lm = dspy.LM('openai/your-model-name', api_key='PROVIDER_API_KEY', api_base='YOUR_PROVIDER_URL')- dspy.configure(lm=lm)- ```---## 1) **Modules** express portable, natural-language-typed behavior.--Building high-quality AI systems requires iterating fast, as you decompose your problem into modular LM components. But it’s really hard to iterate when you have to tinker with each component's messy prompts or synthetic data _every time you change the model, the metrics, or parts of the pipeline_. Having built more than a dozen state-of-the-art compound LM systems over the past five years, we learned this the hard way—and we built DSPy so you don't have to.--DSPy shifts your focus from tinkering with prompts to programming modular AI systems. It does that by **raising the level of AI abstraction from brittle strings to structured, declarative, and natural-language-typed modules**. For every module, you define a _signature_, a spec of input/output types and behavior, and you select an inference-time strategy. DSPy handles expanding your signatures into prompts and parsing your typed outputs, so you can write ergonomic, portable, and optimizable AI systems.---!!! info "Getting Started II: Build DSPy modules for various tasks"- After you've configured your preferred LM above, you can try the examples below. Adjust the field names or types to explore. Naturally, the ability of each LM to correctly solve a given task varies. Each example sets up a DSPy module, like `dspy.Predict`, `dspy.ChainOfThought`, or `dspy.ReAct`. The behavior of a module is defined by a _signature_. For example, `question -> answer: float` tells the module to take a question and to produce a `float` answer.-- === "Math"-- ```python linenums="1"- math = dspy.ChainOfThought("question -> answer: float")- math(question="Two dice are tossed. What is the probability that the sum equals two?")- ```-- **Possible Output:**- ```text- Prediction(- reasoning='When two dice are tossed, each die has 6 faces, resulting in a total of 6 x 6 = 36 possible outcomes. The sum of the numbers on the two dice equals two only when both dice show a 1. This is just one specific outcome: (1, 1). Therefore, there is only 1 favorable outcome. The probability of the sum being two is the number of favorable outcomes divided by the total number of possible outcomes, which is 1/36.',- answer=0.0277776- )- ```-- === "Retrieval-Augmented Generation"-- ```python linenums="1"- def search(query: str) -> list[str]:- """Retrieves abstracts from Wikipedia."""- results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)- return [x['text'] for x in results]-- rag = dspy.ChainOfThought('context, question -> response')-- question = "What's the name of the castle that David Gregory inherited?"- rag(context=search(question), question=question)- ```-- **Possible Output:**- ```text- Prediction(- reasoning='The context provides information about David Gregory, a Scottish physician and inventor. It specifically mentions that he inherited Kinnairdy Castle in 1664. This detail directly answers the question about the name of the castle that David Gregory inherited.',- response='Kinnairdy Castle'- )- ```-- === "Classification"-- ```python linenums="1"- from typing import Literal-- class Classify(dspy.Signature):- """Classify sentiment of a given sentence."""-- sentence: str = dspy.InputField()- sentiment: Literal['positive', 'negative', 'neutral'] = dspy.OutputField()- confidence: float = dspy.OutputField()-- classify = dspy.Predict(Classify)- classify(sentence="This book was super fun to read, though not the last chapter.")- ```-- **Possible Output:**-- ```text- Prediction(- sentiment='positive',- confidence=0.75- )- ```-- === "Information Extraction"-- ```python linenums="1"- text = "Apple Inc. announced its latest iPhone 14 today. The CEO, Tim Cook, highlighted its new features in a press release."-- module = dspy.Predict("text -> title, headings: list[str], entities_and_metadata: list[dict[str, str]]")- response = module(text=text)-- print(response.title)- print(response.headings)- print(response.entities_and_metadata)- ```-- **Possible Output:**- ```text- Apple Unveils iPhone 14- ['Introduction', 'Key Features', "CEO's Statement"]- [{'entity': 'Apple Inc.', 'type': 'Organization'}, {'entity': 'iPhone 14', 'type': 'Product'}, {'entity': 'Tim Cook', 'type': 'Person'}]- ```-- === "Agents"-- ```python linenums="1"- def evaluate_math(expression: str) -> float:- return dspy.PythonInterpreter({}).execute(expression)-- def search_wikipedia(query: str) -> str:- results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)- return [x['text'] for x in results]-- react = dspy.ReAct("question -> answer: float", tools=[evaluate_math, search_wikipedia])-- pred = react(question="What is 9362158 divided by the year of birth of David Gregory of Kinnairdy castle?")- print(pred.answer)- ```-- **Possible Output:**-- ```text- 5761.328- ```--??? "Using DSPy in practice: from quick scripting to building sophisticated systems."-- Although DSPy's _optimizers_ are often considered its most powerful feature, the ergonomic and portable nature of DSPy's _modules_ make them incredibly powerful for quick LM scripting, even when you don't have the data or metrics to optimize. We maintain large _signature test suites_, across many tasks and LMs, to assess the reliability of the built-in DSPy Adapters. Adapters are the components that map signatures to prompts prior to optimization. If you find a task where a simple prompt consistently outperforms idiomatic DSPy for your LM, consider that a bug and [file an issue](https://github.com/stanfordnlp/dspy/issues). We'll use this to improve the built-in adapters.---## 2) **Optimizers** tune the prompts and weights of your Modules.--Given a few tens or hundreds of representative _inputs_ of your task and a _metric_ that can measure the quality of your system's outputs, you can use DSPy Optimizers. Different optimizers in DSPy will tune your program's quality by **synthesizing good few-shot examples** for every module like `dspy.BootstrapRS`,<sup>[1](https://arxiv.org/abs/2310.03714)</sup> **proposing and intelligently exploring better natural-language instructions** for every prompt like `dspy.MIPROv2`,<sup>[2](https://arxiv.org/abs/2406.11695)</sup> and **finetuning the LM weights** in your system like `dspy.BootstrapFinetune`.<sup>[3](https://arxiv.org/abs/2407.10930)</sup>---!!! info "Getting Started III: Optimizing the LM prompts or weights in DSPy programs"- Be careful when running optimizers with very large LMs or very large datasets. A typical simple optimizer run costs- on the order of $2 USD and takes less than ten minutes, but optimizer runs can vary from as little as a few cents to- tens of dollars, depending on your LM, dataset, and configuration.-- === "Optimizing prompts for a ReAct agent"- This is a minimal but fully runnable example of setting up a `dspy.ReAct` agent that answers questions via- search from Wikipedia and then optimizing it using `dspy.MIPROv2` in the cheap `light` mode on 500- question-answer pairs sampled from the `HotPotQA` dataset.-- ```python linenums="1"- import dspy- from dspy.datasets import HotPotQA-- dspy.configure(lm=dspy.LM('openai/gpt-4o-mini'))-- def search(query: str) -> list[str]:- """Retrieves abstracts from Wikipedia."""- results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3)- return [x['text'] for x in results]-- trainset = [x.with_inputs('question') for x in HotPotQA(train_seed=2024, train_size=500).train]- react = dspy.ReAct("question -> answer", tools=[search])-- tp = dspy.MIPROv2(metric=dspy.evaluate.answer_exact_match, auto="light", num_threads=24)- optimized_react = tp.compile(react, trainset=trainset)- ```-- An informal run very similar to this on DSPy 2.5.29 raises ReAct's score from 24% to 51%.-- === "Optimizing prompts for RAG"- Given a retrieval index to `search`, your favorite `dspy.LM`, and a small `trainset` of questions and ground-truth responses, the following code snippet can optimize your RAG system with long outputs against the built-in `dspy.SemanticF1` metric, which is implemented as a DSPy module.-- ```python linenums="1"- class RAG(dspy.Module):- def __init__(self, num_docs=5):- self.num_docs = num_docs- self.respond = dspy.ChainOfThought('context, question -> response')-- def forward(self, question):- context = search(question, k=self.num_docs) # not defined in this snippet, see link above- return self.respond(context=context, question=question)-- tp = dspy.MIPROv2(metric=dspy.SemanticF1(), auto="medium", num_threads=24)- optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)- ```-- For a complete RAG example that you can run, start this [tutorial](http://127.0.0.1:8000/quick-start/getting-started-01/). It improves the quality of a RAG system over a subset of StackExchange communities from 53% to 61%.-- === "Optimizing weights for Classification"- This is a minimal but fully runnable example of setting up a `dspy.ChainOfThought` module that classifies- short texts into one of 77 banking labels and then using `dspy.BootstrapFinetune` with 2000 text-label pairs- from the `Banking77` to finetune the weights of GPT-4o-mini for this task. We use the variant- `dspy.ChainOfThoughtWithHint`, which takes an optional `hint` at bootstrapping time, to maximize the utility of- the training data. Naturally, hints are not available at test time.-- <details><summary>Click to show dataset setup code.</summary>-- ```python linenums="1"- import random- from typing import Literal- from dspy.datasets import DataLoader- from datasets import load_dataset-- # Load the Banking77 dataset.- CLASSES = load_dataset("PolyAI/banking77", split="train", trust_remote_code=True).features['label'].names- kwargs = dict(fields=("text", "label"), input_keys=("text",), split="train", trust_remote_code=True)-- # Load the first 2000 examples from the dataset, and assign a hint to each *training* example.- trainset = [- dspy.Example(x, hint=CLASSES[x.label], label=CLASSES[x.label]).with_inputs("text", "hint")- for x in DataLoader().from_huggingface(dataset_name="PolyAI/banking77", **kwargs)[:2000]- ]- random.Random(0).shuffle(trainset)- ```- </details>-- ```python linenums="1"- import dspy- dspy.configure(lm=dspy.LM('gpt-4o-mini-2024-07-18'))-- # Define the DSPy module for classification. It will use the hint at training time, if available.- signature = dspy.Signature("text -> label").with_updated_fields('label', type_=Literal[tuple(CLASSES)])- classify = dspy.ChainOfThoughtWithHint(signature)-- # Optimize via BootstrapFinetune.- optimizer = dspy.BootstrapFinetune(metric=(lambda x, y, trace=None: x.label == y.label), num_threads=24)- optimized = optimizer.compile(classify, trainset=trainset)-- optimized_classifier(text="What does a pending cash withdrawal mean?")- ```-- **Possible Output (from the last line):**- ```text- Prediction(- reasoning='A pending cash withdrawal indicates that a request to withdraw cash has been initiated but has not yet been completed or processed. This status means that the transaction is still in progress and the funds have not yet been deducted from the account or made available to the user.',- label='pending_cash_withdrawal'- )- ```-- An informal run very similar to this on DSPy 2.5.29 raises GPT-4o-mini's score 66% to 87%.---??? "What's an example of a DSPy optimizer? How do different optimizers work?"-- Take the `dspy.MIPROv2` optimizer as an example. First, MIPRO starts with the bootstrapping stage. It takes your initial program, which may be unoptimized, and running it many times across different inputs to collect traces of input/output behavior for each module in your program. These traces are filtered to keep only the traces that appear in trajectories that are scored highly by your metric. Second, MIPRO enters its grounded proposal stage. It previews your DSPy program's code, samples of your data, and traces from running your program, and uses them to draft many potential instructions for every prompt in your program. Third, MIPRO launches the search stage. It samples mini-batches from your training set, proposes a combination of instructions and traces to use for constructing every prompt in the pipeline, and evaluates the candidate program on the mini-batch. Using the resulting score, MIPRO updates a surrogate model that helps the proposals get better over time.-- One of the most powerful things about DSPy optimizers is that they can be composed. You can run `dspy.MIPROv2` and use the produced program as an input to `dspy.MIPROv2` again or, say, to `dspy.BootstrapFinetune` to get better results. This is partly the essence of `dspy.BetterTogether`. Alternatively, you can run the optimizer and then extract the top-5 candidate programs and build a `dspy.Ensemble` of them. This allows you to scale _inference-time compute_ (e.g., ensembles) and DSPy's unique _pre-inference time compute_ (i.e., optimization budget) in highly systematic ways.----<!-- Future:-BootstrapRS or MIPRO on ??? with a local SGLang LM-BootstrapFS on MATH with a tiny LM like Llama-3.2 with Ollama (maybe with a big teacher) -->----## 3) **DSPy's Ecosystem** advances open-source AI research.--By introducing structure into the space of LM programming, DSPy aims to enable a large, distributed community to independently improve program architectures, inference-time strategies, and LM program optimizers.--The initial DSPy research started at Stanford NLP in Feb 2022, building on what we learned from developing early compound LM systems like [ColBERT-QA](https://arxiv.org/abs/2007.00814), [Baleen](https://arxiv.org/abs/2101.00436), and [Hindsight](https://arxiv.org/abs/2110.07752). This work was first released as [Demonstrate-Search-Predict](https://arxiv.org/abs/2212.14024) (DSP) in Dec 2022 and then evolved in Oct 2023 into [DSPy](https://arxiv.org/abs/2310.03714), or Declarative Self-improving Python. Thanks to [nearly 250 wonderful contributors](https://github.com/stanfordnlp/dspy/graphs/contributors), DSPy has introduced tens of thousands of people to building modular LM programs and optimizing their prompts and weights automatically.--Since then, the broad community around DSPy has produced a large body of research. This includes work on optimizers, like [MIPROv2](https://arxiv.org/abs/2406.11695), [BetterTogether](https://arxiv.org/abs/2407.10930), and [LeReT](https://arxiv.org/abs/2410.23214). It also includes novel program architectures, like [STORM](https://arxiv.org/abs/2402.14207), [IReRa](https://arxiv.org/abs/2401.12178), and [DSPy Assertions](https://arxiv.org/abs/2312.13382). And it includes a large number of successful applications to new problems, like [PAPILLON](https://arxiv.org/abs/2410.17127), [PATH](https://arxiv.org/abs/2406.11706), [UofT's WangLab at MEDIQA-CORR 2024 winning system](https://arxiv.org/abs/2404.14544), [UMD's Prompt Engineering Case Study](https://arxiv.org/abs/2406.06608), and [Haize's Red-Teaming DSPy System](https://blog.haizelabs.com/posts/dspy/), in addition to many open-source projects and numerous production applications. You can learn about a few of these in the [Use Cases](/dspy-usecases/) page.diff --git a/docs/docs/quick-start/getting-started-01.md b/docs/docs/quick-start/getting-started-01.mddeleted file mode 100644index bd65cd5f2..000000000--- a/docs/docs/quick-start/getting-started-01.md+++ /dev/null@@ -1,229 +0,0 @@-----sidebar_position: 2------# Getting Started I: Basic Question Answering--Let's walk through a quick example of **basic question answering** in DSPy. Specifically, let's build **a system for answering Tech questions**, e.g. about Linux or iPhone apps.--Install the latest DSPy via `pip install -U dspy` and follow along. If you're looking instead for a conceptual overview of DSPy, this [recent lecture](https://www.youtube.com/live/JEMYuzrKLUw) is a good place to start.--## Configuring the DSPy environment--Let's tell DSPy that we will use OpenAI's `gpt-4o-mini` in our modules. To authenticate, DSPy will look into your `OPENAI_API_KEY`. You can easily swap this out for [other providers or local models](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb).--```python-import dspy--lm = dspy.LM('openai/gpt-4o-mini')-dspy.configure(lm=lm)-```--## Exploring some basic DSPy `Module`s.--You can always prompt the LM directly via `lm(prompt="prompt")` or `lm(messages=[...])`. However, DSPy gives you `Modules` as a better way to define your LM functions.--The simplest module is `dspy.Predict`. It takes a [DSPy Signature](/building-blocks/2-signatures), i.e. a structured input/output schema, and gives you back a callable function for the behavior you specified. Let's use the "in-line" notation for signatures to declare a module that takes a `question` (of type `str`) as input and produces a `response` as an output.--```python-qa = dspy.Predict('question: str -> response: str')-qa(question="what are high memory and low memory on linux?").response-```--**Output:**-```-'In Linux, "high memory" and "low memory" refer to different regions of the system\'s memory address space, particularly in the context of 32-bit architectures.\n\n- **Low Memory**: This typically refers to the first 896 MB of memory in a 32-bit system. It is directly accessible by the kernel and is used for kernel data structures and user processes. The low memory region is where most of the system\'s memory management occurs, and it is where the kernel can allocate memory for processes without needing special handling.\n\n- **High Memory**: This refers to memory above the 896 MB threshold in a 32-bit system. The kernel cannot directly access this memory without special mechanisms because of the limitations of the 32-bit address space. High memory is used for user processes that require more memory than what is available in the low memory region. The kernel can manage high memory through techniques like "highmem" support, which allows it to map high memory pages into the kernel\'s address space when needed.\n\nIn summary, low memory is directly accessible by the kernel, while high memory requires additional handling for the kernel to access it, especially in 32-bit systems. In 64-bit systems, this distinction is less relevant as the addressable memory space is significantly larger.'-```--Notice how the variable names we specified in the signature defined our input and output argument names and their role.--Now, what did DSPy do to build this `qa` module? Nothing fancy in this example, yet. The module passed your signature, LM, and inputs to an [Adapter](/building-blocks/1-language_models#structured-lm-output-with-adapters), which is a layer that handles structuring the inputs and parsing structured outputs to fit your signature.--Let's see it directly. You can inspect the `n` last prompts sent by DSPy easily.--```python-dspy.inspect_history(n=1)-```--**Output:**-See this [gist](https://gist.github.com/okhat/aff3c9788ccddf726fdfeb78e40e5d22).---DSPy has various built-in modules, e.g. `dspy.ChainOfThought`, `dspy.ProgramOfThought`, and `dspy.ReAct`. These are interchangeable with basic `dspy.Predict`: they take your signature, which is specific to your task, and they apply general-purpose prompting techniques and inference-time strategies to it.--For example, `dspy.ChainOfThought` is an easy way to elicit `reasoning` out of your LM before it commits to the outputs requested in your signature.--In the example below, we'll omit `str` types (as the default type is string). You should feel free to experiment with other fields and types, e.g. try `topics: list[str]` or `is_realistic: bool`.--```python-cot = dspy.ChainOfThought('question -> response')-cot(question="should curly braces appear on their own line?")-```--**Output:**-```-Prediction(- reasoning="The placement of curly braces on their own line is largely a matter of coding style and conventions. In some programming languages and style guides, such as those used in C, C++, and Java, it is common to place opening curly braces on the same line as the control statement (like `if`, `for`, etc.) and closing braces on a new line. However, other styles, such as the Allman style, advocate for placing both opening and closing braces on their own lines. Ultimately, the decision should be based on the team's coding standards or personal preference, as long as it maintains readability and consistency throughout the code.",- response="Curly braces can either appear on their own line or not, depending on the coding style you choose to follow. It's important to be consistent with whichever style you adopt."-)-```---Interestingly, asking for reasoning made the output `response` shorter in this case. Is this a good thing or a bad thing? It depends on what you need: there's no free lunch, but DSPy gives you the tools to experiment with different strategies extremely quickly.--By the way, `dspy.ChainOfThought` is implemented in DSPy, using `dspy.Predict`. This is a good place to `dspy.inspect_history` if you're curious.--## Using DSPy well involves evaluation and iterative development.--You already know a lot about DSPy at this point. If all you want is quick scripting, this much of DSPy already enables a lot. Sprinkling DSPy signatures and modules into your Python control flow is a pretty ergonomic way to just get stuff done with LMs.--That said, you're likely here because you want to build a high-quality system and improve it over time. The way to do that in DSPy is to iterate fast by evaluating the quality of your system and using DSPy's powerful tools, e.g. [Optimizers](/building-blocks/6-optimizers). You can learn about the [appropriate development cycle in DSPy here](/building-blocks/solving_your_task).--## Manipulating `Example`s in DSPy.--To measure the quality of your DSPy system, you need (1) a bunch of input values, like `question`s for example, and (2) a `metric` that can score the quality of an output from your system. Metrics vary widely. Some metrics need ground-truth labels of ideal outputs, e.g. for classification or question answering. Other metrics are self-supervised, e.g. checking faithfulness or lack of hallucination, perhaps using a DSPy program as a judge of these qualities.--Let's load a dataset of questions and their (pretty long) gold answers. Since we started this notebook with the goal of building **a system for answering Tech questions**, we obtained a bunch of StackExchange-based questions and their correct answers from the [RAG-QA Arena](https://arxiv.org/abs/2407.13998) dataset. (Learn more about the [development cycle](/building-blocks/solving_your_task) if you don't have data for your task.)---```python-import ujson--# Download 500 question--answer pairs from the RAG-QA Arena "Tech" dataset.-!wget https://huggingface.co/dspy/cache/resolve/main/ragqa_arena_tech_500.json--with open('ragqa_arena_tech_500.json') as f:- data = ujson.load(f)--# Inspect one datapoint.-data[0]-```--**Output:**-```-{'question': 'how to transfer whatsapp voice message to computer?',- 'response': 'To transfer voice notes from WhatsApp on your device to your computer, you have the option to select the "Share" feature within the app and send the files via Email, Gmail, Bluetooth, or other available services. \nYou can also move the files onto your phone\'s SD card, connect your phone to your computer via a USB cable, then find and transfer the files via File Explorer on your PC. \nAlternatively, you can choose to attach all the desired voice notes to an email and, from your phone, send them to your own email address. \nUpon receiving the email on your computer, you can then download the voice note attachments.'}-```--Given a simple dict like this, let's create a list of `dspy.Example`s, which is the datatype that carries training (or test) datapoints in DSPy.--When you build a `dspy.Example`, you should generally specify `.with_inputs("field1", "field2", ...)` to indicate which fields are inputs. The other fields are treated as labels or metadata.--```python-data = [dspy.Example(**d).with_inputs('question') for d in data]--# Let's pick an `example` here from the data.-example = data[2]-example-```--**Output:**-```-Example({'question': 'what are high memory and low memory on linux?', 'response': '"High Memory" refers to the application or user space, the memory that user programs can use and which isn\'t permanently mapped in the kernel\'s space, while "Low Memory" is the kernel\'s space, which the kernel can address directly and is permanently mapped. \nThe user cannot access the Low Memory as it is set aside for the required kernel programs.'}) (input_keys={'question'})-```--Now, let's divide the data into:--- Training and Validation sets:- - These are the splits you typically give to DSPy optimizers.- - Optimizers typically learn directly from the training examples and check their progress using the validation examples.- - It's good to have 30--300 examples for training and validation each.- - For prompt optimizers in particular, it's often better to pass _more_ validation than training.--- Development and Test sets: The rest, typically on the order of 30--1000, can be used for:- - development (i.e., you can inspect them as you iterate on your system) and- - testing (final held-out evaluation).---```python-trainset, valset, devset, testset = data[:50], data[50:150], data[150:300], data[300:500]--len(trainset), len(valset), len(devset), len(testset)-```--**Output:**-```-(50, 100, 150, 200)-```---## Evaluation in DSPy.--What kind of metric can suit our question-answering task? There are many choices, but since the answers are long, we may ask: How well does the system response _cover_ all key facts in the gold response? And the other way around, how well is the system response _not saying things_ that aren't in the gold response?--That metric is essentially a **semantic F1**, so let's load a `SemanticF1` metric from DSPy. This metric is actually implemented as a [very simple DSPy module](https://github.com/stanfordnlp/dspy/blob/77c2e1cceba427c7f91edb2ed5653276fb0c6de7/dspy/evaluate/auto_evaluation.py#L21) using whatever LM we're working with.---```python-from dspy.evaluate import SemanticF1--# Instantiate the metric.-metric = SemanticF1()--# Produce a prediction from our `cot` module, using the `example` above as input.-pred = cot(**example.inputs())--# Compute the metric score for the prediction.-score = metric(example, pred)--print(f"Question: \t {example.question}\n")-print(f"Gold Response: \t {example.response}\n")-print(f"Predicted Response: \t {pred.response}\n")-print(f"Semantic F1 Score: {score:.2f}")-```--**Output:**-```-Question: what are high memory and low memory on linux?--Gold Response: "High Memory" refers to the application or user space, the memory that user programs can use and which isn't permanently mapped in the kernel's space, while "Low Memory" is the kernel's space, which the kernel can address directly and is permanently mapped.-The user cannot access the Low Memory as it is set aside for the required kernel programs.--Predicted Response: In Linux, "low memory" refers to the memory that is directly accessible by the kernel and user processes, typically the first 4GB on a 32-bit system. "High memory" refers to memory above this limit, which is not directly accessible by the kernel in a 32-bit environment. This distinction is crucial for memory management, particularly in systems with large amounts of RAM, as it influences how memory is allocated and accessed.--Semantic F1 Score: 0.80-```--The final DSPy module call above actually happens inside `metric`. You might be curious how it measured the semantic F1 for this example.---```python-dspy.inspect_history(n=1)-```--**Output:**-See this [gist](https://gist.github.com/okhat/57bf86472d1e14812c0ae33fba5353f8).--For evaluation, you could use the metric above in a simple loop and just average the score. But for nice parallelism and utilities, we can rely on `dspy.Evaluate`.--```python-# Define an evaluator that we can re-use.-evaluate = dspy.Evaluate(devset=devset, metric=metric, num_threads=24,- display_progress=True, display_table=3)--# Evaluate the Chain-of-Thought program.-evaluate(cot)-```--**Output:**-```-Average Metric: 59.565342393613165 / 150 (39.7): 100%|██████████| 150/150 [00:00<00:00, 432.92it/s]-```--The table you'll get in the output would look like:--| | question | example_response | reasoning | pred_response | SemanticF1 |-|---|---|---|---|---|---|-| 0 | why is mercurial considered to be easier than git? | Mercurial's syntax is considered more familiar, especially for those accustomed to SVN, and is well documented. It focuses on interface aspects, which initially makes learning... | Mercurial is often considered easier than Git for several reasons. Firstly, Mercurial has a simpler command structure and a more consistent user interface, which can... | Mercurial is considered easier than Git primarily due to its simpler command structure and more consistent user interface, making it more approachable for beginners. Its... | ✔️ [0.545] |-| 1 | open finder window from current terminal location? | If you type 'open .' in Terminal, it will open the current directory in a Finder window. Alternatively, you can execute the command open `pwd`... | To open a Finder window from the current terminal location on a Mac, you can use the `open` command followed by a dot (`.`) which... | You can open a Finder window from your current terminal location by using the following command:\n```\nopen .\n``` | ✔️ [0.667] |-| 2 | how to import secret gpg key (copied from one machine to another)? | It is advised that it is necessary to add `--import` to the command line to import the private key and that according to the man... | To import a secret GPG key that has been copied from one machine to another, you need to ensure that the key is in the... | To import a secret GPG key that you have copied from one machine to another, follow these steps: 1. **Transfer the Key**: Ensure that the... | ✔️ [0.708] |--## What's next?--In this guide, we built a very simple chain-of-thought module for question answering and evaluated it on a small dataset.--Can we do better? In the next guide, we will build a retrieval-augmented generation (RAG) program in DSPy for the same task.--We'll see how this can boost the score substantially, then we'll use one of the DSPy Optimizers to _compile_ our RAG program to higher-quality prompts, raising our scores even more.--Continue here. [Getting Started II: An Example for Basic RAG](/quick-start/getting-started-02)diff --git a/docs/docs/quick-start/getting-started-02.md b/docs/docs/quick-start/getting-started-02.mddeleted file mode 100644index a292fee83..000000000--- a/docs/docs/quick-start/getting-started-02.md+++ /dev/null@@ -1,270 +0,0 @@-----sidebar_position: 3------# Getting Started II: Basic RAG--Let's walk through a quick example of **basic retrieval-augmented generation (RAG)** in DSPy. Specifically, let's build **a system for answering Tech questions**, e.g. about Linux or iPhone apps.--Install the latest DSPy via `pip install -U dspy` and follow along. You may also need to install PyTorch via `pip install torch`.--## Continue from Getting Started I.--In [Getting Started I: Basic Question Answering](/quick-start/getting-started-01), we've set up the DSPy LM, loaded some data, and loaded a metric for evaluation.--First, let's download the corpus data that we will use for RAG search. The next cell will seek to download 4 GBs, so it may take a few minutes. A future version of this notebook will come with a cache that allows you to skip downloads and the PyTorch installation.--```python-import os-import requests--urls = [- 'https://huggingface.co/dspy/cache/resolve/main/ragqa_arena_tech_500.json',- 'https://huggingface.co/datasets/colbertv2/lotte_passages/resolve/main/technology/test_collection.jsonl',- 'https://huggingface.co/dspy/cache/resolve/main/index.pt'-]--for url in urls:- filename = os.path.basename(url)- remote_size = int(requests.head(url, allow_redirects=True).headers.get('Content-Length', 0))- local_size = os.path.getsize(filename) if os.path.exists(filename) else 0-- if local_size != remote_size:- print(f"Downloading '{filename}'...")- with requests.get(url, stream=True) as r, open(filename, 'wb') as f:- for chunk in r.iter_content(chunk_size=8192): f.write(chunk)-```--Having downloaded these items, let's set up the data and other objects from the previous guide.--```python-import ujson-import dspy-from dspy.evaluate import SemanticF1--lm = dspy.LM('openai/gpt-4o-mini')-dspy.configure(lm=lm)--with open('ragqa_arena_tech_500.json') as f:- data = [dspy.Example(**d).with_inputs('question') for d in ujson.load(f)]- trainset, valset, devset, testset = data[:50], data[50:150], data[150:300], data[300:500]--metric = SemanticF1()-evaluate = dspy.Evaluate(devset=devset, metric=metric, num_threads=24, display_progress=True, display_table=3)-```--## Set up your system's retriever.--As far as DSPy is concerned, you can plug in any Python code for calling tools or retrievers. Hence, for our RAG system, we can plug any tools for the search step. Here, we'll just use OpenAI Embeddings and PyTorch for top-K search, but this is not a special choice, just a convenient one.--```python-import torch-import functools-from litellm import embedding as Embed--with open("test_collection.jsonl") as f:- corpus = [ujson.loads(line) for line in f]--index = torch.load('index.pt', weights_only=True)-max_characters = 4000 # >98th percentile of document lengths-[email protected]_cache(maxsize=None)-def search(query, k=5):- query_embedding = torch.tensor(Embed(input=query, model="text-embedding-3-small").data[0]['embedding'])- topk_scores, topk_indices = torch.matmul(index, query_embedding).topk(k)- topK = [dict(score=score.item(), **corpus[idx]) for idx, score in zip(topk_indices, topk_scores)]- return [doc['text'][:max_characters] for doc in topK]-```--## Build your first RAG `Module`.--In the previous guide, we looked at individual DSPy modules in isolation, e.g. `dspy.Predict("question -> answer")`.--What if we want to build a DSPy _program_ that has multiple steps? The syntax below with `dspy.Module` allows you to connect a few pieces together, in this case, our retriever and a generation module, so the whole system can be optimized.--Concretely, in the `__init__` method, you declare any sub-module you'll need, which in this case is just a `dspy.ChainOfThought('context, question -> response')` module that takes retrieved context, a question, and produces a response. In the `forward` method, you simply express any Python control flow you like, possibly using your modules. In this case, we first invoke the `search` function defined earlier and then invoke the `self.respond` ChainOfThought module.--```python-class RAG(dspy.Module):- def __init__(self, num_docs=5):- self.num_docs = num_docs- self.respond = dspy.ChainOfThought('context, question -> response')-- def forward(self, question):- context = search(question, k=self.num_docs)- return self.respond(context=context, question=question)-```--Let's use the RAG module.--```-rag = RAG()-rag(question="what are high memory and low memory on linux?")-```--**Output:**-```-Prediction(- reasoning="High memory and low memory in Linux refer to the organization of memory in the system, particularly in the context of the Linux kernel's virtual memory management. High memory is the portion of physical memory that is not directly mapped by the kernel's page tables, meaning that user-space applications cannot access it directly. Low memory, on the other hand, is the part of memory that the kernel can access directly. In a typical 32-bit architecture, the virtual memory is split into 3 GB for user space (low memory) and 1 GB for kernel space (high memory). The distinction is important for memory management, especially when dealing with physical memory that cannot be mapped contiguously. Understanding this split is crucial for developers working with the Linux kernel, as it affects how memory is allocated and accessed.",- response="In Linux, high memory refers to the portion of physical memory that is not directly mapped by the kernel's page tables, making it inaccessible to user-space applications. Low memory is the segment that the kernel can access directly. In a typical 32-bit architecture, the memory is divided into 3 GB for user space (low memory) and 1 GB for kernel space (high memory). This organization is essential for efficient memory management and affects how the kernel interacts with physical memory, especially in scenarios where contiguous memory is required."-)-```--```python-dspy.inspect_history()-```--**Output:**-See this [gist](https://gist.github.com/okhat/d807032e138862bb54616dcd2f4d481c).---In the previous guide with a CoT module, we got nearly 40% in terms of semantic F1 on our `devset`. Would this `RAG` module score better?---```python-evaluate(RAG())-```--**Output:**-```-Average Metric: 79.44815807647615 / 150 (53.0): 100%|██████████| 150/150 [00:05<00:00, 29.11it/s]-```--| | question | example_response | reasoning | pred_response | SemanticF1 |-|---|---|---|---|---|---|-| 0 | why is mercurial considered to be easier than git? | Mercurial's syntax is considered more familiar, especially for those accustomed to SVN, and is well documented. It focuses on interface aspects, which initially makes learning... | Mercurial is considered easier than Git for several reasons. Firstly, its syntax is more familiar to users coming from Subversion (SVN), making the transition smoother.... | Mercurial is considered easier than Git primarily because of its more familiar syntax for users transitioning from systems like Subversion (SVN), its straightforward workflow without... | ✔️ [0.797] |-| 1 | open finder window from current terminal location? | If you type 'open .' in Terminal, it will open the current directory in a Finder window. Alternatively, you can execute the command open `pwd`... | To open a Finder window from the current terminal location, you can use the command `open .` in the terminal. This command will open a... | You can open a Finder window from your current terminal location by typing the command `open .` in the terminal. This will launch a Finder... | ✔️ [0.857] |-| 2 | how to import secret gpg key (copied from one machine to another)? | It is advised that it is necessary to add `--import` to the command line to import the private key and that according to the man... | To import a secret GPG key that has been copied from one machine to another, you typically need to use the `gpg --import` command. The... | To import a secret GPG key that you have copied from one machine to another, follow these steps: 1. On the original machine, export your... | |---## Using a DSPy `Optimizer` to improve your RAG prompt.--Off the shelf, our `RAG` module scores 53%. What are our options to make it stronger? One of the various choices DSPy offers is optimizing the prompts in our pipeline.--If there are many sub-modules in your program, all of them will be optimized together. In this case, there's only one: `self.respond = dspy.ChainOfThought('context, question -> response')`--Let's set up and use DSPy's [MIPRO (v2) optimizer](/deep-dive/optimizers/miprov2). The run below has a cost around $1.5 (for the `medium` auto setting) and may take some 20-30 minutes depending on your number of threads.---```python-tp = dspy.MIPROv2(metric=metric, auto="medium", num_threads=24) # use fewer threads if your rate limit is small--optimized_rag = tp.compile(RAG(), trainset=trainset, valset=valset,- max_bootstrapped_demos=2, max_labeled_demos=2,- requires_permission_to_run=False)-```--**Output:**-See this [gist](https://gist.github.com/okhat/d6606e480a94c88180441617342699eb).---The prompt optimization process here is pretty systematic, you can learn about it for example in this paper. Importantly, it's not a magic button. It's very possible that it can overfit your training set for instance and not generalize well to a held-out set, making it essential that we iteratively validate our programs.--Let's check on an example here, asking the same question to the baseline `rag = RAG()` program, which was not optimized, and to the `optimized_rag = MIPROv2(..)(..)` program, after prompt optimization.---```python-baseline = rag(question="cmd+tab does not work on hidden or minimized windows")-print(baseline.response)-```--**Output:**-```-You are correct; cmd+Tab does not activate hidden or minimized windows in macOS. It functions as an application switcher, allowing you to switch between open applications, but it does not bring up minimized windows. To access minimized windows, you would need to click on them directly or use other shortcuts.-```---```python-pred = optimized_rag(question="cmd+tab does not work on hidden or minimized windows")-print(pred.response)-```--**Output:**-```-In macOS, the Command+Tab shortcut is specifically designed to switch between applications rather than individual windows. This means that if an application is minimized or hidden, it will not appear in the Command+Tab application switcher. Therefore, you cannot use Command+Tab to access minimized or hidden windows directly.--If you want to bring a minimized window back into view, you can click on the application's icon in the Dock, or you can use the Command+M shortcut to minimize the current window. For switching between windows of the same application, you can use Command+` (the backtick key) to cycle through open windows of the active application.--For users who prefer a behavior similar to Windows, where minimized windows can be accessed through a single shortcut, third-party applications like HyperSwitch or Witch can provide additional functionality to manage window switching more effectively.-```--You can use `dspy.inspect_history(n=2)` to view the RAG prompt [before optimization](https://gist.github.com/okhat/5d04648f2226e72e66e26a8cb1456ee4) and [after optimization](https://gist.github.com/okhat/79405b8889b4b07da577ee19f1a3479a).--Concretely, the optimized prompt:--1. Constructs the following instruction,-```-Using the provided `context` and `question`, analyze the information step by step to generate a comprehensive and informative `response`. Ensure that the response clearly explains the concepts involved, highlights key distinctions, and addresses any complexities noted in the context.-```--2. And includes two fully worked out RAG examples with synthetic reasoning and answers, e.g. `how to transfer whatsapp voice message to computer?`.--Let's now evaluate on the overall devset.---```python-evaluate(optimized_rag)-```--**Output:**-```-Average Metric: 92.16999654981839 / 150 (61.4): 100%|██████████| 150/150 [00:00<00:00, 399.21it/s]-```---| | question | example_response | reasoning | pred_response | SemanticF1 |-|---|---|---|---|---|---|-| 0 | why is mercurial considered to be easier than git? | Mercurial's syntax is considered more familiar, especially for those accustomed to SVN, and is well documented. It focuses on interface aspects, which initially makes learning... | Mercurial is often considered easier than Git due to its user-friendly design and interface, which is particularly appealing to those new to version control systems... | Mercurial is considered easier than Git for several reasons: 1. **Familiar Syntax**: Mercurial's command syntax is often seen as more intuitive, especially for users coming... | ✔️ [0.874] |-| 1 | open finder window from current terminal location? | If you type 'open .' in Terminal, it will open the current directory in a Finder window. Alternatively, you can execute the command open `pwd`... | To open a Finder window from the current terminal location on a Mac, there are several methods available. The simplest way is to use the... | To open a Finder window from your current terminal location on a Mac, you can use the following methods: 1. **Using Terminal Command**: - Simply... | ✔️ [0.333] |-| 2 | how to import secret gpg key (copied from one machine to another)? | It is advised that it is necessary to add `--import` to the command line to import the private key and that according to the man... | To import a secret GPG key that has been copied from one machine to another, it is essential to follow a series of steps that... | To import a secret GPG key that you have copied from one machine to another, follow these steps: 1. **Export the Secret Key from the... | |---## Keeping an eye on cost.--DSPy allows you to track the cost of your programs, which can be used to monitor the cost of your calls. Here, we'll show you how to track the cost of your programs with DSPy.--```python-sum([x['cost'] for x in lm.history if x['cost'] is not None]) # in USD, as calculated by LiteLLM for certain providers-```--## Saving and loading.--The optimized program has a pretty simple structure on the inside. Feel free to explore it.--Here, we'll save `optimized_rag` so we can load it again later without having to optimize from scratch.--```python-optimized_rag.save("optimized_rag.json")--loaded_rag = RAG()-loaded_rag.load("optimized_rag.json")--loaded_rag(question="cmd+tab does not work on hidden or minimized windows")-```--**Output:**-```-Prediction(- reasoning='The behavior of the Command+Tab shortcut in macOS is designed to switch between applications rather than individual windows. When an application is minimized or hidden, it does not appear in the application switcher, which is why Command+Tab does not work for those windows. Understanding this limitation is important for users who expect similar functionality to that found in other operating systems, such as Windows, where Alt+Tab can switch between all open windows, including minimized ones.',- response="In macOS, the Command+Tab shortcut is specifically designed to switch between applications rather than individual windows. This means that if an application is minimized or hidden, it will not appear in the Command+Tab application switcher. Therefore, you cannot use Command+Tab to access minimized or hidden windows directly.\n\nIf you want to bring a minimized window back into view, you can click on the application's icon in the Dock, or you can use the Command+M shortcut to minimize the current window. For switching between windows of the same application, you can use Command+` (the backtick key) to cycle through open windows of the active application.\n\nFor users who prefer a behavior similar to Windows, where minimized windows can be accessed through a single shortcut, third-party applications like HyperSwitch or Witch can provide additional functionality to manage window switching more effectively."-)-```--## What's next?--Improving from just below 40% to above 60% on this task, in terms of `SemanticF1`, was pretty easy.--But DSPy gives you paths to continue iterating on the quality of your system and we have barely scratched the surface.--In general, you have the following tools:--1. Explore better system architectures for your program, e.g. what if we ask the LM to generate search queries for the retriever? See this [notebook](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/intro.ipynb) or the [STORM pipeline](https://arxiv.org/abs/2402.14207) built in DSPy.-2. Explore different [prompt optimizers](https://arxiv.org/abs/2406.11695) or [weight optimizers](https://arxiv.org/abs/2407.10930). See the **[Optimizers Docs](/building-blocks/6-optimizers)**.-3. Scale inference time compute using DSPy Optimizers, e.g. this [notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/agents/multi_agent.ipynb).-4. Cut cost by distilling to a smaller LM, via prompt or weight optimization, e.g. [this notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/nli/scone/scone.ipynb) or [this notebook](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/qa/hotpot/multihop_finetune.ipynb).--How do you decide which ones to proceed with first?--The first step is to look at your system outputs, which will allow you to identify the sources of lower performance if any. While doing all of this, make sure you continue to refine your metric, e.g. by optimizing against your judgments, and to collect more (or more realistic) data, e.g. from related domains or from putting a demo of your system in front of users.--Learn more about the [development cycle](/building-blocks/solving_your_task) in DSPy.diff --git a/docs/docs/quick-start/installation.md b/docs/docs/quick-start/installation.mddeleted file mode 100644index 86def90dc..000000000--- a/docs/docs/quick-start/installation.md+++ /dev/null@@ -1,15 +0,0 @@-----sidebar_position: 1------# Installation--To install DSPy run:---```text-pip install dspy-```--Or open our intro notebook in Google Colab: [<img align="center" src="https://colab.research.google.com/assets/colab-badge.svg" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/intro.ipynb)-diff --git a/docs/docs/stylesheets/extra.css b/docs/docs/stylesheets/extra.cssindex a5ab84212..326e27fcb 100644--- a/docs/docs/stylesheets/extra.css+++ b/docs/docs/stylesheets/extra.css@@ -32,12 +32,76 @@filter: brightness(0.9);}+/* Default max-width */.md-content {max-width: 1000px;margin: 0 auto;}++/* Adjust width when a sidebar or TOC is present */+@media (min-width: 1700px) {+ /* Increase width when there's enough space (like on desktop or tablet landscape) */+ .md-content {+ min-width: 1000px;+ }+}++/* Fully responsive on smaller screens */+/* @media (max-width: 6px) {+ .md-content {+ max-width: 98%;+ }+} */++/* Justified text for main content */.md-content__inner p {text-align: justify;}+++/* Base styling for the output area */+.jp-Cell-outputWrapper .jp-OutputArea-output pre {+ max-height: 300px;+ overflow-y: auto;+ padding: 10px 15px;+ border-radius: 5px;+ font-family: monospace;+ font-size: 0.9em;+}++/* Light mode specific styling */+:root {+ --output-bg-light: #fafafa;+ --output-border-light: #ddd;+ --output-text-light: #333;+}++body[data-md-color-scheme="default"] .jp-Cell-outputWrapper .jp-OutputArea-output pre {+ background-color: var(--output-bg-light);+ border: 1px solid var(--output-border-light);+ color: var(--output-text-light);+ box-shadow: 0px 1px 4px rgba(0, 0, 0, 0.1);+}++/* Dark mode specific styling */+:root {+ --output-bg-dark: #2e2e2e;+ --output-border-dark: #555;+ --output-text-dark: #e0e0e0;+}++body[data-md-color-scheme="slate"] .jp-Cell-outputWrapper .jp-OutputArea-output pre {+ background-color: var(--output-bg-dark);+ border: 1px solid var(--output-border-dark);+ color: var(--output-text-dark);+ box-shadow: 0px 1px 4px rgba(255, 255, 255, 0.1);+}++++/* Set a fixed width for the sidebar */+.md-sidebar {+ width: 210px;+}diff --git a/docs/docs/tutorials/_category_.json b/docs/docs/tutorials/_category_.jsondeleted file mode 100644index 27038011b..000000000--- a/docs/docs/tutorials/_category_.json+++ /dev/null@@ -1,9 +0,0 @@-{- "label": "Tutorials",- "position": 4,- "link": {- "type": "generated-index",- "description": "Step-by-step illustrations of solving a task in DSPy.",- }-}-diff --git a/docs/docs/tutorials/examples.md b/docs/docs/tutorials/old/examples.mdsimilarity index 100%rename from docs/docs/tutorials/examples.mdrename to docs/docs/tutorials/old/examples.mddiff --git a/docs/docs/tutorials/old/intro-tutorialX.md b/docs/docs/tutorials/old/intro-tutorialX.mdnew file mode 100644index 000000000..6d3f25e80--- /dev/null+++ b/docs/docs/tutorials/old/intro-tutorialX.md@@ -0,0 +1,10 @@+<!-- ---+sidebar_position: 1+hide:+ - navigation+ # - toc+---++::: notebook+intro-tutorial.ipynb+::: -->diff --git a/docs/docs/tutorials/rag.md b/docs/docs/tutorials/old/rag.mdsimilarity index 100%rename from docs/docs/tutorials/rag.mdrename to docs/docs/tutorials/old/rag.mddiff --git a/docs/docs/tutorials/simplified-baleen.md b/docs/docs/tutorials/old/simplified-baleen.mdsimilarity index 100%rename from docs/docs/tutorials/simplified-baleen.mdrename to docs/docs/tutorials/old/simplified-baleen.mddiff --git a/docs/docs/tutorials/summarization.md b/docs/docs/tutorials/old/summarization.mdsimilarity index 100%rename from docs/docs/tutorials/summarization.mdrename to docs/docs/tutorials/old/summarization.mddiff --git a/docs/docs/tutorials/rag/index.ipynb b/docs/docs/tutorials/rag/index.ipynbnew file mode 100644index 000000000..f4adc8935--- /dev/null+++ b/docs/docs/tutorials/rag/index.ipynb@@ -0,0 +1,1289 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Tutorial: Retrieval-Augmented Generation (RAG)\n",+ "\n",+ "Let's walk through a quick example of **basic question answering** with and without **retrieval-augmented generation** (RAG) in DSPy. Specifically, let's build **a system for answering Tech questions**, e.g. about Linux or iPhone apps.\n",+ "\n",+ "Install the latest DSPy via `pip install -U dspy` and follow along. If you're looking instead for a conceptual overview of DSPy, this [recent lecture](https://www.youtube.com/live/JEMYuzrKLUw) is a good place to start.\n",+ "\n",+ "## Configuring the DSPy environment\n",+ "\n",+ "Let's tell DSPy that we will use OpenAI's `gpt-4o-mini` in our modules. To authenticate, DSPy will look into your `OPENAI_API_KEY`. You can easily swap this out for [other providers or local models](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb).\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "metadata": {},+ "outputs": [],+ "source": [+ "import dspy\n",+ "\n",+ "lm = dspy.LM('openai/gpt-4o-mini')\n",+ "dspy.configure(lm=lm)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Exploring some basic DSPy `Module`s.\n",+ "\n",+ "You can always prompt the LM directly via `lm(prompt=\"prompt\")` or `lm(messages=[...])`. However, DSPy gives you `Modules` as a better way to define your LM functions.\n",+ "\n",+ "The simplest module is `dspy.Predict`. It takes a [DSPy Signature](/building-blocks/2-signatures), i.e. a structured input/output schema, and gives you back a callable function for the behavior you specified. Let's use the \"in-line\" notation for signatures to declare a module that takes a `question` (of type `str`) as input and produces a `response` as an output."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "In Linux, \"high memory\" and \"low memory\" refer to different regions of the system's memory address space, particularly in the context of 32-bit architectures.\n",+ "\n",+ "- **Low Memory**: This typically refers to the first 896 MB of memory in a 32-bit system. The kernel can directly access this memory without any special handling. It is used for kernel data structures and for user processes. The low memory region is crucial for the kernel's operation, as it allows for efficient memory management and access.\n",+ "\n",+ "- **High Memory**: This refers to memory above the 896 MB threshold in a 32-bit system. The kernel cannot directly access this memory; instead, it must use special mechanisms to map it into the kernel's address space when needed. High memory is often used for user processes and can be allocated dynamically, but it requires additional overhead for the kernel to manage.\n",+ "\n",+ "In 64-bit systems, the distinction between high and low memory is less relevant, as the addressable memory space is significantly larger, and the kernel can access most of the memory directly.\n"+ ]+ }+ ],+ "source": [+ "qa = dspy.Predict('question: str -> response: str')\n",+ "response = qa(question=\"what are high memory and low memory on linux?\")\n",+ "\n",+ "print(response.response)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Notice how the variable names we specified in the signature defined our input and output argument names and their role.\n",+ "\n",+ "Now, what did DSPy do to build this `qa` module? Nothing fancy in this example, yet. The module passed your signature, LM, and inputs to an [Adapter](/building-blocks/1-language_models#structured-lm-output-with-adapters), which is a layer that handles structuring the inputs and parsing structured outputs to fit your signature.\n",+ "\n",+ "Let's see it directly. You can inspect the `n` last prompts sent by DSPy easily.\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\n",+ "\n",+ "\u001b[34m[2024-11-10T12:39:19.458514]\u001b[0m\n",+ "\n",+ "\u001b[31mSystem message:\u001b[0m\n",+ "\n",+ "Your input fields are:\n",+ "1. `question` (str)\n",+ "\n",+ "Your output fields are:\n",+ "1. `response` (str)\n",+ "\n",+ "All interactions will be structured in the following way, with the appropriate values filled in.\n",+ "\n",+ "[[ ## question ## ]]\n",+ "{question}\n",+ "\n",+ "[[ ## response ## ]]\n",+ "{response}\n",+ "\n",+ "[[ ## completed ## ]]\n",+ "\n",+ "In adhering to this structure, your objective is: \n",+ " Given the fields `question`, produce the fields `response`.\n",+ "\n",+ "\n",+ "\u001b[31mUser message:\u001b[0m\n",+ "\n",+ "[[ ## question ## ]]\n",+ "what are high memory and low memory on linux?\n",+ "\n",+ "Respond with the corresponding output fields, starting with the field `[[ ## response ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`.\n",+ "\n",+ "\n",+ "\u001b[31mResponse:\u001b[0m\n",+ "\n",+ "\u001b[32m[[ ## response ## ]]\n",+ "In Linux, \"high memory\" and \"low memory\" refer to different regions of the system's memory address space, particularly in the context of 32-bit architectures.\n",+ "\n",+ "- **Low Memory**: This typically refers to the first 896 MB of memory in a 32-bit system. The kernel can directly access this memory without any special handling. It is used for kernel data structures and for user processes. The low memory region is crucial for the kernel's operation, as it allows for efficient memory management and access.\n",+ "\n",+ "- **High Memory**: This refers to memory above the 896 MB threshold in a 32-bit system. The kernel cannot directly access this memory; instead, it must use special mechanisms to map it into the kernel's address space when needed. High memory is often used for user processes and can be allocated dynamically, but it requires additional overhead for the kernel to manage.\n",+ "\n",+ "In 64-bit systems, the distinction between high and low memory is less relevant, as the addressable memory space is significantly larger, and the kernel can access most of the memory directly.\n",+ "\n",+ "[[ ## completed ## ]]\u001b[0m\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n"+ ]+ }+ ],+ "source": [+ "dspy.inspect_history(n=1)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "DSPy has various built-in modules, e.g. `dspy.ChainOfThought`, `dspy.ProgramOfThought`, and `dspy.ReAct`. These are interchangeable with basic `dspy.Predict`: they take your signature, which is specific to your task, and they apply general-purpose prompting techniques and inference-time strategies to it.\n",+ "\n",+ "For example, `dspy.ChainOfThought` is an easy way to elicit `reasoning` out of your LM before it commits to the outputs requested in your signature.\n",+ "\n",+ "In the example below, we'll omit `str` types (as the default type is string). You should feel free to experiment with other fields and types, e.g. try `topics: list[str]` or `is_realistic: bool`.\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Prediction(\n",+ " reasoning=\"The placement of curly braces on their own line is largely a matter of coding style and conventions. In some programming languages and style guides, such as those used in C, C++, and Java, it is common to place opening curly braces on the same line as the control statement (like `if`, `for`, etc.) and closing braces on a new line. However, other styles, such as the Allman style, advocate for placing both opening and closing braces on their own lines. Ultimately, the decision should be based on the team's coding standards or personal preference, as long as it maintains readability and consistency.\",\n",+ " response=\"Curly braces can either appear on their own line or not, depending on the coding style you choose to follow. It's important to adhere to a consistent style throughout your codebase.\"\n",+ ")"+ ]+ },+ "execution_count": 4,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "cot = dspy.ChainOfThought('question -> response')\n",+ "cot(question=\"should curly braces appear on their own line?\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "\n",+ "Interestingly, asking for reasoning made the output `response` shorter in this case. Is this a good thing or a bad thing? It depends on what you need: there's no free lunch, but DSPy gives you the tools to experiment with different strategies extremely quickly.\n",+ "\n",+ "By the way, `dspy.ChainOfThought` is implemented in DSPy, using `dspy.Predict`. This is a good place to `dspy.inspect_history` if you're curious.\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Using DSPy well involves evaluation and iterative development.\n",+ "\n",+ "You already know a lot about DSPy at this point. If all you want is quick scripting, this much of DSPy already enables a lot. Sprinkling DSPy signatures and modules into your Python control flow is a pretty ergonomic way to just get stuff done with LMs.\n",+ "\n",+ "That said, you're likely here because you want to build a high-quality system and improve it over time. The way to do that in DSPy is to iterate fast by evaluating the quality of your system and using DSPy's powerful tools, e.g. [Optimizers](/building-blocks/6-optimizers). You can learn about the [appropriate development cycle in DSPy here](/building-blocks/solving_your_task).\n",+ "\n",+ "## Manipulating `Example`s in DSPy.\n",+ "\n",+ "To measure the quality of your DSPy system, you need (1) a bunch of input values, like `question`s for example, and (2) a `metric` that can score the quality of an output from your system. Metrics vary widely. Some metrics need ground-truth labels of ideal outputs, e.g. for classification or question answering. Other metrics are self-supervised, e.g. checking faithfulness or lack of hallucination, perhaps using a DSPy program as a judge of these qualities.\n",+ "\n",+ "Let's load a dataset of questions and their (pretty long) gold answers. Since we started this notebook with the goal of building **a system for answering Tech questions**, we obtained a bunch of StackExchange-based questions and their correct answers from the [RAG-QA Arena](https://arxiv.org/abs/2407.13998) dataset. (Learn more about the [development cycle](/building-blocks/solving_your_task) if you don't have data for your task.)\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {},+ "outputs": [],+ "source": [+ "import os\n",+ "import ujson\n",+ "import requests\n",+ "\n",+ "def download(url):\n",+ " filename = os.path.basename(url)\n",+ " remote_size = int(requests.head(url, allow_redirects=True).headers.get('Content-Length', 0))\n",+ " local_size = os.path.getsize(filename) if os.path.exists(filename) else 0\n",+ "\n",+ " if local_size != remote_size:\n",+ " print(f\"Downloading '{filename}'...\")\n",+ " with requests.get(url, stream=True) as r, open(filename, 'wb') as f:\n",+ " for chunk in r.iter_content(chunk_size=8192): f.write(chunk)\n",+ "\n",+ "# Download 500 question--answer pairs from the RAG-QA Arena \"Tech\" dataset.\n",+ "download(\"https://huggingface.co/dspy/cache/resolve/main/ragqa_arena_tech_500.json\")\n",+ "\n",+ "with open('ragqa_arena_tech_500.json') as f:\n",+ " data = ujson.load(f)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "{'question': 'how to transfer whatsapp voice message to computer?',\n",+ " 'response': 'To transfer voice notes from WhatsApp on your device to your computer, you have the option to select the \"Share\" feature within the app and send the files via Email, Gmail, Bluetooth, or other available services. \\nYou can also move the files onto your phone\\'s SD card, connect your phone to your computer via a USB cable, then find and transfer the files via File Explorer on your PC. \\nAlternatively, you can choose to attach all the desired voice notes to an email and, from your phone, send them to your own email address. \\nUpon receiving the email on your computer, you can then download the voice note attachments.'}"+ ]+ },+ "execution_count": 6,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "# Inspect one datapoint.\n",+ "data[0]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "\n",+ "Given a simple dict like this, let's create a list of `dspy.Example`s, which is the datatype that carries training (or test) datapoints in DSPy.\n",+ "\n",+ "When you build a `dspy.Example`, you should generally specify `.with_inputs(\"field1\", \"field2\", ...)` to indicate which fields are inputs. The other fields are treated as labels or metadata.\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Example({'question': 'what are high memory and low memory on linux?', 'response': '\"High Memory\" refers to the application or user space, the memory that user programs can use and which isn\\'t permanently mapped in the kernel\\'s space, while \"Low Memory\" is the kernel\\'s space, which the kernel can address directly and is permanently mapped. \\nThe user cannot access the Low Memory as it is set aside for the required kernel programs.'}) (input_keys={'question'})"+ ]+ },+ "execution_count": 7,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "data = [dspy.Example(**d).with_inputs('question') for d in data]\n",+ "\n",+ "# Let's pick an `example` here from the data.\n",+ "example = data[2]\n",+ "example"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "Now, let's divide the data into:\n",+ "\n",+ "- Training and Validation sets:\n",+ " - These are the splits you typically give to DSPy optimizers.\n",+ " - Optimizers typically learn directly from the training examples and check their progress using the validation examples.\n",+ " - It's good to have 30--300 examples for training and validation each.\n",+ " - For prompt optimizers in particular, it's often better to pass _more_ validation than training.\n",+ "\n",+ "- Development and Test sets: The rest, typically on the order of 30--1000, can be used for:\n",+ " - development (i.e., you can inspect them as you iterate on your system) and\n",+ " - testing (final held-out evaluation).\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "(50, 100, 150, 200)"+ ]+ },+ "execution_count": 8,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "trainset, valset, devset, testset = data[:50], data[50:150], data[150:300], data[300:500]\n",+ "\n",+ "len(trainset), len(valset), len(devset), len(testset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Evaluation in DSPy.\n",+ "\n",+ "What kind of metric can suit our question-answering task? There are many choices, but since the answers are long, we may ask: How well does the system response _cover_ all key facts in the gold response? And the other way around, how well is the system response _not saying things_ that aren't in the gold response?\n",+ "\n",+ "That metric is essentially a **semantic F1**, so let's load a `SemanticF1` metric from DSPy. This metric is actually implemented as a [very simple DSPy module](https://github.com/stanfordnlp/dspy/blob/77c2e1cceba427c7f91edb2ed5653276fb0c6de7/dspy/evaluate/auto_evaluation.py#L21) using whatever LM we're working with.\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Question: \t what are high memory and low memory on linux?\n",+ "\n",+ "Gold Response: \t \"High Memory\" refers to the application or user space, the memory that user programs can use and which isn't permanently mapped in the kernel's space, while \"Low Memory\" is the kernel's space, which the kernel can address directly and is permanently mapped. \n",+ "The user cannot access the Low Memory as it is set aside for the required kernel programs.\n",+ "\n",+ "Predicted Response: \t In Linux, \"low memory\" refers to the first 896 MB of RAM, which is directly accessible by the kernel and used for kernel operations and user processes. \"High memory\" refers to memory above this limit, which is not directly accessible by the kernel in 32-bit systems and is used for user processes, requiring special handling to access. This distinction is crucial for effective memory management in Linux.\n",+ "\n",+ "Semantic F1 Score: 0.87\n"+ ]+ }+ ],+ "source": [+ "from dspy.evaluate import SemanticF1\n",+ "\n",+ "# Instantiate the metric.\n",+ "metric = SemanticF1()\n",+ "\n",+ "# Produce a prediction from our `cot` module, using the `example` above as input.\n",+ "pred = cot(**example.inputs())\n",+ "\n",+ "# Compute the metric score for the prediction.\n",+ "score = metric(example, pred)\n",+ "\n",+ "print(f\"Question: \\t {example.question}\\n\")\n",+ "print(f\"Gold Response: \\t {example.response}\\n\")\n",+ "print(f\"Predicted Response: \\t {pred.response}\\n\")\n",+ "print(f\"Semantic F1 Score: {score:.2f}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "The final DSPy module call above actually happens inside `metric`. You might be curious how it measured the semantic F1 for this example.\n",+ "\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\n",+ "\n",+ "\u001b[34m[2024-11-10T12:39:19.701005]\u001b[0m\n",+ "\n",+ "\u001b[31mSystem message:\u001b[0m\n",+ "\n",+ "Your input fields are:\n",+ "1. `question` (str)\n",+ "2. `ground_truth` (str)\n",+ "3. `system_response` (str)\n",+ "\n",+ "Your output fields are:\n",+ "1. `reasoning` (str)\n",+ "2. `recall` (float): fraction (out of 1.0) of ground truth covered by the system response\n",+ "3. `precision` (float): fraction (out of 1.0) of system response covered by the ground truth\n",+ "\n",+ "All interactions will be structured in the following way, with the appropriate values filled in.\n",+ "\n",+ "[[ ## question ## ]]\n",+ "{question}\n",+ "\n",+ "[[ ## ground_truth ## ]]\n",+ "{ground_truth}\n",+ "\n",+ "[[ ## system_response ## ]]\n",+ "{system_response}\n",+ "\n",+ "[[ ## reasoning ## ]]\n",+ "{reasoning}\n",+ "\n",+ "[[ ## recall ## ]]\n",+ "{recall} # note: the value you produce must be a single float value\n",+ "\n",+ "[[ ## precision ## ]]\n",+ "{precision} # note: the value you produce must be a single float value\n",+ "\n",+ "[[ ## completed ## ]]\n",+ "\n",+ "In adhering to this structure, your objective is: \n",+ " Compare a system's response to the ground truth to compute its recall and precision.\n",+ " If asked to reason, enumerate key ideas in each response, and whether they are present in the other response.\n",+ "\n",+ "\n",+ "\u001b[31mUser message:\u001b[0m\n",+ "\n",+ "[[ ## question ## ]]\n",+ "what are high memory and low memory on linux?\n",+ "\n",+ "[[ ## ground_truth ## ]]\n",+ "\"High Memory\" refers to the application or user space, the memory that user programs can use and which isn't permanently mapped in the kernel's space, while \"Low Memory\" is the kernel's space, which the kernel can address directly and is permanently mapped. \n",+ "The user cannot access the Low Memory as it is set aside for the required kernel programs.\n",+ "\n",+ "[[ ## system_response ## ]]\n",+ "In Linux, \"low memory\" refers to the first 896 MB of RAM, which is directly accessible by the kernel and used for kernel operations and user processes. \"High memory\" refers to memory above this limit, which is not directly accessible by the kernel in 32-bit systems and is used for user processes, requiring special handling to access. This distinction is crucial for effective memory management in Linux.\n",+ "\n",+ "Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## recall ## ]]` (must be formatted as a valid Python float), then `[[ ## precision ## ]]` (must be formatted as a valid Python float), and then ending with the marker for `[[ ## completed ## ]]`.\n",+ "\n",+ "\n",+ "\u001b[31mResponse:\u001b[0m\n",+ "\n",+ "\u001b[32m[[ ## reasoning ## ]]\n",+ "The system response correctly identifies \"Low Memory\" as the portion of RAM directly accessible by the kernel and used for kernel operations, which aligns with the ground truth. It also mentions that \"High Memory\" refers to memory above a certain limit that is not directly accessible by the kernel, which is consistent with the ground truth's description of user space. However, the ground truth emphasizes that Low Memory is set aside for kernel programs and that users cannot access it, which is not explicitly stated in the system response. Overall, the key ideas are present, but the system response lacks the explicit mention of user access limitations for Low Memory.\n",+ "\n",+ "[[ ## recall ## ]]\n",+ "0.85\n",+ "\n",+ "[[ ## precision ## ]]\n",+ "0.90\n",+ "\n",+ "[[ ## completed ## ]]\u001b[0m\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n"+ ]+ }+ ],+ "source": [+ "dspy.inspect_history(n=1)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "For evaluation, you could use the metric above in a simple loop and just average the score. But for nice parallelism and utilities, we can rely on `dspy.Evaluate`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 11,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 55.380830691218016 / 150 (36.9): 100%|██████████| 150/150 [00:00<00:00, 513.51it/s]\n",+ "2024/11/10 12:39:20 INFO dspy.evaluate.evaluate: Average Metric: 55.380830691218016 / 150 (36.9%)\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>question</th>\n",+ " <th>example_response</th>\n",+ " <th>reasoning</th>\n",+ " <th>pred_response</th>\n",+ " <th>SemanticF1</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>why is mercurial considered to be easier than git?</td>\n",+ " <td>Mercurial's syntax is considered more familiar, especially for tho...</td>\n",+ " <td>Mercurial is often considered easier than Git for several reasons....</td>\n",+ " <td>Mercurial is considered easier than Git primarily due to its simpl...</td>\n",+ " <td>✔️ [0.545]</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>open finder window from current terminal location?</td>\n",+ " <td>If you type 'open .' in Terminal, it will open the current directo...</td>\n",+ " <td>To open a Finder window from the current terminal location on a Ma...</td>\n",+ " <td>You can open a Finder window from your current terminal location b...</td>\n",+ " <td>✔️ [0.667]</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " question \\\n",+ "0 why is mercurial considered to be easier than git? \n",+ "1 open finder window from current terminal location? \n",+ "\n",+ " example_response \\\n",+ "0 Mercurial's syntax is considered more familiar, especially for tho... \n",+ "1 If you type 'open .' in Terminal, it will open the current directo... \n",+ "\n",+ " reasoning \\\n",+ "0 Mercurial is often considered easier than Git for several reasons.... \n",+ "1 To open a Finder window from the current terminal location on a Ma... \n",+ "\n",+ " pred_response \\\n",+ "0 Mercurial is considered easier than Git primarily due to its simpl... \n",+ "1 You can open a Finder window from your current terminal location b... \n",+ "\n",+ " SemanticF1 \n",+ "0 ✔️ [0.545] \n",+ "1 ✔️ [0.667] "+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center;\n",+ " font-size: 16px;\n",+ " font-weight: bold;\n",+ " color: #555;\n",+ " margin: 10px 0;'>\n",+ " ... 148 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "36.92"+ ]+ },+ "execution_count": 11,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "# Define an evaluator that we can re-use.\n",+ "evaluate = dspy.Evaluate(devset=devset, metric=metric, num_threads=24,\n",+ " display_progress=True, display_table=2)\n",+ "\n",+ "# Evaluate the Chain-of-Thought program.\n",+ "evaluate(cot)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "So far, we built a very simple chain-of-thought module for question answering and evaluated it on a small dataset.\n",+ "\n",+ "Can we do better? In the rest of this guide, we will build a retrieval-augmented generation (RAG) program in DSPy for the same task. We'll see how this can boost the score substantially, then we'll use one of the DSPy Optimizers to _compile_ our RAG program to higher-quality prompts, raising our scores even more."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Basic RAG\n",+ "\n",+ "First, let's download the corpus data that we will use for RAG search. The next cell will seek to download 4 GBs, so it may take a few minutes. A future version of this notebook will come with a cache that allows you to skip downloads and the PyTorch installation."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 12,+ "metadata": {},+ "outputs": [],+ "source": [+ "download('https://huggingface.co/datasets/colbertv2/lotte_passages/resolve/main/technology/test_collection.jsonl')\n",+ "download('https://huggingface.co/dspy/cache/resolve/main/index.pt')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Set up your system's retriever.\n",+ "\n",+ "As far as DSPy is concerned, you can plug in any Python code for calling tools or retrievers. Hence, for our RAG system, we can plug any tools for the search step. Here, we'll just use OpenAI Embeddings and PyTorch for top-K search, but this is not a special choice, just a convenient one."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 13,+ "metadata": {},+ "outputs": [],+ "source": [+ "import torch\n",+ "import functools\n",+ "from litellm import embedding as Embed\n",+ "\n",+ "with open(\"test_collection.jsonl\") as f:\n",+ " corpus = [ujson.loads(line) for line in f]\n",+ "\n",+ "index = torch.load('index.pt', weights_only=True)\n",+ "max_characters = 4000 # >98th percentile of document lengths\n",+ "\n",+ "@functools.lru_cache(maxsize=None)\n",+ "def search(query, k=5):\n",+ " query_embedding = torch.tensor(Embed(input=query, model=\"text-embedding-3-small\").data[0]['embedding'])\n",+ " topk_scores, topk_indices = torch.matmul(index, query_embedding).topk(k)\n",+ " topK = [dict(score=score.item(), **corpus[idx]) for idx, score in zip(topk_indices, topk_scores)]\n",+ " return [doc['text'][:max_characters] for doc in topK]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "## Build your first RAG `Module`.\n",+ "\n",+ "In the previous guide, we looked at individual DSPy modules in isolation, e.g. `dspy.Predict(\"question -> answer\")`.\n",+ "\n",+ "What if we want to build a DSPy _program_ that has multiple steps? The syntax below with `dspy.Module` allows you to connect a few pieces together, in this case, our retriever and a generation module, so the whole system can be optimized.\n",+ "\n",+ "Concretely, in the `__init__` method, you declare any sub-module you'll need, which in this case is just a `dspy.ChainOfThought('context, question -> response')` module that takes retrieved context, a question, and produces a response. In the `forward` method, you simply express any Python control flow you like, possibly using your modules. In this case, we first invoke the `search` function defined earlier and then invoke the `self.respond` ChainOfThought module.\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 14,+ "metadata": {},+ "outputs": [],+ "source": [+ "class RAG(dspy.Module):\n",+ " def __init__(self, num_docs=5):\n",+ " self.num_docs = num_docs\n",+ " self.respond = dspy.ChainOfThought('context, question -> response')\n",+ "\n",+ " def forward(self, question):\n",+ " context = search(question, k=self.num_docs)\n",+ " return self.respond(context=context, question=question)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "Let's use the RAG module.\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 15,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Prediction(\n",+ " reasoning=\"High memory and low memory in Linux refer to the way the operating system organizes and manages memory for user-space applications and the kernel. Low memory is the portion of memory that is directly accessible by the kernel, while high memory is the part that is not directly mapped by the kernel's page tables. In a typical 32-bit architecture, low memory usually consists of the lower 3 GB of virtual memory, which is accessible to user-space applications, while high memory comprises the upper 1 GB, which is reserved for the kernel. The distinction is important for memory management, especially in systems with large amounts of RAM, as it affects how the kernel accesses and manages memory resources.\",\n",+ " response=\"In Linux, high memory refers to the portion of memory that is not directly mapped by the kernel's page tables, meaning the kernel cannot access it directly without mapping it into its address space first. Low memory, on the other hand, is the segment of memory that the kernel can access directly. In a typical 32-bit system, low memory consists of the lower 3 GB of virtual memory, while high memory comprises the upper 1 GB. This organization helps manage memory more efficiently, especially in systems with large physical memory.\"\n",+ ")"+ ]+ },+ "execution_count": 15,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "rag = RAG()\n",+ "rag(question=\"what are high memory and low memory on linux?\")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 16,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\n",+ "\n",+ "\u001b[34m[2024-11-10T12:39:22.802994]\u001b[0m\n",+ "\n",+ "\u001b[31mSystem message:\u001b[0m\n",+ "\n",+ "Your input fields are:\n",+ "1. `context` (str)\n",+ "2. `question` (str)\n",+ "\n",+ "Your output fields are:\n",+ "1. `reasoning` (str)\n",+ "2. `response` (str)\n",+ "\n",+ "All interactions will be structured in the following way, with the appropriate values filled in.\n",+ "\n",+ "[[ ## context ## ]]\n",+ "{context}\n",+ "\n",+ "[[ ## question ## ]]\n",+ "{question}\n",+ "\n",+ "[[ ## reasoning ## ]]\n",+ "{reasoning}\n",+ "\n",+ "[[ ## response ## ]]\n",+ "{response}\n",+ "\n",+ "[[ ## completed ## ]]\n",+ "\n",+ "In adhering to this structure, your objective is: \n",+ " Given the fields `context`, `question`, produce the fields `response`.\n",+ "\n",+ "\n",+ "\u001b[31mUser message:\u001b[0m\n",+ "\n",+ "[[ ## context ## ]]\n",+ "[1] «As far as I remember, High Memory is used for application space and Low Memory for the kernel. Advantage is that (user-space) applications cant access kernel-space memory.»\n",+ "[2] «For the people looking for an explanation in the context of Linux kernel memory space, beware that there are two conflicting definitions of the high/low memory split (unfortunately there is no standard, one has to interpret that in context): High memory defined as the totality of kernel space in VIRTUAL memory. This is a region that only the kernel can access and comprises all virtual addresses greater or equal than PAGE_OFFSET. Low memory refers therefore to the region of the remaining addresses, which correspond to the user-space memory accessible from each user process. For example: on 32-bit x86 with a default PAGE_OFFSET, this means that high memory is any address ADDR with ADDR ≥ 0xC0000000 = PAGE_OFFSET (i.e. higher 1 GB). This is the reason why in Linux 32-bit processes are typically limited to 3 GB. Note that PAGE_OFFSET cannot be configured directly, it depends on the configurable VMSPLIT_x options (source). To summarize: in 32-bit archs, virtual memory is by default split into lower 3 GB (user space) and higher 1 GB (kernel space). For 64 bit, PAGE_OFFSET is not configurable and depends on architectural details that are sometimes detected at runtime during kernel load. On x86_64, PAGE_OFFSET is 0xffff888000000000 for 4-level paging (typical) and 0xff11000000000000 for 5-level paging (source). For ARM64 this is usually 0x8000000000000000. Note though, if KASLR is enabled, this value is intentionally unpredictable. High memory defined as the portion of PHYSICAL memory that cannot be mapped contiguously with the rest of the kernel virtual memory. A portion of the kernel virtual address space can be mapped as a single contiguous chunk into the so-called physical low memory. To fully understand what this means, a deeper knowledge of the Linux virtual memory space is required. I would recommend going through these slides. From the slides: This kind of high/low memory split is only applicable to 32-bit architectures where the installed physical RAM size is relatively high (more than ~1 GB). Otherwise, i.e. when the physical address space is small (<1 GB) or when the virtual memory space is large (64 bits), the whole physical space can be accessed from the kernel virtual memory space. In that case, all physical memory is considered low memory. It is preferable that high memory does not exist at all because the whole physical space can be accessed directly from the kernel, which makes memory management a lot simpler and efficient. This is especially important when dealing with DMAs (which typically require physically contiguous memory). See also the answer by @gilles»\n",+ "[3] «Low and High do not refer to whether there is a lot of usage or not. They represent the way it is organized by the system. According to Wikipedia: High Memory is the part of physical memory in a computer which is not directly mapped by the page tables of its operating system kernel. There is no duration for the free command which simply computes a snapshot of the information available. Most people, including programmers, do not need to understand it more clearly as it is managed in a much simpler form through system calls and compiler/interpreter operations.»\n",+ "[4] «This is relevant to the Linux kernel; Im not sure how any Unix kernel handles this. The High Memory is the segment of memory that user-space programs can address. It cannot touch Low Memory. Low Memory is the segment of memory that the Linux kernel can address directly. If the kernel must access High Memory, it has to map it into its own address space first. There was a patch introduced recently that lets you control where the segment is. The tradeoff is that you can take addressable memory away from user space so that the kernel can have more memory that it does not have to map before using. Additional resources: http://tldp.org/HOWTO/KernelAnalysis-HOWTO-7.html http://linux-mm.org/HighMemory»\n",+ "[5] «HIGHMEM is a range of kernels memory space, but it is NOT memory you access but its a place where you put what you want to access. A typical 32bit Linux virtual memory map is like: 0x00000000-0xbfffffff: user process (3GB) 0xc0000000-0xffffffff: kernel space (1GB) (CPU-specific vector and whatsoever are ignored here). Linux splits the 1GB kernel space into 2 pieces, LOWMEM and HIGHMEM. The split varies from installation to installation. If an installation chooses, say, 512MB-512MB for LOW and HIGH mems, the 512MB LOWMEM (0xc0000000-0xdfffffff) is statically mapped at the kernel boot time; usually the first so many bytes of the physical memory is used for this so that virtual and physical addresses in this range have a constant offset of, say, 0xc0000000. On the other hand, the latter 512MB (HIGHMEM) has no static mapping (although you could leave pages semi-permanently mapped there, but you must do so explicitly in your driver code). Instead, pages are temporarily mapped and unmapped here so that virtual and physical addresses in this range have no consistent mapping. Typical uses of HIGHMEM include single-time data buffers.»\n",+ "\n",+ "[[ ## question ## ]]\n",+ "what are high memory and low memory on linux?\n",+ "\n",+ "Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## response ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`.\n",+ "\n",+ "\n",+ "\u001b[31mResponse:\u001b[0m\n",+ "\n",+ "\u001b[32m[[ ## reasoning ## ]]\n",+ "High memory and low memory in Linux refer to the way the operating system organizes and manages memory for user-space applications and the kernel. Low memory is the portion of memory that is directly accessible by the kernel, while high memory is the part that is not directly mapped by the kernel's page tables. In a typical 32-bit architecture, low memory usually consists of the lower 3 GB of virtual memory, which is accessible to user-space applications, while high memory comprises the upper 1 GB, which is reserved for the kernel. The distinction is important for memory management, especially in systems with large amounts of RAM, as it affects how the kernel accesses and manages memory resources.\n",+ "\n",+ "[[ ## response ## ]]\n",+ "In Linux, high memory refers to the portion of memory that is not directly mapped by the kernel's page tables, meaning the kernel cannot access it directly without mapping it into its address space first. Low memory, on the other hand, is the segment of memory that the kernel can access directly. In a typical 32-bit system, low memory consists of the lower 3 GB of virtual memory, while high memory comprises the upper 1 GB. This organization helps manage memory more efficiently, especially in systems with large physical memory.\n",+ "\n",+ "[[ ## completed ## ]]\u001b[0m\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n"+ ]+ }+ ],+ "source": [+ "dspy.inspect_history()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Earlier with a CoT module, we got nearly 37% in terms of semantic F1 on our `devset`. Would this `RAG` module score better?"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 17,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 74.61311832900337 / 150 (49.7): 100%|██████████| 150/150 [00:05<00:00, 27.92it/s] \n",+ "2024/11/10 12:39:28 INFO dspy.evaluate.evaluate: Average Metric: 74.61311832900337 / 150 (49.7%)\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>question</th>\n",+ " <th>example_response</th>\n",+ " <th>reasoning</th>\n",+ " <th>pred_response</th>\n",+ " <th>SemanticF1</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>why is mercurial considered to be easier than git?</td>\n",+ " <td>Mercurial's syntax is considered more familiar, especially for tho...</td>\n",+ " <td>Mercurial is considered easier than Git for several reasons. First...</td>\n",+ " <td>Mercurial is considered easier than Git because it has a more fami...</td>\n",+ " <td>✔️ [0.797]</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>open finder window from current terminal location?</td>\n",+ " <td>If you type 'open .' in Terminal, it will open the current directo...</td>\n",+ " <td>To open a Finder window from the current terminal location, you ca...</td>\n",+ " <td>You can open a Finder window from your current terminal location b...</td>\n",+ " <td>✔️ [0.667]</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " question \\\n",+ "0 why is mercurial considered to be easier than git? \n",+ "1 open finder window from current terminal location? \n",+ "\n",+ " example_response \\\n",+ "0 Mercurial's syntax is considered more familiar, especially for tho... \n",+ "1 If you type 'open .' in Terminal, it will open the current directo... \n",+ "\n",+ " reasoning \\\n",+ "0 Mercurial is considered easier than Git for several reasons. First... \n",+ "1 To open a Finder window from the current terminal location, you ca... \n",+ "\n",+ " pred_response \\\n",+ "0 Mercurial is considered easier than Git because it has a more fami... \n",+ "1 You can open a Finder window from your current terminal location b... \n",+ "\n",+ " SemanticF1 \n",+ "0 ✔️ [0.797] \n",+ "1 ✔️ [0.667] "+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center;\n",+ " font-size: 16px;\n",+ " font-weight: bold;\n",+ " color: #555;\n",+ " margin: 10px 0;'>\n",+ " ... 148 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "49.74"+ ]+ },+ "execution_count": 17,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate(RAG())"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Using a DSPy `Optimizer` to improve your RAG prompt.\n",+ "\n",+ "Off the shelf, our `RAG` module scores nearly 50%. What are our options to make it stronger? One of the various choices DSPy offers is optimizing the prompts in our pipeline.\n",+ "\n",+ "If there are many sub-modules in your program, all of them will be optimized together. In this case, there's only one: `self.respond = dspy.ChainOfThought('context, question -> response')`\n",+ "\n",+ "Let's set up and use DSPy's [MIPRO (v2) optimizer](/deep-dive/optimizers/miprov2). The run below has a cost around $1.5 (for the `medium` auto setting) and may take some 20-30 minutes depending on your number of threads."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "tp = dspy.MIPROv2(metric=metric, auto=\"medium\", num_threads=24) # use fewer threads if your rate limit is small\n",+ "\n",+ "optimized_rag = tp.compile(RAG(), trainset=trainset, valset=valset,\n",+ " max_bootstrapped_demos=2, max_labeled_demos=2,\n",+ " requires_permission_to_run=False)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "The prompt optimization process here is pretty systematic, you can learn about it for example in this paper. Importantly, it's not a magic button. It's very possible that it can overfit your training set for instance and not generalize well to a held-out set, making it essential that we iteratively validate our programs.\n",+ "\n",+ "Let's check on an example here, asking the same question to the baseline `rag = RAG()` program, which was not optimized, and to the `optimized_rag = MIPROv2(..)(..)` program, after prompt optimization."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 19,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "You are correct; cmd+Tab does not work on hidden or minimized windows in macOS. It is designed to switch between applications and will only show non-minimized windows of the active application. To access minimized windows, you need to click on them directly or use other shortcuts.\n"+ ]+ }+ ],+ "source": [+ "baseline = rag(question=\"cmd+tab does not work on hidden or minimized windows\")\n",+ "print(baseline.response)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 20,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "In macOS, the Command+Tab shortcut is specifically designed to switch between applications, not individual windows. This means that if an application is minimized or hidden, it will not be activated using Command+Tab. Here are some alternative methods to manage minimized or hidden windows:\n",+ "\n",+ "1. **Click on the Minimized Window:**\n",+ " - You can directly click on the minimized window in the Dock to restore it.\n",+ "\n",+ "2. **Use Command+M:**\n",+ " - If you want to minimize the current window, you can use Command+M. To restore it, you will need to click on it in the Dock.\n",+ "\n",+ "3. **Use Mission Control:**\n",+ " - You can activate Mission Control (F3 or Control+Up Arrow) to see all open windows and select the one you want to bring to the front.\n",+ "\n",+ "4. **Third-Party Applications:**\n",+ " - Consider using third-party applications like HyperSwitch or Witch, which can provide enhanced window management features, including switching between windows of the same application.\n",+ "\n",+ "5. **Keyboard Shortcuts for Specific Applications:**\n",+ " - Some applications may have their own shortcuts for managing windows. Check the preferences or documentation for the specific application you are using.\n",+ "\n",+ "By using these methods, you can effectively manage and restore minimized or hidden windows in macOS.\n"+ ]+ }+ ],+ "source": [+ "pred = optimized_rag(question=\"cmd+tab does not work on hidden or minimized windows\")\n",+ "print(pred.response)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "You can use `dspy.inspect_history(n=2)` to view the RAG prompt [before optimization](https://gist.github.com/okhat/5d04648f2226e72e66e26a8cb1456ee4) and [after optimization](https://gist.github.com/okhat/79405b8889b4b07da577ee19f1a3479a).\n",+ "\n",+ "Concretely, in of run of this notebook, the optimized prompt:\n",+ "\n",+ "1. Constructs the following instruction,\n",+ "```text\n",+ "Using the provided `context` and `question`, analyze the information step by step to generate a comprehensive and informative `response`. Ensure that the response clearly explains the concepts involved, highlights key distinctions, and addresses any complexities noted in the context.\n",+ "```\n",+ "\n",+ "2. And includes two fully worked out RAG examples with synthetic reasoning and answers, e.g. `how to transfer whatsapp voice message to computer?`.\n",+ "\n",+ "Let's now evaluate on the overall devset."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 21,+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 89.78303512426604 / 150 (59.9): 100%|██████████| 150/150 [00:00<00:00, 424.18it/s]\n",+ "2024/11/10 12:39:36 INFO dspy.evaluate.evaluate: Average Metric: 89.78303512426604 / 150 (59.9%)\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<div>\n",+ "<style scoped>\n",+ " .dataframe tbody tr th:only-of-type {\n",+ " vertical-align: middle;\n",+ " }\n",+ "\n",+ " .dataframe tbody tr th {\n",+ " vertical-align: top;\n",+ " }\n",+ "\n",+ " .dataframe thead th {\n",+ " text-align: right;\n",+ " }\n",+ "</style>\n",+ "<table border=\"1\" class=\"dataframe\">\n",+ " <thead>\n",+ " <tr style=\"text-align: right;\">\n",+ " <th></th>\n",+ " <th>question</th>\n",+ " <th>example_response</th>\n",+ " <th>reasoning</th>\n",+ " <th>pred_response</th>\n",+ " <th>SemanticF1</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th>0</th>\n",+ " <td>why is mercurial considered to be easier than git?</td>\n",+ " <td>Mercurial's syntax is considered more familiar, especially for tho...</td>\n",+ " <td>Mercurial is often considered easier than Git for several reasons,...</td>\n",+ " <td>Mercurial is considered easier than Git for several key reasons: 1...</td>\n",+ " <td>✔️ [0.874]</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th>1</th>\n",+ " <td>open finder window from current terminal location?</td>\n",+ " <td>If you type 'open .' in Terminal, it will open the current directo...</td>\n",+ " <td>To open a Finder window from the current terminal location in macO...</td>\n",+ " <td>To open a Finder window from your current terminal location in mac...</td>\n",+ " <td>✔️ [0.600]</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n",+ "</div>"+ ],+ "text/plain": [+ " question \\\n",+ "0 why is mercurial considered to be easier than git? \n",+ "1 open finder window from current terminal location? \n",+ "\n",+ " example_response \\\n",+ "0 Mercurial's syntax is considered more familiar, especially for tho... \n",+ "1 If you type 'open .' in Terminal, it will open the current directo... \n",+ "\n",+ " reasoning \\\n",+ "0 Mercurial is often considered easier than Git for several reasons,... \n",+ "1 To open a Finder window from the current terminal location in macO... \n",+ "\n",+ " pred_response \\\n",+ "0 Mercurial is considered easier than Git for several key reasons: 1... \n",+ "1 To open a Finder window from your current terminal location in mac... \n",+ "\n",+ " SemanticF1 \n",+ "0 ✔️ [0.874] \n",+ "1 ✔️ [0.600] "+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/html": [+ "\n",+ " <div style='\n",+ " text-align: center;\n",+ " font-size: 16px;\n",+ " font-weight: bold;\n",+ " color: #555;\n",+ " margin: 10px 0;'>\n",+ " ... 148 more rows not displayed ...\n",+ " </div>\n",+ " "+ ],+ "text/plain": [+ "<IPython.core.display.HTML object>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "59.86"+ ]+ },+ "execution_count": 21,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "evaluate(optimized_rag)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Keeping an eye on cost.\n",+ "\n",+ "DSPy allows you to track the cost of your programs, which can be used to monitor the cost of your calls. Here, we'll show you how to track the cost of your programs with DSPy."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 22,+ "metadata": {},+ "outputs": [],+ "source": [+ "cost = sum([x['cost'] for x in lm.history if x['cost'] is not None]) # in USD, as calculated by LiteLLM for certain providers"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Saving and loading.\n",+ "\n",+ "The optimized program has a pretty simple structure on the inside. Feel free to explore it.\n",+ "\n",+ "Here, we'll save `optimized_rag` so we can load it again later without having to optimize from scratch."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 23,+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "Prediction(\n",+ " reasoning='The behavior of the Command+Tab shortcut in macOS is designed to switch between applications rather than individual windows. This means that if an application is minimized or hidden, it will not be brought to the forefront using Command+Tab. Instead, the shortcut will only cycle through applications that are currently open and not minimized. To manage minimized windows, users may need to use different shortcuts or methods to restore them.',\n",+ " response='In macOS, the Command+Tab shortcut is specifically designed to switch between applications, not individual windows. This means that if an application is minimized or hidden, it will not be activated using Command+Tab. Here are some alternative methods to manage minimized or hidden windows:\\n\\n1. **Click on the Minimized Window:**\\n - You can directly click on the minimized window in the Dock to restore it.\\n\\n2. **Use Command+M:**\\n - If you want to minimize the current window, you can use Command+M. To restore it, you will need to click on it in the Dock.\\n\\n3. **Use Mission Control:**\\n - You can activate Mission Control (F3 or Control+Up Arrow) to see all open windows and select the one you want to bring to the front.\\n\\n4. **Third-Party Applications:**\\n - Consider using third-party applications like HyperSwitch or Witch, which can provide enhanced window management features, including switching between windows of the same application.\\n\\n5. **Keyboard Shortcuts for Specific Applications:**\\n - Some applications may have their own shortcuts for managing windows. Check the preferences or documentation for the specific application you are using.\\n\\nBy using these methods, you can effectively manage and restore minimized or hidden windows in macOS.'\n",+ ")"+ ]+ },+ "execution_count": 23,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "optimized_rag.save(\"optimized_rag.json\")\n",+ "\n",+ "loaded_rag = RAG()\n",+ "loaded_rag.load(\"optimized_rag.json\")\n",+ "\n",+ "loaded_rag(question=\"cmd+tab does not work on hidden or minimized windows\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "\n",+ "## What's next?\n",+ "\n",+ "Improving from around 37% to approximately 60% on this task, in terms of `SemanticF1`, was pretty easy.\n",+ "\n",+ "But DSPy gives you paths to continue iterating on the quality of your system and we have barely scratched the surface.\n",+ "\n",+ "In general, you have the following tools:\n",+ "\n",+ "1. Explore better system architectures for your program, e.g. what if we ask the LM to generate search queries for the retriever? See this [notebook](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/intro.ipynb) or the [STORM pipeline](https://arxiv.org/abs/2402.14207) built in DSPy.\n",+ "2. Explore different [prompt optimizers](https://arxiv.org/abs/2406.11695) or [weight optimizers](https://arxiv.org/abs/2407.10930). See the **[Optimizers Docs](/building-blocks/6-optimizers)**.\n",+ "3. Scale inference time compute using DSPy Optimizers, e.g. this [notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/agents/multi_agent.ipynb).\n",+ "4. Cut cost by distilling to a smaller LM, via prompt or weight optimization, e.g. [this notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/nli/scone/scone.ipynb) or [this notebook](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/qa/hotpot/multihop_finetune.ipynb).\n",+ "\n",+ "How do you decide which ones to proceed with first?\n",+ "\n",+ "The first step is to look at your system outputs, which will allow you to identify the sources of lower performance if any. While doing all of this, make sure you continue to refine your metric, e.g. by optimizing against your judgments, and to collect more (or more realistic) data, e.g. from related domains or from putting a demo of your system in front of users.\n",+ "\n",+ "Learn more about the [development cycle](/building-blocks/solving_your_task) in DSPy."+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "py310_sept24_user",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.14"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/docs/mkdocs.yml b/docs/mkdocs.ymlindex 6f12323a1..b29ee1f0f 100644--- a/docs/mkdocs.yml+++ b/docs/mkdocs.yml@@ -3,16 +3,12 @@ site_name: DSPy Documentationrepo_url: https://github.com/stanfordnlp/dspyrepo_name: stanfordnlp/dspy-edit_uri: edit/main/docs/docs/+edit_uri: blob/main/docs/docs/docs_dir: 'docs/'nav:- - Home:- - intro.md- - Quick Start:- - Installation: quick-start/installation.md- - Quick Start: quick-start/getting-started-01.md- - Build your first pipeline: quick-start/getting-started-02.md- - Components:+ - Home: index.md+ # - Build your first pipeline: quick-start/getting-started-02.md+ - Learn DSPy:- Overview: building-blocks/solving_your_task.md- Language Models: building-blocks/1-language_models.md- Signatures: building-blocks/2-signatures.md@@ -28,7 +24,7 @@ nav:- Assertions: deep-dive/assertions.md- Retrieve: deep-dive/modules/retrieve.md- Modules Guide: deep-dive/modules/guide.md- - Metrics and Assertions:+ - Metrics:- Metrics: building-blocks/5-metrics.md- Assertions: building-blocks/7-assertions.md- Assertions II: deep-dive/assertions.md@@ -41,7 +37,7 @@ nav:- MIPROv2: deep-dive/optimizers/miprov2.md- Ensemble: deep-dive/optimizers/Ensemble.md- BootstrapFinetune: deep-dive/optimizers/BootstrapFinetune.md- - Retrieval Model Integrations:+ - Retrieval Clients:- Azure: deep-dive/retrieval_models_clients/Azure.md- ChromadbRM: deep-dive/retrieval_models_clients/ChromadbRM.md- ClarifaiRM: deep-dive/retrieval_models_clients/ClarifaiRM.md@@ -60,11 +56,13 @@ nav:- WeaviateRM: deep-dive/retrieval_models_clients/WeaviateRM.md- YouRM: deep-dive/retrieval_models_clients/YouRM.md- Tutorials:- - Retrieval Augmented Generation: tutorials/rag.md- - Multi-Hop QA: tutorials/simplified-baleen.md- - Summarization: tutorials/summarization.md- - Additional Resources: tutorials/other_tutorial.md- - Examples: tutorials/examples.md+ - Simple RAG: tutorials/rag/index.ipynb+ # - Tutorials:+ # - Retrieval Augmented Generation: tutorials/rag.md+ # - Multi-Hop QA: tutorials/simplified-baleen.md+ # - Summarization: tutorials/summarization.md+ # - Additional Resources: tutorials/other_tutorial.md+ # - Examples: tutorials/examples.md- '' : ''- '' : ''- '' : ''@@ -140,112 +138,9 @@ plugins:- mkdocs-jupyter- redirects:redirect_maps:- 'index.md': 'intro.md'- # Intro page redirects- 'docs/intro.md': 'intro.md'- 'docs/intro/index.md': 'intro.md'--- # Quick Start redirects- 'docs/quick-start/installation.md': 'quick-start/installation.md'- 'docs/quick-start/getting-started-01.md': 'quick-start/getting-started-01.md'- 'docs/quick-start/getting-started-02.md': 'quick-start/getting-started-02.md'-- # Building Blocks redirects- 'docs/building-blocks/solving_your_task.md': 'building-blocks/solving_your_task.md'- 'docs/building-blocks/1-language_models.md': 'building-blocks/1-language_models.md'- 'docs/building-blocks/2-signatures.md': 'building-blocks/2-signatures.md'- 'docs/building-blocks/3-modules.md': 'building-blocks/3-modules.md'- 'docs/building-blocks/4-data.md': 'building-blocks/4-data.md'- 'docs/building-blocks/5-metrics.md': 'building-blocks/5-metrics.md'- 'docs/building-blocks/6-optimizers.md': 'building-blocks/6-optimizers.md'- 'docs/building-blocks/7-assertions.md': 'building-blocks/7-assertions.md'- 'docs/building-blocks/8-typed_predictors.md': 'building-blocks/8-typed_predictors.md'-- # Additional Building Blocks redirects for non-numbered versions- 'docs/building-blocks/0_solving_your_task.md': 'building-blocks/solving_your_task.md'- 'docs/building-blocks/language_models.md': 'building-blocks/1-language_models.md'- 'docs/building-blocks/signatures.md': 'building-blocks/2-signatures.md'- 'docs/building-blocks/modules.md': 'building-blocks/3-modules.md'- 'docs/building-blocks/data.md': 'building-blocks/4-data.md'- 'docs/building-blocks/metrics.md': 'building-blocks/5-metrics.md'- 'docs/building-blocks/optimizers.md': 'building-blocks/6-optimizers.md'- 'docs/building-blocks/assertions.md': 'building-blocks/7-assertions.md'- 'docs/building-blocks/typed_predictors.md': 'building-blocks/8-typed_predictors.md'-- # Tutorials redirects- 'docs/tutorials/rag.md': 'tutorials/rag.md'- 'docs/tutorials/simplified-baleen.md': 'tutorials/simplified-baleen.md'- 'docs/tutorials/summarization.md': 'tutorials/summarization.md'- 'docs/tutorials/other_tutorial.md': 'tutorials/other_tutorial.md'- 'docs/tutorials/examples.md': 'tutorials/examples.md'-- # Deep Dive redirects- # Data Handling- 'docs/deep-dive/data-handling/examples.md': 'deep-dive/data-handling/examples.md'- 'docs/deep-dive/data-handling/built-in-datasets.md': 'deep-dive/data-handling/built-in-datasets.md'- 'docs/deep-dive/data-handling/loading-custom-data.md': 'deep-dive/data-handling/loading-custom-data.md'-- # Signatures- 'docs/deep-dive/signature/understanding-signatures.md': 'deep-dive/signature/understanding-signatures.md'-- # Modules- 'docs/deep-dive/modules/predict.md': 'deep-dive/modules/predict.md'- 'docs/deep-dive/modules/chain-of-thought.md': 'deep-dive/modules/chain-of-thought.md'- 'docs/deep-dive/modules/chain-of-thought-with-hint.md': 'deep-dive/modules/chain-of-thought-with-hint.md'- 'docs/deep-dive/modules/react.md': 'deep-dive/modules/react.md'- 'docs/deep-dive/modules/multi-chain-comparison.md': 'deep-dive/modules/multi-chain-comparison.md'- 'docs/deep-dive/modules/program-of-thought.md': 'deep-dive/modules/program-of-thought.md'- 'docs/deep-dive/modules/assertions.md': 'building-blocks/7-assertions.md'- 'deep-dive/modules/assertions.md': 'building-blocks/7-assertions.md'- 'docs/deep-dive/modules/retrieve.md': 'deep-dive/modules/retrieve.md'- 'docs/deep-dive/modules/guide.md': 'deep-dive/modules/guide.md'-- # Language Model Clients- 'docs/deep-dive/language_model_clients/lm_local_models/HFClientTGI.md': 'deep-dive/language_model_clients/lm_local_models/HFClientTGI.md'- 'docs/deep-dive/language_model_clients/lm_local_models/HFClientVLLM.md': 'deep-dive/language_model_clients/lm_local_models/HFClientVLLM.md'- 'docs/deep-dive/language_model_clients/lm_local_models/MLC.md': 'deep-dive/language_model_clients/lm_local_models/MLC.md'- 'docs/deep-dive/language_model_clients/lm_local_models/Ollama.md': 'deep-dive/language_model_clients/lm_local_models/Ollama.md'- 'docs/deep-dive/language_model_clients/lm_local_models/LlamaCpp.md': 'deep-dive/language_model_clients/lm_local_models/LlamaCpp.md'- 'docs/deep-dive/language_model_clients/lm_local_models/TensorRTLLM.md': 'deep-dive/language_model_clients/lm_local_models/TensorRTLLM.md'-- # Retrieval Model Clients- 'docs/deep-dive/retrieval_models_clients/Azure.md': 'deep-dive/retrieval_models_clients/Azure.md'- 'docs/deep-dive/retrieval_models_clients/ChromadbRM.md': 'deep-dive/retrieval_models_clients/ChromadbRM.md'- 'docs/deep-dive/retrieval_models_clients/ClarifaiRM.md': 'deep-dive/retrieval_models_clients/ClarifaiRM.md'- 'docs/deep-dive/retrieval_models_clients/ColBERTv2.md': 'deep-dive/retrieval_models_clients/ColBERTv2.md'- 'docs/deep-dive/retrieval_models_clients/custom-rm-client.md': 'deep-dive/retrieval_models_clients/custom-rm-client.md'- 'docs/deep-dive/retrieval_models_clients/DatabricksRM.md': 'deep-dive/retrieval_models_clients/DatabricksRM.md'- 'docs/deep-dive/retrieval_models_clients/FaissRM.md': 'deep-dive/retrieval_models_clients/FaissRM.md'- 'docs/deep-dive/retrieval_models_clients/LancedbRM.md': 'deep-dive/retrieval_models_clients/LancedbRM.md'- 'docs/deep-dive/retrieval_models_clients/MilvusRM.md': 'deep-dive/retrieval_models_clients/MilvusRM.md'- 'docs/deep-dive/retrieval_models_clients/MyScaleRM.md': 'deep-dive/retrieval_models_clients/MyScaleRM.md'- 'docs/deep-dive/retrieval_models_clients/Neo4jRM.md': 'deep-dive/retrieval_models_clients/Neo4jRM.md'- 'docs/deep-dive/retrieval_models_clients/QdrantRM.md': 'deep-dive/retrieval_models_clients/QdrantRM.md'- 'docs/deep-dive/retrieval_models_clients/RAGatouilleRM.md': 'deep-dive/retrieval_models_clients/RAGatouilleRM.md'- 'docs/deep-dive/retrieval_models_clients/SnowflakeRM.md': 'deep-dive/retrieval_models_clients/SnowflakeRM.md'- 'docs/deep-dive/retrieval_models_clients/WatsonDiscovery.md': 'deep-dive/retrieval_models_clients/WatsonDiscovery.md'- 'docs/deep-dive/retrieval_models_clients/WeaviateRM.md': 'deep-dive/retrieval_models_clients/WeaviateRM.md'- 'docs/deep-dive/retrieval_models_clients/YouRM.md': 'deep-dive/retrieval_models_clients/YouRM.md'-- # Assertions- 'docs/deep-dive/assertions.md': 'deep-dive/assertions.md'-- # Teleprompters- 'docs/deep-dive/optimizers/LabeledFewShot.md': 'deep-dive/optimizers/LabeledFewShot.md'- 'docs/deep-dive/optimizers/bootstrap-fewshot.md': 'deep-dive/optimizers/bootstrap-fewshot.md'- 'docs/deep-dive/optimizers/BootstrapFinetune.md': 'deep-dive/optimizers/BootstrapFinetune.md'- 'docs/deep-dive/optimizers/Ensemble.md': 'deep-dive/optimizers/Ensemble.md'- 'docs/deep-dive/optimizers/bfrs.md': 'deep-dive/optimizers/bfrs.md'- 'docs/deep-dive/optimizers/copro.md': 'deep-dive/optimizers/copro.md'- 'docs/deep-dive/optimizers/miprov2.md': 'deep-dive/optimizers/miprov2.md'-- # Additional pages redirects- 'docs/faqs.md': 'faqs.md'- 'docs/cheatsheet.md': 'cheatsheet.md'- 'docs/dspy-usecases.md': 'dspy-usecases.md'--+ # Redirect /intro/ to the main page+ 'intro/index.md': 'index.md'+ 'intro.md': 'index.md'diff --git a/dspy/evaluate/evaluate.py b/dspy/evaluate/evaluate.pyindex 7a585a54a..a8af70373 100644--- a/dspy/evaluate/evaluate.py+++ b/dspy/evaluate/evaluate.py@@ -346,24 +346,8 @@ def display_dataframe(df: pd.DataFrame):def configure_dataframe_for_ipython_notebook_display(df: pd.DataFrame) -> pd.DataFrame:"""Set various pandas display options for DataFrame in an IPython notebook environment."""- pd.options.display.max_colwidth = None- pd.set_option("display.max_colwidth", 20) # Adjust the number as needed- pd.set_option("display.width", 400) # Adjust-- # Return styled DataFrame- return df.style.set_table_styles(- [- {"selector": "th", "props": [("text-align", "left")]},- {"selector": "td", "props": [("text-align", "left")]},- ],- ).set_properties(- **{- "text-align": "left",- "white-space": "pre-wrap",- "word-wrap": "break-word",- "max-width": "400px",- },- )+ pd.options.display.max_colwidth = 70+ return dfdef is_in_ipython_notebook_environment():