Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add tutorial of DSPy cache (#8294)

Chen Qian91c4bb417cc0
diff --git a/docs/docs/tutorials/cache/index.md b/docs/docs/tutorials/cache/index.md
new file mode 100644
index 000000000..2989b24cc
--- /dev/null
+++ b/docs/docs/tutorials/cache/index.md
@@ -0,0 +1,185 @@
+# Use and Customize DSPy Cache
+
+In this tutorial, we will explore the design of DSPy's caching mechanism and demonstrate how to effectively use and customize it.
+
+## DSPy Cache Structure
+
+DSPy's caching system is architected in three distinct layers:
+
+1. **In-memory cache**: Implemented using `cachetools.LRUCache`, this layer provides fast access to frequently used data.
+2. **On-disk cache**: Leveraging `diskcache.FanoutCache`, this layer offers persistent storage for cached items.
+3. **Prompt cache (Server-side cache)**: This layer is managed by the LLM service provider (e.g., OpenAI, Anthropic).
+
+While DSPy does not directly control the server-side prompt cache, it offers users the flexibility to enable, disable, and customize the in-memory and on-disk caches to suit their specific requirements.
+
+## Using DSPy Cache
+
+By default, both in-memory and on-disk caching are automatically enabled in DSPy. No specific action is required to start using the cache. When a cache hit occurs, you will observe a significant reduction in the module call's execution time. Furthermore, if usage tracking is enabled, the usage metrics for a cached call will be `None`.
+
+Consider the following example:
+
+```python
+import dspy
+import os
+import time
+
+os.environ["OPENAI_API_KEY"] = "{your_openai_key}"
+
+dspy.settings.configure(lm=dspy.LM("openai/gpt-4o-mini"), track_usage=True)
+
+predict = dspy.Predict("question->answer")
+
+start = time.time()
+result1 = predict(question="Who is the GOAT of basketball?")
+print(f"Time elapse: {time.time() - start: 2f}\n\nTotal usage: {result1.get_lm_usage()}")
+
+start = time.time()
+result2 = predict(question="Who is the GOAT of basketball?")
+print(f"Time elapse: {time.time() - start: 2f}\n\nTotal usage: {result2.get_lm_usage()}")
+```
+
+A sample output looks like:
+
+```
+Time elapse: 4.384113
+Total usage: {'openai/gpt-4o-mini': {'completion_tokens': 97, 'prompt_tokens': 144, 'total_tokens': 241, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0, 'text_tokens': None}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0, 'text_tokens': None, 'image_tokens': None}}}
+
+Time elapse: 0.000529
+Total usage: {}
+```
+
+## Disabling/Enabling DSPy Cache
+
+There are scenarios where you might need to disable caching, either entirely or selectively for in-memory or on-disk caches. For instance:
+
+- You require different responses for identical LM requests.
+- You lack disk write permissions and need to disable the on-disk cache.
+- You have limited memory resources and wish to disable the in-memory cache.
+
+DSPy provides the `dspy.configure_cache()` utility function for this purpose. You can use the corresponding flags to control the enabled/disabled state of each cache type:
+
+```python
+dspy.configure_cache(
+ enable_disk_cache=False,
+ enable_memory_cache=False,
+)
+```
+
+In additions, you can manage the capacity of the in-memory and on-disk caches:
+
+```python
+dspy.configure_cache(
+ enable_disk_cache=True,
+ enable_memory_cache=True,
+ disk_size_limit_bytes=YOUR_DESIRED_VALUE,
+ memory_max_entries=YOUR_DESIRED_VALUE,
+)
+```
+
+Please note that `disk_size_limit_bytes` defines the maximum size in bytes for the on-disk cache, while `memory_max_entries` specifies the maximum number of entries for the in-memory cache.
+
+## Understanding and Customizing the Cache
+
+In specific situations, you might want to implement a custom cache, for example, to gain finer control over how cache keys are generated. By default, the cache key is derived from a hash of all request arguments sent to `litellm`, excluding credentials like `api_key`.
+
+To create a custom cache, you need to subclass `dspy.clients.Cache` and override the relevant methods:
+
+```python
+class CustomCache(dspy.clients.Cache):
+ def __init__(self, **kwargs):
+ {write your own constructor}
+
+ def cache_key(self, request: Dict[str, Any], ignored_args_for_cache_key: Optional[list[str]] = None) -> str:
+ {write your logic of computing cache key}
+
+ def get(self, request: Dict[str, Any], ignored_args_for_cache_key: Optional[list[str]] = None) -> Any:
+ {write your cache read logic}
+
+ def put(
+ self,
+ request: Dict[str, Any],
+ value: Any,
+ ignored_args_for_cache_key: Optional[list[str]] = None,
+ enable_memory_cache: bool = True,
+ ) -> None:
+ {write your cache write logic}
+```
+
+To ensure seamless integration with the rest of DSPy, it is recommended to implement your custom cache using the same method signatures as the base class, or at a minimum, include `**kwargs` in your method definitions to prevent runtime errors during cache read/write operations.
+
+Once your custom cache class is defined, you can instruct DSPy to use it:
+
+```python
+dspy.cache = CustomCache()
+```
+
+Let's illustrate this with a practical example. Suppose we want the cache key computation to depend solely on the request message content, ignoring other parameters like the specific LM being called. We can create a custom cache as follows:
+
+```python
+class CustomCache(dspy.clients.Cache):
+
+ def cache_key(self, request: Dict[str, Any], ignored_args_for_cache_key: Optional[list[str]] = None) -> str:
+ messages = request.get("messages", [])
+ return sha256(ujson.dumps(messages, sort_keys=True).encode()).hexdigest()
+
+dspy.cache = CustomCache(enable_disk_cache=True, enable_memory_cache=True, disk_cache_dir=dspy.clients.DISK_CACHE_DIR)
+```
+
+For comparison, consider executing the code below without the custom cache:
+
+```python
+import dspy
+import os
+import time
+
+os.environ["OPENAI_API_KEY"] = "{your_openai_key}"
+
+dspy.settings.configure(lm=dspy.LM("openai/gpt-4o-mini"))
+
+predict = dspy.Predict("question->answer")
+
+start = time.time()
+result1 = predict(question="Who is the GOAT of soccer?")
+print(f"Time elapse: {time.time() - start: 2f}")
+
+start = time.time()
+with dspy.context(lm=dspy.LM("openai/gpt-4.1-mini")):
+ result2 = predict(question="Who is the GOAT of soccer?")
+print(f"Time elapse: {time.time() - start: 2f}")
+```
+
+The time elapsed will indicate that the cache is not hit on the second call. However, when using the custom cache:
+
+```python
+import dspy
+import os
+import time
+from typing import Dict, Any, Optional
+import ujson
+from hashlib import sha256
+
+os.environ["OPENAI_API_KEY"] = "{your_openai_key}"
+
+dspy.settings.configure(lm=dspy.LM("openai/gpt-4o-mini"))
+
+class CustomCache(dspy.clients.Cache):
+
+ def cache_key(self, request: Dict[str, Any], ignored_args_for_cache_key: Optional[list[str]] = None) -> str:
+ messages = request.get("messages", [])
+ return sha256(ujson.dumps(messages, sort_keys=True).encode()).hexdigest()
+
+dspy.cache = CustomCache(enable_disk_cache=True, enable_memory_cache=True, disk_cache_dir=dspy.clients.DISK_CACHE_DIR)
+
+predict = dspy.Predict("question->answer")
+
+start = time.time()
+result1 = predict(question="Who is the GOAT of volleyball?")
+print(f"Time elapse: {time.time() - start: 2f}")
+
+start = time.time()
+with dspy.context(lm=dspy.LM("openai/gpt-4.1-mini")):
+ result2 = predict(question="Who is the GOAT of volleyball?")
+print(f"Time elapse: {time.time() - start: 2f}")
+```
+
+You will observe that the cache is hit on the second call, demonstrating the effect of the custom cache key logic.
\ No newline at end of file
diff --git a/docs/docs/tutorials/core_development/index.md b/docs/docs/tutorials/core_development/index.md
index 7b174bcd2..b60738d49 100644
--- a/docs/docs/tutorials/core_development/index.md
+++ b/docs/docs/tutorials/core_development/index.md
@@ -1,6 +1,7 @@
- [Use MCP in DSPy](/tutorials/mcp/)
- [Output Refinement](/tutorials/output_refinement/best-of-n-and-refine/)
- [Saving and Loading](/tutorials/saving/)
+- [Cache](/tutorials/cache/)
- [Deployment](/tutorials/deployment/)
- [Debugging & Observability](/tutorials/observability/)
- [Streaming](/tutorials/streaming/)
diff --git a/docs/docs/tutorials/index.md b/docs/docs/tutorials/index.md
index 72fbd5e31..97e48aceb 100644
--- a/docs/docs/tutorials/index.md
+++ b/docs/docs/tutorials/index.md
@@ -34,6 +34,7 @@ Welcome to DSPy tutorials! We've organized our tutorials into three main categor
- [Use MCP in DSPy](/tutorials/mcp/)
- [Output Refinement](/tutorials/output_refinement/best-of-n-and-refine/)
- [Saving and Loading](/tutorials/saving/)
+ - [Cache](/tutorials/cache/)
- [Deployment](/tutorials/deployment/)
- [Debugging & Observability](/tutorials/observability/)
- [Tracking DSPy Optimizers](/tutorials/optimizer_tracking/)
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index 8c3585cf2..c96d305b0 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -51,6 +51,7 @@ nav:
- Use MCP in DSPy: tutorials/mcp/index.md
- Output Refinement: tutorials/output_refinement/best-of-n-and-refine.md
- Saving and Loading: tutorials/saving/index.md
+ - Cache: tutorials/cache/index.md
- Deployment: tutorials/deployment/index.md
- Debugging & Observability: tutorials/observability/index.md
- Tracking DSPy Optimizers: tutorials/optimizer_tracking/index.md