DSPy: The framework for programming—not prompting—language models
Python
Improve assertion document
Shangyintbe07293afeb9
diff --git a/docs/assertions.md b/docs/assertions.mdindex 99da7de07..a1601227d 100644--- a/docs/assertions.md+++ b/docs/assertions.md@@ -13,7 +13,7 @@ We introduce two primary constructs within DSPy Assertions:- **Parameters**:- `constraint (bool)`: Outcome of Python-defined boolean validation check.- `msg (Optional[str])`: User-defined error message providing feedback or correction guidance.- - `backtrack (Optional[module])`: Specifies target module for retry attempts upon constraint failure.+ - `backtrack (Optional[module])`: Specifies target module for retry attempts upon constraint failure. The default backtracking module is the last module before the assertion.- **Behavior**: Initiates retry upon failure, dynamically adjusting the pipeline's execution. If failures persist, it halts execution and raises a `dspy.AssertionError`.- **`dspy.Suggest`**:@@ -33,8 +33,8 @@ If the error continues past the `max_backtracking_attempts`, then `dspy.Assert`- **dspy.Suggest vs. dspy.Assert**: `dspy.Suggest` on the other hand offers a softer approach. It maintains the same retry backtracking as `dspy.Assert` but instead serves as a gentle nudger. If the model outputs cannot pass the model constraints after the `max_backtracking_attempts`, `dspy.Suggest` will log the persistent failure and continue execution of the program on the rest of the data. This ensures the LM pipeline works in a "best-effort" manner without halting execution.-- **`dspy.Suggestions`** are best utilized as "helpers" during the evaluation phase, offering guidance and potential corrections without halting the pipeline.-- **`dspy.Assertions`** are recommended during the development stage as "checkers" to ensure the LM behaves as expected, providing a robust mechanism for identifying and addressing errors early in the development cycle.+- **`dspy.Suggest`** are best utilized as "helpers" during the evaluation phase, offering guidance and potential corrections without halting the pipeline.+- **`dspy.Assert`** are recommended during the development stage as "checkers" to ensure the LM behaves as expected, providing a robust mechanism for identifying and addressing errors early in the development cycle.## Use Case: Including Assertions in DSPy Programs@@ -155,10 +155,15 @@ Now calling programs with DSPy Assertions requires one last step, and that is tr```pythonfrom dspy.primitives.assertions import assert_transform_module, backtrack_handler-baleen_with_assertions = assert_transform_module(SimplifiedBaleenAssertions().map_named_predictors(Retry), backtrack_handler)+baleen_with_assertions = assert_transform_module(SimplifiedBaleenAssertions(), backtrack_handler)++# backtrack_handler is parameterized over a few settings for the backtracking mechanism+# To change the number of max retry attempts, you can do+baleen_with_assertions_retry_once = assert_transform_module(SimplifiedBaleenAssertions(),+ functools.partial(backtrack_handler, max_backtracks=1))```-Alternatively, you can also directly call `activate_assertions` on the program with `dspy.Assert/Suggest` statements:+Alternatively, you can also directly call `activate_assertions` on the program with `dspy.Assert/Suggest` statements using the default backtracking mechanism (`max_backtracks=2`):```pythonbaleen_with_assertions = SimplifiedBaleenAssertions().activate_assertions()@@ -234,7 +239,7 @@ Query: "actress Ophelia RSC Hamlet" + "actress The Shore"DSPy Assertions work with optimizations that DSPy offers, particularly with `BootstrapFewShotWithRandomSearch`, including the following settings:- Compilation with Assertions- This includes assertion-driven example bootstrapping and counterexample bootstrapping during compilation. The teacher model for bootstrapping few-shot demonstrations can make use of DSPy Assertions to offer robust bootstrapped examples for the student model to learn from during inference.+ This includes assertion-driven example bootstrapping and counterexample bootstrapping during compilation. The teacher model for bootstrapping few-shot demonstrations can make use of DSPy Assertions to offer robust bootstrapped examples for the student model to learn from during inference. In this setting, the student model does not perform assertion aware optimizations (backtracking and retry) during inference.- Compilation + Inference with Assertions-This includes assertion-driven optimizations in both compilation and inference. Now the teacher model offers assertion-driven examples but the student can further optimize with assertions of its own during inference time.```python