Skip to content

armgit / burr

Public Imported from GitHub
0

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

Adds more notes on what people need to do regarding testing

Stefan Krawczyk8f7bbcf52deb
diff --git a/burr/cli/__main__.py b/burr/cli/__main__.py
index 6a1d5385..10a1670c 100644
--- a/burr/cli/__main__.py
+++ b/burr/cli/__main__.py
@@ -209,7 +209,13 @@ def test_{ACTION_NAME}(input_state, expected_state):
input_state = state.State(input_state)
expected_state = state.State(expected_state)
_, output_state = {ACTION_NAME}(input_state) # exercise the action
+ # TODO: choose appropriate way to evaluate the output
+ # e.g. exact match, fuzzy match, LLM grade, etc.
+ # this is exact match here on all values in state
assert output_state == expected_state
+ # e.g.
+ # assert 'some value' in output_state["response"]["content"]
+ # assert llm_evaluator(..., ...) == "Y"
"""
diff --git a/docs/examples/creating_tests.rst b/docs/examples/creating_tests.rst
index 08823967..c07fdadb 100644
--- a/docs/examples/creating_tests.rst
+++ b/docs/examples/creating_tests.rst
@@ -7,24 +7,47 @@ for an action involves turning a persister or tracker on and then pulling that s
for a test case. The following example demonstrates how to create a test case
using the `burr-test-case` command.
+Note: writing test cases for GenAI projects can be tricky. The same LLM API calls
+can result in different outputs. This means that 'exact' equality tests may not work,
+and you'll need to resort to more fuzzy tests like checking for the presence of certain
+words or phrases, or using LLMs to grade the output, etc. We aren't opinionated on how you
+do this, but in any case, you'll need to write a test case to exercise things, and this
+is what we're showing you how to do here.
+
Test Case Creation Example
--------------------------
-Prerequisite:
+Steps:
1. You have built some part of your Burr application.
2. You have found some state you want to test / iterate on.
3. Note the project name, partition key, app id, and sequence id for the state you want to test.
+ See orange lines indicating what to pull from the UI in the image below.
+
+ .. image:: test-case-creation-burr.png
+ :alt: What you need to pull from the UI for running the command.
+ :align: center
+
4. Run the following command & cut and paste the test to a test file. This will create a `pytest` test.
+5. Adjust the test as needed, e.g. what you're validating or asserting and how.
+
+.. code-block:: bash
-```bash
-%%sh
-burr-test-case create \
- --project-name "SOME_NAME" \
- --partition-key "SOME_KEY" \
- --app-id "SOME_ID" \
- --sequence-id 0 \
- --target-file-name /tmp/test-case.json
-```
+ burr-test-case create \
+ --project-name "SOME_NAME" \
+ --partition-key "SOME_KEY" \
+ --app-id "SOME_ID" \
+ --sequence-id 0 \
+ --target-file-name /tmp/test-case.json
See `github repository example <https://github.com/DAGWorks-Inc/burr/tree/main/examples/test-case-creation>`_
for an example.
+
+
+Future Work
+-----------
+We see many more improvements here:
+
+1. Annotating data in the UI to make it easier to pull out.
+2. Automatically suggesting tests cases for you to add.
+3. Data export / integration with evaluation tools.
+4. etc. Please let us know what you need!
diff --git a/docs/examples/test-case-creation-burr.png b/docs/examples/test-case-creation-burr.png
new file mode 100644
index 00000000..4fe992f7
Binary files /dev/null and b/docs/examples/test-case-creation-burr.png differ
diff --git a/examples/test-case-creation/README.md b/examples/test-case-creation/README.md
index 0063cf97..d430f6da 100644
--- a/examples/test-case-creation/README.md
+++ b/examples/test-case-creation/README.md
@@ -4,6 +4,14 @@ This example shows you how you can easily create (integration) test cases for Bu
With Burr it is easy to foster a test-driven development mindset when developing actions, which are the building blocks of your Burr
application.
+Note: writing test cases for GenAI projects can be tricky. The same LLM API calls
+can result in different outputs. This means that 'exact' equality tests may not work,
+and you'll need to resort to more fuzzy tests like checking for the presence of certain
+words or phrases, or using LLMs to grade the output, etc. We aren't opinionated on how you
+do this, but in any case, you'll need to write a test case to exercise things, and this
+is what we're showing you how to do here.
+
+
## Motivation
With LLMs non-deterministic behavior can impact your application's behavior. It is important to create test cases to
@@ -16,10 +24,11 @@ real-world behavior of your application -- Burr can make this process easier.
1. Build your project with Burr.
2. Run your Burr application with tracking/persistence.
3. Find a trace that you want to create a test case for.
-4. Note the partition_key, app_id, and sequence_id of the trace.
+4. Note the partition_key, app_id, and sequence_id of the trace. You can find these in the UI.
5. Create a test case using the `burr-test-case create` command passing the project name, partition_key, app_id, and sequence_id as arguments.
-6. Iterate on the test case until it is robust and representative of the real-world behavior of your application.
-7. Profit.
+6. Choose what you're going to test and how. E.g. exact match, similarity, LLM Grader, etc.
+7. Iterate on the test case until it is robust and representative of the real-world behavior of your application.
+8. Profit.
For step (5) above, this is the corresponding command:
```bash
@@ -36,3 +45,12 @@ See [the notebook](./notebook.ipynb) for example usage.
## Testing Actions
In `test_application.py` you'll find examples tests for a simple action
that is found in `application.py`.
+
+
+## Future Work
+We see many more improvements here:
+
+1. Annotating data in the UI to make it easier to pull out.
+2. Automatically suggesting tests cases for you to add.
+3. Data export / integration with evaluation tools.
+4. etc. Please let us know what you need!
diff --git a/examples/test-case-creation/test_application.py b/examples/test-case-creation/test_application.py
index d875e333..d9a3b6b7 100644
--- a/examples/test-case-creation/test_application.py
+++ b/examples/test-case-creation/test_application.py
@@ -2,8 +2,9 @@
This is an example test module showing how you can test an action in your application.
1. There is a very straightforward unit testing style approach.
-2. There is a more complex approach that uses pytest_generate_tests to dynamically generate test cases.
-To use this approach you need to do `from burr.testing import pytest_generate_tests # noqa: F401 `
+2. There is a more sophisticated approach that uses pytest_generate_tests to dynamically generate test cases.
+To use this approach you need to do `from burr.testing import pytest_generate_tests # noqa: F401`.
+
"""
import pytest
from application import prompt_for_more
@@ -36,7 +37,9 @@ def test_prompt_for_more():
}
)
result, output_state = prompt_for_more(input_state)
- # evaluate
+ # evaluate the output
+ # TODO: choose appropriate way to evaluate the output based on your needs.
+ # e.g. exact match, fuzzy match, LLM grade, etc.
assert output_state == expected_state
assert result == {
"response": {
@@ -53,4 +56,12 @@ def test_prompt_for_more_from_file(input_state, expected_state):
input_state = state.State(input_state)
expected_state = state.State(expected_state)
_, output_state = prompt_for_more(input_state) # exercising the action
+ # TODO: choose appropriate way to evaluate the output
+ # e.g. exact match, fuzzy match, LLM grade, etc.
+ # this is exact match here on all values in state
assert output_state == expected_state
+ # for output that varies, you can do something like this
+ # assert 'some value' in output_state["response"]["content"]
+ # or, have an LLM Grade things -- you need to create the llm_evaluator function:
+ # assert llm_evaluator("are these two equivalent responses. Respond with Y for yes, N for no",
+ # output_state["response"]["content"], expected_state["response"]["content"]) == "Y"