Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.
Python aiburrchatbot-frameworkdagsgenerative-aigraphshacktoberfestllmopsllmsmlopspersistent-data-structurestate-machinestate-managementvisibility
Adds more notes on what people need to do regarding testing
Stefan Krawczyk8f7bbcf52deb
diff --git a/burr/cli/__main__.py b/burr/cli/__main__.pyindex 6a1d5385..10a1670c 100644--- a/burr/cli/__main__.py+++ b/burr/cli/__main__.py@@ -209,7 +209,13 @@ def test_{ACTION_NAME}(input_state, expected_state):input_state = state.State(input_state)expected_state = state.State(expected_state)_, output_state = {ACTION_NAME}(input_state) # exercise the action+ # TODO: choose appropriate way to evaluate the output+ # e.g. exact match, fuzzy match, LLM grade, etc.+ # this is exact match here on all values in stateassert output_state == expected_state+ # e.g.+ # assert 'some value' in output_state["response"]["content"]+ # assert llm_evaluator(..., ...) == "Y""""diff --git a/docs/examples/creating_tests.rst b/docs/examples/creating_tests.rstindex 08823967..c07fdadb 100644--- a/docs/examples/creating_tests.rst+++ b/docs/examples/creating_tests.rst@@ -7,24 +7,47 @@ for an action involves turning a persister or tracker on and then pulling that sfor a test case. The following example demonstrates how to create a test caseusing the `burr-test-case` command.+Note: writing test cases for GenAI projects can be tricky. The same LLM API calls+can result in different outputs. This means that 'exact' equality tests may not work,+and you'll need to resort to more fuzzy tests like checking for the presence of certain+words or phrases, or using LLMs to grade the output, etc. We aren't opinionated on how you+do this, but in any case, you'll need to write a test case to exercise things, and this+is what we're showing you how to do here.+Test Case Creation Example---------------------------Prerequisite:+Steps:1. You have built some part of your Burr application.2. You have found some state you want to test / iterate on.3. Note the project name, partition key, app id, and sequence id for the state you want to test.+ See orange lines indicating what to pull from the UI in the image below.++ .. image:: test-case-creation-burr.png+ :alt: What you need to pull from the UI for running the command.+ :align: center+4. Run the following command & cut and paste the test to a test file. This will create a `pytest` test.+5. Adjust the test as needed, e.g. what you're validating or asserting and how.++.. code-block:: bash-```bash-%%sh-burr-test-case create \- --project-name "SOME_NAME" \- --partition-key "SOME_KEY" \- --app-id "SOME_ID" \- --sequence-id 0 \- --target-file-name /tmp/test-case.json-```+ burr-test-case create \+ --project-name "SOME_NAME" \+ --partition-key "SOME_KEY" \+ --app-id "SOME_ID" \+ --sequence-id 0 \+ --target-file-name /tmp/test-case.jsonSee `github repository example <https://github.com/DAGWorks-Inc/burr/tree/main/examples/test-case-creation>`_for an example.+++Future Work+-----------+We see many more improvements here:++1. Annotating data in the UI to make it easier to pull out.+2. Automatically suggesting tests cases for you to add.+3. Data export / integration with evaluation tools.+4. etc. Please let us know what you need!diff --git a/docs/examples/test-case-creation-burr.png b/docs/examples/test-case-creation-burr.pngnew file mode 100644index 00000000..4fe992f7Binary files /dev/null and b/docs/examples/test-case-creation-burr.png differdiff --git a/examples/test-case-creation/README.md b/examples/test-case-creation/README.mdindex 0063cf97..d430f6da 100644--- a/examples/test-case-creation/README.md+++ b/examples/test-case-creation/README.md@@ -4,6 +4,14 @@ This example shows you how you can easily create (integration) test cases for BuWith Burr it is easy to foster a test-driven development mindset when developing actions, which are the building blocks of your Burrapplication.+Note: writing test cases for GenAI projects can be tricky. The same LLM API calls+can result in different outputs. This means that 'exact' equality tests may not work,+and you'll need to resort to more fuzzy tests like checking for the presence of certain+words or phrases, or using LLMs to grade the output, etc. We aren't opinionated on how you+do this, but in any case, you'll need to write a test case to exercise things, and this+is what we're showing you how to do here.++## MotivationWith LLMs non-deterministic behavior can impact your application's behavior. It is important to create test cases to@@ -16,10 +24,11 @@ real-world behavior of your application -- Burr can make this process easier.1. Build your project with Burr.2. Run your Burr application with tracking/persistence.3. Find a trace that you want to create a test case for.-4. Note the partition_key, app_id, and sequence_id of the trace.+4. Note the partition_key, app_id, and sequence_id of the trace. You can find these in the UI.5. Create a test case using the `burr-test-case create` command passing the project name, partition_key, app_id, and sequence_id as arguments.-6. Iterate on the test case until it is robust and representative of the real-world behavior of your application.-7. Profit.+6. Choose what you're going to test and how. E.g. exact match, similarity, LLM Grader, etc.+7. Iterate on the test case until it is robust and representative of the real-world behavior of your application.+8. Profit.For step (5) above, this is the corresponding command:```bash@@ -36,3 +45,12 @@ See [the notebook](./notebook.ipynb) for example usage.## Testing ActionsIn `test_application.py` you'll find examples tests for a simple actionthat is found in `application.py`.+++## Future Work+We see many more improvements here:++1. Annotating data in the UI to make it easier to pull out.+2. Automatically suggesting tests cases for you to add.+3. Data export / integration with evaluation tools.+4. etc. Please let us know what you need!diff --git a/examples/test-case-creation/test_application.py b/examples/test-case-creation/test_application.pyindex d875e333..d9a3b6b7 100644--- a/examples/test-case-creation/test_application.py+++ b/examples/test-case-creation/test_application.py@@ -2,8 +2,9 @@This is an example test module showing how you can test an action in your application.1. There is a very straightforward unit testing style approach.-2. There is a more complex approach that uses pytest_generate_tests to dynamically generate test cases.-To use this approach you need to do `from burr.testing import pytest_generate_tests # noqa: F401 `+2. There is a more sophisticated approach that uses pytest_generate_tests to dynamically generate test cases.+To use this approach you need to do `from burr.testing import pytest_generate_tests # noqa: F401`.+"""import pytestfrom application import prompt_for_more@@ -36,7 +37,9 @@ def test_prompt_for_more():})result, output_state = prompt_for_more(input_state)- # evaluate+ # evaluate the output+ # TODO: choose appropriate way to evaluate the output based on your needs.+ # e.g. exact match, fuzzy match, LLM grade, etc.assert output_state == expected_stateassert result == {"response": {@@ -53,4 +56,12 @@ def test_prompt_for_more_from_file(input_state, expected_state):input_state = state.State(input_state)expected_state = state.State(expected_state)_, output_state = prompt_for_more(input_state) # exercising the action+ # TODO: choose appropriate way to evaluate the output+ # e.g. exact match, fuzzy match, LLM grade, etc.+ # this is exact match here on all values in stateassert output_state == expected_state+ # for output that varies, you can do something like this+ # assert 'some value' in output_state["response"]["content"]+ # or, have an LLM Grade things -- you need to create the llm_evaluator function:+ # assert llm_evaluator("are these two equivalent responses. Respond with Y for yes, N for no",+ # output_state["response"]["content"], expected_state["response"]["content"]) == "Y"