Skip to content

Open source AI coding agent. Designed for large projects and real world tasks.

feat: Add gen-eval target to Makefile for creating promptfoo eval directories

ZanzyTHEbar616e5749ed82
diff --git a/Makefile b/Makefile
index 174fbf8a..11be13b4 100644
--- a/Makefile
+++ b/Makefile
@@ -33,8 +33,8 @@ clean:
test: render
@$(GOTEST) -v ./...
-gen-test:
- @$(GOCMD) run app/scripts/cmd/gen/gen.go $(filter-out $@,$(MAKECMDGOALS))
+gen-eval:
+ @$(GOCMD) run app/scripts/cmd/gen/gen.go test/evals/promptfoo-poc/$(filter-out $@,$(MAKECMDGOALS))
gen-provider:
@$(GOCMD) run app/scripts/cmd/provider/gen_provider.go
@@ -50,7 +50,7 @@ default: build
help:
@echo "Usage:"
@echo " make dev - to run the development scripts"
- @echo " make gen-test <directory_path> - to create a new promptfoo eval directory structure"
+ @echo " make gen-eval <directory_name> - to create a new promptfoo eval directory structure"
@echo " make gen-provider - to create a new promptfoo provider file from the promptfoo diretory structure"
@echo " make clean - to remove generated files and directories"
@echo " make help - to display this help message"
diff --git a/test/evals/promptfoo-poc/README.md b/test/evals/promptfoo-poc/README.md
new file mode 100644
index 00000000..f08fd60b
--- /dev/null
+++ b/test/evals/promptfoo-poc/README.md
@@ -0,0 +1,51 @@
+# Test Driven Development of Prompts
+
+This directory is dedicated to the systematic development of prompts for the plandex project. The prompts are developed in a test-driven manner, where the prompt is first written in a markdown file, and then the prompt is tested by running the prompt through the various evaluations. The output of the prompt is then graded and A/B tested for various metrics (see [metrics](#metrics)). The prompt is then iteratively improved until it meets the desired metrics.
+
+We have decided to write the majority of the evals using the [promptfoo]() framework, as it is robust and contains customizations with a clear ease of setup.
+
+## Usage
+
+Usage will be broken down into [run](#run-evals) and [create](#create-evals) sections:
+
+### Setup
+
+To run or create evals, you will need to have the following installed:
+
+- [Go](https://golang.org/)
+- [Promptfoo]()
+
+### Run Evals
+
+To run the evaluations, you can cd into the relevant directory and use the following command:
+
+```bash
+make evals
+```
+
+Or, you can run all the evaluations by running the following command:
+
+```bash
+make evals all
+```
+
+### Create Evals
+
+To create the evaluations, you can use our `gen-*` commands. The `gen-*` commands are designed to setup the eval environment and will create the evaluations directory structure in the `evals/promptfoo` directory. You have access to the following commands:
+
+```bash
+make gen-eval # Generates the evaluation directory structure
+make gen-provider # Generates a provider file based on the directory config files
+```
+
+> ![IMPORTANT]\
+> Make sure to run the `gen-eval` command before running the `gen-provider` command.
+> You need to have the config files filled out with your details before running the `gen-provider` command.
+> Depending on the provider you use, you will need to setup an environment variable with the provider's API key.
+
+## Metrics
+
+The metrics we are currently tracking are:
+
+> ![NOTE]\
+> COMING SOON