Open source AI coding agent. Designed for large projects and real world tasks.
Go aiai-agentsai-developer-toolsai-toolsclicommand-linedeveloper-toolsgitgolanggpt-4llmopenaipolyglot-programmingterminalterminal-basedterminal-ui
feat: Add gen-eval target to Makefile for creating promptfoo eval directories
ZanzyTHEbar616e5749ed82
diff --git a/Makefile b/Makefileindex 174fbf8a..11be13b4 100644--- a/Makefile+++ b/Makefile@@ -33,8 +33,8 @@ clean:test: render@$(GOTEST) -v ./...-gen-test:- @$(GOCMD) run app/scripts/cmd/gen/gen.go $(filter-out $@,$(MAKECMDGOALS))+gen-eval:+ @$(GOCMD) run app/scripts/cmd/gen/gen.go test/evals/promptfoo-poc/$(filter-out $@,$(MAKECMDGOALS))gen-provider:@$(GOCMD) run app/scripts/cmd/provider/gen_provider.go@@ -50,7 +50,7 @@ default: buildhelp:@echo "Usage:"@echo " make dev - to run the development scripts"- @echo " make gen-test <directory_path> - to create a new promptfoo eval directory structure"+ @echo " make gen-eval <directory_name> - to create a new promptfoo eval directory structure"@echo " make gen-provider - to create a new promptfoo provider file from the promptfoo diretory structure"@echo " make clean - to remove generated files and directories"@echo " make help - to display this help message"diff --git a/test/evals/promptfoo-poc/README.md b/test/evals/promptfoo-poc/README.mdnew file mode 100644index 00000000..f08fd60b--- /dev/null+++ b/test/evals/promptfoo-poc/README.md@@ -0,0 +1,51 @@+# Test Driven Development of Prompts++This directory is dedicated to the systematic development of prompts for the plandex project. The prompts are developed in a test-driven manner, where the prompt is first written in a markdown file, and then the prompt is tested by running the prompt through the various evaluations. The output of the prompt is then graded and A/B tested for various metrics (see [metrics](#metrics)). The prompt is then iteratively improved until it meets the desired metrics.++We have decided to write the majority of the evals using the [promptfoo]() framework, as it is robust and contains customizations with a clear ease of setup.++## Usage++Usage will be broken down into [run](#run-evals) and [create](#create-evals) sections:++### Setup++To run or create evals, you will need to have the following installed:++- [Go](https://golang.org/)+- [Promptfoo]()++### Run Evals++To run the evaluations, you can cd into the relevant directory and use the following command:++```bash+make evals+```++Or, you can run all the evaluations by running the following command:++```bash+make evals all+```++### Create Evals++To create the evaluations, you can use our `gen-*` commands. The `gen-*` commands are designed to setup the eval environment and will create the evaluations directory structure in the `evals/promptfoo` directory. You have access to the following commands:++```bash+make gen-eval # Generates the evaluation directory structure+make gen-provider # Generates a provider file based on the directory config files+```++> ![IMPORTANT]\+> Make sure to run the `gen-eval` command before running the `gen-provider` command.+> You need to have the config files filled out with your details before running the `gen-provider` command.+> Depending on the provider you use, you will need to setup an environment variable with the provider's API key.++## Metrics++The metrics we are currently tracking are:++> ![NOTE]\+> COMING SOON