Skip to content

Open source AI coding agent. Designed for large projects and real world tasks.

Merge branch 'main' into feature/evals

DaOfficialWizardb6f1fe8a7866
diff --cc Makefile
index 252631c5,e9eb25a8..efc12bb4
--- a/Makefile
+++ b/Makefile
@@@ -35,7 -35,6 +35,8 @@@ test: rende
#### Evals and Providers ####
++
+#! No cache is used to ensure that the latest changes are reflected in the eval
# TODO: Implement eval all
eval:
@cd test/evals/promptfoo-poc/$(filter-out $@,$(MAKECMDGOALS)) && promptfoo eval --no-cache
diff --cc test/evals/promptfoo-poc/build/build.prompt.txt
index e0a76064,9d4afc7b..79c71b48
--- a/test/evals/promptfoo-poc/build/build.prompt.txt
+++ b/test/evals/promptfoo-poc/build/build.prompt.txt
@@@ -83,17 -83,17 +83,19 @@@ If the 'hasChange' property is false, t
If *any* change has the 'entireFile' key in the 'old' property set to true, the corresponding 'new' key MUST be the entire updated file, and there MUST only be a single change in the 'changes' array.
Example change object:
- ---
++
+```json
{
- "summary": "Fix syntax error in loop body.",
- "old": {
- "startLineString": "pdx-5: for i := 0; i < 10; i++ { ",
- "endLineString": "pdx-7: }"
+ summary: "Fix syntax error in loop body.",
+ old: {
+ startLineString: "pdx-5: for i := 0; i < 10; i++ { ",
+ endLineString: "pdx-7: }"
},
- "new": "for i := 0; i < 10; i++ {\n execQuery()\n }\n }\n}"
+ new: "for i := 0; i < 10; i++ {\n execQuery()\n }\n }\n}"
}
- ---
+```
+
+
Apply changes intelligently **in order** to avoid syntax errors, breaking code, or removing code from the original file that should not be removed. Consider the reason behind the update and make sure the result is consistent with the intention of the plan.
Changes MUST be ordered based on their position in the original file. ALWAYS go from top to bottom IN ORDER when generating replacements. DO NOT EVER GENERATE AN OVERLAPPING CHANGE. If a change would fall within OR overlap a prior change in the list, SKIP that change and move on to the next one.
diff --cc test/evals/promptfoo-poc/fix/fix.prompt.txt
index b7a16eda,1ca11aa7..09560ce8
--- a/test/evals/promptfoo-poc/fix/fix.prompt.txt
+++ b/test/evals/promptfoo-poc/fix/fix.prompt.txt
@@@ -79,21 -79,20 +79,23 @@@ If *any* change has the 'entireFile' ke
You MUST ensure the line numbers for the 'old' property correctly remove *ALL* code that has problems and that the 'new' property correctly fixes *ALL* the problems present in the updated file. You MUST NOT miss any problems, fail to fix any problems, or introduce any new problems.
- Because you are implementing a fix, be more willing to make larger changes in order to fix all the problems. Smaller changes are more error-prone, and the fact that you are fixing a file means a line-number based change already failed. This likely means there was some tricky structural challenge in applying the changes with line numbers, so be prepared to make a larger change in order to avoid continuing to fail to fix the file.
-
- Example change object:
- ---
- {
- "summary": "Fix syntax error in loop body.",
- "old": {
- "startLineString": "pdx-5: for i := 0; i < 10; i++ { ",
- "endLineString": "pdx-7: }"
- },
- "new": "for i := 0; i < 10; i++ {\n execQuery()\n }\n }\n}"
- }
- ---
++
+Because you are implementing a fix, be more willing to make larger changes in order to fix all the problems. Smaller changes are more error-prone, and the fact that you are fixing a file means a line-number based change already failed. This likely means there was some tricky structural challenge in applying the changes with line numbers, so be prepared to make a larger change in order to avoid continuing to fail to fix the file.
+
+Example change object:
+
+```json
+{
+ "summary": "Fix syntax error in loop body.",
+ "old": {
+ "startLineString": "pdx-5: for i := 0; i < 10; i++ { ",
+ "endLineString": "pdx-7: }"
+ },
+ "new": "for i := 0; i < 10; i++ {\n execQuery()\n }\n }\n}"
+}
+```
+
+
You ABSOLUTELY MUST NOT generate overlapping changes. Group smaller changes together into larger changes where necessary to avoid overlap. Only generate multiple changes when you are ABSOLUTELY CERTAIN that they do not overlap--otherwise group them together into a single change. If changes are close to each other (within several lines), group them together into a single change. You MUST group changes together and make fewer, larger changes rather than many small changes, unless the changes are completely independent of each other and not close to each other in the file. You MUST NEVER generate changes that are adjacent or close to adjacent. Adjacent or closely adjacent changes MUST ALWAYS be grouped into a single larger change.
Furthermore, unless doing so would require a very large change because some changes are far apart in the file, it's ideal to call the 'listChangesWithLineNums' with just a SINGLE change.