Skip to content

SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]

Feat: review on submit prototype

Kilian Lieret5f935edeb058
diff --git a/config/function_calling_replace_late_repro.yaml b/config/function_calling_replace_late_repro.yaml
index 92c3b412..c7e6673b 100644
--- a/config/function_calling_replace_late_repro.yaml
+++ b/config/function_calling_replace_late_repro.yaml
@@ -108,15 +108,47 @@ agent:
- trajectories/demonstrations/function_calling_simple.traj
put_demos_in_history: true
tools:
+ submit_command: "submit -f"
env_variables:
WINDOW: 100
OVERLAP: 2
+ SUBMIT_REVIEW_MESSAGE: |-
+ Thank you for your work on this issue.
+
+ As a refresher, this is the issue we're solving:
+
+ <issue_text>
+ {{problem_statement}}
+ </issue_text>
+
+ Here are your changes:
+
+ <diff>
+ {{diff}}
+ </diff>
+
+ Before we submit your changes, here are some final instructions to review:
+
+ 1. Does the diff <diff> look correct or could it introduce any problems?
+ 2. Does the diff <diff> solve the problem explained in the issue <issue_text>?
+ 3. Were you able to reproduce and then fix the bug? If you had trouble with the reproduction, are there any other
+ ways you could double-check that the bug is fixed?
+ 4. Is there any existing functionality that could be broken by your changes?
+ 5. Did you test or think of any edge cases?
+
+ Please write a DETAILED response to the above questions.
+ You can also run additional checks and tests if you want.
+ If you find anything to be problematic, please fix it.
+
+ Once you're done, run the submit command to submit your changes.
+
+ <IMPORTANT>Please be verbose and think things through step-by-step.</IMPORTANT>
bundles:
- path: tools/registry
- path: tools/defaults
- path: tools/search
- path: tools/edit_replace
- - path: tools/submit
+ - path: tools/review_on_submit
enable_bash_tool: true
parse_function:
type: function_calling
diff --git a/sweagent/agent/agents.py b/sweagent/agent/agents.py
index b9dcdc66..9e3715e6 100644
--- a/sweagent/agent/agents.py
+++ b/sweagent/agent/agents.py
@@ -284,6 +284,7 @@ class Agent:
"""
self._problem_statement = problem_statement
self._env = env
+ self._env.set_env_variables({"PROBLEM_STATEMENT": self._problem_statement.get_problem_statement()})
# Save/reset some attributes
self.info = AgentInfo()
diff --git a/tools/review_on_submit/README.md b/tools/review_on_submit/README.md
new file mode 100644
index 00000000..131667da
--- /dev/null
+++ b/tools/review_on_submit/README.md
@@ -0,0 +1,6 @@
+# Review on submit.
+
+Provides an alternative for `submit` that does not immediately submit, but asks the
+agent to perform additional reviewing steps.
+
+Only `submit -f` will trigger the real submit.
\ No newline at end of file
diff --git a/tools/review_on_submit/bin/submit b/tools/review_on_submit/bin/submit
new file mode 100644
index 00000000..b6865aff
--- /dev/null
+++ b/tools/review_on_submit/bin/submit
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+
+import argparse
+import subprocess
+import sys
+import os
+
+from registry import registry
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Submit changes for review")
+ parser.add_argument("-f", "--force", action="store_true", help="Force submit without review")
+ args = parser.parse_args()
+
+ repo_root = registry.get("ROOT", os.getenv("ROOT"))
+ assert repo_root
+
+ git_process = subprocess.run(
+ "git add -A && git diff --cached > /root/model.patch",
+ shell=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ cwd=repo_root,
+ )
+
+ with open("/root/model.patch") as f:
+ patch = f.read()
+
+ with open("model.patch", "w") as f:
+ f.write(git_process.stdout.decode("utf-8"))
+
+ if not args.force and not registry.get("SUBMIT_TRIGGERED_BEFORE"):
+ message = registry.get("SUBMIT_REVIEW_MESSAGE", "")
+ message = message.replace("{{diff}}", patch)
+ message = message.replace("{{problem_statement}}", "")
+ registry["SUBMIT_TRIGGERED_BEFORE"] = True
+ print(message)
+ sys.exit(0)
+
+ print("<<SUBMISSION||")
+ print(patch)
+ print("||SUBMISSION>>")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/review_on_submit/config.yaml b/tools/review_on_submit/config.yaml
new file mode 100644
index 00000000..082b3d65
--- /dev/null
+++ b/tools/review_on_submit/config.yaml
@@ -0,0 +1,6 @@
+tools:
+ submit:
+ signature: "submit"
+ docstring: "submits the current file"
+ # Do not actually show the -f argument to the model, only
+ # use it from the agent for submissiona after error
diff --git a/tools/review_on_submit/install.sh b/tools/review_on_submit/install.sh
new file mode 100644
index 00000000..e69de29b