SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]
CHANGE: Rename configs (#1144)
Kilian Lieretf6af95e7bb7b
diff --git a/config/README.md b/config/README.mdindex 4c51124b..82914b56 100644--- a/config/README.md+++ b/config/README.md@@ -1,3 +1,9 @@+* Default config: `anthropic_filemap.yaml`+* `swebench_submissions`: Configs that were used for swebench submissions+* `sweagent_0_7`: Configs from SWE-agent 0.7, similar to the one used in the paper+* `exotic`: Various specific configurations that might be more of niche interest+* `human`: Demo/debug configs that have the human type commands and run without a LM+🔗 For more information on config files, visit [our documentation website][docs].You can also find the corresponding markdown files in the [`docs/` folder][source].diff --git a/config/250212_sweagent_heavy_sbl.yaml b/config/benchmarks/250212_sweagent_heavy_sbl.yamlsimilarity index 100%rename from config/250212_sweagent_heavy_sbl.yamlrename to config/benchmarks/250212_sweagent_heavy_sbl.yamldiff --git a/config/250225_anthropic_filemap_simple_review.yaml b/config/benchmarks/250225_anthropic_filemap_simple_review.yamlsimilarity index 100%rename from config/250225_anthropic_filemap_simple_review.yamlrename to config/benchmarks/250225_anthropic_filemap_simple_review.yamldiff --git a/config/anthropic_filemap_ml.yaml b/config/benchmarks/anthropic_filemap_multilingual.yamlsimilarity index 97%rename from config/anthropic_filemap_ml.yamlrename to config/benchmarks/anthropic_filemap_multilingual.yamlindex 348face7..afcf2fab 100644--- a/config/anthropic_filemap_ml.yaml+++ b/config/benchmarks/anthropic_filemap_multilingual.yaml@@ -1,4 +1,4 @@-# This template is heavily inspired by anthropic and openhands. It is almost+# This template is heavily inspired by anthropic, but you can use it with any LM. It is almost# identical to anthropic_filemap.yaml, but it removes python-specific language# and adds the multilingual_setup tool to support evaluation on the Multilingual dataset.agent:diff --git a/config/coding_challenge.yaml b/config/coding_challenge.yamlindex a7925bbe..9bddc180 100644--- a/config/coding_challenge.yaml+++ b/config/coding_challenge.yaml@@ -1,4 +1,5 @@-#This is the template you should use when using SWE-agent to solve a coding challenge (i.e. LeetCode).+# This is the template you should use when using SWE-agent to solve a coding challenge (i.e. LeetCode).+# It also shows how to repurpose the agent to do tasks different from software engineering.agent:templates:system_template: |-diff --git a/config/default.yaml b/config/default.yamlindex b13ec45e..dd6069f3 100644--- a/config/default.yaml+++ b/config/default.yaml@@ -1,126 +1,62 @@+# Formerly called: anthropic_filemap.yaml+# This template is heavily inspired by anthropic's computer use demo, but you can use+# it with any LM.agent:templates:system_template: |-- SETTING: You are an autonomous programmer, and you're working directly in the command line with a special interface.-- The special interface consists of a file editor that shows you {{WINDOW}} lines of a file at a time.- In addition to typical bash commands, you can also use specific commands to help you navigate and edit files.- To call a command, you need to invoke it with a function call/tool call.-- Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION.-- For example, if you are looking at this file:-- def fct():- print("Hello world")-- and you want to edit the file to read:-- def fct():- print("Hello")- print("world")-- you search string should be `Hello world` and your replace string should be `"Hello"\n print("world")`- (note the extra spaces before the print statement!).-- You could also get the same result by search for ` print("Hello world")` and replace with ` print("Hello")\n print("world")`.-- RESPONSE FORMAT:- Your shell prompt is formatted as follows:- (Open file: <path>)- (Current directory: <cwd>)- bash-$-- First, you should _always_ include a general thought about what you're going to do next.- Then, for every response, you must include exactly _ONE_ tool call/function call.-- Remember, you should always include a _SINGLE_ tool call/function call and then wait for a response from the shell before continuing with more discussion and commands. Everything you include in the DISCUSSION section will be saved for future reference.- If you'd like to issue two commands at once, PLEASE DO NOT DO THAT! Please instead first submit just the first tool call, and then after receiving a response you'll be able to issue the second .- Note that the environment does NOT support interactive session commands (e.g. python, vim), so please do not invoke them.+ You are a helpful assistant that can interact with a computer to solve tasks.instance_template: |-- We're currently solving the following issue within our repository. Here's the issue text:- ISSUE:- {{problem_statement}}-- INSTRUCTIONS:- Now, you're going to solve this issue on your own. Your terminal session has started and you're in the repository's root directory. You can use any bash commands or the special interface to help you. Edit all the files you need to and run any checks or tests that you want.- Remember, YOU SHOULD ALWAYS INCLUDE EXACTLY ONE TOOL CALL/FUNCTION CALL PER RESPONSE.- When you're satisfied with all of the changes you've made, you can submit your changes to the code base by simply running the submit command.- Note however that you cannot use any interactive session commands (e.g. python, vim) in this environment, but you can write scripts and run them. E.g. you can write a python script and then run it with the python command.-- NOTE ABOUT THE EDIT COMMAND: Indentation really matters! When editing a file, make sure to insert appropriate indentation before each line!-- GENERAL IMPORTANT TIPS:+ <uploaded_files>+ {{working_dir}}+ </uploaded_files>+ I've uploaded a python code repository in the directory {{working_dir}}. Consider the following PR description:- 1. If you run a command and it doesn't work, try running a different command. A command that did not work once will not work the second time unless you modify it!-- 2. If you open a file and need to get to an area around a specific line that is not in the first 100 lines, say line 583, don't just use the scroll_down command multiple times. Instead, use the goto 583 command. It's much quicker.-- 3. If the bug reproduction script requires inputting/reading a specific file, such as buggy-input.png, and you'd like to understand how to input that file, conduct a search in the existing repo code, to see whether someone else has already done that. Do this by running the command: find_file "buggy-input.png" If that doesn't work, use the linux 'find' command.-- 4. Always make sure to look at the currently open file and the current working directory (which appears right after the currently open file). The currently open file might be in a different directory than the working directory! Note that some commands, such as 'create', open files, so they might change the current open file.-- 5. When editing files, it is easy to accidentally to write code with incorrect indentation or make other mistakes. Always check the code after you issue an edit to make sure that it reflects what you wanted to accomplish. If it didn't, issue another command to fix it.-- 6. When editing files, first explain the code you want to edit and why it is causing the problem. Then explain the edit you want to make and how it fixes the problem. Explain how the edit does not break existing functionality.-- 7. Do not try to install any packages with `pip`, `conda`, or any other way. This will usually not work. If the environment is not set up correctly, try to fix the issue without executing python code or running any tests that require the package installed.-- STRATEGY:-- 1. Always start by trying to replicate the bug that the issues discusses.- If the issue includes code for reproducing the bug, we recommend that you re-implement that in your environment, and run it to make sure you can reproduce the bug.- Then start trying to fix it.-- If the bug reproduction script does not print anything when it successfully runs, we recommend adding a print("Script completed successfully, no errors.") command at the end of the file,- so that you can be sure that the script indeed ran fine all the way through.-- 2. Locate relevant code using the find and search commands. `open` the file you want to edit.-- 3. Use the `edit` command to perform edits.-- 4. When you think you've fixed the bug, re-run the bug reproduction script to make sure that the bug has indeed been fixed.-- 5. Create additional tests to verify the fix in a style similar to the existing reproduction script. In particular, make sure to test edge cases.- If you find any issues, go back to the file you edited and perform further edits.-- (Open file: {{open_file}})- (Current directory: {{working_dir}})- bash-$+ <pr_description>+ {{problem_statement}}+ </pr_description>++ Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?+ I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!+ Your task is to make the minimal changes to non-tests files in the {{working_dir}} directory to ensure the <pr_description> is satisfied.+ Follow these steps to resolve the issue:+ 1. As a first step, it might be a good idea to find and read code relevant to the <pr_description>+ 2. Create a script to reproduce the error and execute it with `python <filename.py>` using the bash tool, to confirm the error+ 3. Edit the sourcecode of the repo to resolve the issue+ 4. Rerun your reproduce script and confirm that the error is fixed!+ 5. Think about edgecases and make sure your fix handles them as well+ Your thinking should be thorough and so it's fine if it's very long.next_step_template: |-+ OBSERVATION:{{observation}}- (Open file: {{open_file}})- (Current directory: {{working_dir}})- bash-$next_step_no_output_template: |-Your command ran successfully and did not produce any output.- (Open file: {{open_file}})- (Current directory: {{working_dir}})- bash-$- demonstration_template: |- Here is a demonstration of how to correctly accomplish this task.- It is included to show you how to correctly use the interface.- You do not need to follow exactly what is done in the demonstration.- --- DEMONSTRATION ---- {{demonstration}}- --- END OF DEMONSTRATION ---- demonstrations:- - trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__function_calling_replace__install-1/marshmallow-code__marshmallow-1867.traj- put_demos_in_history: truetools:- env_variables:- WINDOW: 100- OVERLAP: 2bundles:- path: tools/registry- - path: tools/defaults- - path: tools/search- # - path: tools/edit_linting- - path: tools/edit_replace- - path: tools/submit+ - path: tools/edit_anthropic+ - path: tools/review_on_submit_m+ registry_variables:+ USE_FILEMAP: 'true'+ SUBMIT_REVIEW_MESSAGES:+ - |+ Thank you for your work on this issue. Please carefully follow the steps below to help review your changes.++ 1. If you made any changes to your code after running the reproduction script, please run the reproduction script again.+ If the reproduction script is failing, please revisit your changes and make sure they are correct.+ If you have already removed your reproduction script, please ignore this step.+ 2. Remove your reproduction script (if you haven't done so already).+ 3. If you have modified any TEST files, please revert them to the state they had before you started fixing the issue.+ You can do this with `git checkout -- /path/to/test/file.py`. Use below <diff> to find the files you need to revert.+ 4. Run the submit command again to confirm.++ Here is a list of all of your changes:++ <diff>+ {{diff}}+ </diff>enable_bash_tool: trueparse_function:type: function_callinghistory_processors:- - type: last_n_observations- n: 5+ - type: cache_control+ last_n_messages: 2diff --git a/config/anthropic_filemap.yaml b/config/default_backticks.yamlsimilarity index 96%rename from config/anthropic_filemap.yamlrename to config/default_backticks.yamlindex 671c172b..66bb9677 100644--- a/config/anthropic_filemap.yaml+++ b/config/default_backticks.yaml@@ -1,4 +1,6 @@-# This template is heavily inspired by anthropic's computer use demo+# Formerly called: anthropic_filemap.yaml+# This template is heavily inspired by anthropic's computer use demo, but you can use+# it with any LM.agent:templates:system_template: |-@@ -54,7 +56,7 @@ agent:</diff>enable_bash_tool: trueparse_function:- type: function_calling+ type: thought_actionhistory_processors:- type: cache_controllast_n_messages: 2diff --git a/config/function_calling_replace.yaml b/config/exotic/windowed_replace.yamlsimilarity index 98%rename from config/function_calling_replace.yamlrename to config/exotic/windowed_replace.yamlindex e86b85a0..86d90936 100644--- a/config/function_calling_replace.yaml+++ b/config/exotic/windowed_replace.yaml@@ -1,3 +1,4 @@+# This config uses the windowed-replace tools together with a prompt similar toagent:templates:system_template: |-diff --git a/config/function_calling_replace_late_repro.yaml b/config/exotic/windowed_replace_late_repro.yamlsimilarity index 97%rename from config/function_calling_replace_late_repro.yamlrename to config/exotic/windowed_replace_late_repro.yamlindex b6b4ff82..196f9a58 100644--- a/config/function_calling_replace_late_repro.yaml+++ b/config/exotic/windowed_replace_late_repro.yaml@@ -1,3 +1,5 @@+# This config is similar to windowed_replace.yaml, but with a slightly tweaked prompt that encourages the model+# to write the reproduction script _after_ it has investigated the codebase.agent:templates:system_template: |-diff --git a/config/human.yaml b/config/human/human.yamlsimilarity index 84%rename from config/human.yamlrename to config/human/human.yamlindex d368deec..3be9b8d2 100644--- a/config/human.yaml+++ b/config/human/human.yaml@@ -14,8 +14,5 @@ agent:- path: tools/submitparse_function:type: thought_action- history_processors:- - type: last_n_observations- n: 5model:name: humandiff --git a/config/human_demo.yaml b/config/human/human_demo.yamlsimilarity index 100%rename from config/human_demo.yamlrename to config/human/human_demo.yamldiff --git a/config/default_no_fcalls.yaml b/config/sweagent_0_7/07.yamlsimilarity index 99%rename from config/default_no_fcalls.yamlrename to config/sweagent_0_7/07.yamlindex d0c3d6f0..ffe84e40 100644--- a/config/default_no_fcalls.yaml+++ b/config/sweagent_0_7/07.yaml@@ -1,3 +1,4 @@+# This is the configuration from SWE-agent 0.7agent:templates:system_template: |-diff --git a/config/function_calling.yaml b/config/sweagent_0_7/07_fcalling.yamlsimilarity index 97%rename from config/function_calling.yamlrename to config/sweagent_0_7/07_fcalling.yamlindex 2cb46988..b8d7fad5 100644--- a/config/function_calling.yaml+++ b/config/sweagent_0_7/07_fcalling.yaml@@ -1,3 +1,5 @@+# This config shows the use of the function calling action parser together with the line-range based replace tools+# This config is close to SWE-agent 0.7agent:templates:system_template: |-diff --git a/config/default_from_url.yaml b/config/sweagent_0_7/07_from_url.yamlsimilarity index 96%rename from config/default_from_url.yamlrename to config/sweagent_0_7/07_from_url.yamlindex 6b1895ba..ace0ccac 100644--- a/config/default_from_url.yaml+++ b/config/sweagent_0_7/07_from_url.yaml@@ -1,3 +1,5 @@+# This config is close to SWE-agent 0.7, i.e., using the line-range based replace edit tools.+# This config was specifically used to be pointed to an arbitrary github issue rather than for benchmarking.agent:templates:system_template: |-diff --git a/config/thought_action.yaml b/config/sweagent_0_7/07_thought_action.yamlsimilarity index 97%rename from config/thought_action.yamlrename to config/sweagent_0_7/07_thought_action.yamlindex f4b87e41..ee5df329 100644--- a/config/thought_action.yaml+++ b/config/sweagent_0_7/07_thought_action.yaml@@ -1,3 +1,5 @@+# This config shows the use of the thought_action action parser together with the line-range based replace tools+# This config is close to SWE-agent 0.7agent:templates:system_template: |-diff --git a/config/thought_action_xml.yaml b/config/sweagent_0_7/07_thought_action_xml.yamlsimilarity index 97%rename from config/thought_action_xml.yamlrename to config/sweagent_0_7/07_thought_action_xml.yamlindex 11a70902..4d9bbe17 100644--- a/config/thought_action_xml.yaml+++ b/config/sweagent_0_7/07_thought_action_xml.yaml@@ -1,3 +1,5 @@+# This config shows the use of the thought action xml parser together with the line-range based replace tools+# This config is close to SWE-agent 0.7agent:templates:system_template: |-diff --git a/docs/config/config.md b/docs/config/config.mdindex 2b27e47a..acf374af 100644--- a/docs/config/config.md+++ b/docs/config/config.md@@ -23,10 +23,10 @@ sweagent run-batch --config config/your_config.yamlThis is the current default configuration file which is loaded when no `--config` flag is provided:<details>-<summary><code>default_from_url.yaml</code></summary>+<summary><code>default.yaml</code></summary>-```yaml title="config/default_from_url.yaml"---8<-- "config/default_from_url.yaml"+```yaml title="config/default.yaml"+--8<-- "config/default.yaml"```</details>diff --git a/docs/reference/agent_config.md b/docs/reference/agent_config.mdindex de442a85..372ab144 100644--- a/docs/reference/agent_config.md+++ b/docs/reference/agent_config.md@@ -6,10 +6,10 @@ To learn about the agent class itself, see the [agent class reference page](agenIt might be easiest to simply look at some of our example configurations in the [config dir](https://github.com/SWE-agent/SWE-agent/tree/main/config).<details>-<summary>Example: default config <code>anthropic_filemap.yaml</code></summary>+<summary>Example: default config <code>default.yaml</code></summary>```yaml---8<-- "config/default_from_url.yaml"+--8<-- "config/default.yaml"```</details>diff --git a/docs/usage/batch_mode.md b/docs/usage/batch_mode.mdindex e630c11e..ea948f2f 100644--- a/docs/usage/batch_mode.md+++ b/docs/usage/batch_mode.md@@ -14,7 +14,7 @@ Let's run on three [SWE-bench](https://www.swebench.com/) issues which will be d```bashsweagent run-batch \- --config config/anthropic_filemap.yaml \+ --config config/default.yaml \--agent.model.name gpt-4o \--agent.model.per_instance_cost_limit 2.00 \--instances.type swe_bench \@@ -58,7 +58,7 @@ Let's speed things up and run on 5 instances at once. Only a single line to chan```bash hl_lines="4-4"sweagent run-batch \- --config config/anthropic_filemap.yaml \+ --config config/default.yaml \--agent.model.name gpt-4o \--num_workers 3 \--agent.model.per_instance_cost_limit 2.00 \@@ -88,7 +88,7 @@ thereby easing CPU pressure. Default is 0.3.```bashsweagent run-batch \- --config config/anthropic_filemap.yaml \+ --config config/default.yaml \--agent.model.name gpt-4o \--instances.type file \--instances.path instances.yaml \diff --git a/docs/usage/cl_tutorial.md b/docs/usage/cl_tutorial.mdindex 2b192510..b3ccdff3 100644--- a/docs/usage/cl_tutorial.md+++ b/docs/usage/cl_tutorial.md@@ -110,13 +110,13 @@ But we can also split it up into multiple files and additional command line optisecond config would completely overwrite all `agent` settings of the first config.This is fixed since SWE-agent 1.1.0.-The default config file is `config/anthropic_filemap.yaml`. Let's take a look at it:+The default config file is `config/default.yaml`. Let's take a look at it:<details>-<summary>Example: default config <code>anthropic_filemap.yaml</code></summary>+<summary>Example: default config <code>default.yaml</code></summary>```yaml---8<-- "config/anthropic_filemap.yaml"+--8<-- "config/default.yaml"```</details>@@ -126,7 +126,7 @@ This file is also loaded when no other `--config` options are specified.So to make sure that we get the default templates in the above examples with `--config`, we should have added```bash---config config/anthropic_filemap.yaml+--config config/default.yaml```in addition to all the other `--config` options for the two examples above.diff --git a/sweagent/agent/models.py b/sweagent/agent/models.pyindex ea159fb7..bb6e6720 100644--- a/sweagent/agent/models.py+++ b/sweagent/agent/models.py@@ -543,7 +543,7 @@ class InstantEmptySubmitTestModel(AbstractModel):"DISCUSSION\n""Let's reproduce the bug by creating a `reproduce.py` file.\n\n""```\n"- "create reproduce.py\n"+ "touch reproduce.py\n""```\n")elif self._action_idx == 1:diff --git a/sweagent/api/server.py b/sweagent/api/server.pyindex 0f3f2ad0..0fd29e53 100644--- a/sweagent/api/server.py+++ b/sweagent/api/server.py@@ -114,7 +114,7 @@ def run():test_run: bool = run.extra.test_runif test_run:model_name = "instant_empty_submit"- default_config = yaml.safe_load(Path(CONFIG_DIR / "default_from_url.yaml").read_text())+ default_config = yaml.safe_load(Path(CONFIG_DIR / "default.yaml").read_text())config = {**default_config,"agent": {diff --git a/sweagent/run/common.py b/sweagent/run/common.pyindex 8e45c323..dd2d0c0c 100644--- a/sweagent/run/common.py+++ b/sweagent/run/common.py@@ -289,7 +289,7 @@ class BasicCLI:_loaded = yaml.safe_load(txt)merge_nested_dicts(config_merged, _loaded)elif self.default_settings and not cli_args.no_config_file:- config_file = CONFIG_DIR / "anthropic_filemap.yaml"+ config_file = CONFIG_DIR / "default.yaml"config_files.append(config_file)msg = (f"Loading default config from {config_file}, because no other "diff --git a/sweagent/run/run_batch.py b/sweagent/run/run_batch.pyindex 563d2a6e..bd1ffe92 100644--- a/sweagent/run/run_batch.py+++ b/sweagent/run/run_batch.py@@ -16,8 +16,8 @@ sweagent run-batch \\--instances.split dev \\--instances.slice :50 \\ # first 50 instances--instances.shuffle=True \\ # shuffle instances (with fixed seed)- --config config/anthropic_filemap.yaml \\ # configure model- --agent.model.name gpt-4o+ --config config/default.yaml \\+ --agent.model.name gpt-4o # configure model[/green][cyan][bold]=== LOADING INSTANCES ===[/bold][/cyan]diff --git a/sweagent/run/run_single.py b/sweagent/run/run_single.pyindex 39b58e33..8ba4f77b 100644--- a/sweagent/run/run_single.py+++ b/sweagent/run/run_single.py@@ -11,7 +11,7 @@Basic usage: Run over a [bold][cyan]github issue[/bold][/cyan][green]:-sweagent run --config config/anthropic_filemap.yaml --agent.model.name "gpt-4o" \\+sweagent run --config config/default.yaml --agent.model.name "gpt-4o" \\--env.repo.github_url=https://github.com/SWE-agent/test-repo/ \\--problem_statement.github_url=https://github.com/SWE-agent/test-repo/issues/1[/green]@@ -21,7 +21,7 @@ You can set the image with [green]--env.docker.image[/green].Here's an example that uses [bold][cyan]modal[/bold][/cyan] instead of docker and also a [bold][cyan]local repository[/bold][/cyan]:-[green]sweagent run --config config/anthropic_filemap.yaml --agent.model.name "gpt-4o" \\+[green]sweagent run --config config/default.yaml --agent.model.name "gpt-4o" \\--env.deployment.type=modal --env.repo.path /path/to/repo \\--problem_statement.path=path/to/problem_statement.md[/green]diff --git a/tests/test_agent.py b/tests/test_agent.pyindex 84ef1a87..dc0e610e 100644--- a/tests/test_agent.py+++ b/tests/test_agent.py@@ -49,7 +49,7 @@ def function_calling_agent_config():@pytest.fixturedef default_agent_config():- config = yaml.safe_load((CONFIG_DIR / "default_no_fcalls.yaml").read_text())+ config = yaml.safe_load((CONFIG_DIR / "sweagent_0_7/07.yaml").read_text())config["agent"]["model"] = {"name": "instant_empty_submit"}print(yaml.dump(config))return DefaultAgentConfig.model_validate(config["agent"])diff --git a/tests/test_run_single.py b/tests/test_run_single.pyindex 59e8fff0..893d0f36 100644--- a/tests/test_run_single.py+++ b/tests/test_run_single.py@@ -111,7 +111,7 @@ def test_run_ies_repo_ps_matrix(*ps_args,*repo_args,"--config",- str(CONFIG_DIR / "default_no_fcalls.yaml"),+ str(CONFIG_DIR / "default_backticks.yaml"),]print(args)rs_config = BasicCLI(RunSingleConfig).get_config(args)@@ -121,5 +121,5 @@ def test_run_ies_repo_ps_matrix(# Test that we can run run.py also independently from repo dirrs.run()for fmt in output_formats:- assert len(list(Path(tmpdir).rglob(f"*.{fmt}"))) == 1print(fmt, list(Path(tmpdir).iterdir()))+ assert len(list(Path(tmpdir).rglob(f"*.{fmt}"))) == 1