Skip to content

SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]

Doc: Fix default config & model (#1105)

Kilian Lieret83fe102f1fb2
diff --git a/.devcontainer/bashrc_epilog.sh b/.devcontainer/bashrc_epilog.sh
index 8f3f8855..172c8432 100644
--- a/.devcontainer/bashrc_epilog.sh
+++ b/.devcontainer/bashrc_epilog.sh
@@ -6,7 +6,7 @@ fi
echo "Here's an example SWE-agent command to try out:"
echo "sweagent run \\
- --agent.model.name=claude-3-5-sonnet-20241022 \\
+ --agent.model.name=claude-3-7-sonnet-latest \\
--agent.model.per_instance_cost_limit=2.00 \\
--env.repo.github_url=https://github.com/SWE-agent/test-repo \\
--problem_statement.github_url=https://github.com/SWE-agent/test-repo/issues/1 \\
diff --git a/config/250212_sweagent_heavy_sbl.yaml b/config/250212_sweagent_heavy_sbl.yaml
index a7c0c813..89470625 100644
--- a/config/250212_sweagent_heavy_sbl.yaml
+++ b/config/250212_sweagent_heavy_sbl.yaml
@@ -9,7 +9,7 @@ agent:
# +filemap
- type: default
model: &model
- name: claude-3-5-sonnet-20241022
+ name: claude-3-7-sonnet-latest
api_key: $CLAUDE_API_KEY_ROTATION
per_instance_cost_limit: 1.5
per_instance_call_limit: 75
diff --git a/docs/installation/keys.md b/docs/installation/keys.md
index 5e7855ba..be27a7ac 100644
--- a/docs/installation/keys.md
+++ b/docs/installation/keys.md
@@ -40,7 +40,7 @@ Here are a few options for `--agent.model.name`:
| Model | API key | Comment |
| ----- | ------- | ------- |
-| `claude-3-5-sonnet-20241022` | `ANTHROPIC_API_KEY` | Our recommended model |
+| `claude-3-7-sonnet-latest` | `ANTHROPIC_API_KEY` | Our recommended model |
| `gpt-4o` | `OPENAI_API_KEY` | |
| `o1-preview` | `OPENAI_API_KEY` | You might need to set temperature and sampling to the supported values. |
diff --git a/docs/reference/agent_config.md b/docs/reference/agent_config.md
index 4f6e804b..de442a85 100644
--- a/docs/reference/agent_config.md
+++ b/docs/reference/agent_config.md
@@ -6,7 +6,7 @@ To learn about the agent class itself, see the [agent class reference page](agen
It might be easiest to simply look at some of our example configurations in the [config dir](https://github.com/SWE-agent/SWE-agent/tree/main/config).
<details>
-<summary>Example: default config <code>default.yaml</code></summary>
+<summary>Example: default config <code>anthropic_filemap.yaml</code></summary>
```yaml
--8<-- "config/default_from_url.yaml"
diff --git a/docs/usage/batch_mode.md b/docs/usage/batch_mode.md
index ea948f2f..e630c11e 100644
--- a/docs/usage/batch_mode.md
+++ b/docs/usage/batch_mode.md
@@ -14,7 +14,7 @@ Let's run on three [SWE-bench](https://www.swebench.com/) issues which will be d
```bash
sweagent run-batch \
- --config config/default.yaml \
+ --config config/anthropic_filemap.yaml \
--agent.model.name gpt-4o \
--agent.model.per_instance_cost_limit 2.00 \
--instances.type swe_bench \
@@ -58,7 +58,7 @@ Let's speed things up and run on 5 instances at once. Only a single line to chan
```bash hl_lines="4-4"
sweagent run-batch \
- --config config/default.yaml \
+ --config config/anthropic_filemap.yaml \
--agent.model.name gpt-4o \
--num_workers 3 \
--agent.model.per_instance_cost_limit 2.00 \
@@ -88,7 +88,7 @@ thereby easing CPU pressure. Default is 0.3.
```bash
sweagent run-batch \
- --config config/default.yaml \
+ --config config/anthropic_filemap.yaml \
--agent.model.name gpt-4o \
--instances.type file \
--instances.path instances.yaml \
diff --git a/docs/usage/cl_tutorial.md b/docs/usage/cl_tutorial.md
index 3d88c9bd..59049a7a 100644
--- a/docs/usage/cl_tutorial.md
+++ b/docs/usage/cl_tutorial.md
@@ -103,13 +103,13 @@ But we can also split it up into multiple files and additional command line opti
github_url: https://github.com/SWE-agent/test-repo
```
-The default config file is `config/default.yaml`. Let's take a look at it:
+The default config file is `config/anthropic_filemap.yaml`. Let's take a look at it:
<details>
-<summary>Example: default config <code>default.yaml</code></summary>
+<summary>Example: default config <code>anthropic_filemap.yaml</code></summary>
```yaml
---8<-- "config/default_from_url.yaml"
+--8<-- "config/anthropic_filemap.yaml"
```
</details>
@@ -119,7 +119,7 @@ This file is also loaded when no other `--config` options are specified.
So to make sure that we get the default templates in the above examples with `--config`, we should have added
```bash
---config config/default.yaml
+--config config/anthropic_filemap.yaml
```
in addition to all the other `--config` options for the two examples above.
@@ -207,7 +207,7 @@ In addition, you can also execute additional commands before starting the agent
```bash
sweagent run \
- --agent.model.name=claude-3-5-sonnet-20241022 \
+ --agent.model.name=claude-3-7-sonnet-latest \
--env.post_startup_commands='["pip install flake8"]' \
...
```
diff --git a/docs/usage/competitive_runs.md b/docs/usage/competitive_runs.md
index 84c169da..fac58b1f 100644
--- a/docs/usage/competitive_runs.md
+++ b/docs/usage/competitive_runs.md
@@ -14,7 +14,7 @@ We recently used two configurations for SWE-bench submissions
* [250225_anthropic_filemap_simple_review.yaml](https://github.com/SWE-agent/SWE-agent/blob/main/config/250225_anthropic_filemap_simple_review.yaml):
This is our current default one-attempt config. It uses `claude-3-7-sonnet-20250219`.
* [250212_sweagent_heavy_sbl.yaml](https://github.com/SWE-agent/SWE-agent/blob/main/config/250212_sweagent_heavy_sbl.yaml):
- This config runs 5 attempts with slightly different configurations using `claude-3-5-sonnet-20241022`,
+ This config runs 5 attempts with slightly different configurations using `claude-3-7-sonnet-latest`,
then uses o1 to discriminate between them.
This is a very expensive configuration.
If you use it, also make sure to use Claude 3.7 instead of claude 3.5.
diff --git a/docs/usage/hello_world.md b/docs/usage/hello_world.md
index 15b4a540..824abfb0 100644
--- a/docs/usage/hello_world.md
+++ b/docs/usage/hello_world.md
@@ -20,7 +20,7 @@ Let's start with an absolutely trivial example and solve an issue about a simple
```bash
sweagent run \
- --agent.model.name=claude-3-5-sonnet-20241022 \
+ --agent.model.name=claude-3-7-sonnet-latest \
--agent.model.per_instance_cost_limit=2.00 \
--env.repo.github_url=https://github.com/SWE-agent/test-repo \
--problem_statement.github_url=https://github.com/SWE-agent/test-repo/issues/1
diff --git a/docs/usage/hello_world_output.txt b/docs/usage/hello_world_output.txt
index 9d8f2744..b9b56755 100644
--- a/docs/usage/hello_world_output.txt
+++ b/docs/usage/hello_world_output.txt
@@ -17,7 +17,7 @@
๐ŸŒฑ DEBUG Resetting repository SWE-agent__test-repo to commit HEAD
๐Ÿƒ INFO Running agent
๐Ÿค  INFO Trajectory will be saved to
- /Users/fuchur/Documents/24/git_sync/SWE-agent/trajectories/fuchur/default__claude-3-5-sonnet-20241022__t-0.00__p-1.00__c-2.00___SWE-agent__test-repo-i1/SWE-agent__test-
+ /Users/fuchur/Documents/24/git_sync/SWE-agent/trajectories/fuchur/default__claude-3-7-sonnet-latest__t-0.00__p-1.00__c-2.00___SWE-agent__test-repo-i1/SWE-agent__test-
repo-i1/SWE-agent__test-repo-i1.traj
๐Ÿฆ– DEBUG Uploading file from /Users/fuchur/Documents/24/git_sync/SWE-agent/tools/registry to /root/tools/registry
๐Ÿฆ– DEBUG Created zip file at /var/folders/g3/1y2_hl1j16scls01cxqm76s80000gn/T/tmpswoslekq/zipped_transfer.zip
@@ -363,7 +363,7 @@
(Current directory: /SWE-agent__test-repo)
bash-$
๐Ÿค  INFO Trajectory saved to
- /Users/fuchur/Documents/24/git_sync/SWE-agent/trajectories/fuchur/default__claude-3-5-sonnet-20241022__t-0.00__p-1.00__c-2.00___SWE-agent__test-repo-i1/SWE-agent__test-
+ /Users/fuchur/Documents/24/git_sync/SWE-agent/trajectories/fuchur/default__claude-3-7-sonnet-latest__t-0.00__p-1.00__c-2.00___SWE-agent__test-repo-i1/SWE-agent__test-
repo-i1/SWE-agent__test-repo-i1.traj
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ๐ŸŽ‰ Submission successful ๐ŸŽ‰ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ SWE-agent has produced a patch that it believes will solve the issue you submitted! โ”‚
@@ -371,7 +371,7 @@
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
# The patch has been saved to your local filesystem at:
- PATCH_FILE_PATH='/Users/fuchur/Documents/24/git_sync/SWE-agent/trajectories/fuchur/default__claude-3-5-sonnet-20241022__t-0.00__p-1.00__c-2.00___SWE-agent__test-repo-i1/SWE-agent
+ PATCH_FILE_PATH='/Users/fuchur/Documents/24/git_sync/SWE-agent/trajectories/fuchur/default__claude-3-7-sonnet-latest__t-0.00__p-1.00__c-2.00___SWE-agent__test-repo-i1/SWE-agent
test-repo-i1.patch'
# Inspect it:
cat "${PATCH_FILE_PATH}"
diff --git a/sweagent/run/run_batch.py b/sweagent/run/run_batch.py
index 2b04cc7c..e37816f0 100644
--- a/sweagent/run/run_batch.py
+++ b/sweagent/run/run_batch.py
@@ -16,7 +16,7 @@ sweagent run-batch \\
--instances.split dev \\
--instances.slice :50 \\ # first 50 instances
--instances.shuffle=True \\ # shuffle instances (with fixed seed)
- --config config/default.yaml \\ # configure model
+ --config config/anthropic_filemap.yaml \\ # configure model
--agent.model.name gpt-4o
[/green]
diff --git a/sweagent/run/run_single.py b/sweagent/run/run_single.py
index 8ba4f77b..39b58e33 100644
--- a/sweagent/run/run_single.py
+++ b/sweagent/run/run_single.py
@@ -11,7 +11,7 @@
Basic usage: Run over a [bold][cyan]github issue[/bold][/cyan][green]:
-sweagent run --config config/default.yaml --agent.model.name "gpt-4o" \\
+sweagent run --config config/anthropic_filemap.yaml --agent.model.name "gpt-4o" \\
--env.repo.github_url=https://github.com/SWE-agent/test-repo/ \\
--problem_statement.github_url=https://github.com/SWE-agent/test-repo/issues/1
[/green]
@@ -21,7 +21,7 @@ You can set the image with [green]--env.docker.image[/green].
Here's an example that uses [bold][cyan]modal[/bold][/cyan] instead of docker and also a [bold][cyan]local repository[/bold][/cyan]:
-[green]sweagent run --config config/default.yaml --agent.model.name "gpt-4o" \\
+[green]sweagent run --config config/anthropic_filemap.yaml --agent.model.name "gpt-4o" \\
--env.deployment.type=modal --env.repo.path /path/to/repo \\
--problem_statement.path=path/to/problem_statement.md
[/green]
diff --git a/tests/test_data/trajectories/gpt4__swe-bench-dev-easy_first_only__default__t-0.00__p-0.95__c-3.00__install-1/args.yaml b/tests/test_data/trajectories/gpt4__swe-bench-dev-easy_first_only__default__t-0.00__p-0.95__c-3.00__install-1/args.yaml
index e3a9be65..672bea69 100644
--- a/tests/test_data/trajectories/gpt4__swe-bench-dev-easy_first_only__default__t-0.00__p-0.95__c-3.00__install-1/args.yaml
+++ b/tests/test_data/trajectories/gpt4__swe-bench-dev-easy_first_only__default__t-0.00__p-0.95__c-3.00__install-1/args.yaml
@@ -493,7 +493,7 @@ agent:
end_name: null
name: _constrain_line
signature: _constrain_line
- config_file: config/default.yaml
+ config_file: config/anthropic_filemap.yaml
model:
host_url: localhost:11434
model_name: gpt4