Skip to content

SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]

Docs: Remove 'benchmarking' (part of batch tutorial now) (#903)

Kilian Lieret95e22ebed4f4
diff --git a/docs/usage/batch_mode.md b/docs/usage/batch_mode.md
index cc4c1387..4baf6c92 100644
--- a/docs/usage/batch_mode.md
+++ b/docs/usage/batch_mode.md
@@ -46,6 +46,10 @@ Kind of slow, isn't it?
* See [`RunBatchConfig`](../reference/run_batch_config.md/#sweagent.run.run_batch.RunBatchConfig) for an overview of all options.
* SWE-bench config: [`SWEBenchInstances`](../reference/batch_instances.md/#sweagent.run.batch_instances.SWEBenchInstances).
+!!! tip "Evaluating on SWE-bench"
+ If you are using [`sb-cli`](https://www.swebench.com/sb-cli/), you can automatically evaluate on SWE-bench by adding the `--evaluate=True` flag.
+ This will already submit submissions to `sb-cli` while you are running, so that you should receive results within a minute of finishing your run.
+
## Running in parallel
Let's speed things up and run on 5 instances at once. Only a single line to change:
diff --git a/docs/usage/benchmarking.md b/docs/usage/benchmarking.md
deleted file mode 100644
index 3cc68499..00000000
--- a/docs/usage/benchmarking.md
+++ /dev/null
@@ -1,60 +0,0 @@
-# Benchmarking
-
-!!! abstract "Benchmarking on SWE-bench"
- This page talks about benchmarking on SWE-bench to measure the software engineering capabilities of SWE-agent.
- Benchmarking for the other modes (programming challenges, cybersecurity) are coming soon.
-
-There are two steps to the SWE-agent/SWE-bench pipeline. First SWE-agent takes an input GitHub issue and returns a pull request that attempts to fix it. We call that step *inference*. The second step (currently only available for issues in the SWE-bench benchmark) is to *evaluate* the pull request to verify that it has indeed fixed the issue.
-
-!!! warning "Architectures"
- At this moment, there are known issues with a small number of repositories that don't install properly for `arm64` / `aarch64` architecture computers. We're working on a fix, but if you'd like to run and evaluate on the entirety of SWE-bench, the easiest way is by using an `x86` machine.
-
-## πŸ‘©β€πŸ’» Inference <a name="inference"></a>
-
-Run SWE-agent on [SWE-bench Lite](https://www.swebench.com/lite.html) and generate patches.
-
-```bash
-python run.py --model_name gpt4 \
- --per_instance_cost_limit 2.00 \
- --config_file ./config/default.yaml
-```
-
-If you'd like to run on a *single* issue from SWE-bench, use the `--instance_filter` option as follows:
-```bash
-python run.py --model_name gpt4 \
- --instance_filter marshmallow-code__marshmallow-1359
-```
-
-The above examples use the default value of `--data_path` (`princeton-nlp/SWE-bench_Lite`, which will be looked up from huggingface).
-You can specify any other huggingface datasets as well, or supply the path to a pre-downloaded dataset.
-By default, SWE-agent evaluates on the `dev` split of that dataset.
-You can change that by supplying the `--split` argument to the above commands (obviously you shouldn't tune your model on the `test` dataset).
-
-## πŸ§ͺ Evaluation <a name="evaluation"></a>
-
-!!! note "Previous `evaluation/` scripts"
-
- We have removed the scripts that were previously in the `evaluation/` subfolder.
- They were relatively thin wrappers around `swe-bench`. After the `swe-bench` 2.0 update, we recommend to use `swe-bench` directly.
-
-You can directly run SWE-bench to evaluate the SWE-agent results.
-
-After installing SWE-bench, you can run `run_evaluation` as such:
-
-```bash
-python -m swebench.harness.run_evaluation \
- --predictions_path /path/to/all_preds.jsonl \
- --max_workers 1 \
- --run_id test \
- --split dev
-```
-
-Head over to [SWE-bench](https://github.com/SWE-bench/SWE-bench/) for details.
-
-!!! note "Default split"
-
- When running `swe-agent` it uses the SWE-bench lite `dev` split by default (i.e., when not specifying `--data_path` or `--split`).
- However, `swe-bench` assumes the SWE-bench lite `test` split by default.
- When you get warnings from `swe-bench` about missing instances, make sure you specify `--split test` or `--split dev` appropriately.
-
-{% include-markdown "../_footer.md" %}
diff --git a/docs/usage/cl_tutorial.md b/docs/usage/cl_tutorial.md
index e9a2c50f..bafacee6 100644
--- a/docs/usage/cl_tutorial.md
+++ b/docs/usage/cl_tutorial.md
@@ -5,7 +5,7 @@
* Please read our [hello world](hello_world.md) tutorial before proceeding.
* This tutorial focuses on using SWE-agent as a tool to solve individual issues.
- Benchmarking SWE-agent is covered [separately](benchmarking.md).
+ Benchmarking SWE-agent is covered [separately](batch_mode.md).
Finally, we have a different tutorial for using SWE-agent for [coding challenges](coding_challenges.md).
## A few examples
@@ -128,7 +128,7 @@ in addition to all the other `--config` options for the two examples above.
!!! note "Operating in batch mode: Running on SWE-bench and other benchmark sets"
If you want to run SWE-agent in batch mode on SWE-bench or another whole evaluation set, see
- [benchmarking](benchmarking.md). This tutorial focuses on using SWE-agent on
+ [batch mode](batch_mode.md). This tutorial focuses on using SWE-agent on
individual issues.
We've already seen a few examples of how to specify the problem to solve, namely
diff --git a/docs/usage/inspector.md b/docs/usage/inspector.md
index fe1a4d67..787d7eb6 100644
--- a/docs/usage/inspector.md
+++ b/docs/usage/inspector.md
@@ -37,7 +37,7 @@ The inspector will then be launched in the browser:
## Benchmark results
-If you are running SWE-agent on a benchmark (see [benchmarking](benchmarking.md)), you will see evaluation results as βœ… or ❌.
+If you are running SWE-agent on a benchmark (see [batch mode](batch_mode.md)), you will see evaluation results as βœ… or ❌.
Otherwise, you will see ❓.
!!! tip
diff --git a/docs/usage/trajectories.md b/docs/usage/trajectories.md
index 6f6369d4..0b8683c2 100644
--- a/docs/usage/trajectories.md
+++ b/docs/usage/trajectories.md
@@ -85,8 +85,8 @@ Here are a few more files that are generated with `sweagent run-batch`:
!!! tip
- * Evaluation is not completed by `sweagent run-batch`, it is a separate step (see [benchmarking](benchmarking.md))
- * `preds.json` can be referenced directly into SWE-bench to run evaluation (see [benchmarking](benchmarking.md))
+ * Evaluation is not completed by `sweagent run-batch`, it is a separate step (see [batch mode](batch_mode.md))
+ * `preds.json` can be referenced directly into SWE-bench to run evaluation
* Trajectories can be turned into custom demonstrations for SWE-agent ([more information](../config/demonstrations.md)).
{% include-markdown "../_footer.md" %}