SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]
Docs: Remove 'benchmarking' (part of batch tutorial now) (#903)
Kilian Lieret95e22ebed4f4
diff --git a/docs/usage/batch_mode.md b/docs/usage/batch_mode.mdindex cc4c1387..4baf6c92 100644--- a/docs/usage/batch_mode.md+++ b/docs/usage/batch_mode.md@@ -46,6 +46,10 @@ Kind of slow, isn't it?* See [`RunBatchConfig`](../reference/run_batch_config.md/#sweagent.run.run_batch.RunBatchConfig) for an overview of all options.* SWE-bench config: [`SWEBenchInstances`](../reference/batch_instances.md/#sweagent.run.batch_instances.SWEBenchInstances).+!!! tip "Evaluating on SWE-bench"+ If you are using [`sb-cli`](https://www.swebench.com/sb-cli/), you can automatically evaluate on SWE-bench by adding the `--evaluate=True` flag.+ This will already submit submissions to `sb-cli` while you are running, so that you should receive results within a minute of finishing your run.+## Running in parallelLet's speed things up and run on 5 instances at once. Only a single line to change:diff --git a/docs/usage/benchmarking.md b/docs/usage/benchmarking.mddeleted file mode 100644index 3cc68499..00000000--- a/docs/usage/benchmarking.md+++ /dev/null@@ -1,60 +0,0 @@-# Benchmarking--!!! abstract "Benchmarking on SWE-bench"- This page talks about benchmarking on SWE-bench to measure the software engineering capabilities of SWE-agent.- Benchmarking for the other modes (programming challenges, cybersecurity) are coming soon.--There are two steps to the SWE-agent/SWE-bench pipeline. First SWE-agent takes an input GitHub issue and returns a pull request that attempts to fix it. We call that step *inference*. The second step (currently only available for issues in the SWE-bench benchmark) is to *evaluate* the pull request to verify that it has indeed fixed the issue.--!!! warning "Architectures"- At this moment, there are known issues with a small number of repositories that don't install properly for `arm64` / `aarch64` architecture computers. We're working on a fix, but if you'd like to run and evaluate on the entirety of SWE-bench, the easiest way is by using an `x86` machine.--## π©βπ» Inference <a name="inference"></a>--Run SWE-agent on [SWE-bench Lite](https://www.swebench.com/lite.html) and generate patches.--```bash-python run.py --model_name gpt4 \- --per_instance_cost_limit 2.00 \- --config_file ./config/default.yaml-```--If you'd like to run on a *single* issue from SWE-bench, use the `--instance_filter` option as follows:-```bash-python run.py --model_name gpt4 \- --instance_filter marshmallow-code__marshmallow-1359-```--The above examples use the default value of `--data_path` (`princeton-nlp/SWE-bench_Lite`, which will be looked up from huggingface).-You can specify any other huggingface datasets as well, or supply the path to a pre-downloaded dataset.-By default, SWE-agent evaluates on the `dev` split of that dataset.-You can change that by supplying the `--split` argument to the above commands (obviously you shouldn't tune your model on the `test` dataset).--## π§ͺ Evaluation <a name="evaluation"></a>--!!! note "Previous `evaluation/` scripts"-- We have removed the scripts that were previously in the `evaluation/` subfolder.- They were relatively thin wrappers around `swe-bench`. After the `swe-bench` 2.0 update, we recommend to use `swe-bench` directly.--You can directly run SWE-bench to evaluate the SWE-agent results.--After installing SWE-bench, you can run `run_evaluation` as such:--```bash-python -m swebench.harness.run_evaluation \- --predictions_path /path/to/all_preds.jsonl \- --max_workers 1 \- --run_id test \- --split dev-```--Head over to [SWE-bench](https://github.com/SWE-bench/SWE-bench/) for details.--!!! note "Default split"-- When running `swe-agent` it uses the SWE-bench lite `dev` split by default (i.e., when not specifying `--data_path` or `--split`).- However, `swe-bench` assumes the SWE-bench lite `test` split by default.- When you get warnings from `swe-bench` about missing instances, make sure you specify `--split test` or `--split dev` appropriately.--{% include-markdown "../_footer.md" %}diff --git a/docs/usage/cl_tutorial.md b/docs/usage/cl_tutorial.mdindex e9a2c50f..bafacee6 100644--- a/docs/usage/cl_tutorial.md+++ b/docs/usage/cl_tutorial.md@@ -5,7 +5,7 @@* Please read our [hello world](hello_world.md) tutorial before proceeding.* This tutorial focuses on using SWE-agent as a tool to solve individual issues.- Benchmarking SWE-agent is covered [separately](benchmarking.md).+ Benchmarking SWE-agent is covered [separately](batch_mode.md).Finally, we have a different tutorial for using SWE-agent for [coding challenges](coding_challenges.md).## A few examples@@ -128,7 +128,7 @@ in addition to all the other `--config` options for the two examples above.!!! note "Operating in batch mode: Running on SWE-bench and other benchmark sets"If you want to run SWE-agent in batch mode on SWE-bench or another whole evaluation set, see- [benchmarking](benchmarking.md). This tutorial focuses on using SWE-agent on+ [batch mode](batch_mode.md). This tutorial focuses on using SWE-agent onindividual issues.We've already seen a few examples of how to specify the problem to solve, namelydiff --git a/docs/usage/inspector.md b/docs/usage/inspector.mdindex fe1a4d67..787d7eb6 100644--- a/docs/usage/inspector.md+++ b/docs/usage/inspector.md@@ -37,7 +37,7 @@ The inspector will then be launched in the browser:## Benchmark results-If you are running SWE-agent on a benchmark (see [benchmarking](benchmarking.md)), you will see evaluation results as β or β.+If you are running SWE-agent on a benchmark (see [batch mode](batch_mode.md)), you will see evaluation results as β or β.Otherwise, you will see β.!!! tipdiff --git a/docs/usage/trajectories.md b/docs/usage/trajectories.mdindex 6f6369d4..0b8683c2 100644--- a/docs/usage/trajectories.md+++ b/docs/usage/trajectories.md@@ -85,8 +85,8 @@ Here are a few more files that are generated with `sweagent run-batch`:!!! tip- * Evaluation is not completed by `sweagent run-batch`, it is a separate step (see [benchmarking](benchmarking.md))- * `preds.json` can be referenced directly into SWE-bench to run evaluation (see [benchmarking](benchmarking.md))+ * Evaluation is not completed by `sweagent run-batch`, it is a separate step (see [batch mode](batch_mode.md))+ * `preds.json` can be referenced directly into SWE-bench to run evaluation* Trajectories can be turned into custom demonstrations for SWE-agent ([more information](../config/demonstrations.md)).{% include-markdown "../_footer.md" %}