SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]
Doc: Competitive runs (#1067)
Kilian Lieretb85a6201bfcd
diff --git a/docs/index.md b/docs/index.mdindex b9c34418..141409f3 100644--- a/docs/index.md+++ b/docs/index.md@@ -67,4 +67,5 @@ SWE-agent uses [SWE-ReX](https://swe-rex.com/) for sandboxed code execution.## ✍️ Doc updates+* Apr 8: [Running SWE-agent competitively](usage/competitive_runs.md)* Mar 7: [Updated SWE-agent architecture diagram of 1.0](background/architecture.md)\ No newline at end of filediff --git a/docs/usage/batch_mode.md b/docs/usage/batch_mode.mdindex de4d4df1..ea948f2f 100644--- a/docs/usage/batch_mode.md+++ b/docs/usage/batch_mode.md@@ -193,3 +193,8 @@ data = [{"instance_id": key, **value} for key, value in preds.items()]jsonl = [json.dumps(d) for d in data]Path("all_preds.jsonl").write_text("\n".join(jsonl))```++!!! tip "Next up"++ Take a look at our [competitive runs tutorial](competitive_runs.md)+ for more information on running on SWE-Bench and similar benchmarks.\ No newline at end of filediff --git a/docs/usage/competitive_runs.md b/docs/usage/competitive_runs.mdnew file mode 100644index 00000000..b7ee4970--- /dev/null+++ b/docs/usage/competitive_runs.md@@ -0,0 +1,79 @@+# Competitive runs++!!! abstract "Running swe-agent competitively on benchmarks"+ This page contains information on our competitive runs on SWE-bench, as well as tips and tricks for evaluating on large batches.++ * Please make sure you're familiar with [the command line basics](cl_tutorial.md) and the [batch mode](batch_mode.md)+ * The default examples will be executing code in a Docker sandbox, so make sure you have docker installed ([docker troubleshooting](../installation/tips.md)).+++## Current competitive configurations++We recently used two configurations for SWE-bench submissions++* [250225_anthropic_filemap_simple_review.yaml](https://github.com/SWE-agent/SWE-agent/blob/main/config/250225_anthropic_filemap_simple_review.yaml):+ This is our current default one-attempt config. It uses `claude-3-7-sonnet-20250219`.+* [250212_sweagent_heavy_sbl.yaml](https://github.com/SWE-agent/SWE-agent/blob/main/config/250212_sweagent_heavy_sbl.yaml):+ This config runs 5 attempts with slightly different configurations using `claude-3-5-sonnet-20241022`,+ then uses o1 to discriminate between them.+ This is a very expensive configuration.+ If you use it, also make sure to use Claude 3.7 instead of claude 3.5.++You can find the command with which to run each config at the top of the config file.++In order to run on multiple workers with Claude, you need to use multiple API keys in order to have enough cache break points+For this, please set the following environment variable before running++```bash+# concatenate your keys+export CLAUDE_API_KEY_ROTATION="KEY1:::KEY2:::KEY3"+```++See our [notes on Claude](../config/models.md) for more details.++## Memory consumption++We run our configuration on a machine with 32GB memory and 8 cores.+To avoid out-of-memory (OOM) situations, we recommend setting++```bash+--instances.deployment.docker_args=--memory=10g+```++limiting the maximum amount of memory per worker.++In our case, this completely avoided any instances of running OOM.++However, OOM situations can potentially lock you out of the server, so+you might want to use a script like the following as a second layer+defense to kill any process that hogs too much memory (note that this will affect _any_ script and not just swe-agent):++<details>+<summary>Memory sentinel</summary>++```python+--8<-- "docs/usage/memory_sentinel.py"+```++</details>++If swe-agent dies or you frequently abort it, you might have leftover docker containers+(they are cleaned up by normal termination of swe-agent but can be left over if it is killed).+You can use a sentinel script like the following to clean them up periodically+(note that this will affect _any_ long running container and not just those from swe-agent):++<details>+<summary>Container sentinel</summary>++```bash+--8<-- "docs/usage/containers_sentinel.sh"+```+</details>++## Tradeoffs between resolution rate and cost++* Running multi-attempt configurations will always be _very_ expensive. Don't use them if cost is of importance.+* The simplest setting to keep cost in check is the per instance cost limit or turn limit.+ Without limiting cost, the average cost will also converge to infinity, as the agent will never stop iterating.+ With Claude 3.7, a cost-conservative limit would be $1 instance limit or lower and a turn count limit of 50.+ For our swe-bench submissions we use slightly higher limits (see the configs above).\ No newline at end of filediff --git a/docs/usage/containers_sentinel.sh b/docs/usage/containers_sentinel.shnew file mode 100644index 00000000..1bd9d21e--- /dev/null+++ b/docs/usage/containers_sentinel.sh@@ -0,0 +1,22 @@+#!/bin/bash++while true; do+ echo "Checking for long-running containers..."+ # List all running containers with their uptime+ docker ps --format "{{.ID}} {{.RunningFor}}" | while read -r id running_for; do+ # Extract the number and unit from the running time+ if [[ $running_for =~ ([0-9]+)\ (hour|hours) ]]; then+ hours=${BASH_REMATCH[1]}+ if (( hours >= 2 )); then+ echo "Killing container $id (running for $running_for)..."+ docker kill "$id"+ fi+ elif [[ $running_for =~ ([0-9]+)\ (day|days) ]]; then+ # If it's running for at least a day, it's definitely over 2 hours+ echo "Killing container $id (running for $running_for)..."+ docker kill "$id"+ fi+ done+ echo "Sleeping for 10 minutes..."+ sleep 600 # Wait 600 seconds (10 minutes) before running again+done\ No newline at end of filediff --git a/docs/usage/memory_sentinel.py b/docs/usage/memory_sentinel.pynew file mode 100644index 00000000..aba49bd2--- /dev/null+++ b/docs/usage/memory_sentinel.py@@ -0,0 +1,140 @@+#!/usr/bin/env python3+"""+Memory Sentinel Script++This script monitors the system's RAM usage and kills the process with the highest+memory consumption if available RAM drops below 5GB.++Usage:+ python memory_sentinel.py++The script runs continuously, checking memory usage every second.++Script was generated by Claude 3.7 with the following prompt:++I'm working on a server and I have one script that sometimes consumes so much memory that the entire server becomes unresponsive. This is a hue problem because then I cannot log in to it anymore.++Could you write a sentinel script in python that does the following:++Check the total available RAM of the system (disregarding swap)+Check the currently used RAM of the system+If we have more than 5G left, do nothing+Else, find the process with the highest RAM consumption and kill it. Note: Use the SIGKILL command+Check every second+"""++import logging+import os+import signal+import time++import psutil++# Set up logging+logging.basicConfig(+ level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s", filename="memory_sentinel.log", filemode="a"+)++# Memory threshold in GB+MEMORY_THRESHOLD_GB = 5+# Convert to bytes for comparison+MEMORY_THRESHOLD_BYTES = MEMORY_THRESHOLD_GB * 1024 * 1024 * 1024+++def get_available_ram():+ """Get the available RAM in bytes, excluding swap."""+ return psutil.virtual_memory().available+++def get_total_ram():+ """Get the total RAM in bytes, excluding swap."""+ return psutil.virtual_memory().total+++def get_used_ram():+ """Get the used RAM in bytes."""+ return psutil.virtual_memory().used+++def get_process_with_highest_memory():+ """Find the process with the highest memory consumption."""+ processes = []+ for proc in psutil.process_iter(["pid", "name", "memory_info"]):+ try:+ processes.append((proc.info["pid"], proc.info["name"], proc.info["memory_info"].rss))+ except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):+ pass++ # Sort by memory usage (descending)+ processes.sort(key=lambda x: x[2], reverse=True)++ if processes:+ return processes[0]+ return None+++def kill_process(pid):+ """Kill a process using SIGKILL."""+ try:+ os.kill(pid, signal.SIGKILL)+ return True+ except OSError as e:+ logging.error(f"Failed to kill process {pid}: {e}")+ return False+++def format_bytes(bytes_value):+ """Format bytes to a human-readable string."""+ for unit in ["B", "KB", "MB", "GB", "TB"]:+ if bytes_value < 1024.0:+ return f"{bytes_value:.2f} {unit}"+ bytes_value /= 1024.0+ return f"{bytes_value:.2f} PB"+++def main():+ """Main function that runs the memory monitoring loop."""+ logging.info("Memory Sentinel started")+ logging.info(f"Memory threshold set to {MEMORY_THRESHOLD_GB}GB")++ try:+ while True:+ available_ram = get_available_ram()+ total_ram = get_total_ram()+ used_ram = get_used_ram()++ logging.debug(+ f"Total RAM: {format_bytes(total_ram)}, "+ + f"Used RAM: {format_bytes(used_ram)}, "+ + f"Available RAM: {format_bytes(available_ram)}"+ )++ if available_ram < MEMORY_THRESHOLD_BYTES:+ logging.warning(+ f"Available RAM ({format_bytes(available_ram)}) " + f"below threshold of {MEMORY_THRESHOLD_GB}GB"+ )++ process = get_process_with_highest_memory()+ if process:+ pid, name, memory = process+ logging.warning(f"Killing process {pid} ({name}) " + f"using {format_bytes(memory)}")++ if kill_process(pid):+ logging.info(f"Successfully killed process {pid} ({name})")+ else:+ logging.error(f"Failed to kill process {pid} ({name})")+ else:+ logging.warning("No process found to kill")++ # Sleep for 1 second+ time.sleep(1)++ except KeyboardInterrupt:+ logging.info("Memory Sentinel stopped by user")+ except Exception as e:+ logging.error(f"Unexpected error: {e}")+ raise+++if __name__ == "__main__":+ main()diff --git a/mkdocs.yml b/mkdocs.ymlindex f86eaab9..ebd9c1ed 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -71,6 +71,7 @@ nav:- "Solving coding challenges": "usage/coding_challenges.md"- "Trajectory inspector": "usage/inspector.md"- "Batch mode": "usage/batch_mode.md"+ - "Competitive runs": "usage/benchmarking.md"- "What's next?": "usage/whats_next.md"- "FAQ": "faq.md"- User Guides: