Skip to content

SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive coding challenges. [NeurIPS 2024]

Change: Set SWEBenchInstances.evaluate=False by default

Kilian Lieret46feb1e50efe
diff --git a/sweagent/run/batch_instances.py b/sweagent/run/batch_instances.py
index 55289a3f..c710a563 100644
--- a/sweagent/run/batch_instances.py
+++ b/sweagent/run/batch_instances.py
@@ -272,7 +272,7 @@ class SWEBenchInstances(BaseModel, AbstractInstanceSource):
shuffle: bool = False
"""Shuffle the instances (before filtering and slicing)."""
- evaluate: bool = True
+ evaluate: bool = False
"""Run sb-cli to evaluate"""
def _get_huggingface_name(self) -> str: